Long directory list is truncated

[manning-ncsa]

One of the paths in my bucket has 10170 subdirectories. The rendered web page for this path seems to arbitrarily truncate the directory listing to 637 items with no indication that there are missing items. I am running the docker image ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest-20240113. Is this expected behavior? Is there a config parameter I can adjust that is not listed in this guide?

[4141done]

Hi @manning-ncsa , thanks for bringing this up. It does seem like an arbitrary limit that could benefit from having an explicit sensible default and the ability to specify a higher limit. I've not dug into the listing code too much yet so let me poke around and get back to you once I understand the considerations better. If you've found a workaround yourself we welcome pull requests 🐳

[manning-ncsa]

Have you had a chance to look at the listing functions? If you are familiar with the code perhaps you could point me toward the relevant file(s).

[4141done]

Hey sorry for the delay. I did take a look in to this and confirmed with the original maintainer that we do not intentionally limit the directory listing in any way.

He documented it well in this issue #150.

So the default limit of the AWS listing APIs is 1000 so I'm not exactly sure why you're only seeing 637 - you may want to try calling that endpoint directly with a tool like the AWS cli to verify what AWS returns directly for the prefix you've specified.

In terms of a solution, AWS returns an isTruncated boolean and if that is true there should be a continuationToken that we can append via the directory listing template to allow for simple pagination.

I'm currently looking at a fix for #188 so I won't be able to jump on this work immediately but would welcome a PR.

The primary files to look at for this functionality are the xslt template: https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/etc/nginx/include/listing.xsl

And the two sections that in the configuration that proxy to the directory listing.

nginx-s3-gateway/common/etc/nginx/templates/default.conf.template

Line 151 in 626d29d

location @s3PreListing {

nginx-s3-gateway/common/etc/nginx/templates/default.conf.template

Line 209 in 626d29d

location @s3Directory {

In the javascript this is where the initial directory listing query string is generated

nginx-s3-gateway/common/etc/nginx/include/s3gateway.js

Line 317 in 626d29d

function _s3DirQueryParams(uriPath, method) {

If we added pagination, I think we could just build it in without the need for any additional configuration values. I may be possible to just add a link to the xslt template with the continuationToken added but there could be complications.