Problem
S3FileSystem.find(path, maxdepth=n) returns [] when path is an object, while find(path) without maxdepth returns [path], as fsspec's find() does (if not out and self.isfile(path)).
The maxdepth branch of _find() lists path + "/" and returns only what it finds; the unlimited branch falls back to info(path) when nothing is listed.
A fallback in the maxdepth branch was tried in #956 and removed: fsspec's async _glob() passes the filename stem as prefix, so a glob with no matches reached the fallback with an empty listing and sent a HeadObject and another ListObjectsV2 request (3 requests instead of 1), and could raise PermissionError where HeadObject is denied.
A fix has to avoid that cost for prefixed listings.
Reproduction
fs.touch("s3://BUCKET/d/file")
fs.find("s3://BUCKET/d/file") # ['BUCKET/d/file']
fs.find("s3://BUCKET/d/file", maxdepth=1) # []
Environment
- PyAthena master e0e85da (since
maxdepth support in v3.15.0).
Proposed fix (optional)
Fall back to the object only when no prefix is given, or align the prefix behavior of both branches, and document it in the find() docstring.
Problem
S3FileSystem.find(path, maxdepth=n)returns[]whenpathis an object, whilefind(path)withoutmaxdepthreturns[path], as fsspec'sfind()does (if not out and self.isfile(path)).The
maxdepthbranch of_find()listspath + "/"and returns only what it finds; the unlimited branch falls back toinfo(path)when nothing is listed.A fallback in the
maxdepthbranch was tried in #956 and removed: fsspec's async_glob()passes the filename stem asprefix, so a glob with no matches reached the fallback with an empty listing and sent a HeadObject and another ListObjectsV2 request (3 requests instead of 1), and could raisePermissionErrorwhere HeadObject is denied.A fix has to avoid that cost for prefixed listings.
Reproduction
Environment
maxdepthsupport in v3.15.0).Proposed fix (optional)
Fall back to the object only when no
prefixis given, or align theprefixbehavior of both branches, and document it in thefind()docstring.