Problem
S3FileSystem.find(path, maxdepth=n, prefix=...) does not limit the depth when prefix contains /.
With maxdepth, _find() lists Prefix=f"{key}/{prefix}" with Delimiter="/" (pyathena/filesystem/s3.py:733 on master e0e85da) and treats the returned entries as the first level below path.
S3 groups by the delimiter only after the full prefix, so a prefix such as sub/deep/ makes the "first level" entries of path actually lie two levels down, and every further level is shifted the same way.
fsspec's glob() passes only a slash-free filename stem as prefix, so this affects direct find(..., prefix=...) callers.
Found by the independent review of #956 (#933).
Reproduction
fs.touch("s3://BUCKET/d/sub/deep/x")
fs.find("s3://BUCKET/d", maxdepth=1, prefix="sub/deep/")
# expected: [] (nothing directly under d/ starts with "sub/deep/")
# actual: ['BUCKET/d/sub/deep/x']
Environment
- PyAthena master e0e85da (
maxdepth and prefix together since v3.15.0).
Proposed fix (optional)
Decide what prefix with a / should mean together with maxdepth (reject it, or count the levels from path), and document it in the find() docstring.
Problem
S3FileSystem.find(path, maxdepth=n, prefix=...)does not limit the depth whenprefixcontains/.With
maxdepth,_find()listsPrefix=f"{key}/{prefix}"withDelimiter="/"(pyathena/filesystem/s3.py:733on master e0e85da) and treats the returned entries as the first level belowpath.S3 groups by the delimiter only after the full prefix, so a prefix such as
sub/deep/makes the "first level" entries ofpathactually lie two levels down, and every further level is shifted the same way.fsspec's
glob()passes only a slash-free filename stem asprefix, so this affects directfind(..., prefix=...)callers.Found by the independent review of #956 (#933).
Reproduction
Environment
maxdepthandprefixtogether since v3.15.0).Proposed fix (optional)
Decide what
prefixwith a/should mean together withmaxdepth(reject it, or count the levels frompath), and document it in thefind()docstring.