Skip to content

S3FileSystem.find(maxdepth=..., prefix=...) ignores the depth limit when prefix contains a slash #964

Description

@laughingman7743

Problem

S3FileSystem.find(path, maxdepth=n, prefix=...) does not limit the depth when prefix contains /.
With maxdepth, _find() lists Prefix=f"{key}/{prefix}" with Delimiter="/" (pyathena/filesystem/s3.py:733 on master e0e85da) and treats the returned entries as the first level below path.
S3 groups by the delimiter only after the full prefix, so a prefix such as sub/deep/ makes the "first level" entries of path actually lie two levels down, and every further level is shifted the same way.

fsspec's glob() passes only a slash-free filename stem as prefix, so this affects direct find(..., prefix=...) callers.

Found by the independent review of #956 (#933).

Reproduction

fs.touch("s3://BUCKET/d/sub/deep/x")
fs.find("s3://BUCKET/d", maxdepth=1, prefix="sub/deep/")
# expected: [] (nothing directly under d/ starts with "sub/deep/")
# actual: ['BUCKET/d/sub/deep/x']

Environment

  • PyAthena master e0e85da (maxdepth and prefix together since v3.15.0).

Proposed fix (optional)

Decide what prefix with a / should mean together with maxdepth (reject it, or count the levels from path), and document it in the find() docstring.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions