Skip to content

S3FileSystem.find(maxdepth=...) descends one level deeper than fsspec #933

Description

@laughingman7743

Problem

S3FileSystem.find(path, maxdepth=n) descends one level deeper than fsspec's maxdepth:

maxdepth fsspec AbstractFileSystem.find / walk PyAthena S3FileSystem.find
0 ValueError("maxdepth must be at least 1") entries directly under path
1 entries directly under path also the entries one level below
n n levels n + 1 levels

fsspec documents maxdepth as "the maximum number of levels to descend", and its walk() (fsspec 2026.9.0) rejects maxdepth < 1. Code written against fsspec that passes maxdepth=1 gets more files than expected from PyAthena.

From the code on master (775874c): _find() lists the current level with delimiter="/" and recurses into each subdirectory while maxdepth > 0, with maxdepth - 1. test_find_maxdepth in tests/pyathena/filesystem/test_s3.py encodes the current behavior (maxdepth=0 gives the root files, maxdepth=1 adds level1/).

Found by code reading during the review of #928.

Reproduction

fs.touch("s3://BUCKET/tmp/d/direct")
fs.touch("s3://BUCKET/tmp/d/sub/nested")
print(fs.find("s3://BUCKET/tmp/d", maxdepth=1))
# fsspec semantics: [".../d/direct"]
# PyAthena: [".../d/direct", ".../d/sub/nested"]

Environment

  • PyAthena master 775874c (behavior since maxdepth support was added in v3.15.0), fsspec 2026.9.0.

Proposed fix (optional)

Align with fsspec: reject maxdepth < 1 and recurse only while more than one level remains. This changes results for existing callers that rely on the current numbering, so it needs a release note (and a decision on whether it belongs in 4.0.0 only).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions