Problem
S3FileSystem.find(path, maxdepth=n) descends one level deeper than fsspec's maxdepth:
maxdepth |
fsspec AbstractFileSystem.find / walk |
PyAthena S3FileSystem.find |
0 |
ValueError("maxdepth must be at least 1") |
entries directly under path |
1 |
entries directly under path |
also the entries one level below |
n |
n levels |
n + 1 levels |
fsspec documents maxdepth as "the maximum number of levels to descend", and its walk() (fsspec 2026.9.0) rejects maxdepth < 1. Code written against fsspec that passes maxdepth=1 gets more files than expected from PyAthena.
From the code on master (775874c): _find() lists the current level with delimiter="/" and recurses into each subdirectory while maxdepth > 0, with maxdepth - 1. test_find_maxdepth in tests/pyathena/filesystem/test_s3.py encodes the current behavior (maxdepth=0 gives the root files, maxdepth=1 adds level1/).
Found by code reading during the review of #928.
Reproduction
fs.touch("s3://BUCKET/tmp/d/direct")
fs.touch("s3://BUCKET/tmp/d/sub/nested")
print(fs.find("s3://BUCKET/tmp/d", maxdepth=1))
# fsspec semantics: [".../d/direct"]
# PyAthena: [".../d/direct", ".../d/sub/nested"]
Environment
- PyAthena master 775874c (behavior since
maxdepth support was added in v3.15.0), fsspec 2026.9.0.
Proposed fix (optional)
Align with fsspec: reject maxdepth < 1 and recurse only while more than one level remains. This changes results for existing callers that rely on the current numbering, so it needs a release note (and a decision on whether it belongs in 4.0.0 only).
Problem
S3FileSystem.find(path, maxdepth=n)descends one level deeper than fsspec'smaxdepth:maxdepthAbstractFileSystem.find/walkS3FileSystem.find0ValueError("maxdepth must be at least 1")path1pathnnlevelsn + 1levelsfsspec documents
maxdepthas "the maximum number of levels to descend", and itswalk()(fsspec 2026.9.0) rejectsmaxdepth < 1. Code written against fsspec that passesmaxdepth=1gets more files than expected from PyAthena.From the code on master (775874c):
_find()lists the current level withdelimiter="/"and recurses into each subdirectory whilemaxdepth > 0, withmaxdepth - 1.test_find_maxdepthintests/pyathena/filesystem/test_s3.pyencodes the current behavior (maxdepth=0gives the root files,maxdepth=1addslevel1/).Found by code reading during the review of #928.
Reproduction
Environment
maxdepthsupport was added in v3.15.0), fsspec 2026.9.0.Proposed fix (optional)
Align with fsspec: reject
maxdepth < 1and recurse only while more than one level remains. This changes results for existing callers that rely on the current numbering, so it needs a release note (and a decision on whether it belongs in 4.0.0 only).