Skip to content

S3FileSystem.find(withdirs=True) omits the root directory #963

Description

@laughingman7743

Problem

S3FileSystem.find(path, withdirs=True) does not include path itself when it is a directory.
fsspec's AbstractFileSystem.find() adds the root (if withdirs and path != "" and self.isdir(path)), "needed for posix glob compliance".
Both branches of S3FileSystem._find() omit it: the maxdepth branch adds only the listed directories (pyathena/filesystem/s3.py:741 on master e0e85da), and the unlimited branch adds only the parents derived from the listed keys below the path (s3.py:763-765).

As a consequence, glob() patterns that should match the root directory (for example glob("s3://bucket/dir/**"), which fsspec matches against find(..., withdirs=True)) miss it.

Found by the independent review of #956 (#933).

Reproduction

from fsspec.implementations.memory import MemoryFileSystem
mfs = MemoryFileSystem()
mfs.pipe("/d/direct", b"")
mfs.pipe("/d/sub/nested", b"")
mfs.find("/d", maxdepth=1, withdirs=True)
# ['/d', '/d/direct', '/d/sub']

fs.find("s3://BUCKET/d", maxdepth=1, withdirs=True)
# PyAthena: ['BUCKET/d/sub', 'BUCKET/d/direct']  (no 'BUCKET/d')

mfs.glob("/d/**")
# ['/d', '/d/direct', '/d/sub', '/d/sub/nested']
fs.glob("s3://BUCKET/d/**")
# PyAthena: no 'BUCKET/d'

Environment

  • PyAthena master e0e85da, fsspec 2026.9.0.

Proposed fix (optional)

Include the root directory when withdirs is true and the path is a directory, in both branches, and check the effect on glob(), expand_path(), rm(), and copy(), which call find(..., withdirs=True).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions