Skip to content

S3FileSystem.find(object_path, maxdepth=n) returns an empty list #966

Description

@laughingman7743

Problem

S3FileSystem.find(path, maxdepth=n) returns [] when path is an object, while find(path) without maxdepth returns [path], as fsspec's find() does (if not out and self.isfile(path)).
The maxdepth branch of _find() lists path + "/" and returns only what it finds; the unlimited branch falls back to info(path) when nothing is listed.

A fallback in the maxdepth branch was tried in #956 and removed: fsspec's async _glob() passes the filename stem as prefix, so a glob with no matches reached the fallback with an empty listing and sent a HeadObject and another ListObjectsV2 request (3 requests instead of 1), and could raise PermissionError where HeadObject is denied.
A fix has to avoid that cost for prefixed listings.

Reproduction

fs.touch("s3://BUCKET/d/file")
fs.find("s3://BUCKET/d/file")              # ['BUCKET/d/file']
fs.find("s3://BUCKET/d/file", maxdepth=1)  # []

Environment

  • PyAthena master e0e85da (since maxdepth support in v3.15.0).

Proposed fix (optional)

Fall back to the object only when no prefix is given, or align the prefix behavior of both branches, and document it in the find() docstring.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions