Skip to content

S3FileSystem.info() does not use cached directory listings #965

Description

@laughingman7743

Problem

S3FileSystem.info() cannot use the directory listings that _ls_dirs() caches.
_ls_dirs() stores listings under (path, delimiter) tuple keys (pyathena/filesystem/s3.py:435, s3.py:472 on master e0e85da), but _ls_from_cache() (s3.py:1925) looks up only the string keys path and self._parent(path).
The only string-keyed listing is the bucket list under "", so for keys below a bucket the documented behavior of info() ("a cached listing of the path itself makes it a directory, and a cached listing of its parent without it means it does not exist") does not apply.

As a result, info(), size(), exists(), and du() send a HeadObject request for an object that a previous ls()/find() already listed with its size and metadata.
With list-only permissions, they fail instead of answering from the listing.

The tuple keys date from v3.15.0 (681e749).

Found by the independent review of #956 (#933).

Reproduction

fs.ls("s3://BUCKET/d")          # caches ("BUCKET/d", "/")
fs.info("s3://BUCKET/d/direct") # sends HeadObject anyway

Environment

Proposed fix (optional)

Let _ls_from_cache() consult the (parent, "/") listing (and the path's own (path, "/") listing for directories), or correct the info() docstring if the listing cache is intentionally not used for lookups.
Check the interaction with version_aware, which re-heads version-less cached file entries.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions