Problem
S3FileSystem.info() cannot use the directory listings that _ls_dirs() caches.
_ls_dirs() stores listings under (path, delimiter) tuple keys (pyathena/filesystem/s3.py:435, s3.py:472 on master e0e85da), but _ls_from_cache() (s3.py:1925) looks up only the string keys path and self._parent(path).
The only string-keyed listing is the bucket list under "", so for keys below a bucket the documented behavior of info() ("a cached listing of the path itself makes it a directory, and a cached listing of its parent without it means it does not exist") does not apply.
As a result, info(), size(), exists(), and du() send a HeadObject request for an object that a previous ls()/find() already listed with its size and metadata.
With list-only permissions, they fail instead of answering from the listing.
The tuple keys date from v3.15.0 (681e749).
Found by the independent review of #956 (#933).
Reproduction
fs.ls("s3://BUCKET/d") # caches ("BUCKET/d", "/")
fs.info("s3://BUCKET/d/direct") # sends HeadObject anyway
Environment
Proposed fix (optional)
Let _ls_from_cache() consult the (parent, "/") listing (and the path's own (path, "/") listing for directories), or correct the info() docstring if the listing cache is intentionally not used for lookups.
Check the interaction with version_aware, which re-heads version-less cached file entries.
Problem
S3FileSystem.info()cannot use the directory listings that_ls_dirs()caches._ls_dirs()stores listings under(path, delimiter)tuple keys (pyathena/filesystem/s3.py:435,s3.py:472on master e0e85da), but_ls_from_cache()(s3.py:1925) looks up only the string keyspathandself._parent(path).The only string-keyed listing is the bucket list under
"", so for keys below a bucket the documented behavior ofinfo()("a cached listing of the path itself makes it a directory, and a cached listing of its parent without it means it does not exist") does not apply.As a result,
info(),size(),exists(), anddu()send a HeadObject request for an object that a previousls()/find()already listed with its size and metadata.With list-only permissions, they fail instead of answering from the listing.
The tuple keys date from v3.15.0 (681e749).
Found by the independent review of #956 (#933).
Reproduction
Environment
Proposed fix (optional)
Let
_ls_from_cache()consult the(parent, "/")listing (and the path's own(path, "/")listing for directories), or correct theinfo()docstring if the listing cache is intentionally not used for lookups.Check the interaction with
version_aware, which re-heads version-less cached file entries.