Problem
With version_aware=True, deleting a specific version through rm_file("bucket/dir/key?versionId=...") leaves the cached HeadObject entry of the unversioned path bucket/dir/key. After that, exists() and info() on bucket/dir/key keep returning the cached object:
- If the deleted version was the only one,
exists("bucket/dir/key") stays True.
- If it was the latest version,
info("bucket/dir/key") keeps returning the deleted version's metadata instead of the version that became current.
Expected: after a version is deleted through the filesystem, the next exists()/info() of the object path reflects S3.
From the code on master (775874c):
rm_file() calls invalidate_cache(path) with the version-qualified path (pyathena/filesystem/s3.py, rm_file).
invalidate_cache() pops the path itself and walks _parent(). For bucket/dir/key?versionId=v1, the parent is bucket/dir, so the bucket/dir/key entry that _head_object() stored is never popped.
exists() returns True when _ls_from_cache() finds that entry.
Found by code reading during the review of #928 (not reproduced live yet).
Reproduction
from pyathena import connect
from pyathena.filesystem.s3 import S3FileSystem
fs = S3FileSystem(connect(), version_aware=True, skip_instance_cache=True)
path = "s3://VERSIONED_BUCKET/tmp/key"
fs.pipe(path, b"data")
version_id = fs.info(path)["version_id"] # caches bucket/tmp/key
fs.rm_file(f"{path}?versionId={version_id}")
print(fs.exists(path)) # expected False, returns True from the cache
Environment
- PyAthena master 775874c,
S3FileSystem with version_aware=True.
Proposed fix (optional)
In invalidate_cache() (or in the version-aware delete paths), also invalidate the path without its versionId query. Add a test that deletes the only version of an object and checks exists().
Problem
With
version_aware=True, deleting a specific version throughrm_file("bucket/dir/key?versionId=...")leaves the cached HeadObject entry of the unversioned pathbucket/dir/key. After that,exists()andinfo()onbucket/dir/keykeep returning the cached object:exists("bucket/dir/key")staysTrue.info("bucket/dir/key")keeps returning the deleted version's metadata instead of the version that became current.Expected: after a version is deleted through the filesystem, the next
exists()/info()of the object path reflects S3.From the code on master (775874c):
rm_file()callsinvalidate_cache(path)with the version-qualified path (pyathena/filesystem/s3.py,rm_file).invalidate_cache()pops the path itself and walks_parent(). Forbucket/dir/key?versionId=v1, the parent isbucket/dir, so thebucket/dir/keyentry that_head_object()stored is never popped.exists()returnsTruewhen_ls_from_cache()finds that entry.Found by code reading during the review of #928 (not reproduced live yet).
Reproduction
Environment
S3FileSystemwithversion_aware=True.Proposed fix (optional)
In
invalidate_cache()(or in the version-aware delete paths), also invalidate the path without itsversionIdquery. Add a test that deletes the only version of an object and checksexists().