Skip to content

S3FileSystem keeps the cached object after rm_file() deletes a specific version #931

Description

@laughingman7743

Problem

With version_aware=True, deleting a specific version through rm_file("bucket/dir/key?versionId=...") leaves the cached HeadObject entry of the unversioned path bucket/dir/key. After that, exists() and info() on bucket/dir/key keep returning the cached object:

  • If the deleted version was the only one, exists("bucket/dir/key") stays True.
  • If it was the latest version, info("bucket/dir/key") keeps returning the deleted version's metadata instead of the version that became current.

Expected: after a version is deleted through the filesystem, the next exists()/info() of the object path reflects S3.

From the code on master (775874c):

  • rm_file() calls invalidate_cache(path) with the version-qualified path (pyathena/filesystem/s3.py, rm_file).
  • invalidate_cache() pops the path itself and walks _parent(). For bucket/dir/key?versionId=v1, the parent is bucket/dir, so the bucket/dir/key entry that _head_object() stored is never popped.
  • exists() returns True when _ls_from_cache() finds that entry.

Found by code reading during the review of #928 (not reproduced live yet).

Reproduction

from pyathena import connect
from pyathena.filesystem.s3 import S3FileSystem

fs = S3FileSystem(connect(), version_aware=True, skip_instance_cache=True)
path = "s3://VERSIONED_BUCKET/tmp/key"
fs.pipe(path, b"data")
version_id = fs.info(path)["version_id"]  # caches bucket/tmp/key

fs.rm_file(f"{path}?versionId={version_id}")
print(fs.exists(path))  # expected False, returns True from the cache

Environment

  • PyAthena master 775874c, S3FileSystem with version_aware=True.

Proposed fix (optional)

In invalidate_cache() (or in the version-aware delete paths), also invalidate the path without its versionId query. Add a test that deletes the only version of an object and checks exists().

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions