Make S3Object follow the mapping contract

[laughingman7743]

WHAT

Make S3Object, the entry type of info() and ls(detail=True), follow the mapping contract.

• obj["missing"] raises KeyError, so in, get(key, default), pop(key, default) and setdefault() work like a dictionary.
• Attribute access still returns None for a known S3 object field that the entry does not have, such as content_type, version_id or is_latest.
  The known fields are the property names of the mapped S3 API fields plus name, type, bucket, key, size, version_id and is_latest.
  Any other missing name, including dunder names, raises AttributeError.
• Add S3Object.copy(), which returns a shallow S3Object copy.

Behavior change for the release notes: entry["missing"] now raises KeyError instead of returning None, "missing" in entry is False, and entry.get(key, default) returns default for a missing key.
Attribute access to an unknown name, such as entry.foo, now raises AttributeError instead of returning None.

WHY

Closes #982.

__getitem__() returned None for every missing key and __getattr__() returned None for every missing attribute.
As a result, in was always True, get() ignored its default, and copy.copy(), copy.deepcopy() and pickle failed with TypeError: 'NoneType' object is not callable because __setstate__ resolved to None.
fsspec's DirFileSystem and GenericFileSystem call .copy() on each entry, so info() and ls(detail=True) failed through them with the same TypeError.

Known fields keep returning None through attribute access, as chosen by the maintainer.
Directory entries have no is_latest and ls() file entries have no version_id, so a strict AttributeError would break existing code such as [f.is_latest for f in fs.ls(path, detail=True, versions=True)], which test_ls_versions already relies on for directory entries.
pyathena/ reads optional fields only through .get(); its attribute and subscript reads only touch fields that every entry has.

TEST

Tested commit: 326d2c0 (rebased onto 4950996; first tested on 2b45c15).

• just lint: passed.
• uv run --env-file .env pytest -n 2 -p no:cacheprovider -q tests/pyathena/filesystem/: 396 passed on 326d2c0 (282 on 2b45c15), including the live S3 tests.
• The new tests (TestS3Object::test_mapping, test_attribute, test_copy[copy|deepcopy|pickle|copy_method] and TestS3FileSystem::test_dir_filesystem) are offline.
  All 7 fail with the previous s3_object.py and pass with this change.
• Not covered by a dedicated test: GenericFileSystem and AioS3FileSystem under DirFileSystem.
  Both reach the same S3Object.copy(), because AioS3FileSystem builds its entries through the synchronous filesystem.
  Checked manually with mocked _call: DirFileSystem over AioS3FileSystem (info()/ls(detail=True), and _info()/_ls() with asynchronous=True) and GenericFileSystem(default_method="current") (info()/ls(detail=True)) work with this change and raise TypeError on the base.
• copy.copy, copy.deepcopy, pickle (protocols 0 and default) and copy() checked manually on Python 3.11 and 3.14; pull request CI runs the newest Python only.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): CLEAN

Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 2b45c152535c0eb36fef3802adda9e341bdafa89. Scope: git diff aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe..2b45c152535c0eb36fef3802adda9e341bdafa89 (3 files).

Covered:

• Behavior and failure paths:
  • Every attribute or subscript read of an entry under pyathena/ (s3.py:526, 639-640, 706-708, 801, 846-847, 2028-2031; s3_async.py:339-340) touches name, type or key, which every construction path sets.
  • Optional fields are read through .get() (s3.py:648-649, the copy and checksum paths), so the new KeyError and get() defaults do not change their results.
  • if cache: in info() uses __len__, which is unchanged.
  • No module outside pyathena/filesystem/ reads entries.
• Copy and pickle: copy.copy, copy.deepcopy, pickle protocols 0 and default, and copy() all round-trip on Python 3.11 (the floor) and 3.14. __setstate__ now raises AttributeError, so they fall back to __dict__.update().
• Framework contracts: checked manually against fsspec 2026.9.0 with mocked _call:
  • DirFileSystem over AioS3FileSystem works for info() and ls(detail=True), both synchronously and with asynchronous=True (_info/_ls).
  • GenericFileSystem(default_method="current") works for info() and ls(detail=True).
  • With the base s3_object.py, all of these raise TypeError.
• Test quality: all 7 new tests fail on the base. test_dir_filesystem also asserts that the cached entries keep their full names, which would fail if copy() returned self.

Note: _ls_dirs caches listings under (path, delimiter) (documented at s3.py:420-421), and info() looks up dircache[path] and its parent. So DirFileSystem.info() after ls() sends a HeadObject, and test_dir_filesystem mocks that response instead of relying on the listing cache.

[laughingman7743]

Rebase onto master (no code change)

• Old: base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 2b45c152535c0eb36fef3802adda9e341bdafa89.
• New: base 49509962cb3e81f819c59985d9f6e2ece576ffe0, head 326d2c0821d6a9b678c9caa5df63a00730114fd8.
• The PR conflicted after Fix rm() bulk deletes in S3FileSystem and AioS3FileSystem #986 and Check the part limit in AioS3FileSystem transaction writes #999 merged. The only conflict was the import block in tests/pyathena/filesystem/test_s3.py, where master added from fsspec.dircache import DirCache; both imports are kept.
• git range-diff shows that the patch is unchanged apart from that context line.
• Upstream contract check over aa0fc914..55af09a2, which covers Keep multipart uploads within the 10,000-part limit #968, Fix rm() bulk deletes in S3FileSystem and AioS3FileSystem #986, Align AioS3FileSystem with S3FileSystem in transactions, open() and touch() #988, Read a negative cat_file() start without an end as a suffix range #998 and Check the part limit in AioS3FileSystem transaction writes #999:
  • No new S3Object construction and no new fields.
  • New entry reads use .get("type"), .get("size", 0) and info.key. key is set on every entry (to None for buckets), so the strict __getitem__ and __getattr__ do not affect them.
  • Read a negative cat_file() start without an end as a suffix range #998 (55af09a2) merged after the rebase. It does not conflict, and pull_request CI tests the merge ref.
• Validation on 326d2c08:
  • just lint passed.
  • uv run --env-file .env pytest -n 2 -p no:cacheprovider -q tests/pyathena/filesystem/ gave 396 passed, including the live S3 tests.

[laughingman7743]

Self-review round two (claims and operational behavior): FINDINGS (PR description only, corrected)

Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 2b45c152535c0eb36fef3802adda9e341bdafa89. This is a full claim pass over the PR body, the commit message, and the changed docstrings and comments.

Claims checked:

• "Known fields = mapped property names + name/type/bucket/key/size/version_id/is_latest": matches _S3_OBJECT_FIELDS (s3_object.py:37-48). These are the only kwargs that s3.py passes to S3Object (type, bucket, key, version_id, is_latest), plus the name and size that __init__ sets.
• "Dunder names raise AttributeError": no dunder name is in the set. hasattr(o, "__setstate__") is now False (asserted in test_attribute).
• "setstate resolved to None": on the base, the copy and pickle failures are TypeError: 'NoneType' object is not callable, which matches the __setstate__ call in copy._reconstruct and the unpickler.
• "DirFileSystem/GenericFileSystem call .copy() on each entry": in fsspec 2026.9.0, implementations/dirfs.py:105, 307, 313, 322, 334 and generic.py:202, 216, 231.
• "pyathena/ reads optional fields only through .get()": re-grepped at this head. The attribute and subscript reads are name, type and key only.
• "AioS3FileSystem builds its entries through the synchronous filesystem": s3_async.py constructs no S3Object; _info/_ls delegate to _sync_fs.
• Corrected: the PR body said that f.is_latest on ls(versions=True) entries is "as documented in docs/filesystem.md". The version.is_latest loop at docs/filesystem.md:155-157 iterates object_version_info() (S3ObjectVersion), not ls() entries. The reader that actually exists is test_ls_versions (tests/pyathena/filesystem/test_s3.py:810), and the PR body now cites that test.
• Evidence scope: the PR body now separates manual checks (AioS3FileSystem/GenericFileSystem under fsspec wrappers; copy/pickle on 3.11 and 3.14) from automated tests, and states that PR CI runs the newest Python only.

Existing caller: code that relied on entry["missing"] is None, on "x" in entry being always True, or on entry.unknown is None now gets KeyError, False or AttributeError. The PR body lists these as release-note behavior changes. getattr(entry, name, default) and entry.get(name) keep working.

Documentation reader: docs/api/filesystem.rst:49 autodocs S3Object, so the updated class docstring and copy() are rendered there. No prose in docs/ describes the old None behavior.

AWS operator: no request changes. test_dir_filesystem is offline (2 mocked calls).

[laughingman7743]

Independent review (relayed): CLEAN

• Reviewer: Codex CLI 0.160.0, model gpt-6-astra, reasoning effort max, --sandbox read-only --ephemeral, session 01a100d0-b30d-7ef1-8b69-d5aca3e0e61a.
• Scope: base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 2b45c152535c0eb36fef3802adda9e341bdafa89, in a clean detached snapshot at the head.
• Package: the literal diff, the intended behavior, and repository conventions, plus read-only access to the installed fsspec 2026.9.0 sources. It did not include the PR number, the PR description, the commit message, or the self-review findings.
• Constraints: review only; no edits, builds, tests, network or GitHub access.
• Both the snapshot and the PR worktree were unchanged afterwards (git status clean, HEAD 2b45c152).

This is a static review: the reviewer executed nothing.

Reviewer output (verbatim):

Covered:

• S3Object item/attribute access, known-field completeness, mutation overrides, Mapping mixins, equality, and truthiness.
• copy(), shallow/deep copying, and pickle reconstruction, including interaction with __setattr__ and missing protocol attributes on Python ≥3.11.
• All entry consumers found under pyathena/, including synchronous/asynchronous listing, caching, version handling, file reads/appends, copying, checksums, and timestamps.
• Installed fsspec 2026.9.0 contracts in implementations/dirfs.py, generic.py, spec.py, and asyn.py, including copy-and-rename cache isolation.
• New tests’ observable assertions and expected failures against the old implementation; changed docstrings, import placement, and helper naming.

CLEAN

No actionable defects introduced by this diff found. No caller requires an omitted optional attribute field. The new tests exercise the previous failures and check that renaming copied entries preserves cached names.

Static review only; no tests, builds, or network access. Checkout remains clean at 2b45c152535c0eb36fef3802adda9e341bdafa89.