Bucket lookups: info() misses buckets absent from ls(''), invalidate_cache() keeps the root listing, exists() raises on 403

[laughingman7743]

Problem

Bucket-level lookups do not agree with each other when the bucket is missing from the cached bucket list or when HeadBucket is denied.

1. info() and isdir() of a bucket that ls("") did not return raise FileNotFoundError. ls("") caches the buckets owned by the caller under the "" key. info() then answers every bucket path from _ls_from_cache(), which treats the cached parent listing as complete and raises FileNotFoundError for any bucket not in it, without sending HeadBucket (pyathena/filesystem/s3.py:1937-1950, called from :620-622). Buckets of other accounts that the caller can access are never in that list. exists() already falls back to HeadBucket and returns True for the same bucket (s3.py:845-854), so exists() and info()/isdir() disagree.
2. invalidate_cache(""), invalidate_cache("s3://") and invalidate_cache("/") keep the bucket listing. invalidate_cache() walks up with while path: and stops before the root key "" (s3.py:1916-1923), so the cached bucket list is never removed by a path-based call. mkdir() and rmdir() pop dircache[""] directly to work around this (s3.py:1030, :1103). A caller who creates a bucket outside this instance cannot refresh the bucket list with invalidate_cache("s3://"), and item 1 then reports the new bucket as missing.
3. exists("s3://bucket") raises PermissionError when HeadBucket returns 403. _head_bucket() only catches FileNotFoundError (s3.py:306-319), and exists() calls it without handling other errors (s3.py:845-854). HeadBucket returns 403 for a bucket that exists but that the caller may not head, so exists() raises instead of returning a boolean. mkdir() calls exists(bucket) first (s3.py:998), so makedirs("s3://bucket/prefix", exist_ok=True) and mkdir("s3://bucket/prefix") also raise PermissionError for such a bucket, although they send no request for a key prefix.

Expected:

• info() and isdir() send HeadBucket for a bucket that is not in the cached bucket list, as exists() does;
• invalidate_cache() of the root removes the cached bucket list;
• exists() returns a boolean when HeadBucket is denied (the bucket exists when S3 answers 403), and makedirs(..., exist_ok=True) succeeds for a prefix in such a bucket.

Fix together: #965 (info() does not use cached listings) changes the same info()/_ls_from_cache() lookup. PR #960 rewrites the loop in invalidate_cache(), so whichever lands second needs a rebase.

Reproduction

from unittest.mock import MagicMock
import botocore.exceptions
from pyathena.filesystem.s3 import S3FileSystem
from pyathena.filesystem.s3_errors import S3ClientError

calls = []
fs = S3FileSystem(connection=MagicMock(), skip_instance_cache=True)
def call(method, **request):
    name = method._extract_mock_name().split(".")[-1]
    calls.append(name)
    return {"Buckets": [{"Name": "mine"}]} if name == "list_buckets" else {}
fs._call = call

print(fs.ls("s3://"))
# ['mine']
try:
    fs.info("s3://other-account-bucket")
except FileNotFoundError as e:
    print("info raised FileNotFoundError", e)
# info raised FileNotFoundError other-account-bucket
print(fs.isdir("s3://other-account-bucket"), calls)
# False ['list_buckets']
print(fs.exists("s3://other-account-bucket"), calls)
# True ['list_buckets', 'head_bucket']

for path in ["", "s3://", "/"]:
    fs.invalidate_cache(path)
    print(repr(path), "" in fs.dircache)
# '' True
# 's3://' True
# '/' True

def denied(method, **request):
    # The mapping that S3FileSystem._call applies to a 403 from HeadBucket.
    error = botocore.exceptions.ClientError(
        {"Error": {"Code": "403", "Message": ""}, "ResponseMetadata": {"HTTPStatusCode": 403}},
        "HeadBucket",
    )
    raise S3ClientError(error).os_error
fs = S3FileSystem(connection=MagicMock(), skip_instance_cache=True)
fs._call = denied
for label, func in [
    ("exists", lambda: fs.exists("s3://not-my-bucket")),
    ("makedirs", lambda: fs.makedirs("s3://not-my-bucket/prefix", exist_ok=True)),
]:
    try:
        func()
    except PermissionError as e:
        print(label, "raised", type(e).__name__)
# exists raised PermissionError
# makedirs raised PermissionError

Environment

• PyAthena master e0e85da, fsspec 2026.9.0, botocore from uv.lock, Python 3.13.1.
• Found in a full audit of pyathena/filesystem/. Unless noted, checked offline with mocked S3 responses (fs._call replaced) or botocore's Stubber with dummy credentials.

Proposed fix (optional)

• In _ls_from_cache() (or info()), do not treat a missing entry in the root bucket list as proof that the bucket does not exist; fall through to HeadBucket.
• Let invalidate_cache() pop "" when called with the root path ("", "/", "s3://").
• In exists() for a bucket path, treat PermissionError from HeadBucket as existing.