Objects whose keys end in '/' are listed as files but cannot be opened or read with negative offsets

[laughingman7743]

Problem

An object whose key ends in / and has content (for example dir/ with body abc) cannot be read through info()-based paths. fsspec's _strip_protocol() removes the trailing / (s3://bucket/dir/ and s3://bucket/dir both become bucket/dir). info() then heads dir, finds no object, and reports a directory of size 0, so:

• info("s3://bucket/dir/") returns type="directory", size=0, and isfile() returns False.
• open("s3://bucket/dir/", "rb").read() returns b"" on master. With Clamp byte ranges to the object in cat_file() and S3File reads #987 (S3File reads and cat_file() return the whole object for empty or out-of-bounds ranges #970), it raises FileNotFoundError, because opening a prefix is now rejected.
• cat_file("s3://bucket/dir/", start=-2) returns the whole object b"abc" on master. Negative offsets are resolved against the size of 0, which produces a Range that S3 ignores. With Clamp byte ranges to the object in cat_file() and S3File reads #987, it raises FileNotFoundError.
• cat_file() without a range, or with non-negative offsets, uses the exact key and reads the object (b"abc", b"a" for [0:1]).
• ls("s3://bucket/dir/", detail=True) lists the object as a file of size 3 named bucket/dir/, but that name cannot be passed back to info() or open().

Most keys ending in / are zero-byte folder markers (for example, from the S3 console's "Create folder"), and treating them as directories is the usual interpretation. The problem is limited to such keys that hold data, and to the inconsistency between ls() and info()/open().

Expected: decide whether a key ending in / is addressable as a file. If it is, info(), open() and negative offsets reach it by its exact key. If it is not, ls() should not return it as a file entry, and the limitation should be documented.

Reproduction

Checked on live S3 with three objects: p/only/ (b"abc"), p/slash/ (b"abc") and p/slash/child (b"x").

from pyathena import connect
from pyathena.filesystem.s3 import S3FileSystem

fs = S3FileSystem(connect(), skip_instance_cache=True)
p = "s3://<bucket>/p/slash/"
print(fs.info(p)["type"], fs.info(p)["size"])  # directory 0
print(fs.isfile(p))                            # False
print(fs.cat_file(p))                          # b'abc'
print(fs.cat_file(p, start=0, end=1))          # b'a'
print(fs.cat_file(p, start=-2))                # master: b'abc'; #987: FileNotFoundError
with fs.open(p, "rb") as f:                    # #987: FileNotFoundError
    print(f.read())                            # master: b''
print([(e["name"], e["type"], e["size"]) for e in fs.ls(p, detail=True)])
# [('<bucket>/p/slash/', 'file', 3), ('<bucket>/p/slash/child', 'file', 1)]

p/only/ (no children) gives the same results.

s3fs 2026.9.0 on the same objects: info() reports a directory of size 0, isfile() is False, open().read() returns b"", and ls() returns the same file entry. cat_file(p, start=-2) returns b"bc", because s3fs sends a suffix range (bytes=-2) without looking up the size.

Environment

• PyAthena master aa0fc914 and PR Clamp byte ranges to the object in cat_file() and S3File reads #987 head 02f8ebdd, fsspec 2026.9.0, s3fs 2026.9.0 for comparison, Python 3.13.1.
• Found while reviewing Clamp byte ranges to the object in cat_file() and S3File reads #987 (S3File reads and cat_file() return the whole object for empty or out-of-bounds ranges #970).

Proposed fix (optional)

This needs a maintainer decision before implementation:

• Keep keys ending in / as directories (fsspec path model, as s3fs does for info()/open()). Then exclude such keys from ls() file entries, or document them as unreadable except through cat_file(). Optionally, resolve cat_file(start=-N) with a suffix range (bytes=-N) as s3fs does, which needs no size lookup.
• Make them addressable as files. info(), open() and the directory cache would have to distinguish dir/ from dir, contrary to fsspec's normalization. This touches info, ls, dircache keys, invalidate_cache, S3File paths and pickling, and diverges from s3fs.