pipe_file() rejects compression for small values and uploads an empty object when a compressed write fails

[laughingman7743]

Problem

S3FileSystem.pipe_file() and AioS3FileSystem.pipe_file() handle fsspec's compression argument differently by path:

• Single-request path (data up to the block size, outside a transaction): compression is passed to PutObject, which botocore rejects with ParamValidationError: Unknown parameter in input: "compression".
• Buffered path (in a transaction, or data larger than the block size): compression is passed to open(), which returns a compression wrapper (e.g. GzipFile) around the S3File. A successful write uploads the compressed data. A failed write calls _close_without_commit() on the wrapper, which raises AttributeError: 'GzipFile' object has no attribute '_close_without_commit'. The original error is hidden, and a 10-byte object holding only the gzip header is still uploaded, in and outside a transaction, so the guarantee from Write pipe_file() data without committing a failed write #1003 (a failed write leaves the existing object unchanged) does not hold.

Checked offline on master fe21250 with mocked requests, and with botocore parameter validation for the single-request path.

This also blocks #1034: the write-and-close helper cannot move to S3File while open() may return a compression wrapper (found in review of PR #1035).

Proposed fix

When compression is given, pipe_file() compresses the value in memory with the codec that open() would use (fsspec.core.get_compression() + fsspec.compression.compr, including "infer" from the path), and uploads the compressed bytes through the usual paths without passing compression to open() or S3. The routing and the multipart size check then use the compressed size. The same applies to the transaction path of AioS3FileSystem.

[laughingman7743]

PR #1038 also fixes a related pre-existing defect found in its review, at the maintainer's request: pipe_file() chose between PutObject and the buffered path by len(value), which counts the items of a memoryview, so a typed memoryview (e.g. memoryview(data).cast("I")) larger than the block size in bytes, but not in items, went to PutObject, which fails above 5 GiB. It now routes by the size in bytes.

[laughingman7743]

PR #1038 also makes pipe_file() write a path with a trailing slash to the key without it on both paths, as open() does (maintainer's choice). The single PutObject request used to keep the slash, so the key depended on the size of the value, and compression could switch a call between the two paths (found in the review of #1038).