Multipart uploads with ChecksumAlgorithm cannot complete, and clear_multipart_uploads() fails on finished uploads

[laughingman7743]

Two defects in how the filesystems finish and clean up multipart uploads, found while reviewing #1069. Both are on master 911492c and can be fixed in one PR.

Problem

1. An upload created with ChecksumAlgorithm cannot be completed

A multipart upload created with ChecksumAlgorithm cannot be completed. CompleteMultipartUpload lists each part with only ETag and PartNumber (pyathena/filesystem/s3.py :2334 and s3_async.py :677-679 on master 911492c), but S3 requires the checksum of each part when the upload was created with a checksum algorithm. S3MultipartUploadPart already holds the part checksums that UploadPart returns, and its to_api_repr() builds the entry with them, but nothing uses it.

So fs.open(path, "wb", s3_additional_kwargs={"ChecksumAlgorithm": "SHA256"}) (or put_file()/pipe_file() with it) fails on close for any data larger than the block size, and the upload is aborted. Writes up to the block size use PutObject and are not affected.

2. clear_multipart_uploads() fails on an upload that is already gone

S3FileSystem.clear_multipart_uploads() (and AioS3FileSystem.clear_multipart_uploads(), which calls it) lists the in-progress uploads and then aborts each of them in parallel (pyathena/filesystem/s3.py :2997-3022 on master 911492c). An upload that is completed or aborted between the listing and its abort, for example by the writer that owns it or by another clear_multipart_uploads(), makes AbortMultipartUpload answer NoSuchUpload, which PyAthena raises as FileNotFoundError. future.result() re-raises it, so the whole call fails although that upload is gone, which is what the call wants, and the results of the other aborts are not checked after the first error.

Reproduction

1.

Measured against S3 with one 10-byte part, using the S3Core methods of #1069, which send the same requests as the filesystem's helpers on master 911492c:

upload = core.create_multipart_upload(path, ChecksumAlgorithm="SHA256")
part = core.upload_part(path, upload.upload_id, 1, b"x" * 10, ChecksumAlgorithm="SHA256")
part.checksum_sha256  # '/BHW8o5Z08wzwLFM62RL8JAuvWPWEhjf/p59rHwlRUI='
core.complete_multipart_upload(path, upload.upload_id, [part])
# OSError: [Errno 22] The upload was created using a sha256 checksum. The complete
# request must include the checksum for each part. It was missing for part 1 ...

The same happens with CRC32. Not measured: a multipart copy with ChecksumAlgorithm (CreateMultipartUpload accepts it, so the completion of the copy probably fails the same way), and CRC64NVME, which to_api_repr() does not include.

2.

An abort of an upload that does not exist raises FileNotFoundError (measured against S3 with S3Core.abort_multipart_upload() of #1069, which sends the same request as clear_multipart_uploads() on master 911492c):

core.abort_multipart_upload(path, "nonexistent-upload-id")
# FileNotFoundError: The specified upload does not exist. The upload ID may be
# invalid, or the upload may have been aborted or completed.

The race itself (an upload completed between list_multipart_uploads() and the abort) was not reproduced.

Environment

PyAthena master 911492c, botocore 1.43.102, Python 3.13, S3FileSystem and AioS3FileSystem.

Proposed fix (optional)

1. Send the checksum fields of each part that has them in CompleteMultipartUpload (in S3Core.complete_multipart_upload() once Move the multipart upload requests into S3Core (step 3.2 of #1063) #1069 is merged, which both filesystems use), leaving out the fields that are None, and add ChecksumCRC64NVME. Validate with a Stubber test and a live multipart write with ChecksumAlgorithm.
2. Treat FileNotFoundError from the abort of a listed upload as already cleared, and keep raising other errors. Validate with an offline test whose abort answers NoSuchUpload for one of the listed uploads.