An interrupted multipart copy leaves the upload behind when CreateMultipartUpload is in flight

[laughingman7743]

Problem

A multipart copy that is interrupted while its CreateMultipartUpload request is running leaves the created upload behind, incomplete. It holds no parts, but it stays listed in ListMultipartUploads until something aborts it or a lifecycle rule expires it.

Both filesystems take the upload ID only from the response, and nothing reads that response after the interrupt:

• S3FileSystem._copy_object_with_multipart_upload() sends the request in the caller's thread (pyathena/filesystem/s3.py :1772 on master d989767). A KeyboardInterrupt raised during the request loses the response, so the upload ID is never known.
• AioS3FileSystem._copy_object_with_multipart_upload() awaits asyncio.to_thread(core.create_multipart_upload, ...) (pyathena/filesystem/s3_async.py :588). When the task is cancelled, the coroutine returns at once, while the thread still finishes the request and creates the upload. The cleanup of Cancelling an async multipart copy leaves its multipart upload behind #1046 starts only after the upload ID is known, so it does not run.

#1056 left this case out of scope and documented it ("A cancellation during the CreateMultipartUpload request, before the upload ID is known. The sync path does not handle an interrupt there either."). It still applies to the copy for objects larger than 5 GiB, which cp_file(), copy() and mv() run.

Reproduction

Offline. The test holds the creation until after the interrupt, then lets it return an upload ID:

• aio: start _copy_object_with_multipart_upload() as a task for a source larger than 5 GiB. Mock core.create_multipart_upload to block on an event, cancel the task while it blocks, then release the event. The task is done right after cancel(), and _abort_multipart_upload is never called, although the creation returned upload_id="uploadid".
• sync: mock core.create_multipart_upload to sleep, and deliver SIGINT to the main thread during the sleep. KeyboardInterrupt propagates, and no abort is sent for the upload.

Both checks fail on master d989767. They are the regression tests in the fixing PR.

Environment

PyAthena master d989767, Python 3.13.1, botocore 1.43.102, S3FileSystem and AioS3FileSystem.

Proposed fix (optional)

• aio: run the creation as its own task, await it through asyncio.shield(), and have the existing cleanup of Cancelling an async multipart copy leaves its multipart upload behind #1046 wait for it. The cleanup aborts the upload if the creation returned one, then waits for parts and the completion as it does today.
• sync: send the creation on the copy's executor and wait for its future. On an interrupt while waiting, wait for the creation that cannot be cancelled any more, abort the upload it created, and re-raise. _finish_multipart_upload() handles an interrupt during the parts the same way.

The S3File writer has the same window when it starts its multipart upload. It is tracked separately.