Recursive copy() and mv() send CopyObject for directories, and mv() can delete overlapping copies

[laughingman7743]

Problem

Recursive copy() and mv() pass directory entries to cp_file(), and the recursive mv() that fsspec provides can delete data. The two problems are linked: fixing the first one exposes more of the second.

1. cp_file() sends CopyObject for directory entries

fsspec's recursive copy() passes the source directory and its subdirectories to cp_file(). S3FileSystem.cp_file() and AioS3FileSystem._cp_file() send CopyObject for them, which fails with NoSuchKey because no such object exists.

• copy(recursive=True) still succeeds, because fsspec ignores FileNotFoundError in recursive copies. Every directory entry still costs a failed CopyObject.
• mv(recursive=True) calls copy(..., on_error="raise"), so it raises FileNotFoundError.
  • S3FileSystem copies and deletes nothing.
  • AioS3FileSystem runs the copies concurrently, so the files are copied but the source is not deleted.

This was item 2 of #974; item 1 (get_file()) is fixed separately.

2. fsspec's mv() removes what it copied when the destination overlaps the source

AbstractFileSystem.mv() (fsspec 2026.9.0, spec.py:1292-1301) runs copy(path1, path2, recursive=recursive, on_error="raise") and then rm(path1, recursive=recursive). The removal expands path1 again after the copy. If the copies landed where path1 matches them, they are removed too.

Measured on master 28826be against real S3:

Call	Before	After
mv("p/src/*", "p/src/archive/", recursive=True)	src/a, src/b	nothing left; no error
mv("p/data/*.csv", "p/data/", recursive=True)	data/x.csv, data/y.csv	nothing left; no error
mv("p/src/*", "p/src/archive/", recursive=True)	src/a, src/sub/b	FileNotFoundError on the sub directory entry; src/a, src/archive/a, src/sub/b left
mv("p/src", "p/dst", recursive=True)	src/a, src/b	FileNotFoundError; nothing moved (problem 1)

Today, problem 1 stops most directory moves before the removal. Making cp_file() skip directory entries would let these moves reach the removal:

• mv("p/src", "p/src/archive", recursive=True) would remove both trees.
• mv("p/src/sub", "p/src", recursive=True) would remove src/sub/a, which was reported with live S3 in the PR Follow fsspec's get_file() contract for directories and file objects #990 review.

The PR #990 review also found other cases that need a decision. The table above does not measure them:

• maxdepth: the removal deletes files below the depth that were never copied.
• List sources: fsspec flattens a listed directory onto base names, so copies can overwrite each other.
• Duplicate destinations: mv("p/src/*.txt", ["p/dst", "p/dst"], recursive=True).
• Keys ending in /: info() reports such a key as a directory, so skipping directory entries would drop a real object, possibly one larger than a single CopyObject allows.

Expected

• copy(recursive=True) and mv(recursive=True) move a directory tree without a failed CopyObject per directory.
• mv() never deletes data that it has not copied to a separate location. Either it rejects such moves before copying anything, or it moves them safely.

Decision needed

How much of fsspec's copy-then-remove mv() PyAthena should guard. Options:

• Narrow guard: reject a destination that is, or lies under, a path the source matches.
• Overlap guard: reject any destination that overlaps the source in either direction, compared by path segment, plus maxdepth and list sources. PR Follow fsspec's get_file() contract for directories and file objects #990 tried this before it was reduced to get_file(); the attempt is at c6782a5. It also rejects some moves that work on master, such as files-only lists.
• Own implementation: have mv() remove exactly the keys it copied, instead of expanding path1 again.

Environment

• PyAthena master 28826be, fsspec 2026.9.0, Python 3.13.1.
• The table was measured with live S3 in the CI test bucket. The other cases come from mocked responses in the PR Follow fsspec's get_file() contract for directories and file objects #990 review.

[laughingman7743]

A case for this issue from PR #993 (#963, which makes find(path, withdirs=True) include path itself as fsspec does):

With #993, glob("s3://bucket/d/**") returns the directory bucket/d. Recursive copy()/mv() then pass that directory to cp_file() even for a flat directory with no subdirectories. Measured offline with mocked ListObjectsV2/HeadObject/CopyObject on #993 rebased onto c3ddabf (keys d/a, d/b):

Call	master c3ddabf	#993
copy("s3://bucket/d/**", "s3://bucket/out/", recursive=True)	ok, 2 CopyObject	ok, 3 CopyObject (1 fails with NoSuchKey)
mv("s3://bucket/d/**", "s3://bucket/out/", recursive=True)	ok	FileNotFoundError('d'), nothing copied

#993 stays in Draft until this issue's cp_file() directory skip and mv() implementation land. It will then be rebased with a regression test for this mv().

[laughingman7743]

Decision: own implementation. mv() pairs the sources and destinations as copy() does, copies the files, and deletes exactly the source objects that it copied, instead of expanding path1 again. cp_file() skips directories. Conflicting destinations (duplicates or another source) raise ValueError before anything is copied.