Skip to content

perf(io): pipelined tree operations - #4

Merged
yfcyrus merged 5 commits into
mainfrom
perf/pipelined-tree-ops
Nov 2, 2026
Merged

yfcyrus merged 5 commits into
mainfrom
perf/pipelined-tree-ops

Conversation

@yfcyrus

@yfcyrus yfcyrus commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

perf: pipelined tree operations

Why

Recursive delete and copy sent one request per file or folder and waited for each
reply before sending the next. With server-side copy, a tree of small files was
bound by those per-file round trips: about 8 per file (open, stat, resume key, copy,
fsync, two closes), one file after another.

What

  • PipelinedOps keeps several async requests in flight on one held pool slot.
    Recursive delete, folder creation, and file copies now run on it.
  • No API change; results, error messages, and cancel behavior are unchanged.
Operation Before After
deleteItem one file, then one folder, at a time up to 16 requests in flight; a folder is removed as soon as its contents are gone
Copy: folders one mkdir at a time up to 16 in flight; a folder is created as soon as its parent exists
Copy: files one file at a time 8 files at once with server-side copy

Design

  • PipelinedOps: each operation queues one libsmb2 request. Its completion
    handler may submit follow-up operations, which expresses ordering such as rmdir
    after a folder's contents. Single-threaded, so no locking.
  • Stopping: the first failure or a cancel stops new operations and drains the
    ones in flight. Operations marked cleanup still run, so partial results can be
    closed and removed. Queued operations that are never sent report kNotSent, so
    per-file state still settles.
  • Delete: list and delete share one Metadata slot, instead of one slot per phase.
    Each folder counts its unfinished contents and is removed when the count reaches
    zero; the root goes last.
  • Copy files: the first non-empty file is copied alone to find out whether the
    server supports server-side copy.
    • Supported: the remaining files run as independent chains, open → stat →
      resume key → COPYCHUNK_WRITE → fsync → close, 8 files at a time.
    • Not supported: they are copied through the client one after another, as before.
  • Failure or cancel during a copy: files in progress are closed and their
    partial destinations removed; finished files stay.
  • SmbCopyChunk.hpp: the COPYCHUNK wire format, now shared by the
    single-file and concurrent copies.

Notes for reviewers

  • Delete, folder creation, and file copies finish in completion order rather than
    listing order; the final tree is the same.
  • A folder containing only zero-byte files has no file to probe with. It goes
    straight to concurrent copy, which never sends a server-side copy request for
    empty files.
  • If the server rejects server-side copy for a later file even though the probe
    succeeded, the copy fails instead of switching to the client copy. It would take a
    server that accepts one file and rejects another.

Tree operations issue one request per file or folder and wait for each
reply before sending the next. Add a single-threaded runner that keeps up
to N async libsmb2 requests in flight on a held context; completion
handlers submit follow-up operations to express ordering such as rmdir
after a folder's contents.

- first failure or cancel stops new operations and drains the ones in flight
- cleanup operations still run while stopping, to close or remove partial results
- idle timeout and connection reset follow the listing engine
Recursive delete removed one file, then one folder, at a time, each
waiting a full round trip. Delete the listed tree with PipelinedOps, up to
16 requests in flight: files and empty folders at once, every folder as
soon as its contents are gone, the root folder last.

- list and delete on one Metadata slot instead of one per phase
- the first failure stops new deletes; error messages are unchanged
A recursive copy created destination folders one at a time. Create them
with PipelinedOps, up to 16 requests in flight, starting each folder's
subfolders once it exists; the file list now comes from a plain walk of
the listed tree.
With server-side copy, a recursive copy of many small files spent its time
on per-file round trips (open, stat, resume key, copy, fsync, close), one
file after another. Copy up to 8 files at once as pipelined request
chains; the first non-empty file still goes alone to find out whether the
server supports server-side copy, and an unsupported server keeps the
one-by-one client copy.

- extract the COPYCHUNK wire format into SmbCopyChunk.hpp, shared with the single-file copy
- PipelinedOps reports operations dropped while stopping (kNotSent) so per-file state can settle
- on failure or cancel, files in progress are closed and their partial destinations removed; finished files stay
Describe PipelinedOps, the in-flight limits and ordering for delete,
folder creation, and file copies, and how a recursive copy decides between
concurrent server-side copies and one-by-one client copies.
@yfcyrus
yfcyrus merged commit 2ef2e61 into main Nov 2, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant