Repository navigation
perf(io): pipelined tree operations - #4
Merged
Merged
Conversation
Tree operations issue one request per file or folder and wait for each reply before sending the next. Add a single-threaded runner that keeps up to N async libsmb2 requests in flight on a held context; completion handlers submit follow-up operations to express ordering such as rmdir after a folder's contents. - first failure or cancel stops new operations and drains the ones in flight - cleanup operations still run while stopping, to close or remove partial results - idle timeout and connection reset follow the listing engine
Recursive delete removed one file, then one folder, at a time, each waiting a full round trip. Delete the listed tree with PipelinedOps, up to 16 requests in flight: files and empty folders at once, every folder as soon as its contents are gone, the root folder last. - list and delete on one Metadata slot instead of one per phase - the first failure stops new deletes; error messages are unchanged
A recursive copy created destination folders one at a time. Create them with PipelinedOps, up to 16 requests in flight, starting each folder's subfolders once it exists; the file list now comes from a plain walk of the listed tree.
With server-side copy, a recursive copy of many small files spent its time on per-file round trips (open, stat, resume key, copy, fsync, close), one file after another. Copy up to 8 files at once as pipelined request chains; the first non-empty file still goes alone to find out whether the server supports server-side copy, and an unsupported server keeps the one-by-one client copy. - extract the COPYCHUNK wire format into SmbCopyChunk.hpp, shared with the single-file copy - PipelinedOps reports operations dropped while stopping (kNotSent) so per-file state can settle - on failure or cancel, files in progress are closed and their partial destinations removed; finished files stay
Describe PipelinedOps, the in-flight limits and ordering for delete, folder creation, and file copies, and how a recursive copy decides between concurrent server-side copies and one-by-one client copies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf: pipelined tree operations
Why
Recursive delete and copy sent one request per file or folder and waited for each
reply before sending the next. With server-side copy, a tree of small files was
bound by those per-file round trips: about 8 per file (open, stat, resume key, copy,
fsync, two closes), one file after another.
What
PipelinedOpskeeps several async requests in flight on one held pool slot.Recursive delete, folder creation, and file copies now run on it.
deleteItemmkdirat a timeDesign
PipelinedOps: each operation queues one libsmb2 request. Its completionhandler may submit follow-up operations, which expresses ordering such as
rmdirafter a folder's contents. Single-threaded, so no locking.
ones in flight. Operations marked cleanup still run, so partial results can be
closed and removed. Queued operations that are never sent report
kNotSent, soper-file state still settles.
Each folder counts its unfinished contents and is removed when the count reaches
zero; the root goes last.
server supports server-side copy.
resume key →
COPYCHUNK_WRITE→ fsync → close, 8 files at a time.partial destinations removed; finished files stay.
SmbCopyChunk.hpp: theCOPYCHUNKwire format, now shared by thesingle-file and concurrent copies.
Notes for reviewers
listing order; the final tree is the same.
straight to concurrent copy, which never sends a server-side copy request for
empty files.
succeeded, the copy fails instead of switching to the client copy. It would take a
server that accepts one file and rejects another.