Repository navigation
Upload and Watch & Watch Rearchitecture - #254
Merged
Merged
Conversation
5cript
commented
Oct 23, 2026
5cript
commented
Owner
- Upload and Watch: a new entry on the local side of the file explorer uploads a file or directory, then mirrors its local changes to the server.
- Open and Watch through a file tracker per session: the backend tracker owns every watch and decides what each change does on the server. The file tracking panel only shows that state and forwards the user's input.
- Watch options and server-change questions: re-upload on change, mirror moves and deletes, and check the server first. When the server's copy changed meanwhile, the user chooses Replace or Keep.
- Efficient sync: a change is uploaded once it has settled (judged by events and by the file's size and time). Uploads and deletes are batched into bulk operations. A new directory goes up once, and later only what the server lacks.
- Upload lifecycle: deleting a file while it uploads stops the upload and removes its partial file. Renaming it continues the upload under the new name from the partial file. Moves between directories become renames on the server.
- Watches that end on their own: a lost connection ends the session's watches and tells the user why. A downloaded copy that was deleted or renamed away ends its watch after 5 s. A closing session hands its copies to the orphan cleanup.
- Progress that counts files: "Syncing x of y" counts the entries a directory upload carries, and nothing is counted twice when files are reported late.
- No deletes on doubt: a path that cannot be checked (Windows long paths, access errors, pending deletion) never costs its server copy. The destination is compared before anything is sent. A directory moved away, to the trash for example, keeps its server copy.
- Recovery from lost watch events: when efsw reports missed changes, the tracker compares what is there with what the server is known to have. A watched directory deleted on Windows or macOS (told by its identity on the volume) is deleted on the server with its whole tree.
…afely - 87abe59 freed a WatcherFSEvents on removeWatch while its dispatch queue still ran the callback, watch tests crashed in process() and sendFileAction on macOS. 1.7.2 waits for it. - 1.7.2 waits forever in ~FileWatcherWin32 for the read of a watch whose directory was deleted. The fork's 627eb0d frees watches without a read in flight, nothing else differs from 1.7.2. - handleFileAction takes the old name as std::string const& now.
- Windows opens a std::ifstream's file without FILE_SHARE_DELETE, this one opens it with full sharing
- a watched file could not be renamed, deleted or saved atomically by an editor while its upload ran
- UploadOperationOptions::sourceMayVanish: a file or directory that is gone when the queue starts the upload completes it without a transfer instead of failing, also inside a bulk - For an owner that mirrors deletions apart, like a watch whose files come and go quickly
- OperationQueue::discardResumeBackup: a reconnect does not queue the operation again, for an owner that decides anew what goes to the server
- ComparedState::mustStillExist: a path seen present that is gone now counts as changed since the compare, so an upload does not recreate a file the server lost behind its owner's back. A sync leaves it off.
- OperationQueue::addUploadOperation takes the comparedDestination of UploadOperation: what a file or link must find at the destination to replace it, a directory ignores it
…rror - Every backend error rings the notification bell; ChangedSinceCompare is the compare doing its job and its owner reports it, like the sync dialog's list - Upload and download finalize, a bulk's skipped entry and the queue's failed operation use the level
…f the queue - OperationQueue::leaveChangedDestinationsToOwner: such an end completes the operation as canceled with its error, a bulk leaves those entries out of its failed ones and keeps them in its indexed failures - For an owner that tells the user itself; sync does not register, its cards still show those entries as failed
- The change filter and what reports changes need it without the temporary instances
…ttled - One save shows as several watcher events (inotify reports the writes and the close). Content changes of a file are held back until it stayed unchanged for 300 ms, only content changes of the same file merge - Every other change goes at once, after the held back changes at or below its path and where it moved from, which the path order finds without looking at the others - ReportedChange and ChangeSink describe a change of a watch for whoever reports it; isAtOrBelow compares paths
…h service - DirectoryWatchService: one efsw watcher for plain watches, one of its own for each recursive watch (efsw holds its lock while it walks a tree) and for every watch on macOS (efsw rebuilds a shared FSEvents stream whenever a watch comes or goes and drops the events on their way). A directory is watched one way only, its subscribers share the watch; InstanceWatch is a subscription - DirectoryWatcher hands a watch's changes to its strand, filtered and stated on efsw's thread. On macOS it drops what FSEvents replays from before the watch began and the modification efsw reports before every removal, and gives the watched directory its spelling back where efsw precomposes names (eventSpelling) - ChangeReporter drops noise (isTransientChange) and settles content changes before a change reaches its sink - TemporaryDirectoryInstance reports to a sink instead of the frontend. TempDirInstanceManager is a plain class on the strand of the sessions, owned by the session manager, which answers the orphan count and the cleanup; its file states are typed and list only the requested subtree - The panel's Open and Watch called the instance RPCs that are gone now, it works again through the backend tracker
- LocalItemWatch watches an item the user has locally: its directory flat, a directory item recursively, only the item's changes reach its sink and noise is dropped on efsw's thread - It follows renames within its directory once they settled, case-only ones and ones FSEvents merges included, and takes the item back when an editor writes it anew at the old name; a removal waits until the item stays away, a delete and recreate or a Windows rename reported as a removal first is no removal - A rename onto another watch or a temporary name ends it; changes inside a renamed directory move along, and a directory moved out of its directory stops being watched once its last changes arrived - On macOS the item's own NFD name is matched to efsw's precomposed events - LocalWatchManager owns them and refuses overlapping watches and the temporary downloads directory, comparing on disk (DiskPlace) and folding case where the file system does, with localized refusals
- KnownServerContent: what the server has of a tracked item by local path, searched by path prefix, moved along with renames, with the directories that left with their content kept until it turns up elsewhere; on macOS it finds a known name spelled in the other Unicode normalization, in every script - PendingUploads: the uploads on their way per path, what changed again, moved or was deleted meanwhile, and the kind a path has at its latest change - ServerOperationOrder: an operation waits until those decided before it on the same path, above or below it ended - Move meanings, remote paths of changes and server changes as plain functions
- Open and Watch downloads into an instance of its own and watches the copy there; Upload and Watch watches a local item and uploads it under its canonical name. No two watches upload to the same remote path or one below the other, and a downloaded copy never moves or deletes the server's file - Each change decides what the server gets: uploads, renames and deletes follow local edits, renames and removals as the watch's options say, in the order decided per path, also while an upload of the path is on its way - A file goes up with what the tracker last knew of its destination; when the server changed meanwhile the user is asked and the upload keeps its place in the order. Directories and links are compared before their upload - Paths deleted and written anew, paths that turned into the other kind, directories moved within the item, symbolic links and macOS spellings in the other normalization keep the server copy right - Counts the changes on their way and shows its progress at most every 200 ms - TrackerRemote is what it needs of the server and TrackerView what it shows, both faked in its tests
- BatchingRemote sends what the tracker queues within 100 ms of quiet (1 s at most, 500 entries) as one bulk upload and one bulk delete, each tracker callback hears the end of its own path with the server's state after it - A lone change, links, directories and the first upload of a watch keep their single operation
…onId> - The session's remote queues the tracker's uploads, renames and deletes on its operation queue with the vanishing source allowed, the destination compared and its stop left to the tracker, bulks without a resume backup, and stats what an upload left on an SFTP channel's strand - Requests open, checkUploadAndWatch, uploadAndWatch, stop, setOption, discard and answer; FrontendTrackerView pushes the entries, questions, progress and orphan count - The session manager hands every session the instances, the local watches and the default options from the settings; FileTracker::stopAll ends the watches a reloaded frontend left behind
…d input - The panel lists the tracker's entries in a stable row order, asks its questions, shows the orphaned downloads and a "Syncing done of total" bar with how many changes wait for an answer - Open and Watch on the remote side is one request to the tracker; the hover hints say what it does, and the Move and Delete Remote switches of a downloaded copy are off with a tooltip that says why - A reload ends the watches of the frontend before it through FileTracker::stopAll - The panel's own tracking state and its RPCs to the instances are gone
- Uploads a file or directory into the listed remote directory, or into the one directory selected on the remote side (contextMenuItems gets the other side's selection), and has the tracker watch it from then on - The tracker refuses an overlapping watch before the upload confirmation, which Upload shares - The hover hint names where the item goes
…letions - d71ed84 on 5cript/efsw: five fork commits on 627eb0d (e733a67, fix/inotify-nameless-and-win32-removed-watch) and a signed pick of upstream's d5d9828 (SpartanJ/efsw PR #221). - inotify: an event without a name took the next event's bytes as its name, a tar extract reported Modified dNN/\x01. - inotify: the thread waits with poll(), select() cannot take a descriptor past FD_SETSIZE. A watcher whose descriptor was 1024 or higher reported nothing, or aborted the process with _FORTIFY_SOURCE. - Win32: deletions held for ReportCrossDirectoryMoves are reported when the watched directory's read ends, and before an addition of their path, instead of being dropped or coming late. - Win32: the next read is armed before the listener runs. A read the delete-pending watched directory refuses is reported as missed file actions, and the watch lets the directory go.
- Tracker::stopAll ends the instances of watched and only opened copies, a download on its way opens nothing - Session::stop stops the file tracking before the tracker goes - An instance with nothing left in it goes with its directory
…view why - Tracker::connectionLost ends the watches like stopAll, then shows them as ended (SharedData::EndedWatches) - TrackerView::showEnded, sent as FileTracker::<id>::onEntriesEnded - SessionManager::removeSession(connectionLost) calls Session::onConnectionLost before stop() - The tracker's single operations keep no resume backup, like its bulks: a reconnect queued the first upload of an Upload and Watch directory again
- The copy has copyReturnTime (5 s) to come back, saves that delete it or move it aside keep the watch - The view shows the end as WatchEndReason::CopyGone, the instance goes when nothing is left in it
- The file tracking panel turns onEntriesEnded into a warning: a copy deleted or renamed, or the connection lost - Language keys fileTracking.watchEndedCopyGone and watchesEndedConnectionLost (en_US, de_DE)
- Utility::checkedSymlinkStatus: status, nothing only for ENOENT/ENOTDIR/ERROR_FILE_NOT_FOUND/ERROR_PATH_NOT_FOUND - Windows: std::filesystem's not found is looked at again through the \\?\ form, past MAX_PATH a path that exists is ERROR_FILENAME_EXCED_RANGE, an unreachable server keeps its own error - The second look tells there (its status), gone again (nothing) and reachable only as \\?\ (ERROR_INVALID_NAME): git deletes and writes files anew, GetFileAttributesW found the new one in between (watch e2e WIN3-N1) - A path pending deletion on Windows counts as gone: libc++'s remove leaves it so while a watch holds a handle, it and what is below it read as access denied, STATUS_DELETE_PENDING tells it from a real denial - Detail::checkedSymlinkStatus takes the looks from a reader, so a test can make the first one miss
- A server delete only goes when the local path is gone or checked, else it is skipped and logged as an error - A change whose path cannot be checked is not uploaded, nor taken as gone or as another kind (W6: Windows long paths)
…ents - Windows reports the writes to a file kept open only when someone else opens or closes it, an upload does both: each upload of a slowly written file caused the next one (UD10: 7 uploads of 64 MiB, Linux and macOS 1)
- An upload whose source may vanish and is gone already is refused as FileNotFound, which is no error worth logging - addBulkUploadOperation takes sourceMayVanish for each entry, the tracker's gathered uploads set it - The error log capture of the queue tests lives in a header of its own
…nchanged - UploadOperationOptions::onSent hears of each regular file once it replaced the destination, with its size and modification time when those stayed the same from before it was opened - A file that changed while it was read and a continued upload come without a state, a link is not told - addUploadOperation and addBulkUploadOperation hand it to every file they upload, below a directory too
…e reported - An upload's end tells which files it wrote, and in which local state (size, exact modification time) when they stayed unchanged - A change of a file the server got in the state it still has is not uploaded again: the late reports of files a directory upload already carried, on Linux, macOS and Windows alike - A change of a directory the server has uploads only what it lacks below it instead of the whole directory - What a directory upload wrote that was gone locally by its end is remembered, a file without its state and the directory itself as the server tells, so their deletions follow (watch e2e G2: a checkout removed moved-dir while its upload sent m0.txt and m1.txt, the server kept them) - A deletion below a directory upload is looked at again once the upload ended, which may have sent the path before
…rogress - An upload of a directory counts as the entries below it when it goes, as the late reports of its content when those are more, and as what it left on the server once it ended, so "Syncing done of total" counts files - The changes that waited below a directory still count after its upload took them along, the reports that arrive meanwhile count when they go - A directory that changed again counts no extra change, what it lacks afterwards counts on its own (watch e2e G3 on macOS: the total started at 22 per directory of 21 entries) - A change of a path gone by the time it would go counts as nothing done, its deletion counts on its own (watch e2e G3: Windows counted the modified directories its removal had taken) - The view hears at once that nothing is on its way any more, together with the rows that stopped uploading, after a lost connection too
…s why - watch e2e W3 on Windows: each refused name showed the dialog and a red error notification
… teardown - cancel() and cancelAll() tell the owners of what they removed, the tracker freed nothing before - The work loop takes a finished operation out of its queue before its owner hears of it
…a new name - UploadOperation::partialFileOnCancel: a cancel keeps the temporary file it wrote to for a later upload, or removes it, a temporary file it did not open itself always stays - OperationQueue::cancelUpload cancels an upload with its partial file kept or discarded, discardPartialUpload removes one that a cancelled upload kept - UploadOperationOptions::adoptPartialOf: the upload of a file under its new name takes over the temporary file of the upload cancelled under its old name and continues it - A cancel runs on the strand of the transfer: closing the local file from the queue's thread cut a big-file read short, which logged an error and took the flag of the opened partial file, so the stop removed nothing - The removal stays queued behind the close of the file on the server - The cancel and finalize logs name the source before the destination
- A file deleted or moved over another during its upload stops it, its partial file goes - A file renamed during its upload, its first one too, stops it and goes on under the new name from the partial file - A partial file that no upload continues goes in the server order, a file edited since its stop goes anew - A file whose server copy was not renamed is uploaded under its new name - Only an upload sent as a file stops: on Windows a removed directory's last modification arrived after it was gone and counted as a file's, which stopped the directory's upload
- Recursive watches turn on efsw's ReportCrossDirectoryMoves: one move with the absolute old path where the platform tells it, FSEvents pairs the two halves only when they arrive together - A file or directory moved to another directory of the item is renamed on the server, not deleted and uploaded anew
… events - DirectoryWatchService handles handleMissedFileActions on the shared and the own watchers - DirectoryWatcher lets them past its filters, reportMissed feeds one in tests
- Gone, its removal is held back like a reported one; its own watch losing them sends the Missed change along - There, the sink gets the Missed change of the directory, or of the item when its directory's watch lost them - Windows reports a Missed change of its own for a watched directory that turned delete-pending
…w's thread - Windows drops the buffered changes of a watched directory being deleted when efsw reads them too slowly - One worker strand per DirectoryWatcher keeps the order, the strand of the handler stays free for the RPCs
…wn to have - Below a directory that is there, what is gone is deleted like reported and what changed is uploaded - A watched directory gone or delete-pending goes with its known tree, deepest first, as reported deletions would, on Windows only: only a deletion ends the read of a Windows watch, a move keeps it. Elsewhere it may have gone to the trash and keeps its server copy, like any directory that left with its content - A test holds efsw's thread up by a busy millisecond at every event while the watched directory is removed (a sleep takes a tick on Windows): in the VM 9 of 10 runs left 1 to 7 entries without the rescan, 30 of 30 pass with it - The removal tests of larger trees do not run on macOS: FSEvents reports the deletions of the first entries of a watched directory removed with a larger tree, 36 to 55 of 72 server entries went
- lstat of /.vol/<dev>/<ino>: ENOENT once deleted, found after a move (also into ~/.Trash), on APFS and HFS+ only - fsgetpath() still returned the old path of a deleted directory for over a second after its last descriptor closed
- FSEvents stops reporting below a watched directory removed with its content (`rm -rf`): its whole server tree stayed, of a larger tree 36 to 55 of 72 server entries went - Not found on its volume, its known tree goes like one whose deletions were lost; moved away (trash), it stays - The removal tests of a watched directory run on macOS too, one moved away keeps its server tree everywhere
5cript
enabled auto-merge (rebase)
October 23, 2026 21:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.