Repository navigation
feature: p2p mesh-based synchronous collaborative editing, time-travelling edit history, massive optimizations - #41
Merged
Conversation
… into fable-collab
Implemented collaboration-only tracing with non-blocking file logging. What changed: - Added tracing, tracing-subscriber, and tracing-appender. - Added crates/flowstate/src/logging.rs using tracing_appender::non_blocking, so log writing runs on its own worker thread. - Logging is initialized from main and the WorkerGuard is held for app lifetime. - Added structured collaboration logs for: - session create, join, attach, leave, shutdown - gossip subscribe/publish/receive/lag/neighbors - direct snapshot/update/blob/asset pulls and serves - text sync import/export, local edit capture, remote patches - presence publish/apply and peer counts - anti-entropy digest decisions and recovery pulls - asset transfer and validation - offline/reconnect/recovery/self-check drift - relevant join/share workspace errors Defaults: - Logs go to flowstate_data_dir()/logs as daily rolling flowstate.log... files. - Override with FLOWSTATE_LOG_DIR=/path/to/logs. - Default filter is collaboration-only: - flowstate::collab=trace - flowstate::workspace::workspace=trace - flowstate_collab=trace - Override with FLOWSTATE_LOG or RUST_LOG. Validation: - cargo clippy -p flowstate -p flowstate-collab passes. - git diff --check passes.
… into fable-collab
Eliminates the whole-document rewrite/reprojection on every non-trivial edit while preserving the single-root LoroText design and CRDT convergence. - T1: maintained paragraph->byte Fenwick offset index in DocBinding (new body_index.rs); range helpers no longer stringify the whole body. - T2: incremental ReplaceParagraphSpan — span-scoped minimal body splice, mark carry-over, targeted block-row reconcile; replace_document fallback. - T4: delta-driven remote reconcile — single-paragraph fast path via the Loro TextDelta + slice_delta; falls back to full reprojection on mark changes / newline changes / multi-region edits. - T6-T9: editor keystroke-path incrementalization (skip section rebuild on content edits, in-place block update, O(log n)+shift offset update). - T10: O(1) DocumentIdentityMap::paragraph_index via FxHashMap. - T12: remove dead pre-CRDT op-replay path + postcard wire format. - T15: cheap self-check via cached version-vector/projection hash. - T14: mark plan.md/fix.md superseded by FIX_LORO_ROOT.md. Adds a deterministic convergence regression and tests/perf_smoke.rs proving edit-proportional local deltas and O(1) remote paragraph reconcile. T3/T5 (dead-in-prod/optional) and T11/T13 (breaking/higher-risk) deferred.
- logging: default to `error` only; `FLOWSTATE_LOG_LEVEL` (error|warn|info| debug|trace) raises Flowstate's own crates, `FLOWSTATE_LOG`/`RUST_LOG` give full EnvFilter control, `FLOWSTATE_LOG_STDOUT` also mirrors logs to stdout. - README: document the logging environment variables and precedence. - clippy: add missing trailing `;` in session_io asset-serve warn arm (semicolon_if_nothing_returned); `cargo clippy --workspace --all-targets` is now clean.
…kg inspect/restore bins Consolidated from two local WIP commits; drops a stray 139MB Untitled.db8, the regenerable citation datasets, and a __pycache__ artifact from tracking (gitignored).
… + relabel + split Head/body/tail structural parser, sparse JSON training target, natural-distribution gold eval harness, LLM-relabel fold-in, and stratified split. Datasets themselves are gitignored (regenerable from the DuckDB + scripts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Editor/collab perf & fidelity work accumulated since 1b708fc: - vendored loro-internal: retained import diff-calculator (P1.B), env-gated diff-span probe, undo mark-key scoping - flowstate-collab: recorded-inverse two-stack undo/redo, import coalesce/second-drain, regional readback, checkpoint materialize-once - flowstate-document: single-pass projection registry fusion, package foreign-path guard, rebuild_caches_from_loro, search extract - flowstate-docx: overlapped (parallel) direct-property parse - workspace/editor: autosave debounce, docx-clobber write-path reconcile New crates (source only): - flowstate-autoflow: debate-round ASR/audio indexing + Python indexer - flowstate-citation: citation finetune backend / checks / probes .gitignore: exclude the 17 GB autoflow audio corpus + generated rounds.{sqlite,jsonl}, local design/spec scratch docs, and probe run-artifacts (logs, hotpath/peer captures, *.json outputs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014u9Q95R9VsoGwnNZpQNgdV
Checkpoint of the full performance program (open 7.6s->63ms, keystroke 0.1ms, collab mass-op typing stall 240ms->2-4ms, deep-undo 130s->1s, docx fragment-parallel parse, shallow-open + sidecar, incremental checkpoint, and the supporting vendored patches #1-#27 in loro-internal + the vendored generic-btree/rdocx/ct2rs crates). Includes the Act-14 git-checkout recovery: the four vendored richtext/txn files (handler.rs, txn.rs, container+state richtext_state.rs) had their uncommitted patches restored VERBATIM from prior-session transcripts after a `git checkout` reverted them -- T1 Src-value fast path, #15 streaming for_each_richtext_span, #26 get_cursors_batch, T7.2 utf16 loader accumulator + into_state net, T8.15 utf8 accumulator, T8.16 char_at_utf16, T8.3 mark-batch short-circuit. Verified: document 69/0 (debug: accumulator net + T1 verify armed), collab 169/0 (RICHTEXT_VERIFY), corpus 15436/15443. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017aX7peMhAVrYG4zENQtouY
The prior `git add -A` swept 66 leaked temp files into the tree — 76MB of scratchpad_* analysis artifacts (jsonl/log/txt/py/sh) from the citation finetune work plus citation_*.xlsx and the 55MB scratchpad_names/ name-dataset scratch. All regeneratable; the actual vendored gazetteer lists live in crates/flowstate-citation. Added /scratchpad_* and /citation_*.xlsx to .gitignore so they cannot leak back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017aX7peMhAVrYG4zENQtouY
The editor's `sync_projection_from_authority` ran a caret rebase — meant to reposition the caret across a REMOTE edit — on every drain, including local writes. Its fast path (stored CRDT cursors, O(log n)) only fires when the caret anchor matches the current selection; the anchor is not re-armed on caret movement, so the first edit after any click/arrow fell to the `fork_at` fallback (a whole-document fork, ~358ms on a 2.3M-char doc), freezing typing until the fork completed. For a local write the rebase is pure waste: `integrate_outcome` and `apply_undo_outcome` set the caret from their own commit immediately after. Split the drain into `sync_projection_stream(cx, rebase_caret)`; local callers use `sync_projection_local` (no rebase), the remote pump keeps the rebase (concurrent-typing interleave fix untouched). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
…_fast (13x cold-decode) The git-checkout incident that reverted the 4 vendored richtext files (Acts 5-14) restored `str_slice::try_split_at_unicode_pos` but not its caller in `decode_snapshot_fast`, which reverted to the upstream per-span bounds check `s.as_str().chars().count() < len` — an O(remaining-chars) scan PER text span, i.e. O(spans x chars) snapshot decode. On the 2.3M-char / 69k-span reference doc this made the cold richtext decode (the reopen / collab-join / cache-miss projection path) ~3.3s instead of ~230ms — a ~13x PROJECT-cold regression (3479ms -> 379ms after this fix). The dead-code warning on the surviving `try_split_at_unicode_pos` was the tell that its caller had been lost. Restore the intended caller: one O(len) walk that validates bounds AND splits. Validated by the richtext Src==built decode oracle over the full corpus and the convergence fuzz. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
…he caret The caret-anchor fast path (resolve stored CRDT cursors, O(log n)) only fired when the anchor matched the current selection, and the anchor was re-armed only after a write/sync — never on a plain caret move. So the first edit, and any incoming remote edit, after every click/arrow fell to the O(doc) `fork_at` rebase (~350ms on a large doc): a UI freeze on the local path (now also fixed by skipping the rebase on local writes) and, on the collab-receive path, a freeze whenever a co-author edited while the caret sat at a moved-but-not-typed spot. Re-arm on every selection change: `emit_selection_changed` is the one `&mut self` choke point all move sites funnel through. Guard it — `encode_selection_anchor` takes the editor's frontier and returns None when the core has advanced past it (an undrained remote import), so a re-arm racing that window can't anchor the caret to a body position the editor hasn't caught up to (it falls to the correct fork instead). New 2-peer test asserts a move+remote-insert takes the fast path (zero forks) and repositions the caret correctly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
The 3.3 GB real-docx corpus under helpers/corpus/ (local test fixture, gated by FLOWSTATE_CORPUS_DIR) had been committed in 74b7371, bloating .git to 19 GB. It was purged from the 7 unpushed commits via filter-branch and is now ignored so it can never be tracked again. The files remain on disk, untracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
…+ model embedding) The whole citation feature is archived on the citation-archive branch and in ~/flowstate-citation-archive/flowstate-citation-feature.bundle for later revival. This drops the standalone flowstate-citation crate, tools/citation_finetune, the vendored ct2rs backend, and the ct2rs [patch] entry. All collab/materializer/perf work is retained. Workspace builds clean.
… + materializer) Snapshot of the working tree so no in-flight work is loose. This tree interleaves several agents' uncommitted work with no clean file-level split: - Sol (codex): collab discovery/comments feature — new modules admission/bluetooth/discovery/dropbox/identity in flowstate-collab; the comment backend inside crdt_runtime.rs; collab/workspace UI wiring; discovery_runtime/dropbox_oauth/dropbox_checkpoint/comment_dialog/ collaboration_settings; macOS/Windows assets; release + goal notes. - In-flight caret/selection work in gpui-flowtext (owned by another agent). - Ongoing crdt-materializer-refactor changes. Both flowstate and flowstate-collab compile clean (cargo check). This is a squash-merge branch, so intermediate authorship is not preserved on merge. Committed as a safety checkpoint before auditing and extending Sol's work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
…OS BLE rebuild Security/robustness pass over the discovery+comments+Dropbox surface: - comments: store author_user_id on the thread at creation and gate delete on it (was a timestamp-race heuristic); sanitize control chars out of comment bodies; new comment_threads lifecycle test - identity: from_hex no longer panics on non-ASCII seeds; display names are clamped at signing so an oversized name can't mint a profile every peer rejects - settings: settings.toml written 0600 (identity seed + Dropbox tokens) - dropbox: connect/read timeouts on all HTTP clients, 256 MiB download cap, case-insensitive sidecar match, checkpoint-merge trust boundary documented - bluetooth/linux: disconnect after GATT read (peer's peripheral slot) - bluetooth/macos: module never compiled (dispatch2 feature was never enabled); rebuilt publish side as a dedicated peripheral actor thread (CoreBluetooth objects are !Send), dedicated dispatch queues for both managers, manager-state-aware power wait (permission prompt vs off), objc2 API fixes; now cross-compile-checks clean for aarch64-apple-darwin - bluetooth/windows: verified WinRT long-read handles frame reassembly; cross-compile-checks clean for x86_64-pc-windows-msvc - clippy: both crates back to zero (Boxed large enum variants, lock scope tightening, misc pedantic) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
- linux scanner: a MISSING service-data locator no longer skips a peer — CoreBluetooth cannot publish service data, so macOS publishers advertise only the service UUID; connect-and-verify via the signed record instead (a locator for a DIFFERENT document still skips pre-connect) - COLLABORATION_MACOS_WINDOWS_RELEASE.md rewritten as the authoritative handoff: current verified architecture, yellow-zone on-device verification lists for both BLE backends + Dropbox e2e, red-zone packaging/signing/acceptance matrix, and the deliberate trust-boundary decisions; the stale "other incomplete items" section is gone Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
Contributor
|
Important Review skippedToo many files! This PR contains 514 files, which is 364 over the limit of 150. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (11)
📒 Files selected for processing (514)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…raw-authority rename - cargo +nightly fmt --all across the workspace (fmt job) - rename rebase_selection's base_frontier param -> editor_frontier in the LocalWriteAuthority trait + LocalDocHandle impl: it is the caret-rebase anchor (same concept encode_selection_anchor already calls editor_frontier), not the banned write-validity frontier; clears the raw-authority guard without weakening it - cargo-machete: drop unused gpui-flowtext/postcard and flowstate-document/zstd; machete-ignore serde_bytes (used via #[serde(with)] attributes) and the vendored loro-internal/generic-btree leftovers - security: quick-xml 0.36.2 -> 0.41 for our docx parse paths (unescape() -> xml10_content()), lopdf 0.40 -> 0.44 (+ time update it needs), plist -> 1.10, cargo update anyhow/crossbeam-epoch/quinn-proto; ignore RUSTSEC-2026-0194/0195 for the old quick-xml copies pinned by latest upstream releases (xcb, docx-rs, rdocx-opc/oxml, citationberg, wayland-scanner) in deny.toml + new .cargo/audit.toml - deny.toml: allow Unlicense + CDLA-Permissive-2.0 (iroh websocket stack, webpki CA-bundle data) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
- ci.yml: install libdbus-1-dev (bluer/BLE discovery links libdbus via dbus/libdbus-sys; the build job predates that dependency) - delete two dead flowstate-document functions left by the Acts 5-14 refactor (cursor_fields superseded by text_cursor_fields, append_journal_transaction_to_prepared_file by its _with_durability variant; verified callers migrated, not lost) - clear the clippy -D warnings backlog CI never reached: semicolon consistency, format-string inlining, raw-string hashes, range pattern, assert_eq; wrap the four new integration-test files in the repo's #[cfg(test)] mod tests convention; write! instead of push_str(format!) in the fragment-parse fixture; explicit guard drops in comment_threads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
…sion) nightly-2026-07-13 panics in rustc_ast attr token_trees while expanding test entrypoint markers (ICE hit in gpui-flowtext). Pin the three nightly installs to the toolchain the workspace is verified against locally (build + clippy -D warnings + full test suite green); bump the pin deliberately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NDcBhNZgW3kjkJSMQg3T7d
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes previous PR