From 950fc2f6d59dd17bc1ebe437828d7560260de858 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Fri, 12 Jun 2026 01:55:57 -0500 Subject: [PATCH 01/45] created plan.md --- CLAUDE.md | 1 + plan.md | 718 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 719 insertions(+) create mode 120000 CLAUDE.md create mode 100644 plan.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 00000000..47dc3e3d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/plan.md b/plan.md new file mode 100644 index 00000000..ad909ec4 --- /dev/null +++ b/plan.md @@ -0,0 +1,718 @@ +# Flowstate P2P Collaborative Editing — Implementation Plan + +**Status:** approved architecture, ready for implementation +**Target:** Google-Docs-style live co-editing of `.db8` rich-text documents +**Stack:** [iroh](https://docs.rs/iroh) 0.98.x for P2P networking · [loro](https://docs.rs/loro) 1.13.x for the rich-text CRDT +**Authoring date:** 2026-06-12 (API versions verified against crates.io/docs.rs on this date) + +--- + +## 0. How to use this document + +This plan is written for an orchestrating agent that will spawn implementation subagents. Rules of engagement: + +1. **Milestones M0–M6 (§22) are the unit of delegation.** M2 (doc sync) and M1 (networking) are independent and can run in parallel. M3+ depend on M2. +2. **Do not relitigate the decisions in §20.** They were made deliberately with full knowledge of the codebase. If an implementer hits a wall that genuinely invalidates one, stop and surface it rather than silently choosing differently. +3. **§23 lists the handful of API names that must be verified against docs.rs at implementation time** (exact method names in fast-moving crates). Everything else in this document was verified against the codebase or live docs on the authoring date. +4. Per `CLAUDE.md`: new functionality goes in **new files**, keep files **under 1000 LOC** (budgets are given per file), prefer **gpui-component** widgets over raw GPUI, add dependencies **via `cargo add`**, and run **`cargo clippy`** after each milestone. This workspace denies `clippy::pedantic` and `clippy::nursery` — write code accordingly (no silent truncating casts, no `unwrap()` on `get`, document any `unsafe`, etc.). See the allow-list in the root `Cargo.toml` before fighting a lint. +5. Conventions in this repo you must follow: 2-space indent (`.rustfmt.toml`), `#[hotpath::measure]`/`#[hotpath::measure_all]` on functions/impls in hot paths (match surrounding code), modules sometimes assembled via `include!` (don't convert them), `Entity`-based GPUI state. + +--- + +## 1. Product definition (the UX contract) + +* Any user with a rich-text document (`.db8` editor tab) open can **start a collaboration session** for that document. Starting a session produces a **ticket string** (one opaque token, e.g. `fscollab…base32…`) they can paste to anyone over any side channel. +* Another Flowstate user chooses **Join session**, pastes the ticket, and the document opens as a **new tab** in their workspace. Both users now see each other's edits live (sub-second on LAN, ~1 s worst case via relay), each other's carets (colored, named), and a participant list. +* **The host's on-disk file is the single source of truth.** Only the host's editor has a `document_path`; only the host saves/autosaves to disk. Joiners can *Save As* to take a snapshot copy. While the session is live, concurrent edits merge via CRDT (loro) — no locking, no "reload to see changes." +* The host can end the session (or just close the tab/app); joiners keep an orphaned local copy with a "session ended" banner. +* Sessions can be started in **Editor** mode (joiners can write) or **Viewer** mode (joiners are read-only). The editor already understands `CollaborationRole::{Owner, Editor, Viewer}` and blocks mutations for viewers. +* Out of scope for v1 (explicitly): collaborating on Flow (`.fl0`) documents, per-peer role changes mid-session, mesh topology / host migration, persistence of CRDT history into `.db8`, end-to-end key rotation, more than ~8 simultaneous peers. + +--- + +## 2. What already exists (do not rebuild) + +A previous iteration scaffolded collaboration hooks into the editor engine. Inventory, with file references: + +| Capability | Where | Notes | +|---|---|---| +| Canonical edit-op stream with stable IDs | `crates/gpui-flowtext/src/collaboration.rs` | `CanonicalOperation` enum: `InsertText`, `DeleteRange`, `SplitParagraph`, `JoinParagraphs`, `SetParagraphStyle`, `SetRunStyles`, `InsertBlock`, `DeleteBlock`, `MoveBlock`, `ReplaceParagraphSpan`, `ReplaceBlock`, `ReplaceDocument` | +| Stable identity map | `collaboration.rs` → `DocumentIdentityMap` | maps `ParagraphId`/`BlockId`/`TableCellId` (u128 UUIDs) ↔ indices; reconciled after every edit | +| Per-edit op capture | `crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs:238` | `mark_document_changed_with_reconcile` stores `last_collaboration_edit` after **every** mutation; every undo record carries `canonical_operations` | +| Remote-op application (naive) | `crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs:265` | `apply_remote_operations(&[CanonicalOperation])` — bounds-checked, ID-addressed appliers built on `edit_ops` primitives (`insert_text_at`, `delete_cross_paragraph_range`, `split_paragraph_at`, `mutate_runs_in_range`, `apply_document_span_replacement`) | +| Roles + write gating | `editor/mod.rs:182` (`CollaborationRole`), `editor/commands.rs:86,357,368,380` | every mutation command path already checks `can_write_collaboration()` | +| Remote caret rendering | `editor/mod.rs:461` (`ExternalCaret { offset, color_rgb }`), `lifecycle.rs:471` `set_external_carets` | painted per paragraph; presence layer just needs to feed it | +| Full document replace | `lifecycle.rs:274` `replace_document_from_collaboration` | used for join bootstrap / resync | +| Editor events | `crates/gpui-flowtext/src/api.rs:107` | `EditorEvent::{Changed, SelectionChanged, …}` + `EditorEventSink` | +| Document model | `crates/gpui-flowtext/src/document/*` | `Document { text: crop::Rope, paragraphs, blocks, assets, ids, sections, offset_index, theme }`. Rope = paragraph texts joined by `\n` (no trailing newline; see `edit_ops/offsets.rs:19 paragraph_width`). Non-text blocks (`Image`, `Equation`, `Table`) occupy **no** rope bytes. Table cells own their text privately (`TableCellParagraph { paragraph, text }`). | +| Run styles | `document/text.rs:192` | `RunStyles { semantic: RunSemanticStyle, direct_underline: bool, strikethrough: bool, highlight: Option }` — `Copy`, 4 orthogonal axes → 4 CRDT mark keys | +| Save machinery | `editor/style_state.rs:233 save`, `:247 save_as`, autosave in `workspace/workspace/documents.rs:865` | autosave already skips editors with `document_path == None` — joiners get correct behavior for free | +| Wire encoding helper | `collaboration.rs:280` `encode_canonical_operations` (postcard) | **Will NOT be used for sync** (it silently drops `ReplaceParagraphSpan`/block ops). Leave it; the CRDT layer replaces it. | + +**Key consequence:** the integration burden is *translation*, not editor surgery. The editor already emits a canonical op stream and already accepts externally-driven document mutations. What's missing: (a) a CRDT document that both sides converge on, (b) a network layer, (c) session orchestration + UI. + +Other load-bearing facts about the host app: + +* **No tokio anywhere.** GPUI provides its own (smol-based) executors: `cx.background_executor().spawn(...)`, `cx.spawn(async move |entity, cx| …)`. iroh requires tokio → we run a dedicated tokio runtime thread (§12.1). +* `Workspace` (`crates/flowstate/src/workspace/workspace/mod.rs:65`) owns `document_panels: Vec>`, `active_editor: Option>`, and an `editor_subscriptions: Vec<(Uuid, Subscription)>` list — panels are keyed by `Uuid`. +* `cx.observe(&editor, …)` per panel (`documents.rs:178`) is how autosave is driven — the collab session uses the same mechanism to drain local edits. +* The document **render theme is a local user preference** (`documents.rs:153`: "DB8 stores style assignments, not style appearance") — **do not sync `DocumentTheme`**. Only style *slot assignments* (which are part of `ParagraphStyle`/`RunStyles`) sync. +* Commands are declared in `crates/flowstate/src/commands.rs` (`CommandId` + `COMMAND_SPECS`); top-bar menus in `workspace/workspace/top_bar.rs`; status bar in `render_status.rs`; modal/dialog/notification widgets come from `gpui-component` (`dialog.rs`, `notification.rs`, `avatar/`, `clipboard.rs`). + +--- + +## 3. Architecture overview + +``` +┌──────────────────────────── Flowstate process (host or joiner) ───────────────────────────┐ +│ │ +│ GPUI main thread tokio runtime thread (new, lazy) │ +│ ┌──────────────────────────────────────────┐ ┌────────────────────────────────────┐ │ +│ │ Workspace │ │ CollabNet (flowstate-collab::net) │ │ +│ │ └─ DocumentPanel ── RichTextEditor │ │ • iroh Endpoint (one per app) │ │ +│ │ ▲ collab_apply / take_edits │ │ • protocol::Router, ALPN │ │ +│ │ │ │ │ b"flowstate/collab/0" │ │ +│ │ CollabSession (Entity, per panel) │ │ • per-peer send/recv tasks │ │ +│ │ • LoroDoc + EphemeralStore │ async │ • star topology: │ │ +│ │ • DocBinding (ids ↔ containers) │ chan │ host accepts N conns │ │ +│ │ • LocalApplier / RemoteApplier │◄─────►│ joiner dials 1 conn │ │ +│ │ • participant roster, role │ │ • frame codec (postcard+len) │ │ +│ │ CollabManager (Global) │ └────────────────────────────────────┘ │ +│ │ • owns runtime handle + NetCommand tx │ │ +│ └──────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────────────────────────┘ + + host ──── QUIC (iroh: direct holepunched, or n0 relay fallback) ──── joiner(s) +``` + +**Topology: star.** The host is the hub; joiners connect only to the host. The host imports every received update into its own `LoroDoc` and relays the *raw bytes* to all other peers (loro updates are idempotent and commutative, so blind relay is safe). This matches the "host owns the file" authority model, makes permission enforcement a host-side concern, and avoids gossip/mesh complexity. iroh-gossip is deliberately **not** used. + +**Data flow for one keystroke (host or joiner):** + +1. User types → existing edit pipeline mutates `Document`, pushes an `EditRecord`, queues a `CollaborationEdit` (canonical ops). +2. `cx.observe(editor)` fires → `CollabSession::flush_local_edits` drains the queue → `LocalApplier` translates canonical ops into loro container ops inside one `LoroDoc` commit. +3. loro's `subscribe_local_update` callback hands the session a `Vec` update blob → session sends `NetCommand::Broadcast` over the async channel → tokio thread writes a `Frame::DocUpdate` to peer connection(s). +4. Remote side: tokio thread reads frame → `NetEvent::Frame{peer, bytes}` → session entity `doc.import(bytes)` → loro fires container diff events → `RemoteApplier` maps diffs to paragraph/block mutations → editor repaints, remote selection remapped. + +The CRDT doc lives **on the GPUI main thread inside the session entity** (loro ops are microseconds; only network I/O leaves the thread). `LoroDoc` is `Send + Sync`, which we exploit only for the one-time snapshot export/import on join (background executor). + +### 3.1 Crate layout (new code) + +``` +crates/flowstate-collab/ # NEW crate. Core logic. NO gpui dependency → headless-testable. + Cargo.toml + src/lib.rs # pub mod …, shared error type (thiserror not in tree: use anyhow per workspace norm) + src/protocol.rs # Frame enum, PROTOCOL_VERSION, frame codec (≤400 LOC) + src/ticket.rs # SessionTicket (iroh-tickets Ticket impl) (≤150 LOC) + src/schema.rs # loro container layout constants + builders + projection types (≤500 LOC) + src/binding.rs # DocBinding: container ↔ ParagraphId/BlockId table (≤350 LOC) + src/local_apply.rs # CanonicalOperation → loro ops (≤700 LOC) + src/remote_apply.rs # loro DiffEvent batch → DocPatch list (≤700 LOC) + src/projection.rs # LoroDoc → Vec (+ reverse: Document → fresh LoroDoc) (≤450 LOC) + src/presence.rs # EphemeralStore wrapper: PresenceState, cursors codec (≤300 LOC) + src/integrity.rs # projection hash (xxhash via rustc-hash? no → use `twox-hash`), resync decision (≤150 LOC) + src/session_core.rs # role/roster/session-id types shared by host & joiner (≤250 LOC) + src/net/mod.rs # NetCommand/NetEvent enums, channel types (≤200 LOC) + src/net/runtime.rs # tokio runtime thread bootstrap, Endpoint construction (≤250 LOC) + src/net/host.rs # accept loop, per-peer tasks, relay fan-out, auth (≤500 LOC) + src/net/joiner.rs # dial, handshake, reconnect-with-backoff loop (≤400 LOC) + src/net/framing.rs # length-prefixed read/write on iroh send/recv streams (≤150 LOC) + tests/convergence.rs # multi-peer fuzz (THE acceptance test, §21.2) + tests/translation.rs # golden tests per canonical op + tests/loopback.rs # real iroh sockets on localhost, 2–3 peers + +crates/flowstate/src/collab/ # NEW module in the app crate. GPUI glue + UI. + mod.rs # CollabManager global, init, plumbing (≤300 LOC) + session.rs # CollabSession entity (per panel): owns LoroDoc side, drains edits, + # applies remote patches via editor API, roster state (≤700 LOC) + share_dialog.rs # start/join/manage modal (gpui-component Dialog) (≤500 LOC) + status.rs # status-bar pill + tab badge elements (≤250 LOC) + +crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs # NEW: editor-side patch application (≤600 LOC) +``` + +`flowstate-collab` depends on: `loro`, `iroh`, `iroh-tickets`, `tokio`, `async-channel`, `postcard`, `serde`, `uuid`, `anyhow`, `rand`, `twox-hash`, and `gpui-flowtext` (for `CanonicalOperation`, `Document`, `InputBlock` types only — this is a deliberate dependency direction: collab knows the document model; the document engine does *not* know collab/loro). + +--- + +## 4. Dependencies (exact) + +Run from workspace root (per CLAUDE.md, use the CLI — these land in `[workspace.dependencies]` then get referenced from member crates): + +```sh +cargo add --package flowstate-collab iroh@0.98 # default features; relay/discovery wiring per §23-V1 +cargo add --package flowstate-collab iroh-tickets@0.98 +cargo add --package flowstate-collab loro@1.13 +cargo add --package flowstate-collab tokio@1 --features rt-multi-thread,macros,time,sync +cargo add --package flowstate-collab async-channel@2 +cargo add --package flowstate-collab postcard@1 --features use-std +cargo add --package flowstate-collab rand@0.9 +cargo add --package flowstate-collab twox-hash@2 +cargo add --package flowstate-collab --dev proptest@1 +``` + +Version rationale: + +* **iroh 0.98.2** is the latest stable (1.0.0-rc.1 exists as of 2026-05-27). Pin 0.98; the 1.0 surface is the same post-rename API (`Endpoint`, `EndpointId`, `EndpointAddr`, `endpoint.addr()`, `iroh::protocol::{Router, ProtocolHandler, AcceptError}`). Migration to 1.0 final is a version-bump task later, not an architecture change. +* **iroh-tickets** is the post-0.94 home of ticket types (`EndpointTicket`, the `Ticket` trait); keep its version in lockstep with iroh. +* **loro 1.13.1** — stable 1.x line; has everything we need: `LoroDoc`, `LoroText` (+`*_utf8` APIs), `LoroMap`, `LoroMovableList`, `ExportMode::{Snapshot, updates(&vv), all_updates()}`, `subscribe_local_update`, `subscribe` (diff events), `UndoManager`, `Cursor`, `awareness::EphemeralStore`. +* **async-channel** — runtime-agnostic MPMC; both smol-side (GPUI) and tokio-side can await it. This is the only inter-runtime bridge primitive. Do not use `tokio::sync::mpsc` across the boundary. +* **twox-hash** — fast non-crypto hash for projection integrity checks. + +`crates/flowstate` adds `flowstate-collab` as a workspace dep. `gpui-flowtext` gains **no** new dependencies. + +--- + +## 5. CRDT schema (loro container layout) + +The loro document is the shared, convergent representation. Each peer's flowtext `Document` is a *projection* of it. Schema version is stored in the doc so future formats can migrate. + +``` +LoroDoc +├─ "meta": LoroMap +│ ├─ "schema" : i64 = 1 (bump on breaking layout change) +│ ├─ "title" : string = host display name of the doc (informational) +│ └─ "created_by" : string = host EndpointId (z32/hex, informational) +└─ "blocks": LoroMovableList (one entry per flowtext Block, same order) + └─ [i]: LoroMap (block container) + ├─ "kind" : string ∈ {"p", "image", "equation", "table"} + ├─ for "p" (paragraph): + │ ├─ "text" : LoroText (rich text w/ marks, see below) + │ └─ "style" : i64 (ParagraphStyle encoded: Normal = -1, Custom(s) = s) + └─ for "image" | "equation" | "table": + ├─ "data" : binary (postcard-encoded InputBlock payload, §5.3) + └─ "rev" : i64 (monotonic bump; makes replaces visible in events) +``` + +### 5.1 Rich text marks (per-paragraph `LoroText`) + +`RunStyles` has 4 orthogonal axes → 4 mark keys: + +| Mark key | Value | Absent means | +|---|---|---| +| `"sem"` | i64 slot of `RunSemanticStyle::Custom(slot)` | `RunSemanticStyle::Plain` | +| `"ul"` | bool `true` | `direct_underline == false` | +| `"strike"` | bool `true` | `strikethrough == false` | +| `"hl"` | i64 slot of `HighlightStyle::Custom(slot)` | `highlight == None` | + +**Mark expansion: configure ALL four keys as `ExpandType::None`** via `LoroDoc::config_text_style` on every doc before first use (factor into `schema::configure_text_styles(&LoroDoc)`). Rationale: flowtext computes the styles of every inserted character explicitly (`pending_styles` / run inheritance in `edit_pipeline.rs`), so the CRDT must not "helpfully" expand neighboring marks — the LocalApplier always marks inserted ranges itself. This makes mark state deterministic and identical to flowtext's run model. + +Reading a paragraph back: `text.to_delta() -> Vec` (insert segments + attribute maps) → fold into `Vec` (adjacent equal-style segments merge, mirroring `edit_ops` run normalization). + +**Soft line breaks** (`\u{2028}`, `document/core.rs:12`) are ordinary characters inside paragraph text — no special handling. + +### 5.2 Why per-paragraph texts in a list (not one giant LoroText) + +Decision (final — see §20): one `LoroText` per paragraph inside a `LoroMovableList`, **not** a single body text with `\n` separators. + +* The canonical op stream already speaks `(ParagraphId, utf8 byte)` — translation to `(container, offset)` is direct, with **no global offset arithmetic** anywhere. Global-offset bookkeeping across interleaved non-text blocks is where multi-week debugging cycles go to die. +* Blast radius: a translation bug corrupts one paragraph, not the document; per-paragraph regeneration makes remote apply trivially correct. +* Block move/insert/delete map 1:1 onto `LoroMovableList` ops (true move semantics, no delete+reinsert identity loss). +* Known accepted anomaly: if peer A splits a paragraph at offset k while peer B concurrently types at offset j > k in the same paragraph within one round-trip, B's text converges to the end of the first half rather than inside the new second paragraph. No data loss, positions slightly off, self-consistent on all peers. This is the standard per-block CRDT trade-off (Notion-class behavior). Document it in code; revisit only with evidence it bites in practice. + +### 5.3 Opaque block payloads (`"data"`) + +`Image`, `Equation`, `Table` blocks are **atomic last-writer-wins values** in v1. Justification from the codebase: flowtext itself records table/equation/image edits as whole-block replaces (`CanonicalOperation::ReplaceBlock` carries no finer payload — see `editor/tables.rs:174`, `table_equation_editing.rs:66,280,336`, `media.rs:164,224`, `object_selection.rs:250`). A cell-level table CRDT would exceed the granularity of the host editor's own edit stream — pure cost, no fidelity gain, until the editor itself gets finer-grained table ops. + +Payload type (in `schema.rs`): + +```rust +#[derive(Serialize, Deserialize)] +pub enum BlockPayload { // postcard-encoded into the "data" value + Image { asset_id: u128, mime: String, original_name: Option, + content_hash: u64, byte_len: u64, // asset BYTES ARE NOT IN THE CRDT (§16) + alt_text: String, caption: Option, + sizing: InputImageSizing, alignment: InputBlockAlignment }, + Equation { source: String, display: InputEquationDisplay }, // syntax: only Latex exists today + Table(InputTableBlock), +} +``` + +Conversions `Block ↔ BlockPayload` reuse the existing `Input*` types (`document/text.rs:105-189`) and whatever `InputBlock → Block` builders the persistence/clipboard paths already use (locate via `RichClipboardFragment` consumers in `edit_ops/rich_fragment.rs`; do not write a second converter). + +Concurrent edits to the same table = LWW on `"data"` (one peer's table edit wins; the other's is lost *for that block only*). The `"rev"` bump guarantees a map-diff event fires even if values were equal-ish. Acceptable v1 semantics; logged as a known limitation. + +### 5.4 Identity & the DocBinding + +Cross-peer identity = **loro container identity** (each block map / text container has a `ContainerID`). Flowtext's `ParagraphId`/`BlockId` are **peer-local** (they are generated fresh on load and never serialized to db8 — see `document_ids_for_shape` / `reconcile_document_ids` in `document/core.rs`). Never ship them over the wire; never store them in the CRDT. + +Each session maintains the bridge table (in `binding.rs`): + +```rust +pub struct DocBinding { + rows: Vec, // parallel to BOTH document.blocks and loro "blocks" list + by_paragraph: FxHashMap, // rebuilt on structural change + by_block: FxHashMap, + by_container: FxHashMap, +} +pub struct BindingRow { + pub map: LoroMap, // handle to blocks[i] + pub text: Option, // Some for kind=="p" + pub kind: BlockKind, // P | Image | Equation | Table + pub block_id: BlockId, // local flowtext id at this position + pub paragraph_id: Option, // local flowtext id (paragraph blocks only) +} +``` + +**Invariant (assert in debug builds after every local flush and every remote apply):** +`binding.rows.len() == document.blocks.len() == loro_blocks.len()`, `rows[i].kind` matches `document.blocks[i]`, `rows[i].paragraph_id == document.ids.paragraph_ids[paragraph_ordinal(i)]`, and `rows[i].block_id == document.ids.block_ids[i]`. + +When a *remote* change creates a block, the RemoteApplier generates **fresh local ids** (`new_paragraph_id()` / `new_block_id()`) and writes them into both `Document.ids` and the binding row. When a *local* edit creates one, the LocalApplier reads the id flowtext just minted (via `identity_map` / `document.ids`) and records it next to the new container. + +### 5.5 Offset discipline (read this twice) + +* flowtext: **UTF-8 byte offsets** everywhere (`DocumentOffset.byte`, run `len`s, canonical ops). +* loro `LoroText`: default API is **Unicode scalar positions**; UTF-8 variants exist (`mark_utf8`, `len_utf8`, `convert_pos` between `Utf8`/`Unicode`/`Utf16`/`Event` coordinate systems; insert/delete have unicode signatures — use the `*_utf8` variants where provided, otherwise `convert_pos` first; see §23-V3). +* loro **events** report positions in *event* coordinates (unicode by default). The RemoteApplier must convert event deltas to UTF-8 for selection remapping (`convert_pos(.., PosType::Event, PosType::Utf8)` or by walking the materialized string — pick one, write one helper, test it on multibyte text: `"é🌍\u{2028}x"`). +* Every module that crosses the boundary funnels through two helpers in `schema.rs` — `to_loro_pos(text, utf8_byte) -> loro_pos` and `to_utf8_byte(text, loro_pos) -> usize` — **no inline conversions anywhere else**. +* Debug assertion after each paragraph mutation: `text.len_utf8() == paragraph_text_len(paragraph)`. + +--- + +## 6. LocalApplier: `CanonicalOperation` → loro + +`local_apply.rs`. Entry point: + +```rust +pub struct LocalApplier<'doc> { doc: &'doc LoroDoc, binding: &'doc mut DocBinding } +impl LocalApplier<'_> { + /// Apply ONE editor edit-record's ops, then `doc.commit_with(origin = "local-edit")`. + /// One EditRecord == one loro commit == one undo unit. + pub fn apply(&mut self, document: &Document, ops: &[CanonicalOperation]) -> anyhow::Result<()>; +} +``` + +`document` is the **post-edit** flowtext document (canonical ops reference post-edit identity — `identity_map.reconcile` has already run). Per-op translation: + +| Canonical op | Loro translation | +|---|---| +| `InsertText { paragraph, byte, text, styles }` | row = `binding.by_paragraph[paragraph]`; `row.text.insert` at converted pos; then apply non-default marks over the inserted range (`sem`/`ul`/`strike`/`hl` per §5.1). Expansion is `None`, so a plain insert needs marking only when `styles != RunStyles::default()`. | +| `DeleteRange { start_paragraph, start_byte, end_paragraph, end_byte }` | same paragraph → single `delete`. Cross-paragraph (this is a **join**): (1) read tail delta of `end_paragraph` from `end_byte`; (2) delete `[start_byte..]` of start text; (3) delete the whole-paragraph containers strictly between start and end (movable-list `delete`); (4) append the saved tail (text+marks) to start text; (5) delete end container. Update binding rows + `Document`-side ids are already updated by flowtext. | +| `SplitParagraph { paragraph, byte, new_paragraph }` | (1) read tail delta `[byte..]`; (2) delete tail from original; (3) `blocks.insert_container(idx+1, LoroMap)` → set `"kind"="p"`, `"style"` copied from original; create `"text"` and insert tail delta with marks; (4) insert `BindingRow` with `paragraph_id = new_paragraph` and the `BlockId` flowtext minted (read from `document.ids.block_ids[idx+1]`). | +| `JoinParagraphs { first, second }` | degenerate `DeleteRange` (tail append + container delete), same code path. | +| `SetParagraphStyle { paragraph, style }` | `row.map.insert("style", encode(style))`. | +| `SetRunStyles { paragraph, range, styles }` | for each of the 4 keys: if target value non-default → `mark_utf8(range, key, value)`; else → `unmark` over the range (use the utf8 variant per §5.5). | +| `ReplaceParagraphSpan { start_paragraph, before, after }` | **workhorse**, see §6.1. | +| `InsertBlock { block, block_ix }` | read `document.blocks[block_ix]` (payload is NOT in the op); paragraph → create as in split; object → create map with `"kind"`,`"data"`(=postcard `BlockPayload`),`"rev"=0`. Insert binding row with `block_id = block`. | +| `DeleteBlock { block }` | movable-list delete at `binding.by_block[block]`; remove row. | +| `MoveBlock { block, new_block_ix }` | `blocks.mov(old_ix, new_block_ix)`; rotate binding rows. | +| `ReplaceBlock { block: Some(id) }` | re-serialize `document.blocks[ix]` → `map.insert("data", payload)`; `"rev" += 1`. `block: None` → resolve via the *single changed block* by comparing kinds/versions (flowtext bumps `Block::*.version` on edit) — if ambiguous, fall back to ReplaceDocument semantics. | +| `ReplaceDocument` | clear the movable list, rebuild every container from `document` (reuse `projection::populate_from_document`), rebuild binding. Rare (e.g. recovery/import paths, `block_insertion.rs:309`, `object_selection.rs:461`). | + +### 6.1 `ReplaceParagraphSpan` translation + +flowtext wraps most compound edits (paste, backspace at boundary, IME, style commands across paragraphs, drag-drop) into one coarse op carrying full `before`/`after` `DocumentSpan`s (the captured range deliberately includes ±1 neighbor paragraphs — `edit_pipeline.rs:140 edit_capture_range`). Algorithm: + +1. `start = binding.by_paragraph[start_paragraph]` (fallback: `before.start_paragraph` index — keep the same fallback the existing `apply_canonical_operation` uses, `lifecycle.rs:414`). +2. Positional pairing: paragraphs `0..min(n_before, n_after)` are *matched*; surplus `after` paragraphs are *inserted* at the end of the span; surplus `before` paragraphs are *deleted*. (This matches flowtext's own positional identity reconciliation, and because spans include unchanged neighbor paragraphs, edge pairs typically no-op.) +3. For each matched pair `(b, a)`: + * if text and runs are byte-identical → skip; + * else: `row.text.update(a_text, UpdateOptions)` (loro's built-in Myers diff produces minimal CRDT ops); then reconcile marks: compute the desired mark intervals from `a.runs`; clear-and-remark only if runs differ from `b.runs` (compare run vectors; paragraphs are small — full remark of one paragraph is acceptable and *simple*); update `"style"` if changed. +4. Inserted paragraphs: create containers as in `SplitParagraph`, reading `style`/runs/text from the `after` span, binding to the ids flowtext minted at those indices (`document.ids.paragraph_ids[start + i]`). +5. Deleted paragraphs: movable-list delete + drop rows. +6. Debug-assert binding invariant (§5.4). + +**Correctness note:** `text.update()` + concurrent remote edits to the same paragraph can interleave at diff granularity rather than keystroke granularity. That is normal CRDT behavior (same class as Google Docs' transform granularity) and converges; do not try to be cleverer. + +### 6.2 The double-apply trap (must implement exactly) + +`insert_single_grapheme_fast_path` (`edit_pipeline.rs:44-73`) **merges** consecutive single-grapheme inserts into the *previous* undo record by mutating its `canonical_operations[0].text` in place. If the collab layer naively re-reads "the last record's ops" after each notify, the second keystroke would re-apply the whole accumulated string ("h", then "he", then "hel"…). + +Fix (part of M3's editor changes): the editor gets an explicit **pending collab queue** that receives exactly the delta of each mutation: + +```rust +// editor/mod.rs additions +pending_collab_edits: Vec, // drained by the session +``` + +* In the fast path: push `CollaborationEdit { operations: vec![InsertText{ …, text: just_this_grapheme }] }` regardless of undo-record merging. +* In `mark_document_changed_with_reconcile`: push the new record's canonical ops **only when a new record was pushed this call** (pass the ops in, don't re-read `undo_stack.last()`). +* Undo/redo paths must **not** push (they're handled by the loro UndoManager, §8) — gate pushes on "not currently restoring history" (`after_history_restore` path) and "not currently applying remote" (a `suppress_collab_capture: u32` counter the RemoteApplier increments, mirroring the existing `suppress_mutation_notify` pattern). +* `take_pending_collab_edits() -> Vec` drains. Keep `last_collaboration_edit` accessors untouched for compatibility. + +Add a unit test in gpui-flowtext: type "a","b","c" via the fast path → drained queue is exactly three 1-char `InsertText` ops. + +--- + +## 7. RemoteApplier: loro diff → flowtext `Document` + +`remote_apply.rs` (pure: produces patches) + `editor/collab_apply.rs` (applies them via editor internals). + +Subscribe once per session: `doc.subscribe_root(...)` (or `doc.subscribe(&blocks_container_id, ...)`) — the callback receives `DiffEvent`s **synchronously during `import()`/undo**, on the calling thread (main thread — good). Filter by trigger: only process events whose trigger is *Import* or *Checkout/Undo*; *Local* events are echoes of the LocalApplier and must be ignored (§23-V4 for the exact enum name). + +Collect into a patch list (pure data, unit-testable): + +```rust +pub enum DocPatch { + ParagraphText { row: usize, new: InputParagraph, // full regenerated paragraph + delta_utf8: Vec }, // retain/insert/delete in utf8 bytes, for selection remap + ParagraphStyle { row: usize, style: ParagraphStyle }, + BlockData { row: usize, payload: BlockPayload }, + Insert { row: usize, blocks: Vec }, // built from new containers + Delete { row: usize, count: usize }, + Move { from: usize, to: usize }, +} +``` + +Mapping rules: + +* **Text diff on a paragraph container** → regenerate that paragraph wholesale from loro (`to_delta()` → `InputParagraph`) — never incremental-patch flowtext runs from deltas (regeneration is O(paragraph) and immune to drift). Also emit the utf8-converted delta for caret remapping. +* **Movable-list diff** → `Insert`/`Delete`/`Move` patches; for inserts, build `InputBlock`s by reading the new containers (`projection::input_block_from_container`). +* **Map diff** with `"style"` → `ParagraphStyle`; with `"data"`/`"rev"` → `BlockData`. + +`collab_apply.rs` (editor side) applies a batch: + +```rust +impl RichTextEditor { + pub fn apply_collab_patches(&mut self, patches: &[DocPatch], cx: &mut Context) { … } +} +``` + +Implementation requirements (use existing machinery, don't reinvent): + +1. Wrap the whole batch in `suppress_collab_capture += 1` (so these mutations never echo back) and reuse the `layout_invalidation_hint` + `after_text_mutation(cx)` pattern from `apply_remote_operations` (`lifecycle.rs:265-272`) for cache invalidation. +2. `ParagraphText`: build the replacement via existing span machinery — `capture_document_span` + `apply_document_span_replacement` for the single paragraph, or a focused `replace_paragraph_content(document, ix, InputParagraph)` helper added to `edit_ops` (new file `edit_ops/collab.rs` if it doesn't fit an existing one). Preserve `ParagraphId` (identity unchanged), bump paragraph `version`. +3. Structural patches: splice `document.blocks`/`paragraphs`/`ids` with **explicitly provided ids** (from the binding rows the RemoteApplier created) — do not let `reconcile_document_ids` invent them (call it only to assert no-op afterwards in debug). +4. **Selection remap:** if `self.selection` head/anchor sits in a patched paragraph, walk `delta_utf8` (retain/insert advance, delete clamps) to the new byte offset; if its paragraph was deleted, clamp to the nearest surviving offset. Same for `table_cell_*`/`equation_source_*` editing offsets: if the active table/equation block got `BlockData`-replaced remotely, exit the cell/equation editing mode gracefully (clear `selected_block`, keep selection at block boundary). +5. **Undo safety:** remote patches invalidate positional undo records. Per §8, during a session the local undo stack is unused (undo routes through loro). On session start, clear `undo_stack`/`redo_stack`; in `apply_collab_patches`, debug-assert they stay empty. +6. **Deferral:** if `self.selecting || self.active_text_drag.is_some() || self.image_resize_drag.is_some() || self.table_column_resize_drag.is_some()` or IME composition is active (marked-text state in `editor/platform.rs`), the session queues patches and retries on the next observe tick / 16 ms timer instead of mutating mid-gesture. + +### 7.1 Bootstrap & full resync + +Join: import host snapshot into a fresh configured `LoroDoc` (background executor — `LoroDoc` is Send), then on the main thread run `projection::document_from_loro(&doc, theme)` → `Vec` → build `Document` (existing input-builder path) → `editor.replace_document_from_collaboration(document, cx)` → build `DocBinding` by walking containers. Resync (integrity failure or reconnect-too-stale): identical path, plus selection clamp. + +`projection.rs` also provides the inverse, `populate_from_document(&LoroDoc, &Document)` — used by the **host once at session start** (and by `ReplaceDocument`). Round-trip property test: `document → loro → document` is identity on (text, runs, styles, block structure, table/equation/image payloads), modulo `AssetRecord.bytes` (§16). + +### 7.2 Integrity checking (belt & suspenders) + +`integrity.rs`: `projection_hash(&Document) -> u64` — twox-hash over (paragraph styles, run vectors, paragraph texts, block kinds + payload bytes, in order; explicitly NOT ids/theme/sections/offset-index). Host piggybacks `Frame::IntegrityProbe { vv: Vec, hash: u64 }` every 30 s of activity or every 256 relayed updates. Joiner compares **only when version vectors are equal** (otherwise skip — they're mid-flight); on mismatch → log loudly, send `Frame::ResyncRequest`, host responds with fresh snapshot, joiner rebuilds (§7.1) and posts a non-blocking notification ("Document resynced"). This converts any residual translation bug from "silent divergence forever" into "blip + log we can fix." + +--- + +## 8. Undo/redo during a session + +Positional `EditRecord` replay is unsound once remote edits interleave. While a session is active: + +* Session creates `loro::UndoManager::new(&doc)` (tracks only this peer's ops — exactly Google-Docs semantics), with merge interval ≈ 500 ms to mirror flowtext's grapheme coalescing; configure `set_max_undo_steps(undo-stack depth ~ 300)`. +* Editor gains a redirect hook (set by the session, cleared on session end): + +```rust +// editor/mod.rs +pub enum UndoRedirect { Undo, Redo } +collab_undo_redirect: Option>, // Rc: entity is single-threaded +``` + +`commands.rs:191 undo()` / `:205 redo()`: if redirect set → invoke it and return. The session handler calls `undo_manager.undo()` / `.redo()`; resulting doc changes surface through the same DiffEvent → `DocPatch` → `apply_collab_patches` pipeline (trigger = checkout/undo, so they pass the §7 filter; ensure the LocalApplier echo-suppression doesn't eat them). +* Caret restore: wire `UndoManager`'s cursor callbacks (`set_on_push`/`set_on_pop` storing the selection as two loro `Cursor`s; resolve with `doc.get_cursor_pos` after pop). If the cursor API fights back, v1 fallback: leave selection where the patch remap puts it. (§23-V5) +* On session start: clear editor undo/redo stacks. On session end: stacks start fresh from the final state (history across the boundary is intentionally dropped — banner copy mentions it). +* Host before session / after session uses the existing native undo unchanged. + +--- + +## 9. Presence (carets, names, roster) + +`presence.rs` wraps `loro::awareness::EphemeralStore` (timestamped LWW KV with per-key timeout, made for exactly this): + +* Key = peer's `EndpointId` string. Value (postcard → loro binary value): + +```rust +pub struct PresenceState { + pub name: String, // display name + pub color_ix: u8, // palette index assigned by host in Welcome + pub role: Role, // Owner | Editor | Viewer (informational; enforcement is host-side) + pub selection: Option, +} +pub struct PresenceSelection { + pub container: String, // ContainerID of the paragraph text (head) + pub head: Vec, // loro Cursor::encode() — survives concurrent edits + pub anchor_container: String, + pub anchor: Vec, +} +``` + +* Timeout 30 s; the store's local-update subscription yields raw bytes → `Frame::Ephemeral(bytes)`; inbound frames → `store.apply(bytes)`; host relays ephemeral frames like doc updates. +* Update cadence: on `EditorEvent::SelectionChanged` (subscribe via `cx.subscribe` — `RichTextEditor` must be an `EventEmitter`; if it currently only pushes to `EditorEventSink`, add the `EventEmitter` impl in M3), debounced 50 ms; plus a 10 s keepalive refresh. +* Rendering: on ephemeral merge events + after every remote apply, resolve each peer's cursors: `Cursor::decode` → `doc.get_cursor_pos` → unicode pos → utf8 byte → `binding.by_container` → `DocumentOffset` → `editor.set_external_carets(vec![ExternalCaret { offset, color_rgb }], cx)`. v1 renders caret + name-on-hover only (selection-range highlight is v1.1; `ExternalCaret` keeps its current shape). +* Color palette: 8 hardcoded high-contrast RGBs in `session_core.rs`; host assigns by join order, reuses freed slots. + +--- + +## 10–13. Networking, protocol, tickets, lifecycle + +### 10.1 Runtime bridge (`net/runtime.rs`) + +* Lazy global: first share/join spawns `std::thread::Builder::new().name("flowstate-collab-net")` running a `tokio::runtime::Builder::new_multi_thread().worker_threads(2).enable_all()` runtime; thread parks on `runtime.block_on(net_main(cmd_rx, evt_tx))`. +* `net_main` builds **one iroh `Endpoint` for the whole app** (shared by hosting and joining): + +```rust +let endpoint = Endpoint::builder() // use the n0 preset: relays + address lookup + .alpns(vec![ALPN.to_vec()]) // ALPN = b"flowstate/collab/0" + /* preset/address-lookup wiring per §23-V1 */ + .bind().await?; +``` + +Discovery is NOT on by default in iroh 0.98 — apply the n0 preset (`presets::N0` / `Builder::new(preset)`) so relay + address lookup work; additionally the ticket embeds the full `EndpointAddr` (relay URL + direct addrs) so joining works even if lookup is cold (§23-V1). +* Channels: `async_channel::unbounded::()` (UI→net) and `::()` (net→UI). The UI side pumps events with a detached `cx.spawn` loop in `CollabManager` that routes to the right `CollabSession` entity. + +```rust +pub enum NetCommand { + HostSession { session: SessionId, token: SessionToken, default_role: Role, reply: oneshot }, + CloseSession { session: SessionId }, // host: broadcast Bye, close conns + Join { ticket: SessionTicket, hello: HelloInfo }, + LeaveSession { session: SessionId }, + SendTo { session: SessionId, peer: EndpointId, frames: Vec }, + Broadcast { session: SessionId, frame: Frame, except: Option }, + Shutdown, +} +pub enum NetEvent { + HostReady { session: SessionId, ticket: SessionTicket }, + PeerConnected { session: SessionId, peer: EndpointId, hello: HelloInfo }, // host side, post-auth + Frame { session: SessionId, peer: EndpointId, frame: Frame }, + PeerDisconnected{ session: SessionId, peer: EndpointId, reason: String }, + JoinEstablished { session: SessionId }, // joiner side, post-Welcome (Welcome arrives as Frame) + JoinFailed { session: SessionId, reason: JoinError }, + Reconnecting { session: SessionId, attempt: u32 }, + SessionClosed { session: SessionId, reason: CloseReason }, +} +``` + +### 10.2 Host accept path (`net/host.rs`) + +`Router::builder(endpoint).accept(ALPN, FlowstateProtocol{state}).spawn()` with `impl ProtocolHandler` (`async fn accept(&self, connection: Connection) -> Result<(), AcceptError>`). Per connection: `accept_bi()` → read `Frame::Hello` (5 s deadline) → validate `protocol_version`, `session_id` exists, `token` matches (constant-time compare), peer not already connected, roster < MAX_PEERS(8) → register peer (writer task owns the send half + an mpsc inbox; reader task forwards frames as `NetEvent::Frame`) → emit `PeerConnected`. The **session entity** (not the net thread) decides the `Welcome` content because the snapshot must be exported atomically with update-stream attachment: on `PeerConnected`, the session synchronously (a) exports `ExportMode::Snapshot` (or `updates(vv)` for a reconnect Hello carrying `last_vv`), (b) marks the peer live for future broadcasts — both under the same borrow of the session state so no update can fall between snapshot and stream. Then `SendTo(Welcome{...})`. + +Relay rule: every authenticated inbound `DocUpdate`/`Ephemeral` from peer P → import/apply on host AND `Broadcast{except: P}` *of the same bytes*. Updates from a `Viewer` peer: **dropped, not imported, not relayed** (and the host replies `Frame::Rejected{reason: ReadOnly}` so a buggy client can self-correct via resync). + +### 10.3 Joiner path (`net/joiner.rs`) + +`endpoint.connect(ticket.addr, ALPN)` → `open_bi()` → send `Hello` → await `Welcome` (15 s deadline) → emit `JoinEstablished` + deliver Welcome frame → then a read loop + write loop. Reconnect: on connection loss (not explicit `Bye`/`SessionClosed`), loop with backoff 0.5/1/2/4/8…30 s; each attempt sends `Hello { last_vv: Some(doc.oplog_vv().encode()) }`; host answers `Welcome { doc: Updates(...) }` (cheap catch-up) or `Snapshot` if it can't serve the delta. UI shows "Reconnecting…" via `NetEvent::Reconnecting`; user can cancel (→ orphan copy). + +### 11. Wire protocol (`protocol.rs`) + +Framing (`net/framing.rs`): `u32-le length` + postcard bytes, max frame 2 MiB (`FrameTooLarge` error closes the connection); one long-lived bi-stream per peer for control/doc/presence; assets use separate per-transfer bi-streams (§16). + +```rust +pub const PROTOCOL_VERSION: u16 = 1; +pub const ALPN: &[u8] = b"flowstate/collab/0"; + +#[derive(Serialize, Deserialize)] +pub enum Frame { + Hello { protocol: u16, app: String, session: SessionId, token: SessionToken, + name: String, last_vv: Option> }, + Welcome { roster: Vec, your_color: u8, your_role: Role, + doc: DocBootstrap /* Snapshot(Vec) | Updates(Vec) */, + asset_manifest: Vec }, + Rejected { reason: RejectReason }, // BadToken | Version{min,got} | Full | ReadOnly | UnknownSession + DocUpdate(Vec), // loro export bytes (local update or relay) + Ephemeral(Vec), // EphemeralStore encode bytes + IntegrityProbe { vv: Vec, hash: u64 }, + ResyncRequest, + ResyncSnapshot(Vec), + AssetRequest { id: u128 }, // answered on a NEW bi-stream with AssetHeader+chunks + AssetOffer { meta: AssetMeta }, // joiner → host: "I have a new asset, pull it" + RosterUpdate { roster: Vec }, + Bye { reason: CloseReason }, // SessionEnded | Kicked | HostShutdown + Ping(u64), Pong(u64), // 15 s heartbeat (also drives presence staleness) +} +``` + +Version policy: exact `PROTOCOL_VERSION` match required; mismatch → `Rejected::Version` with a human-readable "host is running a newer/older Flowstate" surfaced in the join dialog. + +### 12. Ticket (`ticket.rs`) + +```rust +pub struct SessionTicket { + pub addr: EndpointAddr, // includes relay URL + direct addresses + pub session: SessionId, // u128 random + pub token: SessionToken, // [u8; 16] random (rand::rngs::OsRng) — the join secret + pub title: String, // doc display name, shown in join dialog before connecting +} +``` + +Implement the `iroh-tickets` `Ticket` trait (postcard payload, base32 display) with prefix/kind `"fscollab"`. The string is the only thing users exchange. Threat model note for implementers: the token gates membership; anyone with the ticket can join until the host ends the session — that *is* the product. The host's authority means a malicious joiner can at worst vandalize the shared doc (undoable; host can end session), never touch the host FS. + +### 13. Session lifecycle state machines + +`crates/flowstate/src/collab/session.rs` — `CollabSession` GPUI entity, one per panel, states: + +``` +Host: Idle → Starting(net spawn + populate loro from Document + HostSession cmd) + → Live { ticket, peers } ⇄ (peer churn) + → Ended (user ends / tab closes / app quits → broadcast Bye, drop) +Joiner: Connecting(ticket) → Bootstrapping(snapshot import, build Document, open tab) + → Live ⇄ Reconnecting(attempt n) + → Orphaned(reason) // session over; editor stays, banner shown, collab hooks removed +Errors: Starting/Connecting failures → toast + back to Idle (no tab opened for failed join) +``` + +Wiring on session start (host) / bootstrap (joiner): +1. `editor.set_collaboration_role(Some(role))`, clear undo stacks, set undo redirect (§8). +2. `cx.observe(editor)` → `flush_local_edits` (drain → LocalApplier → commit; the loro local-update subscription then emits broadcast bytes). +3. `cx.subscribe(editor)` for `SelectionChanged` → presence (§9). +4. Register session with `CollabManager` (global) for net-event routing by `SessionId`. +Teardown reverses everything: role `None` (host keeps `Owner`→`None`), redirect cleared, carets cleared, subscriptions dropped. Closing a hosting tab (`remove_document_panel`, `documents.rs:216`) and `request_close_window` must check `CollabManager::session_for_panel(id)` and confirm: "End live session with N participants?". + +--- + +## 14. Save semantics & source of truth + +* **Host:** unchanged save/autosave (`save_active`, `maybe_autosave_document`). Its `Document` projection *is* the file content. CRDT history is **not** persisted into `.db8` — a saved file is a plain snapshot; a future session starts a fresh loro doc (`populate_from_document`). No format changes. +* **Joiner:** editor opens with `document_path = None` → Save triggers Save-As (existing untitled flow), autosave skips, recovery writes skip. The panel title = host's title + " (shared)". +* **Host recovery files** (`schedule_recovery_write`) keep working — they snapshot the projection, which is exactly right. + +--- + +## 15. Editor changes (gpui-flowtext) — complete enumerated list + +All hidden behind no feature flag (inert without a session). Keep each change minimal; **no loro/iroh types may appear in this crate.** + +1. `editor/mod.rs`: add fields `pending_collab_edits: Vec`, `suppress_collab_capture: u32`, `collab_undo_redirect: Option>`; add `UndoRedirect` enum. (~20 LOC) +2. `edit_pipeline.rs`: per §6.2 — push exact deltas into `pending_collab_edits` (fast path pushes the single grapheme; `mark_document_changed_with_reconcile` gains an `ops: Option<&[CanonicalOperation]>` parameter or a small wrapper so it only pushes when a new record was created and `suppress_collab_capture == 0`). (~40 LOC) +3. `lifecycle.rs`: `take_pending_collab_edits()`, `set_collab_undo_redirect`, `clear_collab_session_state` (clears queue+redirect+external carets). Clear the queue in `dispose_for_close`/`release_transient_memory`. (~40 LOC) +4. `commands.rs` `undo()`/`redo()`: redirect check first. (~10 LOC) +5. **New** `editor/collab_apply.rs`: `apply_collab_patches` + selection remap helpers per §7 (≤600 LOC). Reuses `edit_ops` primitives; add `edit_ops/collab.rs` only if a `replace_paragraph_content` helper doesn't fit existing files. +6. `api.rs`/events: ensure `impl EventEmitter for RichTextEditor` exists so `cx.subscribe` works (it may already; verify). `EditorEvent` already carries `SelectionChanged`. +7. `DocPatch` lives in `flowstate-collab`; to keep the dependency direction (editor must not depend on collab), `apply_collab_patches` takes a **gpui-flowtext-owned** patch type: define `CollabPatch` (mirror of §7's `DocPatch`) in `collaboration.rs` (it already holds the collab vocabulary) and have `flowstate-collab` emit that type. (`flowstate-collab` already depends on gpui-flowtext — clean.) +8. Image placeholder: in the image layout/measure path (`rich_text/editor/media.rs` + `layout/block_layout.rs`), treat an `AssetRecord` with empty `bytes` as "loading": fixed 240×160 placeholder box (theme muted bg + spinner glyph). Triggered while an asset transfer is in flight (§16). (~40 LOC) + +Everything else (roles gating, external carets, replace-document) already exists. + +--- + +## 16. Assets (images) over the wire + +Asset bytes never enter the CRDT (snapshots would carry every image forever). Flow: + +* `BlockPayload::Image` carries `asset_id`, `mime`, `content_hash`, `byte_len` (metadata only). +* Host is the **asset authority**: `Welcome.asset_manifest` lists all assets; joiner requests the ones referenced by visible blocks first (then the rest lazily) via `Frame::AssetRequest` → host opens a fresh bi-stream: postcard `AssetHeader { id, mime, len, hash }` then raw 256 KiB chunks → joiner inserts `AssetRecord` into `document.assets` (via a small `collab_apply` patch `AssetArrived { id, record }`) → repaint replaces placeholder. +* A joiner pasting an image: image lands in its local `Document.assets` (existing paste path), block syncs via CRDT with metadata; joiner sends `AssetOffer`; host `AssetRequest`s it back over a stream, stores it, and other peers fetch from host on demand. (Star topology keeps transfers host-centric — no peer-to-peer asset mesh.) +* Dedup by `content_hash` is an optimization, not v1. + +--- + +## 17. UI specification (gpui-component widgets) + +1. **Commands** (`commands.rs`): add `CommandId::ShareDocument` ("Share / Collaborate…", APP context, no default key) and `CommandId::JoinSession` ("Join Collaboration Session…", APP). Add to `COMMAND_SPECS`; wire dispatch in the workspace command handler (`workspace/workspace/keybindings.rs` routing). +2. **Top bar** (`top_bar.rs`): File menu gains "Share Document…" (enabled when active panel is a rich-text doc) and "Join Session…". Plus a dedicated share `icon_button` on the `TitleBar` right cluster (next to settings) showing a filled/colored state while a session is live. +3. **Share dialog** (`collab/share_dialog.rs`, gpui-component `Dialog` + `Input` + `Button` + `clipboard` copy button + `avatar::AvatarGroup` + `switch` for Editor/Viewer default role): + * Idle: role switch + "Start session". + * Live (host): read-only `Input` with ticket string + Copy button; participant list (color dot, name, role); "End session" (danger variant, confirm). + * Join tab: `Input` (paste ticket) + "Join" with inline validation (parse before dialing; show doc title from ticket); progress states Connecting → Receiving document (n KiB) → done (dialog closes, tab opens). +4. **Status bar** (`render_status.rs`): when the *active* panel has a session: pill `● Hosting · 3` / `● Connected` / `◌ Reconnecting…` / `● Session ended`, colored by state; click opens the share dialog. Follow the zoom/status element patterns already in that file. +5. **Tab badge** (`document_panel.rs` render): small colored dot + peer count on collaborating tabs. +6. **Notifications** (gpui-component `notification.rs`): peer joined/left, resync occurred, read-only edit attempt ("You're a viewer in this session"). +7. **Orphan banner**: when a joiner session ends, a slim banner above the editor: "Session ended — this is your local copy. Save As to keep it." with a Save As button. Implement as a `DocumentPanel` child element gated on session state. +8. Errors: join failures via dialog inline error; mid-session fatal errors via `window.prompt` only if the user must act, otherwise notification + status pill. + +--- + +## 18. Failure-mode catalog (each requires explicit handling + a test or QA item) + +| # | Scenario | Required behavior | +|---|---|---| +| F1 | Garbage/truncated ticket pasted | Parse error inline in dialog; never dials | +| F2 | Host unreachable (offline, NAT+relay down) | Connect timeout 15 s → JoinFailed with actionable text | +| F3 | Token wrong / session unknown (stale ticket) | `Rejected{BadToken/UnknownSession}` → "Session no longer available" | +| F4 | Version skew | `Rejected{Version}` both directions, clear message | +| F5 | Host ends session / closes tab / quits app | `Bye{SessionEnded/HostShutdown}` (best-effort on quit) → joiners orphan with banner; net thread closes conns | +| F6 | Joiner vanishes (crash, network) | host: `connection.closed()` → roster update + notification; presence entry times out | +| F7 | Transient disconnect | joiner auto-reconnect w/ `last_vv` delta catch-up; edits made while offline are queued in loro and sync on reconnect (CRDT gives offline-tolerance for free) | +| F8 | Reconnect after host restarted session (new SessionId) | `UnknownSession` → orphan, do not silently join a different session | +| F9 | Self-join (pasting own ticket) | detect `ticket.addr.endpoint_id == endpoint.id()` → friendly error | +| F10 | Session full (>8) | `Rejected{Full}` | +| F11 | Viewer client sends DocUpdate (bug/malice) | host drops + `Rejected{ReadOnly}`; never relayed | +| F12 | Integrity hash mismatch | auto-resync (§7.2), notification, `eprintln!` diagnostics incl. both hashes + vv | +| F13 | Giant document snapshot | frames ≤2 MiB → snapshot sent as one frame up to that, else chunked `ResyncSnapshot`-style continuation (add `DocBootstrap::SnapshotChunk{n,of}` if needed); progress UI | +| F14 | Remote patch arrives mid-drag/IME | deferral queue (§7 item 6) | +| F15 | Asset referenced but transfer fails | placeholder persists + retry on next request; never blocks text sync | +| F16 | Two sessions on one doc / share twice | second Share opens existing session dialog (sessions keyed by panel Uuid) | +| F17 | Joiner opens Share on a joined doc | allowed to *view* roster, cannot start nested session (button disabled, tooltip) | +| F18 | App quit while hosting | window-close confirm includes session warning (extends existing dirty-check prompt flow in `documents.rs:734`) | + +--- + +## 19. Performance budgets + +* Keystroke→local paint: **zero added frame cost** — LocalApplier work happens in the observe callback after paint scheduling; loro insert ≈ 1–5 µs. Assert no network I/O on the main thread (channel sends only). +* Remote keystroke→repaint: ≤ 1 frame after import; per-paragraph regeneration only for touched paragraphs (the existing `layout_invalidation_hint` machinery keeps relayout scoped). +* Join bootstrap on a 5 MB doc: snapshot export+import off-main-thread; only `replace_document_from_collaboration` on-thread (same cost as opening the file — already acceptable). +* Memory: LoroDoc history grows with session length; fine for sessions (hours), irrelevant after close (dropped). No persistence. +* Add a micro-benchmark (optional, `rich_text/benchmarks/` style) only if M6 profiling shows surprises; do not pre-optimize. + +--- + +## 20. Decisions already made — DO NOT relitigate + +1. **loro schema = per-paragraph `LoroText` in a `LoroMovableList`** (§5.2), not a single body text. Accepted split-race anomaly documented. +2. **Tables/equations/images = atomic LWW payloads** matching the editor's own `ReplaceBlock` granularity (§5.3). +3. **Star topology, host authority, host relays raw update bytes.** No iroh-gossip, no mesh, no host migration in v1. +4. **CRDT lives on the main thread inside the session entity**; tokio thread is transport-only. +5. **Wire = loro update bytes**, never `CanonicalOperation`s (the existing `WireCanonicalOperation` codec stays unused). +6. **Theme/zoom/invisibility are local-only**; never synced. +7. **Undo in-session = loro UndoManager** with editor-level redirect; native undo stack cleared at session boundaries. +8. **Asset bytes out-of-band** over dedicated streams; host is asset authority. +9. **Joiners have no `document_path`**; host file is the only source of truth; CRDT history never touches `.db8`. +10. **Flow (`.fl0`) documents excluded from v1.** +11. New core logic in **`flowstate-collab`** crate (GPUI-free); GPUI glue in `crates/flowstate/src/collab/`; editor crate stays loro/iroh-free. + +Deferred (tracked, not designed here): per-peer role changes + kick, selection-range rendering for remote peers, cell-level table CRDT, host migration, ticket QR codes, session persistence across host restarts, >8 peers, e2e-encryption hardening beyond QUIC+token. + +--- + +## 21. Testing strategy + +### 21.1 Unit / golden (in `flowstate-collab`) +* `translation.rs`: for every `CanonicalOperation` variant: apply to a `Document` fixture via the editor's real edit ops (construct via `document_from_input`), run LocalApplier, project loro back, assert equality with the mutated `Document`. Include multibyte text (`"é🌍"`, soft line breaks) in every text fixture. +* Projection round-trip property test (§7.1). +* Offset conversion helpers: exhaustive over a multibyte fixture. +* §6.2 double-apply regression test (in gpui-flowtext). + +### 21.2 Convergence fuzz (THE gate for M2 — do not ship without it green) +`tests/convergence.rs` (proptest): N ∈ {2,3} simulated peers, each owning a `(Document, LoroDoc, DocBinding)` triple. Random op program per peer (insert/delete/split/join/style/block-insert/move/replace, weighted toward text); ops applied through LocalApplier; update bytes exchanged through a virtual network with random delay/reorder/duplication (loro tolerates all three); RemoteApplier applies inbound diffs to each peer's `Document`. After quiescence: **all projections byte-identical** (text, runs, styles, structure) and equal to a fresh projection of any peer's LoroDoc. Shrinkable failures print the op program. + +### 21.3 Loopback integration (`tests/loopback.rs`) +Real iroh endpoints, localhost direct addrs, `RelayMode::Disabled`, no discovery: host + 2 joiners; assert handshake, welcome bootstrap, typed-edit relay, viewer rejection, reconnect-with-delta (kill joiner conn, redial with vv), Bye propagation. Mark `#[ignore]` if CI sandboxing blocks UDP; runnable locally. + +### 21.4 Manual QA script (commit as `helpers/docs/collab_qa.md`) +Two app instances (`cargo run` ×2, different cwd): the 18 failure modes of §18 plus: simultaneous typing same paragraph, Enter-split race, table edit both sides (LWW visible), image paste joiner-side, undo tug-of-war, 30-minute soak with autosave on. + +--- + +## 22. Milestones & work breakdown (for the orchestrating agent) + +Dependency graph: `M1 ∥ M2` → `M3` → `M4` → `M5` → `M6`. Each milestone ends with `cargo clippy` clean (workspace lints) + its tests green. + +**M0 — Scaffolding (small, do first, single agent)** +`cargo new crates/flowstate-collab --lib`; wire workspace deps (§4); empty modules compiling; `ALPN`/`PROTOCOL_VERSION` constants; CI keeps passing. + +**M1 — Networking core (1 agent)** — `net/*`, `protocol.rs`, `ticket.rs`, framing. Deliverable: loopback test (§21.3) passing with an echo-level session (no loro yet — frames round-trip, auth verified, relay fan-out works, reconnect works). Verify §23-V1/V2 against docs.rs first. + +**M2 — Document sync core (1–2 agents; the hard one)** — `schema.rs`, `binding.rs`, `projection.rs`, `local_apply.rs`, `remote_apply.rs`, `integrity.rs`, plus the gpui-flowtext `pending_collab_edits` change (§6.2) it needs for fixtures. Deliverables: §21.1 goldens + §21.2 convergence fuzz green. Start with text-only ops, then structure ops, then blocks. Verify §23-V3/V4 first. + +**M3 — Editor integration (1 agent, after M2)** — §15 items 1–8 (`collab_apply.rs`, undo redirect, deferral, selection remap, placeholder), §8 UndoManager wiring, EventEmitter check. Deliverable: headless GPUI test (`gpui::TestAppContext` if available in gpui 0.2; otherwise unit-test patch application directly on `RichTextEditor` via `cx.new`) applying remote patches around an active selection. + +**M4 — Session orchestration (1 agent)** — `collab/mod.rs` (CollabManager global + net-event pump), `collab/session.rs` (state machines §13, flush/import paths, presence §9, assets §16, resync §7.2), workspace hooks (close-tab/close-window confirms, joiner tab creation). Deliverable: two real app instances collaborate end-to-end (manual), F5/F6/F7 verified. + +**M5 — UI (1 agent, parallel with late M4)** — §17 complete: commands, menus, share dialog, status pill, tab badge, notifications, orphan banner. + +**M6 — Hardening (1 agent)** — full §18 sweep, §21.4 QA script run + fixes, perf sanity (§19), `helpers/docs/collaboration.md` (user-facing how-to + architecture notes for maintainers), final clippy/test pass. + +Acceptance for the whole feature = QA script passes on Linux (primary) with two instances, including one relay-path run (different networks or forced `RelayMode::Relay`). + +--- + +## 23. API verification appendix — check these exact names before coding + +The architecture does not depend on any of these spellings; they are listed so implementers verify rather than guess: + +* **V1 (iroh, M1):** discovery/preset wiring on `endpoint::Builder` in 0.98 — the docs show `Builder::new(preset)` / `preset()` / `address_lookup()` and `presets::N0`; relays default to n0 via `RelayMode::Default`, but **address lookup is off by default**. Confirm the preset that enables n0 DNS lookup + pkarr publish, and how to read `endpoint.addr()` for tickets (may need to await first addr / `online()` watcher). docs.rs/iroh/0.98.2. +* **V2 (iroh-tickets, M1):** exact `Ticket` trait path + required methods (kind/prefix, `to_bytes`/`from_bytes`), and `EndpointTicket` as reference implementation. docs.rs/iroh-tickets. +* **V3 (loro, M2):** UTF-8 variants available on `LoroText`: confirmed `mark_utf8`, `unmark_utf8?`, `len_utf8`, `convert_pos(index, PosType, PosType)`; check whether `insert_utf8`/`delete_utf8`/`splice_utf8` exist (if not, convert via `convert_pos`). Also `UpdateOptions` fields for `text.update`. docs.rs/loro/1.13. +* **V4 (loro, M2):** event subscription: `doc.subscribe_root` vs `doc.subscribe(&ContainerID)`; `DiffEvent` shape (container path, `Diff::{Text, List, Map}` payloads) and the trigger discriminator (`EventTriggerKind::{Local, Import, Checkout}` or similar) used to filter local echoes; `subscribe_local_update` callback signature (`&[u8]` vs `Vec`, and whether it fires per-commit). +* **V5 (loro, M3):** `UndoManager` construction + `set_on_push`/`set_on_pop` cursor types; `Cursor::encode/decode` (or `to_bytes`) for presence; `doc.get_cursor_pos` return shape. +* **V6 (loro, M2):** `LoroMovableList::{insert_container, delete, mov}` exact names; `LoroMap::{insert, insert_container, get}`; binary values (`LoroValue::Binary`) for `"data"`. +* **V7 (loro, M4):** `awareness::EphemeralStore`: constructor timeout param, `set_local_state`-equivalent (`set(key, value)`?), `encode/encode_all/apply`, subscription signatures (`subscribe_local_updates` yields outbound bytes; `subscribe` yields merge events), `remove_outdated`. +* **V8 (gpui 0.2, M3/M4):** `EventEmitter` impl pattern for `RichTextEditor`; `cx.set_global`/`Global` trait for `CollabManager`; `oneshot` equivalent for `NetCommand::HostSession.reply` (use `async_channel::bounded(1)` to stay runtime-agnostic instead of a tokio oneshot). + +--- + +*End of plan. First action for the orchestrator: run M0, then dispatch M1 and M2 in parallel.* From 1df00deb6952dd6acfe866024fa4daad41d88817 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Fri, 12 Jun 2026 03:50:02 -0500 Subject: [PATCH 02/45] plan v2 --- plan.md | 991 +++++++++++++++++++++++++++++++++----------------------- 1 file changed, 580 insertions(+), 411 deletions(-) diff --git a/plan.md b/plan.md index ad909ec4..97655f22 100644 --- a/plan.md +++ b/plan.md @@ -1,9 +1,11 @@ -# Flowstate P2P Collaborative Editing — Implementation Plan +# Flowstate P2P Collaborative Editing — Implementation Plan (v2) **Status:** approved architecture, ready for implementation -**Target:** Google-Docs-style live co-editing of `.db8` rich-text documents -**Stack:** [iroh](https://docs.rs/iroh) 0.98.x for P2P networking · [loro](https://docs.rs/loro) 1.13.x for the rich-text CRDT -**Authoring date:** 2026-06-12 (API versions verified against crates.io/docs.rs on this date) +**Target:** Google-Docs-style live co-editing of `.db8` rich-text documents between symmetric peers +**Stack:** [iroh](https://docs.rs/iroh) 0.98.x (QUIC P2P) · [iroh-gossip](https://docs.rs/iroh-gossip) 0.100.x (swarm membership + broadcast) · [loro](https://docs.rs/loro) 1.13.x (rich-text CRDT) +**Authoring date:** 2026-06-12. All crate versions and API shapes verified against crates.io/docs.rs/source on this date. + +**v2 supersedes v1 entirely.** Headline changes from v1: star topology replaced by a **gossip mesh of fully symmetric peers** (no host, no starter role, no relay hub); **read-only/Viewer support removed** (every peer writes); session **membership and connectivity are orthogonal state axes** (involuntary disconnects keep you in the session and resync on reconnect; only an explicit Leave detaches); sessions **end by attrition** (no end-session authority or ritual); joiners get **crash-recovery files**; explicit **leave→save prompt cascades** on tab close/app quit. The CRDT document-sync core (§5–§8) is unchanged from v1 — it never depended on topology. --- @@ -11,22 +13,26 @@ This plan is written for an orchestrating agent that will spawn implementation subagents. Rules of engagement: -1. **Milestones M0–M6 (§22) are the unit of delegation.** M2 (doc sync) and M1 (networking) are independent and can run in parallel. M3+ depend on M2. -2. **Do not relitigate the decisions in §20.** They were made deliberately with full knowledge of the codebase. If an implementer hits a wall that genuinely invalidates one, stop and surface it rather than silently choosing differently. -3. **§23 lists the handful of API names that must be verified against docs.rs at implementation time** (exact method names in fast-moving crates). Everything else in this document was verified against the codebase or live docs on the authoring date. -4. Per `CLAUDE.md`: new functionality goes in **new files**, keep files **under 1000 LOC** (budgets are given per file), prefer **gpui-component** widgets over raw GPUI, add dependencies **via `cargo add`**, and run **`cargo clippy`** after each milestone. This workspace denies `clippy::pedantic` and `clippy::nursery` — write code accordingly (no silent truncating casts, no `unwrap()` on `get`, document any `unsafe`, etc.). See the allow-list in the root `Cargo.toml` before fighting a lint. -5. Conventions in this repo you must follow: 2-space indent (`.rustfmt.toml`), `#[hotpath::measure]`/`#[hotpath::measure_all]` on functions/impls in hot paths (match surrounding code), modules sometimes assembled via `include!` (don't convert them), `Entity`-based GPUI state. +1. **Milestones M0–M6 (§22) are the unit of delegation.** M1 (networking) and M2 (doc sync) are independent and can run in parallel. M3 depends on M2; M4 depends on M1+M3; M5 on M4; M6 last. +2. **Do not relitigate the decisions in §20.** Each one was argued out deliberately (topology, symmetry, attrition, leave semantics). If an implementer hits a wall that genuinely invalidates one, stop and surface it; do not silently choose differently. +3. **§23 lists every external API name that must be re-verified against docs.rs at implementation time.** Those are the only facts in this document I could not pin exactly; everything else was verified against this codebase (with `file:line` references) or live documentation. +4. Per `CLAUDE.md`: new functionality in **new files**; files **under 1000 LOC** (per-file budgets are given — treat them as hard caps and split files rather than exceed them); prefer **gpui-component** widgets over raw GPUI; add dependencies **via `cargo add`**; run **`cargo clippy`** when a milestone's edits are complete and fix what it reports. This workspace denies `clippy::{pedantic, nursery, correctness, suspicious, style, complexity, perf, cargo}` wholesale with a long allow-list in the root `Cargo.toml` — read that allow-list before fighting a lint. Practical consequences: no bare `unwrap()` on `get` (`get_unwrap` denied), document every `unsafe` (`undocumented_unsafe_blocks` denied), no `mem::forget`, use `if_then_some_else_none` style, every `#[allow]` needs a `reason`. +5. Repo conventions you must follow: 2-space indent (`.rustfmt.toml`); `#[hotpath::measure]` / `#[hotpath::measure_all]` attributes on functions/impls wherever the surrounding file uses them; several modules are assembled via `include!()` concatenation (`document/mod.rs`, `edit_ops/mod.rs`, `workspace/workspace/mod.rs`) — extend them the same way, do not convert to `mod`; GPUI state lives in `Entity` and is mutated via `entity.update(cx, …)`. --- ## 1. Product definition (the UX contract) -* Any user with a rich-text document (`.db8` editor tab) open can **start a collaboration session** for that document. Starting a session produces a **ticket string** (one opaque token, e.g. `fscollab…base32…`) they can paste to anyone over any side channel. -* Another Flowstate user chooses **Join session**, pastes the ticket, and the document opens as a **new tab** in their workspace. Both users now see each other's edits live (sub-second on LAN, ~1 s worst case via relay), each other's carets (colored, named), and a participant list. -* **The host's on-disk file is the single source of truth.** Only the host's editor has a `document_path`; only the host saves/autosaves to disk. Joiners can *Save As* to take a snapshot copy. While the session is live, concurrent edits merge via CRDT (loro) — no locking, no "reload to see changes." -* The host can end the session (or just close the tab/app); joiners keep an orphaned local copy with a "session ended" banner. -* Sessions can be started in **Editor** mode (joiners can write) or **Viewer** mode (joiners are read-only). The editor already understands `CollaborationRole::{Owner, Editor, Viewer}` and blocks mutations for viewers. -* Out of scope for v1 (explicitly): collaborating on Flow (`.fl0`) documents, per-peer role changes mid-session, mesh topology / host migration, persistence of CRDT history into `.db8`, end-to-end key rotation, more than ~8 simultaneous peers. +* Any user with a rich-text document tab (`.db8` editor) open can **start a collaboration session** on it. Starting produces a **ticket** — one opaque pasteable string. Anyone running Flowstate can choose **Join Session**, paste the ticket, and the shared document opens as a **new tab**. Edits, carets, and the participant list propagate live between everyone (sub-second on LAN; ~1 s worst case via relays). +* **Peers are fully symmetric.** The person who started the session has no ongoing role: they minted the session and seeded its content, and that is all. Every peer can edit, every peer can invite others (tickets are re-mintable by any member), every peer can serve a joining peer the document, and the session continues regardless of who comes and goes — including the person who started it. +* **Membership is distinct from connectivity.** + * A network drop, laptop sleep, or relay outage does **not** remove you from a session. Your tab stays attached; your edits accumulate locally in the CRDT; reconnection is automatic; on reconnect both sides exchange what the other missed and converge. The status UI makes the offline state visible ("Offline — changes will sync when reconnected"). + * Only an explicit **Leave** detaches a document from the session. On leave, the document freezes at exactly the state you last witnessed and becomes an ordinary local document (file-backed if it has a path; an unsaved in-memory document otherwise). +* **Joining never touches your existing documents.** A join always produces a fresh tab containing a fresh document built from the session's state. There is no flow, v1 or planned, that merges a local file into a session. Save As at any time writes your current view wherever you like — and **saving does not detach you**; your saved file simply becomes your personal copy (with autosave on, it keeps tracking the session until you leave). +* **Sessions end by attrition.** There is no "end session for everyone" action and no session-ended event. A session exists while at least one peer is attached; when the last peer leaves, it is gone. A peer who finds themselves alone sees "Only you" and chooses to wait (others may drop back in) or leave. +* **No canonical file.** While a session lives, the truth is the swarm's CRDT state. Files are personal copies; which copy a team treats as "the" document is social convention, not enforced by the app. (This deliberately supersedes the original "source of truth on the starter's disk" framing — discussed and decided.) +* **Closing a tab or quitting the app while attached is an implicit leave**, guarded by a two-step prompt: "Leave session?" then the save prompt — transactionally (cancel at either step cancels everything; see §13.4). +* Out of scope for v1 (explicit): collaborating on Flow (`.fl0`) documents; read-only roles; kick/ban (remedy: start a fresh session and re-invite selectively); session resume across app restarts ("Rejoin previous session" is a designed-for v1.1, see §13.6); cell-level table CRDT; >16 peers; version-history restore. --- @@ -36,157 +42,171 @@ A previous iteration scaffolded collaboration hooks into the editor engine. Inve | Capability | Where | Notes | |---|---|---| -| Canonical edit-op stream with stable IDs | `crates/gpui-flowtext/src/collaboration.rs` | `CanonicalOperation` enum: `InsertText`, `DeleteRange`, `SplitParagraph`, `JoinParagraphs`, `SetParagraphStyle`, `SetRunStyles`, `InsertBlock`, `DeleteBlock`, `MoveBlock`, `ReplaceParagraphSpan`, `ReplaceBlock`, `ReplaceDocument` | +| Canonical edit-op stream with stable IDs | `crates/gpui-flowtext/src/collaboration.rs` | `CanonicalOperation` enum: `InsertText`, `DeleteRange`, `SplitParagraph`, `JoinParagraphs`, `SetParagraphStyle`, `SetRunStyles`, `InsertBlock`, `DeleteBlock`, `MoveBlock`, `ReplaceParagraphSpan`, `ReplaceBlock`, `ReplaceDocument`. **Every** editor mutation path emits these (the undo system depends on it), so the local-edit tap is complete by construction. | | Stable identity map | `collaboration.rs` → `DocumentIdentityMap` | maps `ParagraphId`/`BlockId`/`TableCellId` (u128 UUIDs) ↔ indices; reconciled after every edit | -| Per-edit op capture | `crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs:238` | `mark_document_changed_with_reconcile` stores `last_collaboration_edit` after **every** mutation; every undo record carries `canonical_operations` | -| Remote-op application (naive) | `crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs:265` | `apply_remote_operations(&[CanonicalOperation])` — bounds-checked, ID-addressed appliers built on `edit_ops` primitives (`insert_text_at`, `delete_cross_paragraph_range`, `split_paragraph_at`, `mutate_runs_in_range`, `apply_document_span_replacement`) | -| Roles + write gating | `editor/mod.rs:182` (`CollaborationRole`), `editor/commands.rs:86,357,368,380` | every mutation command path already checks `can_write_collaboration()` | -| Remote caret rendering | `editor/mod.rs:461` (`ExternalCaret { offset, color_rgb }`), `lifecycle.rs:471` `set_external_carets` | painted per paragraph; presence layer just needs to feed it | -| Full document replace | `lifecycle.rs:274` `replace_document_from_collaboration` | used for join bootstrap / resync | -| Editor events | `crates/gpui-flowtext/src/api.rs:107` | `EditorEvent::{Changed, SelectionChanged, …}` + `EditorEventSink` | -| Document model | `crates/gpui-flowtext/src/document/*` | `Document { text: crop::Rope, paragraphs, blocks, assets, ids, sections, offset_index, theme }`. Rope = paragraph texts joined by `\n` (no trailing newline; see `edit_ops/offsets.rs:19 paragraph_width`). Non-text blocks (`Image`, `Equation`, `Table`) occupy **no** rope bytes. Table cells own their text privately (`TableCellParagraph { paragraph, text }`). | +| Per-edit op capture | `crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs:233` `mark_document_changed_with_reconcile` | stores `last_collaboration_edit` after every mutation; every undo record carries `canonical_operations` | +| Remote-op application primitives | `crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs:265` `apply_remote_operations` | bounds-checked, ID-addressed appliers built on `edit_ops` primitives (`insert_text_at`, `delete_cross_paragraph_range`, `split_paragraph_at`, `mutate_runs_in_range`, `apply_document_span_replacement`, `capture_document_span`) — reuse these in `collab_apply.rs` | +| Roles + write gating | `editor/mod.rs:182` `CollaborationRole`, `editor/commands.rs:86` | **Unused in v1** (everyone writes — keep `collaboration_role: None` always). Do not delete the enum; do not build UI for it. | +| Remote caret rendering | `editor/mod.rs:461` `ExternalCaret { offset: DocumentOffset, color_rgb: u32 }`, `lifecycle.rs:471` `set_external_carets` | painted per paragraph already; the presence layer just feeds it | +| Full document replace | `lifecycle.rs:274` `replace_document_from_collaboration(Document, cx)` | used for join bootstrap and local self-heal rebuilds | +| Editor events | `crates/gpui-flowtext/src/api.rs:107` | `EditorEvent::{Changed, SelectionChanged, CommandDispatched, Exported}` + `EditorEventSink` trait | +| Document model | `crates/gpui-flowtext/src/document/*` | `Document { text: crop::Rope, paragraphs: Arc>, blocks: Arc>, assets: AssetStore, ids: DocumentIds, sections, offset_index, theme }`. Rope = paragraph texts joined by `\n`, no trailing newline (`edit_ops/offsets.rs:19 paragraph_width`: `newline_len = usize::from(ix + 1 < paragraphs.len())`). Non-text blocks (`Image`, `Equation`, `Table`) occupy **no** rope bytes. Table cells own their text privately (`TableCellParagraph { paragraph, text }`). | | Run styles | `document/text.rs:192` | `RunStyles { semantic: RunSemanticStyle, direct_underline: bool, strikethrough: bool, highlight: Option }` — `Copy`, 4 orthogonal axes → 4 CRDT mark keys | -| Save machinery | `editor/style_state.rs:233 save`, `:247 save_as`, autosave in `workspace/workspace/documents.rs:865` | autosave already skips editors with `document_path == None` — joiners get correct behavior for free | -| Wire encoding helper | `collaboration.rs:280` `encode_canonical_operations` (postcard) | **Will NOT be used for sync** (it silently drops `ReplaceParagraphSpan`/block ops). Leave it; the CRDT layer replaces it. | +| Save machinery | `editor/style_state.rs:233 save`, `:247 save_as`; autosave `workspace/workspace/documents.rs:865 maybe_autosave_document` | autosave keys off `document_path().is_some()` + `has_unsaved_changes()` — pathless session tabs skip it automatically until the user Save-As-es | +| Recovery files | `editor/recovery.rs`, `recovery_path` set in `lifecycle.rs:25` from `document_path` | currently derived from `document_path` at construction → pathless joiner tabs get none. §15.6 adds an explicit setter. | +| Close/quit prompt flow | `documents.rs:670 close_document_panel`, `:734 request_close_window` | existing prompt→spawn→conditional-remove pattern; §13.4 extends it with the leave step | +| Wire encoding helper | `collaboration.rs:280` `encode_canonical_operations` (postcard) | **NOT used for sync** (it silently drops `ReplaceParagraphSpan`/block ops). Leave it in place; the CRDT layer replaces it. | -**Key consequence:** the integration burden is *translation*, not editor surgery. The editor already emits a canonical op stream and already accepts externally-driven document mutations. What's missing: (a) a CRDT document that both sides converge on, (b) a network layer, (c) session orchestration + UI. +**Key consequence:** the integration burden is *translation between flowtext's document model and loro's containers*, not editor surgery and not distributed-systems plumbing (loro owns op history/merge/idempotency; iroh-gossip owns membership/broadcast). What we write: the two-way mirror (§5–§7), a thin transport (§10–§11), and session orchestration + UI (§13, §17). Other load-bearing facts about the host app: -* **No tokio anywhere.** GPUI provides its own (smol-based) executors: `cx.background_executor().spawn(...)`, `cx.spawn(async move |entity, cx| …)`. iroh requires tokio → we run a dedicated tokio runtime thread (§12.1). -* `Workspace` (`crates/flowstate/src/workspace/workspace/mod.rs:65`) owns `document_panels: Vec>`, `active_editor: Option>`, and an `editor_subscriptions: Vec<(Uuid, Subscription)>` list — panels are keyed by `Uuid`. -* `cx.observe(&editor, …)` per panel (`documents.rs:178`) is how autosave is driven — the collab session uses the same mechanism to drain local edits. -* The document **render theme is a local user preference** (`documents.rs:153`: "DB8 stores style assignments, not style appearance") — **do not sync `DocumentTheme`**. Only style *slot assignments* (which are part of `ParagraphStyle`/`RunStyles`) sync. -* Commands are declared in `crates/flowstate/src/commands.rs` (`CommandId` + `COMMAND_SPECS`); top-bar menus in `workspace/workspace/top_bar.rs`; status bar in `render_status.rs`; modal/dialog/notification widgets come from `gpui-component` (`dialog.rs`, `notification.rs`, `avatar/`, `clipboard.rs`). +* **No tokio anywhere.** GPUI provides smol-based executors (`cx.background_executor().spawn`, `cx.spawn(async move |entity, cx| …)`). iroh requires tokio → dedicated tokio runtime thread (§10.1). Bridge with `async-channel` (runtime-agnostic) only. +* `Workspace` (`workspace/workspace/mod.rs:65`) owns `document_panels: Vec>`, `active_editor: Option>`, `editor_subscriptions: Vec<(Uuid, Subscription)>`; panels keyed by `Uuid`. `cx.observe(&editor, …)` per panel (`documents.rs:178`) is how autosave is driven — the session drains local edits from the same observe callback. +* The document **render theme is a local user preference** (`documents.rs:153`: "DB8 stores style assignments, not style appearance") — **never sync `DocumentTheme`**, zoom, or invisibility mode. Only style *slot assignments* (part of `ParagraphStyle`/`RunStyles`) sync. +* Commands: `crates/flowstate/src/commands.rs` (`CommandId` + `COMMAND_SPECS`). Top-bar menus: `workspace/workspace/top_bar.rs`. Status bar: `render_status.rs`. Dialog/notification/avatar/clipboard widgets: vendored `gpui-component`. --- ## 3. Architecture overview ``` -┌──────────────────────────── Flowstate process (host or joiner) ───────────────────────────┐ -│ │ -│ GPUI main thread tokio runtime thread (new, lazy) │ -│ ┌──────────────────────────────────────────┐ ┌────────────────────────────────────┐ │ -│ │ Workspace │ │ CollabNet (flowstate-collab::net) │ │ -│ │ └─ DocumentPanel ── RichTextEditor │ │ • iroh Endpoint (one per app) │ │ -│ │ ▲ collab_apply / take_edits │ │ • protocol::Router, ALPN │ │ -│ │ │ │ │ b"flowstate/collab/0" │ │ -│ │ CollabSession (Entity, per panel) │ │ • per-peer send/recv tasks │ │ -│ │ • LoroDoc + EphemeralStore │ async │ • star topology: │ │ -│ │ • DocBinding (ids ↔ containers) │ chan │ host accepts N conns │ │ -│ │ • LocalApplier / RemoteApplier │◄─────►│ joiner dials 1 conn │ │ -│ │ • participant roster, role │ │ • frame codec (postcard+len) │ │ -│ │ CollabManager (Global) │ └────────────────────────────────────┘ │ -│ │ • owns runtime handle + NetCommand tx │ │ -│ └──────────────────────────────────────────┘ │ -└──────────────────────────────────────────────────────────────────────────────────────────────┘ - - host ──── QUIC (iroh: direct holepunched, or n0 relay fallback) ──── joiner(s) +┌──────────────────────────── Flowstate process (any peer — all peers identical) ───────────────────────────┐ +│ │ +│ GPUI main thread tokio runtime thread (lazy, one per app) │ +│ ┌───────────────────────────────────────────────┐ ┌───────────────────────────────────────────┐ │ +│ │ Workspace │ │ CollabNet (flowstate-collab::net) │ │ +│ │ └─ DocumentPanel ── RichTextEditor │ │ • one iroh Endpoint for the whole app │ │ +│ │ ▲ apply_collab_patches / take_edits │ │ • iroh-gossip Gossip instance │ │ +│ │ │ │ async │ – one subscribed topic per session │ │ +│ │ CollabSession (Entity, one per session tab) │ channels│ – HyParView membership (≤5 active │ │ +│ │ • LoroDoc + EphemeralStore (presence) │◄───────►│ conns/topic) + Plumtree broadcast │ │ +│ │ • DocBinding (local ids ↔ containers) │ │ • direct protocol (own ALPN): serve/pull │ │ +│ │ • LocalApplier / RemoteApplier │ │ snapshots, update-deltas, blobs, │ │ +│ │ • SessionPhase state machine (§13) │ │ assets over per-request bi-streams │ │ +│ │ • anti-entropy timer, self-check timer │ │ • Router accepts BOTH ALPNs │ │ +│ │ CollabManager (gpui Global) │ └───────────────────────────────────────────┘ │ +│ │ • runtime handle, NetCommand tx, │ │ +│ │ NetEvent pump, session registry │ │ +│ └───────────────────────────────────────────────┘ │ +└────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + peer ⇄ peer: iroh QUIC (direct holepunched; n0 relay fallback), wired into a gossip overlay per session ``` -**Topology: star.** The host is the hub; joiners connect only to the host. The host imports every received update into its own `LoroDoc` and relays the *raw bytes* to all other peers (loro updates are idempotent and commutative, so blind relay is safe). This matches the "host owns the file" authority model, makes permission enforcement a host-side concern, and avoids gossip/mesh complexity. iroh-gossip is deliberately **not** used. +**Topology: gossip swarm of equals.** iroh-gossip (HyParView + Plumtree) maintains, per topic, a bounded **active view** (≤5 live connections by default; `active_view_capacity: 5`, `passive_view_capacity: 30` — verified in iroh-gossip source) and disseminates broadcasts along a self-healing spanning tree. At our session sizes (≤16) the overlay is effectively a near-full mesh; we still get membership maintenance, dedup, fan-out, and repair for free. There is **no hub**: every peer broadcasts its own updates, every peer can serve bootstrap/catch-up/assets over the direct protocol, and the protocol contains no role distinctions whatsoever. + +**Why this is safe with zero merge code:** loro updates are commutative, idempotent, and causally self-describing. Redundant gossip delivery → deduped by loro import. Out-of-order arrival → loro queues imports with missing dependencies as *pending* and applies them when gaps fill. Missed-while-offline → repaired by anti-entropy (§10.4), which is a thin loop over `oplog_vv()` / `export(updates(vv))` / `import()`. We never parse, store, transform, or order operations ourselves. -**Data flow for one keystroke (host or joiner):** +**Data flow for one keystroke:** -1. User types → existing edit pipeline mutates `Document`, pushes an `EditRecord`, queues a `CollaborationEdit` (canonical ops). -2. `cx.observe(editor)` fires → `CollabSession::flush_local_edits` drains the queue → `LocalApplier` translates canonical ops into loro container ops inside one `LoroDoc` commit. -3. loro's `subscribe_local_update` callback hands the session a `Vec` update blob → session sends `NetCommand::Broadcast` over the async channel → tokio thread writes a `Frame::DocUpdate` to peer connection(s). -4. Remote side: tokio thread reads frame → `NetEvent::Frame{peer, bytes}` → session entity `doc.import(bytes)` → loro fires container diff events → `RemoteApplier` maps diffs to paragraph/block mutations → editor repaints, remote selection remapped. +1. User types → existing edit pipeline mutates `Document`, pushes an `EditRecord`, and (new, §15.2) pushes the exact op delta into `pending_collab_edits`. +2. `cx.observe(editor)` fires → `CollabSession::flush_local_edits` drains the queue → `LocalApplier` replays the ops onto the `LoroDoc` → `doc.commit_with(origin="local-edit")`. +3. loro's `subscribe_local_update` callback yields the update blob → session sends `NetCommand::Publish{session, bytes}` → tokio thread: blob ≤ `GOSSIP_INLINE_LIMIT` → `sender.broadcast(GossipMsg::Update(bytes))`; larger → blob outbox + `GossipMsg::UpdateAvailable{blob_id, len}` notice (§10.5). +4. Every other peer: gossip receiver yields the message → `NetEvent::Gossip{session, from, msg}` → session entity `doc.import(bytes)` (pulling first if it was an availability notice) → loro fires container diff events tagged as import-triggered → `RemoteApplier` maps diffs to patches → `editor.apply_collab_patches` → repaint, selections remapped. -The CRDT doc lives **on the GPUI main thread inside the session entity** (loro ops are microseconds; only network I/O leaves the thread). `LoroDoc` is `Send + Sync`, which we exploit only for the one-time snapshot export/import on join (background executor). +The `LoroDoc` lives **on the GPUI main thread inside the session entity** (loro ops are microseconds; only I/O leaves the thread). `LoroDoc` is `Send + Sync`; we exploit that only for snapshot export/import on join (background executor) and for the direct-protocol servers, which hold a cloned doc handle (`LoroDoc::clone` shares the same underlying doc — verify, §23-V6; if it deep-copies, route serve requests through the session entity instead). -### 3.1 Crate layout (new code) +### 3.1 Crate layout (new code, with hard LOC caps) ``` -crates/flowstate-collab/ # NEW crate. Core logic. NO gpui dependency → headless-testable. +crates/flowstate-collab/ # NEW crate. Core logic. NO gpui dependency → headless-testable. Cargo.toml - src/lib.rs # pub mod …, shared error type (thiserror not in tree: use anyhow per workspace norm) - src/protocol.rs # Frame enum, PROTOCOL_VERSION, frame codec (≤400 LOC) - src/ticket.rs # SessionTicket (iroh-tickets Ticket impl) (≤150 LOC) - src/schema.rs # loro container layout constants + builders + projection types (≤500 LOC) - src/binding.rs # DocBinding: container ↔ ParagraphId/BlockId table (≤350 LOC) - src/local_apply.rs # CanonicalOperation → loro ops (≤700 LOC) - src/remote_apply.rs # loro DiffEvent batch → DocPatch list (≤700 LOC) - src/projection.rs # LoroDoc → Vec (+ reverse: Document → fresh LoroDoc) (≤450 LOC) - src/presence.rs # EphemeralStore wrapper: PresenceState, cursors codec (≤300 LOC) - src/integrity.rs # projection hash (xxhash via rustc-hash? no → use `twox-hash`), resync decision (≤150 LOC) - src/session_core.rs # role/roster/session-id types shared by host & joiner (≤250 LOC) - src/net/mod.rs # NetCommand/NetEvent enums, channel types (≤200 LOC) - src/net/runtime.rs # tokio runtime thread bootstrap, Endpoint construction (≤250 LOC) - src/net/host.rs # accept loop, per-peer tasks, relay fan-out, auth (≤500 LOC) - src/net/joiner.rs # dial, handshake, reconnect-with-backoff loop (≤400 LOC) - src/net/framing.rs # length-prefixed read/write on iroh send/recv streams (≤150 LOC) - tests/convergence.rs # multi-peer fuzz (THE acceptance test, §21.2) - tests/translation.rs # golden tests per canonical op - tests/loopback.rs # real iroh sockets on localhost, 2–3 peers - -crates/flowstate/src/collab/ # NEW module in the app crate. GPUI glue + UI. - mod.rs # CollabManager global, init, plumbing (≤300 LOC) - session.rs # CollabSession entity (per panel): owns LoroDoc side, drains edits, - # applies remote patches via editor API, roster state (≤700 LOC) - share_dialog.rs # start/join/manage modal (gpui-component Dialog) (≤500 LOC) - status.rs # status-bar pill + tab badge elements (≤250 LOC) - -crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs # NEW: editor-side patch application (≤600 LOC) + src/lib.rs # module decls + crate-level docs (≤80) + src/ids.rs # SessionId(=TopicId bytes), BlobId, PeerId aliases, color palette (≤120) + src/ticket.rs # SessionTicket: mint/parse/display (iroh-tickets Ticket impl) (≤200) + src/proto_gossip.rs # GossipMsg enum + encode/decode + size budget consts (≤200) + src/proto_direct.rs # direct-protocol Request/Response enums + stream codec (≤350) + src/schema.rs # loro container layout, configure_text_styles, offset helpers (≤500) + src/binding.rs # DocBinding table + invariants (≤350) + src/local_apply.rs # CanonicalOperation → loro ops (≤700) + src/remote_apply.rs # loro DiffEvent batch → CollabPatch list (≤700) + src/projection.rs # LoroDoc → Vec; Document → LoroDoc population (≤450) + src/presence.rs # EphemeralStore wrapper, PresenceState codec, roster derivation (≤300) + src/self_check.rs # local projection hash + drift detection (≤150) + src/net/mod.rs # NetCommand/NetEvent enums, channel aliases (≤250) + src/net/runtime.rs # tokio thread bootstrap, Endpoint + Gossip + Router construction (≤300) + src/net/swarm.rs # per-session topic task: subscribe, broadcast, receive, neighbors (≤400) + src/net/direct.rs # direct-protocol server (ProtocolHandler) + client helpers (≤450) + src/net/anti_entropy.rs # digest timer, gap detection, pull scheduling (≤250) + src/net/blobs.rs # outbox ring buffer for oversized updates (≤150) + tests/convergence.rs # multi-peer fuzz (THE acceptance gate, §21.2) + tests/translation.rs # golden tests per canonical op (§21.1) + tests/swarm_loopback.rs # real iroh+gossip on localhost, 3 peers (§21.3) + tests/anti_entropy.rs # gap/heal logic against a fake transport (§21.1) + +crates/flowstate/src/collab/ # NEW module in the app crate. GPUI glue + UI. + mod.rs # CollabManager global, init, NetEvent pump, registry (≤350) + session.rs # CollabSession entity: §13 state machine, flush/import, presence, + # assets, self-heal, prompt helpers (≤800; split session_io.rs if over) + share_dialog.rs # share/join/roster modal (gpui-component Dialog) (≤500) + status.rs # status-bar pill + tab badge + offline indicator elements (≤250) + +crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs # NEW: patch application + selection remap (≤600) ``` -`flowstate-collab` depends on: `loro`, `iroh`, `iroh-tickets`, `tokio`, `async-channel`, `postcard`, `serde`, `uuid`, `anyhow`, `rand`, `twox-hash`, and `gpui-flowtext` (for `CanonicalOperation`, `Document`, `InputBlock` types only — this is a deliberate dependency direction: collab knows the document model; the document engine does *not* know collab/loro). +Dependency direction (deliberate): `flowstate-collab` → `gpui-flowtext` (for `CanonicalOperation`, `Document`, `Input*`, `CollabPatch` types). The editor crate never sees loro or iroh. The app crate sees both. --- -## 4. Dependencies (exact) +## 4. Dependencies (exact, with commands) -Run from workspace root (per CLAUDE.md, use the CLI — these land in `[workspace.dependencies]` then get referenced from member crates): +Run from the workspace root after `cargo new crates/flowstate-collab --lib` (M0). Workspace uses `[workspace.dependencies]`; add there and reference with `{ workspace = true }` per existing convention: ```sh -cargo add --package flowstate-collab iroh@0.98 # default features; relay/discovery wiring per §23-V1 +cargo add --package flowstate-collab iroh@0.98 +cargo add --package flowstate-collab iroh-gossip@0.100 cargo add --package flowstate-collab iroh-tickets@0.98 cargo add --package flowstate-collab loro@1.13 cargo add --package flowstate-collab tokio@1 --features rt-multi-thread,macros,time,sync cargo add --package flowstate-collab async-channel@2 cargo add --package flowstate-collab postcard@1 --features use-std -cargo add --package flowstate-collab rand@0.9 -cargo add --package flowstate-collab twox-hash@2 +cargo add --package flowstate-collab serde@1 --features derive +cargo add --package flowstate-collab anyhow@1 uuid@1 rand@0.9 twox-hash@2 +cargo add --package flowstate-collab gpui-flowtext --path crates/gpui-flowtext cargo add --package flowstate-collab --dev proptest@1 +cargo add --package flowstate flowstate-collab --path crates/flowstate-collab ``` -Version rationale: +Version rationale (verified 2026-06-12): -* **iroh 0.98.2** is the latest stable (1.0.0-rc.1 exists as of 2026-05-27). Pin 0.98; the 1.0 surface is the same post-rename API (`Endpoint`, `EndpointId`, `EndpointAddr`, `endpoint.addr()`, `iroh::protocol::{Router, ProtocolHandler, AcceptError}`). Migration to 1.0 final is a version-bump task later, not an architecture change. -* **iroh-tickets** is the post-0.94 home of ticket types (`EndpointTicket`, the `Ticket` trait); keep its version in lockstep with iroh. -* **loro 1.13.1** — stable 1.x line; has everything we need: `LoroDoc`, `LoroText` (+`*_utf8` APIs), `LoroMap`, `LoroMovableList`, `ExportMode::{Snapshot, updates(&vv), all_updates()}`, `subscribe_local_update`, `subscribe` (diff events), `UndoManager`, `Cursor`, `awareness::EphemeralStore`. -* **async-channel** — runtime-agnostic MPMC; both smol-side (GPUI) and tokio-side can await it. This is the only inter-runtime bridge primitive. Do not use `tokio::sync::mpsc` across the boundary. -* **twox-hash** — fast non-crypto hash for projection integrity checks. +* **iroh 0.98.2** — latest stable (1.0.0-rc.1 exists, 2026-05-27). Post-rename API: `Endpoint`, `EndpointId`, `EndpointAddr`, `endpoint.addr()`, `iroh::protocol::{Router, ProtocolHandler, AcceptError}`. 1.0 migration later is a version bump, not a redesign. +* **iroh-gossip 0.100.0** (2026-05-27) — pairs with the current iroh line. HyParView defaults verified in source: `active_view_capacity: 5`, `passive_view_capacity: 30`. Subscribe API: `gossip.subscribe(topic_id: TopicId /* 32 bytes */, bootstrap: Vec)`. +* **iroh-tickets 0.98** — post-0.94 home of the `Ticket` trait (postcard payload, base32 display) and `EndpointTicket`. +* **loro 1.13.1** — `LoroDoc`, `LoroText` (+`*_utf8` and `convert_pos` APIs), `LoroMap`, `LoroMovableList`, `ExportMode::{Snapshot, updates(&vv), all_updates()}`, `subscribe_local_update`, diff-event subscriptions, `UndoManager`, `Cursor`, `awareness::EphemeralStore` (timestamped LWW KV with per-key timeout, encode/apply, local-update + merge subscriptions — purpose-built for presence). +* **async-channel 2** — runtime-agnostic; the only primitive allowed across the smol↔tokio boundary. Never `tokio::sync::mpsc` across it; for request/reply use `async_channel::bounded(1)` as a oneshot. +* **twox-hash 2** — fast non-crypto hash for the local self-check (§7.2). -`crates/flowstate` adds `flowstate-collab` as a workspace dep. `gpui-flowtext` gains **no** new dependencies. +`gpui-flowtext` gains **no** new dependencies. --- ## 5. CRDT schema (loro container layout) -The loro document is the shared, convergent representation. Each peer's flowtext `Document` is a *projection* of it. Schema version is stored in the doc so future formats can migrate. +The loro document is the shared convergent representation; each peer's flowtext `Document` is a *projection* of it. One loro doc per session, created once by whoever starts the session (`projection::populate_from_document`), then only ever mutated through the appliers. ``` LoroDoc ├─ "meta": LoroMap -│ ├─ "schema" : i64 = 1 (bump on breaking layout change) -│ ├─ "title" : string = host display name of the doc (informational) -│ └─ "created_by" : string = host EndpointId (z32/hex, informational) -└─ "blocks": LoroMovableList (one entry per flowtext Block, same order) - └─ [i]: LoroMap (block container) +│ ├─ "schema" : i64 = 1 (bump on breaking layout change) +│ ├─ "session" : string = hex(topic_id) (lineage guard — see §12.2) +│ └─ "title" : string = doc display name at creation (informational; shown in join dialog) +└─ "blocks": LoroMovableList (one entry per flowtext Block, same order) + └─ [i]: LoroMap (block container) ├─ "kind" : string ∈ {"p", "image", "equation", "table"} ├─ for "p" (paragraph): - │ ├─ "text" : LoroText (rich text w/ marks, see below) - │ └─ "style" : i64 (ParagraphStyle encoded: Normal = -1, Custom(s) = s) + │ ├─ "text" : LoroText (rich text w/ marks, §5.1) + │ └─ "style" : i64 (ParagraphStyle: Normal = -1, Custom(s) = s as i64) └─ for "image" | "equation" | "table": - ├─ "data" : binary (postcard-encoded InputBlock payload, §5.3) - └─ "rev" : i64 (monotonic bump; makes replaces visible in events) + ├─ "data" : LoroValue::Binary (postcard-encoded BlockPayload, §5.3) + └─ "rev" : i64 (monotonic bump per replace; forces a map-diff event) ``` +Call `schema::configure_text_styles(&LoroDoc)` on **every** doc before first use (creation, join import, resync import) — it registers the four mark keys with their expand behavior (§5.1). Factor doc construction into one function so this can't be forgotten: `schema::new_configured_doc() -> LoroDoc`. + ### 5.1 Rich text marks (per-paragraph `LoroText`) -`RunStyles` has 4 orthogonal axes → 4 mark keys: +`RunStyles` (4 orthogonal axes) → 4 mark keys: | Mark key | Value | Absent means | |---|---|---| @@ -195,77 +215,97 @@ LoroDoc | `"strike"` | bool `true` | `strikethrough == false` | | `"hl"` | i64 slot of `HighlightStyle::Custom(slot)` | `highlight == None` | -**Mark expansion: configure ALL four keys as `ExpandType::None`** via `LoroDoc::config_text_style` on every doc before first use (factor into `schema::configure_text_styles(&LoroDoc)`). Rationale: flowtext computes the styles of every inserted character explicitly (`pending_styles` / run inheritance in `edit_pipeline.rs`), so the CRDT must not "helpfully" expand neighboring marks — the LocalApplier always marks inserted ranges itself. This makes mark state deterministic and identical to flowtext's run model. +**Configure ALL four keys as `ExpandType::None`.** Rationale: flowtext computes the styles of every inserted character explicitly (`pending_styles` / run inheritance in `edit_pipeline.rs:25-34`), so the CRDT must never auto-expand neighboring marks — the LocalApplier marks every styled insert itself. Mark state stays deterministic and bit-identical to flowtext's run model. -Reading a paragraph back: `text.to_delta() -> Vec` (insert segments + attribute maps) → fold into `Vec` (adjacent equal-style segments merge, mirroring `edit_ops` run normalization). +Reading a paragraph back: `text.to_delta() -> Vec` (insert segments with attribute maps) → fold into `Vec`: -**Soft line breaks** (`\u{2028}`, `document/core.rs:12`) are ordinary characters inside paragraph text — no special handling. +```rust +// schema.rs +pub fn input_runs_from_delta(delta: &[TextDelta]) -> Vec { + // 1. each insert segment → InputRun { text, styles: run_styles_from_attrs(attrs) } + // 2. merge adjacent runs with equal RunStyles (mirrors edit_ops run normalization) + // 3. empty paragraph → vec![] (flowtext represents empty paragraphs with zero runs — check + // blank_document() and match its shape exactly) +} +pub fn run_styles_from_attrs(attrs: &…) -> RunStyles; // missing key → default axis value +pub fn mark_intervals_from_runs(runs: &[TextRun]) -> [Vec<(Range, LoroValue)>; 4]; // utf8 ranges per key +``` -### 5.2 Why per-paragraph texts in a list (not one giant LoroText) +**Soft line breaks** (`\u{2028}`, `document/core.rs:12`) are ordinary characters inside paragraph text — no special handling anywhere in the sync layer. -Decision (final — see §20): one `LoroText` per paragraph inside a `LoroMovableList`, **not** a single body text with `\n` separators. +### 5.2 Why per-paragraph texts in a list (not one giant LoroText) — decided, final -* The canonical op stream already speaks `(ParagraphId, utf8 byte)` — translation to `(container, offset)` is direct, with **no global offset arithmetic** anywhere. Global-offset bookkeeping across interleaved non-text blocks is where multi-week debugging cycles go to die. -* Blast radius: a translation bug corrupts one paragraph, not the document; per-paragraph regeneration makes remote apply trivially correct. -* Block move/insert/delete map 1:1 onto `LoroMovableList` ops (true move semantics, no delete+reinsert identity loss). -* Known accepted anomaly: if peer A splits a paragraph at offset k while peer B concurrently types at offset j > k in the same paragraph within one round-trip, B's text converges to the end of the first half rather than inside the new second paragraph. No data loss, positions slightly off, self-consistent on all peers. This is the standard per-block CRDT trade-off (Notion-class behavior). Document it in code; revisit only with evidence it bites in practice. +* The canonical op stream already speaks `(ParagraphId, utf8 byte)` — translation to `(container, offset)` is direct, with **no global offset arithmetic anywhere**. Global-offset bookkeeping across interleaved non-text blocks is the classic multi-week-debugging sink; we structurally avoid it. +* Blast-radius containment: a translation bug corrupts one paragraph, not the document; per-paragraph regeneration makes remote apply trivially correct. +* Block insert/delete/move map 1:1 onto `LoroMovableList` ops (true move semantics — concurrent edits inside a moved block survive). +* Known, accepted anomaly: peer A splits a paragraph at offset k while peer B concurrently types at j > k in the same paragraph within one propagation window → B's text converges at the end of the first half instead of inside the new second paragraph. No data loss; all peers agree; B's caret looks slightly wrong once. Standard per-block CRDT trade-off. Document it in `schema.rs` module docs; revisit only with field evidence. ### 5.3 Opaque block payloads (`"data"`) -`Image`, `Equation`, `Table` blocks are **atomic last-writer-wins values** in v1. Justification from the codebase: flowtext itself records table/equation/image edits as whole-block replaces (`CanonicalOperation::ReplaceBlock` carries no finer payload — see `editor/tables.rs:174`, `table_equation_editing.rs:66,280,336`, `media.rs:164,224`, `object_selection.rs:250`). A cell-level table CRDT would exceed the granularity of the host editor's own edit stream — pure cost, no fidelity gain, until the editor itself gets finer-grained table ops. - -Payload type (in `schema.rs`): +`Image`, `Equation`, `Table` blocks are **atomic last-writer-wins values**. Justification from the codebase: flowtext itself records table/equation/image edits as whole-block replaces — `CanonicalOperation::ReplaceBlock` carries no finer payload (`editor/tables.rs:174`, `table_equation_editing.rs:66,280,336`, `media.rs:164,224`, `object_selection.rs:250`). A finer-grained table CRDT would exceed the granularity of the host editor's own edit stream: pure cost, zero fidelity gain, until the editor itself ships finer table ops. ```rust +// schema.rs #[derive(Serialize, Deserialize)] -pub enum BlockPayload { // postcard-encoded into the "data" value +pub enum BlockPayload { // postcard → LoroValue::Binary in "data" Image { asset_id: u128, mime: String, original_name: Option, - content_hash: u64, byte_len: u64, // asset BYTES ARE NOT IN THE CRDT (§16) + content_hash: u64, byte_len: u64, // asset BYTES NEVER enter the CRDT (§16) alt_text: String, caption: Option, sizing: InputImageSizing, alignment: InputBlockAlignment }, Equation { source: String, display: InputEquationDisplay }, // syntax: only Latex exists today Table(InputTableBlock), } +pub fn payload_from_block(block: &Block) -> Option; // None for Block::Paragraph +pub fn block_from_payload(payload: BlockPayload, assets: &AssetStore) -> InputBlock; ``` -Conversions `Block ↔ BlockPayload` reuse the existing `Input*` types (`document/text.rs:105-189`) and whatever `InputBlock → Block` builders the persistence/clipboard paths already use (locate via `RichClipboardFragment` consumers in `edit_ops/rich_fragment.rs`; do not write a second converter). - -Concurrent edits to the same table = LWW on `"data"` (one peer's table edit wins; the other's is lost *for that block only*). The `"rev"` bump guarantees a map-diff event fires even if values were equal-ish. Acceptable v1 semantics; logged as a known limitation. +Conversions reuse the existing `Input*` types (`document/text.rs:105-189`) and the existing `InputBlock → Block` builder used by the clipboard path — find it via `RichClipboardFragment` consumers in `edit_ops/rich_fragment.rs` and call it; do **not** write a second converter. Concurrent edits to the same table = LWW on `"data"` (one peer's edit wins for that block; logged as a known limitation). The `"rev"` bump guarantees a diff event even for value-equal writes. ### 5.4 Identity & the DocBinding -Cross-peer identity = **loro container identity** (each block map / text container has a `ContainerID`). Flowtext's `ParagraphId`/`BlockId` are **peer-local** (they are generated fresh on load and never serialized to db8 — see `document_ids_for_shape` / `reconcile_document_ids` in `document/core.rs`). Never ship them over the wire; never store them in the CRDT. - -Each session maintains the bridge table (in `binding.rs`): +Cross-peer identity = **loro container identity**. Flowtext's `ParagraphId`/`BlockId` are **peer-local** (generated fresh on every load, never serialized to db8 — see `document_ids_for_shape` / `reconcile_document_ids` in `document/core.rs:243,256`). Never put them on the wire or in the CRDT. ```rust +// binding.rs pub struct DocBinding { - rows: Vec, // parallel to BOTH document.blocks and loro "blocks" list - by_paragraph: FxHashMap, // rebuilt on structural change + rows: Vec, // parallel to BOTH document.blocks and loro "blocks" + by_paragraph: FxHashMap, by_block: FxHashMap, - by_container: FxHashMap, + by_container: FxHashMap, // text container AND map container both map to the row } pub struct BindingRow { pub map: LoroMap, // handle to blocks[i] - pub text: Option, // Some for kind=="p" + pub text: Option, // Some iff kind == P pub kind: BlockKind, // P | Image | Equation | Table - pub block_id: BlockId, // local flowtext id at this position - pub paragraph_id: Option, // local flowtext id (paragraph blocks only) + pub block_id: BlockId, + pub paragraph_id: Option, +} +impl DocBinding { + pub fn build(doc: &LoroDoc, document: &Document) -> anyhow::Result; // walk containers + ids in order + pub fn assert_consistent(&self, doc: &LoroDoc, document: &Document); // debug_assert! the invariant below + // splice/insert/remove/move row methods used by both appliers — keep index maps coherent } ``` -**Invariant (assert in debug builds after every local flush and every remote apply):** -`binding.rows.len() == document.blocks.len() == loro_blocks.len()`, `rows[i].kind` matches `document.blocks[i]`, `rows[i].paragraph_id == document.ids.paragraph_ids[paragraph_ordinal(i)]`, and `rows[i].block_id == document.ids.block_ids[i]`. +**Invariant (debug-assert after every local flush and every remote apply batch):** +`rows.len() == document.blocks.len() == loro_blocks.len()`; `rows[i].kind` matches `document.blocks[i]`'s variant; `rows[i].block_id == document.ids.block_ids[i]`; for the j-th paragraph block, `rows[i].paragraph_id == Some(document.ids.paragraph_ids[j])`. -When a *remote* change creates a block, the RemoteApplier generates **fresh local ids** (`new_paragraph_id()` / `new_block_id()`) and writes them into both `Document.ids` and the binding row. When a *local* edit creates one, the LocalApplier reads the id flowtext just minted (via `identity_map` / `document.ids`) and records it next to the new container. +When a **remote** change creates a block, the RemoteApplier mints fresh local ids (`new_paragraph_id()` / `new_block_id()` from `document/core.rs:225,231`) and writes them into both `Document.ids` and the new row. When a **local** edit creates one, the LocalApplier reads the id flowtext just minted (from `document.ids` at the op's index) and records it beside the new container. -### 5.5 Offset discipline (read this twice) +### 5.5 Offset discipline (read twice; most historical CRDT-integration bugs live here) * flowtext: **UTF-8 byte offsets** everywhere (`DocumentOffset.byte`, run `len`s, canonical ops). -* loro `LoroText`: default API is **Unicode scalar positions**; UTF-8 variants exist (`mark_utf8`, `len_utf8`, `convert_pos` between `Utf8`/`Unicode`/`Utf16`/`Event` coordinate systems; insert/delete have unicode signatures — use the `*_utf8` variants where provided, otherwise `convert_pos` first; see §23-V3). -* loro **events** report positions in *event* coordinates (unicode by default). The RemoteApplier must convert event deltas to UTF-8 for selection remapping (`convert_pos(.., PosType::Event, PosType::Utf8)` or by walking the materialized string — pick one, write one helper, test it on multibyte text: `"é🌍\u{2028}x"`). -* Every module that crosses the boundary funnels through two helpers in `schema.rs` — `to_loro_pos(text, utf8_byte) -> loro_pos` and `to_utf8_byte(text, loro_pos) -> usize` — **no inline conversions anywhere else**. -* Debug assertion after each paragraph mutation: `text.len_utf8() == paragraph_text_len(paragraph)`. +* loro `LoroText`: default API positions are **Unicode scalar values**; UTF-8 variants exist (`mark_utf8`, `len_utf8`); `convert_pos(index, PosType, PosType)` converts between `Utf8`/`Unicode`/`Utf16`/`Event` coordinate systems. Use `*_utf8` variants where they exist; otherwise convert at the boundary (§23-V3 lists what to check). +* loro **diff events** report positions in *event* coordinates — convert before remapping selections. +* ALL conversions funnel through exactly two helpers in `schema.rs`; **no inline conversions anywhere else**: + +```rust +pub fn loro_pos(text: &LoroText, utf8_byte: usize) -> usize; // panics in debug on out-of-range +pub fn utf8_byte(text: &LoroText, loro_pos: usize) -> usize; +``` + +* Debug assertion after every paragraph mutation, both directions: `text.len_utf8() == paragraph_text_len(&document.paragraphs[ix])`. +* Mandatory test fixture for every offset-touching test: `"aé🌍\u{2028}x"` (1-, 2-, 4-byte chars + soft break). --- @@ -274,444 +314,573 @@ When a *remote* change creates a block, the RemoteApplier generates **fresh loca `local_apply.rs`. Entry point: ```rust -pub struct LocalApplier<'doc> { doc: &'doc LoroDoc, binding: &'doc mut DocBinding } +pub struct LocalApplier<'s> { pub doc: &'s LoroDoc, pub binding: &'s mut DocBinding } impl LocalApplier<'_> { /// Apply ONE editor edit-record's ops, then `doc.commit_with(origin = "local-edit")`. - /// One EditRecord == one loro commit == one undo unit. + /// One EditRecord == one loro commit == one undo unit == typically one gossip blob. pub fn apply(&mut self, document: &Document, ops: &[CanonicalOperation]) -> anyhow::Result<()>; } ``` -`document` is the **post-edit** flowtext document (canonical ops reference post-edit identity — `identity_map.reconcile` has already run). Per-op translation: +`document` is the **post-edit** flowtext document (canonical ops reference post-edit identity; `identity_map.reconcile` has already run). Per-op translation: | Canonical op | Loro translation | |---|---| -| `InsertText { paragraph, byte, text, styles }` | row = `binding.by_paragraph[paragraph]`; `row.text.insert` at converted pos; then apply non-default marks over the inserted range (`sem`/`ul`/`strike`/`hl` per §5.1). Expansion is `None`, so a plain insert needs marking only when `styles != RunStyles::default()`. | -| `DeleteRange { start_paragraph, start_byte, end_paragraph, end_byte }` | same paragraph → single `delete`. Cross-paragraph (this is a **join**): (1) read tail delta of `end_paragraph` from `end_byte`; (2) delete `[start_byte..]` of start text; (3) delete the whole-paragraph containers strictly between start and end (movable-list `delete`); (4) append the saved tail (text+marks) to start text; (5) delete end container. Update binding rows + `Document`-side ids are already updated by flowtext. | -| `SplitParagraph { paragraph, byte, new_paragraph }` | (1) read tail delta `[byte..]`; (2) delete tail from original; (3) `blocks.insert_container(idx+1, LoroMap)` → set `"kind"="p"`, `"style"` copied from original; create `"text"` and insert tail delta with marks; (4) insert `BindingRow` with `paragraph_id = new_paragraph` and the `BlockId` flowtext minted (read from `document.ids.block_ids[idx+1]`). | -| `JoinParagraphs { first, second }` | degenerate `DeleteRange` (tail append + container delete), same code path. | +| `InsertText { paragraph, byte, text, styles }` | `row = binding.by_paragraph[paragraph]`; insert at `loro_pos(byte)`; if `styles != RunStyles::default()`, apply the non-default mark keys over the inserted utf8 range. (Expand is `None`, so an unmarked insert is plain by construction.) | +| `DeleteRange { start_paragraph, start_byte, end_paragraph, end_byte }` | Same paragraph → single `delete`. Cross-paragraph (= join): (1) read tail delta of the end paragraph from `end_byte` (`to_delta` sliced); (2) delete `[start_byte..]` from the start text; (3) movable-list-delete every whole container strictly between start and end, removing binding rows; (4) append the saved tail into the start text, re-applying its marks segment by segment; (5) delete the end container + row. All inside the one commit. | +| `SplitParagraph { paragraph, byte, new_paragraph }` | (1) read tail delta `[byte..]`; (2) delete tail from original; (3) `blocks.insert_container(idx+1, LoroMap)`; set `"kind"="p"`, `"style"` = original's style; create `"text"`; insert tail segments with marks; (4) insert `BindingRow { paragraph_id: Some(new_paragraph), block_id: document.ids.block_ids[idx+1], … }`. | +| `JoinParagraphs { first, second }` | Degenerate cross-paragraph delete — same code path as `DeleteRange`. | | `SetParagraphStyle { paragraph, style }` | `row.map.insert("style", encode(style))`. | -| `SetRunStyles { paragraph, range, styles }` | for each of the 4 keys: if target value non-default → `mark_utf8(range, key, value)`; else → `unmark` over the range (use the utf8 variant per §5.5). | -| `ReplaceParagraphSpan { start_paragraph, before, after }` | **workhorse**, see §6.1. | -| `InsertBlock { block, block_ix }` | read `document.blocks[block_ix]` (payload is NOT in the op); paragraph → create as in split; object → create map with `"kind"`,`"data"`(=postcard `BlockPayload`),`"rev"=0`. Insert binding row with `block_id = block`. | -| `DeleteBlock { block }` | movable-list delete at `binding.by_block[block]`; remove row. | -| `MoveBlock { block, new_block_ix }` | `blocks.mov(old_ix, new_block_ix)`; rotate binding rows. | -| `ReplaceBlock { block: Some(id) }` | re-serialize `document.blocks[ix]` → `map.insert("data", payload)`; `"rev" += 1`. `block: None` → resolve via the *single changed block* by comparing kinds/versions (flowtext bumps `Block::*.version` on edit) — if ambiguous, fall back to ReplaceDocument semantics. | -| `ReplaceDocument` | clear the movable list, rebuild every container from `document` (reuse `projection::populate_from_document`), rebuild binding. Rare (e.g. recovery/import paths, `block_insertion.rs:309`, `object_selection.rs:461`). | - -### 6.1 `ReplaceParagraphSpan` translation - -flowtext wraps most compound edits (paste, backspace at boundary, IME, style commands across paragraphs, drag-drop) into one coarse op carrying full `before`/`after` `DocumentSpan`s (the captured range deliberately includes ±1 neighbor paragraphs — `edit_pipeline.rs:140 edit_capture_range`). Algorithm: - -1. `start = binding.by_paragraph[start_paragraph]` (fallback: `before.start_paragraph` index — keep the same fallback the existing `apply_canonical_operation` uses, `lifecycle.rs:414`). -2. Positional pairing: paragraphs `0..min(n_before, n_after)` are *matched*; surplus `after` paragraphs are *inserted* at the end of the span; surplus `before` paragraphs are *deleted*. (This matches flowtext's own positional identity reconciliation, and because spans include unchanged neighbor paragraphs, edge pairs typically no-op.) -3. For each matched pair `(b, a)`: - * if text and runs are byte-identical → skip; - * else: `row.text.update(a_text, UpdateOptions)` (loro's built-in Myers diff produces minimal CRDT ops); then reconcile marks: compute the desired mark intervals from `a.runs`; clear-and-remark only if runs differ from `b.runs` (compare run vectors; paragraphs are small — full remark of one paragraph is acceptable and *simple*); update `"style"` if changed. -4. Inserted paragraphs: create containers as in `SplitParagraph`, reading `style`/runs/text from the `after` span, binding to the ids flowtext minted at those indices (`document.ids.paragraph_ids[start + i]`). +| `SetRunStyles { paragraph, range, styles }` | For each of the 4 keys: target value non-default → `mark_utf8(range, key, value)`; default → unmark over the range. | +| `ReplaceParagraphSpan { start_paragraph, before, after }` | The workhorse — §6.1. | +| `InsertBlock { block, block_ix }` | Payload is NOT in the op — read `document.blocks[block_ix]`. Paragraph → as in split. Object → new map: `"kind"`, `"data"` = `payload_from_block`, `"rev"=0`. Bind with `block_id = block`. | +| `DeleteBlock { block }` | Movable-list delete at `binding.by_block[block]`; remove row. | +| `MoveBlock { block, new_block_ix }` | `blocks.mov(old_ix, new_block_ix)` (§23-V6 for exact name); rotate binding rows to match. | +| `ReplaceBlock { block: Some(id) }` | Re-serialize `document.blocks[binding.by_block[id]]` → `map.insert("data", payload)`; `rev += 1`. `block: None` → locate the single changed block by comparing `Block::*.version` fields against a kept shadow, else fall through to `ReplaceDocument` handling. | +| `ReplaceDocument` | Clear the movable list; `projection::populate_from_document(doc, document)`; `binding = DocBinding::build(...)`. Heavy but rare (recovery/import paths: `block_insertion.rs:309`, `object_selection.rs:461`). | + +### 6.1 `ReplaceParagraphSpan` translation (precise algorithm) + +flowtext wraps most compound edits (paste, backspace at a boundary, IME commit, multi-paragraph style commands, drag-drop) in one coarse op carrying full `before`/`after` `DocumentSpan`s. The captured range deliberately includes ±1 neighbor paragraphs (`edit_pipeline.rs:140 edit_capture_range`), so edge pairs usually no-op. + +1. `start = binding.by_paragraph[start_paragraph]`, falling back to `before.start_paragraph` exactly as the existing applier does (`lifecycle.rs:414`). +2. Pair positionally: paragraphs `0..min(n_before, n_after)` are *matched*; surplus `after` → *inserted* after the matched run; surplus `before` → *deleted*. (Matches flowtext's own positional identity reconciliation.) +3. For each matched pair `(b, a)` at row `r`: + a. If text bytes and run vectors are identical → skip. + b. Else if text differs: `row.text.update(a_text, UpdateOptions::default())` — loro's built-in Myers diff produces minimal CRDT ops (§23-V3 for `UpdateOptions` fields). + c. If run vectors differ: full mark refresh for the paragraph — for each of the 4 keys, unmark `[0..len)`, then re-apply that key's intervals from `mark_intervals_from_runs(a.runs)`. Paragraphs are small; simplicity beats a minimal mark diff. (Optimization later only with profiler evidence.) + d. If `b.style != a.style`: update `"style"`. +4. Inserted paragraphs: create containers as in `SplitParagraph`, reading style/runs/text from the `after` span; bind to the ids flowtext minted at those indices (`document.ids.paragraph_ids[start + i]`, `block_ids` likewise). 5. Deleted paragraphs: movable-list delete + drop rows. -6. Debug-assert binding invariant (§5.4). +6. `binding.assert_consistent(doc, document)`. -**Correctness note:** `text.update()` + concurrent remote edits to the same paragraph can interleave at diff granularity rather than keystroke granularity. That is normal CRDT behavior (same class as Google Docs' transform granularity) and converges; do not try to be cleverer. +**Convergence note:** `text.update()` interleaves with concurrent remote edits at diff granularity rather than keystroke granularity. That is normal CRDT behavior; it converges; do not attempt to be cleverer. -### 6.2 The double-apply trap (must implement exactly) - -`insert_single_grapheme_fast_path` (`edit_pipeline.rs:44-73`) **merges** consecutive single-grapheme inserts into the *previous* undo record by mutating its `canonical_operations[0].text` in place. If the collab layer naively re-reads "the last record's ops" after each notify, the second keystroke would re-apply the whole accumulated string ("h", then "he", then "hel"…). - -Fix (part of M3's editor changes): the editor gets an explicit **pending collab queue** that receives exactly the delta of each mutation: - -```rust -// editor/mod.rs additions -pending_collab_edits: Vec, // drained by the session -``` +### 6.2 The double-apply trap (implement exactly; regression test mandatory) -* In the fast path: push `CollaborationEdit { operations: vec![InsertText{ …, text: just_this_grapheme }] }` regardless of undo-record merging. -* In `mark_document_changed_with_reconcile`: push the new record's canonical ops **only when a new record was pushed this call** (pass the ops in, don't re-read `undo_stack.last()`). -* Undo/redo paths must **not** push (they're handled by the loro UndoManager, §8) — gate pushes on "not currently restoring history" (`after_history_restore` path) and "not currently applying remote" (a `suppress_collab_capture: u32` counter the RemoteApplier increments, mirroring the existing `suppress_mutation_notify` pattern). -* `take_pending_collab_edits() -> Vec` drains. Keep `last_collaboration_edit` accessors untouched for compatibility. +`insert_single_grapheme_fast_path` (`edit_pipeline.rs:44-73`) **merges** consecutive single-grapheme inserts into the *previous* undo record by mutating `canonical_operations[0].text` in place. Any design that re-reads "the last record's ops" after each notify will re-apply the accumulated string ("h", then "he", then "hel" …). Therefore the editor gets an explicit **pending queue receiving exactly the per-mutation delta** (§15.2): the fast path pushes a one-grapheme `InsertText` regardless of undo-record merging; the general path pushes a record's ops exactly once, at record creation. The queue is the only thing the session reads. -Add a unit test in gpui-flowtext: type "a","b","c" via the fast path → drained queue is exactly three 1-char `InsertText` ops. +gpui-flowtext unit test (new `rich_text/tests/collab_capture.rs`): type `"a"`, `"b"`, `"c"` through the fast path → drained queue is exactly three 1-byte `InsertText` ops; then one paste → exactly one `ReplaceParagraphSpan`; then undo → queue stays empty. --- ## 7. RemoteApplier: loro diff → flowtext `Document` -`remote_apply.rs` (pure: produces patches) + `editor/collab_apply.rs` (applies them via editor internals). +`remote_apply.rs` (pure: consumes diff events, emits patches) + `editor/collab_apply.rs` (applies patches through editor internals). -Subscribe once per session: `doc.subscribe_root(...)` (or `doc.subscribe(&blocks_container_id, ...)`) — the callback receives `DiffEvent`s **synchronously during `import()`/undo**, on the calling thread (main thread — good). Filter by trigger: only process events whose trigger is *Import* or *Checkout/Undo*; *Local* events are echoes of the LocalApplier and must be ignored (§23-V4 for the exact enum name). +Subscribe once per session (`doc.subscribe_root` or the blocks-container subscription — §23-V4). Loro delivers `DiffEvent`s **synchronously during `import()` and undo, on the calling thread** (the main thread — exactly what we want). Filter by trigger kind: process only *Import* and *Checkout/Undo* events; *Local* events are echoes of the LocalApplier and must be ignored. -Collect into a patch list (pure data, unit-testable): +Patch type — defined in **gpui-flowtext** (`collaboration.rs`), so the editor crate never imports collab types (`flowstate-collab` already depends on gpui-flowtext; the dependency arrow stays one-way): ```rust -pub enum DocPatch { - ParagraphText { row: usize, new: InputParagraph, // full regenerated paragraph - delta_utf8: Vec }, // retain/insert/delete in utf8 bytes, for selection remap +// gpui-flowtext/src/collaboration.rs (addition) +pub enum CollabPatch { + ParagraphText { row: usize, new: InputParagraph, + delta_utf8: Vec }, // Retain(usize)|Insert(usize)|Delete(usize), utf8 units ParagraphStyle { row: usize, style: ParagraphStyle }, - BlockData { row: usize, payload: BlockPayload }, - Insert { row: usize, blocks: Vec }, // built from new containers - Delete { row: usize, count: usize }, - Move { from: usize, to: usize }, + ReplaceObjectBlock { row: usize, block: InputBlock }, // image/equation/table payload swap + InsertBlocks { row: usize, blocks: Vec }, + DeleteBlocks { row: usize, count: usize }, + MoveBlock { from: usize, to: usize }, + AssetArrived { id: AssetId, record: AssetRecord }, // from the asset fetcher, not from loro } ``` -Mapping rules: +Mapping rules (`remote_apply.rs`): -* **Text diff on a paragraph container** → regenerate that paragraph wholesale from loro (`to_delta()` → `InputParagraph`) — never incremental-patch flowtext runs from deltas (regeneration is O(paragraph) and immune to drift). Also emit the utf8-converted delta for caret remapping. -* **Movable-list diff** → `Insert`/`Delete`/`Move` patches; for inserts, build `InputBlock`s by reading the new containers (`projection::input_block_from_container`). -* **Map diff** with `"style"` → `ParagraphStyle`; with `"data"`/`"rev"` → `BlockData`. +* **Text diff on a paragraph container** → regenerate the whole paragraph from loro (`to_delta()` → `input_runs_from_delta`) — never incremental-patch flowtext runs from deltas; regeneration is O(paragraph) and immune to drift. Also emit the event delta converted to utf8 (`CollabTextDelta`) for caret remapping. +* **Movable-list diff** → `InsertBlocks` / `DeleteBlocks` / `MoveBlock`; for inserts, read the new containers (`projection::input_block_from_container`), mint fresh local ids, splice binding rows. +* **Map diff**: `"style"` → `ParagraphStyle`; `"data"`/`"rev"` → `ReplaceObjectBlock` (decode payload; if it's an image whose asset is missing locally, also enqueue an asset fetch, §16). -`collab_apply.rs` (editor side) applies a batch: +`collab_apply.rs` (editor side): ```rust impl RichTextEditor { - pub fn apply_collab_patches(&mut self, patches: &[DocPatch], cx: &mut Context) { … } + pub fn apply_collab_patches(&mut self, patches: &[CollabPatch], cx: &mut Context) { … } + pub fn collab_apply_deferred(&self) -> bool { … } // §7.3 predicate, checked by the session before calling } ``` -Implementation requirements (use existing machinery, don't reinvent): +Implementation requirements (reuse existing machinery, do not reinvent): -1. Wrap the whole batch in `suppress_collab_capture += 1` (so these mutations never echo back) and reuse the `layout_invalidation_hint` + `after_text_mutation(cx)` pattern from `apply_remote_operations` (`lifecycle.rs:265-272`) for cache invalidation. -2. `ParagraphText`: build the replacement via existing span machinery — `capture_document_span` + `apply_document_span_replacement` for the single paragraph, or a focused `replace_paragraph_content(document, ix, InputParagraph)` helper added to `edit_ops` (new file `edit_ops/collab.rs` if it doesn't fit an existing one). Preserve `ParagraphId` (identity unchanged), bump paragraph `version`. -3. Structural patches: splice `document.blocks`/`paragraphs`/`ids` with **explicitly provided ids** (from the binding rows the RemoteApplier created) — do not let `reconcile_document_ids` invent them (call it only to assert no-op afterwards in debug). -4. **Selection remap:** if `self.selection` head/anchor sits in a patched paragraph, walk `delta_utf8` (retain/insert advance, delete clamps) to the new byte offset; if its paragraph was deleted, clamp to the nearest surviving offset. Same for `table_cell_*`/`equation_source_*` editing offsets: if the active table/equation block got `BlockData`-replaced remotely, exit the cell/equation editing mode gracefully (clear `selected_block`, keep selection at block boundary). -5. **Undo safety:** remote patches invalidate positional undo records. Per §8, during a session the local undo stack is unused (undo routes through loro). On session start, clear `undo_stack`/`redo_stack`; in `apply_collab_patches`, debug-assert they stay empty. -6. **Deferral:** if `self.selecting || self.active_text_drag.is_some() || self.image_resize_drag.is_some() || self.table_column_resize_drag.is_some()` or IME composition is active (marked-text state in `editor/platform.rs`), the session queues patches and retries on the next observe tick / 16 ms timer instead of mutating mid-gesture. +1. Wrap the batch in `suppress_collab_capture += 1` (so nothing echoes into `pending_collab_edits`) and reuse the `layout_invalidation_hint` + `after_text_mutation(cx)` pattern from `apply_remote_operations` (`lifecycle.rs:265-272`) for cache invalidation. Set `layout_invalidation_hint` to the smallest covering paragraph range per patch group. +2. `ParagraphText`: replace content via the span machinery (`capture_document_span` + `apply_document_span_replacement` scoped to one paragraph) or a focused `replace_paragraph_content(document, ix, InputParagraph)` helper added in a new `edit_ops/collab.rs`. **Preserve the existing `ParagraphId`** (identity unchanged); bump the paragraph `version` field so layout caches invalidate. +3. Structural patches: splice `document.blocks` / `paragraphs` / `ids` with the **ids provided in the patch path** (minted by the RemoteApplier) — never let `reconcile_document_ids` invent ids here; call it afterwards in debug and assert it was a no-op. +4. **Selection remap:** if `selection.head`/`anchor` sits in a patched paragraph, walk `delta_utf8` (retain advances, insert advances by inserted len if at-or-before the offset, delete clamps) to the new byte; if its paragraph was deleted, clamp to the start of the next surviving block (or end of previous). Apply the same remap to `table_cell_*` / `equation_source_*` offsets; if the actively-edited table/equation block got replaced remotely, exit the cell/equation editing mode cleanly (clear `selected_block`, selection to block boundary). +5. **Undo safety:** while attached, the native undo stack is unused (§8) — on attach both stacks are cleared; debug-assert they remain empty in `apply_collab_patches`. -### 7.1 Bootstrap & full resync +### 7.1 Bootstrap & rebuild -Join: import host snapshot into a fresh configured `LoroDoc` (background executor — `LoroDoc` is Send), then on the main thread run `projection::document_from_loro(&doc, theme)` → `Vec` → build `Document` (existing input-builder path) → `editor.replace_document_from_collaboration(document, cx)` → build `DocBinding` by walking containers. Resync (integrity failure or reconnect-too-stale): identical path, plus selection clamp. +* **Join:** `schema::new_configured_doc()` → `doc.import(snapshot_bytes)` on the background executor (`LoroDoc` is Send) → verify `meta.session == ticket.session` (§12.2) → on the main thread `projection::document_from_loro(&doc, load_document_theme())` → `Vec` → the existing input-builder → `editor.replace_document_from_collaboration(document, cx)` → `DocBinding::build`. Gossip updates received while importing are buffered by the session and imported afterwards (loro pending-import absorbs ordering). +* **Local self-heal rebuild** (§7.2): identical projection path against the *existing* doc, plus selection clamp to nearest valid offset, plus a non-blocking "Document view rebuilt" notification in debug builds. +* `projection.rs` also provides the inverse, `populate_from_document(&LoroDoc, &Document)` — used once at session creation and by `ReplaceDocument`. -`projection.rs` also provides the inverse, `populate_from_document(&LoroDoc, &Document)` — used by the **host once at session start** (and by `ReplaceDocument`). Round-trip property test: `document → loro → document` is identity on (text, runs, styles, block structure, table/equation/image payloads), modulo `AssetRecord.bytes` (§16). +Round-trip property test: `document → populate → project → document'` is identity on (texts, run vectors, styles, block kinds, payload bytes) — everything except `AssetRecord.bytes` and local ids. -### 7.2 Integrity checking (belt & suspenders) +### 7.2 Integrity: local-first self-check (no authority needed — by design) -`integrity.rs`: `projection_hash(&Document) -> u64` — twox-hash over (paragraph styles, run vectors, paragraph texts, block kinds + payload bytes, in order; explicitly NOT ids/theme/sections/offset-index). Host piggybacks `Frame::IntegrityProbe { vv: Vec, hash: u64 }` every 30 s of activity or every 256 relayed updates. Joiner compares **only when version vectors are equal** (otherwise skip — they're mid-flight); on mismatch → log loudly, send `Frame::ResyncRequest`, host responds with fresh snapshot, joiner rebuilds (§7.1) and posts a non-blocking notification ("Document resynced"). This converts any residual translation bug from "silent divergence forever" into "blip + log we can fix." +The flowtext `Document` is derived state; divergence therefore decomposes into two cases with different owners: ---- +* **Projection drift** (an applier bug — the realistic case): the live `Document` no longer equals `project(own LoroDoc)`. Detect **locally**: every 30 s of idle (no edits for 2 s, timer in the session), compute `self_check::projection_hash(&Document)` and compare with the hash of a freshly projected document from the own doc (background executor; compare on main thread). On mismatch: `eprintln!` a structured diagnostic (both hashes, oplog vv, last 5 patch kinds applied), then **self-heal** via the §7.1 rebuild. No peers involved. `projection_hash` = twox-hash over, in order: paragraph styles, run `(len, styles)` vectors, paragraph text bytes, block kinds, payload bytes. Explicitly excluded: ids, theme, sections, offset index, assets. +* **CRDT-level divergence** (same oplog vv, different loro state): impossible unless lineage mixing (§12.2 prevents) or a loro bug. Guarded by the session-id check on every digest (§10.4); a mismatch there is fatal — detach with an error dialog, never attempt merge. Optional deep cross-peer state-hash comparison ships behind `FLOWSTATE_COLLAB_PARANOIA=1` for QA builds only. -## 8. Undo/redo during a session +### 7.3 Deferral around gestures -Positional `EditRecord` replay is unsound once remote edits interleave. While a session is active: +Remote patches must not mutate the document mid-gesture. Before applying, the session checks `editor.collab_apply_deferred()`: +`self.selecting || self.active_text_drag.is_some() || self.image_resize_drag.is_some() || self.table_column_resize_drag.is_some() || ime_composition_active()` (marked-text state lives in `editor/platform.rs` — expose a query). If true, the session queues patches and retries on the next editor observe tick or a 16 ms timer, whichever first. Queue is unbounded in theory; in practice gestures are short — add a debug log if the queue exceeds 256 patches. -* Session creates `loro::UndoManager::new(&doc)` (tracks only this peer's ops — exactly Google-Docs semantics), with merge interval ≈ 500 ms to mirror flowtext's grapheme coalescing; configure `set_max_undo_steps(undo-stack depth ~ 300)`. -* Editor gains a redirect hook (set by the session, cleared on session end): +--- -```rust -// editor/mod.rs -pub enum UndoRedirect { Undo, Redo } -collab_undo_redirect: Option>, // Rc: entity is single-threaded -``` +## 8. Undo/redo while attached -`commands.rs:191 undo()` / `:205 redo()`: if redirect set → invoke it and return. The session handler calls `undo_manager.undo()` / `.redo()`; resulting doc changes surface through the same DiffEvent → `DocPatch` → `apply_collab_patches` pipeline (trigger = checkout/undo, so they pass the §7 filter; ensure the LocalApplier echo-suppression doesn't eat them). -* Caret restore: wire `UndoManager`'s cursor callbacks (`set_on_push`/`set_on_pop` storing the selection as two loro `Cursor`s; resolve with `doc.get_cursor_pos` after pop). If the cursor API fights back, v1 fallback: leave selection where the patch remap puts it. (§23-V5) -* On session start: clear editor undo/redo stacks. On session end: stacks start fresh from the final state (history across the boundary is intentionally dropped — banner copy mentions it). -* Host before session / after session uses the existing native undo unchanged. +Positional `EditRecord` replay is unsound once remote edits interleave. While attached: + +* The session creates `loro::UndoManager::new(&doc)` at attach. It tracks **only this peer's ops** (precisely Google-Docs semantics: you can retract your own edits, never someone else's; positions transform against concurrent remote edits automatically). Configure: merge interval ≈ 500 ms (mirrors flowtext's grapheme coalescing), max steps ≈ 300. +* The editor gets a redirect hook (§15.4). `commands.rs:191 undo()` / `:205 redo()` check it first; when set, they invoke it and return. The session's handler calls `undo_manager.undo()` / `.redo()`; resulting doc changes arrive through the same DiffEvent → patch → `apply_collab_patches` pipeline (trigger = checkout/undo passes the §7 filter; verify the LocalApplier echo-filter does not eat them — they are *not* `origin="local-edit"` commits). +* Caret restore: wire `UndoManager::set_on_push` / `set_on_pop` storing the selection as two loro `Cursor`s and resolving via `doc.get_cursor_pos` after pop (§23-V5). If the cursor API resists, v1 fallback: selection stays where patch remap leaves it. +* On attach: clear `undo_stack`/`redo_stack`. On detach: stacks restart empty from the final state — history does not cross the membership boundary (mention in user docs). +* **Post-merge cleanup semantics (decided):** a reconnect merge is not an operation and cannot be "undone" as a unit. Resolution story: each author can undo their own contributions step-wise; anything else is fixed by ordinary editing (nothing is ever locked). An hour offline ≈ many undo steps — select-and-delete is often the practical tool; that's acceptable and matches the Google Docs model (minus version history, which v1 does not have). --- ## 9. Presence (carets, names, roster) -`presence.rs` wraps `loro::awareness::EphemeralStore` (timestamped LWW KV with per-key timeout, made for exactly this): - -* Key = peer's `EndpointId` string. Value (postcard → loro binary value): +`presence.rs` wraps `loro::awareness::EphemeralStore` (timestamped LWW KV with per-key timeout — purpose-built for this): ```rust -pub struct PresenceState { - pub name: String, // display name - pub color_ix: u8, // palette index assigned by host in Welcome - pub role: Role, // Owner | Editor | Viewer (informational; enforcement is host-side) +pub struct PresenceState { // postcard → store value under key = hex(EndpointId) + pub name: String, // self-asserted display name (app settings; default = OS username) pub selection: Option, } pub struct PresenceSelection { - pub container: String, // ContainerID of the paragraph text (head) - pub head: Vec, // loro Cursor::encode() — survives concurrent edits + pub head_container: String, // ContainerID of the paragraph text holding the head + pub head: Vec, // loro Cursor::encode() — survives concurrent edits pub anchor_container: String, pub anchor: Vec, } ``` -* Timeout 30 s; the store's local-update subscription yields raw bytes → `Frame::Ephemeral(bytes)`; inbound frames → `store.apply(bytes)`; host relays ephemeral frames like doc updates. -* Update cadence: on `EditorEvent::SelectionChanged` (subscribe via `cx.subscribe` — `RichTextEditor` must be an `EventEmitter`; if it currently only pushes to `EditorEventSink`, add the `EventEmitter` impl in M3), debounced 50 ms; plus a 10 s keepalive refresh. -* Rendering: on ephemeral merge events + after every remote apply, resolve each peer's cursors: `Cursor::decode` → `doc.get_cursor_pos` → unicode pos → utf8 byte → `binding.by_container` → `DocumentOffset` → `editor.set_external_carets(vec![ExternalCaret { offset, color_rgb }], cx)`. v1 renders caret + name-on-hover only (selection-range highlight is v1.1; `ExternalCaret` keeps its current shape). -* Color palette: 8 hardcoded high-contrast RGBs in `session_core.rs`; host assigns by join order, reuses freed slots. +* **Colors are not negotiated**: `color_ix = hash(EndpointId) % PALETTE.len()` with `PALETTE: [u32; 8]` of high-contrast RGBs in `ids.rs`. Stable per peer, no coordination, collisions tolerable at session scale. +* Store timeout 30 s. Outbound: the store's local-update subscription yields raw bytes → `GossipMsg::Presence(bytes)` broadcast. Inbound: `store.apply(bytes)`. Refresh own entry every 10 s (keepalive) and within 50 ms (debounced) of `EditorEvent::SelectionChanged` (subscribe via `cx.subscribe`; ensure `RichTextEditor: EventEmitter` — §15.7). +* **Leave is explicit in presence:** before unsubscribing, best-effort broadcast a self-removal (delete own key / `remove_outdated` semantics — §23-V7 for the exact removal API) so others see the departure immediately rather than after 30 s. +* **Roster = presence, nothing else.** Gossip `NeighborUp/Down` events are connectivity hints for the state machine (§13.2), not membership truth — your neighbors are a random subset of the swarm. The participant list, count, and "Only you" detection all read the presence store (entries newer than timeout). +* Rendering: on presence merge events and after every remote apply batch, resolve every peer's cursors — `Cursor::decode` → `doc.get_cursor_pos` → loro pos → `utf8_byte()` → `binding.by_container` row → paragraph ordinal → `DocumentOffset` → `editor.set_external_carets(carets, cx)`. Unresolvable cursors (container deleted) → drop that caret until the peer's next selection update. v1 renders caret + name label on hover; selection-range highlight is v1.1 (`ExternalCaret` keeps its current shape). --- -## 10–13. Networking, protocol, tickets, lifecycle +## 10. Networking ### 10.1 Runtime bridge (`net/runtime.rs`) -* Lazy global: first share/join spawns `std::thread::Builder::new().name("flowstate-collab-net")` running a `tokio::runtime::Builder::new_multi_thread().worker_threads(2).enable_all()` runtime; thread parks on `runtime.block_on(net_main(cmd_rx, evt_tx))`. -* `net_main` builds **one iroh `Endpoint` for the whole app** (shared by hosting and joining): +* Lazy global, started on first share/join: `std::thread::Builder::new().name("flowstate-collab-net".into()).spawn(...)` running `tokio::runtime::Builder::new_multi_thread().worker_threads(2).enable_all()`; the thread parks in `runtime.block_on(net_main(cmd_rx, evt_tx))`. +* `net_main` constructs, once per app: ```rust -let endpoint = Endpoint::builder() // use the n0 preset: relays + address lookup - .alpns(vec![ALPN.to_vec()]) // ALPN = b"flowstate/collab/0" - /* preset/address-lookup wiring per §23-V1 */ +let endpoint = Endpoint::builder() // wire the n0 preset: relays + address lookup/publish (§23-V1) + .alpns(vec![DIRECT_ALPN.to_vec()]) // gossip registers its own ALPN via the Router .bind().await?; +let gossip = Gossip::builder().spawn(endpoint.clone()); // §23-V2 for exact constructor +let _router = Router::builder(endpoint.clone()) + .accept(iroh_gossip::ALPN, gossip.clone()) // Gossip impls ProtocolHandler + .accept(DIRECT_ALPN, DirectProto::new(serve_state)) // §10.3 + .spawn(); ``` -Discovery is NOT on by default in iroh 0.98 — apply the n0 preset (`presets::N0` / `Builder::new(preset)`) so relay + address lookup work; additionally the ticket embeds the full `EndpointAddr` (relay URL + direct addrs) so joining works even if lookup is cold (§23-V1). -* Channels: `async_channel::unbounded::()` (UI→net) and `::()` (net→UI). The UI side pumps events with a detached `cx.spawn` loop in `CollabManager` that routes to the right `CollabSession` entity. +Discovery is **not** on by default in iroh 0.98 — apply the n0 preset (`presets::N0` / `Builder::new(preset)`) for relay + address lookup/publish. Tickets additionally embed the inviter's full `EndpointAddr` (relay URL + direct addrs) so a fresh install can join with zero warm-up. After `bind()`, await the endpoint's first reachable address before minting any ticket (§23-V1 names the watcher API). +* Channels: `async_channel::unbounded::()` (UI→net) and `::()` (net→UI). The UI side pumps events in `CollabManager` with one detached `cx.spawn` loop, routing by `SessionId` to the registered `CollabSession` entity. +* App quit: `CollabManager::shutdown` sends `NetCommand::Shutdown`; net thread broadcasts presence-removals for all attached sessions (best-effort, 500 ms budget), unsubscribes, closes the endpoint, exits. ```rust +// net/mod.rs pub enum NetCommand { - HostSession { session: SessionId, token: SessionToken, default_role: Role, reply: oneshot }, - CloseSession { session: SessionId }, // host: broadcast Bye, close conns - Join { ticket: SessionTicket, hello: HelloInfo }, - LeaveSession { session: SessionId }, - SendTo { session: SessionId, peer: EndpointId, frames: Vec }, - Broadcast { session: SessionId, frame: Frame, except: Option }, + EnsureUp, // idempotent endpoint warm-up + CreateSession { session: SessionId, reply: Tx> }, // subscribe w/ no bootstrap + JoinSession { session: SessionId, bootstrap: Vec }, + LeaveSession { session: SessionId }, // presence-removal already broadcast by UI side + Publish { session: SessionId, payload: PublishPayload }, // Update(Vec) | Presence(Vec) | Digest(..) + PullUpdates { session: SessionId, from: EndpointId, our_vv: Vec, reply: Tx>> }, + PullSnapshot { session: SessionId, from: EndpointId, reply: Tx>> }, + PullBlob { session: SessionId, from: EndpointId, blob: BlobId, reply: Tx>> }, + PullAsset { session: SessionId, from: EndpointId, asset: u128, reply: Tx> }, + MintTicketAddr{ reply: Tx }, // current own addr for re-minted invites Shutdown, } pub enum NetEvent { - HostReady { session: SessionId, ticket: SessionTicket }, - PeerConnected { session: SessionId, peer: EndpointId, hello: HelloInfo }, // host side, post-auth - Frame { session: SessionId, peer: EndpointId, frame: Frame }, - PeerDisconnected{ session: SessionId, peer: EndpointId, reason: String }, - JoinEstablished { session: SessionId }, // joiner side, post-Welcome (Welcome arrives as Frame) - JoinFailed { session: SessionId, reason: JoinError }, - Reconnecting { session: SessionId, attempt: u32 }, - SessionClosed { session: SessionId, reason: CloseReason }, + Gossip { session: SessionId, from: EndpointId, msg: GossipMsg }, + NeighborUp { session: SessionId, peer: EndpointId }, + NeighborDown { session: SessionId, peer: EndpointId }, + GossipLagged { session: SessionId }, // receiver overflow → treat as a gap; trigger §10.4 pull + SubscribeFailed { session: SessionId, error: String }, + EndpointOnline(bool), // connectivity hint for §13.2 } ``` -### 10.2 Host accept path (`net/host.rs`) +(`Tx` = `async_channel::Sender` with `bounded(1)` as the oneshot idiom.) + +### 10.2 Swarm layer (`net/swarm.rs`) + +One tokio task per attached session. Responsibilities: -`Router::builder(endpoint).accept(ALPN, FlowstateProtocol{state}).spawn()` with `impl ProtocolHandler` (`async fn accept(&self, connection: Connection) -> Result<(), AcceptError>`). Per connection: `accept_bi()` → read `Frame::Hello` (5 s deadline) → validate `protocol_version`, `session_id` exists, `token` matches (constant-time compare), peer not already connected, roster < MAX_PEERS(8) → register peer (writer task owns the send half + an mpsc inbox; reader task forwards frames as `NetEvent::Frame`) → emit `PeerConnected`. The **session entity** (not the net thread) decides the `Welcome` content because the snapshot must be exported atomically with update-stream attachment: on `PeerConnected`, the session synchronously (a) exports `ExportMode::Snapshot` (or `updates(vv)` for a reconnect Hello carrying `last_vv`), (b) marks the peer live for future broadcasts — both under the same borrow of the session state so no update can fall between snapshot and stream. Then `SendTo(Welcome{...})`. +* `gossip.subscribe(topic_id, bootstrap_ids)` → split into sender/receiver halves (§23-V2 for exact types: `GossipTopic` → `GossipSender`/`GossipReceiver`, events `Received/NeighborUp/NeighborDown/Lagged`). Before subscribing with bootstrap peers, register their `EndpointAddr`s with the endpoint so dialing can proceed without discovery (§23-V1 for the add-address method). +* Receive loop: decode `GossipMsg` (postcard); forward as `NetEvent::Gossip`; surface `NeighborUp/Down/Lagged`. +* Send path: `Publish` commands → `sender.broadcast(bytes)`; digests → `sender.broadcast_neighbors(bytes)` (neighbor-only, §23-V2). +* Re-subscription on total neighbor loss is **not** the swarm task's job — HyParView self-heals while ≥1 reachable peer remains; the §13.2 Offline logic only re-bootstraps when the overlay is empty *and* a known peer address is reachable again. -Relay rule: every authenticated inbound `DocUpdate`/`Ephemeral` from peer P → import/apply on host AND `Broadcast{except: P}` *of the same bytes*. Updates from a `Viewer` peer: **dropped, not imported, not relayed** (and the host replies `Frame::Rejected{reason: ReadOnly}` so a buggy client can self-correct via resync). +### 10.3 Direct protocol (`net/direct.rs`, ALPN `b"flowstate/collab-direct/0"`) -### 10.3 Joiner path (`net/joiner.rs`) +Request/response over a **fresh bi-stream per request** on a (cached) direct connection. Server side implements `iroh::protocol::ProtocolHandler` (`async fn accept(&self, connection: Connection) -> Result<(), AcceptError>`): loop `accept_bi`, read one length-prefixed `DirectRequest`, answer, finish. -`endpoint.connect(ticket.addr, ALPN)` → `open_bi()` → send `Hello` → await `Welcome` (15 s deadline) → emit `JoinEstablished` + deliver Welcome frame → then a read loop + write loop. Reconnect: on connection loss (not explicit `Bye`/`SessionClosed`), loop with backoff 0.5/1/2/4/8…30 s; each attempt sends `Hello { last_vv: Some(doc.oplog_vv().encode()) }`; host answers `Welcome { doc: Updates(...) }` (cheap catch-up) or `Snapshot` if it can't serve the delta. UI shows "Reconnecting…" via `NetEvent::Reconnecting`; user can cancel (→ orphan copy). +```rust +// proto_direct.rs — all length-prefixed postcard (u32-le len; 2 MiB cap per frame; payload streams chunked) +pub enum DirectRequest { + Snapshot { session: SessionId }, + Updates { session: SessionId, have_vv: Vec }, + Blob { session: SessionId, blob: BlobId }, + Asset { session: SessionId, asset: u128 }, +} +pub enum DirectResponseHeader { + Ok { total_len: u64 }, // followed by raw payload in ≤256 KiB chunks on the same stream + NotAttached, // serving peer left that session + NotFound, // unknown blob/asset + Busy, // over the concurrency limit; caller retries elsewhere +} +``` -### 11. Wire protocol (`protocol.rs`) +Server rules: verify the `session` is one this peer is attached to (the SessionId is the bearer secret — possessing it *is* authorization, §12.1); ≤4 concurrent serves per peer (semaphore; excess → `Busy`); snapshot/updates payloads are produced by calling back into the session's doc handle (cloned `LoroDoc` if clone-shares, else a request channel into the session entity — §23-V6 decides which). -Framing (`net/framing.rs`): `u32-le length` + postcard bytes, max frame 2 MiB (`FrameTooLarge` error closes the connection); one long-lived bi-stream per peer for control/doc/presence; assets use separate per-transfer bi-streams (§16). +Client helper with **peer fallback** (used by join, anti-entropy, blobs, assets): ```rust -pub const PROTOCOL_VERSION: u16 = 1; -pub const ALPN: &[u8] = b"flowstate/collab/0"; +pub async fn pull_with_fallback(req: DirectRequest, candidates: Vec, per_peer_timeout: Duration) + -> anyhow::Result>; +// try candidates in order: (1) the peer the need came from (digest sender / gossip delivered_from / +// ticket inviter), (2) current gossip neighbors, (3) any peer with a live presence entry. +// 10 s per-peer timeout; first success wins; aggregate error names every attempt. +``` + +### 10.4 Anti-entropy (`net/anti_entropy.rs` + session logic) + +Heals every gap gossip can leave: offline windows, `Lagged` receivers, missed blob pulls, joins racing live traffic. + +* Every attached peer, every **10 s ± 20% jitter** (and immediately on `NeighborUp` and on reconnect): `broadcast_neighbors(GossipMsg::Digest { session: SessionId, vv: doc.oplog_vv().encode() })`. +* On receiving a digest: compare version vectors (loro VV comparison — opaque to us beyond `partial_cmp`-style API, §23-V4). If the sender has ops we lack → `PullUpdates { from: sender, our_vv }` → import the returned `export(updates(our_vv))` bytes. **Pull-only** (we never push unsolicited): each peer pulls at most once per detected gap, so no duplicate-suppression protocol is needed. In-flight guard: one outstanding pull per session. +* If our vv ⊋ theirs: do nothing — their own digest cycle will pull from us. +* Digest carries the `SessionId`; a mismatch (≠ this topic's id) is a lineage violation: log + ignore the peer (§12.2). +* `GossipLagged` → schedule an immediate digest broadcast + opportunistic pull from any neighbor. + +### 10.5 Oversized updates (`net/blobs.rs`) + +Gossip messages are size-capped (configurable; default small — §23-V2). Budget: `GOSSIP_INLINE_LIMIT = 2 KiB` post-encoding. + +* Update blob ≤ limit → `GossipMsg::Update(bytes)` inline (the overwhelmingly common case — typing commits are tens of bytes). +* Larger (pastes, big span replaces) → store in the session's **outbox**: `BlobId = u128 random`, ring buffer of the last 64 blobs (or 16 MiB, whichever first); broadcast `GossipMsg::UpdateAvailable { blob: BlobId, len: u64 }`; receivers `PullBlob` from `delivered_from`, fallback per §10.3. A failed/expired pull is harmless: the next anti-entropy round delivers the same ops via `Updates`. (This indirection avoids a chunk-reassembly protocol entirely — decided.) + +### 11. Gossip wire format (`proto_gossip.rs`) + +```rust +pub const PROTOCOL_VERSION: u16 = 1; // first byte(s) of every gossip payload + ticket +pub const DIRECT_ALPN: &[u8] = b"flowstate/collab-direct/0"; #[derive(Serialize, Deserialize)] -pub enum Frame { - Hello { protocol: u16, app: String, session: SessionId, token: SessionToken, - name: String, last_vv: Option> }, - Welcome { roster: Vec, your_color: u8, your_role: Role, - doc: DocBootstrap /* Snapshot(Vec) | Updates(Vec) */, - asset_manifest: Vec }, - Rejected { reason: RejectReason }, // BadToken | Version{min,got} | Full | ReadOnly | UnknownSession - DocUpdate(Vec), // loro export bytes (local update or relay) - Ephemeral(Vec), // EphemeralStore encode bytes - IntegrityProbe { vv: Vec, hash: u64 }, - ResyncRequest, - ResyncSnapshot(Vec), - AssetRequest { id: u128 }, // answered on a NEW bi-stream with AssetHeader+chunks - AssetOffer { meta: AssetMeta }, // joiner → host: "I have a new asset, pull it" - RosterUpdate { roster: Vec }, - Bye { reason: CloseReason }, // SessionEnded | Kicked | HostShutdown - Ping(u64), Pong(u64), // 15 s heartbeat (also drives presence staleness) +pub enum GossipMsg { // postcard, prefixed with PROTOCOL_VERSION u16 + Update(Vec), // loro export bytes (one or more commits) + UpdateAvailable { blob: BlobId, len: u64 }, + Presence(Vec), // EphemeralStore encode bytes + Digest { session: SessionId, vv: Vec }, // broadcast_neighbors only } ``` -Version policy: exact `PROTOCOL_VERSION` match required; mismatch → `Rejected::Version` with a human-readable "host is running a newer/older Flowstate" surfaced in the join dialog. +Version policy: a peer receiving a higher/lower `PROTOCOL_VERSION` ignores the message and surfaces a one-time "a participant is running an incompatible Flowstate version" notification (per-peer, deduped). Joins fail fast instead: the snapshot `DirectResponseHeader` is version-prefixed too, so an incompatible joiner gets a clear error in the join dialog rather than a silent dead session. -### 12. Ticket (`ticket.rs`) +--- + +## 12. Tickets, identity, lineage + +### 12.1 Ticket (`ticket.rs`) ```rust pub struct SessionTicket { - pub addr: EndpointAddr, // includes relay URL + direct addresses - pub session: SessionId, // u128 random - pub token: SessionToken, // [u8; 16] random (rand::rngs::OsRng) — the join secret - pub title: String, // doc display name, shown in join dialog before connecting + pub version: u16, // PROTOCOL_VERSION at mint time + pub session: SessionId, // = the 32-byte gossip TopicId = THE bearer secret + pub inviter: EndpointAddr, // full addr (relay URL + direct addrs) of whoever minted THIS ticket + pub title: String, // doc display name, shown in the join dialog before connecting } ``` -Implement the `iroh-tickets` `Ticket` trait (postcard payload, base32 display) with prefix/kind `"fscollab"`. The string is the only thing users exchange. Threat model note for implementers: the token gates membership; anyone with the ticket can join until the host ends the session — that *is* the product. The host's authority means a malicious joiner can at worst vandalize the shared doc (undoable; host can end session), never touch the host FS. +Implements the `iroh-tickets` `Ticket` trait (postcard payload, base32 display) with prefix `"fscollab"` — the user-visible string looks like `fscollab…`. **Any attached peer can mint a ticket at any time** (share dialog → "Copy invite"); each mint embeds the minter's *current* address (`NetCommand::MintTicketAddr`), so invites never go stale while anyone is around to share them. Threat model: possession of the ticket (specifically the `session` secret) **is** membership — that is the product. The secret never appears in gossip metadata visible to non-members (TopicId is shared only with members by construction; discovery/relay infrastructure sees opaque traffic). -### 13. Session lifecycle state machines +### 12.2 Lineage guard (the one real landmine — implement all three checks) -`crates/flowstate/src/collab/session.rs` — `CollabSession` GPUI entity, one per panel, states: +A loro doc merged with updates from a *different document lineage* produces interleaved garbage (both histories union), silently. Three structural guards: -``` -Host: Idle → Starting(net spawn + populate loro from Document + HostSession cmd) - → Live { ticket, peers } ⇄ (peer churn) - → Ended (user ends / tab closes / app quits → broadcast Bye, drop) -Joiner: Connecting(ticket) → Bootstrapping(snapshot import, build Document, open tab) - → Live ⇄ Reconnecting(attempt n) - → Orphaned(reason) // session over; editor stays, banner shown, collab hooks removed -Errors: Starting/Connecting failures → toast + back to Idle (no tab opened for failed join) -``` +1. **Topics are never reused.** `SessionId` = 32 fresh `OsRng` bytes per session creation. Starting a new session on the same file = new id, new ticket. There is no path that re-subscribes an old topic with new content. +2. **The doc carries its lineage:** `meta.session = hex(session_id)` written once at creation. A joiner verifies `imported_meta.session == ticket.session` immediately after snapshot import, **before** opening the tab; mismatch → join fails with "stale or corrupted invite." +3. **Digests carry the SessionId** (§10.4); a mismatched digest marks the sender incompatible and is ignored. Snapshot/Updates responses are only served for sessions the server is attached to, keyed by the same id. -Wiring on session start (host) / bootstrap (joiner): -1. `editor.set_collaboration_role(Some(role))`, clear undo stacks, set undo redirect (§8). -2. `cx.observe(editor)` → `flush_local_edits` (drain → LocalApplier → commit; the loro local-update subscription then emits broadcast bytes). -3. `cx.subscribe(editor)` for `SelectionChanged` → presence (§9). -4. Register session with `CollabManager` (global) for net-event routing by `SessionId`. -Teardown reverses everything: role `None` (host keeps `Owner`→`None`), redirect cleared, carets cleared, subscriptions dropped. Closing a hosting tab (`remove_document_panel`, `documents.rs:216`) and `request_close_window` must check `CollabManager::session_for_panel(id)` and confirm: "End live session with N participants?". +With these, lineage mixing requires a peer to hold the right 32-byte secret *and* the wrong document — which cannot arise from any v1 flow (joins always build fresh docs; reconnects reuse the same attached doc; v1.1 "resume" must re-verify check 2). --- -## 14. Save semantics & source of truth +## 13. Session lifecycle (the state machine — implement exactly) + +### 13.1 States + +```rust +// session.rs +pub enum SessionPhase { + Creating, // populate loro + subscribe; ~instant + Joining(JoinStage), // §13.3 + Attached(Attachment), // member of the session (the steady state) + Detached(DetachReason), // terminal for this tab; collab machinery torn down +} +pub struct Attachment { + pub connectivity: Connectivity, // §13.2 + pub peers_present: usize, // presence-derived (excluding self) +} +pub enum Connectivity { Online, Offline { since: Instant, retries: u32 } } +pub enum JoinStage { Resolving, Subscribing, FetchingSnapshot { got: u64, total: Option }, Building } +pub enum DetachReason { UserLeft, JoinFailed(String), Fatal(String) } // Fatal: lineage/incompatibility +``` + +UI projection (status pill): `Attached{Online, n>0}` → "● n+1 in session"; `Attached{Online, 0}` → "● Only you"; `Attached{Offline,..}` → "◌ Offline — will sync"; `Joining` → progress; `Detached` → no pill (plain doc). + +### 13.2 Connectivity detection (Attached only) -* **Host:** unchanged save/autosave (`save_active`, `maybe_autosave_document`). Its `Document` projection *is* the file content. CRDT history is **not** persisted into `.db8` — a saved file is a plain snapshot; a future session starts a fresh loro doc (`populate_from_document`). No format changes. -* **Joiner:** editor opens with `document_path = None` → Save triggers Save-As (existing untitled flow), autosave skips, recovery writes skip. The panel title = host's title + " (shared)". -* **Host recovery files** (`schedule_recovery_write`) keep working — they snapshot the projection, which is exactly right. +`Online` ⇨ `Offline` when **all** of: zero gossip neighbors for > 5 s, AND the last 2 digest rounds produced no inbound traffic, AND (`EndpointOnline(false)` OR a probe dial to the most recent known peer fails). Being merely *alone* (peers all left, but the network is fine) stays `Online` with `peers_present == 0` — "Only you," not "Offline." +`Offline` ⇨ recovery loop: backoff 1 / 2 / 4 / … / 30 s (cap), each attempt: re-register known peer addresses (passive view + last presence-seen peers, retained in the session), re-subscribe the topic with them as bootstrap, await first `NeighborUp` (10 s). On success: immediate digest broadcast (pulls what we missed; our accumulated local commits flow out via the peers' own digest pulls and normal broadcast of new edits) ⇨ `Online`. The user can Leave at any time during `Offline`. + +### 13.3 Join sequence (numbered; every step has a failure edge → `Detached(JoinFailed)`) + +1. Parse + version-check the ticket (reject with specific copy: malformed / newer-version / self-join when `inviter.endpoint_id == own endpoint.id()` — friendly "that's your own invite"). +2. `EnsureUp` the runtime/endpoint. +3. Register `ticket.inviter` address; `JoinSession { session, bootstrap: [inviter] }`. Await first `NeighborUp` — 15 s timeout ("couldn't reach anyone in this session"). +4. `PullSnapshot` via `pull_with_fallback([inviter ∪ neighbors])` — 30 s overall budget; progress streamed to `JoinStage::FetchingSnapshot`. +5. Background import into `new_configured_doc()`; verify `meta.session` (§12.2). +6. Main thread: project → `Document` → create the tab — `workspace.create_document_panel(document, /*path*/ None, Some(format!("{} (shared)", meta.title)), …)` — then `editor.set_recovery_path(Some(session_recovery_path(session)))` (§15.6). +7. `DocBinding::build`; attach all hooks (§13.5); flush gossip messages buffered since step 3 into the doc; broadcast own presence; initial digest. ⇨ `Attached`. + +### 13.4 Leave, close, and quit (the prompt contracts — implement exactly) + +**(a) "Leave session" button** (share dialog / status pill context): light confirm — *"Leave this session? Your copy of the document stays open."* `[Leave] [Cancel]`. On Leave: run the detach sequence (§13.5). The tab **remains open** as an ordinary document; no save prompt (the document is still there; saving stays a later decision). Transient notification: "Left session — this copy is now local." + +**(b) Closing an attached tab** (`close_document_panel`, `documents.rs:670`): two-step, **transactional** — nothing detaches until every answer is in; Cancel anywhere = still attached, tab still open. +1. Prompt 1: *"Leave the collaboration session? This tab is live with N other people."* `[Leave] [Cancel]` (N from presence; copy adjusts for N = 0). +2. Prompt 2 (only if leaving): the standard save flow, branched on **`document_path().is_some()`** — NOT on any starter/joiner notion (a joiner who Save-As-ed is identical to anyone else): pathless → `[Save As…] [Don't Save] [Cancel]`; path-backed and dirty → existing `[Save] [Don't Save] [Cancel]`; path-backed and clean → skip. +3. Execute in order: await save (failure → abort, still attached) → detach (§13.5) → `remove_document_panel`. "Don't Save" on a pathless tab also deletes its session recovery file (existing `discard_recovery_file` path). + +**(c) App quit / window close** (`request_close_window`, `documents.rs:734`): extend the existing dirty-panel sweep — for each attached panel run the (b) sequence; any Cancel aborts the quit. Wording for the combined first prompt when multiple sessions: *"You're in N collaboration sessions. Leave all and quit?"* then per-tab save prompts as needed. After all resolve: `CollabManager::shutdown` (presence-removals best-effort) then close. + +**(d) Crash / kill:** no prompts, obviously. Peers see the presence entry age out (≤30 s). The user's protection is the recovery file (§15.6); membership is not resumable in v1 (§13.6). + +### 13.5 Attach / detach hook lists (single source of truth — both must mirror) + +Attach (after doc+binding exist): +1. `editor.update`: clear undo/redo stacks; `set_collab_undo_redirect(Some(…))`; set `collab_capture = true`. +2. `cx.observe(editor)` → `flush_local_edits` (drain → LocalApplier → commit; loro local-update subscription → `Publish`). Keep the `Subscription` in the session. +3. `cx.subscribe(editor)` (`SelectionChanged`) → presence update (50 ms debounce). +4. loro subscriptions: diff events (→ RemoteApplier), local-update (→ publish), undo manager creation. +5. Timers: anti-entropy digest (10 s jittered), presence keepalive (10 s), self-check (30 s idle-gated). +6. Register in `CollabManager` registry (`SessionId ↔ panel Uuid ↔ Entity`). + +Detach (any reason — `UserLeft`, `JoinFailed` post-tab, `Fatal`): +1. Best-effort presence self-removal broadcast; `LeaveSession` to net (unsubscribes topic, drops swarm task, clears outbox). +2. Flush any queued-but-unapplied remote patches (deterministic final state — "the state you witnessed"). +3. Editor teardown: `set_collab_undo_redirect(None)`, `collab_capture = false` + drain-discard the queue, `set_external_carets(vec![], cx)`, undo stacks stay empty (fresh local history from here). +4. Drop loro subscriptions, UndoManager, timers, observe/subscribe Subscriptions, the LoroDoc, the binding. +5. Deregister from `CollabManager`. Pathless tab keeps its session recovery path until closed or Save-As (then switches to the normal derived recovery path). + +### 13.6 Deferred (v1.1, designed-for): session resume across restart + +Persist `{session_id, last peer addrs, loro snapshot}` per attached tab on quit; offer "Rejoin previous session" on next launch (re-verifies §12.2 check 2; rejoin = the §13.2 reconnect path with a pre-warmed doc). Not in v1 — process exit is an implicit leave via the §13.4(c) prompts. Do not build now; do not preclude (keep `SessionId` and snapshot export accessible from the session entity). --- -## 15. Editor changes (gpui-flowtext) — complete enumerated list +## 14. Save semantics -All hidden behind no feature flag (inert without a session). Keep each change minimal; **no loro/iroh types may appear in this crate.** +* **No canonical file exists.** Every peer's save writes *their current view* to *their* file. The app encodes no relationship between any file and the session. +* Path-backed tabs (whoever started from a file, or anyone after Save As): `save`/`save_as`/autosave all work unchanged while attached — autosave naturally tracks the live session state into the user's file. CRDT history is **never** persisted into `.db8` (a saved file is a plain snapshot; format untouched). +* **Save As while attached does not detach** (§15.5 wires `DocumentPanel::set_path` to leave the session untouched). After Save As, the tab is path-backed: autosave engages, the close-prompt branch changes, recovery path switches to the derived one. +* Pathless attached tabs: autosave/Save skip → Save As (existing untitled behavior); recovery via §15.6. -1. `editor/mod.rs`: add fields `pending_collab_edits: Vec`, `suppress_collab_capture: u32`, `collab_undo_redirect: Option>`; add `UndoRedirect` enum. (~20 LOC) -2. `edit_pipeline.rs`: per §6.2 — push exact deltas into `pending_collab_edits` (fast path pushes the single grapheme; `mark_document_changed_with_reconcile` gains an `ops: Option<&[CanonicalOperation]>` parameter or a small wrapper so it only pushes when a new record was created and `suppress_collab_capture == 0`). (~40 LOC) -3. `lifecycle.rs`: `take_pending_collab_edits()`, `set_collab_undo_redirect`, `clear_collab_session_state` (clears queue+redirect+external carets). Clear the queue in `dispose_for_close`/`release_transient_memory`. (~40 LOC) -4. `commands.rs` `undo()`/`redo()`: redirect check first. (~10 LOC) -5. **New** `editor/collab_apply.rs`: `apply_collab_patches` + selection remap helpers per §7 (≤600 LOC). Reuses `edit_ops` primitives; add `edit_ops/collab.rs` only if a `replace_paragraph_content` helper doesn't fit existing files. -6. `api.rs`/events: ensure `impl EventEmitter for RichTextEditor` exists so `cx.subscribe` works (it may already; verify). `EditorEvent` already carries `SelectionChanged`. -7. `DocPatch` lives in `flowstate-collab`; to keep the dependency direction (editor must not depend on collab), `apply_collab_patches` takes a **gpui-flowtext-owned** patch type: define `CollabPatch` (mirror of §7's `DocPatch`) in `collaboration.rs` (it already holds the collab vocabulary) and have `flowstate-collab` emit that type. (`flowstate-collab` already depends on gpui-flowtext — clean.) -8. Image placeholder: in the image layout/measure path (`rich_text/editor/media.rs` + `layout/block_layout.rs`), treat an `AssetRecord` with empty `bytes` as "loading": fixed 240×160 placeholder box (theme muted bg + spinner glyph). Triggered while an asset transfer is in flight (§16). (~40 LOC) +--- + +## 15. Editor changes (gpui-flowtext) — complete enumerated list -Everything else (roles gating, external carets, replace-document) already exists. +No feature flag; all inert without a session. **No loro/iroh types in this crate.** Each item names the exact file and insertion point. + +1. **`editor/mod.rs`** — new fields on `RichTextEditor` (near `last_collaboration_edit`, line ~870): + ```rust + pending_collab_edits: Vec, + collab_capture: bool, // false ⇒ never push (no leak for solo editors) + suppress_collab_capture: u32, // mirrors suppress_mutation_notify pattern + collab_undo_redirect: Option>, // Rc: entities are single-threaded + ``` + plus `pub enum UndoRedirect { Undo, Redo }`. Initialize in `new_with_path` (`lifecycle.rs:16`); reset in `release_transient_memory` and `dispose_for_close`. +2. **`edit_pipeline.rs`** — the §6.2 delta-exact capture: + * In `insert_single_grapheme_fast_path`: after the undo-record logic (merged or not), if `collab_capture && suppress_collab_capture == 0`, push `CollaborationEdit { operations: vec![InsertText { paragraph: paragraph_id, byte: caret.byte, text: text.to_string(), styles }] }` — **the single grapheme only**, never the merged record. + * Change `mark_document_changed_with_reconcile(generation, reconcile, cx)` to take the ops explicitly: add a thin wrapper `mark_document_changed_with_ops(generation, reconcile, ops: Option<&[CanonicalOperation]>, cx)`; call sites that just pushed a *new* `EditRecord` pass `Some(&record.canonical_operations)`; the fast path (which already pushed its own delta) and history-restore paths pass `None`. The wrapper pushes into `pending_collab_edits` under the same gate. Keep `last_collaboration_edit` assignment as-is for compatibility. +3. **`lifecycle.rs`** — accessors: + ```rust + pub fn take_pending_collab_edits(&mut self) -> Vec; // mem::take + pub fn set_collab_capture(&mut self, on: bool); // false also clears the queue + pub fn set_collab_undo_redirect(&mut self, hook: Option>); + pub fn set_recovery_path(&mut self, path: Option); // §15.6 + ``` +4. **`commands.rs:191/205`** — first line of `undo()`/`redo()`: `if let Some(hook) = self.collab_undo_redirect.clone() { hook(UndoRedirect::Undo); return; }` (resp. `Redo`). +5. **`collab_apply.rs` (new)** — `apply_collab_patches` + `collab_apply_deferred` per §7, building on the `edit_ops` primitives; selection-remap helpers; ≤600 LOC. Add `edit_ops/collab.rs` with `replace_paragraph_content` if no existing helper fits (extend the `include!` list in `edit_ops/mod.rs`). +6. **Recovery for pathless session tabs**: `set_recovery_path` (item 3) + the session sets it to `std::env::temp_dir().join("flowstate-collab-recovery").join(format!("{}-{}.db8", hex(session_id_prefix8), sanitized_title))`. The existing `schedule_recovery_write` / `discard_recovery_file` machinery (`recovery.rs`) then works unchanged — verify it only requires `recovery_path.is_some()`, not `document_path`. Crash → next launch recovers a plain local document via the existing recovery-discovery flow. +7. **Events**: ensure `impl EventEmitter for RichTextEditor` exists (add if only `EditorEventSink` is wired today) and that `SelectionChanged` actually emits on selection mutation paths (check `selection.rs` / movement handlers; add the emit where missing). +8. **`collaboration.rs`** — add `CollabPatch`, `CollabTextDelta` (§7) next to `CanonicalOperation`. +9. **Image placeholder**: in the image measure/paint path (`editor/media.rs` + `rich_text/layout/block_layout.rs`), an `AssetRecord` with empty `bytes` renders a fixed 240×160 "loading" box (muted background + spinner glyph) instead of a broken image. Used while §16 transfers are in flight. +10. **Test** `rich_text/tests/collab_capture.rs` per §6.2. --- ## 16. Assets (images) over the wire -Asset bytes never enter the CRDT (snapshots would carry every image forever). Flow: +Asset bytes never enter the CRDT (snapshots would carry every image forever). Symmetric, pull-based: -* `BlockPayload::Image` carries `asset_id`, `mime`, `content_hash`, `byte_len` (metadata only). -* Host is the **asset authority**: `Welcome.asset_manifest` lists all assets; joiner requests the ones referenced by visible blocks first (then the rest lazily) via `Frame::AssetRequest` → host opens a fresh bi-stream: postcard `AssetHeader { id, mime, len, hash }` then raw 256 KiB chunks → joiner inserts `AssetRecord` into `document.assets` (via a small `collab_apply` patch `AssetArrived { id, record }`) → repaint replaces placeholder. -* A joiner pasting an image: image lands in its local `Document.assets` (existing paste path), block syncs via CRDT with metadata; joiner sends `AssetOffer`; host `AssetRequest`s it back over a stream, stores it, and other peers fetch from host on demand. (Star topology keeps transfers host-centric — no peer-to-peer asset mesh.) -* Dedup by `content_hash` is an optimization, not v1. +* `BlockPayload::Image` carries metadata only (`asset_id`, `mime`, `content_hash`, `byte_len`). +* When a `ReplaceObjectBlock`/`InsertBlocks` patch references an asset missing from `document.assets`: insert a placeholder `AssetRecord { bytes: Arc::new(vec![]), … }` (renders as §15.9 placeholder) and enqueue a fetch: `PullAsset` via `pull_with_fallback` — candidates: the peer whose gossip delivered the block change, then neighbors, then presence-known peers. **Any peer that has the bytes serves them** (`DirectRequest::Asset` answered from the local `document.assets` of the attached tab). On arrival: `CollabPatch::AssetArrived` swaps the record in; repaint. +* A peer pasting an image needs no special path: the paste populates its local `AssetStore` (existing path), the block syncs as metadata, and everyone else pulls from *it* — or from anyone who already pulled it (fallback chain makes distribution self-spreading). +* Joins: after snapshot projection, scan blocks for missing assets and enqueue fetches, viewport-visible blocks first (the editor knows visible range; expose it or just fetch in block order — acceptable v1). +* Failed fetch: placeholder persists; retry on next reference or manual reload; never blocks text sync. Dedup by `content_hash` is v1.1. --- ## 17. UI specification (gpui-component widgets) -1. **Commands** (`commands.rs`): add `CommandId::ShareDocument` ("Share / Collaborate…", APP context, no default key) and `CommandId::JoinSession` ("Join Collaboration Session…", APP). Add to `COMMAND_SPECS`; wire dispatch in the workspace command handler (`workspace/workspace/keybindings.rs` routing). -2. **Top bar** (`top_bar.rs`): File menu gains "Share Document…" (enabled when active panel is a rich-text doc) and "Join Session…". Plus a dedicated share `icon_button` on the `TitleBar` right cluster (next to settings) showing a filled/colored state while a session is live. -3. **Share dialog** (`collab/share_dialog.rs`, gpui-component `Dialog` + `Input` + `Button` + `clipboard` copy button + `avatar::AvatarGroup` + `switch` for Editor/Viewer default role): - * Idle: role switch + "Start session". - * Live (host): read-only `Input` with ticket string + Copy button; participant list (color dot, name, role); "End session" (danger variant, confirm). - * Join tab: `Input` (paste ticket) + "Join" with inline validation (parse before dialing; show doc title from ticket); progress states Connecting → Receiving document (n KiB) → done (dialog closes, tab opens). -4. **Status bar** (`render_status.rs`): when the *active* panel has a session: pill `● Hosting · 3` / `● Connected` / `◌ Reconnecting…` / `● Session ended`, colored by state; click opens the share dialog. Follow the zoom/status element patterns already in that file. -5. **Tab badge** (`document_panel.rs` render): small colored dot + peer count on collaborating tabs. -6. **Notifications** (gpui-component `notification.rs`): peer joined/left, resync occurred, read-only edit attempt ("You're a viewer in this session"). -7. **Orphan banner**: when a joiner session ends, a slim banner above the editor: "Session ended — this is your local copy. Save As to keep it." with a Save As button. Implement as a `DocumentPanel` child element gated on session state. -8. Errors: join failures via dialog inline error; mid-session fatal errors via `window.prompt` only if the user must act, otherwise notification + status pill. +1. **Commands** (`commands.rs`): add `CommandId::ShareDocument` ("Share / Collaborate…", APP context) and `CommandId::JoinSession` ("Join Collaboration Session…", APP); entries in `COMMAND_SPECS` (no default keys); route in the workspace command handler (`workspace/workspace/keybindings.rs`). +2. **Top bar** (`top_bar.rs`): File menu gains "Share Document…" (enabled when the active panel is a rich-text doc) and "Join Session…". Plus a share `icon_button` in the `TitleBar` right cluster (pattern: `settings_top_bar_button`), tinted while the active tab is attached. +3. **Share dialog** (`collab/share_dialog.rs`; gpui-component `Dialog`, `Input`, `Button`, clipboard copy button, `avatar` for the roster): + * Active tab not attached: explainer + `[Start session]` → `Creating` → flips to the attached view. + * Attached: read-only `Input` with a **freshly minted** ticket (re-mint on open and on copy — embeds current addr) + Copy; roster (color dot, name, "you" marker) from presence; connectivity line ("Only you — share the invite" / "Offline — reconnecting…"); `[Leave session]` (danger variant → §13.4(a) confirm). + * Join tab: paste `Input` with inline parse validation (shows doc title from the ticket pre-connect), `[Join]`, progress states from `JoinStage`, inline error on failure. + * **No role switch anywhere. No "End for everyone" anywhere.** (Decided: everyone writes; sessions end by attrition.) +4. **Status bar** (`render_status.rs`): the §13.1 pill for the active tab; click opens the share dialog. Follow the existing zoom/status element patterns. +5. **Tab badge** (`document_panel.rs` render): small colored dot + peer count on attached tabs; grey variant when `Offline`. +6. **Notifications** (gpui-component `notification.rs`): peer joined ("Maya joined"), peer left, "Left session — this copy is now local", view-rebuilt (debug), incompatible-version (once per peer). +7. **Prompts**: §13.4 copy verbatim; use the existing `window.prompt` flow (`documents.rs:700` pattern) so styling matches. --- -## 18. Failure-mode catalog (each requires explicit handling + a test or QA item) +## 18. Failure-mode catalog (each row: explicit handling + a test or QA-script item) | # | Scenario | Required behavior | |---|---|---| -| F1 | Garbage/truncated ticket pasted | Parse error inline in dialog; never dials | -| F2 | Host unreachable (offline, NAT+relay down) | Connect timeout 15 s → JoinFailed with actionable text | -| F3 | Token wrong / session unknown (stale ticket) | `Rejected{BadToken/UnknownSession}` → "Session no longer available" | -| F4 | Version skew | `Rejected{Version}` both directions, clear message | -| F5 | Host ends session / closes tab / quits app | `Bye{SessionEnded/HostShutdown}` (best-effort on quit) → joiners orphan with banner; net thread closes conns | -| F6 | Joiner vanishes (crash, network) | host: `connection.closed()` → roster update + notification; presence entry times out | -| F7 | Transient disconnect | joiner auto-reconnect w/ `last_vv` delta catch-up; edits made while offline are queued in loro and sync on reconnect (CRDT gives offline-tolerance for free) | -| F8 | Reconnect after host restarted session (new SessionId) | `UnknownSession` → orphan, do not silently join a different session | -| F9 | Self-join (pasting own ticket) | detect `ticket.addr.endpoint_id == endpoint.id()` → friendly error | -| F10 | Session full (>8) | `Rejected{Full}` | -| F11 | Viewer client sends DocUpdate (bug/malice) | host drops + `Rejected{ReadOnly}`; never relayed | -| F12 | Integrity hash mismatch | auto-resync (§7.2), notification, `eprintln!` diagnostics incl. both hashes + vv | -| F13 | Giant document snapshot | frames ≤2 MiB → snapshot sent as one frame up to that, else chunked `ResyncSnapshot`-style continuation (add `DocBootstrap::SnapshotChunk{n,of}` if needed); progress UI | -| F14 | Remote patch arrives mid-drag/IME | deferral queue (§7 item 6) | -| F15 | Asset referenced but transfer fails | placeholder persists + retry on next request; never blocks text sync | -| F16 | Two sessions on one doc / share twice | second Share opens existing session dialog (sessions keyed by panel Uuid) | -| F17 | Joiner opens Share on a joined doc | allowed to *view* roster, cannot start nested session (button disabled, tooltip) | -| F18 | App quit while hosting | window-close confirm includes session warning (extends existing dirty-check prompt flow in `documents.rs:734`) | +| F1 | Garbage/truncated ticket | Inline parse error in join dialog; never dials | +| F2 | Inviter unreachable at join | 15 s subscribe timeout → JoinFailed with actionable copy | +| F3 | Inviter reachable but snapshot pull fails | fallback to neighbors (§10.3); all candidates fail → JoinFailed | +| F4 | Ticket from an old, dead session | subscribe succeeds but no neighbors ever → 15 s timeout (same as F2; copy mentions "session may be over") | +| F5 | Protocol version skew | join: fast clear error; in-session gossip: ignore + one-time notification (§11) | +| F6 | Lineage violation (wrong doc on right topic) | §12.2: join-time meta check fails ⇒ JoinFailed; digest mismatch ⇒ ignore peer; never merge | +| F7 | Peer vanishes (crash/kill) | presence entry ages out ≤30 s → roster update + notification; overlay self-heals | +| F8 | Transient network loss | `Attached{Offline}` (§13.2): edits accumulate, auto-reconnect, digest-pull catch-up; pill visible; zero data loss | +| F9 | Long offline (hours) | same path; reconnect merge interleaves per CRDT semantics; resolution = own-ops undo + manual edit (§8) | +| F10 | All peers leave while I'm attached | `Attached{Online, peers_present: 0}` → "Only you"; wait (drop-in works) or Leave — user's call; no auto-detach | +| F11 | Gossip receiver `Lagged` | treat as gap: immediate digest + pull (§10.4) | +| F12 | Oversized-update blob pull fails / outbox expired | harmless: next anti-entropy round delivers via `Updates` (§10.5) | +| F13 | Projection drift (applier bug) | local self-check → self-heal rebuild + structured log (§7.2); selection clamped | +| F14 | Remote patch during drag/IME | deferral queue (§7.3) | +| F15 | Asset fetch fails everywhere | placeholder persists; text sync unaffected; retry on next reference (§16) | +| F16 | Self-join (own ticket) | detected at parse (`inviter.endpoint_id == own`) → friendly error | +| F17 | Share invoked on an already-attached tab | dialog shows the attached view (re-mint invite); no nested sessions (registry keyed by panel Uuid) | +| F18 | Close tab / quit while attached | §13.4(b)/(c) transactional prompts; Cancel anywhere fully aborts | +| F19 | Crash while attached, pathless tab | session recovery file (§15.6) → recovered as plain local doc on next launch | +| F20 | Two peers edit the same table cell concurrently | LWW on the block payload — one peer's table edit wins (§5.3); QA item to observe, documented limitation | +| F21 | Clock skew between peers | presence timestamps are loro-internal lamport-ish (EphemeralStore) — verify it doesn't use wall-clock comparisons across peers (§23-V7); doc sync needs no wall clock at all | --- ## 19. Performance budgets -* Keystroke→local paint: **zero added frame cost** — LocalApplier work happens in the observe callback after paint scheduling; loro insert ≈ 1–5 µs. Assert no network I/O on the main thread (channel sends only). -* Remote keystroke→repaint: ≤ 1 frame after import; per-paragraph regeneration only for touched paragraphs (the existing `layout_invalidation_hint` machinery keeps relayout scoped). -* Join bootstrap on a 5 MB doc: snapshot export+import off-main-thread; only `replace_document_from_collaboration` on-thread (same cost as opening the file — already acceptable). -* Memory: LoroDoc history grows with session length; fine for sessions (hours), irrelevant after close (dropped). No persistence. -* Add a micro-benchmark (optional, `rich_text/benchmarks/` style) only if M6 profiling shows surprises; do not pre-optimize. +* **Keystroke → local paint: zero added frame cost.** LocalApplier runs in the observe callback after paint scheduling; loro insert ≈ 1–5 µs; publish is a channel send. Assert no network I/O and no `export` on the keystroke path beyond loro's incremental local-update bytes. +* Remote keystroke → repaint: ≤ 1 frame after import; per-paragraph regeneration only for touched paragraphs (existing `layout_invalidation_hint` scoping). +* Join on a 5 MB doc: export/import off-main-thread; the only main-thread cost is `replace_document_from_collaboration` (≈ cost of opening the file today). +* Anti-entropy: digests are O(peers in vv) bytes every 10 s to ≤5 neighbors — noise. Self-check projection is O(doc) — that's why it's idle-gated at 30 s. +* Memory: LoroDoc history grows for the session's lifetime and is dropped at detach; never persisted. The blob outbox is capped (64 entries / 16 MiB). --- ## 20. Decisions already made — DO NOT relitigate -1. **loro schema = per-paragraph `LoroText` in a `LoroMovableList`** (§5.2), not a single body text. Accepted split-race anomaly documented. -2. **Tables/equations/images = atomic LWW payloads** matching the editor's own `ReplaceBlock` granularity (§5.3). -3. **Star topology, host authority, host relays raw update bytes.** No iroh-gossip, no mesh, no host migration in v1. -4. **CRDT lives on the main thread inside the session entity**; tokio thread is transport-only. -5. **Wire = loro update bytes**, never `CanonicalOperation`s (the existing `WireCanonicalOperation` codec stays unused). -6. **Theme/zoom/invisibility are local-only**; never synced. -7. **Undo in-session = loro UndoManager** with editor-level redirect; native undo stack cleared at session boundaries. -8. **Asset bytes out-of-band** over dedicated streams; host is asset authority. -9. **Joiners have no `document_path`**; host file is the only source of truth; CRDT history never touches `.db8`. -10. **Flow (`.fl0`) documents excluded from v1.** -11. New core logic in **`flowstate-collab`** crate (GPUI-free); GPUI glue in `crates/flowstate/src/collab/`; editor crate stays loro/iroh-free. - -Deferred (tracked, not designed here): per-peer role changes + kick, selection-range rendering for remote peers, cell-level table CRDT, host migration, ticket QR codes, session persistence across host restarts, >8 peers, e2e-encryption hardening beyond QUIC+token. +1. **Gossip mesh of symmetric peers** (iroh-gossip), not star, not hand-rolled full mesh. Chosen with full knowledge of the trade (anti-entropy + direct side-protocol are the price; host-absence liveness and no-special-peer semantics are the prize). +2. **No roles. Everyone writes.** `CollaborationRole` stays in the editor unused; no Viewer UI, no enforcement, no per-peer permissions. +3. **No starter.** Creating a session = minting secret + seeding content; zero ongoing distinction. All UX branches on `document_path.is_some()`, never on who started. +4. **Sessions end by attrition.** No end-session action, no ended flag, no tombstone. "Alone" is a first-class, indefinitely-valid state. +5. **Membership ⊥ connectivity.** Involuntary disconnect keeps you attached (edits accumulate; auto-reconnect; anti-entropy resync). Only explicit Leave (or its close/quit prompt equivalent) detaches; the doc then freezes at the witnessed state as a plain local document. +6. **Join always creates a fresh tab/doc from the session snapshot.** No merge-local-file-into-session flow exists. **Save As while attached does not detach.** +7. **Process exit = leave (v1)**, guarded by the §13.4 transactional prompt cascade; session resume across restarts is v1.1 (§13.6), designed-for but not built. +8. **CRDT layout:** per-paragraph `LoroText` in a `LoroMovableList` (§5.2); tables/equations/images as atomic LWW payloads matching the editor's own `ReplaceBlock` granularity (§5.3); 4 mark keys with `ExpandType::None`. +9. **Wire = loro update bytes + EphemeralStore bytes.** No operation parsing, no transforms, no custom merge logic anywhere. `WireCanonicalOperation` stays unused. +10. **SessionId = TopicId = the bearer secret**; lineage guarded by §12.2's three checks; topics never reused. +11. **Anyone-can-invite, re-minted tickets** carrying the minter's current address. +12. **Pull-only anti-entropy** (10 s jittered digests via `broadcast_neighbors`); oversized updates via blob-availability notice + direct pull, **no chunk-reassembly protocol**. +13. **Integrity is local-first**: Document-vs-own-CRDT self-check with local rebuild; cross-peer state comparison only as an env-gated QA tool. No arbitration, because projections are derived state. +14. **Undo while attached = loro UndoManager** (own ops only); native stacks cleared at attach/detach; merges are not undoable units — own-ops undo + manual editing is the documented resolution story. +15. **Assets out-of-band, any-peer serving, pull-based** (§16); bytes never in the CRDT. +16. **Theme/zoom/invisibility are local-only.** Flow (`.fl0`) docs out of scope. +17. **Threading:** CRDT on the GPUI main thread inside the session entity; one lazy tokio thread owns endpoint+gossip+direct server; `async-channel` is the only cross-runtime primitive. +18. New core logic in **`flowstate-collab`** (GPUI-free); GPUI glue in `crates/flowstate/src/collab/`; the editor crate stays loro/iroh-free; dependency arrow `flowstate-collab → gpui-flowtext` only. + +Deferred (tracked, not designed here beyond noted sketches): session resume (§13.6), selection-range rendering for remote peers, asset `content_hash` dedup, cell-level table CRDT, kick/ban (remedy: rotate session), ticket QR codes, >16 peers, version-history restore. --- ## 21. Testing strategy -### 21.1 Unit / golden (in `flowstate-collab`) -* `translation.rs`: for every `CanonicalOperation` variant: apply to a `Document` fixture via the editor's real edit ops (construct via `document_from_input`), run LocalApplier, project loro back, assert equality with the mutated `Document`. Include multibyte text (`"é🌍"`, soft line breaks) in every text fixture. +### 21.1 Unit / golden (in `flowstate-collab`, all GPUI-free) +* `tests/translation.rs`: for **every** `CanonicalOperation` variant: build a `Document` fixture via `document_from_input`, mutate it through the editor's real `edit_ops` functions, run LocalApplier, project the loro doc back, assert byte-equality with the mutated `Document`. Every text fixture includes `"aé🌍\u{2028}x"`. * Projection round-trip property test (§7.1). -* Offset conversion helpers: exhaustive over a multibyte fixture. -* §6.2 double-apply regression test (in gpui-flowtext). +* `schema.rs` offset helpers: exhaustive positions over the multibyte fixture. +* `tests/anti_entropy.rs`: digest/gap/pull logic against a fake transport (in-memory channel pairs): peer drops offline N ops, digests resume, exactly one pull issued, convergence; `Lagged` path; in-flight pull dedup. +* gpui-flowtext: §6.2 capture regression test; `collab_apply` patch tests (apply around a live selection, IME-deferral predicate, selection remap on multibyte deltas). -### 21.2 Convergence fuzz (THE gate for M2 — do not ship without it green) -`tests/convergence.rs` (proptest): N ∈ {2,3} simulated peers, each owning a `(Document, LoroDoc, DocBinding)` triple. Random op program per peer (insert/delete/split/join/style/block-insert/move/replace, weighted toward text); ops applied through LocalApplier; update bytes exchanged through a virtual network with random delay/reorder/duplication (loro tolerates all three); RemoteApplier applies inbound diffs to each peer's `Document`. After quiescence: **all projections byte-identical** (text, runs, styles, structure) and equal to a fresh projection of any peer's LoroDoc. Shrinkable failures print the op program. +### 21.2 Convergence fuzz — the M2 gate; nothing integrates until this is green +`tests/convergence.rs` (proptest): N ∈ {2, 3} simulated peers, each a `(Document, LoroDoc, DocBinding)` triple. Random per-peer op programs (weighted: 70% text insert/delete, 10% split/join, 10% styles, 10% block ops) applied through the editor's real edit ops + LocalApplier; update bytes exchanged through a virtual network applying random delay, reorder, duplication, and per-peer offline windows (buffer then flush — this directly simulates §13.2 drop-out/drop-in); RemoteApplier applies inbound diffs to each peer's `Document`. After quiescence + full exchange: **all projections byte-identical** and equal to a fresh projection of any peer's LoroDoc. Shrunk failures print the op program. -### 21.3 Loopback integration (`tests/loopback.rs`) -Real iroh endpoints, localhost direct addrs, `RelayMode::Disabled`, no discovery: host + 2 joiners; assert handshake, welcome bootstrap, typed-edit relay, viewer rejection, reconnect-with-delta (kill joiner conn, redial with vv), Bye propagation. Mark `#[ignore]` if CI sandboxing blocks UDP; runnable locally. +### 21.3 Swarm loopback (`tests/swarm_loopback.rs`) +Real iroh endpoints + real gossip on localhost: `RelayMode::Disabled`, no discovery, addresses registered manually. Three peers: A creates (populate + subscribe), B and C join via A's ticket (full §13.3 sequence). Assert: snapshot fetch, live update propagation A→B→C, presence roster on all three, **A leaves → B and C continue editing and converge** (the no-starter property, asserted in code), B kills its gossip subscription + keeps editing + resubscribes → digest pull converges (§13.2 simulation), blob path (force a >2 KiB update). Mark `#[ignore]` if CI blocks UDP; must run locally before M4 merge. ### 21.4 Manual QA script (commit as `helpers/docs/collab_qa.md`) -Two app instances (`cargo run` ×2, different cwd): the 18 failure modes of §18 plus: simultaneous typing same paragraph, Enter-split race, table edit both sides (LWW visible), image paste joiner-side, undo tug-of-war, 30-minute soak with autosave on. +Two-three app instances (`cargo run` ×N, different cwd). Cover all 21 failure rows of §18 plus: simultaneous same-paragraph typing; Enter-split race (observe the §5.2 anomaly, confirm convergence); same-table-cell race (F20, observe LWW); image paste on each peer; undo tug-of-war; leave-and-rejoin via fresh ticket; offline-laptop-lid 5 min then resync; close-tab prompt cancellation at both steps; 30-minute soak with autosave on a path-backed peer. --- ## 22. Milestones & work breakdown (for the orchestrating agent) -Dependency graph: `M1 ∥ M2` → `M3` → `M4` → `M5` → `M6`. Each milestone ends with `cargo clippy` clean (workspace lints) + its tests green. +Dependency graph: **M0 → (M1 ∥ M2) → M3 → M4 → M5 → M6.** Every milestone ends with `cargo clippy` clean and its tests green. Where a milestone says "verify §23-Vn first," the subagent's first action is reading those docs.rs pages and correcting names in its working copy of this plan. -**M0 — Scaffolding (small, do first, single agent)** -`cargo new crates/flowstate-collab --lib`; wire workspace deps (§4); empty modules compiling; `ALPN`/`PROTOCOL_VERSION` constants; CI keeps passing. +**M0 — Scaffolding (single agent, small).** `cargo new crates/flowstate-collab --lib`; §4 dependency commands; empty modules compiling with `lib.rs` docs; `ids.rs` (SessionId/BlobId/palette); `PROTOCOL_VERSION`/`DIRECT_ALPN` consts; CI stays green. -**M1 — Networking core (1 agent)** — `net/*`, `protocol.rs`, `ticket.rs`, framing. Deliverable: loopback test (§21.3) passing with an echo-level session (no loro yet — frames round-trip, auth verified, relay fan-out works, reconnect works). Verify §23-V1/V2 against docs.rs first. +**M1 — Networking (1 agent).** `net/{runtime,swarm,direct,anti_entropy,blobs}.rs`, `proto_gossip.rs`, `proto_direct.rs`, `ticket.rs`. Verify §23-V1/V2 first. Deliverables: `tests/anti_entropy.rs` green; a temporary `examples/echo_swarm.rs` proving 3-peer subscribe/broadcast/direct-pull on localhost (folded into §21.3 later). No loro yet — payloads are opaque bytes at this layer by design. -**M2 — Document sync core (1–2 agents; the hard one)** — `schema.rs`, `binding.rs`, `projection.rs`, `local_apply.rs`, `remote_apply.rs`, `integrity.rs`, plus the gpui-flowtext `pending_collab_edits` change (§6.2) it needs for fixtures. Deliverables: §21.1 goldens + §21.2 convergence fuzz green. Start with text-only ops, then structure ops, then blocks. Verify §23-V3/V4 first. +**M2 — Document sync core (1–2 agents; the hard one).** `schema.rs`, `binding.rs`, `projection.rs`, `local_apply.rs`, `remote_apply.rs`, `self_check.rs`, plus the gpui-flowtext capture change (§15.1–3, §15.8, §15.10) it needs for fixtures. Verify §23-V3/V4/V6 first. Order of attack: text ops → structure ops (split/join/span) → block ops → fuzz hardening. Deliverables: §21.1 goldens + **§21.2 convergence fuzz green**. -**M3 — Editor integration (1 agent, after M2)** — §15 items 1–8 (`collab_apply.rs`, undo redirect, deferral, selection remap, placeholder), §8 UndoManager wiring, EventEmitter check. Deliverable: headless GPUI test (`gpui::TestAppContext` if available in gpui 0.2; otherwise unit-test patch application directly on `RichTextEditor` via `cx.new`) applying remote patches around an active selection. +**M3 — Editor integration (1 agent, after M2).** Remaining §15 items (collab_apply.rs, undo redirect, deferral, selection remap, recovery setter, EventEmitter, placeholder), §8 UndoManager wiring. Verify §23-V5/V8 first. Deliverable: editor-level patch/undo/capture tests green (`cx.new`-based entity tests; check how existing `workspace/workspace/tests.rs` constructs entities and follow that pattern). -**M4 — Session orchestration (1 agent)** — `collab/mod.rs` (CollabManager global + net-event pump), `collab/session.rs` (state machines §13, flush/import paths, presence §9, assets §16, resync §7.2), workspace hooks (close-tab/close-window confirms, joiner tab creation). Deliverable: two real app instances collaborate end-to-end (manual), F5/F6/F7 verified. +**M4 — Session orchestration (1 agent).** `collab/mod.rs` (CollabManager global, NetEvent pump, registry), `collab/session.rs` (§13 state machine *exactly*, §13.5 attach/detach lists, presence, assets, self-heal, buffered-import join), workspace hooks (§13.4 prompts into `documents.rs` close/quit paths, joiner tab creation). Deliverables: §21.3 swarm loopback green end-to-end; manual two-instance session works including leave/offline/rejoin. -**M5 — UI (1 agent, parallel with late M4)** — §17 complete: commands, menus, share dialog, status pill, tab badge, notifications, orphan banner. +**M5 — UI (1 agent, can overlap late M4).** §17 complete: commands, menus, share dialog, status pill, tab badge, notifications, prompt copy. -**M6 — Hardening (1 agent)** — full §18 sweep, §21.4 QA script run + fixes, perf sanity (§19), `helpers/docs/collaboration.md` (user-facing how-to + architecture notes for maintainers), final clippy/test pass. +**M6 — Hardening (1 agent).** Full §18 sweep with the §21.4 QA script; perf sanity against §19; `helpers/docs/collaboration.md` (user how-to + maintainer architecture notes); final clippy/test pass across the workspace. -Acceptance for the whole feature = QA script passes on Linux (primary) with two instances, including one relay-path run (different networks or forced `RelayMode::Relay`). +Feature acceptance = §21.4 passes on Linux with three instances, including one run forced through relays (`RelayMode` forced, or two machines on different networks), exercising drop-out/drop-in and the A-leaves-B-C-continue property. --- -## 23. API verification appendix — check these exact names before coding +## 23. API verification appendix — re-verify these exact names before coding -The architecture does not depend on any of these spellings; they are listed so implementers verify rather than guess: +The architecture depends on none of these spellings; implementers verify rather than guess. Everything *not* listed here was verified on 2026-06-12. -* **V1 (iroh, M1):** discovery/preset wiring on `endpoint::Builder` in 0.98 — the docs show `Builder::new(preset)` / `preset()` / `address_lookup()` and `presets::N0`; relays default to n0 via `RelayMode::Default`, but **address lookup is off by default**. Confirm the preset that enables n0 DNS lookup + pkarr publish, and how to read `endpoint.addr()` for tickets (may need to await first addr / `online()` watcher). docs.rs/iroh/0.98.2. -* **V2 (iroh-tickets, M1):** exact `Ticket` trait path + required methods (kind/prefix, `to_bytes`/`from_bytes`), and `EndpointTicket` as reference implementation. docs.rs/iroh-tickets. -* **V3 (loro, M2):** UTF-8 variants available on `LoroText`: confirmed `mark_utf8`, `unmark_utf8?`, `len_utf8`, `convert_pos(index, PosType, PosType)`; check whether `insert_utf8`/`delete_utf8`/`splice_utf8` exist (if not, convert via `convert_pos`). Also `UpdateOptions` fields for `text.update`. docs.rs/loro/1.13. -* **V4 (loro, M2):** event subscription: `doc.subscribe_root` vs `doc.subscribe(&ContainerID)`; `DiffEvent` shape (container path, `Diff::{Text, List, Map}` payloads) and the trigger discriminator (`EventTriggerKind::{Local, Import, Checkout}` or similar) used to filter local echoes; `subscribe_local_update` callback signature (`&[u8]` vs `Vec`, and whether it fires per-commit). -* **V5 (loro, M3):** `UndoManager` construction + `set_on_push`/`set_on_pop` cursor types; `Cursor::encode/decode` (or `to_bytes`) for presence; `doc.get_cursor_pos` return shape. -* **V6 (loro, M2):** `LoroMovableList::{insert_container, delete, mov}` exact names; `LoroMap::{insert, insert_container, get}`; binary values (`LoroValue::Binary`) for `"data"`. -* **V7 (loro, M4):** `awareness::EphemeralStore`: constructor timeout param, `set_local_state`-equivalent (`set(key, value)`?), `encode/encode_all/apply`, subscription signatures (`subscribe_local_updates` yields outbound bytes; `subscribe` yields merge events), `remove_outdated`. -* **V8 (gpui 0.2, M3/M4):** `EventEmitter` impl pattern for `RichTextEditor`; `cx.set_global`/`Global` trait for `CollabManager`; `oneshot` equivalent for `NetCommand::HostSession.reply` (use `async_channel::bounded(1)` to stay runtime-agnostic instead of a tokio oneshot). +* **V1 (iroh 0.98, M1):** endpoint builder discovery/preset wiring — docs show `Builder::new(preset)` / `.preset()` / `address_lookup()` and `presets::N0`; relays default on (`RelayMode::Default`) but **address lookup is off by default**. Confirm: the preset enabling n0 DNS lookup + pkarr publish; how to await first reachable addr for ticket minting (`endpoint.addr()` + online/watch API); the method to register a remote peer's `EndpointAddr` before dialing (`add_endpoint_addr` or similar). docs.rs/iroh/0.98.2. +* **V2 (iroh-gossip 0.100, M1):** `Gossip::builder()` construction + `max_message_size` config (default is small — confirm the number; our inline limit must stay under it); `iroh_gossip::ALPN` const path; Router registration pattern (`accept(ALPN, gossip)`); `subscribe(TopicId, Vec)` return type and split (`GossipTopic` → sender/receiver); event enum (`Received{content, delivered_from, scope}`, `NeighborUp`, `NeighborDown`, `Lagged`); `broadcast` vs `broadcast_neighbors`; whether subscribe requires the bootstrap peers' addresses pre-registered on the endpoint (it does not dial without addresses or discovery). +* **V3 (loro 1.13, M2):** `LoroText` UTF-8 surface: confirmed `mark_utf8`, `len_utf8`, `convert_pos(index, PosType, PosType)`; check for `insert_utf8`/`delete_utf8`/`splice_utf8` and `unmark` utf8 variant (else convert via `convert_pos`); `update(text, UpdateOptions)` option fields; `to_delta() -> Vec` attribute map shape. +* **V4 (loro 1.13, M2):** diff subscription: `doc.subscribe_root` vs `doc.subscribe(&ContainerID)`; `DiffEvent` shape (container path, `Diff::{Text, List, Map}`); the trigger discriminator (`EventTriggerKind::{Local, Import, Checkout}` or current names) used to filter LocalApplier echoes; `subscribe_local_update` callback signature and per-commit firing; `commit_with`/commit-options origin string API; `oplog_vv().encode()`/decode + VV comparison API; `ExportMode::updates(&vv)` exact constructor; pending-import status type returned by `import`. +* **V5 (loro 1.13, M3):** `UndoManager`: constructor, merge-interval + max-steps config, `set_on_push`/`set_on_pop` cursor callback types; `Cursor` encode/decode; `doc.get_cursor_pos` return shape. +* **V6 (loro 1.13, M2):** `LoroMovableList::{insert_container, delete, mov}` exact names/signatures; `LoroMap::{insert, insert_container, get}`; `LoroValue::Binary` for `"data"`; **`LoroDoc::clone` semantics** (shared handle vs deep copy — determines §10.3's serve-side doc access; `fork()` is the deep copy). +* **V7 (loro 1.13, M4):** `awareness::EphemeralStore`: constructor timeout param; set/get/delete-own-key API (the §9 leave broadcast needs an explicit removal or equivalent); `encode`/`encode_all`/`apply`; both subscription kinds (local-updates-bytes vs merged-events); `remove_outdated`; whether peer timestamps are wall-clock-compared across peers (F21). +* **V8 (gpui 0.2 / workspace, M3–M4):** `EventEmitter` impl pattern for an existing entity type; `Global` trait + `cx.set_global` for `CollabManager`; `window.prompt` multi-step sequencing from a detached `cx.spawn` (the §13.4 transactional flow — follow the `close_document_panel` pattern at `documents.rs:670`); how existing entity tests construct editors (`workspace/workspace/tests.rs`) for M3's tests. --- From 77fa1661a4a233b8bf51425a959708564771c0c2 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Fri, 12 Jun 2026 22:41:18 -0500 Subject: [PATCH 03/45] add p2p collaboration --- Cargo.lock | 2468 ++++++++++++++++- Cargo.toml | 12 + crates/flowstate-collab/Cargo.toml | 27 + crates/flowstate-collab/src/binding.rs | 208 ++ crates/flowstate-collab/src/ids.rs | 100 + crates/flowstate-collab/src/lib.rs | 21 + crates/flowstate-collab/src/local_apply.rs | 639 +++++ .../flowstate-collab/src/net/anti_entropy.rs | 116 + crates/flowstate-collab/src/net/blobs.rs | 83 + crates/flowstate-collab/src/net/direct.rs | 307 ++ crates/flowstate-collab/src/net/mod.rs | 53 + crates/flowstate-collab/src/net/runtime.rs | 179 ++ crates/flowstate-collab/src/net/swarm.rs | 180 ++ crates/flowstate-collab/src/presence.rs | 157 ++ crates/flowstate-collab/src/projection.rs | 176 ++ crates/flowstate-collab/src/proto_direct.rs | 67 + crates/flowstate-collab/src/proto_gossip.rs | 41 + crates/flowstate-collab/src/remote_apply.rs | 293 ++ crates/flowstate-collab/src/schema.rs | 305 ++ crates/flowstate-collab/src/self_check.rs | 37 + crates/flowstate-collab/src/ticket.rs | 57 + crates/flowstate-collab/tests/local_apply.rs | 48 + crates/flowstate-collab/tests/projection.rs | 51 + crates/flowstate/Cargo.toml | 3 + crates/flowstate/src/collab/mod.rs | 390 +++ crates/flowstate/src/collab/session.rs | 768 +++++ crates/flowstate/src/commands.rs | 8 + crates/flowstate/src/commands/keymap.rs | 4 + crates/flowstate/src/lib.rs | 1 + crates/flowstate/src/rich_text_element/mod.rs | 4 + .../src/workspace/workspace/collab.rs | 214 ++ .../src/workspace/workspace/documents.rs | 10 +- .../src/workspace/workspace/keybindings.rs | 4 + .../flowstate/src/workspace/workspace/mod.rs | 1 + .../src/workspace/workspace/render_top_bar.rs | 1 + .../src/workspace/workspace/top_bar.rs | 33 + .../src/workspace/workspace/window.rs | 3 + crates/gpui-flowtext/Cargo.toml | 2 + crates/gpui-flowtext/src/collaboration.rs | 52 +- .../src/rich_text/editor/block_insertion.rs | 25 +- .../src/rich_text/editor/collab_apply.rs | 251 ++ .../src/rich_text/editor/commands.rs | 10 + .../src/rich_text/editor/edit_pipeline.rs | 55 +- .../src/rich_text/editor/lifecycle.rs | 34 + .../src/rich_text/editor/media.rs | 20 +- .../gpui-flowtext/src/rich_text/editor/mod.rs | 13 + .../src/rich_text/editor/mouse.rs | 25 +- .../src/rich_text/editor/movement_core.rs | 1 + .../src/rich_text/editor/object_selection.rs | 23 +- .../src/rich_text/editor/paste.rs | 14 +- .../src/rich_text/editor/serialization.rs | 77 +- .../editor/table_equation_editing.rs | 27 +- .../src/rich_text/editor/tables.rs | 9 +- .../src/rich_text/tests/collab_capture.rs | 59 + .../gpui-flowtext/src/rich_text/tests/mod.rs | 1 + 55 files changed, 7600 insertions(+), 167 deletions(-) create mode 100644 crates/flowstate-collab/Cargo.toml create mode 100644 crates/flowstate-collab/src/binding.rs create mode 100644 crates/flowstate-collab/src/ids.rs create mode 100644 crates/flowstate-collab/src/lib.rs create mode 100644 crates/flowstate-collab/src/local_apply.rs create mode 100644 crates/flowstate-collab/src/net/anti_entropy.rs create mode 100644 crates/flowstate-collab/src/net/blobs.rs create mode 100644 crates/flowstate-collab/src/net/direct.rs create mode 100644 crates/flowstate-collab/src/net/mod.rs create mode 100644 crates/flowstate-collab/src/net/runtime.rs create mode 100644 crates/flowstate-collab/src/net/swarm.rs create mode 100644 crates/flowstate-collab/src/presence.rs create mode 100644 crates/flowstate-collab/src/projection.rs create mode 100644 crates/flowstate-collab/src/proto_direct.rs create mode 100644 crates/flowstate-collab/src/proto_gossip.rs create mode 100644 crates/flowstate-collab/src/remote_apply.rs create mode 100644 crates/flowstate-collab/src/schema.rs create mode 100644 crates/flowstate-collab/src/self_check.rs create mode 100644 crates/flowstate-collab/src/ticket.rs create mode 100644 crates/flowstate-collab/tests/local_apply.rs create mode 100644 crates/flowstate-collab/tests/projection.rs create mode 100644 crates/flowstate/src/collab/mod.rs create mode 100644 crates/flowstate/src/collab/session.rs create mode 100644 crates/flowstate/src/workspace/workspace/collab.rs create mode 100644 crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs create mode 100644 crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs diff --git a/Cargo.lock b/Cargo.lock index 311e03d5..2460d5df 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,12 +2,31 @@ # It is not intended for manual editing. version = 4 +[[package]] +name = "addr2line" +version = "0.25.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b5d307320b3181d6d7954e663bd7c774a838b8220fe0593c86d9fb09f498b4b" +dependencies = [ + "gimli", +] + [[package]] name = "adler2" version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" +[[package]] +name = "aead" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" +dependencies = [ + "crypto-common 0.1.7", + "generic-array", +] + [[package]] name = "aes" version = "0.8.4" @@ -31,6 +50,20 @@ dependencies = [ "cpufeatures 0.3.0", ] +[[package]] +name = "aes-gcm" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" +dependencies = [ + "aead", + "aes 0.8.4", + "cipher 0.4.4", + "ctr", + "ghash", + "subtle", +] + [[package]] name = "ahash" version = "0.7.8" @@ -154,6 +187,12 @@ version = "1.0.102" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" +[[package]] +name = "append-only-bytes" +version = "0.1.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac436601d6bdde674a0d7fb593e829ffe7b3387c351b356dd20e2d40f5bf3ee5" + [[package]] name = "approx" version = "0.5.1" @@ -177,6 +216,9 @@ name = "arbitrary" version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" +dependencies = [ + "derive_arbitrary", +] [[package]] name = "arc-swap" @@ -210,6 +252,12 @@ version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" +[[package]] +name = "arref" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ccd462b64c3c72f1be8305905a85d85403d768e8690c9b8bd3b9009a5761679" + [[package]] name = "as-raw-xcb-connection" version = "1.0.1" @@ -508,6 +556,17 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "async_io_stream" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d7b9decdf35d8908a7e3ef02f64c5e9b1695e230154c0e8de3969142d9b94c" +dependencies = [ + "futures", + "pharos", + "rustc_version", +] + [[package]] name = "async_zip" version = "0.0.17" @@ -542,6 +601,18 @@ version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" +[[package]] +name = "attohttpc" +version = "0.30.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "16e2cdb6d5ed835199484bb92bb8b3edd526effe995c61732580439c1a67e2e9" +dependencies = [ + "base64", + "http", + "log", + "url", +] + [[package]] name = "autocfg" version = "1.5.1" @@ -597,6 +668,38 @@ version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be5eb007b7cacc6c660343e96f650fedf4b5a77512399eb952ca6642cf8d13f7" +[[package]] +name = "backon" +version = "1.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef" +dependencies = [ + "fastrand 2.4.1", + "gloo-timers", + "tokio", +] + +[[package]] +name = "backtrace" +version = "0.3.76" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb531853791a215d7c62a30daf0dde835f381ab5de4589cfe7c649d2cbe92bd6" +dependencies = [ + "addr2line", + "cfg-if", + "libc", + "miniz_oxide", + "object 0.37.3", + "rustc-demangle", + "windows-link 0.2.1", +] + +[[package]] +name = "base16ct" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd307490d624467aa6f74b0eabb77633d1f758a7b25f12bceb0b22e08d9726f6" + [[package]] name = "base62" version = "2.2.4" @@ -609,6 +712,12 @@ version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + [[package]] name = "beef" version = "0.5.2" @@ -694,6 +803,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "bitmaps" +version = "2.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "031043d04099746d8db04daf1fa424b2bc8bd69d92b25962dcde24da39ab64a2" +dependencies = [ + "typenum", +] + [[package]] name = "bitpacking" version = "0.9.3" @@ -868,7 +986,7 @@ version = "3.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "519bd3116aeeb42d5372c29d982d16d0170d3d4a5ed85fc7dd91642ffff3c67c" dependencies = [ - "darling", + "darling 0.23.0", "ident_case", "prettyplease", "proc-macro2", @@ -970,6 +1088,9 @@ name = "bytes" version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" +dependencies = [ + "serde", +] [[package]] name = "bzip2" @@ -1119,6 +1240,7 @@ dependencies = [ "iana-time-zone", "js-sys", "num-traits", + "serde", "wasm-bindgen", "windows-link 0.2.1", ] @@ -1342,6 +1464,16 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + [[package]] name = "comemo" version = "0.5.1" @@ -1441,6 +1573,25 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" +[[package]] +name = "convert_case" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "633458d4ef8c78b72454de2d54fd6ab2e60f9e02be22f3c6104cdc8a4e0fceb9" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "cordyceps" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "688d7fbb8092b8de775ef2536f36c8c31f2bc4006ece2e8d8ad2d17d00ce0a2a" +dependencies = [ + "loom", + "tracing", +] + [[package]] name = "core-foundation" version = "0.9.4" @@ -1759,6 +1910,15 @@ version = "0.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e2931af7e13dc045d8e9d26afccc6fa115d64e115c9c84b1166288b46f6782c2" +[[package]] +name = "ctr" +version = "0.9.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" +dependencies = [ + "cipher 0.4.4", +] + [[package]] name = "ctutils" version = "0.4.2" @@ -1768,14 +1928,67 @@ dependencies = [ "cmov", ] +[[package]] +name = "curve25519-dalek" +version = "5.0.0-pre.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "335f1947f241137a14106b6f5acc5918a5ede29c9d71d3f2cb1678d5075d9fc3" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "curve25519-dalek-derive", + "digest 0.11.3", + "fiat-crypto", + "rand_core 0.10.1", + "rustc_version", + "serde", + "subtle", + "zeroize", +] + +[[package]] +name = "curve25519-dalek-derive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "darling" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee" +dependencies = [ + "darling_core 0.20.11", + "darling_macro 0.20.11", +] + [[package]] name = "darling" version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" dependencies = [ - "darling_core", - "darling_macro", + "darling_core 0.23.0", + "darling_macro 0.23.0", +] + +[[package]] +name = "darling_core" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e" +dependencies = [ + "fnv", + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn 2.0.117", ] [[package]] @@ -1791,17 +2004,54 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "darling_macro" +version = "0.20.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" +dependencies = [ + "darling_core 0.20.11", + "quote", + "syn 2.0.117", +] + [[package]] name = "darling_macro" version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" dependencies = [ - "darling_core", + "darling_core 0.23.0", "quote", "syn 2.0.117", ] +[[package]] +name = "data-encoding" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" + +[[package]] +name = "data-encoding-macro" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3259c913752a86488b501ed8680446a5ed2d5aeac6e596cb23ba3800768ea32c" +dependencies = [ + "data-encoding", + "data-encoding-macro-internal", +] + +[[package]] +name = "data-encoding-macro-internal" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccc2776f0c61eca1ca32528f85548abd1a4be8fb53d1b21c013e4f18da1e7090" +dependencies = [ + "data-encoding", + "syn 2.0.117", +] + [[package]] name = "data-url" version = "0.3.2" @@ -1820,6 +2070,17 @@ version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" +[[package]] +name = "der" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "71fd89660b2dc699704064e59e9dba0147b903e85319429e131620d022be411b" +dependencies = [ + "const-oid", + "pem-rfc7468", + "zeroize", +] + [[package]] name = "deranged" version = "0.5.8" @@ -1830,19 +2091,96 @@ dependencies = [ "serde_core", ] +[[package]] +name = "derive_arbitrary" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "derive_builder" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947" +dependencies = [ + "derive_builder_macro", +] + +[[package]] +name = "derive_builder_core" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8" +dependencies = [ + "darling 0.20.11", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "derive_builder_macro" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" +dependencies = [ + "derive_builder_core", + "syn 2.0.117", +] + [[package]] name = "derive_more" version = "0.99.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" dependencies = [ - "convert_case", + "convert_case 0.4.0", + "proc-macro2", + "quote", + "rustc_version", + "syn 2.0.117", +] + +[[package]] +name = "derive_more" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134" +dependencies = [ + "derive_more-impl", +] + +[[package]] +name = "derive_more-impl" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb" +dependencies = [ + "convert_case 0.10.0", "proc-macro2", "quote", "rustc_version", "syn 2.0.117", + "unicode-xid", ] +[[package]] +name = "diatomic-waker" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab03c107fafeb3ee9f5925686dbb7a73bc76e3932abb0d2b365cb64b169cf04c" + +[[package]] +name = "diff" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8" + [[package]] name = "digest" version = "0.10.7" @@ -1963,6 +2301,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" dependencies = [ "bitflags 2.13.0", + "block2", + "libc", "objc2", ] @@ -1986,6 +2326,17 @@ dependencies = [ "libloading", ] +[[package]] +name = "dlopen2" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e2c5bd4158e66d1e215c49b837e11d62f3267b30c92f1d171c4d3105e3dc4d4" +dependencies = [ + "libc", + "once_cell", + "winapi", +] + [[package]] name = "docx-rs" version = "0.4.20" @@ -2098,13 +2449,40 @@ dependencies = [ ] [[package]] -name = "either" -version = "1.16.0" +name = "ed25519" +version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" +checksum = "29fcf32e6c73d1079f83ab4d782de2d81620346a5f38c6237a86a22f8368980a" +dependencies = [ + "pkcs8", + "serdect", + "signature", +] [[package]] -name = "embed-resource" +name = "ed25519-dalek" +version = "3.0.0-pre.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "20449acd54b660981ae5caa2bcb56d1fe7f25f2e37a38ec507400fab034d4bb6" +dependencies = [ + "curve25519-dalek", + "ed25519", + "rand_core 0.10.1", + "serde", + "sha2 0.11.0", + "signature", + "subtle", + "zeroize", +] + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "embed-resource" version = "3.0.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c31a88c8d26de40ed18fe748c547845aa39de1db3afd958f8cb91579f3644bcb" @@ -2159,6 +2537,47 @@ version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "66b7e2430c6dff6a955451e2cfc438f09cea1965a9d6f87f7e3b90decc014099" +[[package]] +name = "ensure-cov" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33753185802e107b8fa907192af1f0eca13b1fb33327a59266d650fef29b2b4e" + +[[package]] +name = "enum-as-inner" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9720bba047d567ffc8a3cba48bf19126600e249ab7f128e9233e6376976a116" +dependencies = [ + "heck 0.4.1", + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "enum-as-inner" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1e6a265c649f3f5979b601d26f1d05ada116434c87741c9493cb56218f76cbc" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "enum-assoc" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ed8956bd5c1f0415200516e78ff07ec9e16415ade83c056c230d7b7ea0d55b7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "enum-iterator" version = "2.3.0" @@ -2199,6 +2618,18 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "enum_dispatch" +version = "0.3.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aa18ce2bc66555b3218614519ac839ddb759a7d6720732f979ef8d13be147ecd" +dependencies = [ + "once_cell", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "enumflags2" version = "0.7.12" @@ -2460,6 +2891,12 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "fiat-crypto" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "64cd1e32ddd350061ae6edb1b082d7c54915b5c672c389143b9a63403a109f24" + [[package]] name = "file-id" version = "0.2.3" @@ -2496,6 +2933,12 @@ version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + [[package]] name = "flate2" version = "1.1.9" @@ -2530,10 +2973,12 @@ name = "flowstate" version = "0.1.0" dependencies = [ "anyhow", + "async-channel 2.5.0", "clap", "cosmic-text", "dirs 6.0.0", "fff-search", + "flowstate-collab", "flowstate-document", "flowstate-docx", "flowstate-flow", @@ -2543,6 +2988,7 @@ dependencies = [ "gpui-component-assets", "gpui-symbols", "hotpath", + "loro", "memchr", "mimalloc", "rustc-hash 2.1.2", @@ -2553,6 +2999,27 @@ dependencies = [ "uuid", ] +[[package]] +name = "flowstate-collab" +version = "0.1.0" +dependencies = [ + "anyhow", + "async-channel 2.5.0", + "gpui-flowtext", + "iroh", + "iroh-gossip", + "iroh-tickets", + "loro", + "n0-future", + "postcard", + "proptest", + "rand 0.9.4", + "serde", + "tokio", + "twox-hash", + "uuid", +] + [[package]] name = "flowstate-document" version = "0.1.0" @@ -2624,7 +3091,7 @@ dependencies = [ "futures-core", "futures-sink", "nanorand", - "spin", + "spin 0.9.8", ] [[package]] @@ -2797,6 +3264,19 @@ dependencies = [ "futures-util", ] +[[package]] +name = "futures-buffered" +version = "0.2.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4421cb78ee172b6b06080093479d3c50f058e7c81b7d577bbb8d118d551d4cd5" +dependencies = [ + "cordyceps", + "diatomic-waker", + "futures-core", + "pin-project-lite", + "spin 0.10.0", +] + [[package]] name = "futures-channel" version = "0.3.32" @@ -2807,6 +3287,19 @@ dependencies = [ "futures-sink", ] +[[package]] +name = "futures-concurrency" +version = "7.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "175cd8cca9e1d45b87f18ffa75088f2099e3c4fe5e2f83e42de112560bea8ea6" +dependencies = [ + "fixedbitset", + "futures-core", + "futures-lite 2.6.1", + "pin-project", + "smallvec", +] + [[package]] name = "futures-core" version = "0.3.32" @@ -2907,6 +3400,21 @@ dependencies = [ "byteorder", ] +[[package]] +name = "generator" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b3b854b0e584ead1a33f18b2fcad7cf7be18b3875c78816b753639aa501513ae" +dependencies = [ + "cc", + "cfg-if", + "libc", + "log", + "rustversion", + "windows-link 0.2.1", + "windows-result 0.4.1", +] + [[package]] name = "generic-array" version = "0.14.7" @@ -2917,6 +3425,20 @@ dependencies = [ "version_check", ] +[[package]] +name = "generic-btree" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0c1bce85c110ab718fd139e0cc89c51b63bd647b14a767e24bdfc77c83df79b" +dependencies = [ + "arref", + "heapless 0.9.3", + "itertools 0.11.0", + "loro-thunderdome", + "proc-macro2", + "rustc-hash 2.1.2", +] + [[package]] name = "gethostname" version = "1.1.0" @@ -2970,6 +3492,16 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "ghash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" +dependencies = [ + "opaque-debug", + "polyval", +] + [[package]] name = "gif" version = "0.13.3" @@ -2990,6 +3522,12 @@ dependencies = [ "weezl", ] +[[package]] +name = "gimli" +version = "0.32.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e629b9b98ef3dd8afe6ca2bd0f89306cec16d43d907889945bc5d6687f2f13c7" + [[package]] name = "git2" version = "0.20.4" @@ -3103,6 +3641,7 @@ dependencies = [ "as-raw-xcb-connection", "ashpd 0.11.1", "async-task", + "backtrace", "bindgen", "blade-graphics", "blade-macros", @@ -3121,7 +3660,7 @@ dependencies = [ "core-video", "cosmic-text", "ctor", - "derive_more", + "derive_more 0.99.20", "embed-resource", "etagere", "filedescriptor", @@ -3179,7 +3718,7 @@ dependencies = [ "wayland-protocols-plasma", "windows 0.61.3", "windows-core 0.61.2", - "windows-numerics", + "windows-numerics 0.2.0", "windows-registry 0.5.3", "x11-clipboard", "x11rb", @@ -3330,7 +3869,7 @@ dependencies = [ "async-compression", "async-fs", "bytes", - "derive_more", + "derive_more 0.99.20", "futures", "gpui_util", "http", @@ -3417,12 +3956,15 @@ dependencies = [ "dunce", "futures", "futures-lite 1.13.0", + "git2", "globset", "gpui_collections", + "gpui_util_macros", "itertools 0.14.0", "libc", "log", "nix 0.29.0", + "rand 0.9.4", "regex", "rust-embed", "schemars", @@ -3496,6 +4038,15 @@ dependencies = [ "byteorder", ] +[[package]] +name = "hash32" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" +dependencies = [ + "byteorder", +] + [[package]] name = "hashbrown" version = "0.12.3" @@ -3537,6 +4088,8 @@ version = "0.17.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" dependencies = [ + "allocator-api2", + "equivalent", "foldhash 0.2.0", ] @@ -3657,10 +4210,30 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cdc6457c0eb62c71aac4bc17216026d8410337c4126773b9c5daba343f17964f" dependencies = [ "atomic-polyfill", - "hash32", + "hash32 0.2.1", "rustc_version", "serde", - "spin", + "spin 0.9.8", + "stable_deref_trait", +] + +[[package]] +name = "heapless" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +dependencies = [ + "hash32 0.3.1", + "stable_deref_trait", +] + +[[package]] +name = "heapless" +version = "0.9.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "25ba4bd83f9415b58b4ed8dc5714c76e626a105be4646c02630ad730ad3b5aa4" +dependencies = [ + "hash32 0.3.1", "stable_deref_trait", ] @@ -3732,6 +4305,83 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dfa686283ad6dd069f105e5ab091b04c62850d3e4cf5d67debad1933f55023df" +[[package]] +name = "hickory-net" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2295ed2f9c31e471e1428a8f88a3f0e1f4b27c15049592138d1eebe9c35b183" +dependencies = [ + "async-trait", + "bytes", + "cfg-if", + "data-encoding", + "futures-channel", + "futures-io", + "futures-util", + "h2", + "hickory-proto", + "http", + "idna", + "ipnet", + "jni", + "rand 0.10.1", + "rustls", + "thiserror 2.0.18", + "tinyvec", + "tokio", + "tokio-rustls", + "tracing", + "url", +] + +[[package]] +name = "hickory-proto" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bab31817bfb44672a252e97fe81cd0c18d1b2cf892108922f6818820df8c643" +dependencies = [ + "data-encoding", + "idna", + "ipnet", + "jni", + "once_cell", + "prefix-trie", + "rand 0.10.1", + "ring", + "thiserror 2.0.18", + "tinyvec", + "tracing", + "url", +] + +[[package]] +name = "hickory-resolver" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0d58d28879ceecde6607729660c2667a081ccdc082e082675042793960f178c" +dependencies = [ + "cfg-if", + "futures-util", + "hickory-net", + "hickory-proto", + "ipconfig", + "ipnet", + "jni", + "moka", + "ndk-context", + "once_cell", + "parking_lot", + "rand 0.10.1", + "resolv-conf", + "rustls", + "smallvec", + "system-configuration 0.7.0", + "thiserror 2.0.18", + "tokio", + "tokio-rustls", + "tracing", +] + [[package]] name = "hidden-trait" version = "0.1.2" @@ -3920,6 +4570,7 @@ dependencies = [ "http", "http-body", "httparse", + "httpdate", "itoa", "pin-project-lite", "smallvec", @@ -3949,13 +4600,16 @@ version = "0.1.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" dependencies = [ + "base64", "bytes", "futures-channel", "futures-util", "http", "http-body", "hyper", + "ipnet", "libc", + "percent-encoding", "pin-project-lite", "socket2", "tokio", @@ -4235,6 +4889,12 @@ version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" +[[package]] +name = "identity-hash" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfdd7caa900436d8f13b2346fe10257e0c05c1f1f9e351f4f5d57c03bd5f45da" + [[package]] name = "idna" version = "1.1.0" @@ -4256,6 +4916,26 @@ dependencies = [ "icu_properties 2.2.0", ] +[[package]] +name = "igd-next" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de7238d487a9aff61f81b5ab41c0a841532a115a398b5fa92a2fadd0885e2581" +dependencies = [ + "attohttpc", + "bytes", + "futures", + "http", + "http-body-util", + "hyper", + "hyper-util", + "log", + "rand 0.10.1", + "tokio", + "url", + "xmltree", +] + [[package]] name = "ignore" version = "0.4.26" @@ -4272,6 +4952,21 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "im" +version = "15.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0acd33ff0285af998aaf9b57342af478078f53492322fafc47450e09397e0e9" +dependencies = [ + "bitmaps", + "rand_core 0.6.4", + "rand_xoshiro", + "serde", + "sized-chunks", + "typenum", + "version_check", +] + [[package]] name = "image" version = "0.25.10" @@ -4303,7 +4998,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "525e9ff3e1a4be2fbea1fdf0e98686a6d98b4d8f937e1bf7402245af1909e8c3" dependencies = [ "byteorder-lite", - "quick-error", + "quick-error 2.0.1", ] [[package]] @@ -4421,12 +5116,270 @@ dependencies = [ "leaky-cow", ] +[[package]] +name = "ipconfig" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d40460c0ce33d6ce4b0630ad68ff63d6661961c48b6dba35e5a4d81cfb48222" +dependencies = [ + "socket2", + "widestring", + "windows-registry 0.6.1", + "windows-result 0.4.1", + "windows-sys 0.61.2", +] + [[package]] name = "ipnet" version = "2.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" - +dependencies = [ + "serde", +] + +[[package]] +name = "iroh" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef865dc2d11a19fe670ff217b68ffc3b511bddf473dc3a3e120090b9f691803" +dependencies = [ + "backon", + "blake3", + "bytes", + "cfg_aliases", + "ctutils", + "data-encoding", + "derive_more 2.1.1", + "ed25519-dalek", + "futures-util", + "getrandom 0.4.2", + "hickory-resolver", + "http", + "ipnet", + "iroh-base", + "iroh-dns", + "iroh-metrics", + "iroh-relay", + "n0-error", + "n0-future", + "n0-watcher", + "netwatch", + "noq", + "noq-proto", + "noq-udp", + "papaya", + "pin-project", + "portable-atomic", + "portmapper", + "rand 0.10.1", + "reqwest", + "rustc-hash 2.1.2", + "rustls", + "rustls-pki-types", + "rustls-webpki", + "serde", + "smallvec", + "strum 0.28.0", + "time", + "tokio", + "tokio-stream", + "tokio-util", + "tracing", + "url", + "wasm-bindgen-futures", + "webpki-roots", +] + +[[package]] +name = "iroh-base" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af93d67701c00c504982154569192ad384738c0450ba1196930314b955100552" +dependencies = [ + "curve25519-dalek", + "data-encoding", + "data-encoding-macro", + "derive_more 2.1.1", + "digest 0.11.3", + "ed25519-dalek", + "getrandom 0.4.2", + "n0-error", + "rand 0.10.1", + "serde", + "sha2 0.11.0", + "url", + "zeroize", + "zeroize_derive", +] + +[[package]] +name = "iroh-dns" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de4112c91eb64094d77df9d3112606dcf7ff216421afccd2dc762fda5a7b2879" +dependencies = [ + "arc-swap", + "cfg_aliases", + "derive_more 2.1.1", + "hickory-resolver", + "iroh-base", + "n0-error", + "n0-future", + "ndk-context", + "rand 0.10.1", + "reqwest", + "rustls", + "simple-dns", + "strum 0.28.0", + "tokio", + "tracing", + "url", +] + +[[package]] +name = "iroh-gossip" +version = "0.100.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f5cada641f84a2ff3d1acbd43e16490cf3b2a4f675e580c5c2caa5d68ee63b5" +dependencies = [ + "blake3", + "bytes", + "constant_time_eq", + "data-encoding", + "derive_more 2.1.1", + "ed25519-dalek", + "futures-concurrency", + "hex", + "indexmap", + "iroh", + "iroh-base", + "iroh-metrics", + "irpc", + "n0-error", + "n0-future", + "postcard", + "rand 0.10.1", + "serde", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "iroh-metrics" +version = "1.0.0-rc.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d102597d0ee523f17fdb672c532395e634dbe945429284c811430d63bacc0d8a" +dependencies = [ + "iroh-metrics-derive", + "itoa", + "n0-error", + "portable-atomic", + "ryu", + "serde", + "tracing", +] + +[[package]] +name = "iroh-metrics-derive" +version = "1.0.0-rc.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91c8e0c97f1dc787107f388433c349397c565572fe6406d600ff7bb7b7fe3b30" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "iroh-relay" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a70030b9e71c1183bd4f88fbdbebfa1af2a5be549dd6f20a1e8ac3cd0202ee9d" +dependencies = [ + "blake3", + "bytes", + "cfg_aliases", + "data-encoding", + "derive_more 2.1.1", + "getrandom 0.4.2", + "hickory-resolver", + "http", + "http-body-util", + "hyper", + "hyper-util", + "iroh-base", + "iroh-dns", + "iroh-metrics", + "lru 0.18.0", + "n0-error", + "n0-future", + "noq", + "noq-proto", + "num_enum", + "pin-project", + "postcard", + "rand 0.10.1", + "reqwest", + "rustls", + "rustls-pki-types", + "serde", + "serde_bytes", + "strum 0.28.0", + "tokio", + "tokio-rustls", + "tokio-util", + "tokio-websockets", + "tracing", + "url", + "vergen-gitcl", + "webpki-roots", + "ws_stream_wasm", +] + +[[package]] +name = "iroh-tickets" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b5d4051ac9825cd834d88377dab71c1d9d483b65e9740029b2d624cc17ac58d" +dependencies = [ + "data-encoding", + "derive_more 2.1.1", + "iroh-base", + "n0-error", + "postcard", + "serde", +] + +[[package]] +name = "irpc" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a05799eb70acd04843c327ef939233ccf80f607d30e9ca94857ac7f3d8f18b46" +dependencies = [ + "futures-util", + "irpc-derive", + "n0-error", + "n0-future", + "serde", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "irpc-derive" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "445d81dbc1eed4dab6379bf7f97d12ac28ce8e6f3f7d6660c9f333b7b5d8d03b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "is-docker" version = "0.2.0" @@ -4472,6 +5425,15 @@ dependencies = [ "either", ] +[[package]] +name = "itertools" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569" +dependencies = [ + "either", +] + [[package]] name = "itertools" version = "0.13.0" @@ -4537,6 +5499,55 @@ dependencies = [ "jiff-tzdb", ] +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror 2.0.18", + "walkdir", + "windows-link 0.2.1", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -4647,7 +5658,7 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" dependencies = [ - "spin", + "spin 0.9.8", ] [[package]] @@ -4665,6 +5676,12 @@ dependencies = [ "leak", ] +[[package]] +name = "leb128" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6cc46bac87ef8093eed6f272babb833b6443374399985ac8ed28471ee0918545" + [[package]] name = "leb128fmt" version = "0.1.0" @@ -4879,6 +5896,21 @@ dependencies = [ "logos-codegen", ] +[[package]] +name = "loom" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca" +dependencies = [ + "cfg-if", + "generator", + "scoped-tls", + "serde", + "serde_json", + "tracing", + "tracing-subscriber", +] + [[package]] name = "loop9" version = "0.1.5" @@ -4920,6 +5952,144 @@ dependencies = [ "weezl", ] +[[package]] +name = "loro" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7bd1b63cd2f0cf66acbbb3191c41feaff03c94c3f6c6bb1e61d2ab3062c301c7" +dependencies = [ + "enum-as-inner 0.6.1", + "generic-btree", + "loro-common", + "loro-delta", + "loro-internal", + "loro-kv-store", + "rustc-hash 2.1.2", + "tracing", +] + +[[package]] +name = "loro-common" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b51042936156a1d537df8a38938ec8ed24b45de14530af933c1d02de39c9e056" +dependencies = [ + "arbitrary", + "enum-as-inner 0.6.1", + "leb128", + "loro-rle", + "nonmax", + "rustc-hash 2.1.2", + "serde", + "serde_columnar", + "serde_json", + "thiserror 1.0.69", +] + +[[package]] +name = "loro-delta" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96f381e7fb5d4bf5605b4f4a9241f3adf61403ded978d757552004c8302cccbb" +dependencies = [ + "arrayvec", + "enum-as-inner 0.5.1", + "generic-btree", + "heapless 0.8.0", +] + +[[package]] +name = "loro-internal" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "252a84196402ac55a7dd064af87a747a430fc176ec744d5bb99f4ed92e639fa8" +dependencies = [ + "append-only-bytes", + "arref", + "bytes", + "either", + "ensure-cov", + "enum-as-inner 0.6.1", + "enum_dispatch", + "generic-btree", + "getrandom 0.2.17", + "im", + "itertools 0.12.1", + "leb128", + "loom", + "loro-common", + "loro-delta", + "loro-kv-store", + "loro-rle", + "loro_fractional_index", + "md5", + "nonmax", + "num", + "num-traits", + "once_cell", + "parking_lot", + "pest", + "pest_derive", + "postcard", + "pretty_assertions", + "rand 0.8.6", + "rustc-hash 2.1.2", + "serde", + "serde_columnar", + "serde_json", + "smallvec", + "thiserror 1.0.69", + "thread_local", + "tracing", + "wasm-bindgen", + "xxhash-rust", +] + +[[package]] +name = "loro-kv-store" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "950cc8bcc64dcff536e949fbc56ccc3d0759646fa441c7f76b3cd7c3eafa2096" +dependencies = [ + "bytes", + "ensure-cov", + "loro-common", + "lz4_flex 0.11.6", + "once_cell", + "quick_cache", + "rustc-hash 2.1.2", + "tracing", + "xxhash-rust", +] + +[[package]] +name = "loro-rle" +version = "1.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76400c3eea6bb39b013406acce964a8db39311534e308286c8d8721baba8ee20" +dependencies = [ + "append-only-bytes", + "num", + "smallvec", +] + +[[package]] +name = "loro-thunderdome" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f3d053a135388e6b1df14e8af1212af5064746e9b87a06a345a7a779ee9695a" + +[[package]] +name = "loro_fractional_index" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aca7180674d0273ddf37049a5efcde4547fd5330d24abb7519bb9d9eb6780d5b" +dependencies = [ + "once_cell", + "rand 0.8.6", + "serde", +] + [[package]] name = "lru" version = "0.16.4" @@ -4929,6 +6099,15 @@ dependencies = [ "hashbrown 0.16.1", ] +[[package]] +name = "lru" +version = "0.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a860605968fce16869fd239cf4237a82f3ac470723415db603b0e8b6c8d4fb9" +dependencies = [ + "hashbrown 0.17.1", +] + [[package]] name = "lru-slab" version = "0.1.2" @@ -5000,6 +6179,15 @@ dependencies = [ "num-traits", ] +[[package]] +name = "lz4_flex" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "373f5eceeeab7925e0c1098212f2fbc4d416adec9d35051a6ab251e824c1854a" +dependencies = [ + "twox-hash", +] + [[package]] name = "lz4_flex" version = "0.13.1" @@ -5008,11 +6196,11 @@ checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" [[package]] name = "lzma-rust2" -version = "0.16.4" +version = "0.16.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce716bf1a316f47a280fc76295f6495b5bea4752bca01c3b3885e101b1c23c02" +checksum = "47bb1e988e6fb779cf720ad431242d3f03167c1b3f2b1aae7f1a94b2495b36ae" dependencies = [ - "sha2 0.11.0", + "sha2 0.10.9", ] [[package]] @@ -5021,6 +6209,12 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4" +[[package]] +name = "mac-addr" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3d25b0e0b648a86960ac23b7ad4abb9717601dec6f66c165f5b037f3f03065f" + [[package]] name = "malloc_buf" version = "0.0.6" @@ -5094,6 +6288,12 @@ dependencies = [ "digest 0.10.7", ] +[[package]] +name = "md5" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "490cc448043f947bae3cbee9c203358d62dbee0db12107a74be5c30ccfd09771" + [[package]] name = "measure_time" version = "0.9.0" @@ -5270,6 +6470,23 @@ dependencies = [ "which 5.0.0", ] +[[package]] +name = "moka" +version = "0.12.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046" +dependencies = [ + "crossbeam-channel", + "crossbeam-epoch", + "crossbeam-utils", + "equivalent", + "parking_lot", + "portable-atomic", + "smallvec", + "tagptr", + "uuid", +] + [[package]] name = "moxcms" version = "0.7.11" @@ -5297,53 +6514,233 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2195bf6aa996a481483b29d62a7663eed3fe39600c460e323f8ff41e90bdd89b" [[package]] -name = "mutate_once" -version = "0.1.2" +name = "mutate_once" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13d2233c9842d08cfe13f9eac96e207ca6a2ea10b80259ebe8ad0268be27d2af" + +[[package]] +name = "n0-error" +version = "1.0.0-rc.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "223e946a84aa91644507a6b7865cfebbb9a231ace499041c747ab0fd30408212" +dependencies = [ + "anyhow", + "n0-error-macros", + "spez", +] + +[[package]] +name = "n0-error-macros" +version = "1.0.0-rc.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "565305a21e6b3bf26640ad98f05a0fda12d3ab4315394566b52a7bddb8b34828" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "n0-future" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2ab99dfb861450e68853d34ae665243a88b8c493d01ba957321a1e9b2312bbe" +dependencies = [ + "cfg_aliases", + "derive_more 2.1.1", + "futures-buffered", + "futures-lite 2.6.1", + "futures-util", + "js-sys", + "pin-project", + "send_wrapper", + "tokio", + "tokio-util", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-time", +] + +[[package]] +name = "n0-watcher" +version = "1.0.0-rc.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "928d8039a66cce5efcfd35e88b32d3defc8eba630b3ac451522997f563956a52" +dependencies = [ + "derive_more 2.1.1", + "n0-error", + "n0-future", +] + +[[package]] +name = "naga" +version = "25.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b977c445f26e49757f9aca3631c3b8b836942cb278d69a92e7b80d3b24da632" +dependencies = [ + "arrayvec", + "bit-set", + "bitflags 2.13.0", + "cfg_aliases", + "codespan-reporting", + "half", + "hashbrown 0.15.5", + "hexf-parse", + "indexmap", + "log", + "num-traits", + "once_cell", + "rustc-hash 1.1.0", + "spirv", + "strum 0.26.3", + "thiserror 2.0.18", + "unicode-ident", +] + +[[package]] +name = "nanorand" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a51313c5820b0b02bd422f4b44776fbf47961755c74ce64afc73bfad10226c3" +dependencies = [ + "getrandom 0.2.17", +] + +[[package]] +name = "ndk-context" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27b02d87554356db9e9a873add8782d4ea6e3e58ea071a9adb9a2e8ddb884a8b" + +[[package]] +name = "neo_frizbee" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728e0731ad3a0083b9f72a82df72c298641ba970d4e308e0897ec4c89212c31d" +dependencies = [ + "itertools 0.14.0", + "raw-cpuid", +] + +[[package]] +name = "netdev" +version = "0.43.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57bacaf873ee4eab5646f99b381b271ec75e716902a67cf962c0f328c5eb5bfb" +dependencies = [ + "block2", + "dispatch2", + "dlopen2", + "ipnet", + "libc", + "mac-addr", + "netlink-packet-core", + "netlink-packet-route 0.29.0", + "netlink-sys", + "objc2-core-foundation", + "objc2-core-wlan", + "objc2-foundation", + "objc2-system-configuration", + "once_cell", + "plist", + "windows-sys 0.61.2", +] + +[[package]] +name = "netlink-packet-core" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3463cbb78394cb0141e2c926b93fc2197e473394b761986eca3b9da2c63ae0f4" +dependencies = [ + "paste", +] + +[[package]] +name = "netlink-packet-route" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df9854ea6ad14e3f4698a7f03b65bce0833dd2d81d594a0e4a984170537146b6" +dependencies = [ + "bitflags 2.13.0", + "libc", + "log", + "netlink-packet-core", +] + +[[package]] +name = "netlink-packet-route" +version = "0.30.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13d2233c9842d08cfe13f9eac96e207ca6a2ea10b80259ebe8ad0268be27d2af" +checksum = "be8919612f6028ab4eacbbfe1234a9a43e3722c6e0915e7ff519066991905092" +dependencies = [ + "bitflags 2.13.0", + "libc", + "log", + "netlink-packet-core", +] [[package]] -name = "naga" -version = "25.0.1" +name = "netlink-proto" +version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b977c445f26e49757f9aca3631c3b8b836942cb278d69a92e7b80d3b24da632" +checksum = "b65d130ee111430e47eed7896ea43ca693c387f097dd97376bffafbf25812128" dependencies = [ - "arrayvec", - "bit-set", - "bitflags 2.13.0", - "cfg_aliases", - "codespan-reporting", - "half", - "hashbrown 0.15.5", - "hexf-parse", - "indexmap", + "bytes", + "futures", "log", - "num-traits", - "once_cell", - "rustc-hash 1.1.0", - "spirv", - "strum 0.26.3", + "netlink-packet-core", + "netlink-sys", "thiserror 2.0.18", - "unicode-ident", ] [[package]] -name = "nanorand" -version = "0.7.0" +name = "netlink-sys" +version = "0.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a51313c5820b0b02bd422f4b44776fbf47961755c74ce64afc73bfad10226c3" +checksum = "cd6c30ed10fa69cc491d491b85cc971f6bdeb8e7367b7cde2ee6cc878d583fae" dependencies = [ - "getrandom 0.2.17", + "bytes", + "futures-util", + "libc", + "log", + "tokio", ] [[package]] -name = "neo_frizbee" -version = "0.10.2" +name = "netwatch" +version = "0.18.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "728e0731ad3a0083b9f72a82df72c298641ba970d4e308e0897ec4c89212c31d" +checksum = "2071e0c2b5b229622c459096b84f1ad51afa150cdeeefdad491ef3704e581d91" dependencies = [ - "itertools 0.14.0", - "raw-cpuid", + "atomic-waker", + "bytes", + "cfg_aliases", + "derive_more 2.1.1", + "js-sys", + "libc", + "n0-error", + "n0-future", + "n0-watcher", + "netdev", + "netlink-packet-core", + "netlink-packet-route 0.30.0", + "netlink-proto", + "netlink-sys", + "noq-udp", + "objc2-core-foundation", + "objc2-system-configuration", + "pin-project-lite", + "serde", + "socket2", + "time", + "tokio", + "tokio-util", + "tracing", + "web-sys", + "windows 0.62.2", + "windows-result 0.4.1", + "wmi", ] [[package]] @@ -5415,12 +6812,80 @@ dependencies = [ "nom 8.0.0", ] +[[package]] +name = "nonmax" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "610a5acd306ec67f907abe5567859a3c693fb9886eb1f012ab8f2a47bef3db51" + [[package]] name = "noop_proc_macro" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0676bb32a98c1a483ce53e500a81ad9c3d5b3f7c920c28c24e9cb0980d0b5bc8" +[[package]] +name = "noq" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "198b99fc085a5db1f7d259edb5ede8311e59f28cdd2687920b4313613d21a73f" +dependencies = [ + "bytes", + "cfg_aliases", + "derive_more 2.1.1", + "noq-proto", + "noq-udp", + "pin-project-lite", + "rustc-hash 2.1.2", + "rustls", + "socket2", + "thiserror 2.0.18", + "tokio", + "tokio-stream", + "tracing", + "web-time", +] + +[[package]] +name = "noq-proto" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ab0ac774795ce1e42a7e61266e71f3be8110210630441169ac8dda403dd23f1" +dependencies = [ + "aes-gcm", + "bytes", + "derive_more 2.1.1", + "enum-assoc", + "getrandom 0.4.2", + "identity-hash", + "lru-slab", + "rand 0.10.1", + "rand_pcg", + "ring", + "rustc-hash 2.1.2", + "rustls", + "rustls-pki-types", + "slab", + "sorted-index-buffer", + "thiserror 2.0.18", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "noq-udp" +version = "1.0.0-rc.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b3c1520eacd33fd6b009e2e70116b05508ade51db5e0d315ff8bf6b702148c2b" +dependencies = [ + "cfg_aliases", + "libc", + "socket2", + "tracing", + "windows-sys 0.61.2", +] + [[package]] name = "normpath" version = "1.5.1" @@ -5641,6 +7106,37 @@ dependencies = [ "libc", ] +[[package]] +name = "num_enum" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d0bca838442ec211fa11de3a8b0e0e8f3a4522575b5c4c06ed722e005036f26" +dependencies = [ + "num_enum_derive", + "rustversion", +] + +[[package]] +name = "num_enum_derive" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8" +dependencies = [ + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "num_threads" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c7398b9c8b70908f6371f47ed36737907c87c52af34c268fed0bf0ceb92ead9" +dependencies = [ + "libc", +] + [[package]] name = "objc" version = "0.2.7" @@ -5691,7 +7187,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536" dependencies = [ "bitflags 2.13.0", + "block2", "dispatch2", + "libc", "objc2", ] @@ -5705,6 +7203,20 @@ dependencies = [ "objc2-core-foundation", ] +[[package]] +name = "objc2-core-wlan" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c71e34919aba0d701380d911702455038a8a3587467fe0141d6a71501e7ffe48" +dependencies = [ + "bitflags 2.13.0", + "objc2", + "objc2-core-foundation", + "objc2-foundation", + "objc2-security", + "objc2-security-foundation", +] + [[package]] name = "objc2-encode" version = "4.1.0" @@ -5718,6 +7230,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3e0adef53c21f888deb4fa59fc59f7eb17404926ee8a6f59f5df0fd7f9f3272" dependencies = [ "bitflags 2.13.0", + "block2", + "libc", "objc2", "objc2-core-foundation", ] @@ -5747,6 +7261,41 @@ dependencies = [ "objc2-metal", ] +[[package]] +name = "objc2-security" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe137109bd1e8b5a99390f77a7d8b2961dafc1a1c5db8f2e60329ad6d895a" +dependencies = [ + "bitflags 2.13.0", + "objc2", + "objc2-core-foundation", +] + +[[package]] +name = "objc2-security-foundation" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef76382e9cedd18123099f17638715cc3d81dba3637d4c0d39ab69df2ef345a5" +dependencies = [ + "objc2", + "objc2-foundation", +] + +[[package]] +name = "objc2-system-configuration" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7216bd11cbda54ccabcab84d523dc93b858ec75ecfb3a7d89513fa22464da396" +dependencies = [ + "bitflags 2.13.0", + "dispatch2", + "libc", + "objc2", + "objc2-core-foundation", + "objc2-security", +] + [[package]] name = "objc2-ui-kit" version = "0.3.2" @@ -5801,6 +7350,10 @@ name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "critical-section", + "portable-atomic", +] [[package]] name = "once_cell_polyfill" @@ -5849,6 +7402,12 @@ dependencies = [ "zvariant", ] +[[package]] +name = "opaque-debug" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" + [[package]] name = "open" version = "5.3.5" @@ -5934,6 +7493,16 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "papaya" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "997ee03cd38c01469a7046643714f0ad28880bcb9e6679ff0666e24817ca19b7" +dependencies = [ + "equivalent", + "seize", +] + [[package]] name = "parking" version = "2.2.1" @@ -6032,12 +7601,74 @@ dependencies = [ "ryu", ] +[[package]] +name = "pem-rfc7468" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6305423e0e7738146434843d1694d621cce767262b2a86910beab705e4493d9" +dependencies = [ + "base64ct", +] + [[package]] name = "percent-encoding" version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" +[[package]] +name = "pest" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0848c601009d37dfa3430c4666e147e49cdcf1b92ecd3e63657d8a5f19da662" +dependencies = [ + "memchr", + "ucd-trie", +] + +[[package]] +name = "pest_derive" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11f486f1ea21e6c10ed15d5a7c77165d0ee443402f0780849d1768e7d9d6fe77" +dependencies = [ + "pest", + "pest_generator", +] + +[[package]] +name = "pest_generator" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8040c4647b13b210a963c1ed407c1ff4fdfa01c31d6d2a098218702e6664f94f" +dependencies = [ + "pest", + "pest_meta", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "pest_meta" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89815c69d36021a140146f26659a81d6c2afa33d216d736dd4be5381a7362220" +dependencies = [ + "pest", + "sha2 0.10.9", +] + +[[package]] +name = "pharos" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9567389417feee6ce15dd6527a8a1ecac205ef62c2932bcf3d9f6fc5b78b414" +dependencies = [ + "futures", + "rustc_version", +] + [[package]] name = "phf" version = "0.11.3" @@ -6182,6 +7813,16 @@ dependencies = [ "futures-io", ] +[[package]] +name = "pkcs8" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "451913da69c775a56034ea8d9003d27ee8948e12443eae7c038ba100a4f21cb7" +dependencies = [ + "der", + "spki", +] + [[package]] name = "pkg-config" version = "0.3.33" @@ -6256,19 +7897,63 @@ dependencies = [ "arrayvec", ] +[[package]] +name = "polyval" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "opaque-debug", + "universal-hash", +] + [[package]] name = "portable-atomic" version = "1.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" +dependencies = [ + "serde", +] [[package]] name = "portable-atomic-util" version = "0.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "portmapper" +version = "0.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "64959cbabf952c8ffcbaea13745308508f1f825922f4068353f3de08d42cf214" dependencies = [ - "portable-atomic", + "base64", + "bytes", + "derive_more 2.1.1", + "hyper-util", + "igd-next", + "iroh-metrics", + "libc", + "n0-error", + "n0-future", + "netwatch", + "num_enum", + "rand 0.10.1", + "serde", + "smallvec", + "socket2", + "time", + "tokio", + "tokio-util", + "tower-layer", + "tracing", + "url", ] [[package]] @@ -6297,10 +7982,22 @@ dependencies = [ "cobs", "embedded-io 0.4.0", "embedded-io 0.6.1", - "heapless", + "heapless 0.7.17", + "postcard-derive", "serde", ] +[[package]] +name = "postcard-derive" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0232bd009a197ceec9cc881ba46f727fcd8060a2d8d6a9dde7a69030a6fe2bb" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "potential_utf" version = "0.1.5" @@ -6337,6 +8034,27 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "925383efa346730478fb4838dbe9137d2a47675ad789c546d150a6e1dd4ab31c" +[[package]] +name = "prefix-trie" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cf6e3177f0684016a5c209b00882e15f8bdd3f3bb48f0491df10cd102d0c6e7" +dependencies = [ + "either", + "ipnet", + "num-traits", +] + +[[package]] +name = "pretty_assertions" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d" +dependencies = [ + "diff", + "yansi", +] + [[package]] name = "prettyplease" version = "0.2.37" @@ -6425,6 +8143,25 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "proptest" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744" +dependencies = [ + "bit-set", + "bit-vec", + "bitflags 2.13.0", + "num-traits", + "rand 0.9.4", + "rand_chacha 0.9.0", + "rand_xorshift", + "regex-syntax 0.8.10", + "rusty-fork", + "tempfile", + "unarray", +] + [[package]] name = "psm" version = "0.1.31" @@ -6491,6 +8228,12 @@ dependencies = [ "winapi", ] +[[package]] +name = "quick-error" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0" + [[package]] name = "quick-error" version = "2.0.1" @@ -6544,6 +8287,18 @@ dependencies = [ "memchr", ] +[[package]] +name = "quick_cache" +version = "0.6.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a3db184a8b66cfe87f0263a1de147a6b554c864d1767c6f7fa4eb0e5497b565" +dependencies = [ + "ahash 0.8.12", + "equivalent", + "hashbrown 0.16.1", + "parking_lot", +] + [[package]] name = "quinn" version = "0.11.9" @@ -6702,6 +8457,33 @@ version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + +[[package]] +name = "rand_xorshift" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a" +dependencies = [ + "rand_core 0.9.5", +] + +[[package]] +name = "rand_xoshiro" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f97cdb2a36ed4183de61b2f824cc45c9f1037f28afe0a322e9fff4c108b5aaa" +dependencies = [ + "rand_core 0.6.4", +] + [[package]] name = "rangemap" version = "1.7.1" @@ -6752,7 +8534,7 @@ dependencies = [ "avif-serialize", "imgref", "loop9", - "quick-error", + "quick-error 2.0.1", "rav1e", "rayon", "rgb", @@ -7014,6 +8796,49 @@ dependencies = [ "bytecheck", ] +[[package]] +name = "reqwest" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" +dependencies = [ + "base64", + "bytes", + "futures-core", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "rustls", + "rustls-pki-types", + "rustls-platform-verifier", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams 0.5.0", + "web-sys", +] + +[[package]] +name = "resolv-conf" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e061d1b48cb8d38042de4ae0a7a6401009d6143dc80d2e2d6f31f0bdd6470c7" + [[package]] name = "resvg" version = "0.45.1" @@ -7320,6 +9145,7 @@ version = "0.23.40" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" dependencies = [ + "log", "once_cell", "ring", "rustls-pki-types", @@ -7359,6 +9185,33 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rustls-platform-verifier" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0" +dependencies = [ + "core-foundation 0.10.0", + "core-foundation-sys", + "jni", + "log", + "once_cell", + "rustls", + "rustls-native-certs", + "rustls-platform-verifier-android", + "rustls-webpki", + "security-framework", + "security-framework-sys", + "webpki-root-certs", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls-platform-verifier-android" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" + [[package]] name = "rustls-webpki" version = "0.103.13" @@ -7376,6 +9229,18 @@ version = "1.0.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" +[[package]] +name = "rusty-fork" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc6bf79ff24e648f6da1f8d1f011e9cac26491b619e6b9280f2b47f1774e6ee2" +dependencies = [ + "fnv", + "quick-error 1.2.3", + "tempfile", + "wait-timeout", +] + [[package]] name = "rustybuzz" version = "0.14.1" @@ -7525,6 +9390,16 @@ dependencies = [ "libc", ] +[[package]] +name = "seize" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b55fb86dfd3a2f5f76ea78310a88f96c4ea21a3031f8d212443d56123fd0521" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + [[package]] name = "self_cell" version = "1.2.2" @@ -7537,6 +9412,12 @@ version = "1.0.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" +[[package]] +name = "send_wrapper" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd0b0ec5f1c1ca621c432a25813d8d60c88abe6d3e08a3eb9cf37d97a0fe3d73" + [[package]] name = "serde" version = "1.0.228" @@ -7547,6 +9428,41 @@ dependencies = [ "serde_derive", ] +[[package]] +name = "serde_bytes" +version = "0.11.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" +dependencies = [ + "serde", + "serde_core", +] + +[[package]] +name = "serde_columnar" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a16e404f17b16d0273460350e29b02d76ba0d70f34afdc9a4fa034c97d6c6eb" +dependencies = [ + "itertools 0.11.0", + "postcard", + "serde", + "serde_columnar_derive", + "thiserror 1.0.69", +] + +[[package]] +name = "serde_columnar_derive" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "45958fce4903f67e871fbf15ac78e289269b21ebd357d6fecacdba233629112e" +dependencies = [ + "darling 0.20.11", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "serde_core" version = "1.0.228" @@ -7668,6 +9584,16 @@ dependencies = [ "unsafe-libyaml", ] +[[package]] +name = "serdect" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66cf8fedced2fcf12406bcb34223dffb92eaf34908ede12fed414c82b7f00b3e" +dependencies = [ + "base16ct", + "serde", +] + [[package]] name = "sfsymbols" version = "0.2.1" @@ -7753,12 +9679,28 @@ dependencies = [ "libc", ] +[[package]] +name = "signature" +version = "3.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28d567dcbaf0049cb8ac2608a76cd95ff9e4412e1899d389ee400918ca7537f5" + [[package]] name = "simd-adler32" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" +[[package]] +name = "simd_cesu8" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94f90157bb87cddf702797c5dadfa0be7d266cdf49e22da2fcaa32eff75b2c33" +dependencies = [ + "rustc_version", + "simdutf8", +] + [[package]] name = "simd_helpers" version = "0.1.0" @@ -7774,6 +9716,15 @@ version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" +[[package]] +name = "simple-dns" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a75cbde1bf934313596a004973e462f9a82caa814dcf1a5f507bdf51597eeb4" +dependencies = [ + "bitflags 2.13.0", +] + [[package]] name = "simplecss" version = "0.2.2" @@ -7789,6 +9740,16 @@ version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" +[[package]] +name = "sized-chunks" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "16d69225bde7a69b235da73377861095455d298f2b970996eec25ddbb42b3d1e" +dependencies = [ + "bitmaps", + "typenum", +] + [[package]] name = "sketches-ddsketch" version = "0.4.0" @@ -7848,6 +9809,9 @@ name = "smallvec" version = "1.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" +dependencies = [ + "serde", +] [[package]] name = "smol" @@ -7882,6 +9846,23 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "sorted-index-buffer" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea06cc588e43c632923a55450401b8f25e628131571d4e1baea1bdfdb2b5ed06" + +[[package]] +name = "spez" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c87e960f4dca2788eeb86bbdde8dd246be8948790b7618d656e68f9b720a86e8" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "spin" version = "0.9.8" @@ -7891,6 +9872,12 @@ dependencies = [ "lock_api", ] +[[package]] +name = "spin" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d5fe4ccb98d9c292d56fec89a5e07da7fc4cf0dc11e156b41793132775d3e591" + [[package]] name = "spirv" version = "0.3.0+sdk-1.3.268.0" @@ -7900,6 +9887,16 @@ dependencies = [ "bitflags 2.13.0", ] +[[package]] +name = "spki" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d9efca8738c78ee9484207732f728b1ef517bbb1833d6fc0879ca898a522f6f" +dependencies = [ + "base64ct", + "der", +] + [[package]] name = "sqlite-wasm-rs" version = "0.5.5" @@ -8039,6 +10036,15 @@ dependencies = [ "strum_macros 0.27.2", ] +[[package]] +name = "strum" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9628de9b8791db39ceda2b119bbe13134770b56c138ec1d3af810d045c04f9bd" +dependencies = [ + "strum_macros 0.28.0", +] + [[package]] name = "strum_macros" version = "0.26.4" @@ -8064,6 +10070,18 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "strum_macros" +version = "0.28.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "subsetter" version = "0.2.6" @@ -8299,6 +10317,17 @@ dependencies = [ "system-configuration-sys", ] +[[package]] +name = "system-configuration" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b" +dependencies = [ + "bitflags 2.13.0", + "core-foundation 0.9.4", + "system-configuration-sys", +] + [[package]] name = "system-configuration-sys" version = "0.6.0" @@ -8321,6 +10350,12 @@ dependencies = [ "slotmap", ] +[[package]] +name = "tagptr" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" + [[package]] name = "take-until" version = "0.2.0" @@ -8351,8 +10386,8 @@ dependencies = [ "itertools 0.14.0", "levenshtein_automata", "log", - "lru", - "lz4_flex", + "lru 0.16.4", + "lz4_flex 0.13.1", "measure_time", "memmap2", "once_cell", @@ -8607,7 +10642,7 @@ dependencies = [ "fax", "flate2", "half", - "quick-error", + "quick-error 2.0.1", "weezl", "zune-jpeg 0.5.15", ] @@ -8621,7 +10656,9 @@ dependencies = [ "deranged", "itoa", "js-sys", + "libc", "num-conv", + "num_threads", "powerfmt", "serde_core", "time-core", @@ -8738,10 +10775,23 @@ dependencies = [ "libc", "mio", "pin-project-lite", + "signal-hook-registry", "socket2", + "tokio-macros", "windows-sys 0.61.2", ] +[[package]] +name = "tokio-macros" +version = "2.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "tokio-rustls" version = "0.26.4" @@ -8764,6 +10814,18 @@ dependencies = [ "tokio", ] +[[package]] +name = "tokio-stream" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", + "tokio-util", +] + [[package]] name = "tokio-util" version = "0.7.18" @@ -8773,10 +10835,34 @@ dependencies = [ "bytes", "futures-core", "futures-sink", + "futures-util", "pin-project-lite", "tokio", ] +[[package]] +name = "tokio-websockets" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dad543404f98bfc969aeb71994105c592acfc6c43323fddcd016bb208d1c65cb" +dependencies = [ + "base64", + "bytes", + "futures-core", + "futures-sink", + "getrandom 0.4.2", + "http", + "httparse", + "rand 0.10.1", + "ring", + "rustls-pki-types", + "sha1_smol", + "simdutf8", + "tokio", + "tokio-rustls", + "tokio-util", +] + [[package]] name = "toml" version = "0.8.23" @@ -8884,6 +10970,24 @@ dependencies = [ "tower-service", ] +[[package]] +name = "tower-http" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" +dependencies = [ + "bitflags 2.13.0", + "bytes", + "futures-util", + "http", + "http-body", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", + "url", +] + [[package]] name = "tower-layer" version = "0.3.3" @@ -8902,6 +11006,7 @@ version = "0.1.44" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" dependencies = [ + "log", "pin-project-lite", "tracing-attributes", "tracing-core", @@ -9049,6 +11154,15 @@ dependencies = [ "syntect", ] +[[package]] +name = "twox-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" +dependencies = [ + "rand 0.9.4", +] + [[package]] name = "typed-arena" version = "2.0.2" @@ -9365,6 +11479,12 @@ dependencies = [ "unicode-math-class", ] +[[package]] +name = "ucd-trie" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971" + [[package]] name = "uds_windows" version = "1.2.1" @@ -9376,6 +11496,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "unarray" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94" + [[package]] name = "unic-langid" version = "0.9.6" @@ -9531,6 +11657,16 @@ version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" +[[package]] +name = "universal-hash" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" +dependencies = [ + "crypto-common 0.1.7", + "subtle", +] + [[package]] name = "unsafe-libyaml" version = "0.2.11" @@ -9685,6 +11821,43 @@ version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" +[[package]] +name = "vergen" +version = "9.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b849a1f6d8639e8de261e81ee0fc881e3e3620db1af9f2e0da015d4382ceaf75" +dependencies = [ + "anyhow", + "derive_builder", + "rustversion", + "vergen-lib", +] + +[[package]] +name = "vergen-gitcl" +version = "9.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ff3b5300a085d6bcd8fc96a507f706a28ae3814693236c9b409db71a1d15b9" +dependencies = [ + "anyhow", + "derive_builder", + "rustversion", + "time", + "vergen", + "vergen-lib", +] + +[[package]] +name = "vergen-lib" +version = "9.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b34a29ba7e9c59e62f229ae1932fb1b8fb8a6fdcc99215a641913f5f5a59a569" +dependencies = [ + "anyhow", + "derive_builder", + "rustversion", +] + [[package]] name = "version_check" version = "0.9.5" @@ -9711,6 +11884,15 @@ dependencies = [ "libc", ] +[[package]] +name = "wait-timeout" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11" +dependencies = [ + "libc", +] + [[package]] name = "waker-fn" version = "1.2.0" @@ -9850,13 +12032,26 @@ dependencies = [ "web-sys", ] +[[package]] +name = "wasm-streams" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + [[package]] name = "wasmi" version = "0.51.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bb321403ce594274827657a908e13d1d9918aa02257b8bf8391949d9764023ff" dependencies = [ - "spin", + "spin 0.9.8", "wasmi_collections", "wasmi_core", "wasmi_ir", @@ -10025,6 +12220,24 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "webpki-root-certs" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "webpki-roots" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52f5ee44c96cf55f1b349600768e3ece3a8f26010c05265ab73f945bb1a2eb9d" +dependencies = [ + "rustls-pki-types", +] + [[package]] name = "weezl" version = "0.1.12" @@ -10056,6 +12269,12 @@ dependencies = [ "winsafe", ] +[[package]] +name = "widestring" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72069c3113ab32ab29e5584db3c6ec55d416895e60715417b5b883a357c3e471" + [[package]] name = "winapi" version = "0.3.9" @@ -10103,11 +12322,23 @@ version = "0.61.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9babd3a767a4c1aef6900409f85f5d53ce2544ccdfaa86dad48c91782c6d6893" dependencies = [ - "windows-collections", + "windows-collections 0.2.0", "windows-core 0.61.2", - "windows-future", + "windows-future 0.2.1", "windows-link 0.1.3", - "windows-numerics", + "windows-numerics 0.2.0", +] + +[[package]] +name = "windows" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580" +dependencies = [ + "windows-collections 0.3.2", + "windows-core 0.62.2", + "windows-future 0.3.2", + "windows-numerics 0.3.1", ] [[package]] @@ -10120,7 +12351,7 @@ dependencies = [ "rayon", "thiserror 2.0.18", "windows 0.61.3", - "windows-future", + "windows-future 0.2.1", ] [[package]] @@ -10132,6 +12363,15 @@ dependencies = [ "windows-core 0.61.2", ] +[[package]] +name = "windows-collections" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b2d95af1a8a14a3c7367e1ed4fc9c20e0a26e79551b1454d72583c97cc6610" +dependencies = [ + "windows-core 0.62.2", +] + [[package]] name = "windows-core" version = "0.57.0" @@ -10178,7 +12418,18 @@ checksum = "fc6a41e98427b19fe4b73c550f060b59fa592d7d686537eebf9385621bfbad8e" dependencies = [ "windows-core 0.61.2", "windows-link 0.1.3", - "windows-threading", + "windows-threading 0.1.0", +] + +[[package]] +name = "windows-future" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb" +dependencies = [ + "windows-core 0.62.2", + "windows-link 0.2.1", + "windows-threading 0.2.1", ] [[package]] @@ -10247,6 +12498,16 @@ dependencies = [ "windows-link 0.1.3", ] +[[package]] +name = "windows-numerics" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26" +dependencies = [ + "windows-core 0.62.2", + "windows-link 0.2.1", +] + [[package]] name = "windows-registry" version = "0.4.0" @@ -10269,6 +12530,17 @@ dependencies = [ "windows-strings 0.4.2", ] +[[package]] +name = "windows-registry" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720" +dependencies = [ + "windows-link 0.2.1", + "windows-result 0.4.1", + "windows-strings 0.5.1", +] + [[package]] name = "windows-result" version = "0.1.2" @@ -10425,6 +12697,15 @@ dependencies = [ "windows-link 0.1.3", ] +[[package]] +name = "windows-threading" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3949bd5b99cafdf1c7ca86b43ca564028dfe27d66958f2470940f73d86d75b37" +dependencies = [ + "windows-link 0.2.1", +] + [[package]] name = "windows_aarch64_gnullvm" version = "0.48.5" @@ -10700,6 +12981,21 @@ dependencies = [ "wasmparser 0.244.0", ] +[[package]] +name = "wmi" +version = "0.18.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c81b85c57a57500e56669586496bf2abd5cf082b9d32995251185d105208b64" +dependencies = [ + "chrono", + "futures", + "log", + "serde", + "thiserror 2.0.18", + "windows 0.62.2", + "windows-core 0.62.2", +] + [[package]] name = "workspace-hack" version = "0.1.0" @@ -10731,6 +13027,25 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" +[[package]] +name = "ws_stream_wasm" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c173014acad22e83f16403ee360115b38846fe754e735c5d9d3803fe70c6abc" +dependencies = [ + "async_io_stream", + "futures", + "js-sys", + "log", + "pharos", + "rustc_version", + "send_wrapper", + "thiserror 2.0.18", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + [[package]] name = "wyz" version = "0.5.1" @@ -10843,6 +13158,12 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9cc00251562a284751c9973bace760d86c0276c471b4be569fe6b068ee97a56" +[[package]] +name = "xml-rs" +version = "0.8.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ae8337f8a065cfc972643663ea4279e04e7256de865aa66fe25cec5fb912d3f" + [[package]] name = "xml5ever" version = "0.18.1" @@ -10860,6 +13181,15 @@ version = "0.13.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4" +[[package]] +name = "xmltree" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7d8a75eaf6557bb84a65ace8609883db44a29951042ada9b393151532e41fcb" +dependencies = [ + "xml-rs", +] + [[package]] name = "xmlwriter" version = "0.1.0" @@ -10887,6 +13217,12 @@ dependencies = [ "linked-hash-map", ] +[[package]] +name = "yansi" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" + [[package]] name = "yazi" version = "0.2.1" @@ -11086,7 +13422,7 @@ dependencies = [ "serde_json", "serde_urlencoded", "sync_wrapper", - "system-configuration", + "system-configuration 0.6.1", "tokio", "tokio-rustls", "tokio-socks", @@ -11096,7 +13432,7 @@ dependencies = [ "url", "wasm-bindgen", "wasm-bindgen-futures", - "wasm-streams", + "wasm-streams 0.4.2", "web-sys", "windows-registry 0.4.0", ] diff --git a/Cargo.toml b/Cargo.toml index 7cdeca66..3ab5cbee 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -12,6 +12,7 @@ publish = false [workspace.dependencies] anyhow = "1.0.102" +async-channel = "2" clap = { version = "4.5", features = ["derive"] } cosmic-text = { version = "0.14.2", default-features = false, features = [ "shape-run-cache", @@ -23,6 +24,7 @@ docxide-pdf = { version = "0.15.1", default-features = false, git = "https://git fff-search = "0.8.2-nightly.ffb39a8" flowstate-document = { version = "0.1.0", path = "crates/flowstate-document" } flowstate-docx = { version = "0.1.0", path = "crates/flowstate-docx" } +flowstate-collab = { version = "0.1.0", path = "crates/flowstate-collab" } flowstate-flow = { version = "0.1.0", path = "crates/flowstate-flow" } flowstate-tub = { version = "0.1.0", path = "crates/flowstate-tub" } gpui-flowtext = { version = "0.1.0", path = "crates/gpui-flowtext" } @@ -33,11 +35,19 @@ gpui-symbols = { version = "0.6.1", features = ["component"] } hotpath = "0.16.1" ignore = "0.4.25" imagesize = "0.13" +iroh = "1.0.0-rc.1" +iroh-gossip = "0.100" +iroh-tickets = "1.0.0-rc.1" lopdf = "0.40.0" +loro = "1.13" mimalloc = "0.1.52" mitex = "0.2" notify = "8.2.0" +n0-future = "0.3" +postcard = { version = "1", features = ["use-std"] } +proptest = "1" quick-xml = "0.36.2" +rand = "0.9" rdocx = "0.1.2" rdocx-opc = "0.1.2" rdocx-oxml = "0.1.2" @@ -48,7 +58,9 @@ serde = { version = "1.0.228", features = ["derive"] } serde_json = "1.0.150" tantivy = "0.26.1" tempfile = "3" +tokio = { version = "1", features = ["rt-multi-thread", "macros", "time", "sync"] } toml = "1.1.2" +twox-hash = "2" typst = "0.14" typst-assets = "0.14" typst-library = "0.14" diff --git a/crates/flowstate-collab/Cargo.toml b/crates/flowstate-collab/Cargo.toml new file mode 100644 index 00000000..62a7b68d --- /dev/null +++ b/crates/flowstate-collab/Cargo.toml @@ -0,0 +1,27 @@ +[package] +name = "flowstate-collab" +version.workspace = true +edition.workspace = true +publish.workspace = true + +[dependencies] +anyhow = { workspace = true } +async-channel = { workspace = true } +gpui-flowtext = { workspace = true } +iroh = { workspace = true } +iroh-gossip = { workspace = true } +iroh-tickets = { workspace = true } +loro = { workspace = true } +n0-future = { workspace = true } +postcard = { workspace = true } +rand = { workspace = true } +serde = { workspace = true } +tokio = { workspace = true } +twox-hash = { workspace = true } +uuid = { workspace = true } + +[dev-dependencies] +proptest = { workspace = true } + +[lints] +workspace = true diff --git a/crates/flowstate-collab/src/binding.rs b/crates/flowstate-collab/src/binding.rs new file mode 100644 index 00000000..2ed3923c --- /dev/null +++ b/crates/flowstate-collab/src/binding.rs @@ -0,0 +1,208 @@ +//! Local flowtext identity bindings for shared CRDT containers. + +use std::collections::HashMap; + +use anyhow::{Context as _, Result, bail}; +use gpui_flowtext::{Block, BlockId, Document, ParagraphId}; +use loro::{Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroText, ValueOrContainer}; + +use crate::schema::BLOCKS; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum BlockKind { + Paragraph, + Image, + Equation, + Table, +} + +#[derive(Clone, Debug)] +pub struct BindingRow { + pub map: LoroMap, + pub text: Option, + pub kind: BlockKind, + pub block_id: BlockId, + pub paragraph_id: Option, + pub version: u64, +} + +#[derive(Clone, Debug, Default)] +pub struct DocBinding { + pub rows: Vec, + pub by_paragraph: HashMap, + pub by_block: HashMap, + pub by_container: HashMap, +} + +impl DocBinding { + pub fn build(doc: &LoroDoc, document: &Document) -> Result { + let blocks = doc.get_movable_list(BLOCKS); + if blocks.len() != document.blocks.len() { + bail!( + "binding row count mismatch: loro has {} rows, document has {} blocks", + blocks.len(), + document.blocks.len() + ); + } + + let mut binding = Self::default(); + let mut paragraph_ix = 0; + for (row_ix, block) in document.blocks.iter().enumerate() { + let map = map_at(&blocks, row_ix)?; + let kind = BlockKind::from_block(block); + let paragraph_id = if matches!(kind, BlockKind::Paragraph) { + let id = document + .ids + .paragraph_ids + .get(paragraph_ix) + .copied() + .context("document is missing a paragraph id for a paragraph block")?; + paragraph_ix += 1; + Some(id) + } else { + None + }; + let block_id = document + .ids + .block_ids + .get(row_ix) + .copied() + .context("document is missing a block id for a block row")?; + let text = if matches!(kind, BlockKind::Paragraph) { + Some(text_child(&map)?) + } else { + None + }; + binding.push_row(BindingRow { + map, + text, + kind, + block_id, + paragraph_id, + version: block_version(block), + }); + } + binding.assert_consistent(doc, document); + Ok(binding) + } + + pub fn assert_consistent(&self, doc: &LoroDoc, document: &Document) { + let blocks = doc.get_movable_list(BLOCKS); + debug_assert_eq!(self.rows.len(), blocks.len()); + debug_assert_eq!(self.rows.len(), document.blocks.len()); + + let mut paragraph_ix = 0; + for (row_ix, row) in self.rows.iter().enumerate() { + let Some(block) = document.blocks.get(row_ix) else { + debug_assert!(false, "binding row has no matching document block"); + continue; + }; + debug_assert_eq!(row.kind, BlockKind::from_block(block)); + debug_assert_eq!(document.ids.block_ids.get(row_ix).copied(), Some(row.block_id)); + debug_assert_eq!(row.version, block_version(block)); + if matches!(row.kind, BlockKind::Paragraph) { + debug_assert_eq!(document.ids.paragraph_ids.get(paragraph_ix).copied(), row.paragraph_id); + paragraph_ix += 1; + } else { + debug_assert!(row.paragraph_id.is_none()); + } + } + } + + pub fn push_row(&mut self, row: BindingRow) { + let ix = self.rows.len(); + self.index_row(ix, &row); + self.rows.push(row); + } + + pub fn insert_row(&mut self, ix: usize, row: BindingRow) { + let ix = ix.min(self.rows.len()); + self.rows.insert(ix, row); + self.rebuild_indexes(); + } + + pub fn remove_row(&mut self, ix: usize) -> Option { + if ix >= self.rows.len() { + return None; + } + let row = self.rows.remove(ix); + self.rebuild_indexes(); + Some(row) + } + + pub fn move_row(&mut self, from: usize, to: usize) { + if from >= self.rows.len() || from == to { + return; + } + let row = self.rows.remove(from); + self.rows.insert(to.min(self.rows.len()), row); + self.rebuild_indexes(); + } + + pub fn refresh_versions(&mut self, document: &Document) { + for (row, block) in self.rows.iter_mut().zip(document.blocks.iter()) { + row.version = block_version(block); + } + } + + pub fn rebuild_indexes(&mut self) { + self.by_paragraph.clear(); + self.by_block.clear(); + self.by_container.clear(); + for ix in 0..self.rows.len() { + let row = self.rows[ix].clone(); + self.index_row(ix, &row); + } + } + + fn index_row(&mut self, ix: usize, row: &BindingRow) { + self.by_block.insert(row.block_id, ix); + self.by_container.insert(row.map.id(), ix); + if let Some(text) = &row.text { + self.by_container.insert(text.id(), ix); + } + if let Some(paragraph_id) = row.paragraph_id { + self.by_paragraph.insert(paragraph_id, ix); + } + } +} + +#[must_use] +pub const fn block_version(block: &Block) -> u64 { + match block { + Block::Paragraph(paragraph) => paragraph.version, + Block::Image(image) => image.version, + Block::Equation(equation) => equation.version, + Block::Table(table) => table.version, + } +} + +impl BlockKind { + #[must_use] + pub const fn from_block(block: &Block) -> Self { + match block { + Block::Paragraph(_) => Self::Paragraph, + Block::Image(_) => Self::Image, + Block::Equation(_) => Self::Equation, + Block::Table(_) => Self::Table, + } + } +} + +fn map_at(blocks: &LoroMovableList, ix: usize) -> Result { + match blocks.get(ix) { + Some(ValueOrContainer::Container(Container::Map(map))) => Ok(map), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("binding row {ix} is not a map container") + }, + } +} + +fn text_child(map: &LoroMap) -> Result { + match map.get(crate::schema::TEXT) { + Some(ValueOrContainer::Container(Container::Text(text))) => Ok(text), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("paragraph block map is missing its text container") + }, + } +} diff --git a/crates/flowstate-collab/src/ids.rs b/crates/flowstate-collab/src/ids.rs new file mode 100644 index 00000000..df1df075 --- /dev/null +++ b/crates/flowstate-collab/src/ids.rs @@ -0,0 +1,100 @@ +use std::{fmt, str::FromStr}; + +use serde::{Deserialize, Serialize}; + +pub type PeerId = iroh::EndpointId; + +pub const SESSION_ID_LEN: usize = 32; + +pub const PALETTE: [u32; 8] = [ + 0x3b82f6, 0xef4444, 0x22c55e, 0xf59e0b, 0x8b5cf6, 0x06b6d4, 0xec4899, 0x84cc16, +]; + +#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize, Deserialize)] +pub struct SessionId([u8; SESSION_ID_LEN]); + +impl SessionId { + #[must_use] + pub fn new() -> Self { + Self(rand::random()) + } + + #[must_use] + pub const fn from_bytes(bytes: [u8; SESSION_ID_LEN]) -> Self { + Self(bytes) + } + + #[must_use] + pub const fn as_bytes(&self) -> &[u8; SESSION_ID_LEN] { + &self.0 + } + + #[must_use] + pub fn color_index_for_peer(peer: &PeerId) -> usize { + let bytes = peer.as_bytes(); + let mut hash = 0usize; + for byte in bytes.iter().take(8) { + hash = hash.wrapping_mul(257).wrapping_add(usize::from(*byte)); + } + hash % PALETTE.len() + } +} + +impl Default for SessionId { + fn default() -> Self { + Self::new() + } +} + +impl fmt::Display for SessionId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + for byte in self.0 { + write!(f, "{byte:02x}")?; + } + Ok(()) + } +} + +impl FromStr for SessionId { + type Err = SessionIdParseError; + + fn from_str(value: &str) -> Result { + if value.len() != SESSION_ID_LEN * 2 { + return Err(SessionIdParseError); + } + + let mut bytes = [0; SESSION_ID_LEN]; + for (ix, chunk) in value.as_bytes().chunks_exact(2).enumerate() { + let text = std::str::from_utf8(chunk).map_err(|_| SessionIdParseError)?; + bytes[ix] = u8::from_str_radix(text, 16).map_err(|_| SessionIdParseError)?; + } + Ok(Self(bytes)) + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct SessionIdParseError; + +impl fmt::Display for SessionIdParseError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("expected a 64-character lowercase hex session id") + } +} + +impl std::error::Error for SessionIdParseError {} + +#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize, Deserialize)] +pub struct BlobId(pub u128); + +impl BlobId { + #[must_use] + pub fn new() -> Self { + Self(uuid::Uuid::new_v4().as_u128()) + } +} + +impl Default for BlobId { + fn default() -> Self { + Self::new() + } +} diff --git a/crates/flowstate-collab/src/lib.rs b/crates/flowstate-collab/src/lib.rs new file mode 100644 index 00000000..d930ae3f --- /dev/null +++ b/crates/flowstate-collab/src/lib.rs @@ -0,0 +1,21 @@ +//! Peer-to-peer collaboration support for Flowstate rich-text documents. +//! +//! This crate contains the GPUI-free collaboration core: CRDT projection, +//! transport protocol types, and networking state. Application/UI integration +//! lives in `crates/flowstate/src/collab`. + +pub mod binding; +pub mod ids; +pub mod local_apply; +pub mod net; +pub mod presence; +pub mod projection; +pub mod proto_direct; +pub mod proto_gossip; +pub mod remote_apply; +pub mod schema; +pub mod self_check; +pub mod ticket; + +pub use ids::{BlobId, SessionId}; +pub use proto_gossip::{DIRECT_ALPN, PROTOCOL_VERSION}; diff --git a/crates/flowstate-collab/src/local_apply.rs b/crates/flowstate-collab/src/local_apply.rs new file mode 100644 index 00000000..e7b11fc1 --- /dev/null +++ b/crates/flowstate-collab/src/local_apply.rs @@ -0,0 +1,639 @@ +//! Translation from flowtext canonical operations into Loro operations. + +use anyhow::{Context as _, Result, bail}; +use gpui_flowtext::{ + Block, BlockId, CanonicalOperation, Document, DocumentSpan, Paragraph, ParagraphId, RunStyles, TextRun, paragraph_text, paragraph_text_len, +}; +use loro::{CommitOptions, LoroDoc, LoroMap, LoroMovableList, LoroText, LoroValue, TextDelta, UpdateOptions, ValueOrContainer, cursor::PosType}; + +use crate::{ + binding::{BindingRow, BlockKind, DocBinding, block_version}, + projection::{insert_object_container, insert_paragraph_container, replace_blocks_from_document}, + schema::{ + BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_TABLE, MARK_HIGHLIGHT, MARK_SEMANTIC, MARK_STRIKE, MARK_UNDERLINE, REV, + STYLE, apply_mark_intervals, decode_paragraph_style, encode_paragraph_style, mark_intervals_from_runs, payload_from_block, + run_styles_from_attrs, set_run_styles_utf8, unmark_utf8, + }, +}; + +pub struct LocalApplier<'s> { + pub doc: &'s LoroDoc, + pub binding: &'s mut DocBinding, +} + +impl LocalApplier<'_> { + pub fn apply(&mut self, document: &Document, ops: &[CanonicalOperation]) -> Result<()> { + for op in ops { + self.apply_one(document, op)?; + } + if !ops.is_empty() { + self + .doc + .commit_with(CommitOptions::new().origin("local-edit")); + } + self.binding.refresh_versions(document); + self.binding.assert_consistent(self.doc, document); + Ok(()) + } + + fn apply_one(&mut self, document: &Document, op: &CanonicalOperation) -> Result<()> { + match op { + CanonicalOperation::InsertText { + paragraph, + byte, + text, + styles, + } => self.insert_text(*paragraph, *byte, text, *styles), + CanonicalOperation::DeleteRange { + start_paragraph, + start_byte, + end_paragraph, + end_byte, + } => self.delete_range(*start_paragraph, *start_byte, *end_paragraph, *end_byte), + CanonicalOperation::SplitParagraph { + paragraph, + byte, + new_paragraph, + } => self.split_paragraph(document, *paragraph, *byte, *new_paragraph), + CanonicalOperation::JoinParagraphs { first, second } => self.join_paragraphs(*first, *second), + CanonicalOperation::SetParagraphStyle { paragraph, style } => { + let row = self.row_for_paragraph(*paragraph)?; + row.map.insert(STYLE, encode_paragraph_style(*style))?; + Ok(()) + }, + CanonicalOperation::SetRunStyles { paragraph, range, styles } => { + let row = self.row_for_paragraph(*paragraph)?; + let text = row + .text + .as_ref() + .context("paragraph row is missing its LoroText")?; + set_run_styles_utf8(text, range.clone(), *styles)?; + Ok(()) + }, + CanonicalOperation::InsertBlock { block, block_ix } => self.insert_block(document, *block, *block_ix), + CanonicalOperation::DeleteBlock { block } => self.delete_block(*block), + CanonicalOperation::MoveBlock { block, new_block_ix } => self.move_block(*block, *new_block_ix), + CanonicalOperation::ReplaceParagraphSpan { + start_paragraph, + before, + after, + } => self.replace_paragraph_span(document, *start_paragraph, before, after), + CanonicalOperation::ReplaceBlock { block } => self.replace_block(document, *block), + CanonicalOperation::ReplaceDocument => self.replace_document(document), + } + } + + fn insert_text(&mut self, paragraph: ParagraphId, byte: usize, value: &str, styles: RunStyles) -> Result<()> { + let row = self.row_for_paragraph(paragraph)?; + let text = row + .text + .as_ref() + .context("paragraph row is missing its LoroText")?; + text.insert_utf8(byte, value)?; + if styles != RunStyles::default() { + set_run_styles_utf8(text, byte..byte + value.len(), styles)?; + } + Ok(()) + } + + fn delete_range(&mut self, start_paragraph: ParagraphId, start: usize, end_paragraph: ParagraphId, end: usize) -> Result<()> { + let start_ix = self.row_ix_for_paragraph(start_paragraph)?; + let end_ix = self.row_ix_for_paragraph(end_paragraph)?; + if start_ix == end_ix { + return self.delete_text_at_row(start_ix, start, end); + } + if start_ix > end_ix { + bail!("cross-paragraph delete start row is after end row"); + } + + let start_text = self.paragraph_text_at_row(start_ix)?; + let end_text = self.paragraph_text_at_row(end_ix)?; + let tail = end_text.slice_delta(end, end_text.len_utf8(), PosType::Bytes)?; + let start_len = start_text.len_utf8(); + if start < start_len { + start_text.delete_utf8(start, start_len - start)?; + } + + let blocks = self.blocks(); + for row_ix in ((start_ix + 1)..end_ix).rev() { + blocks.delete(row_ix, 1)?; + self + .binding + .remove_row(row_ix) + .context("DocBinding row disappeared during cross-paragraph delete")?; + } + + insert_delta_at(&start_text, start, &tail)?; + blocks.delete(start_ix + 1, 1)?; + self + .binding + .remove_row(start_ix + 1) + .context("DocBinding end row disappeared during cross-paragraph delete")?; + Ok(()) + } + + fn split_paragraph(&mut self, document: &Document, paragraph: ParagraphId, byte: usize, new_paragraph: ParagraphId) -> Result<()> { + let row_ix = self.row_ix_for_paragraph(paragraph)?; + let row = self + .binding + .rows + .get(row_ix) + .context("DocBinding paragraph index is out of bounds")?; + let style = decode_paragraph_style(map_i64(&row.map, STYLE)?); + let text = row + .text + .as_ref() + .context("paragraph row is missing its LoroText")? + .clone(); + let tail = text.slice_delta(byte, text.len_utf8(), PosType::Bytes)?; + if byte < text.len_utf8() { + text.delete_utf8(byte, text.len_utf8() - byte)?; + } + + let insert_ix = row_ix + 1; + let block_id = document + .ids + .block_ids + .get(insert_ix) + .copied() + .context("post-split document is missing the new block id")?; + let version = document + .blocks + .get(insert_ix) + .map(block_version) + .context("post-split document is missing the new block")?; + let blocks = self.blocks(); + let (map, new_text) = insert_paragraph_container(&blocks, insert_ix, style, &[], "")?; + insert_delta_at(&new_text, 0, &tail)?; + self.binding.insert_row( + insert_ix, + BindingRow { + map, + text: Some(new_text), + kind: BlockKind::Paragraph, + block_id, + paragraph_id: Some(new_paragraph), + version, + }, + ); + Ok(()) + } + + fn join_paragraphs(&mut self, first: ParagraphId, second: ParagraphId) -> Result<()> { + let first_ix = self.row_ix_for_paragraph(first)?; + let first_len = self.paragraph_text_at_row(first_ix)?.len_utf8(); + self.delete_range(first, first_len, second, 0) + } + + fn insert_block(&mut self, document: &Document, block_id: BlockId, block_ix: usize) -> Result<()> { + let block = document + .blocks + .get(block_ix) + .context("inserted block index is out of bounds in the post-edit document")?; + match block { + Block::Paragraph(paragraph) => { + let paragraph_ix = document_paragraph_ix_for_block(document, block_ix)?; + let paragraph_id = document + .ids + .paragraph_ids + .get(paragraph_ix) + .copied() + .context("inserted paragraph block is missing a paragraph id")?; + let text = paragraph_text(document, paragraph_ix); + self.insert_paragraph_row( + block_ix, + block_id, + paragraph_id, + paragraph.style, + ¶graph.runs, + &text, + block_version(block), + ) + }, + Block::Image(_) | Block::Equation(_) | Block::Table(_) => { + let blocks = self.blocks(); + let map = insert_object_container(&blocks, block_ix, block, document)?; + self.binding.insert_row( + block_ix, + BindingRow { + map, + text: None, + kind: BlockKind::from_block(block), + block_id, + paragraph_id: None, + version: block_version(block), + }, + ); + Ok(()) + }, + } + } + + fn delete_block(&mut self, block: BlockId) -> Result<()> { + let row_ix = self.row_ix_for_block(block)?; + self.blocks().delete(row_ix, 1)?; + self + .binding + .remove_row(row_ix) + .context("DocBinding block row disappeared during block delete")?; + Ok(()) + } + + fn move_block(&mut self, block: BlockId, new_block_ix: usize) -> Result<()> { + let old_ix = self.row_ix_for_block(block)?; + self.blocks().mov(old_ix, new_block_ix)?; + self.binding.move_row(old_ix, new_block_ix); + Ok(()) + } + + fn replace_paragraph_span( + &mut self, + document: &Document, + start_paragraph: Option, + before: &DocumentSpan, + after: &DocumentSpan, + ) -> Result<()> { + let start = self.replace_span_start_ordinal(start_paragraph, before); + let old_rows = self.paragraph_row_indices(start, before.paragraphs.len())?; + let before_texts = span_paragraph_texts(before)?; + let after_texts = span_paragraph_texts(after)?; + let matched = before.paragraphs.len().min(after.paragraphs.len()); + + for ix in 0..matched { + self.update_matched_paragraph( + old_rows[ix], + &before.paragraphs[ix], + &before_texts[ix], + &after.paragraphs[ix], + &after_texts[ix], + )?; + } + + if after.paragraphs.len() > matched { + let base_insert_ix = if matched > 0 { + old_rows[matched - 1] + 1 + } else { + old_rows + .first() + .copied() + .unwrap_or_else(|| self.row_ix_for_paragraph_insert_ordinal(start)) + }; + for (ix, (paragraph, text)) in after + .paragraphs + .iter() + .zip(after_texts.iter()) + .enumerate() + .skip(matched) + { + let insert_ix = base_insert_ix + ix - matched; + let paragraph_ix = start + ix; + let block_ix = document_block_ix_for_paragraph(document, paragraph_ix)?; + let block_id = document + .ids + .block_ids + .get(block_ix) + .copied() + .context("inserted paragraph span block is missing a block id")?; + let paragraph_id = document + .ids + .paragraph_ids + .get(paragraph_ix) + .copied() + .context("inserted paragraph span block is missing a paragraph id")?; + self.insert_paragraph_row( + insert_ix, + block_id, + paragraph_id, + paragraph.style, + ¶graph.runs, + text, + block_version(&document.blocks[block_ix]), + )?; + } + } + + if before.paragraphs.len() > matched { + let blocks = self.blocks(); + for row_ix in old_rows[matched..].iter().rev().copied() { + blocks.delete(row_ix, 1)?; + self + .binding + .remove_row(row_ix) + .context("DocBinding paragraph span row disappeared during delete")?; + } + } + + self.binding.refresh_versions(document); + self.binding.assert_consistent(self.doc, document); + Ok(()) + } + + fn replace_block(&mut self, document: &Document, block: Option) -> Result<()> { + let row_ix = match block { + Some(block) => self.binding.by_block.get(&block).copied(), + None => self.single_changed_object_row(document), + }; + if let Some(row_ix) = row_ix + && self.replace_block_at(document, row_ix)? + { + return Ok(()); + } + self.replace_document(document) + } + + fn replace_document(&mut self, document: &Document) -> Result<()> { + replace_blocks_from_document(self.doc, document)?; + *self.binding = DocBinding::build(self.doc, document)?; + Ok(()) + } + + fn update_matched_paragraph(&self, row_ix: usize, before: &Paragraph, before_text: &str, after: &Paragraph, after_text: &str) -> Result<()> { + let row = self + .binding + .rows + .get(row_ix) + .context("matched paragraph row is out of bounds")?; + let text = row + .text + .as_ref() + .context("matched paragraph row is missing its LoroText")?; + if before_text != after_text { + text.update(after_text, UpdateOptions::default())?; + } + if before.runs != after.runs { + refresh_marks(text, &after.runs)?; + } + if before.style != after.style { + row.map.insert(STYLE, encode_paragraph_style(after.style))?; + } + Ok(()) + } + + fn replace_block_at(&mut self, document: &Document, row_ix: usize) -> Result { + let Some(block) = document.blocks.get(row_ix) else { + return Ok(false); + }; + let Some(payload) = payload_from_block(block, &document.assets) else { + return Ok(false); + }; + let row = self + .binding + .rows + .get_mut(row_ix) + .context("replacement block row is out of bounds")?; + row.map.insert(KIND, kind_for_payload(&payload))?; + row + .map + .insert(DATA, LoroValue::Binary(postcard::to_stdvec(&payload)?.into()))?; + row + .map + .insert(REV, map_i64_or(&row.map, REV, 0)?.saturating_add(1))?; + row.kind = BlockKind::from_block(block); + row.text = None; + row.paragraph_id = None; + row.version = block_version(block); + self.binding.rebuild_indexes(); + Ok(true) + } + + fn insert_paragraph_row( + &mut self, + row_ix: usize, + block_id: BlockId, + paragraph_id: ParagraphId, + style: gpui_flowtext::ParagraphStyle, + runs: &[TextRun], + text: &str, + version: u64, + ) -> Result<()> { + let blocks = self.blocks(); + let (map, loro_text) = insert_paragraph_container(&blocks, row_ix, style, runs, text)?; + self.binding.insert_row( + row_ix, + BindingRow { + map, + text: Some(loro_text), + kind: BlockKind::Paragraph, + block_id, + paragraph_id: Some(paragraph_id), + version, + }, + ); + Ok(()) + } + + fn single_changed_object_row(&self, document: &Document) -> Option { + let mut changed = None; + for (row_ix, row) in self.binding.rows.iter().enumerate() { + let block = document.blocks.get(row_ix)?; + if matches!(block, Block::Paragraph(_)) || row.version == block_version(block) { + continue; + } + if changed.replace(row_ix).is_some() { + return None; + } + } + changed + } + + fn delete_text_at_row(&self, row_ix: usize, start: usize, end: usize) -> Result<()> { + let text = self.paragraph_text_at_row(row_ix)?; + if end > start { + text.delete_utf8(start, end - start)?; + } + Ok(()) + } + + fn paragraph_text_at_row(&self, row_ix: usize) -> Result { + self + .binding + .rows + .get(row_ix) + .and_then(|row| row.text.clone()) + .context("paragraph row is missing its LoroText") + } + + fn row_for_paragraph(&self, paragraph: ParagraphId) -> Result<&BindingRow> { + let ix = self.row_ix_for_paragraph(paragraph)?; + self + .binding + .rows + .get(ix) + .context("DocBinding paragraph index is out of bounds") + } + + fn row_ix_for_paragraph(&self, paragraph: ParagraphId) -> Result { + self + .binding + .by_paragraph + .get(¶graph) + .copied() + .context("paragraph id is not present in DocBinding") + } + + fn row_ix_for_block(&self, block: BlockId) -> Result { + self + .binding + .by_block + .get(&block) + .copied() + .context("block id is not present in DocBinding") + } + + fn replace_span_start_ordinal(&self, start_paragraph: Option, before: &DocumentSpan) -> usize { + start_paragraph + .and_then(|paragraph| self.binding.by_paragraph.get(¶graph).copied()) + .and_then(|row_ix| self.paragraph_ordinal_for_row(row_ix)) + .unwrap_or(before.start_paragraph) + } + + fn paragraph_ordinal_for_row(&self, target_row: usize) -> Option { + let mut paragraph_ix = 0; + for (row_ix, row) in self.binding.rows.iter().enumerate() { + if row_ix == target_row { + return row.paragraph_id.map(|_| paragraph_ix); + } + if row.paragraph_id.is_some() { + paragraph_ix += 1; + } + } + None + } + + fn paragraph_row_indices(&self, start: usize, count: usize) -> Result> { + let mut rows = Vec::with_capacity(count); + let mut paragraph_ix = 0; + for (row_ix, row) in self.binding.rows.iter().enumerate() { + if row.paragraph_id.is_none() { + continue; + } + if paragraph_ix >= start && rows.len() < count { + rows.push(row_ix); + } + paragraph_ix += 1; + } + if rows.len() != count { + bail!("paragraph span references rows outside DocBinding"); + } + Ok(rows) + } + + fn row_ix_for_paragraph_insert_ordinal(&self, target: usize) -> usize { + let mut paragraph_ix = 0; + for (row_ix, row) in self.binding.rows.iter().enumerate() { + if row.paragraph_id.is_none() { + continue; + } + if paragraph_ix == target { + return row_ix; + } + paragraph_ix += 1; + } + self.binding.rows.len() + } + + fn blocks(&self) -> LoroMovableList { + self.doc.get_movable_list(BLOCKS) + } +} + +fn insert_delta_at(text: &LoroText, byte: usize, delta: &[TextDelta]) -> Result<()> { + let mut cursor = byte; + for item in delta { + let TextDelta::Insert { insert, attributes } = item else { + continue; + }; + if insert.is_empty() { + continue; + } + text.insert_utf8(cursor, insert)?; + let styles = run_styles_from_attrs(attributes.as_ref()); + if styles != RunStyles::default() { + set_run_styles_utf8(text, cursor..cursor + insert.len(), styles)?; + } + cursor += insert.len(); + } + Ok(()) +} + +fn refresh_marks(text: &LoroText, runs: &[TextRun]) -> Result<()> { + let len = text.len_utf8(); + if len == 0 { + return Ok(()); + } + for key in [MARK_SEMANTIC, MARK_UNDERLINE, MARK_STRIKE, MARK_HIGHLIGHT] { + unmark_utf8(text, 0..len, key)?; + } + apply_mark_intervals(text, &mark_intervals_from_runs(runs)) +} + +fn span_paragraph_texts(span: &DocumentSpan) -> Result> { + let mut texts = Vec::with_capacity(span.paragraphs.len()); + let mut byte = 0; + for (ix, paragraph) in span.paragraphs.iter().enumerate() { + let end = byte + paragraph_text_len(paragraph); + texts.push( + span + .text + .get(byte..end) + .context("DocumentSpan text is shorter than its paragraph runs")? + .to_string(), + ); + byte = end + usize::from(ix + 1 < span.paragraphs.len()); + } + Ok(texts) +} + +fn document_paragraph_ix_for_block(document: &Document, target_block_ix: usize) -> Result { + let mut paragraph_ix = 0; + for (block_ix, block) in document.blocks.iter().enumerate() { + match block { + Block::Paragraph(_) if block_ix == target_block_ix => return Ok(paragraph_ix), + Block::Paragraph(_) => paragraph_ix += 1, + Block::Image(_) | Block::Equation(_) | Block::Table(_) if block_ix == target_block_ix => { + bail!("target block is not a paragraph") + }, + Block::Image(_) | Block::Equation(_) | Block::Table(_) => {}, + } + } + bail!("target block index is out of bounds") +} + +fn document_block_ix_for_paragraph(document: &Document, target_paragraph_ix: usize) -> Result { + let mut paragraph_ix = 0; + for (block_ix, block) in document.blocks.iter().enumerate() { + match block { + Block::Paragraph(_) if paragraph_ix == target_paragraph_ix => return Ok(block_ix), + Block::Paragraph(_) => paragraph_ix += 1, + Block::Image(_) | Block::Equation(_) | Block::Table(_) => {}, + } + } + bail!("target paragraph index is out of bounds") +} + +fn kind_for_payload(payload: &BlockPayload) -> &'static str { + match payload { + BlockPayload::Image { .. } => KIND_IMAGE, + BlockPayload::Equation { .. } => KIND_EQUATION, + BlockPayload::Table(_) => KIND_TABLE, + } +} + +fn map_i64(map: &LoroMap, key: &str) -> Result { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("collaboration map key {key} is not an i64") + }, + } +} + +fn map_i64_or(map: &LoroMap, key: &str, default: i64) -> Result { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), + None => Ok(default), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) => { + bail!("collaboration map key {key} is not an i64") + }, + } +} diff --git a/crates/flowstate-collab/src/net/anti_entropy.rs b/crates/flowstate-collab/src/net/anti_entropy.rs new file mode 100644 index 00000000..0c915d9c --- /dev/null +++ b/crates/flowstate-collab/src/net/anti_entropy.rs @@ -0,0 +1,116 @@ +use std::time::{Duration, Instant}; + +use iroh::EndpointId; + +use crate::ids::SessionId; + +pub const DIGEST_INTERVAL: Duration = Duration::from_secs(10); +pub const DIGEST_JITTER_PERCENT: u32 = 20; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum VersionVectorRelation { + Equal, + SenderHasMissingOps, + WeHaveMissingOps, + Concurrent, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum GapAction { + None, + Pull { from: EndpointId, our_vv: Vec }, + LineageMismatch { from: EndpointId, expected: SessionId, got: SessionId }, +} + +#[derive(Clone, Debug)] +pub struct AntiEntropyState { + session: SessionId, + next_digest: Instant, + pull_in_flight: bool, +} + +impl AntiEntropyState { + #[must_use] + pub fn new(session: SessionId, now: Instant) -> Self { + Self { + session, + next_digest: next_jittered_deadline(now), + pull_in_flight: false, + } + } + + #[must_use] + pub const fn session(&self) -> SessionId { + self.session + } + + #[must_use] + pub fn digest_due(&self, now: Instant) -> bool { + now >= self.next_digest + } + + pub fn mark_digest_sent(&mut self, now: Instant) { + self.next_digest = next_jittered_deadline(now); + } + + pub fn schedule_immediate_digest(&mut self) { + self.next_digest = Instant::now(); + } + + pub fn on_neighbor_up(&mut self) { + self.schedule_immediate_digest(); + } + + pub fn on_reconnect(&mut self) { + self.schedule_immediate_digest(); + } + + #[must_use] + pub fn on_lagged(&mut self, fallback_peer: Option, our_vv: Vec) -> GapAction { + self.schedule_immediate_digest(); + let Some(from) = fallback_peer else { + return GapAction::None; + }; + self.begin_pull(from, our_vv) + } + + #[must_use] + pub fn consider_digest(&mut self, from: EndpointId, digest_session: SessionId, relation: VersionVectorRelation, our_vv: Vec) -> GapAction { + if digest_session != self.session { + return GapAction::LineageMismatch { + from, + expected: self.session, + got: digest_session, + }; + } + + match relation { + VersionVectorRelation::SenderHasMissingOps | VersionVectorRelation::Concurrent => self.begin_pull(from, our_vv), + VersionVectorRelation::Equal | VersionVectorRelation::WeHaveMissingOps => GapAction::None, + } + } + + pub fn finish_pull(&mut self) { + self.pull_in_flight = false; + } + + fn begin_pull(&mut self, from: EndpointId, our_vv: Vec) -> GapAction { + if self.pull_in_flight { + return GapAction::None; + } + self.pull_in_flight = true; + GapAction::Pull { from, our_vv } + } +} + +#[must_use] +pub fn next_jittered_deadline(now: Instant) -> Instant { + let base_millis = DIGEST_INTERVAL.as_millis() as u64; + let jitter_millis = base_millis * u64::from(DIGEST_JITTER_PERCENT) / 100; + let spread = jitter_millis.saturating_mul(2).saturating_add(1); + let offset = rand::random::() % spread; + let delay_millis = base_millis + .saturating_add(offset) + .saturating_sub(jitter_millis); + now + Duration::from_millis(delay_millis) +} diff --git a/crates/flowstate-collab/src/net/blobs.rs b/crates/flowstate-collab/src/net/blobs.rs new file mode 100644 index 00000000..9a579d64 --- /dev/null +++ b/crates/flowstate-collab/src/net/blobs.rs @@ -0,0 +1,83 @@ +use std::{collections::VecDeque, num::NonZeroUsize}; + +use crate::ids::BlobId; + +pub const DEFAULT_MAX_BLOBS: usize = 64; +pub const DEFAULT_MAX_BYTES: usize = 16 * 1024 * 1024; + +#[derive(Debug)] +pub struct BlobOutbox { + max_blobs: NonZeroUsize, + max_bytes: usize, + total_bytes: usize, + entries: VecDeque<(BlobId, Vec)>, +} + +impl Default for BlobOutbox { + fn default() -> Self { + Self { + max_blobs: NonZeroUsize::new(DEFAULT_MAX_BLOBS).expect("constant is non-zero"), + max_bytes: DEFAULT_MAX_BYTES, + total_bytes: 0, + entries: VecDeque::new(), + } + } +} + +impl BlobOutbox { + #[must_use] + pub fn new(max_blobs: NonZeroUsize, max_bytes: usize) -> Self { + Self { + max_blobs, + max_bytes, + total_bytes: 0, + entries: VecDeque::new(), + } + } + + #[must_use] + pub fn insert(&mut self, bytes: Vec) -> BlobId { + let id = BlobId::new(); + self.insert_with_id(id, bytes); + id + } + + pub fn insert_with_id(&mut self, id: BlobId, bytes: Vec) { + self.total_bytes = self.total_bytes.saturating_add(bytes.len()); + self.entries.push_back((id, bytes)); + self.trim(); + } + + #[must_use] + pub fn get(&self, id: BlobId) -> Option<&[u8]> { + self + .entries + .iter() + .find_map(|(candidate, bytes)| (*candidate == id).then_some(bytes.as_slice())) + } + + #[must_use] + pub fn len(&self) -> usize { + self.entries.len() + } + + #[must_use] + pub const fn total_bytes(&self) -> usize { + self.total_bytes + } + + #[must_use] + pub fn is_empty(&self) -> bool { + self.entries.is_empty() + } + + fn trim(&mut self) { + while self.entries.len() > self.max_blobs.get() || self.total_bytes > self.max_bytes { + let Some((_, bytes)) = self.entries.pop_front() else { + self.total_bytes = 0; + return; + }; + self.total_bytes = self.total_bytes.saturating_sub(bytes.len()); + } + } +} diff --git a/crates/flowstate-collab/src/net/direct.rs b/crates/flowstate-collab/src/net/direct.rs new file mode 100644 index 00000000..429f5f0b --- /dev/null +++ b/crates/flowstate-collab/src/net/direct.rs @@ -0,0 +1,307 @@ +use std::{ + collections::{HashMap, HashSet}, + sync::{Arc, OnceLock}, + time::Duration, +}; + +use anyhow::{Context as _, Result, anyhow, ensure}; +use async_channel::Sender; +use iroh::{ + Endpoint, EndpointId, + endpoint::{Connection, RecvStream, SendStream}, + protocol::{AcceptError, ProtocolHandler}, +}; +use tokio::{ + sync::{RwLock, Semaphore}, + time::timeout, +}; + +use crate::{ + ids::{BlobId, SessionId}, + proto_direct::{AssetBytes, DirectRequest, DirectResponseHeader, MAX_FRAME_LEN, MAX_PAYLOAD_CHUNK_LEN, decode_frame, encode_frame}, + proto_gossip::DIRECT_ALPN, +}; + +use super::blobs::BlobOutbox; + +const DIRECT_SERVE_CONCURRENCY: usize = 4; +static CLIENT_ENDPOINT: OnceLock = OnceLock::new(); + +#[derive(Clone, Debug)] +pub struct DirectSessionHandler { + requests: Sender, +} + +impl DirectSessionHandler { + #[must_use] + pub fn new(requests: Sender) -> Self { + Self { requests } + } +} + +#[derive(Debug)] +pub enum DirectServeRequest { + Snapshot { reply: Sender>> }, + Updates { have_vv: Vec, reply: Sender>> }, + Asset { asset: u128, reply: Sender> }, +} + +#[derive(Clone, Debug, Default)] +pub struct DirectServeState { + inner: Arc>, +} + +#[derive(Debug, Default)] +struct DirectServeInner { + attached: HashSet, + blobs: HashMap, + handlers: HashMap, +} + +impl DirectServeState { + pub async fn attach_session(&self, session: SessionId) { + self.inner.write().await.attached.insert(session); + } + + pub async fn detach_session(&self, session: SessionId) { + let mut inner = self.inner.write().await; + inner.attached.remove(&session); + inner.blobs.remove(&session); + inner.handlers.remove(&session); + } + + pub async fn register_handler(&self, session: SessionId, handler: DirectSessionHandler) { + let mut inner = self.inner.write().await; + inner.attached.insert(session); + inner.handlers.insert(session, handler); + } + + pub async fn insert_blob(&self, session: SessionId, bytes: Vec) -> BlobId { + let mut inner = self.inner.write().await; + inner.attached.insert(session); + inner.blobs.entry(session).or_default().insert(bytes) + } + + async fn serve(&self, request: DirectRequest) -> ServeOutcome { + let session = request.session(); + let handler = { + let inner = self.inner.read().await; + if !inner.attached.contains(&session) { + return ServeOutcome::Header(DirectResponseHeader::NotAttached); + } + if let DirectRequest::Blob { blob, .. } = request { + return inner + .blobs + .get(&session) + .and_then(|outbox| outbox.get(blob)) + .map_or(ServeOutcome::Header(DirectResponseHeader::NotFound), |bytes| { + ServeOutcome::Payload(bytes.to_vec()) + }); + } + inner.handlers.get(&session).cloned() + }; + + let Some(handler) = handler else { + return ServeOutcome::Header(DirectResponseHeader::NotFound); + }; + + match request { + DirectRequest::Snapshot { .. } => request_payload(handler.requests, |reply| DirectServeRequest::Snapshot { reply }).await, + DirectRequest::Updates { have_vv, .. } => request_payload(handler.requests, |reply| DirectServeRequest::Updates { have_vv, reply }).await, + DirectRequest::Asset { asset, .. } => request_asset(handler.requests, asset).await, + DirectRequest::Blob { .. } => ServeOutcome::Header(DirectResponseHeader::NotFound), + } + } +} + +#[derive(Debug)] +enum ServeOutcome { + Header(DirectResponseHeader), + Payload(Vec), +} + +#[derive(Clone, Debug)] +pub struct DirectProto { + state: DirectServeState, + permits: Arc, +} + +impl DirectProto { + #[must_use] + pub fn new(state: DirectServeState) -> Self { + Self { + state, + permits: Arc::new(Semaphore::new(DIRECT_SERVE_CONCURRENCY)), + } + } + + async fn handle_stream(&self, mut send: SendStream, mut recv: RecvStream) -> Result<()> { + let request = read_frame::(&mut recv).await?; + let Ok(permit) = self.permits.clone().try_acquire_owned() else { + write_response(&mut send, ServeOutcome::Header(DirectResponseHeader::Busy)).await?; + return Ok(()); + }; + + let outcome = self.state.serve(request).await; + write_response(&mut send, outcome).await?; + drop(permit); + Ok(()) + } +} + +impl ProtocolHandler for DirectProto { + async fn accept(&self, connection: Connection) -> Result<(), AcceptError> { + while let Ok((send, recv)) = connection.accept_bi().await { + let proto = self.clone(); + tokio::spawn(async move { + if let Err(error) = proto.handle_stream(send, recv).await { + eprintln!("flowstate collab direct stream failed: {error:#}"); + } + }); + } + Ok(()) + } +} + +pub(crate) fn install_endpoint(endpoint: Endpoint) { + let _ = CLIENT_ENDPOINT.set(endpoint); +} + +pub async fn pull_with_fallback(req: DirectRequest, candidates: Vec, per_peer_timeout: Duration) -> Result> { + let endpoint = CLIENT_ENDPOINT + .get() + .cloned() + .ok_or_else(|| anyhow!("collaboration direct client endpoint is not running"))?; + pull_with_endpoint(&endpoint, req, candidates, per_peer_timeout).await +} + +pub async fn pull_with_endpoint( + endpoint: &Endpoint, + req: DirectRequest, + candidates: Vec, + per_peer_timeout: Duration, +) -> Result> { + ensure!(!candidates.is_empty(), "direct pull has no candidate peers"); + + let mut errors = Vec::new(); + for peer in candidates { + match timeout(per_peer_timeout, pull_once(endpoint, peer, req.clone())).await { + Ok(Ok(bytes)) => return Ok(bytes), + Ok(Err(error)) => errors.push(format!("{peer}: {error:#}")), + Err(_) => errors.push(format!("{peer}: timed out after {per_peer_timeout:?}")), + } + } + + Err(anyhow!("direct pull failed for all candidates: {}", errors.join("; "))) +} + +async fn pull_once(endpoint: &Endpoint, peer: EndpointId, req: DirectRequest) -> Result> { + let connection = endpoint + .connect(peer, DIRECT_ALPN) + .await + .context("direct dial failed")?; + let (mut send, mut recv) = connection + .open_bi() + .await + .context("opening direct request stream failed")?; + write_frame(&mut send, &encode_frame(&req)?).await?; + send.finish()?; + + let header = read_frame::(&mut recv).await?; + match header { + DirectResponseHeader::Ok { total_len } => read_payload(&mut recv, total_len).await, + DirectResponseHeader::NotAttached => Err(anyhow!("peer is not attached to this session")), + DirectResponseHeader::NotFound => Err(anyhow!("peer does not have the requested collaboration data")), + DirectResponseHeader::Busy => Err(anyhow!("peer is busy serving collaboration data")), + } +} + +async fn request_payload(requests: Sender, make_request: F) -> ServeOutcome +where + F: FnOnce(Sender>>) -> DirectServeRequest, +{ + let (reply_tx, reply_rx) = async_channel::bounded(1); + if requests.send(make_request(reply_tx)).await.is_err() { + return ServeOutcome::Header(DirectResponseHeader::NotAttached); + } + match reply_rx.recv().await { + Ok(Ok(bytes)) => ServeOutcome::Payload(bytes), + Ok(Err(_)) | Err(_) => ServeOutcome::Header(DirectResponseHeader::NotFound), + } +} + +async fn request_asset(requests: Sender, asset: u128) -> ServeOutcome { + let (reply_tx, reply_rx) = async_channel::bounded(1); + if requests + .send(DirectServeRequest::Asset { asset, reply: reply_tx }) + .await + .is_err() + { + return ServeOutcome::Header(DirectResponseHeader::NotAttached); + } + match reply_rx.recv().await { + Ok(Ok(asset)) => ServeOutcome::Payload(asset.bytes), + Ok(Err(_)) | Err(_) => ServeOutcome::Header(DirectResponseHeader::NotFound), + } +} + +async fn read_frame(recv: &mut RecvStream) -> Result +where + T: for<'de> serde::Deserialize<'de>, +{ + let mut len_bytes = [0; 4]; + recv.read_exact(&mut len_bytes).await?; + let payload_len = u32::from_le_bytes(len_bytes) as usize; + ensure!(payload_len <= MAX_FRAME_LEN, "direct frame exceeds {MAX_FRAME_LEN} bytes"); + let mut frame = len_bytes.to_vec(); + frame.resize(4 + payload_len, 0); + recv.read_exact(&mut frame[4..]).await?; + decode_frame(&frame) +} + +async fn write_response(send: &mut SendStream, outcome: ServeOutcome) -> Result<()> { + match outcome { + ServeOutcome::Header(header) => write_frame(send, &encode_frame(&header)?).await?, + ServeOutcome::Payload(payload) => { + let header = DirectResponseHeader::Ok { + total_len: payload.len() as u64, + }; + write_frame(send, &encode_frame(&header)?).await?; + write_payload(send, &payload).await?; + }, + } + send.finish()?; + Ok(()) +} + +async fn write_frame(send: &mut SendStream, frame: &[u8]) -> Result<()> { + send.write_all(frame).await?; + Ok(()) +} + +async fn write_payload(send: &mut SendStream, payload: &[u8]) -> Result<()> { + for chunk in payload.chunks(MAX_PAYLOAD_CHUNK_LEN) { + send.write_all(chunk).await?; + } + Ok(()) +} + +async fn read_payload(recv: &mut RecvStream, total_len: u64) -> Result> { + let total_len = usize::try_from(total_len).context("direct payload is too large for this platform")?; + let mut payload = vec![0; total_len]; + let mut offset = 0; + while offset < total_len { + let next = (offset + MAX_PAYLOAD_CHUNK_LEN).min(total_len); + recv.read_exact(&mut payload[offset..next]).await?; + offset = next; + } + Ok(payload) +} + +impl DirectRequest { + fn session(&self) -> SessionId { + match self { + Self::Snapshot { session } | Self::Updates { session, .. } | Self::Blob { session, .. } | Self::Asset { session, .. } => *session, + } + } +} diff --git a/crates/flowstate-collab/src/net/mod.rs b/crates/flowstate-collab/src/net/mod.rs new file mode 100644 index 00000000..6f118a62 --- /dev/null +++ b/crates/flowstate-collab/src/net/mod.rs @@ -0,0 +1,53 @@ +pub mod anti_entropy; +pub mod blobs; +pub mod direct; +pub mod runtime; +pub mod swarm; + +use anyhow::Result; +use async_channel::Sender; +use iroh::{EndpointAddr, EndpointId}; + +use crate::{ids::{BlobId, SessionId}, proto_direct::AssetBytes, proto_gossip::GossipMsg}; + +use self::direct::DirectSessionHandler; + +pub type Reply = Sender; + +#[derive(Clone, Debug)] +pub enum PublishPayload { + Update(Vec), + Presence(Vec), + Digest { vv: Vec }, +} + +#[derive(Debug)] +pub enum NetCommand { + EnsureUp, + RegisterDirectHandler { session: SessionId, handler: DirectSessionHandler }, + CreateSession { session: SessionId, reply: Reply> }, + JoinSession { session: SessionId, bootstrap: Vec }, + LeaveSession { session: SessionId }, + Publish { session: SessionId, payload: PublishPayload }, + PullUpdates { session: SessionId, from: EndpointId, our_vv: Vec, reply: Reply>> }, + PullSnapshot { session: SessionId, from: EndpointId, reply: Reply>> }, + PullBlob { session: SessionId, from: EndpointId, blob: BlobId, reply: Reply>> }, + PullAsset { session: SessionId, from: EndpointId, asset: u128, reply: Reply> }, + MintTicketAddr { reply: Reply }, + Shutdown, +} + +#[derive(Clone, Debug)] +pub struct TicketSeed { + pub inviter: EndpointAddr, +} + +#[derive(Clone, Debug)] +pub enum NetEvent { + Gossip { session: SessionId, from: EndpointId, msg: GossipMsg }, + NeighborUp { session: SessionId, peer: EndpointId }, + NeighborDown { session: SessionId, peer: EndpointId }, + GossipLagged { session: SessionId }, + SubscribeFailed { session: SessionId, error: String }, + EndpointOnline(bool), +} diff --git a/crates/flowstate-collab/src/net/runtime.rs b/crates/flowstate-collab/src/net/runtime.rs new file mode 100644 index 00000000..56c6c36b --- /dev/null +++ b/crates/flowstate-collab/src/net/runtime.rs @@ -0,0 +1,179 @@ +use std::{collections::HashMap, sync::OnceLock, thread, time::Duration}; + +use anyhow::{Context as _, Result}; +use iroh::{Endpoint, EndpointAddr, endpoint::presets}; +use iroh_gossip::net::Gossip; + +use crate::proto_direct::{AssetBytes, DirectRequest}; + +use super::{ + NetCommand, NetEvent, TicketSeed, + direct::{self, DirectProto, DirectServeState}, + swarm::SwarmHandle, +}; + +pub type CommandSender = async_channel::Sender; +pub type EventReceiver = async_channel::Receiver; + +const DIRECT_PULL_TIMEOUT: Duration = Duration::from_secs(10); +static RUNTIME: OnceLock = OnceLock::new(); + +#[derive(Clone)] +struct RuntimeBridge { + commands: CommandSender, + events: EventReceiver, +} + +pub fn start() -> Result<(CommandSender, EventReceiver)> { + if let Some(runtime) = RUNTIME.get() { + return Ok((runtime.commands.clone(), runtime.events.clone())); + } + + let runtime = spawn_runtime()?; + let set_failed = RUNTIME.set(runtime.clone()).is_err(); + if set_failed + && let Some(runtime) = RUNTIME.get() + { + return Ok((runtime.commands.clone(), runtime.events.clone())); + } + Ok((runtime.commands, runtime.events)) +} + +fn spawn_runtime() -> Result { + let (cmd_tx, cmd_rx) = async_channel::unbounded(); + let (evt_tx, evt_rx) = async_channel::unbounded(); + let thread_evt_tx = evt_tx.clone(); + + thread::Builder::new() + .name("flowstate-collab-net".into()) + .spawn(move || { + let runtime = match tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + { + Ok(runtime) => runtime, + Err(error) => { + let _ = thread_evt_tx.try_send(NetEvent::EndpointOnline(false)); + eprintln!("flowstate collaboration tokio runtime failed: {error:#}"); + return; + }, + }; + + if let Err(error) = runtime.block_on(net_main(cmd_rx, evt_tx)) { + let _ = thread_evt_tx.try_send(NetEvent::EndpointOnline(false)); + eprintln!("flowstate collaboration networking stopped: {error:#}"); + } + }) + .context("spawning collaboration networking thread failed")?; + + Ok(RuntimeBridge { + commands: cmd_tx, + events: evt_rx, + }) +} + +async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_channel::Sender) -> Result<()> { + let endpoint = Endpoint::builder(presets::N0) + .bind() + .await + .context("binding collaboration endpoint failed")?; + direct::install_endpoint(endpoint.clone()); + + let gossip = Gossip::builder().spawn(endpoint.clone()); + let direct_state = DirectServeState::default(); + let _router = iroh::protocol::Router::builder(endpoint.clone()) + .accept(iroh_gossip::ALPN, gossip.clone()) + .accept(crate::DIRECT_ALPN, DirectProto::new(direct_state.clone())) + .spawn(); + + let _ = evt_tx.send(NetEvent::EndpointOnline(true)).await; + let mut swarms = HashMap::new(); + + while let Ok(command) = cmd_rx.recv().await { + match command { + NetCommand::EnsureUp => { + let _ = evt_tx.send(NetEvent::EndpointOnline(true)).await; + }, + NetCommand::RegisterDirectHandler { session, handler } => { + direct_state.register_handler(session, handler).await; + }, + NetCommand::CreateSession { session, reply } => { + direct_state.attach_session(session).await; + replace_swarm( + &mut swarms, + SwarmHandle::spawn(endpoint.clone(), gossip.clone(), direct_state.clone(), session, Vec::new(), evt_tx.clone())?, + ) + .await; + let _ = reply + .send(Ok(TicketSeed { + inviter: endpoint_addr(&endpoint), + })) + .await; + }, + NetCommand::JoinSession { session, bootstrap } => { + direct_state.attach_session(session).await; + replace_swarm( + &mut swarms, + SwarmHandle::spawn(endpoint.clone(), gossip.clone(), direct_state.clone(), session, bootstrap, evt_tx.clone())?, + ) + .await; + }, + NetCommand::LeaveSession { session } => { + if let Some(handle) = swarms.remove(&session) { + handle.stop().await; + } + direct_state.detach_session(session).await; + }, + NetCommand::Publish { session, payload } => { + if let Some(handle) = swarms.get(&session) { + handle.publish(payload).await?; + } + }, + NetCommand::PullUpdates { + session, + from, + our_vv, + reply, + } => { + let result = direct::pull_with_fallback(DirectRequest::Updates { session, have_vv: our_vv }, vec![from], DIRECT_PULL_TIMEOUT).await; + let _ = reply.send(result).await; + }, + NetCommand::PullSnapshot { session, from, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, vec![from], DIRECT_PULL_TIMEOUT).await; + let _ = reply.send(result).await; + }, + NetCommand::PullBlob { session, from, blob, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Blob { session, blob }, vec![from], DIRECT_PULL_TIMEOUT).await; + let _ = reply.send(result).await; + }, + NetCommand::PullAsset { session, from, asset, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Asset { session, asset }, vec![from], DIRECT_PULL_TIMEOUT) + .await + .map(|bytes| AssetBytes { bytes }); + let _ = reply.send(result).await; + }, + NetCommand::MintTicketAddr { reply } => { + let _ = reply.send(endpoint_addr(&endpoint)).await; + }, + NetCommand::Shutdown => break, + } + } + + for (_, handle) in swarms { + handle.stop().await; + } + endpoint.close().await; + Ok(()) +} + +async fn replace_swarm(swarms: &mut HashMap, handle: SwarmHandle) { + let session = handle.session; + if let Some(previous) = swarms.insert(session, handle) { + previous.stop().await; + } +} + +fn endpoint_addr(endpoint: &Endpoint) -> EndpointAddr { + endpoint.addr() +} diff --git a/crates/flowstate-collab/src/net/swarm.rs b/crates/flowstate-collab/src/net/swarm.rs new file mode 100644 index 00000000..6cd10bb9 --- /dev/null +++ b/crates/flowstate-collab/src/net/swarm.rs @@ -0,0 +1,180 @@ +use anyhow::{Context as _, Result}; +use async_channel::{Receiver, Sender}; +use iroh::{Endpoint, EndpointAddr, EndpointId, address_lookup::memory::MemoryLookup}; +use iroh_gossip::{ + api::{Event, GossipSender}, + net::Gossip, + proto::TopicId, +}; +use n0_future::StreamExt as _; + +use crate::{ + ids::SessionId, + proto_gossip::{self, GOSSIP_INLINE_LIMIT, GossipMsg}, +}; + +use super::{NetEvent, PublishPayload, direct::DirectServeState}; + +#[derive(Debug)] +enum SwarmCommand { + Publish(PublishPayload), + Stop, +} + +#[derive(Clone, Debug)] +pub struct SwarmHandle { + pub session: SessionId, + tx: Sender, +} + +impl SwarmHandle { + pub fn spawn( + endpoint: Endpoint, + gossip: Gossip, + direct_state: DirectServeState, + session: SessionId, + bootstrap: Vec, + evt_tx: Sender, + ) -> Result { + register_bootstrap_addrs(&endpoint, &bootstrap)?; + let bootstrap_ids = bootstrap.iter().map(endpoint_id).collect::>(); + let (tx, rx) = async_channel::unbounded(); + tokio::spawn(async move { + if let Err(error) = run_session(gossip, direct_state, session, bootstrap_ids, rx, evt_tx.clone()).await { + let _ = evt_tx + .send(NetEvent::SubscribeFailed { + session, + error: format!("{error:#}"), + }) + .await; + } + }); + Ok(Self { session, tx }) + } + + pub async fn publish(&self, payload: PublishPayload) -> Result<()> { + self + .tx + .send(SwarmCommand::Publish(payload)) + .await + .context("collaboration swarm task stopped") + } + + pub async fn stop(&self) { + let _ = self.tx.send(SwarmCommand::Stop).await; + } +} + +async fn run_session( + gossip: Gossip, + direct_state: DirectServeState, + session: SessionId, + bootstrap: Vec, + rx: Receiver, + evt_tx: Sender, +) -> Result<()> { + let topic = gossip + .subscribe(topic_id(session), bootstrap) + .await + .context("subscribing to collaboration gossip topic failed")?; + let (sender, mut receiver) = topic.split(); + + loop { + tokio::select! { + command = rx.recv() => { + match command { + Ok(SwarmCommand::Publish(payload)) => publish(&sender, &direct_state, session, payload).await?, + Ok(SwarmCommand::Stop) | Err(_) => break, + } + }, + event = receiver.next() => { + let event = event.context("collaboration gossip receiver ended")??; + handle_event(event, session, &evt_tx).await?; + }, + } + } + + Ok(()) +} + +async fn publish( + sender: &GossipSender, + direct_state: &DirectServeState, + session: SessionId, + payload: PublishPayload, +) -> Result<()> { + let (message, neighbors_only) = match payload { + PublishPayload::Update(bytes) => (update_message(direct_state, session, bytes).await?, false), + PublishPayload::Presence(bytes) => (GossipMsg::Presence(bytes), false), + PublishPayload::Digest { vv } => (GossipMsg::Digest { session, vv }, true), + }; + + let frame = if neighbors_only { + proto_gossip::encode(&message)? + } else { + proto_gossip::encode_inline(&message)? + }; + + if neighbors_only { + sender.broadcast_neighbors(frame.into()).await?; + } else { + sender.broadcast(frame.into()).await?; + } + Ok(()) +} + +async fn update_message(direct_state: &DirectServeState, session: SessionId, bytes: Vec) -> Result { + let inline = GossipMsg::Update(bytes.clone()); + if proto_gossip::encoded_len(&inline)? <= GOSSIP_INLINE_LIMIT { + return Ok(inline); + } + + let len = bytes.len() as u64; + let blob = direct_state.insert_blob(session, bytes).await; + Ok(GossipMsg::UpdateAvailable { blob, len }) +} + +async fn handle_event(event: Event, session: SessionId, evt_tx: &Sender) -> Result<()> { + match event { + Event::Received(message) => match proto_gossip::decode(&message.content) { + Ok(msg) => { + let _ = evt_tx + .send(NetEvent::Gossip { + session, + from: message.delivered_from, + msg, + }) + .await; + }, + Err(error) => eprintln!("flowstate collab ignored gossip frame: {error:#}"), + }, + Event::NeighborUp(peer) => { + let _ = evt_tx.send(NetEvent::NeighborUp { session, peer }).await; + }, + Event::NeighborDown(peer) => { + let _ = evt_tx.send(NetEvent::NeighborDown { session, peer }).await; + }, + Event::Lagged => { + let _ = evt_tx.send(NetEvent::GossipLagged { session }).await; + }, + } + Ok(()) +} + +fn register_bootstrap_addrs(endpoint: &Endpoint, bootstrap: &[EndpointAddr]) -> Result<()> { + if bootstrap.is_empty() { + return Ok(()); + } + endpoint + .address_lookup()? + .add(MemoryLookup::from_endpoint_info(bootstrap.iter().cloned())); + Ok(()) +} + +fn endpoint_id(addr: &EndpointAddr) -> EndpointId { + addr.id +} + +fn topic_id(session: SessionId) -> TopicId { + TopicId::from_bytes(*session.as_bytes()) +} diff --git a/crates/flowstate-collab/src/presence.rs b/crates/flowstate-collab/src/presence.rs new file mode 100644 index 00000000..a2d87bda --- /dev/null +++ b/crates/flowstate-collab/src/presence.rs @@ -0,0 +1,157 @@ +//! Presence state and roster derivation. + +use anyhow::{Context as _, Result}; +use loro::{LoroValue, Subscription, awareness::{EphemeralStore, EphemeralStoreEvent}}; +use serde::{Deserialize, Serialize}; + +use crate::ids::{PALETTE, PeerId, SessionId}; + +pub const PRESENCE_TIMEOUT_MS: i64 = 30_000; +pub const PRESENCE_KEEPALIVE_SECS: u64 = 10; + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub struct PresenceState { + pub name: String, + pub selection: Option, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub struct PresenceSelection { + pub head_container: String, + pub head: Vec, + pub anchor_container: String, + pub anchor: Vec, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct RosterEntry { + pub key: String, + pub name: String, + pub color_rgb: u32, + pub selection: Option, +} + +#[derive(Clone, Debug)] +pub struct PresenceStore { + store: EphemeralStore, + self_key: String, + self_color: u32, +} + +impl PresenceStore { + #[must_use] + pub fn new(self_peer: &PeerId) -> Self { + Self { + store: EphemeralStore::new(PRESENCE_TIMEOUT_MS), + self_key: peer_key(self_peer), + self_color: color_for_peer(self_peer), + } + } + + #[must_use] + pub fn self_key(&self) -> &str { + &self.self_key + } + + #[must_use] + pub const fn self_color(&self) -> u32 { + self.self_color + } + + pub fn set_self(&self, state: &PresenceState) -> Result<()> { + self + .store + .set(&self.self_key, LoroValue::Binary(encode_state(state)?.into())); + Ok(()) + } + + pub fn delete_self(&self) { + self.store.delete(&self.self_key); + } + + #[must_use] + pub fn encode_self(&self) -> Vec { + self.store.encode(&self.self_key) + } + + #[must_use] + pub fn encode_all(&self) -> Vec { + self.store.encode_all() + } + + pub fn apply(&self, bytes: &[u8]) -> Result<()> { + self + .store + .apply(bytes) + .map_err(|error| anyhow::anyhow!(error.to_string())) + .context("applying collaboration presence update failed") + } + + pub fn remove_outdated(&self) { + self.store.remove_outdated(); + } + + #[must_use] + pub fn roster(&self) -> Vec { + let mut entries = self + .store + .get_all_states() + .into_iter() + .filter_map(|(key, value)| roster_entry_from_value(key, value)) + .collect::>(); + entries.sort_by(|left, right| left.name.cmp(&right.name).then_with(|| left.key.cmp(&right.key))); + entries + } + + pub fn subscribe_local_updates(&self, callback: impl Fn(&Vec) -> bool + Send + Sync + 'static) -> Subscription { + self.store.subscribe_local_updates(Box::new(callback)) + } + + pub fn subscribe(&self, callback: impl Fn(&EphemeralStoreEvent) -> bool + Send + Sync + 'static) -> Subscription { + self.store.subscribe(Box::new(callback)) + } +} + +pub fn encode_state(state: &PresenceState) -> Result> { + postcard::to_stdvec(state).context("encoding collaboration presence state failed") +} + +pub fn decode_state(bytes: &[u8]) -> Result { + postcard::from_bytes(bytes).context("decoding collaboration presence state failed") +} + +#[must_use] +pub fn peer_key(peer: &PeerId) -> String { + let mut key = String::with_capacity(peer.as_bytes().len() * 2); + for byte in peer.as_bytes() { + use std::fmt::Write as _; + let _ = write!(&mut key, "{byte:02x}"); + } + key +} + +#[must_use] +pub fn color_for_peer(peer: &PeerId) -> u32 { + PALETTE[SessionId::color_index_for_peer(peer)] +} + +fn roster_entry_from_value(key: String, value: LoroValue) -> Option { + let LoroValue::Binary(bytes) = value else { + return None; + }; + let state = decode_state(bytes.as_ref()).ok()?; + Some(RosterEntry { + color_rgb: color_for_key(&key), + key, + name: state.name, + selection: state.selection, + }) +} + +fn color_for_key(key: &str) -> u32 { + let mut hash = 0usize; + for byte in key.as_bytes().iter().take(16) { + hash = hash.wrapping_mul(257).wrapping_add(usize::from(*byte)); + } + PALETTE[hash % PALETTE.len()] +} diff --git a/crates/flowstate-collab/src/projection.rs b/crates/flowstate-collab/src/projection.rs new file mode 100644 index 00000000..961fc657 --- /dev/null +++ b/crates/flowstate-collab/src/projection.rs @@ -0,0 +1,176 @@ +//! Projection between flowtext documents and the shared Loro document. + +use anyhow::{Context as _, Result, bail}; +use gpui_flowtext::{AssetStore, Block, Document, DocumentTheme, InputBlock, document_from_input_blocks, paragraph_text}; +use loro::{Container, LoroDoc, LoroMap, LoroMovableList, LoroText, LoroValue, ValueOrContainer}; + +use crate::{ + SessionId, + schema::{ + BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_PARAGRAPH, KIND_TABLE, META, META_SCHEMA, META_SESSION, META_TITLE, REV, + SCHEMA_VERSION, STYLE, TEXT, apply_mark_intervals, block_from_payload, configure_text_styles, decode_paragraph_style, + encode_paragraph_style, input_paragraph_from_text, mark_intervals_from_runs, payload_from_block, + }, +}; + +pub fn populate_from_document(doc: &LoroDoc, session: SessionId, title: &str, document: &Document) -> Result<()> { + configure_text_styles(doc); + populate_meta(doc, session, title)?; + replace_blocks_from_document(doc, document)?; + doc.commit(); + Ok(()) +} + +pub fn replace_blocks_from_document(doc: &LoroDoc, document: &Document) -> Result<()> { + configure_text_styles(doc); + let blocks = doc.get_movable_list(BLOCKS); + clear_blocks(&blocks)?; + + let mut paragraph_ix = 0; + for block in document.blocks.iter() { + match block { + Block::Paragraph(paragraph) => { + let text = paragraph_text(document, paragraph_ix); + insert_paragraph_container(&blocks, blocks.len(), paragraph.style, ¶graph.runs, &text)?; + paragraph_ix += 1; + }, + Block::Image(_) | Block::Equation(_) | Block::Table(_) => { + insert_object_container(&blocks, blocks.len(), block, document)?; + }, + } + } + + Ok(()) +} + +pub fn input_blocks_from_loro(doc: &LoroDoc) -> Result> { + configure_text_styles(doc); + let blocks = doc.get_movable_list(BLOCKS); + let mut input_blocks = Vec::with_capacity(blocks.len()); + for ix in 0..blocks.len() { + let map = map_at(&blocks, ix)?; + input_blocks.push(input_block_from_container(&map)?); + } + Ok(input_blocks) +} + +pub fn document_from_loro(doc: &LoroDoc, theme: DocumentTheme) -> Result { + Ok(document_from_input_blocks(theme, input_blocks_from_loro(doc)?)) +} + +pub fn input_block_from_container(map: &LoroMap) -> Result { + match map_string(map, KIND)?.as_str() { + KIND_PARAGRAPH => { + let text = text_at(map, TEXT)?; + let style = decode_paragraph_style(map_i64(map, STYLE)?); + Ok(InputBlock::Paragraph(input_paragraph_from_text(&text, style))) + }, + KIND_IMAGE | KIND_EQUATION | KIND_TABLE => { + let payload = map_binary(map, DATA)?; + let payload = postcard::from_bytes::(&payload)?; + Ok(block_from_payload(payload, &AssetStore::default())) + }, + kind => bail!("unknown collaboration block kind {kind}"), + } +} + +fn populate_meta(doc: &LoroDoc, session: SessionId, title: &str) -> Result<()> { + let meta = doc.get_map(META); + meta.insert(META_SCHEMA, SCHEMA_VERSION)?; + meta.insert(META_SESSION, session.to_string())?; + meta.insert(META_TITLE, title)?; + Ok(()) +} + +fn clear_blocks(blocks: &LoroMovableList) -> Result<()> { + while !blocks.is_empty() { + blocks.delete(0, 1)?; + } + Ok(()) +} + +pub fn insert_paragraph_container( + blocks: &LoroMovableList, + ix: usize, + style: gpui_flowtext::ParagraphStyle, + runs: &[gpui_flowtext::TextRun], + paragraph_text: &str, +) -> Result<(LoroMap, LoroText)> { + let map = blocks.insert_container(ix, LoroMap::new())?; + map.insert(KIND, KIND_PARAGRAPH)?; + map.insert(STYLE, encode_paragraph_style(style))?; + let text = map.insert_container(TEXT, LoroText::new())?; + text.insert_utf8(0, paragraph_text)?; + apply_mark_intervals(&text, &mark_intervals_from_runs(runs))?; + Ok((map, text)) +} + +pub fn insert_object_container(blocks: &LoroMovableList, ix: usize, block: &Block, document: &Document) -> Result { + let Some(payload) = payload_from_block(block, &document.assets) else { + bail!("paragraph block reached object projection path"); + }; + let kind = match &payload { + BlockPayload::Image { .. } => KIND_IMAGE, + BlockPayload::Equation { .. } => KIND_EQUATION, + BlockPayload::Table(_) => KIND_TABLE, + }; + let map = blocks.insert_container(ix, LoroMap::new())?; + map.insert(KIND, kind)?; + map.insert(DATA, LoroValue::Binary(postcard::to_stdvec(&payload)?.into()))?; + map.insert(REV, 0_i64)?; + Ok(map) +} + +fn map_at(blocks: &LoroMovableList, ix: usize) -> Result { + match blocks.get(ix) { + Some(ValueOrContainer::Container(Container::Map(map))) => Ok(map), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("collaboration block row {ix} is not a map container") + }, + } +} + +fn text_at(map: &LoroMap, key: &str) -> Result { + match map.get(key) { + Some(ValueOrContainer::Container(Container::Text(text))) => Ok(text), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("collaboration map key {key} is not a text container") + }, + } +} + +fn map_string(map: &LoroMap, key: &str) -> Result { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::String(value))) => Ok(value.to_string()), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("collaboration map key {key} is not a string") + }, + } +} + +fn map_i64(map: &LoroMap, key: &str) -> Result { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("collaboration map key {key} is not an i64") + }, + } +} + +fn map_binary(map: &LoroMap, key: &str) -> Result> { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::Binary(value))) => Ok(value.to_vec()), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { + bail!("collaboration map key {key} is not binary") + }, + } +} + +pub fn verify_lineage(doc: &LoroDoc, session: SessionId) -> Result<()> { + let meta = doc.get_map(META); + let stored = map_string(&meta, META_SESSION).context("collaboration document has no session lineage")?; + if stored != session.to_string() { + bail!("collaboration document lineage does not match the ticket session") + } + Ok(()) +} diff --git a/crates/flowstate-collab/src/proto_direct.rs b/crates/flowstate-collab/src/proto_direct.rs new file mode 100644 index 00000000..c1a12a48 --- /dev/null +++ b/crates/flowstate-collab/src/proto_direct.rs @@ -0,0 +1,67 @@ +use anyhow::{Result, anyhow, ensure}; +use serde::{Deserialize, Serialize}; + +use crate::ids::{BlobId, SessionId}; +use crate::proto_gossip::PROTOCOL_VERSION; + +pub const MAX_FRAME_LEN: usize = 2 * 1024 * 1024; +pub const MAX_PAYLOAD_CHUNK_LEN: usize = 256 * 1024; + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub enum DirectRequest { + Snapshot { session: SessionId }, + Updates { session: SessionId, have_vv: Vec }, + Blob { session: SessionId, blob: BlobId }, + Asset { session: SessionId, asset: u128 }, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub enum DirectResponseHeader { + Ok { total_len: u64 }, + NotAttached, + NotFound, + Busy, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct AssetBytes { + pub bytes: Vec, +} + +pub fn encode_frame(value: &T) -> Result> { + let payload = postcard::to_stdvec(&(PROTOCOL_VERSION, value))?; + ensure!(payload.len() <= MAX_FRAME_LEN, "direct frame exceeds {MAX_FRAME_LEN} bytes"); + let len = u32::try_from(payload.len())?; + let mut frame = len.to_le_bytes().to_vec(); + frame.extend(payload); + Ok(frame) +} + +pub fn decode_frame(frame: &[u8]) -> Result +where + T: for<'de> Deserialize<'de>, +{ + ensure!(frame.len() >= 4, "direct frame is missing its length prefix"); + let payload_len = u32::from_le_bytes([frame[0], frame[1], frame[2], frame[3]]) as usize; + ensure!(payload_len <= MAX_FRAME_LEN, "direct frame exceeds {MAX_FRAME_LEN} bytes"); + let payload = frame + .get(4..) + .ok_or_else(|| anyhow!("direct frame payload is missing"))?; + ensure!(payload.len() == payload_len, "direct frame length prefix does not match payload length"); + let (version, value): (u16, T) = postcard::from_bytes(payload)?; + if version != PROTOCOL_VERSION { + return Err(anyhow!("unsupported direct protocol version {version}")); + } + Ok(value) +} + +pub fn split_frame(frame: &[u8]) -> Result<(usize, &[u8])> { + ensure!(frame.len() >= 4, "direct frame is missing its length prefix"); + let payload_len = u32::from_le_bytes([frame[0], frame[1], frame[2], frame[3]]) as usize; + ensure!(payload_len <= MAX_FRAME_LEN, "direct frame exceeds {MAX_FRAME_LEN} bytes"); + let payload = frame + .get(4..) + .ok_or_else(|| anyhow!("direct frame payload is missing"))?; + ensure!(payload.len() == payload_len, "direct frame length prefix does not match payload length"); + Ok((payload_len, payload)) +} diff --git a/crates/flowstate-collab/src/proto_gossip.rs b/crates/flowstate-collab/src/proto_gossip.rs new file mode 100644 index 00000000..54aa6e71 --- /dev/null +++ b/crates/flowstate-collab/src/proto_gossip.rs @@ -0,0 +1,41 @@ +use anyhow::{Result, anyhow, ensure}; +use serde::{Deserialize, Serialize}; + +use crate::ids::{BlobId, SessionId}; + +pub const PROTOCOL_VERSION: u16 = 1; +pub const DIRECT_ALPN: &[u8] = b"flowstate/collab-direct/0"; +pub const GOSSIP_INLINE_LIMIT: usize = 2 * 1024; + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +pub enum GossipMsg { + Update(Vec), + UpdateAvailable { blob: BlobId, len: u64 }, + Presence(Vec), + Digest { session: SessionId, vv: Vec }, +} + +pub fn encode(msg: &GossipMsg) -> Result> { + let mut bytes = PROTOCOL_VERSION.to_le_bytes().to_vec(); + bytes.extend(postcard::to_stdvec(msg)?); + Ok(bytes) +} + +pub fn encode_inline(msg: &GossipMsg) -> Result> { + let bytes = encode(msg)?; + ensure!(bytes.len() <= GOSSIP_INLINE_LIMIT, "gossip frame exceeds {GOSSIP_INLINE_LIMIT} bytes"); + Ok(bytes) +} + +pub fn encoded_len(msg: &GossipMsg) -> Result { + Ok(encode(msg)?.len()) +} + +pub fn decode(bytes: &[u8]) -> Result { + ensure!(bytes.len() >= 2, "gossip frame is missing protocol version"); + let version = u16::from_le_bytes([bytes[0], bytes[1]]); + if version != PROTOCOL_VERSION { + return Err(anyhow!("unsupported gossip protocol version {version}")); + } + Ok(postcard::from_bytes(&bytes[2..])?) +} diff --git a/crates/flowstate-collab/src/remote_apply.rs b/crates/flowstate-collab/src/remote_apply.rs new file mode 100644 index 00000000..24bdc4d6 --- /dev/null +++ b/crates/flowstate-collab/src/remote_apply.rs @@ -0,0 +1,293 @@ +//! Translation from Loro diffs into flowtext collaboration patches. + +use anyhow::{Context as _, Result, bail}; +use gpui_flowtext::{ + BlockId, CollabPatch, CollabStructuralBlock, CollabTextDelta, Document, InputBlock, ParagraphId, new_block_id, new_paragraph_id, + paragraph_text, +}; +use loro::{ + Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroText, LoroValue, TextDelta, ValueOrContainer, + event::{Diff, DiffEvent, ListDiffItem}, +}; + +use crate::{ + binding::{BindingRow, BlockKind, DocBinding}, + projection::input_block_from_container, + schema::{BLOCKS, DATA, REV, STYLE, TEXT, decode_paragraph_style}, +}; + +pub struct RemoteApplier<'s> { + pub doc: &'s LoroDoc, + pub binding: &'s mut DocBinding, +} + +impl RemoteApplier<'_> { + pub fn apply_event(&mut self, document: &Document, event: &DiffEvent<'_>) -> Result> { + if !should_process_event(event) { + return Ok(Vec::new()); + } + + let mut patches = Vec::new(); + for diff in &event.events { + match &diff.diff { + Diff::Text(delta) => self.apply_text_diff(document, diff.target, delta, &mut patches)?, + Diff::Map(delta) => { + for key in delta.updated.keys() { + self.apply_map_diff(diff.target, key.as_ref(), &mut patches)?; + } + }, + Diff::List(delta) => self.apply_list_diff(diff.target, delta, &mut patches)?, + Diff::Tree(_) | Diff::Counter(_) | Diff::Unknown => {}, + } + } + Ok(patches) + } + + fn apply_text_diff( + &self, + document: &Document, + target: &ContainerID, + delta: &[TextDelta], + patches: &mut Vec, + ) -> Result<()> { + let Some(row_ix) = self.binding.by_container.get(target).copied() else { + return Ok(()); + }; + let row = self + .binding + .rows + .get(row_ix) + .context("text diff row is outside DocBinding")?; + if !matches!(row.kind, BlockKind::Paragraph) { + return Ok(()); + } + let text = row + .text + .as_ref() + .context("paragraph diff row is missing its LoroText")?; + let style = decode_paragraph_style(map_i64(&row.map, STYLE)?); + let paragraph_ix = paragraph_ordinal_for_row(self.binding, row_ix).context("paragraph diff row has no paragraph ordinal")?; + let old_text = paragraph_text(document, paragraph_ix); + patches.push(CollabPatch::ParagraphText { + row: row_ix, + new: crate::schema::input_paragraph_from_text(text, style), + delta_utf8: text_delta_to_collab_utf8(&old_text, delta), + }); + Ok(()) + } + + fn apply_map_diff(&mut self, target: &ContainerID, key: &str, patches: &mut Vec) -> Result<()> { + let Some(row_ix) = self.binding.by_container.get(target).copied() else { + return Ok(()); + }; + let row = self + .binding + .rows + .get_mut(row_ix) + .context("map diff row is outside DocBinding")?; + match key { + STYLE if matches!(row.kind, BlockKind::Paragraph) => { + patches.push(CollabPatch::ParagraphStyle { + row: row_ix, + style: decode_paragraph_style(map_i64(&row.map, STYLE)?), + }); + }, + DATA | REV => { + let input = input_block_from_container(&row.map)?; + row.kind = block_kind_for_input(&input); + row.text = None; + row.paragraph_id = None; + row.version = 0; + patches.push(CollabPatch::ReplaceObjectBlock { + row: row_ix, + block: CollabStructuralBlock { + block_id: row.block_id, + paragraph_id: None, + block: input, + }, + }); + }, + _ => {}, + } + self.binding.rebuild_indexes(); + Ok(()) + } + + fn apply_list_diff(&mut self, target: &ContainerID, delta: &[ListDiffItem], patches: &mut Vec) -> Result<()> { + let blocks = self.doc.get_movable_list(BLOCKS); + if *target != blocks.id() { + return Ok(()); + } + + let mut row_ix = 0usize; + for item in delta { + match item { + ListDiffItem::Retain { retain } => row_ix = row_ix.saturating_add(*retain), + ListDiffItem::Delete { delete } => { + let count = (*delete).min(self.binding.rows.len().saturating_sub(row_ix)); + if count == 0 { + continue; + } + for _ in 0..count { + self + .binding + .remove_row(row_ix) + .context("DocBinding row disappeared during remote block delete")?; + } + patches.push(CollabPatch::DeleteBlocks { row: row_ix, count }); + }, + ListDiffItem::Insert { insert, is_move } => { + for value in insert { + let map = map_from_insert(value)?; + if *is_move && let Some(from) = self.binding.by_container.get(&map.id()).copied() { + patches.push(CollabPatch::MoveBlock { from, to: row_ix }); + self.binding.move_row(from, row_ix); + row_ix += 1; + continue; + } + + let input = input_block_from_container(&map)?; + let structural = structural_block_for_insert(&input); + let row = binding_row_from_insert(map, &input, structural.block_id, structural.paragraph_id)?; + self.binding.insert_row(row_ix, row); + patches.push(CollabPatch::InsertBlocks { + row: row_ix, + blocks: vec![structural], + }); + row_ix += 1; + } + }, + } + } + Ok(()) + } +} + +#[must_use] +pub fn should_process_event(event: &DiffEvent<'_>) -> bool { + event.triggered_by.is_import() || event.triggered_by.is_checkout() +} + +fn text_delta_to_collab_utf8(old_text: &str, delta: &[TextDelta]) -> Vec { + let mut output = Vec::with_capacity(delta.len()); + let mut old_chars = 0usize; + for item in delta { + match item { + TextDelta::Retain { retain, .. } => { + let len = utf8_len_for_char_span(old_text, old_chars, *retain); + push_text_delta(&mut output, CollabTextDelta::Retain(len)); + old_chars = old_chars.saturating_add(*retain); + }, + TextDelta::Insert { insert, .. } => push_text_delta(&mut output, CollabTextDelta::Insert(insert.len())), + TextDelta::Delete { delete } => { + let len = utf8_len_for_char_span(old_text, old_chars, *delete); + push_text_delta(&mut output, CollabTextDelta::Delete(len)); + old_chars = old_chars.saturating_add(*delete); + }, + } + } + output +} + +fn push_text_delta(output: &mut Vec, delta: CollabTextDelta) { + match delta { + CollabTextDelta::Retain(0) | CollabTextDelta::Insert(0) | CollabTextDelta::Delete(0) => {}, + CollabTextDelta::Retain(len) => match output.last_mut() { + Some(CollabTextDelta::Retain(existing)) => *existing += len, + Some(CollabTextDelta::Insert(_)) | Some(CollabTextDelta::Delete(_)) | None => output.push(CollabTextDelta::Retain(len)), + }, + CollabTextDelta::Insert(len) => match output.last_mut() { + Some(CollabTextDelta::Insert(existing)) => *existing += len, + Some(CollabTextDelta::Retain(_)) | Some(CollabTextDelta::Delete(_)) | None => output.push(CollabTextDelta::Insert(len)), + }, + CollabTextDelta::Delete(len) => match output.last_mut() { + Some(CollabTextDelta::Delete(existing)) => *existing += len, + Some(CollabTextDelta::Retain(_)) | Some(CollabTextDelta::Insert(_)) | None => output.push(CollabTextDelta::Delete(len)), + }, + } +} + +fn utf8_len_for_char_span(text: &str, start_char: usize, char_len: usize) -> usize { + let start = byte_for_char(text, start_char); + let end = byte_for_char(text, start_char.saturating_add(char_len)); + end.saturating_sub(start) +} + +fn byte_for_char(text: &str, target_char: usize) -> usize { + text + .char_indices() + .nth(target_char) + .map_or(text.len(), |(byte, _)| byte) +} + +fn map_from_insert(value: &ValueOrContainer) -> Result { + match value { + ValueOrContainer::Container(Container::Map(map)) => Ok(map.clone()), + ValueOrContainer::Value(_) | ValueOrContainer::Container(_) => bail!("remote block insert is not a map container"), + } +} + +fn structural_block_for_insert(input: &InputBlock) -> CollabStructuralBlock { + CollabStructuralBlock { + block_id: new_block_id(), + paragraph_id: matches!(input, InputBlock::Paragraph(_)).then(new_paragraph_id), + block: input.clone(), + } +} + +fn binding_row_from_insert( + map: LoroMap, + input: &InputBlock, + block_id: BlockId, + paragraph_id: Option, +) -> Result { + let text = if matches!(input, InputBlock::Paragraph(_)) { + Some(text_child(&map)?) + } else { + None + }; + Ok(BindingRow { + map, + text, + kind: block_kind_for_input(input), + block_id, + paragraph_id, + version: 0, + }) +} + +fn block_kind_for_input(input: &InputBlock) -> BlockKind { + match input { + InputBlock::Paragraph(_) => BlockKind::Paragraph, + InputBlock::Image(_) => BlockKind::Image, + InputBlock::Equation(_) => BlockKind::Equation, + InputBlock::Table(_) => BlockKind::Table, + } +} + +fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option { + let mut paragraph_ix = 0; + for (row_ix, row) in binding.rows.iter().enumerate() { + if row_ix == target_row { + return row.paragraph_id.map(|_| paragraph_ix); + } + if row.paragraph_id.is_some() { + paragraph_ix += 1; + } + } + None +} + +fn text_child(map: &LoroMap) -> Result { + match map.get(TEXT) { + Some(ValueOrContainer::Container(Container::Text(text))) => Ok(text), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => bail!("paragraph block map is missing its text container"), + } +} + +fn map_i64(map: &LoroMap, key: &str) -> Result { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => bail!("collaboration map key {key} is not an i64"), + } +} diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs new file mode 100644 index 00000000..ac1e861d --- /dev/null +++ b/crates/flowstate-collab/src/schema.rs @@ -0,0 +1,305 @@ +//! Shared Loro container schema for Flowstate rich-text documents. +//! +//! The schema uses one `LoroText` per paragraph inside a movable block list. +//! Concurrent edits after a split can therefore converge into the first half of +//! the split paragraph; this is the accepted per-paragraph CRDT trade-off from +//! the implementation plan. + +use std::{collections::HashMap, hash::BuildHasher, ops::Range}; + +use anyhow::Result; +use gpui_flowtext::{ + AssetStore, Block, HighlightStyle, InputBlock, InputBlockAlignment, InputEquationDisplay, InputImageSizing, InputParagraph, InputRun, + InputTableBlock, RunSemanticStyle, RunStyles, TextRun, input_block_from_block, +}; +use loro::{ExpandType, LoroDoc, LoroResult, LoroText, LoroValue, StyleConfig, StyleConfigMap, TextDelta, cursor::PosType}; +use serde::{Deserialize, Serialize}; + +pub const META: &str = "meta"; +pub const BLOCKS: &str = "blocks"; +pub const META_SCHEMA: &str = "schema"; +pub const META_SESSION: &str = "session"; +pub const META_TITLE: &str = "title"; +pub const SCHEMA_VERSION: i64 = 1; + +pub const KIND: &str = "kind"; +pub const KIND_PARAGRAPH: &str = "p"; +pub const KIND_IMAGE: &str = "image"; +pub const KIND_EQUATION: &str = "equation"; +pub const KIND_TABLE: &str = "table"; +pub const TEXT: &str = "text"; +pub const STYLE: &str = "style"; +pub const DATA: &str = "data"; +pub const REV: &str = "rev"; + +pub const MARK_SEMANTIC: &str = "sem"; +pub const MARK_UNDERLINE: &str = "ul"; +pub const MARK_STRIKE: &str = "strike"; +pub const MARK_HIGHLIGHT: &str = "hl"; + +pub type MarkIntervals = [Vec<(Range, LoroValue)>; 4]; + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub enum BlockPayload { + Image { + asset_id: u128, + mime: String, + original_name: Option, + content_hash: u64, + byte_len: u64, + alt_text: String, + caption: Option, + sizing: InputImageSizing, + alignment: InputBlockAlignment, + }, + Equation { + source: String, + display: InputEquationDisplay, + }, + Table(InputTableBlock), +} + +#[must_use] +pub fn payload_from_block(block: &Block, assets: &AssetStore) -> Option { + match input_block_from_block(block) { + InputBlock::Paragraph(_) => None, + InputBlock::Image(image) => { + let asset = assets.assets.get(&image.asset_id); + Some(BlockPayload::Image { + asset_id: image.asset_id.0, + mime: asset.map_or_else(String::new, |asset| asset.mime_type.to_string()), + original_name: asset.and_then(|asset| asset.original_name.as_ref().map(ToString::to_string)), + content_hash: asset.map_or(0, |asset| asset.content_hash), + byte_len: asset.map_or(0, |asset| asset.bytes.len() as u64), + alt_text: image.alt_text, + caption: image.caption, + sizing: image.sizing, + alignment: image.alignment, + }) + }, + InputBlock::Equation(equation) => Some(BlockPayload::Equation { + source: equation.source, + display: equation.display, + }), + InputBlock::Table(table) => Some(BlockPayload::Table(table)), + } +} + +#[must_use] +pub fn block_from_payload(payload: BlockPayload, _assets: &AssetStore) -> InputBlock { + match payload { + BlockPayload::Image { + asset_id, + alt_text, + caption, + sizing, + alignment, + mime: _, + original_name: _, + content_hash: _, + byte_len: _, + } => InputBlock::Image(gpui_flowtext::InputImageBlock { + asset_id: gpui_flowtext::AssetId(asset_id), + alt_text, + caption, + sizing, + alignment, + }), + BlockPayload::Equation { source, display } => InputBlock::Equation(gpui_flowtext::InputEquationBlock { + source, + syntax: gpui_flowtext::InputEquationSyntax::Latex, + display, + }), + BlockPayload::Table(table) => InputBlock::Table(table), + } +} + +#[must_use] +pub fn new_configured_doc() -> LoroDoc { + let doc = LoroDoc::new(); + configure_text_styles(&doc); + doc +} + +pub fn configure_text_styles(doc: &LoroDoc) { + let mut styles = StyleConfigMap::new(); + let no_expand = StyleConfig { expand: ExpandType::None }; + styles.insert(MARK_SEMANTIC.into(), no_expand); + styles.insert(MARK_UNDERLINE.into(), no_expand); + styles.insert(MARK_STRIKE.into(), no_expand); + styles.insert(MARK_HIGHLIGHT.into(), no_expand); + doc.config_text_style(styles); +} + +#[must_use] +pub fn loro_pos(text: &LoroText, utf8_byte: usize) -> usize { + text + .convert_pos(utf8_byte, PosType::Bytes, PosType::Unicode) + .expect("UTF-8 byte offset must be in bounds for LoroText") +} + +#[must_use] +pub fn utf8_byte(text: &LoroText, loro_pos: usize) -> usize { + text + .convert_pos(loro_pos, PosType::Unicode, PosType::Bytes) + .expect("Unicode offset must be in bounds for LoroText") +} + +pub fn set_run_styles_utf8(text: &LoroText, range: Range, styles: RunStyles) -> LoroResult<()> { + if range.is_empty() { + return Ok(()); + } + + match styles.semantic { + RunSemanticStyle::Plain => unmark_utf8(text, range.clone(), MARK_SEMANTIC)?, + RunSemanticStyle::Custom(slot) => text.mark_utf8(range.clone(), MARK_SEMANTIC, LoroValue::I64(i64::from(slot)))?, + } + if styles.direct_underline { + text.mark_utf8(range.clone(), MARK_UNDERLINE, LoroValue::Bool(true))?; + } else { + unmark_utf8(text, range.clone(), MARK_UNDERLINE)?; + } + if styles.strikethrough { + text.mark_utf8(range.clone(), MARK_STRIKE, LoroValue::Bool(true))?; + } else { + unmark_utf8(text, range.clone(), MARK_STRIKE)?; + } + if let Some(HighlightStyle::Custom(slot)) = styles.highlight { + text.mark_utf8(range, MARK_HIGHLIGHT, LoroValue::I64(i64::from(slot)))?; + } else { + unmark_utf8(text, range, MARK_HIGHLIGHT)?; + } + Ok(()) +} + +pub fn unmark_utf8(text: &LoroText, range: Range, key: &str) -> LoroResult<()> { + if range.is_empty() { + return Ok(()); + } + let start = loro_pos(text, range.start); + let end = loro_pos(text, range.end); + text.unmark(start..end, key) +} + +#[must_use] +pub fn input_runs_from_delta(delta: &[TextDelta]) -> Vec { + let mut runs: Vec = Vec::new(); + for item in delta { + let TextDelta::Insert { insert, attributes } = item else { + continue; + }; + if insert.is_empty() { + continue; + } + let styles = run_styles_from_attrs(attributes.as_ref()); + if let Some(last) = runs.last_mut() + && last.styles == styles + { + last.text.push_str(insert); + continue; + } + runs.push(InputRun { text: insert.clone(), styles }); + } + runs +} + +#[must_use] +pub fn input_paragraph_from_text(text: &LoroText, style: gpui_flowtext::ParagraphStyle) -> InputParagraph { + InputParagraph { + style, + runs: input_runs_from_delta(&text.to_delta()), + } +} + +#[must_use] +pub fn run_styles_from_attrs(attrs: Option<&HashMap>) -> RunStyles +where + S: BuildHasher, +{ + RunStyles { + semantic: attrs + .and_then(|attrs| attrs.get(MARK_SEMANTIC)) + .and_then(loro_i64) + .and_then(|slot| u8::try_from(slot).ok()) + .map_or(RunSemanticStyle::Plain, RunSemanticStyle::Custom), + direct_underline: attrs + .and_then(|attrs| attrs.get(MARK_UNDERLINE)) + .is_some_and(loro_true), + strikethrough: attrs + .and_then(|attrs| attrs.get(MARK_STRIKE)) + .is_some_and(loro_true), + highlight: attrs + .and_then(|attrs| attrs.get(MARK_HIGHLIGHT)) + .and_then(loro_i64) + .and_then(|slot| u8::try_from(slot).ok()) + .map(HighlightStyle::Custom), + } +} + +#[must_use] +pub fn mark_intervals_from_runs(runs: &[TextRun]) -> MarkIntervals { + let mut intervals: MarkIntervals = [Vec::new(), Vec::new(), Vec::new(), Vec::new()]; + let mut byte = 0; + for run in runs { + let range = byte..byte + run.len; + byte = range.end; + match run.styles.semantic { + RunSemanticStyle::Plain => {}, + RunSemanticStyle::Custom(slot) => intervals[0].push((range.clone(), LoroValue::I64(i64::from(slot)))), + } + if run.styles.direct_underline { + intervals[1].push((range.clone(), LoroValue::Bool(true))); + } + if run.styles.strikethrough { + intervals[2].push((range.clone(), LoroValue::Bool(true))); + } + if let Some(HighlightStyle::Custom(slot)) = run.styles.highlight { + intervals[3].push((range, LoroValue::I64(i64::from(slot)))); + } + } + intervals +} + +pub fn apply_mark_intervals(text: &LoroText, intervals: &MarkIntervals) -> Result<()> { + let keys = [MARK_SEMANTIC, MARK_UNDERLINE, MARK_STRIKE, MARK_HIGHLIGHT]; + for (key, intervals) in keys.into_iter().zip(intervals) { + for (range, value) in intervals { + text.mark_utf8(range.clone(), key, value.clone())?; + } + } + Ok(()) +} + +#[must_use] +pub const fn encode_paragraph_style(style: gpui_flowtext::ParagraphStyle) -> i64 { + match style { + gpui_flowtext::ParagraphStyle::Normal => -1, + gpui_flowtext::ParagraphStyle::Custom(slot) => slot as i64, + } +} + +#[must_use] +pub fn decode_paragraph_style(value: i64) -> gpui_flowtext::ParagraphStyle { + if value < 0 { + return gpui_flowtext::ParagraphStyle::Normal; + } + u8::try_from(value).map_or(gpui_flowtext::ParagraphStyle::Normal, gpui_flowtext::ParagraphStyle::Custom) +} + +fn loro_i64(value: &LoroValue) -> Option { + match value { + LoroValue::I64(value) => Some(*value), + LoroValue::Null + | LoroValue::Bool(_) + | LoroValue::Double(_) + | LoroValue::Binary(_) + | LoroValue::String(_) + | LoroValue::List(_) + | LoroValue::Map(_) + | LoroValue::Container(_) => None, + } +} + +fn loro_true(value: &LoroValue) -> bool { + matches!(value, LoroValue::Bool(true)) +} diff --git a/crates/flowstate-collab/src/self_check.rs b/crates/flowstate-collab/src/self_check.rs new file mode 100644 index 00000000..86d25c17 --- /dev/null +++ b/crates/flowstate-collab/src/self_check.rs @@ -0,0 +1,37 @@ +//! Local projection drift detection. + +use std::hash::{Hash, Hasher}; + +use gpui_flowtext::{Block, Document, paragraph_text}; +use twox_hash::XxHash64; + +use crate::schema::payload_from_block; + +#[must_use] +pub fn projection_hash(document: &Document) -> u64 { + let mut hasher = XxHash64::default(); + let mut paragraph_ix = 0; + for block in document.blocks.iter() { + match block { + Block::Paragraph(paragraph) => { + "p".hash(&mut hasher); + paragraph.style.hash(&mut hasher); + paragraph_text(document, paragraph_ix).hash(&mut hasher); + for run in ¶graph.runs { + run.len.hash(&mut hasher); + run.styles.hash(&mut hasher); + } + paragraph_ix += 1; + }, + Block::Image(_) | Block::Equation(_) | Block::Table(_) => { + std::mem::discriminant(block).hash(&mut hasher); + if let Some(payload) = payload_from_block(block, &document.assets) + && let Ok(bytes) = postcard::to_stdvec(&payload) + { + bytes.hash(&mut hasher); + } + }, + } + } + hasher.finish() +} diff --git a/crates/flowstate-collab/src/ticket.rs b/crates/flowstate-collab/src/ticket.rs new file mode 100644 index 00000000..9d10092c --- /dev/null +++ b/crates/flowstate-collab/src/ticket.rs @@ -0,0 +1,57 @@ +use iroh::EndpointAddr; +use iroh_tickets::{ParseError, Ticket}; +use serde::{Deserialize, Serialize}; + +use crate::{SessionId, proto_gossip::PROTOCOL_VERSION}; + +pub const TICKET_KIND: &str = "fscollab"; + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct SessionTicket { + pub version: u16, + pub session: SessionId, + pub inviter: EndpointAddr, + pub title: String, +} + +impl SessionTicket { + #[must_use] + pub fn new(session: SessionId, inviter: EndpointAddr, title: String) -> Self { + Self { + version: PROTOCOL_VERSION, + session, + inviter, + title, + } + } + + #[must_use] + pub const fn is_supported_version(&self) -> bool { + self.version == PROTOCOL_VERSION + } + + #[must_use] + pub fn encode_text(&self) -> String { + self.encode_string() + } + + pub fn decode_text(text: &str) -> Result { + Self::decode_string(text.trim()) + } +} + +impl Ticket for SessionTicket { + const KIND: &'static str = TICKET_KIND; + + fn encode_bytes(&self) -> Vec { + postcard::to_stdvec(self).expect("SessionTicket serialization should not fail") + } + + fn decode_bytes(bytes: &[u8]) -> Result { + let ticket: Self = postcard::from_bytes(bytes).map_err(|_| ParseError::verification_failed("invalid collaboration ticket payload"))?; + if !ticket.is_supported_version() { + return Err(ParseError::verification_failed("unsupported collaboration protocol version")); + } + Ok(ticket) + } +} diff --git a/crates/flowstate-collab/tests/local_apply.rs b/crates/flowstate-collab/tests/local_apply.rs new file mode 100644 index 00000000..27d02ea2 --- /dev/null +++ b/crates/flowstate-collab/tests/local_apply.rs @@ -0,0 +1,48 @@ +#[cfg(test)] +mod tests { + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; + use gpui_flowtext::{ + CanonicalOperation, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyle, RunStyles, insert_text_at, + paragraph_text, plain, + }; + + #[test] + fn local_apply_insert_text_matches_flowtext_edit_op() { + let mut document = gpui_flowtext::document_from_input_blocks( + DocumentTheme::default(), + vec![InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain("aé🌍\u{2028}x")], + })], + ); + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, SessionId::from_bytes([9; 32]), "local-apply", &document).expect("populate should succeed"); + let mut binding = DocBinding::build(&loro, &document).expect("binding should build"); + let paragraph = document.ids.paragraph_ids[0]; + let styles = RunStyles::default().with(RunStyle::Semantic(4)).with_strikethrough(); + + insert_text_at(&mut document, 0, "a".len(), "Z", styles); + LocalApplier { doc: &loro, binding: &mut binding } + .apply( + &document, + &[CanonicalOperation::InsertText { + paragraph, + byte: "a".len(), + text: "Z".to_string(), + styles, + }], + ) + .expect("local apply should succeed"); + + let projected = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); + assert_document_text_and_runs_eq(&document, &projected); + } + + fn assert_document_text_and_runs_eq(left: &Document, right: &Document) { + assert_eq!(left.paragraphs.len(), right.paragraphs.len()); + for ix in 0..left.paragraphs.len() { + assert_eq!(paragraph_text(left, ix), paragraph_text(right, ix)); + assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); + } + } +} diff --git a/crates/flowstate-collab/tests/projection.rs b/crates/flowstate-collab/tests/projection.rs new file mode 100644 index 00000000..dc34aa51 --- /dev/null +++ b/crates/flowstate-collab/tests/projection.rs @@ -0,0 +1,51 @@ +#[cfg(test)] +mod tests { + use flowstate_collab::binding::DocBinding; + use flowstate_collab::{SessionId, projection, schema}; + use gpui_flowtext::{ + Document, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, + RunStyles, document_from_input_blocks, paragraph_text, plain, run, + }; + + #[test] + fn projection_round_trips_text_marks_and_equation_blocks() { + let input_blocks = vec![ + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Custom(2), + runs: vec![ + plain("aé"), + run("🌍\u{2028}x", RunStyles::default().with(RunStyle::Semantic(3)).with_direct_underline()), + ], + }), + InputBlock::Equation(InputEquationBlock { + source: "E = mc^2".to_string(), + syntax: InputEquationSyntax::Latex, + display: InputEquationDisplay::Display, + }), + ]; + let original = document_from_input_blocks(DocumentTheme::default(), input_blocks); + let loro = schema::new_configured_doc(); + let session = SessionId::from_bytes([7; 32]); + + projection::populate_from_document(&loro, session, "projection-test", &original).expect("populate should succeed"); + projection::verify_lineage(&loro, session).expect("lineage should match"); + let projected = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); + let binding = DocBinding::build(&loro, &projected).expect("binding should match projected document"); + + assert_document_shape_eq(&original, &projected); + assert_eq!(binding.rows.len(), projected.blocks.len()); + } + + fn assert_document_shape_eq(left: &Document, right: &Document) { + assert_eq!(left.blocks.len(), right.blocks.len()); + assert_eq!(left.paragraphs.len(), right.paragraphs.len()); + for ix in 0..left.paragraphs.len() { + assert_eq!(paragraph_text(left, ix), paragraph_text(right, ix)); + assert_eq!(left.paragraphs[ix].style, right.paragraphs[ix].style); + assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); + } + for (left, right) in left.blocks.iter().zip(right.blocks.iter()) { + assert_eq!(std::mem::discriminant(left), std::mem::discriminant(right)); + } + } +} diff --git a/crates/flowstate/Cargo.toml b/crates/flowstate/Cargo.toml index 7368610c..a06fe97f 100644 --- a/crates/flowstate/Cargo.toml +++ b/crates/flowstate/Cargo.toml @@ -10,17 +10,20 @@ workspace = true [dependencies] anyhow = { workspace = true } +async-channel = { workspace = true } clap = { workspace = true } dirs = { workspace = true } fff-search = { workspace = true } flowstate-document = { workspace = true } flowstate-docx = { workspace = true } +flowstate-collab = { workspace = true } flowstate-flow = { workspace = true } flowstate-tub = { workspace = true } gpui = { workspace = true } gpui-component = { workspace = true } gpui-component-assets = { workspace = true } hotpath = { workspace = true } +loro = { workspace = true } memchr = "2.8.1" mimalloc = { workspace = true } rustc-hash = { workspace = true } diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs new file mode 100644 index 00000000..2de5c1d6 --- /dev/null +++ b/crates/flowstate/src/collab/mod.rs @@ -0,0 +1,390 @@ +mod session; + +use std::collections::HashMap; + +use anyhow::{Context as _, Result, anyhow, ensure}; +use async_channel::Receiver; +use flowstate_collab::{ + SessionId, + net::{NetCommand, NetEvent, TicketSeed, runtime::{self, CommandSender}}, + ticket::SessionTicket, +}; +use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global}; +use uuid::Uuid; + +use crate::rich_text_element::RichTextEditor; + +pub use session::{Attachment, CollabSession, Connectivity, DetachReason, JoinStage, JoinedDocument, SessionPhase}; + +pub struct JoinRequest { + pub session: SessionId, + pub completed: Receiver>, +} + +#[derive(Default)] +pub struct CollabManager { + runtime: Option, + sessions_by_id: HashMap>, + session_by_panel: HashMap, + endpoint_online: bool, + event_pump_started: bool, +} + +impl Global for CollabManager {} + +#[derive(Clone)] +struct CollabRuntime { + commands: CommandSender, +} + +impl CollabManager { + pub fn init(cx: &mut C) + where + C: BorrowAppContext, + { + cx.update_default_global::(|_, _| {}); + } + + pub fn start_session_for_panel( + &mut self, + panel_id: Uuid, + editor: Entity, + title: String, + cx: &mut Context, + ) -> Result + where + T: 'static, + { + if let Some(session) = self.session_by_panel.get(&panel_id).copied() { + return Ok(session); + } + + let commands = self.ensure_runtime(cx)?; + let session = SessionId::new(); + let document = editor.read(cx).document().clone(); + let collab = CollabSession::from_local_document(session, panel_id, editor, title, document, commands.clone())?; + let direct_handler = collab.direct_handler(); + let entity = cx.new(|_| collab); + entity.update(cx, |session, cx| session.attach(cx)); + + self.register_session(entity.clone(), cx); + commands + .try_send(NetCommand::RegisterDirectHandler { session, handler: direct_handler }) + .context("registering collaboration direct handler failed")?; + + let (reply_tx, reply_rx) = async_channel::bounded(1); + commands + .try_send(NetCommand::CreateSession { session, reply: reply_tx }) + .context("creating collaboration network session failed")?; + Self::finish_create_session(entity, reply_rx, cx); + Ok(session) + } + + pub fn join_session(&mut self, ticket: SessionTicket, cx: &mut Context) -> Result + where + T: 'static, + { + ensure!(ticket.is_supported_version(), "unsupported collaboration protocol version"); + ensure!(!self.sessions_by_id.contains_key(&ticket.session), "collaboration session is already open"); + + let commands = self.ensure_runtime(cx)?; + let session = ticket.session; + let inviter = ticket.inviter.id; + let collab = CollabSession::joining(session, ticket.title.clone(), commands.clone()); + let entity = cx.new(|_| collab); + self.register_session(entity.clone(), cx); + + if let Err(error) = commands.try_send(NetCommand::JoinSession { + session, + bootstrap: vec![ticket.inviter], + }) { + entity.update(cx, |session, cx| { + session.detach(DetachReason::JoinFailed(format!("joining collaboration network session failed: {error}")), cx); + }); + self.unregister_session(session); + return Err(error).context("joining collaboration network session failed"); + } + let completed = entity.update(cx, |session, cx| { + session.set_join_stage(JoinStage::Subscribing, cx); + session.begin_join_bootstrap(inviter, cx) + }); + Ok(JoinRequest { session, completed }) + } + + pub fn attach_joined_session( + &mut self, + session_id: SessionId, + panel_id: Uuid, + editor: Entity, + cx: &mut Context, + ) -> Result<()> + where + T: 'static, + { + let commands = self.ensure_runtime(cx)?; + let session = self + .sessions_by_id + .get(&session_id) + .cloned() + .ok_or_else(|| anyhow!("collaboration session is not registered"))?; + let direct_handler = session.read(cx).direct_handler(); + session.update(cx, |session, cx| session.attach_joined_editor(panel_id, editor, cx))?; + self.session_by_panel.insert(panel_id, session_id); + commands + .try_send(NetCommand::RegisterDirectHandler { + session: session_id, + handler: direct_handler, + }) + .context("registering collaboration direct handler failed")?; + Self::establish_joined_peer(session, commands, cx)?; + Ok(()) + } + + pub fn leave_session_for_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> bool + where + T: 'static, + { + let Some(session_id) = self.session_by_panel.get(&panel_id).copied() else { + return false; + }; + let Some(session) = self.sessions_by_id.get(&session_id).cloned() else { + self.unregister_session(session_id); + return false; + }; + + let _ = session.update(cx, |session, cx| session.detach(DetachReason::UserLeft, cx)); + self.unregister_session(session_id); + true + } + + pub fn request_ticket_for_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> Option>> + where + T: 'static, + { + let session_id = self.session_by_panel.get(&panel_id).copied()?; + let session = self.sessions_by_id.get(&session_id)?.clone(); + let title = session.read(cx).title().to_string(); + let commands = self.ensure_runtime(cx).ok()?; + let (ticket_tx, ticket_rx) = async_channel::bounded(1); + let (reply_tx, reply_rx) = async_channel::bounded(1); + commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }).ok()?; + + cx.spawn(async move |_, _| { + let ticket = match reply_rx.recv().await { + Ok(addr) => Ok(SessionTicket::new(session_id, addr, title)), + Err(error) => Err(anyhow!("collaboration endpoint address unavailable: {error}")), + }; + let _ = ticket_tx.send(ticket).await; + }) + .detach(); + Some(ticket_rx) + } + + pub fn session_for_panel(&self, panel_id: Uuid) -> Option> { + self + .session_by_panel + .get(&panel_id) + .and_then(|session| self.sessions_by_id.get(session)) + .cloned() + } + + pub fn phase_for_panel(&self, panel_id: Uuid, cx: &App) -> Option { + self + .session_for_panel(panel_id) + .map(|session| session.read(cx).phase().clone()) + } + + fn register_session(&mut self, session: Entity, cx: &mut Context) { + let id = session.read(cx).session_id(); + if let Some(panel_id) = session.read(cx).panel_id() { + self.session_by_panel.insert(panel_id, id); + } + self.sessions_by_id.insert(id, session); + } + + fn unregister_session(&mut self, session: SessionId) { + self.sessions_by_id.remove(&session); + self.session_by_panel.retain(|_, active| *active != session); + } + + fn ensure_runtime(&mut self, cx: &mut Context) -> Result + where + T: 'static, + { + if let Some(runtime) = &self.runtime { + return Ok(runtime.commands.clone()); + } + + let (commands, events) = runtime::start()?; + self.runtime = Some(CollabRuntime { + commands: commands.clone(), + }); + self.start_event_pump(events, cx); + Ok(commands) + } + + fn start_event_pump(&mut self, events: runtime::EventReceiver, cx: &mut Context) + where + T: 'static, + { + if self.event_pump_started { + return; + } + self.event_pump_started = true; + cx.spawn(async move |_, cx| { + while let Ok(event) = events.recv().await { + if cx + .update_global::(|manager, cx| manager.handle_net_event(event, cx)) + .is_err() + { + break; + } + } + }) + .detach(); + } + + fn handle_net_event(&mut self, event: NetEvent, cx: &mut App) { + match event { + NetEvent::EndpointOnline(online) => { + self.endpoint_online = online; + for session in self.sessions_by_id.values() { + session.update(cx, |session, cx| session.set_endpoint_online(online, cx)); + } + }, + NetEvent::Gossip { session, from, msg } => { + self.update_session(session, cx, |session, cx| session.handle_gossip(from, msg, cx)); + }, + NetEvent::NeighborUp { session, peer } => { + self.update_session(session, cx, |session, cx| session.neighbor_up(peer, cx)); + }, + NetEvent::NeighborDown { session, peer } => { + self.update_session(session, cx, |session, cx| session.neighbor_down(peer, cx)); + }, + NetEvent::GossipLagged { session } => { + self.update_session(session, cx, |session, cx| session.handle_gossip_lagged(cx)); + }, + NetEvent::SubscribeFailed { session, error } => { + let detached = self.update_session(session, cx, |session, cx| { + session.detach(DetachReason::Fatal(error), cx) + }); + if detached.unwrap_or(false) { + self.unregister_session(session); + } + }, + } + } + + fn update_session( + &mut self, + session: SessionId, + cx: &mut App, + update: impl FnOnce(&mut CollabSession, &mut Context) -> R, + ) -> Option { + self + .sessions_by_id + .get(&session) + .cloned() + .map(|session| session.update(cx, update)) + } + + fn finish_create_session(session: Entity, reply_rx: Receiver>, cx: &mut Context) + where + T: 'static, + { + cx.spawn(async move |_, cx| { + match reply_rx.recv().await { + Ok(Ok(seed)) => { + let _ = session.update(cx, |session, cx| session.establish_local_peer(&seed.inviter.id, cx)); + }, + Ok(Err(error)) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error:#}")), cx); + }); + }, + Err(error) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error}")), cx); + }); + }, + } + }) + .detach(); + } + + fn establish_joined_peer(session: Entity, commands: CommandSender, cx: &mut Context) -> Result<()> + where + T: 'static, + { + let (reply_tx, reply_rx) = async_channel::bounded(1); + commands + .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) + .context("requesting collaboration endpoint address failed")?; + cx.spawn(async move |_, cx| { + match reply_rx.recv().await { + Ok(addr) => { + let _ = session.update(cx, |session, cx| session.establish_local_peer(&addr.id, cx)); + }, + Err(error) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error}")), cx); + }); + }, + } + }) + .detach(); + Ok(()) + } +} + +pub fn init(cx: &mut C) +where + C: BorrowAppContext, +{ + CollabManager::init(cx); +} + +pub fn start_session_for_panel( + panel_id: Uuid, + editor: Entity, + title: String, + cx: &mut Context, +) -> Result +where + T: 'static, +{ + cx.update_default_global::(|manager, cx| manager.start_session_for_panel(panel_id, editor, title, cx)) +} + +pub fn join_session(ticket: SessionTicket, cx: &mut Context) -> Result +where + T: 'static, +{ + cx.update_default_global::(|manager, cx| manager.join_session(ticket, cx)) +} + +pub fn attach_joined_session( + session_id: SessionId, + panel_id: Uuid, + editor: Entity, + cx: &mut Context, +) -> Result<()> +where + T: 'static, +{ + cx.update_default_global::(|manager, cx| manager.attach_joined_session(session_id, panel_id, editor, cx)) +} + +pub fn leave_session_for_panel(panel_id: Uuid, cx: &mut Context) -> bool +where + T: 'static, +{ + cx.update_default_global::(|manager, cx| manager.leave_session_for_panel(panel_id, cx)) +} + +pub fn request_ticket_for_panel(panel_id: Uuid, cx: &mut Context) -> Option>> +where + T: 'static, +{ + cx.update_default_global::(|manager, cx| manager.request_ticket_for_panel(panel_id, cx)) +} diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs new file mode 100644 index 00000000..a81dd7dd --- /dev/null +++ b/crates/flowstate/src/collab/session.rs @@ -0,0 +1,768 @@ +use std::{ + cmp::Ordering, + collections::HashSet, + rc::Rc, + sync::{Arc, Mutex}, + time::Instant, +}; + +use anyhow::{Context as _, Result, anyhow, bail}; +use flowstate_collab::{ + SessionId, + binding::DocBinding, + local_apply::LocalApplier, + net::{NetCommand, PublishPayload, runtime::CommandSender, direct::{DirectServeRequest, DirectSessionHandler}}, + presence::{PresenceState, PresenceStore}, + projection, + proto_direct::AssetBytes, + proto_gossip::GossipMsg, + remote_apply::RemoteApplier, + schema, +}; +use gpui::{Context, Entity, Subscription}; +use loro::{ExportMode, LoroDoc, Subscription as LoroSubscription, VersionVector, event::Subscriber}; +use uuid::Uuid; + +use crate::app_settings::load_document_theme; +use crate::rich_text_element::{AssetId, CollabPatch, Document, EditorEvent, RichTextEditor, UndoRedirect}; + +#[derive(Clone, Debug)] +pub enum SessionPhase { + Creating, + Joining(JoinStage), + Attached(Attachment), + Detached(DetachReason), +} + +#[derive(Clone, Debug)] +pub struct Attachment { + pub connectivity: Connectivity, + pub peers_present: usize, +} + +#[derive(Clone, Debug)] +pub enum Connectivity { + Online, + Offline { since: Instant, retries: u32 }, +} + +#[derive(Clone, Debug)] +pub enum JoinStage { + Resolving, + Subscribing, + FetchingSnapshot { got: u64, total: Option }, + Building, +} + +#[derive(Clone, Debug)] +pub enum DetachReason { + UserLeft, + JoinFailed(String), + Fatal(String), +} + +pub struct JoinedDocument { + pub session: SessionId, + pub title: String, + pub document: Document, +} + +pub struct CollabSession { + session: SessionId, + title: String, + phase: SessionPhase, + doc: Option, + binding: Option, + editor: Option>, + panel_id: Option, + pending_remote_patches: Vec, + pending_remote_updates: Vec>, + presence: Option, + net_tx: CommandSender, + direct_tx: async_channel::Sender, + direct_rx: async_channel::Receiver, + editor_subscriptions: Vec, + loro_subscriptions: Vec, + neighbors: HashSet, + update_pull_in_flight: bool, + direct_pump_started: bool, + local_update_publish_attached: bool, +} + +impl CollabSession { + pub fn from_local_document( + session: SessionId, + panel_id: Uuid, + editor: Entity, + title: String, + document: Document, + net_tx: CommandSender, + ) -> Result { + let doc = schema::new_configured_doc(); + projection::populate_from_document(&doc, session, &title, &document)?; + let binding = DocBinding::build(&doc, &document)?; + let (direct_tx, direct_rx) = async_channel::unbounded(); + + Ok(Self { + session, + title, + phase: SessionPhase::Creating, + doc: Some(doc), + binding: Some(binding), + editor: Some(editor), + panel_id: Some(panel_id), + pending_remote_patches: Vec::new(), + pending_remote_updates: Vec::new(), + presence: None, + net_tx, + direct_tx, + direct_rx, + editor_subscriptions: Vec::new(), + loro_subscriptions: Vec::new(), + neighbors: HashSet::new(), + update_pull_in_flight: false, + direct_pump_started: false, + local_update_publish_attached: false, + }) + } + + pub fn joining(session: SessionId, title: String, net_tx: CommandSender) -> Self { + let (direct_tx, direct_rx) = async_channel::unbounded(); + Self { + session, + title, + phase: SessionPhase::Joining(JoinStage::Resolving), + doc: None, + binding: None, + editor: None, + panel_id: None, + pending_remote_patches: Vec::new(), + pending_remote_updates: Vec::new(), + presence: None, + net_tx, + direct_tx, + direct_rx, + editor_subscriptions: Vec::new(), + loro_subscriptions: Vec::new(), + neighbors: HashSet::new(), + update_pull_in_flight: false, + direct_pump_started: false, + local_update_publish_attached: false, + } + } + + pub fn session_id(&self) -> SessionId { + self.session + } + + pub fn panel_id(&self) -> Option { + self.panel_id + } + + pub fn title(&self) -> &str { + &self.title + } + + pub fn phase(&self) -> &SessionPhase { + &self.phase + } + + pub fn direct_handler(&self) -> DirectSessionHandler { + DirectSessionHandler::new(self.direct_tx.clone()) + } + + pub fn set_join_stage(&mut self, stage: JoinStage, cx: &mut Context) { + self.phase = SessionPhase::Joining(stage); + cx.notify(); + } + + pub fn begin_join_bootstrap( + &mut self, + inviter: flowstate_collab::ids::PeerId, + cx: &mut Context, + ) -> async_channel::Receiver> { + let (result_tx, result_rx) = async_channel::bounded(1); + self.phase = SessionPhase::Joining(JoinStage::FetchingSnapshot { got: 0, total: None }); + cx.notify(); + + let (reply_tx, reply_rx) = async_channel::bounded(1); + if let Err(error) = self.net_tx.try_send(NetCommand::PullSnapshot { + session: self.session, + from: inviter, + reply: reply_tx, + }) { + let detail = format!("requesting collaboration snapshot failed: {error}"); + self.detach(DetachReason::JoinFailed(detail.clone()), cx); + let _ = result_tx.try_send(Err(anyhow!(detail))); + return result_rx; + } + + cx.spawn(async move |session, cx| { + let joined = match reply_rx.recv().await { + Ok(Ok(bytes)) => session + .update(cx, |session, cx| match session.finish_join_snapshot(&bytes, cx) { + Ok(joined) => Ok(joined), + Err(error) => { + let detail = format!("building collaboration document failed: {error:#}"); + session.detach(DetachReason::JoinFailed(detail), cx); + Err(error.context("building collaboration document failed")) + }, + }) + .unwrap_or_else(|error| Err(anyhow!("collaboration join session disappeared: {error}"))), + Ok(Err(error)) => { + let detail = format!("pulling collaboration snapshot failed: {error:#}"); + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::JoinFailed(detail.clone()), cx); + }); + Err(anyhow!(detail)) + }, + Err(error) => { + let detail = format!("collaboration snapshot reply channel closed: {error}"); + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::JoinFailed(detail.clone()), cx); + }); + Err(anyhow!(detail)) + }, + }; + let _ = result_tx.send(joined).await; + }) + .detach(); + + result_rx + } + + pub fn attach_joined_editor( + &mut self, + panel_id: Uuid, + editor: Entity, + cx: &mut Context, + ) -> Result<()> { + if self.doc.is_none() || self.binding.is_none() { + bail!("collaboration snapshot has not finished loading"); + } + + self.panel_id = Some(panel_id); + self.editor = Some(editor); + self.attach(cx); + + let pending_updates = std::mem::take(&mut self.pending_remote_updates); + for update in pending_updates { + self.import_update_bytes(&update, cx)?; + } + self.flush_pending_remote_patches(cx); + self.publish_digest(); + cx.notify(); + Ok(()) + } + + pub fn attach(&mut self, cx: &mut Context) { + self.attach_editor_hooks(cx); + self.attach_loro_publish_hook(); + self.attach_direct_request_pump(cx); + } + + pub fn establish_local_peer(&mut self, peer: &flowstate_collab::ids::PeerId, cx: &mut Context) { + if self.presence.is_none() { + let presence = PresenceStore::new(peer); + let session = self.session; + let net_tx = self.net_tx.clone(); + self.loro_subscriptions.push(presence.subscribe_local_updates(move |bytes| { + let _ = net_tx.try_send(NetCommand::Publish { + session, + payload: PublishPayload::Presence(bytes.clone()), + }); + true + })); + self.presence = Some(presence); + } + self.refresh_own_presence(cx); + self.phase = SessionPhase::Attached(Attachment { + connectivity: Connectivity::Online, + peers_present: self.peers_present(), + }); + self.publish_presence_snapshot(); + self.publish_digest(); + cx.notify(); + } + + fn finish_join_snapshot(&mut self, snapshot: &[u8], cx: &mut Context) -> Result { + self.phase = SessionPhase::Joining(JoinStage::Building); + cx.notify(); + + let doc = schema::new_configured_doc(); + doc + .import_with(snapshot, "remote") + .context("importing collaboration snapshot failed")?; + projection::verify_lineage(&doc, self.session)?; + let document = projection::document_from_loro(&doc, load_document_theme())?; + let binding = DocBinding::build(&doc, &document)?; + + self.doc = Some(doc); + self.binding = Some(binding); + Ok(JoinedDocument { + session: self.session, + title: self.title.clone(), + document, + }) + } + + pub fn detach(&mut self, reason: DetachReason, cx: &mut Context) -> bool { + if matches!(self.phase, SessionPhase::Detached(_)) { + return false; + } + + if let Some(presence) = &self.presence { + presence.delete_self(); + self.publish_presence_bytes(presence.encode_self()); + } + let _ = self.net_tx.try_send(NetCommand::LeaveSession { session: self.session }); + self.flush_pending_remote_patches(cx); + + if let Some(editor) = self.editor.clone() { + editor.update(cx, |editor, cx| { + editor.set_collab_undo_redirect(None); + editor.set_collab_capture(false); + let _ = editor.take_pending_collab_edits(); + editor.set_external_carets(Vec::new(), cx); + }); + } + + self.editor_subscriptions.clear(); + self.loro_subscriptions.clear(); + self.presence = None; + self.binding = None; + self.doc = None; + self.pending_remote_patches.clear(); + self.pending_remote_updates.clear(); + self.neighbors.clear(); + self.local_update_publish_attached = false; + self.phase = SessionPhase::Detached(reason); + cx.notify(); + true + } + + pub fn flush_local_edits(&mut self, editor: Entity, cx: &mut Context) -> Result<()> { + if matches!(self.phase, SessionPhase::Detached(_) | SessionPhase::Joining(_)) { + return Ok(()); + } + + let document = editor.read(cx).document().clone(); + let edits = editor.update(cx, |editor, _| editor.take_pending_collab_edits()); + let Some(doc) = &self.doc else { + return Ok(()); + }; + let Some(binding) = &mut self.binding else { + return Ok(()); + }; + + for edit in edits { + if edit.operations.is_empty() { + continue; + } + LocalApplier { doc, binding }.apply(&document, &edit.operations)?; + } + Ok(()) + } + + pub fn handle_gossip(&mut self, from: flowstate_collab::ids::PeerId, msg: GossipMsg, cx: &mut Context) { + let result = match msg { + GossipMsg::Update(bytes) => self.import_update_bytes(&bytes, cx), + GossipMsg::UpdateAvailable { blob, .. } => { + self.pull_blob(from, blob, cx); + Ok(()) + }, + GossipMsg::Presence(bytes) => { + self.apply_presence(&bytes, cx); + Ok(()) + }, + GossipMsg::Digest { session, vv } => self.handle_digest(from, session, &vv, cx), + }; + if let Err(error) = result { + self.detach(DetachReason::Fatal(format!("collaboration update failed: {error:#}")), cx); + } + } + + pub fn neighbor_up(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { + self.neighbors.insert(peer); + if let SessionPhase::Attached(attachment) = &mut self.phase { + attachment.connectivity = Connectivity::Online; + } + self.publish_digest(); + cx.notify(); + } + + pub fn neighbor_down(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { + self.neighbors.remove(&peer); + cx.notify(); + } + + pub fn handle_gossip_lagged(&mut self, cx: &mut Context) { + self.publish_digest(); + if let Some(peer) = self.neighbors.iter().next().copied() { + let vv = self.doc.as_ref().map(|doc| doc.oplog_vv().encode()).unwrap_or_default(); + self.pull_updates(peer, vv, cx); + } + } + + pub fn set_endpoint_online(&mut self, online: bool, cx: &mut Context) { + if let SessionPhase::Attached(attachment) = &mut self.phase { + attachment.connectivity = if online { + Connectivity::Online + } else { + Connectivity::Offline { since: Instant::now(), retries: 0 } + }; + cx.notify(); + } + } + + pub fn import_update_bytes(&mut self, bytes: &[u8], cx: &mut Context) -> Result<()> { + if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() { + self.pending_remote_updates.push(bytes.to_vec()); + return Ok(()); + } + + let editor = self.editor.clone().context("collaboration session has no editor")?; + let document = Arc::new(editor.read(cx).document().clone()); + let doc = self.doc.clone().context("collaboration session has no Loro document")?; + let binding = self.binding.take().context("collaboration session has no document binding")?; + let binding = Arc::new(Mutex::new(binding)); + let patches = Arc::new(Mutex::new(Vec::::new())); + let sub = self.diff_subscription(doc.clone(), document, binding.clone(), patches.clone()); + + let import_result = doc.import_with(bytes, "remote"); + drop(sub); + self.binding = Some(take_mutex_value(binding, "document binding")?); + let patches = take_mutex_value(patches, "remote patches")?; + import_result.context("importing collaboration update failed")?; + self.apply_or_queue_patches(patches, cx); + Ok(()) + } + + fn attach_editor_hooks(&mut self, cx: &mut Context) { + if !self.editor_subscriptions.is_empty() { + return; + } + let Some(editor) = self.editor.clone() else { + return; + }; + + editor.update(cx, |editor, _| { + editor.set_collab_capture(true); + editor.set_collab_undo_redirect(Some(Rc::new(|_redirect: UndoRedirect| { + // TODO(Flowstate P2P §8/M3): replace this guard with Loro UndoManager undo/redo routing. + }))); + }); + + self.editor_subscriptions.push(cx.observe(&editor, |session, editor, cx| { + if let Err(error) = session.flush_local_edits(editor.clone(), cx) { + session.detach(DetachReason::Fatal(format!("capturing local collaboration edit failed: {error:#}")), cx); + return; + } + session.flush_pending_remote_patches(cx); + })); + + self.editor_subscriptions.push(cx.subscribe(&editor, |session, _, event: &EditorEvent, cx| { + if matches!(event, EditorEvent::SelectionChanged { .. }) { + session.refresh_own_presence(cx); + } + })); + } + + fn attach_loro_publish_hook(&mut self) { + if self.doc.is_none() || self.local_update_publish_attached { + return; + } + let Some(doc) = &self.doc else { + return; + }; + let session = self.session; + let net_tx = self.net_tx.clone(); + self.loro_subscriptions.push(doc.subscribe_local_update(Box::new(move |bytes| { + let _ = net_tx.try_send(NetCommand::Publish { + session, + payload: PublishPayload::Update(bytes.clone()), + }); + true + }))); + self.local_update_publish_attached = true; + } + + fn attach_direct_request_pump(&mut self, cx: &mut Context) { + if self.direct_pump_started { + return; + } + self.direct_pump_started = true; + let requests = self.direct_rx.clone(); + cx.spawn(async move |session, cx| { + while let Ok(request) = requests.recv().await { + if session.update(cx, |session, cx| session.handle_direct_request(request, cx)).is_err() { + break; + } + } + }) + .detach(); + } + + fn handle_direct_request(&mut self, request: DirectServeRequest, cx: &mut Context) { + match request { + DirectServeRequest::Snapshot { reply } => { + let _ = reply.try_send(self.snapshot_bytes()); + }, + DirectServeRequest::Updates { have_vv, reply } => { + let _ = reply.try_send(self.update_bytes(&have_vv)); + }, + DirectServeRequest::Asset { asset, reply } => { + let _ = reply.try_send(self.asset_bytes(asset, cx)); + }, + } + } + + fn snapshot_bytes(&self) -> Result> { + self + .doc + .as_ref() + .context("collaboration session is not attached")? + .export(ExportMode::Snapshot) + .context("exporting collaboration snapshot failed") + } + + fn update_bytes(&self, have_vv: &[u8]) -> Result> { + let vv = VersionVector::decode(have_vv).context("decoding collaboration version vector failed")?; + self + .doc + .as_ref() + .context("collaboration session is not attached")? + .export(ExportMode::updates(&vv)) + .context("exporting collaboration updates failed") + } + + fn asset_bytes(&self, asset: u128, cx: &mut Context) -> Result { + let editor = self.editor.as_ref().context("collaboration session has no editor")?; + let bytes = editor + .read(cx) + .document() + .assets + .assets + .get(&AssetId(asset)) + .map(|record| record.bytes.as_ref().clone()) + .ok_or_else(|| anyhow!("collaboration asset {asset} is not available"))?; + Ok(AssetBytes { bytes }) + } + + fn diff_subscription( + &self, + doc: LoroDoc, + document: Arc, + binding: Arc>, + patches: Arc>>, + ) -> LoroSubscription { + let subscribed_doc = doc.clone(); + let callback: Subscriber = Arc::new(move |event| { + let produced = { + let mut binding = match binding.lock() { + Ok(binding) => binding, + Err(error) => { + eprintln!("flowstate collab binding lock failed: {error}"); + return; + }, + }; + let result = { + let mut applier = RemoteApplier { + doc: &doc, + binding: &mut binding, + }; + applier.apply_event(&document, &event) + }; + drop(binding); + result + }; + match produced { + Ok(mut produced) => { + if let Ok(mut patches) = patches.lock() { + patches.append(&mut produced); + } + }, + Err(error) => eprintln!("flowstate collab remote apply failed: {error:#}"), + } + }); + subscribed_doc.subscribe_root(callback) + } + + fn apply_or_queue_patches(&mut self, mut patches: Vec, cx: &mut Context) { + if patches.is_empty() { + return; + } + self.pending_remote_patches.append(&mut patches); + self.flush_pending_remote_patches(cx); + } + + fn flush_pending_remote_patches(&mut self, cx: &mut Context) -> bool { + let Some(editor) = self.editor.clone() else { + return false; + }; + if self.pending_remote_patches.is_empty() || editor.read(cx).collab_apply_deferred() { + return false; + } + let patches = std::mem::take(&mut self.pending_remote_patches); + editor.update(cx, |editor, cx| editor.apply_collab_patches(&patches, cx)); + true + } + + fn handle_digest( + &mut self, + from: flowstate_collab::ids::PeerId, + digest_session: SessionId, + vv: &[u8], + cx: &mut Context, + ) -> Result<()> { + if digest_session != self.session { + bail!("received a collaboration digest for a different session"); + } + let Some(doc) = &self.doc else { + return Ok(()); + }; + let sender_vv = VersionVector::decode(vv).context("decoding collaboration digest failed")?; + let our_vv = doc.oplog_vv(); + match sender_vv.partial_cmp(&our_vv) { + Some(Ordering::Greater) | None => self.pull_updates(from, our_vv.encode(), cx), + Some(Ordering::Equal | Ordering::Less) => {}, + } + Ok(()) + } + + fn pull_updates(&mut self, from: flowstate_collab::ids::PeerId, our_vv: Vec, cx: &mut Context) { + if self.update_pull_in_flight { + return; + } + self.update_pull_in_flight = true; + let (reply_tx, reply_rx) = async_channel::bounded(1); + let send_result = self.net_tx.try_send(NetCommand::PullUpdates { + session: self.session, + from, + our_vv, + reply: reply_tx, + }); + if send_result.is_err() { + self.update_pull_in_flight = false; + return; + } + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + session.update_pull_in_flight = false; + if let Ok(Ok(bytes)) = result + && let Err(error) = session.import_update_bytes(&bytes, cx) + { + session.detach(DetachReason::Fatal(format!("pulling collaboration updates failed: {error:#}")), cx); + } + }); + }) + .detach(); + } + + fn pull_blob(&mut self, from: flowstate_collab::ids::PeerId, blob: flowstate_collab::BlobId, cx: &mut Context) { + let (reply_tx, reply_rx) = async_channel::bounded(1); + if self + .net_tx + .try_send(NetCommand::PullBlob { + session: self.session, + from, + blob, + reply: reply_tx, + }) + .is_err() + { + return; + } + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + if let Ok(Ok(bytes)) = result + && let Err(error) = session.import_update_bytes(&bytes, cx) + { + session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + } + }); + }) + .detach(); + } + + fn publish_digest(&self) { + if let Some(doc) = &self.doc { + let _ = self.net_tx.try_send(NetCommand::Publish { + session: self.session, + payload: PublishPayload::Digest { vv: doc.oplog_vv().encode() }, + }); + } + } + + fn apply_presence(&mut self, bytes: &[u8], cx: &mut Context) { + if let Some(presence) = &self.presence { + if let Err(error) = presence.apply(bytes) { + eprintln!("flowstate collab presence update failed: {error:#}"); + } + self.refresh_peer_count(); + cx.notify(); + } + } + + fn refresh_own_presence(&mut self, _cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let state = PresenceState { + name: default_presence_name(), + selection: None, + }; + if let Err(error) = presence.set_self(&state) { + eprintln!("flowstate collab presence encode failed: {error:#}"); + } + self.refresh_peer_count(); + } + + fn publish_presence_snapshot(&self) { + if let Some(presence) = &self.presence { + self.publish_presence_bytes(presence.encode_all()); + } + } + + fn publish_presence_bytes(&self, bytes: Vec) { + if bytes.is_empty() { + return; + } + let _ = self.net_tx.try_send(NetCommand::Publish { + session: self.session, + payload: PublishPayload::Presence(bytes), + }); + } + + fn refresh_peer_count(&mut self) { + let peers_present = self.peers_present(); + if let SessionPhase::Attached(attachment) = &mut self.phase { + attachment.peers_present = peers_present; + } + } + + fn peers_present(&self) -> usize { + self + .presence + .as_ref() + .map_or(0, |presence| presence.roster().len().saturating_sub(1)) + } +} + +fn take_mutex_value(value: Arc>, label: &str) -> Result { + match Arc::try_unwrap(value) { + Ok(mutex) => mutex + .into_inner() + .map_err(|error| anyhow!("collaboration {label} lock was poisoned: {error}")), + Err(_) => Err(anyhow!("collaboration {label} is still referenced")), + } +} + +fn default_presence_name() -> String { + std::env::var("FLOWSTATE_COLLAB_NAME") + .or_else(|_| std::env::var("USER")) + .or_else(|_| std::env::var("USERNAME")) + .unwrap_or_else(|_| "Flowstate user".to_string()) +} diff --git a/crates/flowstate/src/commands.rs b/crates/flowstate/src/commands.rs index a629516d..02e5b684 100644 --- a/crates/flowstate/src/commands.rs +++ b/crates/flowstate/src/commands.rs @@ -76,6 +76,10 @@ pub enum CommandId { OpenDocument, OpenDemoDocument, CloseDocument, + StartCollaboration, + CopyCollaborationTicket, + JoinCollaborationFromClipboard, + LeaveCollaboration, FindInDocument, ToggleRibbon, NextTab, @@ -201,6 +205,10 @@ pub const COMMAND_SPECS: &[CommandSpec] = &[ CommandSpec::new(CommandId::OpenDocument, "Open Document", APP, &[]), CommandSpec::new(CommandId::OpenDemoDocument, "Open Demo Document", APP, &[]), CommandSpec::new(CommandId::CloseDocument, "Close Document", APP, &[]), + CommandSpec::new(CommandId::StartCollaboration, "Start Collaboration", APP, &[]), + CommandSpec::new(CommandId::CopyCollaborationTicket, "Copy Collaboration Invite", APP, &[]), + CommandSpec::new(CommandId::JoinCollaborationFromClipboard, "Join Collaboration from Clipboard", APP, &[]), + CommandSpec::new(CommandId::LeaveCollaboration, "Leave Collaboration", APP, &[]), CommandSpec::new(CommandId::FindInDocument, "Find in Document", APP, &["cmd-f", "ctrl-f"]), CommandSpec::new(CommandId::ToggleRibbon, "Toggle Ribbon", APP, &[]), CommandSpec::new(CommandId::NextTab, "Next Tab", APP, &["ctrl-tab"]), diff --git a/crates/flowstate/src/commands/keymap.rs b/crates/flowstate/src/commands/keymap.rs index 4de74ee0..6ed22b1d 100644 --- a/crates/flowstate/src/commands/keymap.rs +++ b/crates/flowstate/src/commands/keymap.rs @@ -151,6 +151,10 @@ fn action_for_command(command: CommandId) -> Option> { | CommandId::OpenDocument | CommandId::OpenDemoDocument | CommandId::CloseDocument + | CommandId::StartCollaboration + | CommandId::CopyCollaborationTicket + | CommandId::JoinCollaborationFromClipboard + | CommandId::LeaveCollaboration | CommandId::ToggleRibbon | CommandId::NextTab | CommandId::PreviousTab diff --git a/crates/flowstate/src/lib.rs b/crates/flowstate/src/lib.rs index 1b6957bd..b65baa3c 100644 --- a/crates/flowstate/src/lib.rs +++ b/crates/flowstate/src/lib.rs @@ -6,6 +6,7 @@ pub mod app; pub mod app_settings; +pub mod collab; pub mod commands; pub mod docx_conversion; pub mod file_search; diff --git a/crates/flowstate/src/rich_text_element/mod.rs b/crates/flowstate/src/rich_text_element/mod.rs index 16f42a9e..3bece707 100644 --- a/crates/flowstate/src/rich_text_element/mod.rs +++ b/crates/flowstate/src/rich_text_element/mod.rs @@ -64,6 +64,10 @@ pub fn flowstate_command_to_rich_text(command: CommandId) -> Option) -> Option { + let panel_id = self.active_document_id?; + let editor = self.active_editor.clone()?; + let title = self + .document_panels + .iter() + .find(|panel| panel.read(cx).id() == panel_id) + .map(|panel| panel.read(cx).title_text().to_string()) + .unwrap_or_else(|| "Shared document".to_string()); + + match crate::collab::start_session_for_panel(panel_id, editor, title, cx) { + Ok(session) => Some(session), + Err(error) => { + eprintln!("starting collaboration session failed: {error:#}"); + None + }, + } + } + + pub fn request_active_collaboration_ticket( + &mut self, + cx: &mut Context, + ) -> Option>> { + crate::collab::request_ticket_for_panel(self.active_document_id?, cx) + } + + pub fn copy_active_collaboration_ticket(&mut self, window: &mut Window, cx: &mut Context) -> bool { + let Some(panel_id) = self.active_document_id else { + return false; + }; + let mut ticket_rx = crate::collab::request_ticket_for_panel(panel_id, cx); + if ticket_rx.is_none() { + if self.start_collaboration_on_active_document(cx).is_none() { + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Share failed", + Some("The active document could not be shared."), + &[PromptButton::ok("Ok")], + cx, + )); + return true; + } + ticket_rx = crate::collab::request_ticket_for_panel(panel_id, cx); + } + + let Some(ticket_rx) = ticket_rx else { + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Share failed", + Some("The collaboration ticket is not available yet."), + &[PromptButton::ok("Ok")], + cx, + )); + return true; + }; + + let window_handle = window.window_handle(); + cx.spawn(async move |_, cx| { + let result = ticket_rx.recv().await; + let _ = window_handle.update(cx, |_, window, cx| match result { + Ok(Ok(ticket)) => { + cx.write_to_clipboard(gpui::ClipboardItem::new_string(ticket.encode_text())); + std::mem::drop(window.prompt( + PromptLevel::Info, + "Invite copied", + Some("The collaboration invite ticket was copied to the clipboard."), + &[PromptButton::ok("Ok")], + cx, + )); + }, + Ok(Err(error)) => { + let detail = format!("Creating collaboration invite failed: {error:#}"); + std::mem::drop(window.prompt(PromptLevel::Critical, "Share failed", Some(&detail), &[PromptButton::ok("Ok")], cx)); + }, + Err(error) => { + let detail = format!("Creating collaboration invite failed: {error}"); + std::mem::drop(window.prompt(PromptLevel::Critical, "Share failed", Some(&detail), &[PromptButton::ok("Ok")], cx)); + }, + }); + }) + .detach(); + true + } + + pub fn join_collaboration_from_clipboard(&mut self, window: &mut Window, cx: &mut Context) -> bool { + let Some(item) = cx.read_from_clipboard() else { + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Join failed", + Some("The clipboard does not contain a collaboration invite ticket."), + &[PromptButton::ok("Ok")], + cx, + )); + return true; + }; + let Some(text) = item.text() else { + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Join failed", + Some("The clipboard does not contain text."), + &[PromptButton::ok("Ok")], + cx, + )); + return true; + }; + let ticket = match flowstate_collab::ticket::SessionTicket::decode_text(&text) { + Ok(ticket) => ticket, + Err(error) => { + let detail = format!("The clipboard text is not a valid collaboration invite: {error:#}"); + std::mem::drop(window.prompt(PromptLevel::Critical, "Join failed", Some(&detail), &[PromptButton::ok("Ok")], cx)); + return true; + }, + }; + self.join_collaboration_session(ticket, window, cx).is_some() + } + + pub fn join_collaboration_session( + &mut self, + ticket: flowstate_collab::ticket::SessionTicket, + window: &mut Window, + cx: &mut Context, + ) -> Option { + let request = match crate::collab::join_session(ticket, cx) { + Ok(request) => request, + Err(error) => { + eprintln!("joining collaboration session failed: {error:#}"); + return None; + }, + }; + let session = request.session; + let completed = request.completed; + let window_handle = window.window_handle(); + + cx.spawn(async move |workspace, cx| { + let result = completed.recv().await; + let _ = window_handle.update(cx, |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| { + match result { + Ok(Ok(joined)) => { + let panel = workspace.add_joined_collaboration_panel(joined.document, joined.title, window, cx); + let panel_id = panel.read(cx).id(); + let editor = panel.read(cx).editor(); + if let Err(error) = crate::collab::attach_joined_session(joined.session, panel_id, editor, cx) { + let detail = format!("Joined document opened, but collaboration attachment failed: {error:#}"); + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Join failed", + Some(&detail), + &[PromptButton::ok("Ok")], + cx, + )); + } + }, + Ok(Err(error)) => { + let detail = format!("Joining collaboration session failed: {error:#}"); + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Join failed", + Some(&detail), + &[PromptButton::ok("Ok")], + cx, + )); + }, + Err(error) => { + let detail = format!("Joining collaboration session failed: {error}"); + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Join failed", + Some(&detail), + &[PromptButton::ok("Ok")], + cx, + )); + }, + } + }); + }); + }) + .detach(); + + Some(session) + } + + fn add_joined_collaboration_panel( + &mut self, + document: Document, + title: String, + window: &mut Window, + cx: &mut Context, + ) -> Entity { + let panel = self.create_document_panel(document, None, Some(title), window, cx); + self.persist_temporary_workspace_session(cx); + cx.notify(); + panel + } + + pub fn leave_collaboration_on_active_document(&mut self, cx: &mut Context) -> bool { + self + .active_document_id + .is_some_and(|panel_id| crate::collab::leave_session_for_panel(panel_id, cx)) + } + + pub fn leave_all_collaboration_sessions(&mut self, cx: &mut Context) { + let panel_ids = self + .document_panels + .iter() + .map(|panel| panel.read(cx).id()) + .collect::>(); + for panel_id in panel_ids { + crate::collab::leave_session_for_panel(panel_id, cx); + } + } +} diff --git a/crates/flowstate/src/workspace/workspace/documents.rs b/crates/flowstate/src/workspace/workspace/documents.rs index 7255ce76..8e225b38 100644 --- a/crates/flowstate/src/workspace/workspace/documents.rs +++ b/crates/flowstate/src/workspace/workspace/documents.rs @@ -1,6 +1,7 @@ #[hotpath::measure_all] impl Workspace { pub fn new(initial_path: Option, window: &mut Window, cx: &mut Context) -> Self { + crate::collab::init(cx); let zoom_slider = cx.new(|_| { SliderState::new() .min(25.0) @@ -214,6 +215,7 @@ impl Workspace { } pub fn remove_document_panel(&mut self, panel_id: Uuid, _: &mut Window, cx: &mut Context) { + crate::collab::leave_session_for_panel(panel_id, cx); let closing_active_document = self.active_document_id == Some(panel_id); if let Some(panel) = self .document_panels @@ -734,6 +736,7 @@ impl Workspace { fn request_close_window(&mut self, window: &mut Window, cx: &mut Context) { let dirty_panels = self.dirty_panels(cx); if dirty_panels.is_empty() { + self.leave_all_collaboration_sessions(cx); window.remove_window(); return; } @@ -752,7 +755,7 @@ impl Workspace { ); let window_handle = window.window_handle(); - cx.spawn(async move |_, cx| { + cx.spawn(async move |workspace, cx| { let should_close = match answer.await { Ok(0) => { let mut ok = true; @@ -782,7 +785,10 @@ impl Workspace { }; if should_close { - let _ = window_handle.update(cx, |_, window, _| window.remove_window()); + let _ = window_handle.update(cx, |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); + window.remove_window(); + }); } }) .detach(); diff --git a/crates/flowstate/src/workspace/workspace/keybindings.rs b/crates/flowstate/src/workspace/workspace/keybindings.rs index 83fd8507..9cfed3fb 100644 --- a/crates/flowstate/src/workspace/workspace/keybindings.rs +++ b/crates/flowstate/src/workspace/workspace/keybindings.rs @@ -52,6 +52,10 @@ impl Workspace { self.close_active_document(window, cx); true }, + CommandId::StartCollaboration => self.start_collaboration_on_active_document(cx).is_some(), + CommandId::CopyCollaborationTicket => self.copy_active_collaboration_ticket(window, cx), + CommandId::JoinCollaborationFromClipboard => self.join_collaboration_from_clipboard(window, cx), + CommandId::LeaveCollaboration => self.leave_collaboration_on_active_document(cx), CommandId::FindInDocument => self.open_active_document_search_bar(window, cx), CommandId::ZoomIn => { if let Some(editor) = self.active_editor.clone() { diff --git a/crates/flowstate/src/workspace/workspace/mod.rs b/crates/flowstate/src/workspace/workspace/mod.rs index 230b9d30..b0399622 100644 --- a/crates/flowstate/src/workspace/workspace/mod.rs +++ b/crates/flowstate/src/workspace/workspace/mod.rs @@ -261,6 +261,7 @@ impl DocumentStyleSection { } include!("documents.rs"); +include!("collab.rs"); include!("workspace_state.rs"); include!("load.rs"); include!("traits.rs"); diff --git a/crates/flowstate/src/workspace/workspace/render_top_bar.rs b/crates/flowstate/src/workspace/workspace/render_top_bar.rs index 0990071f..aaf3b073 100644 --- a/crates/flowstate/src/workspace/workspace/render_top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/render_top_bar.rs @@ -16,6 +16,7 @@ impl Workspace { .child(file_top_bar_button(self.active_document_id.is_some(), cx)) .child(insert_top_bar_button(cx, self.active_editor.is_some())) .child(document_top_bar_button(cx)) + .child(collaboration_top_bar_button(cx, self.active_document_id.is_some())) .child(view_top_bar_button( cx, !self.outline_collapsed, diff --git a/crates/flowstate/src/workspace/workspace/top_bar.rs b/crates/flowstate/src/workspace/workspace/top_bar.rs index d9397c6e..ac3b2ed5 100644 --- a/crates/flowstate/src/workspace/workspace/top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/top_bar.rs @@ -106,6 +106,39 @@ fn document_top_bar_button(cx: &mut Context) -> impl IntoElement { ) } +fn collaboration_top_bar_button(cx: &mut Context, has_document: bool) -> impl IntoElement { + let workspace = cx.entity().downgrade(); + div() + .h_full() + .flex_none() + .flex() + .items_center() + .justify_center() + .on_mouse_down(MouseButton::Left, |_, _, cx| cx.stop_propagation()) + .child( + Button::new("top-collaborate") + .label("Collaborate") + .xsmall() + .ghost() + .dropdown_menu(move |menu, _, _| { + menu + .item(file_menu_item(workspace.clone(), "Start Sharing", !has_document, |workspace, _, cx| { + let _ = workspace.start_collaboration_on_active_document(cx); + })) + .item(file_menu_item(workspace.clone(), "Copy Invite Ticket", !has_document, |workspace, window, cx| { + workspace.copy_active_collaboration_ticket(window, cx); + })) + .separator() + .item(file_menu_item(workspace.clone(), "Join from Clipboard", false, |workspace, window, cx| { + workspace.join_collaboration_from_clipboard(window, cx); + })) + .item(file_menu_item(workspace.clone(), "Leave Shared Session", !has_document, |workspace, _, cx| { + workspace.leave_collaboration_on_active_document(cx); + })) + }), + ) +} + #[hotpath::measure] fn file_top_bar_button(has_document: bool, cx: &mut Context) -> impl IntoElement { let workspace = cx.entity().downgrade(); diff --git a/crates/flowstate/src/workspace/workspace/window.rs b/crates/flowstate/src/workspace/workspace/window.rs index 62829fb0..663c2385 100644 --- a/crates/flowstate/src/workspace/workspace/window.rs +++ b/crates/flowstate/src/workspace/workspace/window.rs @@ -11,6 +11,7 @@ pub fn install_workspace_close_prompt(workspace: Entity, window: &mut let dirty_panels = workspace.read(cx).dirty_panels(cx); if dirty_panels.is_empty() { + workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); return true; } @@ -33,6 +34,7 @@ pub fn install_workspace_close_prompt(workspace: Entity, window: &mut ); let prompt_open = prompt_open.clone(); let allow_close = allow_close.clone(); + let workspace = workspace.clone(); cx.spawn(async move |cx| { let should_close = match answer.await { @@ -65,6 +67,7 @@ pub fn install_workspace_close_prompt(workspace: Entity, window: &mut prompt_open.set(false); if should_close { + let _ = workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); allow_close.set(true); let _ = window_handle.update(cx, |_, window, _| window.remove_window()); } diff --git a/crates/gpui-flowtext/Cargo.toml b/crates/gpui-flowtext/Cargo.toml index 59b84aa0..e80e631b 100644 --- a/crates/gpui-flowtext/Cargo.toml +++ b/crates/gpui-flowtext/Cargo.toml @@ -31,6 +31,8 @@ uuid = { workspace = true } [target.'cfg(any(target_os = "macos", target_os = "linux"))'.dependencies] cosmic-text = { workspace = true } +[dev-dependencies] +gpui = { workspace = true, features = ["test-support"] } [features] hotpath = ["hotpath/hotpath"] diff --git a/crates/gpui-flowtext/src/collaboration.rs b/crates/gpui-flowtext/src/collaboration.rs index 308b31a8..48950216 100644 --- a/crates/gpui-flowtext/src/collaboration.rs +++ b/crates/gpui-flowtext/src/collaboration.rs @@ -2,7 +2,10 @@ use std::ops::Range; use serde::{Deserialize, Serialize}; -use super::{Block, BlockId, Document, DocumentSpan, ParagraphId, ParagraphStyle, RunStyles, new_block_id, new_paragraph_id}; +use super::{ + AssetId, AssetRecord, Block, BlockId, Document, DocumentSpan, InputBlock, InputParagraph, ParagraphId, ParagraphStyle, RunStyles, + new_block_id, new_paragraph_id, +}; #[derive(Clone, Copy, Debug, Eq, PartialEq, Hash, Serialize, Deserialize)] pub struct TableCellId(pub u128); @@ -303,3 +306,50 @@ pub fn decode_canonical_operations(bytes: &[u8]) -> Option, } + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum CollabTextDelta { + Retain(usize), + Insert(usize), + Delete(usize), +} + +#[derive(Clone, Debug)] +pub struct CollabStructuralBlock { + pub block_id: BlockId, + pub paragraph_id: Option, + pub block: InputBlock, +} + +#[derive(Clone, Debug)] +pub enum CollabPatch { + ParagraphText { + row: usize, + new: InputParagraph, + delta_utf8: Vec, + }, + ParagraphStyle { + row: usize, + style: ParagraphStyle, + }, + ReplaceObjectBlock { + row: usize, + block: CollabStructuralBlock, + }, + InsertBlocks { + row: usize, + blocks: Vec, + }, + DeleteBlocks { + row: usize, + count: usize, + }, + MoveBlock { + from: usize, + to: usize, + }, + AssetArrived { + id: AssetId, + record: AssetRecord, + }, +} diff --git a/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs b/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs index bb01a648..0f4823af 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs @@ -50,6 +50,7 @@ impl RichTextEditor { } let caret = insert_rich_fragment_at(&mut self.document, self.selection.head, &fragment); self.selection = EditorSelection { anchor: caret, head: caret }; + self.emit_selection_changed(cx); self.after_text_mutation(cx); } @@ -71,12 +72,12 @@ impl RichTextEditor { }, ); } - self.insert_ordered_block_fragment_after_caret(&fragment.blocks); + self.insert_ordered_block_fragment_after_caret(&fragment.blocks, cx); self.push_replace_document_history(before_document, before_selection, cx); } - fn insert_ordered_block_fragment_after_caret(&mut self, input_blocks: &[InputBlock]) { - let insert_ix = self.prepare_block_insertion_index(); + fn insert_ordered_block_fragment_after_caret(&mut self, input_blocks: &[InputBlock], cx: &mut Context) { + let insert_ix = self.prepare_block_insertion_index(cx); let insert_paragraph_ix = self .document .blocks @@ -155,15 +156,15 @@ impl RichTextEditor { } let before_document = self.document.clone(); let before_selection = self.selection.clone(); - self.insert_blocks_after_caret_without_history(blocks); + self.insert_blocks_after_caret_without_history(blocks, cx); self.push_replace_document_history(before_document, before_selection, cx); } - fn insert_blocks_after_caret_without_history(&mut self, blocks: Vec) { + fn insert_blocks_after_caret_without_history(&mut self, blocks: Vec, cx: &mut Context) { if blocks.is_empty() { return; } - let insert_ix = self.prepare_block_insertion_index(); + let insert_ix = self.prepare_block_insertion_index(cx); let inserted_count = blocks.len(); Arc::make_mut(&mut self.document.blocks).splice(insert_ix..insert_ix, blocks); for relative_ix in 0..inserted_count { @@ -177,7 +178,7 @@ impl RichTextEditor { self.paragraph_height_cache_revision = self.paragraph_height_cache_revision.wrapping_add(1); } - fn prepare_block_insertion_index(&mut self) -> usize { + fn prepare_block_insertion_index(&mut self, cx: &mut Context) -> usize { if let Some( BlockSelection::Image(block_ix) | BlockSelection::Equation(block_ix) @@ -188,7 +189,7 @@ impl RichTextEditor { return (block_ix + 1).min(self.document.blocks.len()); } - if let Some(insert_ix) = self.remove_empty_caret_paragraph_for_block_insertion() { + if let Some(insert_ix) = self.remove_empty_caret_paragraph_for_block_insertion(cx) { return insert_ix; } @@ -220,7 +221,7 @@ impl RichTextEditor { self.document.blocks.len() } - fn remove_empty_caret_paragraph_for_block_insertion(&mut self) -> Option { + fn remove_empty_caret_paragraph_for_block_insertion(&mut self, cx: &mut Context) -> Option { if !self.selection.is_caret() { return None; } @@ -261,6 +262,7 @@ impl RichTextEditor { byte: 0, }, }; + self.emit_selection_changed(cx); } Some(block_ix) } @@ -297,6 +299,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceDocument]; self.undo_stack.push(EditRecord { before_selection, before_generation, @@ -306,11 +309,11 @@ impl RichTextEditor { before: Box::new(before_document), after: Box::new(self.document.clone()), }], - canonical_operations: vec![CanonicalOperation::ReplaceDocument], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); } fn insert_plain_text_fragment(&mut self, text: &str, cx: &mut Context) { diff --git a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs new file mode 100644 index 00000000..417a6f67 --- /dev/null +++ b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs @@ -0,0 +1,251 @@ +#[hotpath::measure_all] +impl RichTextEditor { + pub fn apply_collab_patches(&mut self, patches: &[CollabPatch], cx: &mut Context) { + if patches.is_empty() { + return; + } + debug_assert!(self.undo_stack.is_empty()); + debug_assert!(self.redo_stack.is_empty()); + + self.suppress_collab_capture = self.suppress_collab_capture.saturating_add(1); + let mut invalidation: Option> = None; + for patch in patches { + self.apply_one_collab_patch(patch, &mut invalidation); + } + self.suppress_collab_capture = self.suppress_collab_capture.saturating_sub(1); + self.identity_map.reconcile(&self.document); + self.last_collaboration_edit = None; + self.layout_invalidation_hint = invalidation; + self.after_text_mutation(cx); + self.layout_invalidation_hint = None; + } + + pub fn collab_apply_deferred(&self) -> bool { + self.selecting || self.active_text_drag.is_some() || self.image_resize_drag.is_some() || self.table_column_resize_drag.is_some() + } + + fn apply_one_collab_patch(&mut self, patch: &CollabPatch, invalidation: &mut Option>) { + match patch { + CollabPatch::ParagraphText { row, new, delta_utf8 } => { + if let Some(paragraph_ix) = self.paragraph_ix_for_block(*row) { + remap_selection_for_text_delta(&mut self.selection, paragraph_ix, delta_utf8); + replace_paragraph_content(&mut self.document, paragraph_ix, new); + extend_invalidation(invalidation, paragraph_ix..paragraph_ix + 1); + } + }, + CollabPatch::ParagraphStyle { row, style } => { + if let Some(paragraph_ix) = self.paragraph_ix_for_block(*row) + && let Some(paragraph) = paragraphs_mut(&mut self.document).get_mut(paragraph_ix) + { + paragraph.style = *style; + bump_paragraph_version(paragraph); + update_paragraph_block(&mut self.document, paragraph_ix); + rebuild_document_sections(&mut self.document); + extend_invalidation(invalidation, paragraph_ix..paragraph_ix + 1); + } + }, + CollabPatch::ReplaceObjectBlock { row, block } => { + let mut blocks = collab_structural_blocks_from_document(&self.document); + if *row < blocks.len() { + blocks[*row] = block.clone(); + rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + extend_invalidation(invalidation, 0..self.document.paragraphs.len()); + } + }, + CollabPatch::InsertBlocks { row, blocks: inserted } => { + let mut blocks = collab_structural_blocks_from_document(&self.document); + let row = (*row).min(blocks.len()); + blocks.splice(row..row, inserted.iter().cloned()); + rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + extend_invalidation(invalidation, 0..self.document.paragraphs.len()); + }, + CollabPatch::DeleteBlocks { row, count } => { + let mut blocks = collab_structural_blocks_from_document(&self.document); + let start = (*row).min(blocks.len()); + let end = start.saturating_add(*count).min(blocks.len()); + blocks.drain(start..end); + rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + clamp_selection_to_document(&self.document, &mut self.selection); + extend_invalidation(invalidation, 0..self.document.paragraphs.len()); + }, + CollabPatch::MoveBlock { from, to } => { + let mut blocks = collab_structural_blocks_from_document(&self.document); + if *from < blocks.len() { + let block = blocks.remove(*from); + blocks.insert((*to).min(blocks.len()), block); + rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + clamp_selection_to_document(&self.document, &mut self.selection); + extend_invalidation(invalidation, 0..self.document.paragraphs.len()); + } + }, + CollabPatch::AssetArrived { id, record } => { + self.document.assets.assets.insert(*id, record.clone()); + }, + } + } +} + +#[hotpath::measure] +fn replace_paragraph_content(document: &mut Document, paragraph_ix: usize, paragraph: &InputParagraph) { + if paragraph_ix >= document.paragraphs.len() { + return; + } + let text = input_paragraph_text(paragraph); + let byte_range = paragraph_byte_range(document, paragraph_ix); + document.text.delete(byte_range.clone()); + document.text.insert(byte_range.start, &text); + let mut replacement = paragraph_from_input_paragraph(paragraph); + replacement.version = document.paragraphs[paragraph_ix].version.wrapping_add(1); + paragraphs_mut(document)[paragraph_ix] = replacement; + rebuild_document_offset_index(document); + update_paragraph_block(document, paragraph_ix); + rebuild_document_sections(document); +} + +#[hotpath::measure] +fn collab_structural_blocks_from_document(document: &Document) -> Vec { + let mut paragraph_ix = 0; + document + .blocks + .iter() + .enumerate() + .map(|(block_ix, block)| match block { + Block::Paragraph(paragraph) => { + let input = input_paragraph_from_document_paragraph(document, paragraph_ix, paragraph); + let structural = CollabStructuralBlock { + block_id: document.ids.block_ids.get(block_ix).copied().unwrap_or_else(new_block_id), + paragraph_id: Some( + document + .ids + .paragraph_ids + .get(paragraph_ix) + .copied() + .unwrap_or_else(new_paragraph_id), + ), + block: InputBlock::Paragraph(input), + }; + paragraph_ix += 1; + structural + }, + Block::Image(_) | Block::Equation(_) | Block::Table(_) => CollabStructuralBlock { + block_id: document.ids.block_ids.get(block_ix).copied().unwrap_or_else(new_block_id), + paragraph_id: None, + block: input_block_from_block(block), + }, + }) + .collect() +} + +#[hotpath::measure] +fn input_paragraph_from_document_paragraph(document: &Document, paragraph_ix: usize, paragraph: &Paragraph) -> InputParagraph { + let text = paragraph_text(document, paragraph_ix); + let mut byte = 0; + InputParagraph { + style: paragraph.style, + runs: paragraph + .runs + .iter() + .map(|run| { + let start = byte; + let end = (start + run.len).min(text.len()); + byte = end; + InputRun { + text: text.get(start..end).unwrap_or("").to_string(), + styles: run.styles, + } + }) + .collect(), + } +} + +#[hotpath::measure] +fn rebuild_document_from_collab_structural_blocks(document: &mut Document, blocks: Vec) { + let assets = document.assets.clone(); + let theme = document.theme.clone(); + let document_id = document.ids.document_id; + let input_blocks = blocks + .iter() + .map(|block| block.block.clone()) + .collect::>(); + let block_ids = blocks.iter().map(|block| block.block_id).collect::>(); + let paragraph_ids = blocks + .iter() + .filter_map(|block| match &block.block { + InputBlock::Paragraph(_) => Some(block.paragraph_id.unwrap_or_else(new_paragraph_id)), + InputBlock::Image(_) | InputBlock::Equation(_) | InputBlock::Table(_) => None, + }) + .collect::>(); + let mut rebuilt = document_from_input_blocks(theme, input_blocks); + rebuilt.assets = assets; + rebuilt.ids.document_id = document_id; + rebuilt.ids.block_ids = block_ids; + rebuilt.ids.paragraph_ids = paragraph_ids; + debug_assert_eq!(rebuilt.ids.block_ids.len(), rebuilt.blocks.len()); + debug_assert_eq!(rebuilt.ids.paragraph_ids.len(), rebuilt.paragraphs.len()); + rebuild_document_sections(&mut rebuilt); + *document = rebuilt; +} + +#[hotpath::measure] +fn remap_selection_for_text_delta(selection: &mut EditorSelection, paragraph_ix: usize, delta: &[CollabTextDelta]) { + if selection.anchor.paragraph == paragraph_ix { + selection.anchor.byte = remap_byte(selection.anchor.byte, delta); + } + if selection.head.paragraph == paragraph_ix { + selection.head.byte = remap_byte(selection.head.byte, delta); + } +} + +#[hotpath::measure] +fn remap_byte(byte: usize, delta: &[CollabTextDelta]) -> usize { + let mut old = 0usize; + let mut new = 0usize; + for item in delta { + match *item { + CollabTextDelta::Retain(len) => { + if byte <= old + len { + return new + (byte - old); + } + old += len; + new += len; + }, + CollabTextDelta::Insert(len) => { + new += len; + }, + CollabTextDelta::Delete(len) => { + if byte <= old + len { + return new; + } + old += len; + }, + } + } + new + byte.saturating_sub(old) +} + +#[hotpath::measure] +fn clamp_selection_to_document(document: &Document, selection: &mut EditorSelection) { + selection.anchor = clamp_offset_to_document(document, selection.anchor); + selection.head = clamp_offset_to_document(document, selection.head); +} + +#[hotpath::measure] +fn clamp_offset_to_document(document: &Document, offset: DocumentOffset) -> DocumentOffset { + let paragraph = offset.paragraph.min(document.paragraphs.len().saturating_sub(1)); + let byte = document.paragraphs.get(paragraph).map_or(0, paragraph_text_len).min(offset.byte); + DocumentOffset { paragraph, byte } +} + +#[hotpath::measure] +fn extend_invalidation(target: &mut Option>, range: Range) { + if range.is_empty() { + return; + } + match target { + Some(existing) => { + existing.start = existing.start.min(range.start); + existing.end = existing.end.max(range.end); + }, + None => *target = Some(range), + } +} diff --git a/crates/gpui-flowtext/src/rich_text/editor/commands.rs b/crates/gpui-flowtext/src/rich_text/editor/commands.rs index 2e3a3026..a2f70730 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/commands.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/commands.rs @@ -189,6 +189,10 @@ impl RichTextEditor { } pub fn undo(&mut self, cx: &mut Context) { + if let Some(hook) = self.collab_undo_redirect.clone() { + hook(UndoRedirect::Undo); + return; + } let Some(record) = self.undo_stack.pop() else { return; }; @@ -197,12 +201,17 @@ impl RichTextEditor { operation.undo(&mut self.document); } self.selection = record.before_selection.clone(); + self.emit_selection_changed(cx); self.edit_generation = restored_generation; self.redo_stack.push(record); self.after_history_restore(cx); } pub fn redo(&mut self, cx: &mut Context) { + if let Some(hook) = self.collab_undo_redirect.clone() { + hook(UndoRedirect::Redo); + return; + } let Some(record) = self.redo_stack.pop() else { return; }; @@ -211,6 +220,7 @@ impl RichTextEditor { operation.redo(&mut self.document); } self.selection = record.after_selection.clone(); + self.emit_selection_changed(cx); self.edit_generation = restored_generation; self.undo_stack.push(record); self.after_history_restore(cx); diff --git a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs index 7c0bda2a..c4416d04 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs @@ -1,6 +1,6 @@ #[hotpath::measure_all] impl RichTextEditor { - fn insert_single_grapheme_fast_path(&mut self, text: &str, cx: &mut Context) -> bool { + pub(super) fn insert_single_grapheme_fast_path(&mut self, text: &str, cx: &mut Context) -> bool { if !is_single_grapheme_text_insert(text) || !self.selection.is_caret() || self.selected_block.is_some() { return false; } @@ -39,6 +39,7 @@ impl RichTextEditor { byte: caret.byte + text.len(), }; self.selection = EditorSelection { anchor: after, head: after }; + self.emit_selection_changed(cx); let mut merged_into_previous = false; if let Some(record) = self.undo_stack.last_mut() @@ -92,12 +93,22 @@ impl RichTextEditor { }], }); } + if self.collab_capture && self.suppress_collab_capture == 0 { + self.pending_collab_edits.push(CollaborationEdit { + operations: vec![CanonicalOperation::InsertText { + paragraph: paragraph_id, + byte: caret.byte, + text: text.to_string(), + styles, + }], + }); + } self.redo_stack.clear(); self.layout_invalidation_hint = Some(caret.paragraph..caret.paragraph + 1); self.suppress_mutation_notify += 1; self.after_text_mutation(cx); self.suppress_mutation_notify = self.suppress_mutation_notify.saturating_sub(1); - self.mark_document_changed_with_reconcile(after_generation, false, cx); + self.mark_document_changed_with_ops(after_generation, false, None, cx); true } @@ -169,6 +180,7 @@ impl RichTextEditor { before: before_span.clone(), after: after_span.clone(), }]; + let collab_operations = canonical_operations.clone(); let identity_shape_changed = before_span.paragraphs.len() != after_span.paragraphs.len() || before_block_count != self.document.blocks.len(); let record = EditRecord { before_selection, @@ -183,7 +195,7 @@ impl RichTextEditor { }; self.undo_stack.push(record); self.redo_stack.clear(); - self.mark_document_changed_with_reconcile(after_generation, identity_shape_changed, cx); + self.mark_document_changed_with_ops(after_generation, identity_shape_changed, Some(&collab_operations), cx); } fn insert_paragraph_break_at_caret(&mut self, caret: DocumentOffset, block_ix: usize, cx: &mut Context) { @@ -206,6 +218,11 @@ impl RichTextEditor { return; } + let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { + start_paragraph: self.identity_map.paragraph_id(caret.paragraph), + before: before_span.clone(), + after: after_span.clone(), + }]; let record = EditRecord { before_selection, before_generation, @@ -215,29 +232,33 @@ impl RichTextEditor { before: before_span.clone(), after: after_span.clone(), }], - canonical_operations: vec![CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: self.identity_map.paragraph_id(caret.paragraph), - before: before_span, - after: after_span, - }], + canonical_operations, }; + let collab_operations = record.canonical_operations.clone(); self.undo_stack.push(record); self.redo_stack.clear(); - self.mark_document_changed_with_reconcile(after_generation, false, cx); + self.mark_document_changed_with_ops(after_generation, false, Some(&collab_operations), cx); } - fn mark_document_changed(&mut self, generation: u64, cx: &mut Context) { - self.mark_document_changed_with_reconcile(generation, true, cx); - } - - fn mark_document_changed_with_reconcile(&mut self, generation: u64, reconcile_identity: bool, cx: &mut Context) { + fn mark_document_changed_with_ops( + &mut self, + generation: u64, + reconcile_identity: bool, + operations: Option<&[CanonicalOperation]>, + cx: &mut Context, + ) { self.edit_generation = generation; if reconcile_identity { self.identity_map.reconcile(&self.document); } - self.last_collaboration_edit = self.undo_stack.last().map(|record| CollaborationEdit { - operations: record.canonical_operations.clone(), - }); + self.last_collaboration_edit = operations.map(|operations| CollaborationEdit { operations: operations.to_vec() }); + if self.collab_capture + && self.suppress_collab_capture == 0 + && let Some(operations) = operations + && !operations.is_empty() + { + self.pending_collab_edits.push(CollaborationEdit { operations: operations.to_vec() }); + } self.refresh_save_status(); self.schedule_recovery_write(cx); cx.notify(); diff --git a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs index 315f3d34..99bdde75 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs @@ -38,6 +38,10 @@ impl RichTextEditor { redo_stack: Vec::new(), identity_map, last_collaboration_edit: None, + pending_collab_edits: Vec::new(), + collab_capture: false, + suppress_collab_capture: 0, + collab_undo_redirect: None, collaboration_role: None, recovery_write_in_progress: false, recovery_write_pending: false, @@ -114,6 +118,12 @@ impl RichTextEditor { &self.document } + fn emit_selection_changed(&self, cx: &mut Context) { + cx.emit(EditorEvent::SelectionChanged { + selection: self.selection.clone(), + }); + } + pub fn dispose_for_close(&mut self) { if self.disposed { return; @@ -146,6 +156,10 @@ impl RichTextEditor { self.undo_stack = Vec::new(); self.redo_stack = Vec::new(); self.last_collaboration_edit = None; + self.pending_collab_edits.clear(); + self.collab_capture = false; + self.suppress_collab_capture = 0; + self.collab_undo_redirect = None; self.collaboration_role = None; self.recovery_write_in_progress = false; self.recovery_write_pending = false; @@ -232,6 +246,26 @@ impl RichTextEditor { pub fn clear_collaboration_edit(&mut self) { self.last_collaboration_edit = None; } + + pub fn take_pending_collab_edits(&mut self) -> Vec { + std::mem::take(&mut self.pending_collab_edits) + } + + pub fn set_collab_capture(&mut self, on: bool) { + self.collab_capture = on; + if !on { + self.pending_collab_edits.clear(); + } + } + + pub fn set_collab_undo_redirect(&mut self, hook: Option>) { + self.collab_undo_redirect = hook; + } + + pub fn set_recovery_path(&mut self, path: Option) { + self.recovery_path = path; + } + pub fn collaboration_role(&self) -> Option { self.collaboration_role } diff --git a/crates/gpui-flowtext/src/rich_text/editor/media.rs b/crates/gpui-flowtext/src/rich_text/editor/media.rs index b7dbd8ae..24ddeb37 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/media.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/media.rs @@ -151,6 +151,9 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + block: self.identity_map.block_id(drag.block_ix), + }]; self.undo_stack.push(EditRecord { before_selection: self.selection.clone(), before_generation, @@ -161,13 +164,11 @@ impl RichTextEditor { before: Block::Image(drag.before), after: Block::Image(after), }], - canonical_operations: vec![CanonicalOperation::ReplaceBlock { - block: self.identity_map.block_id(drag.block_ix), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); true } @@ -215,19 +216,20 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + block: self.identity_map.block_id(block_ix), + }]; self.undo_stack.push(EditRecord { before_selection: self.selection.clone(), before_generation, after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: vec![CanonicalOperation::ReplaceBlock { - block: self.identity_map.block_id(block_ix), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); } pub fn insert_equation(&mut self, source: impl Into, cx: &mut Context) { @@ -265,7 +267,7 @@ impl RichTextEditor { version: 0, })); } - self.insert_blocks_after_caret_without_history(blocks); + self.insert_blocks_after_caret_without_history(blocks, cx); self.push_replace_document_history(before_document, before_selection, cx); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/mod.rs b/crates/gpui-flowtext/src/rich_text/editor/mod.rs index 646f429d..7a05e431 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mod.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mod.rs @@ -185,6 +185,12 @@ pub enum CollaborationRole { Viewer, } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum UndoRedirect { + Undo, + Redo, +} + impl CollaborationRole { #[must_use] pub const fn can_write(self) -> bool { @@ -868,6 +874,10 @@ pub struct RichTextEditor { redo_stack: Vec, identity_map: DocumentIdentityMap, last_collaboration_edit: Option, + pending_collab_edits: Vec, + collab_capture: bool, + suppress_collab_capture: u32, + collab_undo_redirect: Option>, collaboration_role: Option, recovery_write_in_progress: bool, recovery_write_pending: bool, @@ -944,7 +954,10 @@ pub struct RichTextEditor { goal_x: Option, } +impl gpui::EventEmitter for RichTextEditor {} + include!("lifecycle.rs"); +include!("collab_apply.rs"); include!("object_selection.rs"); include!("style_state.rs"); include!("search_highlights.rs"); diff --git a/crates/gpui-flowtext/src/rich_text/editor/mouse.rs b/crates/gpui-flowtext/src/rich_text/editor/mouse.rs index d5444fa7..0e553a7a 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mouse.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mouse.rs @@ -275,6 +275,17 @@ impl RichTextEditor { anchor: inserted_end, head: inserted_end, }; + let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { + start_paragraph: self.identity_map.paragraph_id(inserted_start.paragraph), + before: capture_document_span( + &self.document, + inserted_start.paragraph..(inserted_start.paragraph + 1).min(self.document.paragraphs.len()), + ), + after: capture_document_span( + &self.document, + inserted_start.paragraph..(inserted_end.paragraph + 1).min(self.document.paragraphs.len()), + ), + }]; self.undo_stack.push(EditRecord { before_selection, before_generation, @@ -286,21 +297,11 @@ impl RichTextEditor { inserted_range: inserted_start..inserted_end, fragment: drag.fragment, }], - canonical_operations: vec![CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: self.identity_map.paragraph_id(inserted_start.paragraph), - before: capture_document_span( - &self.document, - inserted_start.paragraph..(inserted_start.paragraph + 1).min(self.document.paragraphs.len()), - ), - after: capture_document_span( - &self.document, - inserted_start.paragraph..(inserted_end.paragraph + 1).min(self.document.paragraphs.len()), - ), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.after_text_mutation(cx); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); self.clear_drop_preview(); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/movement_core.rs b/crates/gpui-flowtext/src/rich_text/editor/movement_core.rs index 92ae74f0..42271054 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/movement_core.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/movement_core.rs @@ -8,6 +8,7 @@ impl RichTextEditor { return; } self.selection = selection; + self.emit_selection_changed(cx); self.goal_x = None; self.scroll_head_into_view(); self.reset_caret_blink(cx); diff --git a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs index 93d85a77..d29c9dd9 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs @@ -237,6 +237,9 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + block: self.identity_map.block_id(drag.block_ix), + }]; self.undo_stack.push(EditRecord { before_selection: self.selection.clone(), before_generation, @@ -247,13 +250,11 @@ impl RichTextEditor { before: Block::Table(drag.before), after: Block::Table(after), }], - canonical_operations: vec![CanonicalOperation::ReplaceBlock { - block: self.identity_map.block_id(drag.block_ix), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); true } @@ -449,6 +450,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceDocument]; self.undo_stack.push(EditRecord { before_selection, before_generation, @@ -458,11 +460,11 @@ impl RichTextEditor { before: Box::new(before_document), after: Box::new(after_document), }], - canonical_operations: vec![CanonicalOperation::ReplaceDocument], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); true } @@ -493,21 +495,22 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::DeleteBlock { + block: self.identity_map.block_id(block_ix).unwrap_or(BlockId(0)), + }]; self.undo_stack.push(EditRecord { before_selection: before_selection.clone(), before_generation, after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::DeleteBlock { block_ix, block }], - canonical_operations: vec![CanonicalOperation::DeleteBlock { - block: self.identity_map.block_id(block_ix).unwrap_or(BlockId(0)), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.clear_layout_work_caches(); self.item_sizes_cache = None; self.paragraph_height_cache_revision = self.paragraph_height_cache_revision.wrapping_add(1); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); true } diff --git a/crates/gpui-flowtext/src/rich_text/editor/paste.rs b/crates/gpui-flowtext/src/rich_text/editor/paste.rs index 3be8aca1..10f9506d 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/paste.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/paste.rs @@ -20,7 +20,7 @@ impl RichTextEditor { self.insert_rich_fragment_paste_at_caret(&fragment, cx) } - fn insert_rich_fragment_paste_at_caret(&mut self, fragment: &RichClipboardFragment, cx: &mut Context) -> bool { + pub(super) fn insert_rich_fragment_paste_at_caret(&mut self, fragment: &RichClipboardFragment, cx: &mut Context) -> bool { if !self.selection.is_caret() || self.selected_block.is_some() || !fragment.blocks.is_empty() @@ -45,11 +45,19 @@ impl RichTextEditor { let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); let offset = self.selection.head; + let before_span = capture_document_span(&self.document, offset.paragraph..offset.paragraph + 1); let inserted_end = insert_rich_fragment_at(&mut self.document, offset, fragment); + let after_span = capture_document_span(&self.document, offset.paragraph..inserted_end.paragraph + 1); self.selection = EditorSelection { anchor: inserted_end, head: inserted_end, }; + self.emit_selection_changed(cx); + let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { + start_paragraph: Some(paragraph_id), + before: before_span.clone(), + after: after_span.clone(), + }]; self.undo_stack.push(EditRecord { before_selection, before_generation, @@ -60,12 +68,12 @@ impl RichTextEditor { inserted_end, fragment: fragment.clone(), }], - canonical_operations: canonical_insert_text_operations(paragraph_id, offset.byte, paragraph), + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.layout_invalidation_hint = Some(offset.paragraph..offset.paragraph + 1); self.after_text_mutation(cx); - self.mark_document_changed_with_reconcile(after_generation, false, cx); + self.mark_document_changed_with_ops(after_generation, false, Some(&canonical_operations), cx); true } diff --git a/crates/gpui-flowtext/src/rich_text/editor/serialization.rs b/crates/gpui-flowtext/src/rich_text/editor/serialization.rs index 4049c090..ff9f556f 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/serialization.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/serialization.rs @@ -1,21 +1,3 @@ -#[hotpath::measure] -fn canonical_insert_text_operations(paragraph_id: ParagraphId, start_byte: usize, paragraph: &InputParagraph) -> Vec { - let mut byte = start_byte; - let mut operations = Vec::with_capacity(paragraph.runs.len()); - for run in ¶graph.runs { - if !run.text.is_empty() { - operations.push(CanonicalOperation::InsertText { - paragraph: paragraph_id, - byte, - text: run.text.clone(), - styles: run.styles, - }); - } - byte += run.text.len(); - } - operations -} - #[hotpath::measure] pub(super) fn input_paragraph_text(paragraph: &InputParagraph) -> String { paragraph.runs.iter().map(|run| run.text.as_str()).collect() @@ -51,7 +33,7 @@ fn collect_block_assets(block: &Block, assets: &AssetStore, output: &mut Vec InputBlock { +pub fn input_block_from_block(block: &Block) -> InputBlock { match block { Block::Paragraph(paragraph) => InputBlock::Paragraph(input_paragraph_from_paragraph(paragraph)), Block::Image(image) => InputBlock::Image(InputImageBlock { @@ -78,7 +60,7 @@ fn input_block_from_block(block: &Block) -> InputBlock { } #[hotpath::measure] -fn block_from_input_block(block: &InputBlock) -> Block { +pub fn block_from_input_block(block: &InputBlock) -> Block { match block { InputBlock::Paragraph(paragraph) => Block::Paragraph(paragraph_from_input_paragraph(paragraph)), InputBlock::Image(image) => Block::Image(ImageBlock { @@ -106,6 +88,61 @@ fn block_from_input_block(block: &InputBlock) -> Block { } } +#[hotpath::measure] +#[must_use] +pub fn document_from_input_blocks(theme: DocumentTheme, input_blocks: Vec) -> Document { + let mut text = String::new(); + let mut paragraphs = Vec::new(); + let mut blocks = Vec::with_capacity(input_blocks.len()); + + for input_block in input_blocks { + match input_block { + InputBlock::Paragraph(paragraph) => { + if !paragraphs.is_empty() { + text.push('\n'); + } + let start = text.len(); + text.push_str(&input_paragraph_text(¶graph)); + let mut paragraph = paragraph_from_input_paragraph(¶graph); + paragraph.byte_range = start..text.len(); + paragraphs.push(paragraph.clone()); + blocks.push(Block::Paragraph(paragraph)); + }, + InputBlock::Image(image) => blocks.push(block_from_input_block(&InputBlock::Image(image))), + InputBlock::Equation(equation) => blocks.push(block_from_input_block(&InputBlock::Equation(equation))), + InputBlock::Table(table) => blocks.push(block_from_input_block(&InputBlock::Table(table))), + } + } + + if paragraphs.is_empty() { + let paragraph = Paragraph { + style: ParagraphStyle::Normal, + byte_range: 0..0, + runs: Vec::new(), + version: 0, + }; + paragraphs.push(paragraph.clone()); + blocks.push(Block::Paragraph(paragraph)); + } + + let offset_index = ParagraphOffsetIndex::new(¶graphs); + let paragraph_count = paragraphs.len(); + let block_count = blocks.len(); + let mut document = Document { + text: Rope::from(text), + paragraphs: Arc::new(paragraphs), + blocks: Arc::new(blocks), + assets: AssetStore::default(), + ids: document_ids_for_shape(paragraph_count, block_count), + sections: Arc::new(Vec::new()), + offset_index, + theme, + }; + reconcile_document_ids(&mut document); + rebuild_document_sections(&mut document); + document +} + #[hotpath::measure] fn input_paragraph_from_paragraph(paragraph: &Paragraph) -> InputParagraph { InputParagraph { diff --git a/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs b/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs index a5fe2d24..87d9386a 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs @@ -57,21 +57,22 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + block: self.identity_map.block_id(block_ix), + }]; self.undo_stack.push(EditRecord { before_selection: self.selection.clone(), before_generation, after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: vec![CanonicalOperation::ReplaceBlock { - block: self.identity_map.block_id(block_ix), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.table_cell_block_ix = new_paragraph_ix; self.table_cell_caret = 0; self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); true } @@ -271,19 +272,20 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + block: self.identity_map.block_id(block_ix), + }]; self.undo_stack.push(EditRecord { before_selection: self.selection.clone(), before_generation, after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: vec![CanonicalOperation::ReplaceBlock { - block: self.identity_map.block_id(block_ix), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); } pub(super) fn edit_table_cell_paragraph( @@ -327,19 +329,20 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + block: self.identity_map.block_id(block_ix), + }]; self.undo_stack.push(EditRecord { before_selection: self.selection.clone(), before_generation, after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: vec![CanonicalOperation::ReplaceBlock { - block: self.identity_map.block_id(block_ix), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); } } diff --git a/crates/gpui-flowtext/src/rich_text/editor/tables.rs b/crates/gpui-flowtext/src/rich_text/editor/tables.rs index 29fb890c..ace99622 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/tables.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/tables.rs @@ -165,19 +165,20 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + block: self.identity_map.block_id(block_ix), + }]; self.undo_stack.push(EditRecord { before_selection: self.selection.clone(), before_generation, after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: vec![CanonicalOperation::ReplaceBlock { - block: self.identity_map.block_id(block_ix), - }], + canonical_operations: canonical_operations.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed(after_generation, cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); } fn selected_table_block_ix(&self) -> Option { diff --git a/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs new file mode 100644 index 00000000..eb44cefb --- /dev/null +++ b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs @@ -0,0 +1,59 @@ +use gpui::AppContext as _; + +#[gpui::test] +fn collab_capture_fast_path_emits_single_grapheme_deltas(cx: &mut gpui::TestAppContext) { + let editor = cx.update(|cx| cx.new(|cx| RichTextEditor::new_with_path(blank_document(), None, cx))); + + let edits = cx.update(|cx| { + editor.update(cx, |editor, cx| { + editor.set_collab_capture(true); + assert!(editor.insert_single_grapheme_fast_path("a", cx)); + assert!(editor.insert_single_grapheme_fast_path("b", cx)); + assert!(editor.insert_single_grapheme_fast_path("c", cx)); + editor.take_pending_collab_edits() + }) + }); + + assert_eq!(edits.len(), 3); + for (edit, (expected_text, expected_byte)) in edits.iter().zip([("a", 0), ("b", 1), ("c", 2)]) { + let [CanonicalOperation::InsertText { byte, text, .. }] = edit.operations.as_slice() else { + panic!("expected one InsertText op, got {:?}", edit.operations); + }; + assert_eq!(text, expected_text); + assert_eq!(*byte, expected_byte); + } + + let edits_after_undo = cx.update(|cx| { + editor.update(cx, |editor, cx| { + editor.undo(cx); + editor.take_pending_collab_edits() + }) + }); + assert!(edits_after_undo.is_empty()); + + let paste_edits = cx.update(|cx| { + editor.update(cx, |editor, cx| { + let fragment = RichClipboardFragment { + format: RICH_TEXT_CLIPBOARD_FORMAT.to_string(), + paragraphs: vec![InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain("paste")], + }], + blocks: Vec::new(), + assets: Vec::new(), + }; + assert!(editor.insert_rich_fragment_paste_at_caret(&fragment, cx)); + editor.take_pending_collab_edits() + }) + }); + assert_eq!(paste_edits.len(), 1); + assert!(matches!(paste_edits[0].operations.as_slice(), [CanonicalOperation::ReplaceParagraphSpan { .. }])); + + let edits_after_paste_undo = cx.update(|cx| { + editor.update(cx, |editor, cx| { + editor.undo(cx); + editor.take_pending_collab_edits() + }) + }); + assert!(edits_after_paste_undo.is_empty()); +} diff --git a/crates/gpui-flowtext/src/rich_text/tests/mod.rs b/crates/gpui-flowtext/src/rich_text/tests/mod.rs index cebe5a31..b77c6bac 100644 --- a/crates/gpui-flowtext/src/rich_text/tests/mod.rs +++ b/crates/gpui-flowtext/src/rich_text/tests/mod.rs @@ -5,3 +5,4 @@ include!("history_tables.rs"); include!("positions_formatting.rs"); include!("history_fragments.rs"); include!("decorations_drag_search.rs"); +include!("collab_capture.rs"); From 759ef587f1f1708f8255196dbd736a36403e7d76 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Fri, 12 Jun 2026 23:17:56 -0500 Subject: [PATCH 04/45] add collaboration presence status --- crates/flowstate-collab/src/presence.rs | 2 - crates/flowstate/src/collab/mod.rs | 8 +- crates/flowstate/src/collab/presence_view.rs | 100 ++++++++++++++++++ crates/flowstate/src/collab/session.rs | 36 ++++++- crates/flowstate/src/collab/status.rs | 66 ++++++++++++ .../src/workspace/workspace/render_status.rs | 10 ++ 6 files changed, 216 insertions(+), 6 deletions(-) create mode 100644 crates/flowstate/src/collab/presence_view.rs create mode 100644 crates/flowstate/src/collab/status.rs diff --git a/crates/flowstate-collab/src/presence.rs b/crates/flowstate-collab/src/presence.rs index a2d87bda..b3cdce47 100644 --- a/crates/flowstate-collab/src/presence.rs +++ b/crates/flowstate-collab/src/presence.rs @@ -17,9 +17,7 @@ pub struct PresenceState { #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] pub struct PresenceSelection { - pub head_container: String, pub head: Vec, - pub anchor_container: String, pub anchor: Vec, } diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index 2de5c1d6..3bff0775 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -1,4 +1,6 @@ mod session; +mod presence_view; +pub mod status; use std::collections::HashMap; @@ -9,7 +11,7 @@ use flowstate_collab::{ net::{NetCommand, NetEvent, TicketSeed, runtime::{self, CommandSender}}, ticket::SessionTicket, }; -use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global}; +use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global, ReadGlobal}; use uuid::Uuid; use crate::rich_text_element::RichTextEditor; @@ -388,3 +390,7 @@ where { cx.update_default_global::(|manager, cx| manager.request_ticket_for_panel(panel_id, cx)) } + +pub fn phase_for_panel(panel_id: Uuid, cx: &App) -> Option { + CollabManager::global(cx).phase_for_panel(panel_id, cx) +} diff --git a/crates/flowstate/src/collab/presence_view.rs b/crates/flowstate/src/collab/presence_view.rs new file mode 100644 index 00000000..387d5ba8 --- /dev/null +++ b/crates/flowstate/src/collab/presence_view.rs @@ -0,0 +1,100 @@ +use std::path::PathBuf; + +use flowstate_collab::{SessionId, binding::DocBinding, presence::{PresenceSelection, PresenceStore}}; +use loro::{LoroDoc, cursor::{Cursor, PosType, Side}}; + +use crate::rich_text_element::{DocumentOffset, ExternalCaret, ParagraphId, RichTextEditor}; + +pub fn selection_for_editor(editor: &RichTextEditor, binding: &DocBinding) -> Option { + let selection = editor.selection().clone(); + let anchor_paragraph = editor.paragraph_id(selection.anchor.paragraph)?; + let head_paragraph = editor.paragraph_id(selection.head.paragraph)?; + Some(PresenceSelection { + anchor: cursor_bytes_for_offset(binding, anchor_paragraph, selection.anchor.byte)?, + head: cursor_bytes_for_offset(binding, head_paragraph, selection.head.byte)?, + }) +} + +pub fn external_carets(doc: &LoroDoc, binding: &DocBinding, presence: &PresenceStore) -> Vec { + let self_key = presence.self_key(); + presence + .roster() + .into_iter() + .filter(|entry| entry.key != self_key) + .filter_map(|entry| { + entry + .selection + .as_ref() + .and_then(|selection| external_caret_for_presence(doc, binding, selection, entry.color_rgb)) + }) + .collect() +} + +pub fn collaboration_recovery_path(session: SessionId, title: &str) -> PathBuf { + let dir = std::env::temp_dir().join("flowstate-collab-recovery"); + let _ = std::fs::create_dir_all(&dir); + let session_hex = session.to_string(); + let prefix = session_hex.get(..16).unwrap_or(&session_hex); + dir.join(format!("{prefix}-{}.db8", sanitized_recovery_title(title))) +} + +fn cursor_bytes_for_offset(binding: &DocBinding, paragraph: ParagraphId, byte: usize) -> Option> { + let row_ix = binding.by_paragraph.get(¶graph).copied()?; + let row = binding.rows.get(row_ix)?; + let text = row.text.as_ref()?; + let byte = byte.min(text.len_utf8()); + let pos = text.convert_pos(byte, PosType::Bytes, PosType::Unicode)?; + text.get_cursor(pos, Side::Middle).map(|cursor| cursor.encode()) +} + +fn external_caret_for_presence( + doc: &LoroDoc, + binding: &DocBinding, + selection: &PresenceSelection, + color_rgb: u32, +) -> Option { + let cursor = Cursor::decode(&selection.head).ok()?; + let row_ix = binding.by_container.get(&cursor.container).copied()?; + let row = binding.rows.get(row_ix)?; + let text = row.text.as_ref()?; + let resolved = doc.get_cursor_pos(&cursor).ok()?; + let byte = text.convert_pos(resolved.current.pos, PosType::Unicode, PosType::Bytes)?; + let paragraph = paragraph_ordinal_for_row(binding, row_ix)?; + Some(ExternalCaret { + offset: DocumentOffset { paragraph, byte }, + color_rgb, + }) +} + +fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option { + let mut paragraph = 0; + for (row_ix, row) in binding.rows.iter().enumerate() { + if row_ix == target_row { + return row.paragraph_id.map(|_| paragraph); + } + if row.paragraph_id.is_some() { + paragraph += 1; + } + } + None +} + +fn sanitized_recovery_title(title: &str) -> String { + let mut out = String::new(); + for ch in title.chars() { + if out.len() >= 48 { + break; + } + if ch.is_ascii_alphanumeric() || matches!(ch, '-' | '_' | '.') { + out.push(ch); + } else if ch.is_whitespace() && !out.ends_with('_') { + out.push('_'); + } + } + let trimmed = out.trim_matches(['_', '.']).to_string(); + if trimmed.is_empty() { + "shared-document".to_string() + } else { + trimmed + } +} diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index a81dd7dd..9a9446ef 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -12,7 +12,7 @@ use flowstate_collab::{ binding::DocBinding, local_apply::LocalApplier, net::{NetCommand, PublishPayload, runtime::CommandSender, direct::{DirectServeRequest, DirectSessionHandler}}, - presence::{PresenceState, PresenceStore}, + presence::{PresenceSelection, PresenceState, PresenceStore}, projection, proto_direct::AssetBytes, proto_gossip::GossipMsg, @@ -26,6 +26,8 @@ use uuid::Uuid; use crate::app_settings::load_document_theme; use crate::rich_text_element::{AssetId, CollabPatch, Document, EditorEvent, RichTextEditor, UndoRedirect}; +use super::presence_view; + #[derive(Clone, Debug)] pub enum SessionPhase { Creating, @@ -447,6 +449,9 @@ impl CollabSession { }; editor.update(cx, |editor, _| { + if editor.document_path().is_none() { + editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title))); + } editor.set_collab_capture(true); editor.set_collab_undo_redirect(Some(Rc::new(|_redirect: UndoRedirect| { // TODO(Flowstate P2P §8/M3): replace this guard with Loro UndoManager undo/redo routing. @@ -605,6 +610,7 @@ impl CollabSession { } let patches = std::mem::take(&mut self.pending_remote_patches); editor.update(cx, |editor, cx| editor.apply_collab_patches(&patches, cx)); + self.refresh_external_carets(cx); true } @@ -702,17 +708,18 @@ impl CollabSession { eprintln!("flowstate collab presence update failed: {error:#}"); } self.refresh_peer_count(); + self.refresh_external_carets(cx); cx.notify(); } } - fn refresh_own_presence(&mut self, _cx: &mut Context) { + fn refresh_own_presence(&mut self, cx: &mut Context) { let Some(presence) = &self.presence else { return; }; let state = PresenceState { name: default_presence_name(), - selection: None, + selection: self.own_presence_selection(cx), }; if let Err(error) = presence.set_self(&state) { eprintln!("flowstate collab presence encode failed: {error:#}"); @@ -720,6 +727,29 @@ impl CollabSession { self.refresh_peer_count(); } + fn own_presence_selection(&self, cx: &mut Context) -> Option { + let editor = self.editor.as_ref()?.read(cx); + let binding = self.binding.as_ref()?; + presence_view::selection_for_editor(editor, binding) + } + + fn refresh_external_carets(&mut self, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let Some(doc) = &self.doc else { + return; + }; + let Some(binding) = &self.binding else { + return; + }; + let Some(editor) = self.editor.clone() else { + return; + }; + let carets = presence_view::external_carets(doc, binding, presence); + editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); + } + fn publish_presence_snapshot(&self) { if let Some(presence) = &self.presence { self.publish_presence_bytes(presence.encode_all()); diff --git a/crates/flowstate/src/collab/status.rs b/crates/flowstate/src/collab/status.rs new file mode 100644 index 00000000..4be211ae --- /dev/null +++ b/crates/flowstate/src/collab/status.rs @@ -0,0 +1,66 @@ +use gpui::{AnyElement, App, IntoElement, div, px, prelude::*}; +use gpui_component::{ActiveTheme as _, h_flex}; + +use super::{Attachment, Connectivity, JoinStage, SessionPhase}; + +pub fn status_pill(phase: &SessionPhase, cx: &App) -> AnyElement { + let (label, color) = status_label_and_color(phase, cx); + h_flex() + .flex_none() + .items_center() + .gap_1() + .px_2() + .py_0p5() + .rounded_full() + .border_1() + .border_color(color.opacity(0.38)) + .bg(color.opacity(0.10)) + .child( + div() + .size(px(6.0)) + .rounded_full() + .bg(color), + ) + .child( + div() + .text_size(px(10.0)) + .text_color(color) + .child(label), + ) + .into_any_element() +} + +fn status_label_and_color(phase: &SessionPhase, cx: &App) -> (String, gpui::Hsla) { + match phase { + SessionPhase::Creating => ("Starting share".to_string(), cx.theme().info), + SessionPhase::Joining(stage) => (join_label(stage), cx.theme().info), + SessionPhase::Attached(attachment) => attached_label_and_color(attachment, cx), + SessionPhase::Detached(_) => ("Not shared".to_string(), cx.theme().muted_foreground), + } +} + +fn join_label(stage: &JoinStage) -> String { + match stage { + JoinStage::Resolving => "Resolving invite".to_string(), + JoinStage::Subscribing => "Joining session".to_string(), + JoinStage::FetchingSnapshot { got, total } => match total { + Some(total) => format!("Fetching {got}/{total} bytes"), + None => "Fetching document".to_string(), + }, + JoinStage::Building => "Opening shared doc".to_string(), + } +} + +fn attached_label_and_color(attachment: &Attachment, cx: &App) -> (String, gpui::Hsla) { + match attachment.connectivity { + Connectivity::Online => { + let participants = attachment.peers_present + 1; + if attachment.peers_present == 0 { + ("Only you".to_string(), cx.theme().success) + } else { + (format!("{participants} in session"), cx.theme().success) + } + }, + Connectivity::Offline { .. } => ("Offline - will sync".to_string(), cx.theme().warning), + } +} diff --git a/crates/flowstate/src/workspace/workspace/render_status.rs b/crates/flowstate/src/workspace/workspace/render_status.rs index 7fede050..1f62927a 100644 --- a/crates/flowstate/src/workspace/workspace/render_status.rs +++ b/crates/flowstate/src/workspace/workspace/render_status.rs @@ -264,6 +264,9 @@ impl Workspace { None }; let zoom = self.active_editor.as_ref().map(|editor| editor.read(cx).zoom_percent()); + let collab_phase = self + .active_document_id + .and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)); if let Some(percent) = zoom { self.sync_zoom_slider(percent, window, cx); } @@ -277,6 +280,13 @@ impl Workspace { .border_color(cx.theme().border) .bg(cx.theme().background) .child(div().flex_1()) + .when_some(collab_phase, |this, phase| { + if matches!(phase, crate::collab::SessionPhase::Detached(_)) { + this + } else { + this.child(crate::collab::status::status_pill(&phase, cx)) + } + }) .when_some(zoom, |this, percent| this.child(self.render_zoom_slider(percent, cx))) .when_some(speech_word_count, |this, count| { this.child( From cf7a44976a4acb713a33e5cdd8eec6edee23774a Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Sat, 13 Jun 2026 18:20:56 -0500 Subject: [PATCH 05/45] finished off first pass plan implemnentation --- .../flowstate-collab/src/net/anti_entropy.rs | 5 + crates/flowstate-collab/src/net/mod.rs | 9 +- crates/flowstate-collab/src/net/runtime.rs | 16 +- crates/flowstate-collab/src/schema.rs | 4 + crates/flowstate-collab/tests/anti_entropy.rs | 115 ++++ crates/flowstate-collab/tests/convergence.rs | 158 ++++++ crates/flowstate-collab/tests/projection.rs | 65 ++- crates/flowstate-collab/tests/translation.rs | 327 +++++++++++ crates/flowstate/src/collab/asset_transfer.rs | 198 +++++++ crates/flowstate/src/collab/mod.rs | 17 +- crates/flowstate/src/collab/session.rs | 509 ++++++----------- crates/flowstate/src/collab/session_io.rs | 313 +++++++++++ .../flowstate/src/collab/session_presence.rs | 92 +++ crates/flowstate/src/collab/session_timers.rs | 272 +++++++++ crates/flowstate/src/collab/share_dialog.rs | 530 ++++++++++++++++++ crates/flowstate/src/collab/shutdown.rs | 23 + crates/flowstate/src/collab/status.rs | 38 +- crates/flowstate/src/commands.rs | 4 + crates/flowstate/src/commands/keymap.rs | 2 + crates/flowstate/src/rich_text_element/mod.rs | 2 + .../src/workspace/workspace/collab.rs | 85 ++- .../src/workspace/workspace/collab_prompts.rs | 324 +++++++++++ .../src/workspace/workspace/documents.rs | 7 + .../src/workspace/workspace/keybindings.rs | 16 +- .../flowstate/src/workspace/workspace/mod.rs | 2 + .../workspace/workspace/render_documents.rs | 4 + .../src/workspace/workspace/render_status.rs | 8 +- .../src/workspace/workspace/render_top_bar.rs | 6 + .../src/workspace/workspace/top_bar.rs | 50 +- .../src/workspace/workspace/traits.rs | 1 + .../src/workspace/workspace/window.rs | 84 +++ crates/gpui-flowtext/src/document/blocks.rs | 10 + .../src/rich_text/editor/caret_movement.rs | 3 + .../src/rich_text/editor/collab_apply.rs | 43 ++ .../src/rich_text/editor/lifecycle.rs | 8 +- .../src/rich_text/editor/object_assets.rs | 30 +- .../src/rich_text/editor/render_blocks.rs | 32 +- .../src/rich_text/layout/block_layout.rs | 11 + helpers/docs/collab_qa.md | 59 ++ 39 files changed, 3101 insertions(+), 381 deletions(-) create mode 100644 crates/flowstate-collab/tests/anti_entropy.rs create mode 100644 crates/flowstate-collab/tests/convergence.rs create mode 100644 crates/flowstate-collab/tests/translation.rs create mode 100644 crates/flowstate/src/collab/asset_transfer.rs create mode 100644 crates/flowstate/src/collab/session_io.rs create mode 100644 crates/flowstate/src/collab/session_presence.rs create mode 100644 crates/flowstate/src/collab/session_timers.rs create mode 100644 crates/flowstate/src/collab/share_dialog.rs create mode 100644 crates/flowstate/src/collab/shutdown.rs create mode 100644 crates/flowstate/src/workspace/workspace/collab_prompts.rs create mode 100644 helpers/docs/collab_qa.md diff --git a/crates/flowstate-collab/src/net/anti_entropy.rs b/crates/flowstate-collab/src/net/anti_entropy.rs index 0c915d9c..ecb80adf 100644 --- a/crates/flowstate-collab/src/net/anti_entropy.rs +++ b/crates/flowstate-collab/src/net/anti_entropy.rs @@ -49,6 +49,11 @@ impl AntiEntropyState { now >= self.next_digest } + #[must_use] + pub fn duration_until_digest(&self, now: Instant) -> Duration { + self.next_digest.saturating_duration_since(now) + } + pub fn mark_digest_sent(&mut self, now: Instant) { self.next_digest = next_jittered_deadline(now); } diff --git a/crates/flowstate-collab/src/net/mod.rs b/crates/flowstate-collab/src/net/mod.rs index 6f118a62..9cef1b23 100644 --- a/crates/flowstate-collab/src/net/mod.rs +++ b/crates/flowstate-collab/src/net/mod.rs @@ -13,6 +13,7 @@ use crate::{ids::{BlobId, SessionId}, proto_direct::AssetBytes, proto_gossip::Go use self::direct::DirectSessionHandler; pub type Reply = Sender; +pub type PeerAddr = EndpointAddr; #[derive(Clone, Debug)] pub enum PublishPayload { @@ -29,10 +30,10 @@ pub enum NetCommand { JoinSession { session: SessionId, bootstrap: Vec }, LeaveSession { session: SessionId }, Publish { session: SessionId, payload: PublishPayload }, - PullUpdates { session: SessionId, from: EndpointId, our_vv: Vec, reply: Reply>> }, - PullSnapshot { session: SessionId, from: EndpointId, reply: Reply>> }, - PullBlob { session: SessionId, from: EndpointId, blob: BlobId, reply: Reply>> }, - PullAsset { session: SessionId, from: EndpointId, asset: u128, reply: Reply> }, + PullUpdates { session: SessionId, candidates: Vec, our_vv: Vec, reply: Reply>> }, + PullSnapshot { session: SessionId, candidates: Vec, reply: Reply>> }, + PullBlob { session: SessionId, candidates: Vec, blob: BlobId, reply: Reply>> }, + PullAsset { session: SessionId, candidates: Vec, asset: u128, reply: Reply> }, MintTicketAddr { reply: Reply }, Shutdown, } diff --git a/crates/flowstate-collab/src/net/runtime.rs b/crates/flowstate-collab/src/net/runtime.rs index 56c6c36b..f09ca235 100644 --- a/crates/flowstate-collab/src/net/runtime.rs +++ b/crates/flowstate-collab/src/net/runtime.rs @@ -132,23 +132,23 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha }, NetCommand::PullUpdates { session, - from, + candidates, our_vv, reply, } => { - let result = direct::pull_with_fallback(DirectRequest::Updates { session, have_vv: our_vv }, vec![from], DIRECT_PULL_TIMEOUT).await; + let result = direct::pull_with_fallback(DirectRequest::Updates { session, have_vv: our_vv }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullSnapshot { session, from, reply } => { - let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, vec![from], DIRECT_PULL_TIMEOUT).await; + NetCommand::PullSnapshot { session, candidates, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullBlob { session, from, blob, reply } => { - let result = direct::pull_with_fallback(DirectRequest::Blob { session, blob }, vec![from], DIRECT_PULL_TIMEOUT).await; + NetCommand::PullBlob { session, candidates, blob, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Blob { session, blob }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullAsset { session, from, asset, reply } => { - let result = direct::pull_with_fallback(DirectRequest::Asset { session, asset }, vec![from], DIRECT_PULL_TIMEOUT) + NetCommand::PullAsset { session, candidates, asset, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Asset { session, asset }, candidates, DIRECT_PULL_TIMEOUT) .await .map(|bytes| AssetBytes { bytes }); let _ = reply.send(result).await; diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs index ac1e861d..925e1290 100644 --- a/crates/flowstate-collab/src/schema.rs +++ b/crates/flowstate-collab/src/schema.rs @@ -59,6 +59,10 @@ pub enum BlockPayload { Table(InputTableBlock), } +pub fn decode_block_payload(bytes: &[u8]) -> Result { + Ok(postcard::from_bytes(bytes)?) +} + #[must_use] pub fn payload_from_block(block: &Block, assets: &AssetStore) -> Option { match input_block_from_block(block) { diff --git a/crates/flowstate-collab/tests/anti_entropy.rs b/crates/flowstate-collab/tests/anti_entropy.rs new file mode 100644 index 00000000..932b678d --- /dev/null +++ b/crates/flowstate-collab/tests/anti_entropy.rs @@ -0,0 +1,115 @@ +#[cfg(test)] +mod tests { + use std::time::{Duration, Instant}; + + use flowstate_collab::{ + SessionId, + net::anti_entropy::{AntiEntropyState, GapAction, VersionVectorRelation}, + }; + use iroh::{EndpointId, SecretKey}; + + #[test] + fn digest_gap_starts_one_pull_until_the_pull_finishes() { + let session = SessionId::from_bytes([1; 32]); + let peer = peer(9); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::SenderHasMissingOps, vec![1, 2, 3]), + GapAction::Pull { + from: peer, + our_vv: vec![1, 2, 3], + } + ); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::Concurrent, vec![4, 5]), + GapAction::None + ); + + state.finish_pull(); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::Concurrent, vec![6]), + GapAction::Pull { + from: peer, + our_vv: vec![6], + } + ); + } + + #[test] + fn equal_or_sender_behind_digest_does_not_pull() { + let session = SessionId::from_bytes([2; 32]); + let peer = peer(10); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::Equal, vec![1]), + GapAction::None + ); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::WeHaveMissingOps, vec![2]), + GapAction::None + ); + } + + #[test] + fn lineage_mismatch_is_reported_without_starting_a_pull() { + let session = SessionId::from_bytes([3; 32]); + let other = SessionId::from_bytes([4; 32]); + let peer = peer(11); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!( + state.consider_digest(peer, other, VersionVectorRelation::SenderHasMissingOps, vec![1]), + GapAction::LineageMismatch { + from: peer, + expected: session, + got: other, + } + ); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::SenderHasMissingOps, vec![2]), + GapAction::Pull { + from: peer, + our_vv: vec![2], + } + ); + } + + #[test] + fn lagged_uses_the_fallback_peer_and_deduplicates_in_flight_pulls() { + let session = SessionId::from_bytes([5; 32]); + let peer = peer(12); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!(state.on_lagged(None, vec![1]), GapAction::None); + assert_eq!( + state.on_lagged(Some(peer), vec![2]), + GapAction::Pull { + from: peer, + our_vv: vec![2], + } + ); + assert_eq!(state.on_lagged(Some(peer), vec![3]), GapAction::None); + } + + #[test] + fn neighbor_and_reconnect_events_schedule_an_immediate_digest() { + let session = SessionId::from_bytes([6; 32]); + let now = Instant::now(); + let mut state = AntiEntropyState::new(session, now); + + assert!(!state.digest_due(now + Duration::from_secs(1))); + state.on_neighbor_up(); + assert!(state.digest_due(Instant::now() + Duration::from_millis(1))); + + state.mark_digest_sent(Instant::now()); + assert!(!state.digest_due(Instant::now() + Duration::from_secs(1))); + state.on_reconnect(); + assert!(state.digest_due(Instant::now() + Duration::from_millis(1))); + } + + fn peer(seed: u8) -> EndpointId { + SecretKey::from_bytes(&[seed; 32]).public() + } +} diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs new file mode 100644 index 00000000..3e6cf790 --- /dev/null +++ b/crates/flowstate-collab/tests/convergence.rs @@ -0,0 +1,158 @@ +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; + use gpui_flowtext::{ + CanonicalOperation, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyle, RunStyles, document_from_input_blocks, + insert_text_at, mutate_runs_in_range, paragraph_text, plain, + }; + use loro::{ExportMode, LoroDoc, Subscription}; + + const MULTIBYTE: &str = "aé🌍\u{2028}x"; + + #[test] + fn two_peers_converge_with_reordered_and_duplicated_update_imports() { + let session = SessionId::from_bytes([77; 32]); + let initial = document_from_input_blocks( + DocumentTheme::default(), + vec![ + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(MULTIBYTE)], + }), + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain("second aé🌍")], + }), + ], + ); + let mut peer_a = Peer::from_initial(session, initial); + let mut peer_b = Peer::from_snapshot(session, &peer_a.snapshot()); + + peer_a.insert_text(0, "a".len(), "A🌍", RunStyles::default().with(RunStyle::Semantic(2))); + let inserted = "A🌍".len(); + peer_a.set_run_styles( + 0, + "a".len().."a".len() + inserted, + RunStyles::default().with(RunStyle::Highlight(4)).with_direct_underline(), + ); + peer_b.insert_text(1, "second ".len(), "Bé", RunStyles::default().with_strikethrough()); + + let updates_a = peer_a.updates(); + let updates_b = peer_b.updates(); + assert_eq!(updates_a.len(), 2); + assert_eq!(updates_b.len(), 1); + + // Import A's second commit before its first commit to exercise Loro's pending-import path, + // then replay duplicates to assert idempotency. + import_sequence(&peer_b.loro, [&updates_a[1], &updates_a[1], &updates_a[0], &updates_a[0]]); + import_sequence(&peer_a.loro, [&updates_b[0], &updates_b[0]]); + + let projected_a = projection::document_from_loro(&peer_a.loro, DocumentTheme::default()).expect("projection A should succeed"); + let projected_b = projection::document_from_loro(&peer_b.loro, DocumentTheme::default()).expect("projection B should succeed"); + assert_documents_eq(&projected_a, &projected_b); + } + + struct Peer { + document: Document, + loro: LoroDoc, + binding: DocBinding, + updates: Arc>>>, + _subscription: Subscription, + } + + impl Peer { + fn from_initial(session: SessionId, document: Document) -> Self { + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, session, "convergence", &document).expect("populate should succeed"); + let binding = DocBinding::build(&loro, &document).expect("binding should build"); + Self::with_subscription(document, loro, binding) + } + + fn from_snapshot(session: SessionId, snapshot: &[u8]) -> Self { + let loro = schema::new_configured_doc(); + loro.import(snapshot).expect("snapshot import should succeed"); + projection::verify_lineage(&loro, session).expect("lineage should match"); + let document = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); + let binding = DocBinding::build(&loro, &document).expect("binding should build"); + Self::with_subscription(document, loro, binding) + } + + fn with_subscription(document: Document, loro: LoroDoc, binding: DocBinding) -> Self { + let updates = Arc::new(Mutex::new(Vec::new())); + let captured = updates.clone(); + let subscription = loro.subscribe_local_update(Box::new(move |bytes| { + captured.lock().expect("update capture lock should not be poisoned").push(bytes.clone()); + true + })); + Self { + document, + loro, + binding, + updates, + _subscription: subscription, + } + } + + fn snapshot(&self) -> Vec { + self.loro.export(ExportMode::Snapshot).expect("snapshot export should succeed") + } + + fn updates(&self) -> Vec> { + self.updates.lock().expect("update capture lock should not be poisoned").clone() + } + + fn insert_text(&mut self, paragraph_ix: usize, byte: usize, text: &str, styles: RunStyles) { + let paragraph = self.document.ids.paragraph_ids[paragraph_ix]; + insert_text_at(&mut self.document, paragraph_ix, byte, text, styles); + self.apply(&[CanonicalOperation::InsertText { + paragraph, + byte, + text: text.to_string(), + styles, + }]); + } + + fn set_run_styles(&mut self, paragraph_ix: usize, range: std::ops::Range, styles: RunStyles) { + let paragraph = self.document.ids.paragraph_ids[paragraph_ix]; + mutate_runs_in_range( + &mut self.document, + gpui_flowtext::DocumentOffset { + paragraph: paragraph_ix, + byte: range.start, + }..gpui_flowtext::DocumentOffset { + paragraph: paragraph_ix, + byte: range.end, + }, + |run_styles| *run_styles = styles, + ); + self.apply(&[CanonicalOperation::SetRunStyles { paragraph, range, styles }]); + } + + fn apply(&mut self, ops: &[CanonicalOperation]) { + LocalApplier { + doc: &self.loro, + binding: &mut self.binding, + } + .apply(&self.document, ops) + .expect("local apply should succeed"); + } + } + + fn import_sequence<'a>(doc: &LoroDoc, updates: impl IntoIterator>) { + for update in updates { + doc.import(update).expect("update import should succeed"); + } + } + + fn assert_documents_eq(left: &Document, right: &Document) { + assert_eq!(left.blocks.len(), right.blocks.len()); + assert_eq!(left.paragraphs.len(), right.paragraphs.len()); + for ix in 0..left.paragraphs.len() { + assert_eq!(paragraph_text(left, ix), paragraph_text(right, ix)); + assert_eq!(left.paragraphs[ix].style, right.paragraphs[ix].style); + assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); + } + } +} diff --git a/crates/flowstate-collab/tests/projection.rs b/crates/flowstate-collab/tests/projection.rs index dc34aa51..4acee34c 100644 --- a/crates/flowstate-collab/tests/projection.rs +++ b/crates/flowstate-collab/tests/projection.rs @@ -3,10 +3,13 @@ mod tests { use flowstate_collab::binding::DocBinding; use flowstate_collab::{SessionId, projection, schema}; use gpui_flowtext::{ - Document, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, - RunStyles, document_from_input_blocks, paragraph_text, plain, run, + Block, Document, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, InputTableBlock, + InputTableCell, InputTableCellBlock, InputTableColumnWidth, InputTableRow, InputTableStyle, ParagraphStyle, RunStyle, RunStyles, + document_from_input_blocks, paragraph_text, plain, run, }; + const MULTIBYTE: &str = "aé🌍\u{2028}x"; + #[test] fn projection_round_trips_text_marks_and_equation_blocks() { let input_blocks = vec![ @@ -22,6 +25,20 @@ mod tests { syntax: InputEquationSyntax::Latex, display: InputEquationDisplay::Display, }), + InputBlock::Table(InputTableBlock { + rows: vec![InputTableRow { + cells: vec![InputTableCell { + blocks: vec![InputTableCellBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(MULTIBYTE)], + })], + row_span: 1, + col_span: 1, + }], + }], + column_widths: vec![InputTableColumnWidth::Auto], + style: InputTableStyle { header_row: true }, + }), ]; let original = document_from_input_blocks(DocumentTheme::default(), input_blocks); let loro = schema::new_configured_doc(); @@ -36,6 +53,28 @@ mod tests { assert_eq!(binding.rows.len(), projected.blocks.len()); } + #[test] + fn offset_helpers_round_trip_all_multibyte_boundaries() { + let original = document_from_input_blocks( + DocumentTheme::default(), + vec![InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(MULTIBYTE)], + })], + ); + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, SessionId::from_bytes([8; 32]), "offsets", &original).expect("populate should succeed"); + let binding = DocBinding::build(&loro, &original).expect("binding should build"); + let text = binding.rows[0].text.as_ref().expect("first row should be paragraph text"); + let byte_to_unicode = [(0, 0), ("a".len(), 1), ("aé".len(), 2), ("aé🌍".len(), 3), ("aé🌍\u{2028}".len(), 4), (MULTIBYTE.len(), 5)]; + + assert_eq!(text.len_utf8(), MULTIBYTE.len()); + for (byte, unicode) in byte_to_unicode { + assert_eq!(schema::loro_pos(text, byte), unicode); + assert_eq!(schema::utf8_byte(text, unicode), byte); + } + } + fn assert_document_shape_eq(left: &Document, right: &Document) { assert_eq!(left.blocks.len(), right.blocks.len()); assert_eq!(left.paragraphs.len(), right.paragraphs.len()); @@ -45,7 +84,27 @@ mod tests { assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); } for (left, right) in left.blocks.iter().zip(right.blocks.iter()) { - assert_eq!(std::mem::discriminant(left), std::mem::discriminant(right)); + match (left, right) { + (Block::Paragraph(_), Block::Paragraph(_)) => {}, + (Block::Equation(left), Block::Equation(right)) => { + assert_eq!(left.source, right.source); + assert_eq!(left.syntax, right.syntax); + assert_eq!(left.display, right.display); + }, + (Block::Table(left), Block::Table(right)) => { + assert_eq!(left.rows, right.rows); + assert_eq!(left.column_widths, right.column_widths); + assert_eq!(left.style, right.style); + }, + (Block::Image(left), Block::Image(right)) => { + assert_eq!(left.asset_id, right.asset_id); + assert_eq!(left.alt_text, right.alt_text); + assert_eq!(left.caption, right.caption); + assert_eq!(left.sizing, right.sizing); + assert_eq!(left.alignment, right.alignment); + }, + _ => panic!("projected block kind changed"), + } } } } diff --git a/crates/flowstate-collab/tests/translation.rs b/crates/flowstate-collab/tests/translation.rs new file mode 100644 index 00000000..358dbc26 --- /dev/null +++ b/crates/flowstate-collab/tests/translation.rs @@ -0,0 +1,327 @@ +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; + use gpui_flowtext::{ + Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, + InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, RunStyles, block_from_input_block, capture_document_span, + delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, + mutate_runs_in_range, paragraph_text, paragraphs_mut, plain, remove_block_ids, run, split_paragraph_at, update_paragraph_block, + }; + use loro::LoroDoc; + + const MULTIBYTE: &str = "aé🌍\u{2028}x"; + + #[test] + fn delete_range_uses_utf8_byte_offsets() { + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain(MULTIBYTE)])]); + let paragraph = document.ids.paragraph_ids[0]; + let start = "a".len(); + let end = "aé🌍".len(); + + delete_range_in_paragraph(&mut document, 0, start..end); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::DeleteRange { + start_paragraph: paragraph, + start_byte: start, + end_paragraph: paragraph, + end_byte: end, + }], + ); + } + + #[test] + fn split_and_join_paragraphs_preserve_multibyte_text_and_styles() { + let semantic = RunStyles::default().with(RunStyle::Semantic(2)).with_direct_underline(); + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("aé"), run("🌍\u{2028}x", semantic)])]); + let original = document.ids.paragraph_ids[0]; + let split_byte = "aé".len(); + + split_paragraph_at(&mut document, 0, split_byte); + let split = document.ids.paragraph_ids[1]; + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SplitParagraph { + paragraph: original, + byte: split_byte, + new_paragraph: split, + }], + ); + + let first_len = paragraph_text(&document, 0).len(); + delete_cross_paragraph_range( + &mut document, + DocumentOffset { + paragraph: 0, + byte: first_len, + }..DocumentOffset { paragraph: 1, byte: 0 }, + ); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::JoinParagraphs { + first: original, + second: split, + }], + ); + } + + #[test] + fn paragraph_and_run_style_updates_match_flowtext_runs() { + let base_style = RunStyles::default().with(RunStyle::Semantic(1)).with_strikethrough(); + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![run(MULTIBYTE, base_style)])]); + let paragraph = document.ids.paragraph_ids[0]; + + paragraphs_mut(&mut document)[0].style = ParagraphStyle::Custom(6); + update_paragraph_block(&mut document, 0); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SetParagraphStyle { + paragraph, + style: ParagraphStyle::Custom(6), + }], + ); + + let unstyled = "a".len().."aé".len(); + mutate_runs_in_range( + &mut document, + DocumentOffset { + paragraph: 0, + byte: unstyled.start, + }..DocumentOffset { + paragraph: 0, + byte: unstyled.end, + }, + |styles| *styles = RunStyles::default(), + ); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SetRunStyles { + paragraph, + range: unstyled, + styles: RunStyles::default(), + }], + ); + + let highlighted = "aé".len().."aé🌍".len(); + let highlighted_styles = RunStyles::default() + .with(RunStyle::Semantic(4)) + .with(RunStyle::Highlight(3)) + .with_direct_underline() + .with_strikethrough(); + mutate_runs_in_range( + &mut document, + DocumentOffset { + paragraph: 0, + byte: highlighted.start, + }..DocumentOffset { + paragraph: 0, + byte: highlighted.end, + }, + |styles| *styles = highlighted_styles, + ); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SetRunStyles { + paragraph, + range: highlighted, + styles: highlighted_styles, + }], + ); + } + + #[test] + fn replace_paragraph_span_refreshes_text_runs_and_styles() { + let (mut document, loro, mut binding) = collab_fixture(vec![ + paragraph_block(vec![plain(MULTIBYTE)]), + paragraph_block(vec![plain("second aé")]), + ]); + let start_paragraph = document.ids.paragraph_ids[0]; + let before = capture_document_span(&document, 0..2); + let inserted_styles = RunStyles::default().with(RunStyle::Semantic(5)).with_direct_underline(); + + insert_text_at(&mut document, 0, "a".len(), "Zé", inserted_styles); + paragraphs_mut(&mut document)[1].style = ParagraphStyle::Custom(2); + update_paragraph_block(&mut document, 1); + let after = capture_document_span(&document, 0..2); + + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::ReplaceParagraphSpan { + start_paragraph: Some(start_paragraph), + before, + after, + }], + ); + } + + #[test] + fn insert_move_replace_and_delete_object_blocks_match_flowtext_document() { + let (mut document, loro, mut binding) = collab_fixture(vec![ + paragraph_block(vec![plain("one aé")]), + paragraph_block(vec![plain("two 🌍")]), + ]); + let equation = equation_block("x = aé + 🌍"); + + let equation_id = insert_object_block(&mut document, 1, equation); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::InsertBlock { + block: equation_id, + block_ix: 1, + }], + ); + + move_block(&mut document, 1, 2); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::MoveBlock { + block: equation_id, + new_block_ix: 2, + }], + ); + + replace_object_block(&mut document, 2, equation_block("y = 🌍^2")); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::ReplaceBlock { block: Some(equation_id) }], + ); + + delete_object_block(&mut document, 2); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::DeleteBlock { block: equation_id }], + ); + } + + #[test] + fn replace_document_rebuilds_the_loro_projection() { + let (_, loro, mut binding) = collab_fixture(vec![ + paragraph_block(vec![plain("old")]), + equation_block("old = 1"), + ]); + let document = document_from_input_blocks( + DocumentTheme::default(), + vec![ + paragraph_block(vec![plain("new aé"), run("🌍", RunStyles::default().with(RunStyle::Highlight(1)))]), + equation_block("new = aé + 🌍"), + paragraph_block(vec![plain("tail")]), + ], + ); + + apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::ReplaceDocument]); + } + + fn collab_fixture(blocks: Vec) -> (Document, LoroDoc, DocBinding) { + let document = document_from_input_blocks(DocumentTheme::default(), blocks); + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, SessionId::from_bytes([42; 32]), "translation", &document) + .expect("populate should succeed"); + let binding = DocBinding::build(&loro, &document).expect("binding should build"); + (document, loro, binding) + } + + fn apply_and_assert_projection(document: &Document, loro: &LoroDoc, binding: &mut DocBinding, ops: &[CanonicalOperation]) { + LocalApplier { doc: loro, binding } + .apply(document, ops) + .expect("local apply should succeed"); + let projected = projection::document_from_loro(loro, DocumentTheme::default()).expect("projection should succeed"); + assert_document_projection_eq(document, &projected); + } + + fn paragraph_block(runs: Vec) -> InputBlock { + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs, + }) + } + + fn equation_block(source: &str) -> InputBlock { + InputBlock::Equation(InputEquationBlock { + source: source.to_string(), + syntax: InputEquationSyntax::Latex, + display: InputEquationDisplay::Display, + }) + } + + fn insert_object_block(document: &mut Document, block_ix: usize, input: InputBlock) -> gpui_flowtext::BlockId { + Arc::make_mut(&mut document.blocks).insert(block_ix, block_from_input_block(&input)); + insert_block_id(document, block_ix) + } + + fn move_block(document: &mut Document, from: usize, to: usize) { + let blocks = Arc::make_mut(&mut document.blocks); + let block = blocks.remove(from); + blocks.insert(to.min(blocks.len()), block); + let block_id = document.ids.block_ids.remove(from); + document.ids.block_ids.insert(to.min(document.ids.block_ids.len()), block_id); + } + + fn replace_object_block(document: &mut Document, block_ix: usize, input: InputBlock) { + Arc::make_mut(&mut document.blocks)[block_ix] = block_from_input_block(&input); + } + + fn delete_object_block(document: &mut Document, block_ix: usize) { + Arc::make_mut(&mut document.blocks).remove(block_ix); + remove_block_ids(document, block_ix..block_ix + 1); + } + + fn assert_document_projection_eq(expected: &Document, actual: &Document) { + assert_eq!(expected.blocks.len(), actual.blocks.len()); + assert_eq!(expected.paragraphs.len(), actual.paragraphs.len()); + let mut paragraph_ix = 0; + for (expected_block, actual_block) in expected.blocks.iter().zip(actual.blocks.iter()) { + match (expected_block, actual_block) { + (Block::Paragraph(expected_paragraph), Block::Paragraph(actual_paragraph)) => { + assert_eq!(paragraph_text(expected, paragraph_ix), paragraph_text(actual, paragraph_ix)); + assert_eq!(expected_paragraph.style, actual_paragraph.style); + assert_eq!(expected_paragraph.runs, actual_paragraph.runs); + }, + (Block::Equation(expected), Block::Equation(actual)) => { + assert_eq!(expected.source, actual.source); + assert_eq!(expected.syntax, actual.syntax); + assert_eq!(expected.display, actual.display); + }, + (Block::Image(expected), Block::Image(actual)) => { + assert_eq!(expected.asset_id, actual.asset_id); + assert_eq!(expected.alt_text, actual.alt_text); + assert_eq!(expected.caption, actual.caption); + assert_eq!(expected.sizing, actual.sizing); + assert_eq!(expected.alignment, actual.alignment); + }, + (Block::Table(expected), Block::Table(actual)) => { + assert_eq!(expected.rows, actual.rows); + assert_eq!(expected.column_widths, actual.column_widths); + assert_eq!(expected.style, actual.style); + }, + _ => panic!("projected block kind changed"), + } + if matches!(expected_block, Block::Paragraph(_)) { + paragraph_ix += 1; + } + } + } +} diff --git a/crates/flowstate/src/collab/asset_transfer.rs b/crates/flowstate/src/collab/asset_transfer.rs new file mode 100644 index 00000000..fd75720b --- /dev/null +++ b/crates/flowstate/src/collab/asset_transfer.rs @@ -0,0 +1,198 @@ +use std::{collections::hash_map::DefaultHasher, hash::{Hash, Hasher}, sync::Arc}; + +use anyhow::{Context as _, Result, anyhow, ensure}; +use flowstate_collab::{ + net::NetCommand, + schema::{BLOCKS, BlockPayload, DATA, KIND, KIND_IMAGE, decode_block_payload}, +}; +use gpui::{Context, SharedString}; +use loro::{Container, LoroValue, ValueOrContainer}; + +use crate::rich_text_element::{AssetId, AssetRecord, CollabPatch}; + +use super::CollabSession; + +pub(super) fn schedule_missing_assets( + session: &mut CollabSession, + preferred_peer: Option, + cx: &mut Context, +) { + let Some(editor) = session.editor.clone() else { + return; + }; + let Some(doc) = &session.doc else { + return; + }; + + let assets = match image_assets_in_loro(doc) { + Ok(assets) => assets, + Err(error) => { + eprintln!("flowstate collab asset scan failed: {error:#}"); + return; + }, + }; + if assets.is_empty() { + return; + } + + let missing_assets = { + let editor = editor.read(cx); + assets + .iter() + .filter(|meta| { + let id = AssetId(meta.asset_id); + editor + .document() + .assets + .assets + .get(&id) + .is_none_or(AssetRecord::is_loading_placeholder) + }) + .cloned() + .collect::>() + }; + if missing_assets.is_empty() { + return; + } + + let placeholders = missing_assets + .iter() + .map(|meta| CollabPatch::AssetArrived { + id: AssetId(meta.asset_id), + record: meta.placeholder_record(), + }) + .collect::>(); + session.apply_or_queue_patches(placeholders, cx); + + let candidates = session.pull_candidates(preferred_peer); + if candidates.is_empty() { + return; + } + + for meta in missing_assets { + let id = AssetId(meta.asset_id); + if !session.asset_pulls_in_flight.insert(id) { + continue; + } + start_asset_pull(session, candidates.clone(), meta, cx); + } +} + +fn start_asset_pull( + session: &mut CollabSession, + candidates: Vec, + meta: ImageAssetMeta, + cx: &mut Context, +) { + let (reply_tx, reply_rx) = async_channel::bounded(1); + let id = AssetId(meta.asset_id); + if session + .net_tx + .try_send(NetCommand::PullAsset { + session: session.session, + candidates, + asset: meta.asset_id, + reply: reply_tx, + }) + .is_err() + { + session.asset_pulls_in_flight.remove(&id); + return; + } + + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + session.asset_pulls_in_flight.remove(&id); + match result { + Ok(Ok(bytes)) => match meta.record_from_bytes(bytes.bytes) { + Ok(record) => { + session.apply_or_queue_patches(vec![CollabPatch::AssetArrived { id, record }], cx); + }, + Err(error) => eprintln!("flowstate collab rejected fetched asset {id:?}: {error:#}"), + }, + Ok(Err(error)) => eprintln!("flowstate collab asset pull failed for {id:?}: {error:#}"), + Err(error) => eprintln!("flowstate collab asset pull channel closed for {id:?}: {error}"), + } + }); + }) + .detach(); +} + +#[derive(Clone, Debug)] +struct ImageAssetMeta { + asset_id: u128, + mime: String, + original_name: Option, + content_hash: u64, + byte_len: u64, +} + +impl ImageAssetMeta { + fn placeholder_record(&self) -> AssetRecord { + AssetRecord { + id: AssetId(self.asset_id), + mime_type: SharedString::from(self.mime.clone()), + original_name: self.original_name.clone().map(SharedString::from), + content_hash: self.content_hash, + bytes: Arc::new(Vec::new()), + } + } + + fn record_from_bytes(&self, bytes: Vec) -> Result { + ensure!(bytes.len() as u64 == self.byte_len, "asset byte length mismatch"); + let mut hasher = DefaultHasher::new(); + bytes.hash(&mut hasher); + let content_hash = hasher.finish(); + ensure!(content_hash == self.content_hash, "asset content hash mismatch"); + Ok(AssetRecord { + id: AssetId(self.asset_id), + mime_type: SharedString::from(self.mime.clone()), + original_name: self.original_name.clone().map(SharedString::from), + content_hash, + bytes: Arc::new(bytes), + }) + } +} + +fn image_assets_in_loro(doc: &loro::LoroDoc) -> Result> { + let blocks = doc.get_movable_list(BLOCKS); + let mut assets = Vec::new(); + for ix in 0..blocks.len() { + let Some(ValueOrContainer::Container(Container::Map(map))) = blocks.get(ix) else { + continue; + }; + let Some(ValueOrContainer::Value(LoroValue::String(kind))) = map.get(KIND) else { + continue; + }; + if kind.as_ref() != KIND_IMAGE { + continue; + } + let Some(ValueOrContainer::Value(LoroValue::Binary(data))) = map.get(DATA) else { + return Err(anyhow!("image block {ix} is missing asset metadata")); + }; + let payload = decode_block_payload(&data).context("decoding image asset metadata failed")?; + let BlockPayload::Image { + asset_id, + mime, + original_name, + content_hash, + byte_len, + .. + } = payload + else { + continue; + }; + if byte_len == 0 { + continue; + } + assets.push(ImageAssetMeta { + asset_id, + mime, + original_name, + content_hash, + byte_len, + }); + } + Ok(assets) +} diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index 3bff0775..c041cb16 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -1,5 +1,7 @@ mod session; mod presence_view; +pub mod share_dialog; +mod shutdown; pub mod status; use std::collections::HashMap; @@ -16,7 +18,8 @@ use uuid::Uuid; use crate::rich_text_element::RichTextEditor; -pub use session::{Attachment, CollabSession, Connectivity, DetachReason, JoinStage, JoinedDocument, SessionPhase}; +pub use session::{Attachment, CollabSession, Connectivity, DetachReason, JoinStage, JoinedDocument, SessionPhase, SessionRosterEntry}; +pub use shutdown::shutdown; pub struct JoinRequest { pub session: SessionId, @@ -92,7 +95,7 @@ impl CollabManager { let commands = self.ensure_runtime(cx)?; let session = ticket.session; let inviter = ticket.inviter.id; - let collab = CollabSession::joining(session, ticket.title.clone(), commands.clone()); + let collab = CollabSession::joining(session, ticket.title.clone(), commands.clone(), vec![ticket.inviter.clone()]); let entity = cx.new(|_| collab); self.register_session(entity.clone(), cx); @@ -196,6 +199,12 @@ impl CollabManager { .map(|session| session.read(cx).phase().clone()) } + pub fn roster_for_panel(&self, panel_id: Uuid, cx: &App) -> Vec { + self + .session_for_panel(panel_id) + .map_or_else(Vec::new, |session| session.read(cx).roster()) + } + fn register_session(&mut self, session: Entity, cx: &mut Context) { let id = session.read(cx).session_id(); if let Some(panel_id) = session.read(cx).panel_id() { @@ -394,3 +403,7 @@ where pub fn phase_for_panel(panel_id: Uuid, cx: &App) -> Option { CollabManager::global(cx).phase_for_panel(panel_id, cx) } + +pub fn roster_for_panel(panel_id: Uuid, cx: &App) -> Vec { + CollabManager::global(cx).roster_for_panel(panel_id, cx) +} diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index 9a9446ef..92443787 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -1,8 +1,6 @@ use std::{ - cmp::Ordering, collections::HashSet, rc::Rc, - sync::{Arc, Mutex}, time::Instant, }; @@ -11,16 +9,14 @@ use flowstate_collab::{ SessionId, binding::DocBinding, local_apply::LocalApplier, - net::{NetCommand, PublishPayload, runtime::CommandSender, direct::{DirectServeRequest, DirectSessionHandler}}, - presence::{PresenceSelection, PresenceState, PresenceStore}, + net::{NetCommand, PeerAddr, PublishPayload, anti_entropy::AntiEntropyState, direct::{DirectServeRequest, DirectSessionHandler}, runtime::CommandSender}, + presence::PresenceStore, projection, - proto_direct::AssetBytes, proto_gossip::GossipMsg, - remote_apply::RemoteApplier, schema, }; use gpui::{Context, Entity, Subscription}; -use loro::{ExportMode, LoroDoc, Subscription as LoroSubscription, VersionVector, event::Subscriber}; +use loro::{LoroDoc, Subscription as LoroSubscription, UndoManager}; use uuid::Uuid; use crate::app_settings::load_document_theme; @@ -28,6 +24,15 @@ use crate::rich_text_element::{AssetId, CollabPatch, Document, EditorEvent, Rich use super::presence_view; +#[path = "asset_transfer.rs"] +mod asset_transfer; +#[path = "session_io.rs"] +mod session_io; +#[path = "session_presence.rs"] +mod session_presence; +#[path = "session_timers.rs"] +mod session_timers; + #[derive(Clone, Debug)] pub enum SessionPhase { Creating, @@ -69,6 +74,13 @@ pub struct JoinedDocument { pub document: Document, } +#[derive(Clone, Debug)] +pub struct SessionRosterEntry { + pub name: String, + pub color_rgb: u32, + pub is_self: bool, +} + pub struct CollabSession { session: SessionId, title: String, @@ -83,12 +95,25 @@ pub struct CollabSession { net_tx: CommandSender, direct_tx: async_channel::Sender, direct_rx: async_channel::Receiver, + undo_tx: async_channel::Sender, + undo_rx: async_channel::Receiver, editor_subscriptions: Vec, loro_subscriptions: Vec, neighbors: HashSet, - update_pull_in_flight: bool, + bootstrap_addrs: Vec, + asset_pulls_in_flight: HashSet, + anti_entropy: AntiEntropyState, + undo_manager: Option, direct_pump_started: bool, + undo_pump_started: bool, local_update_publish_attached: bool, + timers_started: bool, + endpoint_online: bool, + zero_neighbors_since: Option, + inbound_since_last_digest: bool, + quiet_digest_rounds: u8, + next_recovery_at: Option, + last_document_activity: Instant, } impl CollabSession { @@ -104,6 +129,8 @@ impl CollabSession { projection::populate_from_document(&doc, session, &title, &document)?; let binding = DocBinding::build(&doc, &document)?; let (direct_tx, direct_rx) = async_channel::unbounded(); + let (undo_tx, undo_rx) = async_channel::unbounded(); + let now = Instant::now(); Ok(Self { session, @@ -119,17 +146,32 @@ impl CollabSession { net_tx, direct_tx, direct_rx, + undo_tx, + undo_rx, editor_subscriptions: Vec::new(), loro_subscriptions: Vec::new(), neighbors: HashSet::new(), - update_pull_in_flight: false, + bootstrap_addrs: Vec::new(), + asset_pulls_in_flight: HashSet::new(), + anti_entropy: AntiEntropyState::new(session, now), + undo_manager: None, direct_pump_started: false, + undo_pump_started: false, local_update_publish_attached: false, + timers_started: false, + endpoint_online: true, + zero_neighbors_since: Some(now), + inbound_since_last_digest: false, + quiet_digest_rounds: 0, + next_recovery_at: None, + last_document_activity: now, }) } - pub fn joining(session: SessionId, title: String, net_tx: CommandSender) -> Self { + pub fn joining(session: SessionId, title: String, net_tx: CommandSender, bootstrap_addrs: Vec) -> Self { let (direct_tx, direct_rx) = async_channel::unbounded(); + let (undo_tx, undo_rx) = async_channel::unbounded(); + let now = Instant::now(); Self { session, title, @@ -144,12 +186,25 @@ impl CollabSession { net_tx, direct_tx, direct_rx, + undo_tx, + undo_rx, editor_subscriptions: Vec::new(), loro_subscriptions: Vec::new(), neighbors: HashSet::new(), - update_pull_in_flight: false, + bootstrap_addrs, + asset_pulls_in_flight: HashSet::new(), + anti_entropy: AntiEntropyState::new(session, now), + undo_manager: None, direct_pump_started: false, + undo_pump_started: false, local_update_publish_attached: false, + timers_started: false, + endpoint_online: true, + zero_neighbors_since: Some(now), + inbound_since_last_digest: false, + quiet_digest_rounds: 0, + next_recovery_at: None, + last_document_activity: now, } } @@ -169,10 +224,35 @@ impl CollabSession { &self.phase } + pub fn roster(&self) -> Vec { + let Some(presence) = &self.presence else { + return Vec::new(); + }; + let self_key = presence.self_key().to_string(); + presence + .roster() + .into_iter() + .map(|entry| SessionRosterEntry { + is_self: entry.key == self_key, + name: entry.name, + color_rgb: entry.color_rgb, + }) + .collect() + } + pub fn direct_handler(&self) -> DirectSessionHandler { DirectSessionHandler::new(self.direct_tx.clone()) } + pub(super) fn pull_candidates(&self, preferred: Option) -> Vec { + let mut candidates = Vec::with_capacity(self.neighbors.len() + usize::from(preferred.is_some())); + if let Some(peer) = preferred { + candidates.push(peer); + } + candidates.extend(self.neighbors.iter().copied().filter(|peer| Some(*peer) != preferred)); + candidates + } + pub fn set_join_stage(&mut self, stage: JoinStage, cx: &mut Context) { self.phase = SessionPhase::Joining(stage); cx.notify(); @@ -190,7 +270,7 @@ impl CollabSession { let (reply_tx, reply_rx) = async_channel::bounded(1); if let Err(error) = self.net_tx.try_send(NetCommand::PullSnapshot { session: self.session, - from: inviter, + candidates: self.pull_candidates(Some(inviter)), reply: reply_tx, }) { let detail = format!("requesting collaboration snapshot failed: {error}"); @@ -252,15 +332,19 @@ impl CollabSession { self.import_update_bytes(&update, cx)?; } self.flush_pending_remote_patches(cx); + asset_transfer::schedule_missing_assets(self, None, cx); self.publish_digest(); cx.notify(); Ok(()) } pub fn attach(&mut self, cx: &mut Context) { + self.attach_undo_manager(); self.attach_editor_hooks(cx); self.attach_loro_publish_hook(); self.attach_direct_request_pump(cx); + self.attach_undo_request_pump(cx); + self.attach_timers(cx); } pub fn establish_local_peer(&mut self, peer: &flowstate_collab::ids::PeerId, cx: &mut Context) { @@ -278,12 +362,14 @@ impl CollabSession { self.presence = Some(presence); } self.refresh_own_presence(cx); + self.endpoint_online = true; self.phase = SessionPhase::Attached(Attachment { connectivity: Connectivity::Online, peers_present: self.peers_present(), }); self.publish_presence_snapshot(); self.publish_digest(); + asset_transfer::schedule_missing_assets(self, None, cx); cx.notify(); } @@ -324,6 +410,7 @@ impl CollabSession { editor.update(cx, |editor, cx| { editor.set_collab_undo_redirect(None); editor.set_collab_capture(false); + editor.clear_collab_history(); let _ = editor.take_pending_collab_edits(); editor.set_external_carets(Vec::new(), cx); }); @@ -331,12 +418,18 @@ impl CollabSession { self.editor_subscriptions.clear(); self.loro_subscriptions.clear(); + self.undo_manager = None; self.presence = None; self.binding = None; self.doc = None; self.pending_remote_patches.clear(); self.pending_remote_updates.clear(); self.neighbors.clear(); + self.asset_pulls_in_flight.clear(); + self.zero_neighbors_since = Some(Instant::now()); + self.inbound_since_last_digest = false; + self.quiet_digest_rounds = 0; + self.next_recovery_at = None; self.local_update_publish_attached = false; self.phase = SessionPhase::Detached(reason); cx.notify(); @@ -349,7 +442,11 @@ impl CollabSession { } let document = editor.read(cx).document().clone(); - let edits = editor.update(cx, |editor, _| editor.take_pending_collab_edits()); + let edits = editor.update(cx, |editor, _| { + let edits = editor.take_pending_collab_edits(); + editor.clear_collab_history(); + edits + }); let Some(doc) = &self.doc else { return Ok(()); }; @@ -357,18 +454,26 @@ impl CollabSession { return Ok(()); }; + let mut applied = false; for edit in edits { if edit.operations.is_empty() { continue; } LocalApplier { doc, binding }.apply(&document, &edit.operations)?; + applied = true; + } + if applied { + self.last_document_activity = Instant::now(); } Ok(()) } pub fn handle_gossip(&mut self, from: flowstate_collab::ids::PeerId, msg: GossipMsg, cx: &mut Context) { + self.note_inbound_traffic(cx); let result = match msg { - GossipMsg::Update(bytes) => self.import_update_bytes(&bytes, cx), + GossipMsg::Update(bytes) => self + .import_update_bytes(&bytes, cx) + .map(|()| asset_transfer::schedule_missing_assets(self, Some(from), cx)), GossipMsg::UpdateAvailable { blob, .. } => { self.pull_blob(from, blob, cx); Ok(()) @@ -386,58 +491,41 @@ impl CollabSession { pub fn neighbor_up(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { self.neighbors.insert(peer); - if let SessionPhase::Attached(attachment) = &mut self.phase { - attachment.connectivity = Connectivity::Online; - } + self.zero_neighbors_since = None; + self.anti_entropy.on_neighbor_up(); + self.mark_online(cx); self.publish_digest(); + asset_transfer::schedule_missing_assets(self, Some(peer), cx); cx.notify(); } pub fn neighbor_down(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { self.neighbors.remove(&peer); + if self.neighbors.is_empty() && self.zero_neighbors_since.is_none() { + self.zero_neighbors_since = Some(Instant::now()); + } + self.evaluate_connectivity(cx); cx.notify(); } pub fn handle_gossip_lagged(&mut self, cx: &mut Context) { self.publish_digest(); - if let Some(peer) = self.neighbors.iter().next().copied() { - let vv = self.doc.as_ref().map(|doc| doc.oplog_vv().encode()).unwrap_or_default(); - self.pull_updates(peer, vv, cx); - } + let peer = self.neighbors.iter().next().copied(); + let vv = self.doc.as_ref().map(|doc| doc.oplog_vv().encode()).unwrap_or_default(); + let action = self.anti_entropy.on_lagged(peer, vv); + self.handle_gap_action(action, cx); } pub fn set_endpoint_online(&mut self, online: bool, cx: &mut Context) { - if let SessionPhase::Attached(attachment) = &mut self.phase { - attachment.connectivity = if online { - Connectivity::Online - } else { - Connectivity::Offline { since: Instant::now(), retries: 0 } - }; - cx.notify(); - } - } - - pub fn import_update_bytes(&mut self, bytes: &[u8], cx: &mut Context) -> Result<()> { - if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() { - self.pending_remote_updates.push(bytes.to_vec()); - return Ok(()); + self.endpoint_online = online; + if online { + if self.peers_present() == 0 || !self.neighbors.is_empty() { + self.mark_online(cx); + } + } else { + self.evaluate_connectivity(cx); } - - let editor = self.editor.clone().context("collaboration session has no editor")?; - let document = Arc::new(editor.read(cx).document().clone()); - let doc = self.doc.clone().context("collaboration session has no Loro document")?; - let binding = self.binding.take().context("collaboration session has no document binding")?; - let binding = Arc::new(Mutex::new(binding)); - let patches = Arc::new(Mutex::new(Vec::::new())); - let sub = self.diff_subscription(doc.clone(), document, binding.clone(), patches.clone()); - - let import_result = doc.import_with(bytes, "remote"); - drop(sub); - self.binding = Some(take_mutex_value(binding, "document binding")?); - let patches = take_mutex_value(patches, "remote patches")?; - import_result.context("importing collaboration update failed")?; - self.apply_or_queue_patches(patches, cx); - Ok(()) + cx.notify(); } fn attach_editor_hooks(&mut self, cx: &mut Context) { @@ -448,13 +536,15 @@ impl CollabSession { return; }; - editor.update(cx, |editor, _| { + editor.update(cx, |editor, cx| { if editor.document_path().is_none() { - editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title))); + editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title)), cx); } + editor.clear_collab_history(); editor.set_collab_capture(true); - editor.set_collab_undo_redirect(Some(Rc::new(|_redirect: UndoRedirect| { - // TODO(Flowstate P2P §8/M3): replace this guard with Loro UndoManager undo/redo routing. + let undo_tx = self.undo_tx.clone(); + editor.set_collab_undo_redirect(Some(Rc::new(move |redirect: UndoRedirect| { + let _ = undo_tx.try_send(redirect); }))); }); @@ -492,307 +582,40 @@ impl CollabSession { self.local_update_publish_attached = true; } - fn attach_direct_request_pump(&mut self, cx: &mut Context) { - if self.direct_pump_started { + fn attach_undo_manager(&mut self) { + if self.undo_manager.is_some() { return; } - self.direct_pump_started = true; - let requests = self.direct_rx.clone(); - cx.spawn(async move |session, cx| { - while let Ok(request) = requests.recv().await { - if session.update(cx, |session, cx| session.handle_direct_request(request, cx)).is_err() { - break; - } - } - }) - .detach(); - } - - fn handle_direct_request(&mut self, request: DirectServeRequest, cx: &mut Context) { - match request { - DirectServeRequest::Snapshot { reply } => { - let _ = reply.try_send(self.snapshot_bytes()); - }, - DirectServeRequest::Updates { have_vv, reply } => { - let _ = reply.try_send(self.update_bytes(&have_vv)); - }, - DirectServeRequest::Asset { asset, reply } => { - let _ = reply.try_send(self.asset_bytes(asset, cx)); - }, - } - } - - fn snapshot_bytes(&self) -> Result> { - self - .doc - .as_ref() - .context("collaboration session is not attached")? - .export(ExportMode::Snapshot) - .context("exporting collaboration snapshot failed") - } - - fn update_bytes(&self, have_vv: &[u8]) -> Result> { - let vv = VersionVector::decode(have_vv).context("decoding collaboration version vector failed")?; - self - .doc - .as_ref() - .context("collaboration session is not attached")? - .export(ExportMode::updates(&vv)) - .context("exporting collaboration updates failed") - } - - fn asset_bytes(&self, asset: u128, cx: &mut Context) -> Result { - let editor = self.editor.as_ref().context("collaboration session has no editor")?; - let bytes = editor - .read(cx) - .document() - .assets - .assets - .get(&AssetId(asset)) - .map(|record| record.bytes.as_ref().clone()) - .ok_or_else(|| anyhow!("collaboration asset {asset} is not available"))?; - Ok(AssetBytes { bytes }) - } - - fn diff_subscription( - &self, - doc: LoroDoc, - document: Arc, - binding: Arc>, - patches: Arc>>, - ) -> LoroSubscription { - let subscribed_doc = doc.clone(); - let callback: Subscriber = Arc::new(move |event| { - let produced = { - let mut binding = match binding.lock() { - Ok(binding) => binding, - Err(error) => { - eprintln!("flowstate collab binding lock failed: {error}"); - return; - }, - }; - let result = { - let mut applier = RemoteApplier { - doc: &doc, - binding: &mut binding, - }; - applier.apply_event(&document, &event) - }; - drop(binding); - result - }; - match produced { - Ok(mut produced) => { - if let Ok(mut patches) = patches.lock() { - patches.append(&mut produced); - } - }, - Err(error) => eprintln!("flowstate collab remote apply failed: {error:#}"), - } - }); - subscribed_doc.subscribe_root(callback) - } - - fn apply_or_queue_patches(&mut self, mut patches: Vec, cx: &mut Context) { - if patches.is_empty() { - return; - } - self.pending_remote_patches.append(&mut patches); - self.flush_pending_remote_patches(cx); - } - - fn flush_pending_remote_patches(&mut self, cx: &mut Context) -> bool { - let Some(editor) = self.editor.clone() else { - return false; - }; - if self.pending_remote_patches.is_empty() || editor.read(cx).collab_apply_deferred() { - return false; - } - let patches = std::mem::take(&mut self.pending_remote_patches); - editor.update(cx, |editor, cx| editor.apply_collab_patches(&patches, cx)); - self.refresh_external_carets(cx); - true - } - - fn handle_digest( - &mut self, - from: flowstate_collab::ids::PeerId, - digest_session: SessionId, - vv: &[u8], - cx: &mut Context, - ) -> Result<()> { - if digest_session != self.session { - bail!("received a collaboration digest for a different session"); - } let Some(doc) = &self.doc else { - return Ok(()); - }; - let sender_vv = VersionVector::decode(vv).context("decoding collaboration digest failed")?; - let our_vv = doc.oplog_vv(); - match sender_vv.partial_cmp(&our_vv) { - Some(Ordering::Greater) | None => self.pull_updates(from, our_vv.encode(), cx), - Some(Ordering::Equal | Ordering::Less) => {}, - } - Ok(()) - } - - fn pull_updates(&mut self, from: flowstate_collab::ids::PeerId, our_vv: Vec, cx: &mut Context) { - if self.update_pull_in_flight { return; - } - self.update_pull_in_flight = true; - let (reply_tx, reply_rx) = async_channel::bounded(1); - let send_result = self.net_tx.try_send(NetCommand::PullUpdates { - session: self.session, - from, - our_vv, - reply: reply_tx, - }); - if send_result.is_err() { - self.update_pull_in_flight = false; - return; - } - cx.spawn(async move |session, cx| { - let result = reply_rx.recv().await; - let _ = session.update(cx, |session, cx| { - session.update_pull_in_flight = false; - if let Ok(Ok(bytes)) = result - && let Err(error) = session.import_update_bytes(&bytes, cx) - { - session.detach(DetachReason::Fatal(format!("pulling collaboration updates failed: {error:#}")), cx); - } - }); - }) - .detach(); + }; + let mut undo_manager = UndoManager::new(doc); + undo_manager.set_merge_interval(500); + undo_manager.set_max_undo_steps(300); + self.undo_manager = Some(undo_manager); } - fn pull_blob(&mut self, from: flowstate_collab::ids::PeerId, blob: flowstate_collab::BlobId, cx: &mut Context) { - let (reply_tx, reply_rx) = async_channel::bounded(1); - if self - .net_tx - .try_send(NetCommand::PullBlob { - session: self.session, - from, - blob, - reply: reply_tx, - }) - .is_err() - { + fn attach_undo_request_pump(&mut self, cx: &mut Context) { + if self.undo_pump_started { return; } + self.undo_pump_started = true; + let requests = self.undo_rx.clone(); cx.spawn(async move |session, cx| { - let result = reply_rx.recv().await; - let _ = session.update(cx, |session, cx| { - if let Ok(Ok(bytes)) = result - && let Err(error) = session.import_update_bytes(&bytes, cx) + while let Ok(redirect) = requests.recv().await { + if session + .update(cx, |session, cx| { + if let Err(error) = session.apply_loro_undo_redirect(redirect, cx) { + session.detach(DetachReason::Fatal(format!("collaboration undo failed: {error:#}")), cx); + } + }) + .is_err() { - session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + break; } - }); + } }) .detach(); } - fn publish_digest(&self) { - if let Some(doc) = &self.doc { - let _ = self.net_tx.try_send(NetCommand::Publish { - session: self.session, - payload: PublishPayload::Digest { vv: doc.oplog_vv().encode() }, - }); - } - } - - fn apply_presence(&mut self, bytes: &[u8], cx: &mut Context) { - if let Some(presence) = &self.presence { - if let Err(error) = presence.apply(bytes) { - eprintln!("flowstate collab presence update failed: {error:#}"); - } - self.refresh_peer_count(); - self.refresh_external_carets(cx); - cx.notify(); - } - } - - fn refresh_own_presence(&mut self, cx: &mut Context) { - let Some(presence) = &self.presence else { - return; - }; - let state = PresenceState { - name: default_presence_name(), - selection: self.own_presence_selection(cx), - }; - if let Err(error) = presence.set_self(&state) { - eprintln!("flowstate collab presence encode failed: {error:#}"); - } - self.refresh_peer_count(); - } - - fn own_presence_selection(&self, cx: &mut Context) -> Option { - let editor = self.editor.as_ref()?.read(cx); - let binding = self.binding.as_ref()?; - presence_view::selection_for_editor(editor, binding) - } - - fn refresh_external_carets(&mut self, cx: &mut Context) { - let Some(presence) = &self.presence else { - return; - }; - let Some(doc) = &self.doc else { - return; - }; - let Some(binding) = &self.binding else { - return; - }; - let Some(editor) = self.editor.clone() else { - return; - }; - let carets = presence_view::external_carets(doc, binding, presence); - editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); - } - - fn publish_presence_snapshot(&self) { - if let Some(presence) = &self.presence { - self.publish_presence_bytes(presence.encode_all()); - } - } - - fn publish_presence_bytes(&self, bytes: Vec) { - if bytes.is_empty() { - return; - } - let _ = self.net_tx.try_send(NetCommand::Publish { - session: self.session, - payload: PublishPayload::Presence(bytes), - }); - } - - fn refresh_peer_count(&mut self) { - let peers_present = self.peers_present(); - if let SessionPhase::Attached(attachment) = &mut self.phase { - attachment.peers_present = peers_present; - } - } - - fn peers_present(&self) -> usize { - self - .presence - .as_ref() - .map_or(0, |presence| presence.roster().len().saturating_sub(1)) - } -} - -fn take_mutex_value(value: Arc>, label: &str) -> Result { - match Arc::try_unwrap(value) { - Ok(mutex) => mutex - .into_inner() - .map_err(|error| anyhow!("collaboration {label} lock was poisoned: {error}")), - Err(_) => Err(anyhow!("collaboration {label} is still referenced")), - } -} - -fn default_presence_name() -> String { - std::env::var("FLOWSTATE_COLLAB_NAME") - .or_else(|_| std::env::var("USER")) - .or_else(|_| std::env::var("USERNAME")) - .unwrap_or_else(|_| "Flowstate user".to_string()) } diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs new file mode 100644 index 00000000..09fcc0bd --- /dev/null +++ b/crates/flowstate/src/collab/session_io.rs @@ -0,0 +1,313 @@ +use std::sync::{Arc, Mutex}; + +use anyhow::{Context as _, Result, anyhow}; +use flowstate_collab::{ + SessionId, + binding::DocBinding, + net::{ + NetCommand, PublishPayload, + anti_entropy::{GapAction, VersionVectorRelation}, + direct::DirectServeRequest, + }, + proto_direct::AssetBytes, + remote_apply::RemoteApplier, +}; +use gpui::Context; +use loro::{ExportMode, LoroDoc, Subscription as LoroSubscription, VersionVector, event::Subscriber}; + +use crate::rich_text_element::{AssetId, CollabPatch, Document, UndoRedirect}; + +use super::{CollabSession, DetachReason}; + +impl CollabSession { + pub fn import_update_bytes(&mut self, bytes: &[u8], cx: &mut Context) -> Result<()> { + if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() { + self.pending_remote_updates.push(bytes.to_vec()); + return Ok(()); + } + + let editor = self.editor.clone().context("collaboration session has no editor")?; + let document = Arc::new(editor.read(cx).document().clone()); + let doc = self.doc.clone().context("collaboration session has no Loro document")?; + let binding = self.binding.take().context("collaboration session has no document binding")?; + let binding = Arc::new(Mutex::new(binding)); + let patches = Arc::new(Mutex::new(Vec::::new())); + let sub = self.diff_subscription(doc.clone(), document, binding.clone(), patches.clone()); + + let import_result = doc.import_with(bytes, "remote"); + drop(sub); + self.binding = Some(take_mutex_value(binding, "document binding")?); + let patches = take_mutex_value(patches, "remote patches")?; + import_result.context("importing collaboration update failed")?; + self.apply_or_queue_patches(patches, cx); + Ok(()) + } + + pub(super) fn attach_direct_request_pump(&mut self, cx: &mut Context) { + if self.direct_pump_started { + return; + } + self.direct_pump_started = true; + let requests = self.direct_rx.clone(); + cx.spawn(async move |session, cx| { + while let Ok(request) = requests.recv().await { + if session.update(cx, |session, cx| session.handle_direct_request(request, cx)).is_err() { + break; + } + } + }) + .detach(); + } + + pub(super) fn handle_digest( + &mut self, + from: flowstate_collab::ids::PeerId, + digest_session: SessionId, + vv: &[u8], + cx: &mut Context, + ) -> Result<()> { + let Some(doc) = &self.doc else { + return Ok(()); + }; + let sender_vv = VersionVector::decode(vv).context("decoding collaboration digest failed")?; + let our_vv = doc.oplog_vv(); + let relation = match sender_vv.partial_cmp(&our_vv) { + Some(std::cmp::Ordering::Equal) => VersionVectorRelation::Equal, + Some(std::cmp::Ordering::Greater) => VersionVectorRelation::SenderHasMissingOps, + Some(std::cmp::Ordering::Less) => VersionVectorRelation::WeHaveMissingOps, + None => VersionVectorRelation::Concurrent, + }; + let action = self + .anti_entropy + .consider_digest(from, digest_session, relation, our_vv.encode()); + self.handle_gap_action(action, cx); + Ok(()) + } + + pub(super) fn pull_blob(&mut self, from: flowstate_collab::ids::PeerId, blob: flowstate_collab::BlobId, cx: &mut Context) { + let (reply_tx, reply_rx) = async_channel::bounded(1); + if self + .net_tx + .try_send(NetCommand::PullBlob { + session: self.session, + candidates: self.pull_candidates(Some(from)), + blob, + reply: reply_tx, + }) + .is_err() + { + return; + } + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + if let Ok(Ok(bytes)) = result + && let Err(error) = session.import_update_bytes(&bytes, cx) + { + session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + } + }); + }) + .detach(); + } + + pub(super) fn publish_digest(&self) { + if let Some(doc) = &self.doc { + let _ = self.net_tx.try_send(NetCommand::Publish { + session: self.session, + payload: PublishPayload::Digest { vv: doc.oplog_vv().encode() }, + }); + } + } + + pub(super) fn flush_pending_remote_patches(&mut self, cx: &mut Context) -> bool { + let Some(editor) = self.editor.clone() else { + return false; + }; + if self.pending_remote_patches.is_empty() || editor.read(cx).collab_apply_deferred() { + return false; + } + let patches = std::mem::take(&mut self.pending_remote_patches); + editor.update(cx, |editor, cx| { + editor.clear_collab_history(); + editor.apply_collab_patches(&patches, cx); + }); + self.last_document_activity = std::time::Instant::now(); + self.refresh_external_carets(cx); + true + } + + pub(super) fn apply_loro_undo_redirect(&mut self, redirect: UndoRedirect, cx: &mut Context) -> Result<()> { + if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() || self.undo_manager.is_none() { + return Ok(()); + } + + let editor = self.editor.clone().context("collaboration session has no editor")?; + let document = Arc::new(editor.read(cx).document().clone()); + let doc = self.doc.clone().context("collaboration session has no Loro document")?; + let binding = self.binding.take().context("collaboration session has no document binding")?; + let binding = Arc::new(Mutex::new(binding)); + let patches = Arc::new(Mutex::new(Vec::::new())); + let sub = self.diff_subscription(doc, document, binding.clone(), patches.clone()); + + let undo_result = match redirect { + UndoRedirect::Undo => self + .undo_manager + .as_mut() + .context("collaboration session has no undo manager")? + .undo(), + UndoRedirect::Redo => self + .undo_manager + .as_mut() + .context("collaboration session has no undo manager")? + .redo(), + }; + drop(sub); + self.binding = Some(take_mutex_value(binding, "document binding")?); + let patches = take_mutex_value(patches, "undo patches")?; + let applied = undo_result.context("applying collaboration undo operation failed")?; + if applied { + self.apply_or_queue_patches(patches, cx); + self.publish_digest(); + } + Ok(()) + } + + pub(super) fn handle_gap_action(&mut self, action: GapAction, cx: &mut Context) { + match action { + GapAction::None => {}, + GapAction::Pull { from, our_vv } => self.start_update_pull(from, our_vv, cx), + GapAction::LineageMismatch { from, expected, got } => { + eprintln!("flowstate collab ignored mismatched digest from {from}: expected session {expected}, got {got}"); + }, + } + } + + fn handle_direct_request(&mut self, request: DirectServeRequest, cx: &mut Context) { + match request { + DirectServeRequest::Snapshot { reply } => { + let _ = reply.try_send(self.snapshot_bytes()); + }, + DirectServeRequest::Updates { have_vv, reply } => { + let _ = reply.try_send(self.update_bytes(&have_vv)); + }, + DirectServeRequest::Asset { asset, reply } => { + let _ = reply.try_send(self.asset_bytes(asset, cx)); + }, + } + } + + fn snapshot_bytes(&self) -> Result> { + self + .doc + .as_ref() + .context("collaboration session is not attached")? + .export(ExportMode::Snapshot) + .context("exporting collaboration snapshot failed") + } + + fn update_bytes(&self, have_vv: &[u8]) -> Result> { + let vv = VersionVector::decode(have_vv).context("decoding collaboration version vector failed")?; + self + .doc + .as_ref() + .context("collaboration session is not attached")? + .export(ExportMode::updates(&vv)) + .context("exporting collaboration updates failed") + } + + fn asset_bytes(&self, asset: u128, cx: &mut Context) -> Result { + let editor = self.editor.as_ref().context("collaboration session has no editor")?; + let bytes = editor + .read(cx) + .document() + .assets + .assets + .get(&AssetId(asset)) + .map(|record| record.bytes.as_ref().clone()) + .ok_or_else(|| anyhow!("collaboration asset {asset} is not available"))?; + Ok(AssetBytes { bytes }) + } + + fn diff_subscription( + &self, + doc: LoroDoc, + document: Arc, + binding: Arc>, + patches: Arc>>, + ) -> LoroSubscription { + let subscribed_doc = doc.clone(); + let callback: Subscriber = Arc::new(move |event| { + let produced = { + let mut binding = match binding.lock() { + Ok(binding) => binding, + Err(error) => { + eprintln!("flowstate collab binding lock failed: {error}"); + return; + }, + }; + let result = { + let mut applier = RemoteApplier { + doc: &doc, + binding: &mut binding, + }; + applier.apply_event(&document, &event) + }; + drop(binding); + result + }; + match produced { + Ok(mut produced) => { + if let Ok(mut patches) = patches.lock() { + patches.append(&mut produced); + } + }, + Err(error) => eprintln!("flowstate collab remote apply failed: {error:#}"), + } + }); + subscribed_doc.subscribe_root(callback) + } + + pub(super) fn apply_or_queue_patches(&mut self, mut patches: Vec, cx: &mut Context) { + if patches.is_empty() { + return; + } + self.pending_remote_patches.append(&mut patches); + self.flush_pending_remote_patches(cx); + } + + fn start_update_pull(&mut self, from: flowstate_collab::ids::PeerId, our_vv: Vec, cx: &mut Context) { + let (reply_tx, reply_rx) = async_channel::bounded(1); + let send_result = self.net_tx.try_send(NetCommand::PullUpdates { + session: self.session, + candidates: self.pull_candidates(Some(from)), + our_vv, + reply: reply_tx, + }); + if send_result.is_err() { + self.anti_entropy.finish_pull(); + return; + } + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + session.anti_entropy.finish_pull(); + if let Ok(Ok(bytes)) = result + && let Err(error) = session.import_update_bytes(&bytes, cx) + { + session.detach(DetachReason::Fatal(format!("pulling collaboration updates failed: {error:#}")), cx); + } + }); + }) + .detach(); + } +} + +fn take_mutex_value(value: Arc>, label: &str) -> Result { + match Arc::try_unwrap(value) { + Ok(mutex) => mutex + .into_inner() + .map_err(|error| anyhow!("collaboration {label} lock was poisoned: {error}")), + Err(_) => Err(anyhow!("collaboration {label} is still referenced")), + } +} diff --git a/crates/flowstate/src/collab/session_presence.rs b/crates/flowstate/src/collab/session_presence.rs new file mode 100644 index 00000000..e78010e3 --- /dev/null +++ b/crates/flowstate/src/collab/session_presence.rs @@ -0,0 +1,92 @@ +use flowstate_collab::presence::{PresenceSelection, PresenceState}; +use gpui::Context; + +use super::{CollabSession, presence_view}; + +impl CollabSession { + pub(super) fn apply_presence(&mut self, bytes: &[u8], cx: &mut Context) { + if let Some(presence) = &self.presence { + if let Err(error) = presence.apply(bytes) { + eprintln!("flowstate collab presence update failed: {error:#}"); + } + self.refresh_peer_count(); + self.evaluate_connectivity(cx); + self.refresh_external_carets(cx); + cx.notify(); + } + } + + pub(super) fn refresh_own_presence(&mut self, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let state = PresenceState { + name: default_presence_name(), + selection: self.own_presence_selection(cx), + }; + if let Err(error) = presence.set_self(&state) { + eprintln!("flowstate collab presence encode failed: {error:#}"); + } + self.refresh_peer_count(); + } + + pub(super) fn refresh_external_carets(&mut self, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let Some(doc) = &self.doc else { + return; + }; + let Some(binding) = &self.binding else { + return; + }; + let Some(editor) = self.editor.clone() else { + return; + }; + let carets = presence_view::external_carets(doc, binding, presence); + editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); + } + + pub(super) fn publish_presence_snapshot(&self) { + if let Some(presence) = &self.presence { + self.publish_presence_bytes(presence.encode_all()); + } + } + + pub(super) fn publish_presence_bytes(&self, bytes: Vec) { + if bytes.is_empty() { + return; + } + let _ = self.net_tx.try_send(flowstate_collab::net::NetCommand::Publish { + session: self.session, + payload: flowstate_collab::net::PublishPayload::Presence(bytes), + }); + } + + pub(super) fn refresh_peer_count(&mut self) { + let peers_present = self.peers_present(); + if let super::SessionPhase::Attached(attachment) = &mut self.phase { + attachment.peers_present = peers_present; + } + } + + pub(super) fn peers_present(&self) -> usize { + self + .presence + .as_ref() + .map_or(0, |presence| presence.roster().len().saturating_sub(1)) + } + + fn own_presence_selection(&self, cx: &mut Context) -> Option { + let editor = self.editor.as_ref()?.read(cx); + let binding = self.binding.as_ref()?; + presence_view::selection_for_editor(editor, binding) + } +} + +fn default_presence_name() -> String { + std::env::var("FLOWSTATE_COLLAB_NAME") + .or_else(|_| std::env::var("USER")) + .or_else(|_| std::env::var("USERNAME")) + .unwrap_or_else(|_| "Flowstate user".to_string()) +} diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs new file mode 100644 index 00000000..7c88a4b1 --- /dev/null +++ b/crates/flowstate/src/collab/session_timers.rs @@ -0,0 +1,272 @@ +use std::time::{Duration, Instant}; + +use anyhow::{Context as _, Result}; +use flowstate_collab::{ + binding::DocBinding, + net::NetCommand, + presence::PRESENCE_KEEPALIVE_SECS, + projection, self_check, +}; +use gpui::{Context, Timer}; + +use crate::{app_settings::load_document_theme, rich_text_element::Document}; + +use super::{Attachment, CollabSession, Connectivity, SessionPhase}; + +const ZERO_NEIGHBOR_OFFLINE_GRACE: Duration = Duration::from_secs(5); +const QUIET_DIGEST_ROUNDS_OFFLINE: u8 = 2; +const SELF_CHECK_INTERVAL: Duration = Duration::from_secs(30); +const SELF_CHECK_IDLE: Duration = Duration::from_secs(2); +const RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30); + +impl CollabSession { + pub(super) fn attach_timers(&mut self, cx: &mut Context) { + if self.timers_started { + return; + } + self.timers_started = true; + self.spawn_digest_timer(cx); + self.spawn_presence_timer(cx); + self.spawn_self_check_timer(cx); + } + + pub(super) fn note_inbound_traffic(&mut self, cx: &mut Context) { + self.inbound_since_last_digest = true; + self.quiet_digest_rounds = 0; + if self.mark_online(cx) { + self.anti_entropy.on_reconnect(); + self.publish_digest(); + } + } + + pub(super) fn mark_online(&mut self, cx: &mut Context) -> bool { + let mut changed = false; + if let SessionPhase::Attached(attachment) = &mut self.phase + && matches!(attachment.connectivity, Connectivity::Offline { .. }) + { + attachment.connectivity = Connectivity::Online; + self.next_recovery_at = None; + changed = true; + } + if changed { + cx.notify(); + } + changed + } + + pub(super) fn evaluate_connectivity(&mut self, cx: &mut Context) { + if !matches!(self.phase, SessionPhase::Attached(_)) { + return; + } + + self.refresh_peer_count(); + if self.peers_present() == 0 || !self.neighbors.is_empty() { + self.mark_online(cx); + return; + } + + let now = Instant::now(); + let zero_since = self.zero_neighbors_since.get_or_insert(now); + let quiet_enough = self.quiet_digest_rounds >= QUIET_DIGEST_ROUNDS_OFFLINE || !self.endpoint_online; + if quiet_enough && now.saturating_duration_since(*zero_since) >= ZERO_NEIGHBOR_OFFLINE_GRACE { + self.mark_offline(now, cx); + } + } + + fn spawn_digest_timer(&mut self, cx: &mut Context) { + cx.spawn(async move |session, cx| { + loop { + let delay = match session.update(cx, |session, _| session.next_digest_delay()) { + Ok(Some(delay)) => delay, + Ok(None) | Err(_) => break, + }; + Timer::after(delay).await; + match session.update(cx, |session, cx| session.run_digest_tick(cx)) { + Ok(true) => {}, + Ok(false) | Err(_) => break, + } + } + }) + .detach(); + } + + fn spawn_presence_timer(&mut self, cx: &mut Context) { + cx.spawn(async move |session, cx| { + loop { + Timer::after(Duration::from_secs(PRESENCE_KEEPALIVE_SECS)).await; + match session.update(cx, |session, cx| session.run_presence_tick(cx)) { + Ok(true) => {}, + Ok(false) | Err(_) => break, + } + } + }) + .detach(); + } + + fn spawn_self_check_timer(&mut self, cx: &mut Context) { + cx.spawn(async move |session, cx| { + loop { + Timer::after(SELF_CHECK_INTERVAL).await; + match session.update(cx, |session, cx| session.run_self_check_tick(cx)) { + Ok(true) => {}, + Ok(false) | Err(_) => break, + } + } + }) + .detach(); + } + + fn next_digest_delay(&self) -> Option { + self.timer_live().then(|| self.anti_entropy.duration_until_digest(Instant::now())) + } + + fn run_digest_tick(&mut self, cx: &mut Context) -> bool { + if !self.timer_live() { + return false; + } + + let now = Instant::now(); + if self.anti_entropy.digest_due(now) { + if matches!(self.phase, SessionPhase::Attached(_)) { + self.publish_digest(); + self.finish_digest_round(now, cx); + } + self.anti_entropy.mark_digest_sent(now); + } + true + } + + fn run_presence_tick(&mut self, cx: &mut Context) -> bool { + if !self.timer_live() { + return false; + } + if self.presence.is_some() { + if let Some(presence) = &self.presence { + presence.remove_outdated(); + } + self.refresh_own_presence(cx); + self.refresh_peer_count(); + self.evaluate_connectivity(cx); + self.refresh_external_carets(cx); + cx.notify(); + } + true + } + + fn run_self_check_tick(&mut self, cx: &mut Context) -> bool { + if !self.timer_live() { + return false; + } + if !matches!(self.phase, SessionPhase::Attached(_)) || self.last_document_activity.elapsed() < SELF_CHECK_IDLE { + return true; + } + if let Err(error) = self.run_self_check(cx) { + eprintln!("flowstate collab self-check failed: {error:#}"); + } + true + } + + fn finish_digest_round(&mut self, now: Instant, cx: &mut Context) { + if self.inbound_since_last_digest { + self.quiet_digest_rounds = 0; + } else { + self.quiet_digest_rounds = self.quiet_digest_rounds.saturating_add(1); + } + self.inbound_since_last_digest = false; + self.evaluate_connectivity(cx); + self.run_recovery_if_due(now, cx); + } + + fn run_recovery_if_due(&mut self, now: Instant, cx: &mut Context) { + let Some(next_recovery_at) = self.next_recovery_at else { + return; + }; + if now < next_recovery_at { + return; + } + + let mut attempted = false; + if let SessionPhase::Attached(Attachment { + connectivity: Connectivity::Offline { retries, .. }, + .. + }) = &mut self.phase + { + let delay = recovery_delay(*retries); + *retries = retries.saturating_add(1); + self.next_recovery_at = Some(now + delay); + attempted = true; + } + + if attempted { + if self.bootstrap_addrs.is_empty() { + let _ = self.net_tx.try_send(NetCommand::EnsureUp); + } else { + let _ = self.net_tx.try_send(NetCommand::JoinSession { + session: self.session, + bootstrap: self.bootstrap_addrs.clone(), + }); + } + self.publish_digest(); + cx.notify(); + } + } + + fn run_self_check(&mut self, cx: &mut Context) -> Result<()> { + let Some(doc) = &self.doc else { + return Ok(()); + }; + let Some(editor) = self.editor.clone() else { + return Ok(()); + }; + + let live_document = editor.read(cx).document().clone(); + let live_hash = self_check::projection_hash(&live_document); + let mut projected = projection::document_from_loro(doc, load_document_theme())?; + projected.assets = live_document.assets.clone(); + let projected_hash = self_check::projection_hash(&projected); + if live_hash == projected_hash { + return Ok(()); + } + + eprintln!( + "flowstate collab projection drift detected: live_hash={live_hash:016x} projected_hash={projected_hash:016x} vv_bytes={}", + doc.oplog_vv().encode().len() + ); + self.rebuild_from_projection(projected, cx) + } + + fn rebuild_from_projection(&mut self, projected: Document, cx: &mut Context) -> Result<()> { + let Some(doc) = &self.doc else { + return Ok(()); + }; + let Some(editor) = self.editor.clone() else { + return Ok(()); + }; + editor.update(cx, |editor, cx| editor.replace_document_from_collaboration(projected, cx)); + let document = editor.read(cx).document().clone(); + self.binding = Some(DocBinding::build(doc, &document).context("rebuilding collaboration binding after self-check failed")?); + self.last_document_activity = Instant::now(); + self.refresh_external_carets(cx); + Ok(()) + } + + fn mark_offline(&mut self, now: Instant, cx: &mut Context) { + if let SessionPhase::Attached(attachment) = &mut self.phase + && matches!(attachment.connectivity, Connectivity::Online) + { + attachment.connectivity = Connectivity::Offline { since: now, retries: 0 }; + self.next_recovery_at = Some(now); + cx.notify(); + } + } + + fn timer_live(&self) -> bool { + !matches!(self.phase, SessionPhase::Detached(_)) + } +} + +fn recovery_delay(retries: u32) -> Duration { + let shift = retries.min(5); + let secs = 1_u64.checked_shl(shift).unwrap_or(RECOVERY_MAX_BACKOFF.as_secs()); + Duration::from_secs(secs).min(RECOVERY_MAX_BACKOFF) +} diff --git a/crates/flowstate/src/collab/share_dialog.rs b/crates/flowstate/src/collab/share_dialog.rs new file mode 100644 index 00000000..f0008b09 --- /dev/null +++ b/crates/flowstate/src/collab/share_dialog.rs @@ -0,0 +1,530 @@ +use flowstate_collab::ticket::SessionTicket; +use gpui::{ + AnyElement, AnyWindowHandle, App, ClipboardItem, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, + KeyDownEvent, MouseButton, ParentElement, Render, SharedString, Subscription, WeakEntity, Window, div, prelude::*, px, relative, rgb, +}; +use gpui_component::{ + ActiveTheme as _, Disableable, IconName, Sizable, + button::{Button, ButtonVariants as _}, + h_flex, + input::{Input, InputEvent, InputState}, + scroll::ScrollableElement, + v_flex, +}; +use uuid::Uuid; + +use crate::workspace::Workspace; + +use super::{Connectivity, SessionPhase, status}; + +#[derive(Clone, Copy, PartialEq, Eq)] +pub enum CollabDialogMode { + Share, + Join, +} + +pub struct CollabShareDialog { + workspace: WeakEntity, + panel_id: Option, + mode: CollabDialogMode, + join_input: Entity, + ticket_text: Option, + ticket_loading: bool, + ticket_error: Option, + copy_notice: Option, + join_error: Option, + _input_subscription: Subscription, +} + +#[hotpath::measure_all] +impl CollabShareDialog { + pub fn new( + workspace: WeakEntity, + panel_id: Option, + mode: CollabDialogMode, + window: &mut Window, + cx: &mut Context, + ) -> Self { + let join_input = cx.new(|cx| InputState::new(window, cx).placeholder("Paste a Flowstate collaboration invite")); + let _input_subscription = cx.subscribe(&join_input, |dialog, _, event: &InputEvent, cx| { + if let InputEvent::Change = event { + dialog.validate_join_ticket(cx); + } + }); + + let mut dialog = Self { + workspace, + panel_id, + mode, + join_input, + ticket_text: None, + ticket_loading: false, + ticket_error: None, + copy_notice: None, + join_error: None, + _input_subscription, + }; + if mode == CollabDialogMode::Share { + dialog.refresh_ticket(None, cx); + } + dialog + } + + pub fn focus(&self, window: &mut Window, cx: &mut Context) { + if self.mode == CollabDialogMode::Join { + self.join_input.focus_handle(cx).focus(window); + } + } + + fn close(&mut self, cx: &mut Context) { + let _ = self.workspace.update(cx, |workspace, cx| workspace.close_collaboration_dialog(cx)); + } + + fn set_mode(&mut self, mode: CollabDialogMode, cx: &mut Context) { + if self.mode == mode { + return; + } + self.mode = mode; + self.join_error = None; + self.copy_notice = None; + if mode == CollabDialogMode::Share { + self.refresh_ticket(None, cx); + } + cx.notify(); + } + + fn start_session(&mut self, cx: &mut Context) { + let Some(panel_id) = self.panel_id else { + self.ticket_error = Some("Open a document before starting a session.".into()); + cx.notify(); + return; + }; + + match self + .workspace + .update(cx, |workspace, cx| workspace.start_collaboration_on_document(panel_id, cx)) + { + Ok(Some(_)) => self.refresh_ticket(None, cx), + Ok(None) => { + self.ticket_error = Some("The active document could not be shared.".into()); + cx.notify(); + }, + Err(error) => { + self.ticket_error = Some(format!("The workspace is no longer available: {error}").into()); + cx.notify(); + }, + } + } + + fn refresh_ticket(&mut self, copy_to_clipboard: Option, cx: &mut Context) { + let Some(panel_id) = self.panel_id else { + self.ticket_text = None; + self.ticket_loading = false; + return; + }; + let Some(ticket_rx) = crate::collab::request_ticket_for_panel(panel_id, cx) else { + self.ticket_text = None; + self.ticket_loading = false; + self.ticket_error = None; + return; + }; + + self.ticket_loading = true; + self.ticket_error = None; + self.copy_notice = None; + cx.notify(); + + cx.spawn(async move |dialog, cx| { + let result = ticket_rx.recv().await; + let encoded = match result { + Ok(Ok(ticket)) => Ok(ticket.encode_text()), + Ok(Err(error)) => Err(format!("Creating collaboration invite failed: {error:#}")), + Err(error) => Err(format!("Creating collaboration invite failed: {error}")), + }; + let should_copy = copy_to_clipboard.is_some(); + + if let Ok(text) = &encoded + && let Some(window_handle) = copy_to_clipboard + { + let text = text.clone(); + let _ = window_handle.update(cx, |_, _, cx| cx.write_to_clipboard(ClipboardItem::new_string(text))); + } + + let _ = dialog.update(cx, |dialog, cx| { + dialog.ticket_loading = false; + match encoded { + Ok(text) => { + dialog.ticket_text = Some(text.into()); + dialog.ticket_error = None; + if should_copy { + dialog.copy_notice = Some("Fresh invite copied to clipboard.".into()); + } + }, + Err(error) => { + dialog.ticket_error = Some(error.into()); + }, + } + cx.notify(); + }); + }) + .detach(); + } + + fn copy_invite(&mut self, window: &mut Window, cx: &mut Context) { + self.refresh_ticket(Some(window.window_handle()), cx); + } + + fn validate_join_ticket(&mut self, cx: &mut Context) { + self.join_error = match self.parse_join_ticket(cx) { + Ok(_) => None, + Err(error) => Some(error.into()), + }; + cx.notify(); + } + + fn parse_join_ticket(&self, cx: &App) -> Result, String> { + let text = self.join_input.read(cx).value().trim().to_string(); + if text.is_empty() { + return Ok(None); + } + SessionTicket::decode_text(&text) + .map(Some) + .map_err(|error| format!("Invalid invite: {error}")) + } + + fn join_session(&mut self, window: &mut Window, cx: &mut Context) { + let ticket = match self.parse_join_ticket(cx) { + Ok(Some(ticket)) => ticket, + Ok(None) => { + self.join_error = Some("Paste an invite before joining.".into()); + cx.notify(); + return; + }, + Err(error) => { + self.join_error = Some(error.into()); + cx.notify(); + return; + }, + }; + + match self + .workspace + .update(cx, |workspace, cx| workspace.join_collaboration_session(ticket, window, cx)) + { + Ok(Some(_)) => self.close(cx), + Ok(None) => { + self.join_error = Some("Joining collaboration session failed.".into()); + cx.notify(); + }, + Err(error) => { + self.join_error = Some(format!("The workspace is no longer available: {error}").into()); + cx.notify(); + }, + } + } + + fn confirm_leave(&mut self, window: &mut Window, cx: &mut Context) { + let Some(panel_id) = self.panel_id else { + return; + }; + let _ = self + .workspace + .update(cx, |workspace, cx| workspace.confirm_leave_collaboration_on_panel(panel_id, window, cx)); + } + + fn on_key_down(&mut self, event: &KeyDownEvent, window: &mut Window, cx: &mut Context) { + match event.keystroke.key.as_str() { + "escape" => { + self.close(cx); + cx.stop_propagation(); + }, + "enter" if self.mode == CollabDialogMode::Join => { + self.join_session(window, cx); + cx.stop_propagation(); + }, + _ => {}, + } + } + + fn render_share_pane(&self, phase: Option<&SessionPhase>, cx: &mut Context) -> AnyElement { + if self.panel_id.is_none() { + return v_flex() + .gap_3() + .child(section_title("Share this document", cx)) + .child(helper_text("Open a rich-text document before starting a collaboration session.", cx)) + .into_any_element(); + } + + if phase.is_none_or(|phase| matches!(phase, SessionPhase::Detached(_))) { + return v_flex() + .gap_3() + .child(section_title("Share this document", cx)) + .child(helper_text( + "Start a session to create one invite ticket. Everyone who joins can edit, invite others, and leave independently.", + cx, + )) + .child( + Button::new("start-collaboration-session") + .label("Start session") + .primary() + .on_click(cx.listener(|dialog, _, _, cx| dialog.start_session(cx))), + ) + .when_some(self.ticket_error.clone(), |this, error| this.child(error_text(error, cx))) + .into_any_element(); + } + + let Some(phase) = phase else { + return v_flex().into_any_element(); + }; + let roster = self + .panel_id + .map_or_else(Vec::new, |panel_id| crate::collab::roster_for_panel(panel_id, cx)); + v_flex() + .gap_3() + .child(h_flex().items_center().justify_between().child(section_title("Invite people", cx)).child(status::status_pill(phase, cx))) + .child(helper_text(connectivity_text(phase).as_str(), cx)) + .child(roster_list(roster, cx)) + .child(ticket_box(self.ticket_text.clone(), self.ticket_loading, cx)) + .when_some(self.ticket_error.clone(), |this, error| this.child(error_text(error, cx))) + .when_some(self.copy_notice.clone(), |this, notice| this.child(success_text(notice, cx))) + .child( + h_flex() + .gap_2() + .child( + Button::new("copy-collaboration-invite") + .label(if self.ticket_loading { "Minting..." } else { "Copy invite" }) + .primary() + .disabled(self.ticket_loading) + .on_click(cx.listener(|dialog, _, window, cx| dialog.copy_invite(window, cx))), + ) + .child( + Button::new("refresh-collaboration-invite") + .label("Refresh invite") + .outline() + .disabled(self.ticket_loading) + .on_click(cx.listener(|dialog, _, _, cx| dialog.refresh_ticket(None, cx))), + ) + .child(div().flex_1()) + .child( + Button::new("leave-collaboration-session") + .label("Leave session") + .danger() + .on_click(cx.listener(|dialog, _, window, cx| dialog.confirm_leave(window, cx))), + ), + ) + .into_any_element() + } + + fn render_join_pane(&self, cx: &mut Context) -> AnyElement { + let parsed = self.parse_join_ticket(cx).ok().flatten(); + v_flex() + .gap_3() + .child(section_title("Join a session", cx)) + .child(helper_text("Paste an invite ticket. Joining opens the shared document as a new tab and never merges into your existing files.", cx)) + .child(Input::new(&self.join_input).w_full().cleanable(true)) + .when_some(parsed.as_ref().map(|ticket| ticket.title.clone()), |this, title| { + this.child(success_text(format!("Invite for: {title}").into(), cx)) + }) + .when_some(self.join_error.clone(), |this, error| this.child(error_text(error, cx))) + .child( + h_flex() + .justify_end() + .gap_2() + .child( + Button::new("join-collaboration-session") + .label("Join") + .primary() + .on_click(cx.listener(|dialog, _, window, cx| dialog.join_session(window, cx))), + ), + ) + .into_any_element() + } +} + +#[hotpath::measure_all] +impl Focusable for CollabShareDialog { + fn focus_handle(&self, cx: &App) -> FocusHandle { + self.join_input.focus_handle(cx) + } +} + +#[hotpath::measure_all] +impl Render for CollabShareDialog { + fn render(&mut self, _: &mut Window, cx: &mut Context) -> impl IntoElement { + let phase = self.panel_id.and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)); + + div() + .absolute() + .top_0() + .right_0() + .bottom_0() + .left_0() + .bg(cx.theme().background.opacity(0.72)) + .flex() + .items_center() + .justify_center() + .occlude() + .on_key_down(cx.listener(Self::on_key_down)) + .on_mouse_down(MouseButton::Left, cx.listener(|dialog, _, _, cx| { + dialog.close(cx); + cx.stop_propagation(); + })) + .on_scroll_wheel(|_, _, cx| cx.stop_propagation()) + .child( + v_flex() + .w(px(620.0)) + .max_w_full() + .max_h(px(620.0)) + .overflow_hidden() + .rounded_lg() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().popover) + .shadow_lg() + .on_mouse_down(MouseButton::Left, |_, _, cx| cx.stop_propagation()) + .child( + h_flex() + .h(px(44.0)) + .items_center() + .justify_between() + .px_4() + .border_b_1() + .border_color(cx.theme().border) + .child(div().font_weight(gpui::FontWeight::SEMIBOLD).child("Share / Collaborate")) + .child( + Button::new("close-collaboration-dialog") + .icon(IconName::Close) + .xsmall() + .ghost() + .tooltip("Close") + .on_click(cx.listener(|dialog, _, _, cx| dialog.close(cx))), + ), + ) + .child( + h_flex() + .gap_2() + .px_4() + .pt_4() + .child(tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { + dialog.set_mode(CollabDialogMode::Share, cx); + }))) + .child(tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { + dialog.set_mode(CollabDialogMode::Join, cx); + dialog.focus(window, cx); + }))), + ) + .child( + div() + .flex_1() + .overflow_y_scrollbar() + .p_4() + .child(match self.mode { + CollabDialogMode::Share => self.render_share_pane(phase.as_ref(), cx), + CollabDialogMode::Join => self.render_join_pane(cx), + }), + ), + ) + } +} + +fn tab_button(id: &'static str, label: &'static str, selected: bool) -> Button { + let button = Button::new(id).label(label).small(); + if selected { button.primary() } else { button.outline() } +} + +fn section_title(title: &'static str, cx: &App) -> impl IntoElement { + div() + .text_lg() + .font_weight(gpui::FontWeight::SEMIBOLD) + .text_color(cx.theme().foreground) + .child(title) +} + +fn helper_text(text: &str, cx: &App) -> impl IntoElement { + div() + .text_sm() + .line_height(relative(1.45)) + .text_color(cx.theme().muted_foreground) + .child(text.to_string()) +} + +fn ticket_box(ticket: Option, loading: bool, cx: &App) -> impl IntoElement { + div() + .w_full() + .min_h(px(84.0)) + .max_h(px(132.0)) + .overflow_y_scrollbar() + .rounded_md() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().background) + .p_3() + .text_xs() + .line_height(relative(1.35)) + .text_color(cx.theme().foreground) + .child(if loading { + SharedString::from("Minting a fresh invite...") + } else { + ticket.unwrap_or_else(|| "Start sharing to mint an invite.".into()) + }) +} + +fn error_text(text: SharedString, cx: &App) -> impl IntoElement { + div() + .text_sm() + .text_color(cx.theme().danger) + .child(text) +} + +fn success_text(text: SharedString, cx: &App) -> impl IntoElement { + div() + .text_sm() + .text_color(cx.theme().success) + .child(text) +} + +fn roster_list(entries: Vec, cx: &App) -> AnyElement { + let mut list = v_flex() + .gap_2() + .rounded_md() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().background) + .p_3() + .child(div().text_xs().text_color(cx.theme().muted_foreground).child("Participants")); + + if entries.is_empty() { + return list + .child(div().text_sm().text_color(cx.theme().muted_foreground).child("Presence is starting...")) + .into_any_element(); + } + + for entry in entries { + let label = if entry.is_self { + format!("{} (you)", entry.name) + } else { + entry.name + }; + list = list.child( + h_flex() + .items_center() + .gap_2() + .child(div().size(px(8.0)).rounded_full().bg(rgb(entry.color_rgb))) + .child(div().text_sm().text_color(cx.theme().foreground).child(label)), + ); + } + list.into_any_element() +} + +fn connectivity_text(phase: &SessionPhase) -> String { + match phase { + SessionPhase::Creating => "Starting collaboration...".to_string(), + SessionPhase::Joining(_) => "Joining collaboration...".to_string(), + SessionPhase::Attached(attachment) => match &attachment.connectivity { + Connectivity::Online if attachment.peers_present == 0 => "Only you - share the invite.".to_string(), + Connectivity::Online => format!("{} people in session.", attachment.peers_present + 1), + Connectivity::Offline { .. } => "Offline - reconnecting. Changes will sync when connected.".to_string(), + }, + SessionPhase::Detached(_) => "This document is local.".to_string(), + } +} diff --git a/crates/flowstate/src/collab/shutdown.rs b/crates/flowstate/src/collab/shutdown.rs new file mode 100644 index 00000000..6db912e0 --- /dev/null +++ b/crates/flowstate/src/collab/shutdown.rs @@ -0,0 +1,23 @@ +use flowstate_collab::net::NetCommand; +use gpui::BorrowAppContext; + +use super::CollabManager; + +impl CollabManager { + pub(super) fn shutdown_runtime(&mut self) { + if let Some(runtime) = self.runtime.take() { + let _ = runtime.commands.try_send(NetCommand::Shutdown); + } + self.sessions_by_id.clear(); + self.session_by_panel.clear(); + self.event_pump_started = false; + self.endpoint_online = false; + } +} + +pub fn shutdown(cx: &mut C) +where + C: BorrowAppContext, +{ + cx.update_default_global::(|manager, _| manager.shutdown_runtime()); +} diff --git a/crates/flowstate/src/collab/status.rs b/crates/flowstate/src/collab/status.rs index 4be211ae..ea59dc14 100644 --- a/crates/flowstate/src/collab/status.rs +++ b/crates/flowstate/src/collab/status.rs @@ -30,6 +30,42 @@ pub fn status_pill(phase: &SessionPhase, cx: &App) -> AnyElement { .into_any_element() } +pub fn tab_badge(phase: &SessionPhase, cx: &App) -> Option { + let (count, color) = match phase { + SessionPhase::Attached(attachment) => { + let color = match &attachment.connectivity { + Connectivity::Online => cx.theme().success, + Connectivity::Offline { .. } => cx.theme().muted_foreground, + }; + (Some(attachment.peers_present + 1), color) + }, + SessionPhase::Creating | SessionPhase::Joining(_) => (None, cx.theme().info), + SessionPhase::Detached(_) => return None, + }; + + Some( + h_flex() + .items_center() + .gap_0p5() + .child( + div() + .size(px(6.0)) + .rounded_full() + .bg(color), + ) + .when_some(count, |this, count| { + this.child( + div() + .text_size(px(9.0)) + .font_weight(gpui::FontWeight::SEMIBOLD) + .text_color(color) + .child(count.to_string()), + ) + }) + .into_any_element(), + ) +} + fn status_label_and_color(phase: &SessionPhase, cx: &App) -> (String, gpui::Hsla) { match phase { SessionPhase::Creating => ("Starting share".to_string(), cx.theme().info), @@ -52,7 +88,7 @@ fn join_label(stage: &JoinStage) -> String { } fn attached_label_and_color(attachment: &Attachment, cx: &App) -> (String, gpui::Hsla) { - match attachment.connectivity { + match &attachment.connectivity { Connectivity::Online => { let participants = attachment.peers_present + 1; if attachment.peers_present == 0 { diff --git a/crates/flowstate/src/commands.rs b/crates/flowstate/src/commands.rs index 02e5b684..f7a8b887 100644 --- a/crates/flowstate/src/commands.rs +++ b/crates/flowstate/src/commands.rs @@ -76,6 +76,8 @@ pub enum CommandId { OpenDocument, OpenDemoDocument, CloseDocument, + ShareDocument, + JoinSession, StartCollaboration, CopyCollaborationTicket, JoinCollaborationFromClipboard, @@ -205,6 +207,8 @@ pub const COMMAND_SPECS: &[CommandSpec] = &[ CommandSpec::new(CommandId::OpenDocument, "Open Document", APP, &[]), CommandSpec::new(CommandId::OpenDemoDocument, "Open Demo Document", APP, &[]), CommandSpec::new(CommandId::CloseDocument, "Close Document", APP, &[]), + CommandSpec::new(CommandId::ShareDocument, "Share / Collaborate...", APP, &[]), + CommandSpec::new(CommandId::JoinSession, "Join Collaboration Session...", APP, &[]), CommandSpec::new(CommandId::StartCollaboration, "Start Collaboration", APP, &[]), CommandSpec::new(CommandId::CopyCollaborationTicket, "Copy Collaboration Invite", APP, &[]), CommandSpec::new(CommandId::JoinCollaborationFromClipboard, "Join Collaboration from Clipboard", APP, &[]), diff --git a/crates/flowstate/src/commands/keymap.rs b/crates/flowstate/src/commands/keymap.rs index 6ed22b1d..4e69f6ea 100644 --- a/crates/flowstate/src/commands/keymap.rs +++ b/crates/flowstate/src/commands/keymap.rs @@ -151,6 +151,8 @@ fn action_for_command(command: CommandId) -> Option> { | CommandId::OpenDocument | CommandId::OpenDemoDocument | CommandId::CloseDocument + | CommandId::ShareDocument + | CommandId::JoinSession | CommandId::StartCollaboration | CommandId::CopyCollaborationTicket | CommandId::JoinCollaborationFromClipboard diff --git a/crates/flowstate/src/rich_text_element/mod.rs b/crates/flowstate/src/rich_text_element/mod.rs index 3bece707..4bf4a4b9 100644 --- a/crates/flowstate/src/rich_text_element/mod.rs +++ b/crates/flowstate/src/rich_text_element/mod.rs @@ -64,6 +64,8 @@ pub fn flowstate_command_to_rich_text(command: CommandId) -> Option) { + self.open_collaboration_dialog_with_mode(crate::collab::share_dialog::CollabDialogMode::Share, window, cx); + } + + pub fn open_join_collaboration_dialog(&mut self, window: &mut Window, cx: &mut Context) { + self.open_collaboration_dialog_with_mode(crate::collab::share_dialog::CollabDialogMode::Join, window, cx); + } + + pub fn close_collaboration_dialog(&mut self, cx: &mut Context) { + self.collaboration_dialog = None; + cx.notify(); + } + + fn open_collaboration_dialog_with_mode( + &mut self, + mode: crate::collab::share_dialog::CollabDialogMode, + window: &mut Window, + cx: &mut Context, + ) { + let workspace = cx.entity().downgrade(); + let panel_id = self.active_editor.as_ref().and(self.active_document_id); + let dialog = cx.new(|cx| crate::collab::share_dialog::CollabShareDialog::new(workspace, panel_id, mode, window, cx)); + dialog.update(cx, |dialog, cx| dialog.focus(window, cx)); + self.collaboration_dialog = Some(dialog); + cx.notify(); + } + pub fn start_collaboration_on_active_document(&mut self, cx: &mut Context) -> Option { - let panel_id = self.active_document_id?; - let editor = self.active_editor.clone()?; + self.start_collaboration_on_document(self.active_document_id?, cx) + } + + pub fn start_collaboration_on_document(&mut self, panel_id: Uuid, cx: &mut Context) -> Option { + let panel = self + .document_panels + .iter() + .find(|panel| panel.read(cx).id() == panel_id)?; + let editor = panel.read(cx).editor(); let title = self .document_panels .iter() @@ -116,6 +150,51 @@ impl Workspace { self.join_collaboration_session(ticket, window, cx).is_some() } + pub fn confirm_leave_collaboration_on_active_document(&mut self, window: &mut Window, cx: &mut Context) -> bool { + self + .active_document_id + .is_some_and(|panel_id| self.confirm_leave_collaboration_on_panel(panel_id, window, cx)) + } + + pub fn confirm_leave_collaboration_on_panel(&mut self, panel_id: Uuid, window: &mut Window, cx: &mut Context) -> bool { + let phase = crate::collab::phase_for_panel(panel_id, cx); + if !collaboration_phase_blocks_close(phase.as_ref()) { + return false; + } + let detail = format!("{} Your copy of the document stays open.", collaboration_leave_detail(phase.as_ref())); + let answer = window.prompt( + PromptLevel::Warning, + "Leave this session?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ); + let window_handle = window.window_handle(); + cx.spawn(async move |workspace, cx| { + if !matches!(answer.await, Ok(0)) { + return; + } + let _ = workspace.update(cx, |workspace, cx| { + workspace.leave_collaboration_on_panel(panel_id, cx); + }); + let _ = window_handle.update(cx, |_, window, cx| { + std::mem::drop(window.prompt( + PromptLevel::Info, + "Left session", + Some("Left session - this copy is now local."), + &[PromptButton::ok("Ok")], + cx, + )); + }); + }) + .detach(); + true + } + + pub fn leave_collaboration_on_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> bool { + crate::collab::leave_session_for_panel(panel_id, cx) + } + pub fn join_collaboration_session( &mut self, ticket: flowstate_collab::ticket::SessionTicket, @@ -198,7 +277,7 @@ impl Workspace { pub fn leave_collaboration_on_active_document(&mut self, cx: &mut Context) -> bool { self .active_document_id - .is_some_and(|panel_id| crate::collab::leave_session_for_panel(panel_id, cx)) + .is_some_and(|panel_id| self.leave_collaboration_on_panel(panel_id, cx)) } pub fn leave_all_collaboration_sessions(&mut self, cx: &mut Context) { diff --git a/crates/flowstate/src/workspace/workspace/collab_prompts.rs b/crates/flowstate/src/workspace/workspace/collab_prompts.rs new file mode 100644 index 00000000..14960b67 --- /dev/null +++ b/crates/flowstate/src/workspace/workspace/collab_prompts.rs @@ -0,0 +1,324 @@ +#[derive(Clone)] +struct CollaborationClosePanel { + id: Uuid, + panel: PanelKind, + save_prompt: CollaborationSavePrompt, +} + +#[derive(Clone, Copy)] +enum CollaborationSavePrompt { + Clean, + Dirty, + Pathless, +} + +enum CollaborationCloseResolution { + Proceed { discard: Option }, + Cancelled, +} + +#[hotpath::measure_all] +impl Workspace { + fn close_collaboration_document_panel( + &mut self, + panel_id: Uuid, + panel_kind: PanelKind, + window: &mut Window, + cx: &mut Context, + ) -> bool { + let phase = crate::collab::phase_for_panel(panel_id, cx); + if !collaboration_phase_blocks_close(phase.as_ref()) { + return false; + } + + let detail = collaboration_leave_detail(phase.as_ref()); + let save_prompt = collaboration_save_prompt_for_panel(&panel_kind, cx); + let answer = window.prompt( + PromptLevel::Warning, + "Leave the collaboration session?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ); + let window_handle = window.window_handle(); + cx.spawn(async move |workspace, cx| { + if !matches!(answer.await, Ok(0)) { + return; + } + + let resolution = resolve_collaboration_save(panel_kind, save_prompt, window_handle, cx).await; + let CollaborationCloseResolution::Proceed { discard } = resolution else { + return; + }; + if let Some(panel) = discard { + panel.discard(cx); + } + + let _ = window_handle.update(cx, |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| { + workspace.leave_collaboration_on_panel(panel_id, cx); + workspace.remove_document_panel(panel_id, window, cx); + }); + }); + }) + .detach(); + true + } + + fn request_close_window_with_collaboration(&mut self, window: &mut Window, cx: &mut Context) -> bool { + let collaboration_panels = self.collaboration_close_panels(cx); + if collaboration_panels.is_empty() { + return false; + } + + let collaboration_ids = collaboration_panels + .iter() + .map(|panel| panel.id) + .collect::>(); + let dirty_panels = self + .dirty_panels(cx) + .into_iter() + .filter(|panel| !collaboration_ids.contains(&panel.panel_id(cx))) + .collect::>(); + + let answer = if collaboration_panels.len() == 1 { + let phase = crate::collab::phase_for_panel(collaboration_panels[0].id, cx); + window.prompt( + PromptLevel::Warning, + "Leave the collaboration session and quit?", + Some(&collaboration_leave_detail(phase.as_ref())), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + } else { + let detail = format!( + "You're in {} collaboration sessions. Leave all and quit?", + collaboration_panels.len() + ); + window.prompt( + PromptLevel::Warning, + "Leave collaboration sessions?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + }; + let window_handle = window.window_handle(); + + cx.spawn(async move |workspace, cx| { + if !matches!(answer.await, Ok(0)) { + return; + } + + let mut discards = Vec::new(); + for panel in collaboration_panels { + match resolve_collaboration_save(panel.panel, panel.save_prompt, window_handle, cx).await { + CollaborationCloseResolution::Proceed { discard } => { + if let Some(discard) = discard { + discards.push(discard); + } + }, + CollaborationCloseResolution::Cancelled => return, + } + } + + if !resolve_dirty_window_close(dirty_panels, window_handle, cx).await { + return; + } + + for panel in discards { + panel.discard(cx); + } + + let _ = window_handle.update(cx, |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); + crate::collab::shutdown(cx); + window.remove_window(); + }); + }) + .detach(); + true + } + + fn collaboration_close_panels(&self, cx: &App) -> Vec { + self + .document_panels + .iter() + .filter_map(|panel| { + let panel_state = panel.read(cx); + let id = panel_state.id(); + let phase = crate::collab::phase_for_panel(id, cx); + collaboration_phase_blocks_close(phase.as_ref()).then(|| { + let panel_kind = PanelKind::Document { + panel: panel.clone(), + editor: panel_state.editor(), + }; + CollaborationClosePanel { + id, + save_prompt: collaboration_save_prompt_for_panel(&panel_kind, cx), + panel: panel_kind, + } + }) + }) + .collect() + } +} + +#[hotpath::measure_all] +impl PanelKind { + fn panel_id(&self, cx: &App) -> Uuid { + match self { + PanelKind::Document { panel, .. } => panel.read(cx).id(), + PanelKind::Flow { panel, .. } => panel.read(cx).id(), + } + } +} + +#[hotpath::measure] +async fn resolve_collaboration_save( + panel: PanelKind, + prompt: CollaborationSavePrompt, + window_handle: AnyWindowHandle, + cx: &mut gpui::AsyncApp, +) -> CollaborationCloseResolution { + match prompt { + CollaborationSavePrompt::Clean => CollaborationCloseResolution::Proceed { discard: None }, + CollaborationSavePrompt::Dirty | CollaborationSavePrompt::Pathless => { + let answer = match window_handle.update(cx, |_, window, cx| { + let (detail, save_label) = match prompt { + CollaborationSavePrompt::Pathless => ("This shared tab has no saved file.", "Save As..."), + CollaborationSavePrompt::Dirty => ("This document has unsaved changes.", "Save"), + CollaborationSavePrompt::Clean => unreachable!(), + }; + window.prompt( + PromptLevel::Warning, + "Save changes before closing?", + Some(detail), + &[PromptButton::ok(save_label), PromptButton::new("Don't Save"), PromptButton::cancel("Cancel")], + cx, + ) + }) { + Ok(answer) => answer, + Err(error) => { + show_save_failed(window_handle, cx, format!("failed to show save prompt: {error}")); + return CollaborationCloseResolution::Cancelled; + }, + }; + + match answer.await { + Ok(0) => match panel.save(window_handle, cx).await { + PanelSaveOutcome::Saved => CollaborationCloseResolution::Proceed { discard: None }, + PanelSaveOutcome::Cancelled => CollaborationCloseResolution::Cancelled, + PanelSaveOutcome::Failed(error) => { + show_save_failed(window_handle, cx, error); + CollaborationCloseResolution::Cancelled + }, + }, + Ok(1) => CollaborationCloseResolution::Proceed { discard: Some(panel) }, + _ => CollaborationCloseResolution::Cancelled, + } + }, + } +} + +#[hotpath::measure] +async fn resolve_dirty_window_close( + dirty_panels: Vec, + window_handle: AnyWindowHandle, + cx: &mut gpui::AsyncApp, +) -> bool { + if dirty_panels.is_empty() { + return true; + } + let message = if dirty_panels.len() == 1 { + "This document has unsaved changes." + } else { + "One or more documents have unsaved changes." + }; + let answer = match window_handle.update(cx, |_, window, cx| { + window.prompt( + PromptLevel::Warning, + "Save changes before closing?", + Some(message), + &[PromptButton::ok("Save"), PromptButton::new("Don't Save"), PromptButton::cancel("Cancel")], + cx, + ) + }) { + Ok(answer) => answer, + Err(error) => { + show_save_failed(window_handle, cx, format!("failed to show save prompt: {error}")); + return false; + }, + }; + + match answer.await { + Ok(0) => { + for panel in dirty_panels { + match panel.save(window_handle, cx).await { + PanelSaveOutcome::Saved => {}, + PanelSaveOutcome::Cancelled => return false, + PanelSaveOutcome::Failed(error) => { + show_save_failed(window_handle, cx, error); + return false; + }, + } + } + true + }, + Ok(1) => { + for panel in dirty_panels { + panel.discard(cx); + } + true + }, + _ => false, + } +} + +#[hotpath::measure] +fn collaboration_save_prompt_for_panel(panel: &PanelKind, cx: &App) -> CollaborationSavePrompt { + match panel { + PanelKind::Document { editor, .. } => { + let editor = editor.read(cx); + if editor.document_path().is_none() { + CollaborationSavePrompt::Pathless + } else if editor.has_unsaved_changes() { + CollaborationSavePrompt::Dirty + } else { + CollaborationSavePrompt::Clean + } + }, + PanelKind::Flow { editor, .. } => { + let editor = editor.read(cx); + if editor.document_path().is_none() { + CollaborationSavePrompt::Pathless + } else if editor.has_unsaved_changes() { + CollaborationSavePrompt::Dirty + } else { + CollaborationSavePrompt::Clean + } + }, + } +} + +#[hotpath::measure] +fn collaboration_phase_blocks_close(phase: Option<&crate::collab::SessionPhase>) -> bool { + phase.is_some_and(|phase| !matches!(phase, crate::collab::SessionPhase::Detached(_))) +} + +#[hotpath::measure] +fn collaboration_leave_detail(phase: Option<&crate::collab::SessionPhase>) -> String { + match phase { + Some(crate::collab::SessionPhase::Attached(attachment)) if attachment.peers_present == 0 => { + "This tab is live, but no other people are present right now.".to_string() + }, + Some(crate::collab::SessionPhase::Attached(attachment)) => format!( + "This tab is live with {} other {}.", + attachment.peers_present, + if attachment.peers_present == 1 { "person" } else { "people" } + ), + Some(crate::collab::SessionPhase::Creating) => "This tab is starting a collaboration session.".to_string(), + Some(crate::collab::SessionPhase::Joining(_)) => "This tab is joining a collaboration session.".to_string(), + _ => "This tab is connected to a collaboration session.".to_string(), + } +} diff --git a/crates/flowstate/src/workspace/workspace/documents.rs b/crates/flowstate/src/workspace/workspace/documents.rs index 8e225b38..a7879e2c 100644 --- a/crates/flowstate/src/workspace/workspace/documents.rs +++ b/crates/flowstate/src/workspace/workspace/documents.rs @@ -89,6 +89,7 @@ impl Workspace { autosave_enabled: load_autosave(), autosave_document_generations: HashMap::new(), autosave_flow_in_flight: HashSet::new(), + collaboration_dialog: None, file_search_overlay: None, tub_root: None, tub_index: None, @@ -694,6 +695,9 @@ impl Workspace { else { return; }; + if self.close_collaboration_document_panel(panel_id, panel_kind.clone(), window, cx) { + return; + } if !panel_kind.is_dirty(cx) { self.remove_document_panel(panel_id, window, cx); return; @@ -734,6 +738,9 @@ impl Workspace { } fn request_close_window(&mut self, window: &mut Window, cx: &mut Context) { + if self.request_close_window_with_collaboration(window, cx) { + return; + } let dirty_panels = self.dirty_panels(cx); if dirty_panels.is_empty() { self.leave_all_collaboration_sessions(cx); diff --git a/crates/flowstate/src/workspace/workspace/keybindings.rs b/crates/flowstate/src/workspace/workspace/keybindings.rs index 9cfed3fb..9578568f 100644 --- a/crates/flowstate/src/workspace/workspace/keybindings.rs +++ b/crates/flowstate/src/workspace/workspace/keybindings.rs @@ -52,10 +52,18 @@ impl Workspace { self.close_active_document(window, cx); true }, + CommandId::ShareDocument => { + self.open_collaboration_dialog(window, cx); + true + }, + CommandId::JoinSession => { + self.open_join_collaboration_dialog(window, cx); + true + }, CommandId::StartCollaboration => self.start_collaboration_on_active_document(cx).is_some(), CommandId::CopyCollaborationTicket => self.copy_active_collaboration_ticket(window, cx), CommandId::JoinCollaborationFromClipboard => self.join_collaboration_from_clipboard(window, cx), - CommandId::LeaveCollaboration => self.leave_collaboration_on_active_document(cx), + CommandId::LeaveCollaboration => self.confirm_leave_collaboration_on_active_document(window, cx), CommandId::FindInDocument => self.open_active_document_search_bar(window, cx), CommandId::ZoomIn => { if let Some(editor) = self.active_editor.clone() { @@ -211,7 +219,7 @@ impl Workspace { } fn non_document_keybinding_surface_is_open(&self) -> bool { - self.settings_overlay.is_some() || self.file_search_overlay.is_some() + self.settings_overlay.is_some() || self.file_search_overlay.is_some() || self.collaboration_dialog.is_some() } fn focused_workspace_input_is_focused(&self, window: &Window, cx: &App) -> bool { @@ -229,6 +237,10 @@ impl Workspace { .file_search_overlay .as_ref() .is_some_and(|overlay| overlay.read(cx).focus_handle(cx).is_focused(window)) + || self + .collaboration_dialog + .as_ref() + .is_some_and(|dialog| dialog.read(cx).focus_handle(cx).is_focused(window)) || self .document_panels .iter() diff --git a/crates/flowstate/src/workspace/workspace/mod.rs b/crates/flowstate/src/workspace/workspace/mod.rs index b0399622..6b27bbbc 100644 --- a/crates/flowstate/src/workspace/workspace/mod.rs +++ b/crates/flowstate/src/workspace/workspace/mod.rs @@ -103,6 +103,7 @@ pub struct Workspace { autosave_enabled: bool, autosave_document_generations: HashMap, autosave_flow_in_flight: HashSet, + collaboration_dialog: Option>, file_search_overlay: Option>, tub_root: Option, tub_index: Option>, @@ -261,6 +262,7 @@ impl DocumentStyleSection { } include!("documents.rs"); +include!("collab_prompts.rs"); include!("collab.rs"); include!("workspace_state.rs"); include!("load.rs"); diff --git a/crates/flowstate/src/workspace/workspace/render_documents.rs b/crates/flowstate/src/workspace/workspace/render_documents.rs index 187be03c..fd0b30bb 100644 --- a/crates/flowstate/src/workspace/workspace/render_documents.rs +++ b/crates/flowstate/src/workspace/workspace/render_documents.rs @@ -75,6 +75,7 @@ impl Workspace { .children(tabs.into_iter().map(|tab| { let panel_id = tab.id; let workspace = workspace.clone(); + let collab_phase = crate::collab::phase_for_panel(panel_id, cx); let tab_prefix = h_flex() .ml(px(5.0)) .mr(px(-3.0)) @@ -104,6 +105,9 @@ impl Workspace { .border_color(cx.theme().warning.opacity(0.72)) .child(pin_label), ) + }) + .when_some(collab_phase.as_ref().and_then(|phase| crate::collab::status::tab_badge(phase, cx)), |this, badge| { + this.child(badge) }); let close_button = icon_button(("close-tab", panel_id.as_u128() as u64), AppIcon::Close) .tooltip("Close document") diff --git a/crates/flowstate/src/workspace/workspace/render_status.rs b/crates/flowstate/src/workspace/workspace/render_status.rs index 1f62927a..30f27a3e 100644 --- a/crates/flowstate/src/workspace/workspace/render_status.rs +++ b/crates/flowstate/src/workspace/workspace/render_status.rs @@ -284,7 +284,13 @@ impl Workspace { if matches!(phase, crate::collab::SessionPhase::Detached(_)) { this } else { - this.child(crate::collab::status::status_pill(&phase, cx)) + this.child( + Button::new("collaboration-status-pill") + .text() + .compact() + .child(crate::collab::status::status_pill(&phase, cx)) + .on_click(cx.listener(|workspace, _, window, cx| workspace.open_collaboration_dialog(window, cx))), + ) } }) .when_some(zoom, |this, percent| this.child(self.render_zoom_slider(percent, cx))) diff --git a/crates/flowstate/src/workspace/workspace/render_top_bar.rs b/crates/flowstate/src/workspace/workspace/render_top_bar.rs index aaf3b073..c4a50afb 100644 --- a/crates/flowstate/src/workspace/workspace/render_top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/render_top_bar.rs @@ -2,6 +2,10 @@ impl Workspace { fn render_top_bar(&mut self, _window: &Window, cx: &mut Context) -> impl IntoElement { let workspace = cx.entity().downgrade(); + let active_collaborating = self + .active_document_id + .and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)) + .is_some_and(|phase| !matches!(phase, crate::collab::SessionPhase::Detached(_))); TitleBar::new() .on_close_window(move |_, window, cx| { let _ = workspace.update(cx, |workspace, cx| workspace.request_close_window(window, cx)); @@ -23,6 +27,8 @@ impl Workspace { !self.ribbon_collapsed, !self.toolkit_collapsed, )) + .child(div().flex_1()) + .child(share_top_bar_button(cx, self.active_editor.is_some(), active_collaborating)) .child(settings_top_bar_button(cx)), ) } diff --git a/crates/flowstate/src/workspace/workspace/top_bar.rs b/crates/flowstate/src/workspace/workspace/top_bar.rs index ac3b2ed5..c24f701e 100644 --- a/crates/flowstate/src/workspace/workspace/top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/top_bar.rs @@ -122,18 +122,21 @@ fn collaboration_top_bar_button(cx: &mut Context, has_document: bool) .ghost() .dropdown_menu(move |menu, _, _| { menu - .item(file_menu_item(workspace.clone(), "Start Sharing", !has_document, |workspace, _, cx| { - let _ = workspace.start_collaboration_on_active_document(cx); + .item(file_menu_item(workspace.clone(), "Share / Collaborate...", !has_document, |workspace, window, cx| { + workspace.open_collaboration_dialog(window, cx); })) .item(file_menu_item(workspace.clone(), "Copy Invite Ticket", !has_document, |workspace, window, cx| { workspace.copy_active_collaboration_ticket(window, cx); })) .separator() + .item(file_menu_item(workspace.clone(), "Join Session...", false, |workspace, window, cx| { + workspace.open_join_collaboration_dialog(window, cx); + })) .item(file_menu_item(workspace.clone(), "Join from Clipboard", false, |workspace, window, cx| { workspace.join_collaboration_from_clipboard(window, cx); })) - .item(file_menu_item(workspace.clone(), "Leave Shared Session", !has_document, |workspace, _, cx| { - workspace.leave_collaboration_on_active_document(cx); + .item(file_menu_item(workspace.clone(), "Leave Shared Session", !has_document, |workspace, window, cx| { + workspace.confirm_leave_collaboration_on_active_document(window, cx); })) }), ) @@ -176,6 +179,13 @@ fn file_top_bar_button(has_document: bool, cx: &mut Context) -> impl workspace.save_active_as(window, cx); })) .separator() + .item(file_menu_item(workspace.clone(), "Share Document...", !has_document, |workspace, window, cx| { + workspace.open_collaboration_dialog(window, cx); + })) + .item(file_menu_item(workspace.clone(), "Join Session...", false, |workspace, window, cx| { + workspace.open_join_collaboration_dialog(window, cx); + })) + .separator() .item(file_menu_item(workspace.clone(), "Close File", !has_document, |workspace, window, cx| { workspace.close_active_document(window, cx); })) @@ -246,6 +256,38 @@ fn insert_top_bar_button(cx: &mut Context, has_document: bool) -> imp ) } +#[hotpath::measure] +fn share_top_bar_button(cx: &mut Context, has_document: bool, collaborating: bool) -> impl IntoElement { + let workspace = cx.entity().downgrade(); + let button = Button::new("top-share-document") + .label("Share") + .xsmall() + .ghost() + .disabled(!has_document) + .tooltip("Share / Collaborate") + .on_click(move |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| workspace.open_collaboration_dialog(window, cx)); + }); + + div() + .h_full() + .flex_none() + .flex() + .items_center() + .justify_center() + .on_mouse_down(MouseButton::Left, |_, _, cx| cx.stop_propagation()) + .child(if collaborating { + button.custom( + ButtonCustomVariant::new(cx) + .foreground(cx.theme().success) + .hover(cx.theme().success.opacity(0.12)) + .active(cx.theme().success.opacity(0.18)), + ) + } else { + button + }) +} + #[hotpath::measure] fn insert_image_from_top_bar(workspace: &WeakEntity, cx: &mut App) { let _ = workspace.update(cx, |workspace, cx| { diff --git a/crates/flowstate/src/workspace/workspace/traits.rs b/crates/flowstate/src/workspace/workspace/traits.rs index 4111d93b..1c61fdcc 100644 --- a/crates/flowstate/src/workspace/workspace/traits.rs +++ b/crates/flowstate/src/workspace/workspace/traits.rs @@ -45,6 +45,7 @@ impl Render for Workspace { this.child(self.render_settings_overlay(overlay, cx)) }) .when_some(self.file_search_overlay.clone(), |this, overlay| this.child(overlay)) + .when_some(self.collaboration_dialog.clone(), |this, dialog| this.child(dialog)) .when_some(self.outline_context_menu.as_ref(), |this, ctx| { let workspace = cx.entity().downgrade(); let menu = ctx.menu_view.clone(); diff --git a/crates/flowstate/src/workspace/workspace/window.rs b/crates/flowstate/src/workspace/workspace/window.rs index 663c2385..eae10e7d 100644 --- a/crates/flowstate/src/workspace/workspace/window.rs +++ b/crates/flowstate/src/workspace/workspace/window.rs @@ -9,6 +9,90 @@ pub fn install_workspace_close_prompt(workspace: Entity, window: &mut return true; } + let collaboration_panels = workspace.read(cx).collaboration_close_panels(cx); + if !collaboration_panels.is_empty() { + if prompt_open.get() { + return false; + } + prompt_open.set(true); + + let collaboration_ids = collaboration_panels + .iter() + .map(|panel| panel.id) + .collect::>(); + let dirty_panels = workspace + .read(cx) + .dirty_panels(cx) + .into_iter() + .filter(|panel| !collaboration_ids.contains(&panel.panel_id(cx))) + .collect::>(); + let answer = if collaboration_panels.len() == 1 { + let phase = crate::collab::phase_for_panel(collaboration_panels[0].id, cx); + window.prompt( + PromptLevel::Warning, + "Leave the collaboration session and quit?", + Some(&collaboration_leave_detail(phase.as_ref())), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + } else { + let detail = format!( + "You're in {} collaboration sessions. Leave all and quit?", + collaboration_panels.len() + ); + window.prompt( + PromptLevel::Warning, + "Leave collaboration sessions?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + }; + let prompt_open = prompt_open.clone(); + let allow_close = allow_close.clone(); + let workspace = workspace.clone(); + + cx.spawn(async move |cx| { + let mut should_close = false; + if matches!(answer.await, Ok(0)) { + let mut discards = Vec::new(); + let mut ok = true; + for panel in collaboration_panels { + match resolve_collaboration_save(panel.panel, panel.save_prompt, window_handle, cx).await { + CollaborationCloseResolution::Proceed { discard } => { + if let Some(discard) = discard { + discards.push(discard); + } + }, + CollaborationCloseResolution::Cancelled => { + ok = false; + break; + }, + } + } + if ok && resolve_dirty_window_close(dirty_panels, window_handle, cx).await { + for panel in discards { + panel.discard(cx); + } + should_close = true; + } + } + + prompt_open.set(false); + if should_close { + allow_close.set(true); + let _ = window_handle.update(cx, |_, window, cx| { + workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); + crate::collab::shutdown(cx); + window.remove_window(); + }); + } + }) + .detach(); + + return false; + } + let dirty_panels = workspace.read(cx).dirty_panels(cx); if dirty_panels.is_empty() { workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); diff --git a/crates/gpui-flowtext/src/document/blocks.rs b/crates/gpui-flowtext/src/document/blocks.rs index de88397a..f26c0997 100644 --- a/crates/gpui-flowtext/src/document/blocks.rs +++ b/crates/gpui-flowtext/src/document/blocks.rs @@ -24,6 +24,16 @@ pub struct AssetRecord { pub bytes: Arc>, } +pub const IMAGE_LOADING_PLACEHOLDER_WIDTH_PX: f32 = 240.0; +pub const IMAGE_LOADING_PLACEHOLDER_HEIGHT_PX: f32 = 160.0; + +impl AssetRecord { + #[must_use] + pub fn is_loading_placeholder(&self) -> bool { + self.bytes.is_empty() + } +} + #[derive(Clone, Debug, PartialEq, Eq)] pub struct ImageBlock { pub asset_id: AssetId, diff --git a/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs b/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs index 09d57050..4c76e189 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs @@ -115,6 +115,7 @@ impl RichTextEditor { let _ = self.paragraph_item_sizes(window, cx); self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } @@ -199,6 +200,7 @@ impl RichTextEditor { self.goal_x = Some(used_goal_x); self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } @@ -256,6 +258,7 @@ impl RichTextEditor { self.goal_x = Some(goal_x); self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs index 417a6f67..e288ab42 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs @@ -47,22 +47,33 @@ impl RichTextEditor { CollabPatch::ReplaceObjectBlock { row, block } => { let mut blocks = collab_structural_blocks_from_document(&self.document); if *row < blocks.len() { + if selected_block_in_range(self.selected_block, *row..row.saturating_add(1)) { + self.clear_remote_object_editing_state(); + } blocks[*row] = block.clone(); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + clamp_selection_to_document(&self.document, &mut self.selection); extend_invalidation(invalidation, 0..self.document.paragraphs.len()); } }, CollabPatch::InsertBlocks { row, blocks: inserted } => { let mut blocks = collab_structural_blocks_from_document(&self.document); let row = (*row).min(blocks.len()); + if selected_block_ix(self.selected_block).is_some_and(|block_ix| block_ix >= row) { + self.clear_remote_object_editing_state(); + } blocks.splice(row..row, inserted.iter().cloned()); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + clamp_selection_to_document(&self.document, &mut self.selection); extend_invalidation(invalidation, 0..self.document.paragraphs.len()); }, CollabPatch::DeleteBlocks { row, count } => { let mut blocks = collab_structural_blocks_from_document(&self.document); let start = (*row).min(blocks.len()); let end = start.saturating_add(*count).min(blocks.len()); + if selected_block_ix(self.selected_block).is_some_and(|block_ix| block_ix >= start) { + self.clear_remote_object_editing_state(); + } blocks.drain(start..end); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); clamp_selection_to_document(&self.document, &mut self.selection); @@ -71,6 +82,11 @@ impl RichTextEditor { CollabPatch::MoveBlock { from, to } => { let mut blocks = collab_structural_blocks_from_document(&self.document); if *from < blocks.len() { + let first = (*from).min(*to); + let last = (*from).max(*to); + if selected_block_ix(self.selected_block).is_some_and(|block_ix| (first..=last).contains(&block_ix)) { + self.clear_remote_object_editing_state(); + } let block = blocks.remove(*from); blocks.insert((*to).min(blocks.len()), block); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); @@ -83,6 +99,33 @@ impl RichTextEditor { }, } } + + fn clear_remote_object_editing_state(&mut self) { + self.selected_block = None; + self.image_resize_drag = None; + self.table_column_resize_drag = None; + self.table_cell_block_ix = 0; + self.table_cell_anchor = 0; + self.table_cell_caret = 0; + self.equation_source_anchor = 0; + self.equation_source_caret = 0; + } +} + +#[hotpath::measure] +fn selected_block_ix(selection: Option) -> Option { + match selection { + Some(BlockSelection::Image(block_ix) + | BlockSelection::Equation(block_ix) + | BlockSelection::Table(block_ix) + | BlockSelection::TableCell { block_ix, .. }) => Some(block_ix), + None => None, + } +} + +#[hotpath::measure] +fn selected_block_in_range(selection: Option, range: Range) -> bool { + selected_block_ix(selection).is_some_and(|block_ix| range.contains(&block_ix)) } #[hotpath::measure] diff --git a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs index 99bdde75..147bd3f7 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs @@ -262,8 +262,14 @@ impl RichTextEditor { self.collab_undo_redirect = hook; } - pub fn set_recovery_path(&mut self, path: Option) { + pub fn clear_collab_history(&mut self) { + self.undo_stack.clear(); + self.redo_stack.clear(); + } + + pub fn set_recovery_path(&mut self, path: Option, cx: &mut Context) { self.recovery_path = path; + self.schedule_recovery_write(cx); } pub fn collaboration_role(&self) -> Option { diff --git a/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs b/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs index d4e4c77a..cc0ef168 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs @@ -19,12 +19,17 @@ fn reserved_object_frame(document: &Document, row_size: Size, selected: fn image_object_frame(document: &Document, image: &ImageBlock, asset: &AssetRecord, row_size: Size, selected: bool) -> gpui::Div { let available_width = (row_size.width - document.theme.pageless_inset_x * 2.0).max(px(1.0)); let intrinsic = image_asset_intrinsic_size(asset); - let object_width = match image.sizing { - ImageSizing::Fixed { width_px, .. } => px(width_px as f32).min(available_width), - ImageSizing::FitWidth => available_width, - ImageSizing::Intrinsic => intrinsic - .map(|(width, _)| width.min(available_width)) - .unwrap_or(available_width), + let loading = asset.is_loading_placeholder(); + let object_width = if loading { + px(IMAGE_LOADING_PLACEHOLDER_WIDTH_PX).min(available_width) + } else { + match image.sizing { + ImageSizing::Fixed { width_px, .. } => px(width_px as f32).min(available_width), + ImageSizing::FitWidth => available_width, + ImageSizing::Intrinsic => intrinsic + .map(|(width, _)| width.min(available_width)) + .unwrap_or(available_width), + } }; let object_height = (row_size.height - document.theme.paragraph_after).max(px(1.0)); let left_margin = document.theme.pageless_inset_x @@ -41,13 +46,22 @@ fn image_object_frame(document: &Document, image: &ImageBlock, asset: &AssetReco .mr(document.theme.pageless_inset_x) .mb(document.theme.paragraph_after) .overflow_hidden() - .bg(rgb(0xffffff)) + .bg(if loading { rgb(0xf3f4f6) } else { rgb(0xffffff) }) .border_1() - .border_color(if selected { rgb(0x0969da) } else { rgb(0xffffff) }) + .border_color(if selected { + rgb(0x0969da) + } else if loading { + rgb(0xd0d7de) + } else { + rgb(0xffffff) + }) } #[hotpath::measure] fn image_asset_intrinsic_size(asset: &AssetRecord) -> Option<(Pixels, Pixels)> { + if asset.is_loading_placeholder() { + return None; + } let size = imagesize::blob_size(asset.bytes.as_ref()).ok()?; if size.width == 0 || size.height == 0 { return None; diff --git a/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs b/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs index 9faa7b9e..3fa523ea 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs @@ -13,6 +13,9 @@ fn render_image_block( .child("Missing image") .into_any_element(); }; + if asset.is_loading_placeholder() { + return render_loading_image_placeholder(document, image, asset, row_size, selected, editor, block_ix); + } let Some(format) = ImageFormat::from_mime_type(asset.mime_type.as_ref()) else { return reserved_object_frame(document, row_size, selected) .child("Unsupported image") @@ -37,6 +40,34 @@ fn render_image_block( .into_any_element() } +#[hotpath::measure] +fn render_loading_image_placeholder( + document: &Document, + image: &ImageBlock, + asset: &AssetRecord, + row_size: Size, + selected: bool, + editor: Entity, + block_ix: usize, +) -> gpui::AnyElement { + image_object_frame(document, image, asset, row_size, selected) + .child( + div() + .size_full() + .flex() + .flex_col() + .items_center() + .justify_center() + .gap_1() + .text_sm() + .text_color(rgb(0x6b7280)) + .child(div().text_xs().text_color(rgb(0x9ca3af)).child("...")) + .child("Loading image"), + ) + .when(selected, |this| this.children(image_resize_handles(editor, block_ix))) + .into_any_element() +} + #[hotpath::measure] fn image_resize_handles(editor: Entity, block_ix: usize) -> Vec { [ @@ -254,4 +285,3 @@ type EquationRenderCache = FxHashMap>, Stri static EQUATION_SVG_CACHE: OnceLock> = OnceLock::new(); static EQUATION_PNG_CACHE: OnceLock> = OnceLock::new(); - diff --git a/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs b/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs index 9d45f715..75d9183e 100644 --- a/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs +++ b/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs @@ -274,6 +274,14 @@ fn structural_block_bounds(document: &Document, width: Pixels, y: Pixels, height #[hotpath::measure] fn image_placeholder_height(document: &Document, image: &ImageBlock, width: Pixels) -> Pixels { + if document + .assets + .assets + .get(&image.asset_id) + .is_some_and(AssetRecord::is_loading_placeholder) + { + return px(IMAGE_LOADING_PLACEHOLDER_HEIGHT_PX); + } let available_width = (width - document.theme.pageless_inset_x * 2.0).max(px(1.0)); let intrinsic = image_intrinsic_size(document, image); match image.sizing { @@ -295,6 +303,9 @@ pub(super) fn image_layout_height_for_test(document: &Document, image: &ImageBlo #[hotpath::measure] fn image_intrinsic_size(document: &Document, image: &ImageBlock) -> Option<(Pixels, Pixels)> { let asset = document.assets.assets.get(&image.asset_id)?; + if asset.is_loading_placeholder() { + return None; + } let size = imagesize::blob_size(asset.bytes.as_ref()).ok()?; if size.width == 0 || size.height == 0 { return None; diff --git a/helpers/docs/collab_qa.md b/helpers/docs/collab_qa.md new file mode 100644 index 00000000..3e22751b --- /dev/null +++ b/helpers/docs/collab_qa.md @@ -0,0 +1,59 @@ +# Flowstate Collaboration QA + +Manual QA for P2P `.db8` collaboration. Run with two or three app instances using separate working directories. Use fresh documents unless a scenario says otherwise. + +## Setup + +1. Start instance A and open or create a `.db8` document. +2. Use `Share / Collaborate...` to start a session and copy an invite. +3. Start instance B, use `Join Collaboration Session...`, paste the invite, and verify a new shared tab opens. +4. For three-peer cases, repeat join from instance C using a fresh invite from either A or B. + +## Core Sync + +1. Type in A and verify B/C update live. +2. Type in B and verify A/C update live. +3. Paste multi-paragraph rich text and verify convergence. +4. Exercise multibyte text with the fixture `a\u{00e9}\u{1f30d}\u{2028}x`. +5. Apply paragraph styles, run styles, highlights, underline, and strikethrough from different peers. +6. Insert, move, replace, and delete image/equation/table blocks. +7. Paste an image on A; verify B/C show a loading placeholder then the image. +8. Paste a different image on B; verify A/C can serve it after B has it. + +## Presence + +1. Move the caret with keyboard arrows in A; verify B sees the remote caret move. +2. Select text in B; verify A sees the caret position update. +3. Leave C or kill it; verify presence ages out within 30 seconds. +4. Reopen the share dialog and verify participant count/status remains sensible. + +## Undo And Conflicts + +1. Type several edits on A, then undo and redo; verify only A's edits are undone/redone and all peers converge. +2. Interleave A and B edits in the same paragraph, then undo on A and B independently. +3. Simultaneously edit the same table cell from A and B; verify the block converges with last-writer-wins behavior. +4. Simultaneously split a paragraph on A while B types later in the paragraph; verify no data loss and convergence. + +## Network And Recovery + +1. Disconnect B's network or block traffic, keep editing on B, then reconnect; verify anti-entropy catches up. +2. Let A leave while B and C remain; verify B and C continue editing and converging. +3. Force a large paste above the gossip inline limit; verify blob pull or later anti-entropy convergence. +4. Quit or crash a pathless joined tab; verify a recovery file exists under the temp `flowstate-collab-recovery` directory and opens as a local document on recovery. +5. Test a stale/dead invite; verify join fails with actionable copy. + +## Prompts And Leave + +1. Click `Leave session`; cancel, then leave. Verify the tab remains open as a local document. +2. Close an attached clean path-backed tab; verify leave prompt, then close. +3. Close an attached dirty path-backed tab; verify leave prompt, save prompt, and cancel at each step keeps the session attached. +4. Close a pathless attached tab; verify leave prompt, `Save As...` / `Don't Save` / `Cancel` behavior. +5. Quit with multiple attached tabs; verify the combined leave prompt, then per-tab save prompts. Cancel anywhere must abort quit. + +## Compatibility And Error Cases + +1. Paste a malformed invite; verify inline parse error and no network dial. +2. Use an unsupported-version invite if available; verify clear rejection. +3. Try joining a session already open in the same app; verify it is rejected. +4. Kill the inviter before join; verify join fails without modifying existing documents. +5. Confirm saved files are plain `.db8` snapshots and do not contain CRDT history. From 7e7433d8c3a7baa645c3975c104c649ba5942a05 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Fri, 12 Jun 2026 23:17:56 -0500 Subject: [PATCH 06/45] add collaboration presence status --- crates/flowstate-collab/src/presence.rs | 2 - crates/flowstate/src/collab/mod.rs | 8 +- crates/flowstate/src/collab/presence_view.rs | 100 ++++++++++++++++++ crates/flowstate/src/collab/session.rs | 36 ++++++- crates/flowstate/src/collab/status.rs | 66 ++++++++++++ .../src/workspace/workspace/render_status.rs | 10 ++ 6 files changed, 216 insertions(+), 6 deletions(-) create mode 100644 crates/flowstate/src/collab/presence_view.rs create mode 100644 crates/flowstate/src/collab/status.rs diff --git a/crates/flowstate-collab/src/presence.rs b/crates/flowstate-collab/src/presence.rs index a2d87bda..b3cdce47 100644 --- a/crates/flowstate-collab/src/presence.rs +++ b/crates/flowstate-collab/src/presence.rs @@ -17,9 +17,7 @@ pub struct PresenceState { #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] pub struct PresenceSelection { - pub head_container: String, pub head: Vec, - pub anchor_container: String, pub anchor: Vec, } diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index 2de5c1d6..3bff0775 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -1,4 +1,6 @@ mod session; +mod presence_view; +pub mod status; use std::collections::HashMap; @@ -9,7 +11,7 @@ use flowstate_collab::{ net::{NetCommand, NetEvent, TicketSeed, runtime::{self, CommandSender}}, ticket::SessionTicket, }; -use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global}; +use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global, ReadGlobal}; use uuid::Uuid; use crate::rich_text_element::RichTextEditor; @@ -388,3 +390,7 @@ where { cx.update_default_global::(|manager, cx| manager.request_ticket_for_panel(panel_id, cx)) } + +pub fn phase_for_panel(panel_id: Uuid, cx: &App) -> Option { + CollabManager::global(cx).phase_for_panel(panel_id, cx) +} diff --git a/crates/flowstate/src/collab/presence_view.rs b/crates/flowstate/src/collab/presence_view.rs new file mode 100644 index 00000000..387d5ba8 --- /dev/null +++ b/crates/flowstate/src/collab/presence_view.rs @@ -0,0 +1,100 @@ +use std::path::PathBuf; + +use flowstate_collab::{SessionId, binding::DocBinding, presence::{PresenceSelection, PresenceStore}}; +use loro::{LoroDoc, cursor::{Cursor, PosType, Side}}; + +use crate::rich_text_element::{DocumentOffset, ExternalCaret, ParagraphId, RichTextEditor}; + +pub fn selection_for_editor(editor: &RichTextEditor, binding: &DocBinding) -> Option { + let selection = editor.selection().clone(); + let anchor_paragraph = editor.paragraph_id(selection.anchor.paragraph)?; + let head_paragraph = editor.paragraph_id(selection.head.paragraph)?; + Some(PresenceSelection { + anchor: cursor_bytes_for_offset(binding, anchor_paragraph, selection.anchor.byte)?, + head: cursor_bytes_for_offset(binding, head_paragraph, selection.head.byte)?, + }) +} + +pub fn external_carets(doc: &LoroDoc, binding: &DocBinding, presence: &PresenceStore) -> Vec { + let self_key = presence.self_key(); + presence + .roster() + .into_iter() + .filter(|entry| entry.key != self_key) + .filter_map(|entry| { + entry + .selection + .as_ref() + .and_then(|selection| external_caret_for_presence(doc, binding, selection, entry.color_rgb)) + }) + .collect() +} + +pub fn collaboration_recovery_path(session: SessionId, title: &str) -> PathBuf { + let dir = std::env::temp_dir().join("flowstate-collab-recovery"); + let _ = std::fs::create_dir_all(&dir); + let session_hex = session.to_string(); + let prefix = session_hex.get(..16).unwrap_or(&session_hex); + dir.join(format!("{prefix}-{}.db8", sanitized_recovery_title(title))) +} + +fn cursor_bytes_for_offset(binding: &DocBinding, paragraph: ParagraphId, byte: usize) -> Option> { + let row_ix = binding.by_paragraph.get(¶graph).copied()?; + let row = binding.rows.get(row_ix)?; + let text = row.text.as_ref()?; + let byte = byte.min(text.len_utf8()); + let pos = text.convert_pos(byte, PosType::Bytes, PosType::Unicode)?; + text.get_cursor(pos, Side::Middle).map(|cursor| cursor.encode()) +} + +fn external_caret_for_presence( + doc: &LoroDoc, + binding: &DocBinding, + selection: &PresenceSelection, + color_rgb: u32, +) -> Option { + let cursor = Cursor::decode(&selection.head).ok()?; + let row_ix = binding.by_container.get(&cursor.container).copied()?; + let row = binding.rows.get(row_ix)?; + let text = row.text.as_ref()?; + let resolved = doc.get_cursor_pos(&cursor).ok()?; + let byte = text.convert_pos(resolved.current.pos, PosType::Unicode, PosType::Bytes)?; + let paragraph = paragraph_ordinal_for_row(binding, row_ix)?; + Some(ExternalCaret { + offset: DocumentOffset { paragraph, byte }, + color_rgb, + }) +} + +fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option { + let mut paragraph = 0; + for (row_ix, row) in binding.rows.iter().enumerate() { + if row_ix == target_row { + return row.paragraph_id.map(|_| paragraph); + } + if row.paragraph_id.is_some() { + paragraph += 1; + } + } + None +} + +fn sanitized_recovery_title(title: &str) -> String { + let mut out = String::new(); + for ch in title.chars() { + if out.len() >= 48 { + break; + } + if ch.is_ascii_alphanumeric() || matches!(ch, '-' | '_' | '.') { + out.push(ch); + } else if ch.is_whitespace() && !out.ends_with('_') { + out.push('_'); + } + } + let trimmed = out.trim_matches(['_', '.']).to_string(); + if trimmed.is_empty() { + "shared-document".to_string() + } else { + trimmed + } +} diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index a81dd7dd..9a9446ef 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -12,7 +12,7 @@ use flowstate_collab::{ binding::DocBinding, local_apply::LocalApplier, net::{NetCommand, PublishPayload, runtime::CommandSender, direct::{DirectServeRequest, DirectSessionHandler}}, - presence::{PresenceState, PresenceStore}, + presence::{PresenceSelection, PresenceState, PresenceStore}, projection, proto_direct::AssetBytes, proto_gossip::GossipMsg, @@ -26,6 +26,8 @@ use uuid::Uuid; use crate::app_settings::load_document_theme; use crate::rich_text_element::{AssetId, CollabPatch, Document, EditorEvent, RichTextEditor, UndoRedirect}; +use super::presence_view; + #[derive(Clone, Debug)] pub enum SessionPhase { Creating, @@ -447,6 +449,9 @@ impl CollabSession { }; editor.update(cx, |editor, _| { + if editor.document_path().is_none() { + editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title))); + } editor.set_collab_capture(true); editor.set_collab_undo_redirect(Some(Rc::new(|_redirect: UndoRedirect| { // TODO(Flowstate P2P §8/M3): replace this guard with Loro UndoManager undo/redo routing. @@ -605,6 +610,7 @@ impl CollabSession { } let patches = std::mem::take(&mut self.pending_remote_patches); editor.update(cx, |editor, cx| editor.apply_collab_patches(&patches, cx)); + self.refresh_external_carets(cx); true } @@ -702,17 +708,18 @@ impl CollabSession { eprintln!("flowstate collab presence update failed: {error:#}"); } self.refresh_peer_count(); + self.refresh_external_carets(cx); cx.notify(); } } - fn refresh_own_presence(&mut self, _cx: &mut Context) { + fn refresh_own_presence(&mut self, cx: &mut Context) { let Some(presence) = &self.presence else { return; }; let state = PresenceState { name: default_presence_name(), - selection: None, + selection: self.own_presence_selection(cx), }; if let Err(error) = presence.set_self(&state) { eprintln!("flowstate collab presence encode failed: {error:#}"); @@ -720,6 +727,29 @@ impl CollabSession { self.refresh_peer_count(); } + fn own_presence_selection(&self, cx: &mut Context) -> Option { + let editor = self.editor.as_ref()?.read(cx); + let binding = self.binding.as_ref()?; + presence_view::selection_for_editor(editor, binding) + } + + fn refresh_external_carets(&mut self, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let Some(doc) = &self.doc else { + return; + }; + let Some(binding) = &self.binding else { + return; + }; + let Some(editor) = self.editor.clone() else { + return; + }; + let carets = presence_view::external_carets(doc, binding, presence); + editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); + } + fn publish_presence_snapshot(&self) { if let Some(presence) = &self.presence { self.publish_presence_bytes(presence.encode_all()); diff --git a/crates/flowstate/src/collab/status.rs b/crates/flowstate/src/collab/status.rs new file mode 100644 index 00000000..4be211ae --- /dev/null +++ b/crates/flowstate/src/collab/status.rs @@ -0,0 +1,66 @@ +use gpui::{AnyElement, App, IntoElement, div, px, prelude::*}; +use gpui_component::{ActiveTheme as _, h_flex}; + +use super::{Attachment, Connectivity, JoinStage, SessionPhase}; + +pub fn status_pill(phase: &SessionPhase, cx: &App) -> AnyElement { + let (label, color) = status_label_and_color(phase, cx); + h_flex() + .flex_none() + .items_center() + .gap_1() + .px_2() + .py_0p5() + .rounded_full() + .border_1() + .border_color(color.opacity(0.38)) + .bg(color.opacity(0.10)) + .child( + div() + .size(px(6.0)) + .rounded_full() + .bg(color), + ) + .child( + div() + .text_size(px(10.0)) + .text_color(color) + .child(label), + ) + .into_any_element() +} + +fn status_label_and_color(phase: &SessionPhase, cx: &App) -> (String, gpui::Hsla) { + match phase { + SessionPhase::Creating => ("Starting share".to_string(), cx.theme().info), + SessionPhase::Joining(stage) => (join_label(stage), cx.theme().info), + SessionPhase::Attached(attachment) => attached_label_and_color(attachment, cx), + SessionPhase::Detached(_) => ("Not shared".to_string(), cx.theme().muted_foreground), + } +} + +fn join_label(stage: &JoinStage) -> String { + match stage { + JoinStage::Resolving => "Resolving invite".to_string(), + JoinStage::Subscribing => "Joining session".to_string(), + JoinStage::FetchingSnapshot { got, total } => match total { + Some(total) => format!("Fetching {got}/{total} bytes"), + None => "Fetching document".to_string(), + }, + JoinStage::Building => "Opening shared doc".to_string(), + } +} + +fn attached_label_and_color(attachment: &Attachment, cx: &App) -> (String, gpui::Hsla) { + match attachment.connectivity { + Connectivity::Online => { + let participants = attachment.peers_present + 1; + if attachment.peers_present == 0 { + ("Only you".to_string(), cx.theme().success) + } else { + (format!("{participants} in session"), cx.theme().success) + } + }, + Connectivity::Offline { .. } => ("Offline - will sync".to_string(), cx.theme().warning), + } +} diff --git a/crates/flowstate/src/workspace/workspace/render_status.rs b/crates/flowstate/src/workspace/workspace/render_status.rs index 7fede050..1f62927a 100644 --- a/crates/flowstate/src/workspace/workspace/render_status.rs +++ b/crates/flowstate/src/workspace/workspace/render_status.rs @@ -264,6 +264,9 @@ impl Workspace { None }; let zoom = self.active_editor.as_ref().map(|editor| editor.read(cx).zoom_percent()); + let collab_phase = self + .active_document_id + .and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)); if let Some(percent) = zoom { self.sync_zoom_slider(percent, window, cx); } @@ -277,6 +280,13 @@ impl Workspace { .border_color(cx.theme().border) .bg(cx.theme().background) .child(div().flex_1()) + .when_some(collab_phase, |this, phase| { + if matches!(phase, crate::collab::SessionPhase::Detached(_)) { + this + } else { + this.child(crate::collab::status::status_pill(&phase, cx)) + } + }) .when_some(zoom, |this, percent| this.child(self.render_zoom_slider(percent, cx))) .when_some(speech_word_count, |this, count| { this.child( From 71570bd92a8b0ff3b54e4b60b868240b40b3537c Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Sat, 13 Jun 2026 18:20:56 -0500 Subject: [PATCH 07/45] finished off first pass plan implemnentation --- .../flowstate-collab/src/net/anti_entropy.rs | 5 + crates/flowstate-collab/src/net/mod.rs | 9 +- crates/flowstate-collab/src/net/runtime.rs | 16 +- crates/flowstate-collab/src/schema.rs | 4 + crates/flowstate-collab/tests/anti_entropy.rs | 115 ++++ crates/flowstate-collab/tests/convergence.rs | 158 ++++++ crates/flowstate-collab/tests/projection.rs | 65 ++- crates/flowstate-collab/tests/translation.rs | 327 +++++++++++ crates/flowstate/src/collab/asset_transfer.rs | 198 +++++++ crates/flowstate/src/collab/mod.rs | 17 +- crates/flowstate/src/collab/session.rs | 509 ++++++----------- crates/flowstate/src/collab/session_io.rs | 313 +++++++++++ .../flowstate/src/collab/session_presence.rs | 92 +++ crates/flowstate/src/collab/session_timers.rs | 272 +++++++++ crates/flowstate/src/collab/share_dialog.rs | 530 ++++++++++++++++++ crates/flowstate/src/collab/shutdown.rs | 23 + crates/flowstate/src/collab/status.rs | 38 +- crates/flowstate/src/commands.rs | 4 + crates/flowstate/src/commands/keymap.rs | 2 + crates/flowstate/src/rich_text_element/mod.rs | 2 + .../src/workspace/workspace/collab.rs | 85 ++- .../src/workspace/workspace/collab_prompts.rs | 324 +++++++++++ .../src/workspace/workspace/documents.rs | 7 + .../src/workspace/workspace/keybindings.rs | 16 +- .../flowstate/src/workspace/workspace/mod.rs | 2 + .../workspace/workspace/render_documents.rs | 4 + .../src/workspace/workspace/render_status.rs | 8 +- .../src/workspace/workspace/render_top_bar.rs | 6 + .../src/workspace/workspace/top_bar.rs | 50 +- .../src/workspace/workspace/traits.rs | 1 + .../src/workspace/workspace/window.rs | 84 +++ crates/gpui-flowtext/src/document/blocks.rs | 10 + .../src/rich_text/editor/caret_movement.rs | 3 + .../src/rich_text/editor/collab_apply.rs | 43 ++ .../src/rich_text/editor/lifecycle.rs | 8 +- .../src/rich_text/editor/object_assets.rs | 30 +- .../src/rich_text/editor/render_blocks.rs | 32 +- .../src/rich_text/layout/block_layout.rs | 11 + helpers/docs/collab_qa.md | 59 ++ 39 files changed, 3101 insertions(+), 381 deletions(-) create mode 100644 crates/flowstate-collab/tests/anti_entropy.rs create mode 100644 crates/flowstate-collab/tests/convergence.rs create mode 100644 crates/flowstate-collab/tests/translation.rs create mode 100644 crates/flowstate/src/collab/asset_transfer.rs create mode 100644 crates/flowstate/src/collab/session_io.rs create mode 100644 crates/flowstate/src/collab/session_presence.rs create mode 100644 crates/flowstate/src/collab/session_timers.rs create mode 100644 crates/flowstate/src/collab/share_dialog.rs create mode 100644 crates/flowstate/src/collab/shutdown.rs create mode 100644 crates/flowstate/src/workspace/workspace/collab_prompts.rs create mode 100644 helpers/docs/collab_qa.md diff --git a/crates/flowstate-collab/src/net/anti_entropy.rs b/crates/flowstate-collab/src/net/anti_entropy.rs index 0c915d9c..ecb80adf 100644 --- a/crates/flowstate-collab/src/net/anti_entropy.rs +++ b/crates/flowstate-collab/src/net/anti_entropy.rs @@ -49,6 +49,11 @@ impl AntiEntropyState { now >= self.next_digest } + #[must_use] + pub fn duration_until_digest(&self, now: Instant) -> Duration { + self.next_digest.saturating_duration_since(now) + } + pub fn mark_digest_sent(&mut self, now: Instant) { self.next_digest = next_jittered_deadline(now); } diff --git a/crates/flowstate-collab/src/net/mod.rs b/crates/flowstate-collab/src/net/mod.rs index 6f118a62..9cef1b23 100644 --- a/crates/flowstate-collab/src/net/mod.rs +++ b/crates/flowstate-collab/src/net/mod.rs @@ -13,6 +13,7 @@ use crate::{ids::{BlobId, SessionId}, proto_direct::AssetBytes, proto_gossip::Go use self::direct::DirectSessionHandler; pub type Reply = Sender; +pub type PeerAddr = EndpointAddr; #[derive(Clone, Debug)] pub enum PublishPayload { @@ -29,10 +30,10 @@ pub enum NetCommand { JoinSession { session: SessionId, bootstrap: Vec }, LeaveSession { session: SessionId }, Publish { session: SessionId, payload: PublishPayload }, - PullUpdates { session: SessionId, from: EndpointId, our_vv: Vec, reply: Reply>> }, - PullSnapshot { session: SessionId, from: EndpointId, reply: Reply>> }, - PullBlob { session: SessionId, from: EndpointId, blob: BlobId, reply: Reply>> }, - PullAsset { session: SessionId, from: EndpointId, asset: u128, reply: Reply> }, + PullUpdates { session: SessionId, candidates: Vec, our_vv: Vec, reply: Reply>> }, + PullSnapshot { session: SessionId, candidates: Vec, reply: Reply>> }, + PullBlob { session: SessionId, candidates: Vec, blob: BlobId, reply: Reply>> }, + PullAsset { session: SessionId, candidates: Vec, asset: u128, reply: Reply> }, MintTicketAddr { reply: Reply }, Shutdown, } diff --git a/crates/flowstate-collab/src/net/runtime.rs b/crates/flowstate-collab/src/net/runtime.rs index 56c6c36b..f09ca235 100644 --- a/crates/flowstate-collab/src/net/runtime.rs +++ b/crates/flowstate-collab/src/net/runtime.rs @@ -132,23 +132,23 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha }, NetCommand::PullUpdates { session, - from, + candidates, our_vv, reply, } => { - let result = direct::pull_with_fallback(DirectRequest::Updates { session, have_vv: our_vv }, vec![from], DIRECT_PULL_TIMEOUT).await; + let result = direct::pull_with_fallback(DirectRequest::Updates { session, have_vv: our_vv }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullSnapshot { session, from, reply } => { - let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, vec![from], DIRECT_PULL_TIMEOUT).await; + NetCommand::PullSnapshot { session, candidates, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullBlob { session, from, blob, reply } => { - let result = direct::pull_with_fallback(DirectRequest::Blob { session, blob }, vec![from], DIRECT_PULL_TIMEOUT).await; + NetCommand::PullBlob { session, candidates, blob, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Blob { session, blob }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullAsset { session, from, asset, reply } => { - let result = direct::pull_with_fallback(DirectRequest::Asset { session, asset }, vec![from], DIRECT_PULL_TIMEOUT) + NetCommand::PullAsset { session, candidates, asset, reply } => { + let result = direct::pull_with_fallback(DirectRequest::Asset { session, asset }, candidates, DIRECT_PULL_TIMEOUT) .await .map(|bytes| AssetBytes { bytes }); let _ = reply.send(result).await; diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs index ac1e861d..925e1290 100644 --- a/crates/flowstate-collab/src/schema.rs +++ b/crates/flowstate-collab/src/schema.rs @@ -59,6 +59,10 @@ pub enum BlockPayload { Table(InputTableBlock), } +pub fn decode_block_payload(bytes: &[u8]) -> Result { + Ok(postcard::from_bytes(bytes)?) +} + #[must_use] pub fn payload_from_block(block: &Block, assets: &AssetStore) -> Option { match input_block_from_block(block) { diff --git a/crates/flowstate-collab/tests/anti_entropy.rs b/crates/flowstate-collab/tests/anti_entropy.rs new file mode 100644 index 00000000..932b678d --- /dev/null +++ b/crates/flowstate-collab/tests/anti_entropy.rs @@ -0,0 +1,115 @@ +#[cfg(test)] +mod tests { + use std::time::{Duration, Instant}; + + use flowstate_collab::{ + SessionId, + net::anti_entropy::{AntiEntropyState, GapAction, VersionVectorRelation}, + }; + use iroh::{EndpointId, SecretKey}; + + #[test] + fn digest_gap_starts_one_pull_until_the_pull_finishes() { + let session = SessionId::from_bytes([1; 32]); + let peer = peer(9); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::SenderHasMissingOps, vec![1, 2, 3]), + GapAction::Pull { + from: peer, + our_vv: vec![1, 2, 3], + } + ); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::Concurrent, vec![4, 5]), + GapAction::None + ); + + state.finish_pull(); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::Concurrent, vec![6]), + GapAction::Pull { + from: peer, + our_vv: vec![6], + } + ); + } + + #[test] + fn equal_or_sender_behind_digest_does_not_pull() { + let session = SessionId::from_bytes([2; 32]); + let peer = peer(10); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::Equal, vec![1]), + GapAction::None + ); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::WeHaveMissingOps, vec![2]), + GapAction::None + ); + } + + #[test] + fn lineage_mismatch_is_reported_without_starting_a_pull() { + let session = SessionId::from_bytes([3; 32]); + let other = SessionId::from_bytes([4; 32]); + let peer = peer(11); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!( + state.consider_digest(peer, other, VersionVectorRelation::SenderHasMissingOps, vec![1]), + GapAction::LineageMismatch { + from: peer, + expected: session, + got: other, + } + ); + assert_eq!( + state.consider_digest(peer, session, VersionVectorRelation::SenderHasMissingOps, vec![2]), + GapAction::Pull { + from: peer, + our_vv: vec![2], + } + ); + } + + #[test] + fn lagged_uses_the_fallback_peer_and_deduplicates_in_flight_pulls() { + let session = SessionId::from_bytes([5; 32]); + let peer = peer(12); + let mut state = AntiEntropyState::new(session, Instant::now()); + + assert_eq!(state.on_lagged(None, vec![1]), GapAction::None); + assert_eq!( + state.on_lagged(Some(peer), vec![2]), + GapAction::Pull { + from: peer, + our_vv: vec![2], + } + ); + assert_eq!(state.on_lagged(Some(peer), vec![3]), GapAction::None); + } + + #[test] + fn neighbor_and_reconnect_events_schedule_an_immediate_digest() { + let session = SessionId::from_bytes([6; 32]); + let now = Instant::now(); + let mut state = AntiEntropyState::new(session, now); + + assert!(!state.digest_due(now + Duration::from_secs(1))); + state.on_neighbor_up(); + assert!(state.digest_due(Instant::now() + Duration::from_millis(1))); + + state.mark_digest_sent(Instant::now()); + assert!(!state.digest_due(Instant::now() + Duration::from_secs(1))); + state.on_reconnect(); + assert!(state.digest_due(Instant::now() + Duration::from_millis(1))); + } + + fn peer(seed: u8) -> EndpointId { + SecretKey::from_bytes(&[seed; 32]).public() + } +} diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs new file mode 100644 index 00000000..3e6cf790 --- /dev/null +++ b/crates/flowstate-collab/tests/convergence.rs @@ -0,0 +1,158 @@ +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; + use gpui_flowtext::{ + CanonicalOperation, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyle, RunStyles, document_from_input_blocks, + insert_text_at, mutate_runs_in_range, paragraph_text, plain, + }; + use loro::{ExportMode, LoroDoc, Subscription}; + + const MULTIBYTE: &str = "aé🌍\u{2028}x"; + + #[test] + fn two_peers_converge_with_reordered_and_duplicated_update_imports() { + let session = SessionId::from_bytes([77; 32]); + let initial = document_from_input_blocks( + DocumentTheme::default(), + vec![ + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(MULTIBYTE)], + }), + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain("second aé🌍")], + }), + ], + ); + let mut peer_a = Peer::from_initial(session, initial); + let mut peer_b = Peer::from_snapshot(session, &peer_a.snapshot()); + + peer_a.insert_text(0, "a".len(), "A🌍", RunStyles::default().with(RunStyle::Semantic(2))); + let inserted = "A🌍".len(); + peer_a.set_run_styles( + 0, + "a".len().."a".len() + inserted, + RunStyles::default().with(RunStyle::Highlight(4)).with_direct_underline(), + ); + peer_b.insert_text(1, "second ".len(), "Bé", RunStyles::default().with_strikethrough()); + + let updates_a = peer_a.updates(); + let updates_b = peer_b.updates(); + assert_eq!(updates_a.len(), 2); + assert_eq!(updates_b.len(), 1); + + // Import A's second commit before its first commit to exercise Loro's pending-import path, + // then replay duplicates to assert idempotency. + import_sequence(&peer_b.loro, [&updates_a[1], &updates_a[1], &updates_a[0], &updates_a[0]]); + import_sequence(&peer_a.loro, [&updates_b[0], &updates_b[0]]); + + let projected_a = projection::document_from_loro(&peer_a.loro, DocumentTheme::default()).expect("projection A should succeed"); + let projected_b = projection::document_from_loro(&peer_b.loro, DocumentTheme::default()).expect("projection B should succeed"); + assert_documents_eq(&projected_a, &projected_b); + } + + struct Peer { + document: Document, + loro: LoroDoc, + binding: DocBinding, + updates: Arc>>>, + _subscription: Subscription, + } + + impl Peer { + fn from_initial(session: SessionId, document: Document) -> Self { + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, session, "convergence", &document).expect("populate should succeed"); + let binding = DocBinding::build(&loro, &document).expect("binding should build"); + Self::with_subscription(document, loro, binding) + } + + fn from_snapshot(session: SessionId, snapshot: &[u8]) -> Self { + let loro = schema::new_configured_doc(); + loro.import(snapshot).expect("snapshot import should succeed"); + projection::verify_lineage(&loro, session).expect("lineage should match"); + let document = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); + let binding = DocBinding::build(&loro, &document).expect("binding should build"); + Self::with_subscription(document, loro, binding) + } + + fn with_subscription(document: Document, loro: LoroDoc, binding: DocBinding) -> Self { + let updates = Arc::new(Mutex::new(Vec::new())); + let captured = updates.clone(); + let subscription = loro.subscribe_local_update(Box::new(move |bytes| { + captured.lock().expect("update capture lock should not be poisoned").push(bytes.clone()); + true + })); + Self { + document, + loro, + binding, + updates, + _subscription: subscription, + } + } + + fn snapshot(&self) -> Vec { + self.loro.export(ExportMode::Snapshot).expect("snapshot export should succeed") + } + + fn updates(&self) -> Vec> { + self.updates.lock().expect("update capture lock should not be poisoned").clone() + } + + fn insert_text(&mut self, paragraph_ix: usize, byte: usize, text: &str, styles: RunStyles) { + let paragraph = self.document.ids.paragraph_ids[paragraph_ix]; + insert_text_at(&mut self.document, paragraph_ix, byte, text, styles); + self.apply(&[CanonicalOperation::InsertText { + paragraph, + byte, + text: text.to_string(), + styles, + }]); + } + + fn set_run_styles(&mut self, paragraph_ix: usize, range: std::ops::Range, styles: RunStyles) { + let paragraph = self.document.ids.paragraph_ids[paragraph_ix]; + mutate_runs_in_range( + &mut self.document, + gpui_flowtext::DocumentOffset { + paragraph: paragraph_ix, + byte: range.start, + }..gpui_flowtext::DocumentOffset { + paragraph: paragraph_ix, + byte: range.end, + }, + |run_styles| *run_styles = styles, + ); + self.apply(&[CanonicalOperation::SetRunStyles { paragraph, range, styles }]); + } + + fn apply(&mut self, ops: &[CanonicalOperation]) { + LocalApplier { + doc: &self.loro, + binding: &mut self.binding, + } + .apply(&self.document, ops) + .expect("local apply should succeed"); + } + } + + fn import_sequence<'a>(doc: &LoroDoc, updates: impl IntoIterator>) { + for update in updates { + doc.import(update).expect("update import should succeed"); + } + } + + fn assert_documents_eq(left: &Document, right: &Document) { + assert_eq!(left.blocks.len(), right.blocks.len()); + assert_eq!(left.paragraphs.len(), right.paragraphs.len()); + for ix in 0..left.paragraphs.len() { + assert_eq!(paragraph_text(left, ix), paragraph_text(right, ix)); + assert_eq!(left.paragraphs[ix].style, right.paragraphs[ix].style); + assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); + } + } +} diff --git a/crates/flowstate-collab/tests/projection.rs b/crates/flowstate-collab/tests/projection.rs index dc34aa51..4acee34c 100644 --- a/crates/flowstate-collab/tests/projection.rs +++ b/crates/flowstate-collab/tests/projection.rs @@ -3,10 +3,13 @@ mod tests { use flowstate_collab::binding::DocBinding; use flowstate_collab::{SessionId, projection, schema}; use gpui_flowtext::{ - Document, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, - RunStyles, document_from_input_blocks, paragraph_text, plain, run, + Block, Document, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, InputTableBlock, + InputTableCell, InputTableCellBlock, InputTableColumnWidth, InputTableRow, InputTableStyle, ParagraphStyle, RunStyle, RunStyles, + document_from_input_blocks, paragraph_text, plain, run, }; + const MULTIBYTE: &str = "aé🌍\u{2028}x"; + #[test] fn projection_round_trips_text_marks_and_equation_blocks() { let input_blocks = vec![ @@ -22,6 +25,20 @@ mod tests { syntax: InputEquationSyntax::Latex, display: InputEquationDisplay::Display, }), + InputBlock::Table(InputTableBlock { + rows: vec![InputTableRow { + cells: vec![InputTableCell { + blocks: vec![InputTableCellBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(MULTIBYTE)], + })], + row_span: 1, + col_span: 1, + }], + }], + column_widths: vec![InputTableColumnWidth::Auto], + style: InputTableStyle { header_row: true }, + }), ]; let original = document_from_input_blocks(DocumentTheme::default(), input_blocks); let loro = schema::new_configured_doc(); @@ -36,6 +53,28 @@ mod tests { assert_eq!(binding.rows.len(), projected.blocks.len()); } + #[test] + fn offset_helpers_round_trip_all_multibyte_boundaries() { + let original = document_from_input_blocks( + DocumentTheme::default(), + vec![InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(MULTIBYTE)], + })], + ); + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, SessionId::from_bytes([8; 32]), "offsets", &original).expect("populate should succeed"); + let binding = DocBinding::build(&loro, &original).expect("binding should build"); + let text = binding.rows[0].text.as_ref().expect("first row should be paragraph text"); + let byte_to_unicode = [(0, 0), ("a".len(), 1), ("aé".len(), 2), ("aé🌍".len(), 3), ("aé🌍\u{2028}".len(), 4), (MULTIBYTE.len(), 5)]; + + assert_eq!(text.len_utf8(), MULTIBYTE.len()); + for (byte, unicode) in byte_to_unicode { + assert_eq!(schema::loro_pos(text, byte), unicode); + assert_eq!(schema::utf8_byte(text, unicode), byte); + } + } + fn assert_document_shape_eq(left: &Document, right: &Document) { assert_eq!(left.blocks.len(), right.blocks.len()); assert_eq!(left.paragraphs.len(), right.paragraphs.len()); @@ -45,7 +84,27 @@ mod tests { assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); } for (left, right) in left.blocks.iter().zip(right.blocks.iter()) { - assert_eq!(std::mem::discriminant(left), std::mem::discriminant(right)); + match (left, right) { + (Block::Paragraph(_), Block::Paragraph(_)) => {}, + (Block::Equation(left), Block::Equation(right)) => { + assert_eq!(left.source, right.source); + assert_eq!(left.syntax, right.syntax); + assert_eq!(left.display, right.display); + }, + (Block::Table(left), Block::Table(right)) => { + assert_eq!(left.rows, right.rows); + assert_eq!(left.column_widths, right.column_widths); + assert_eq!(left.style, right.style); + }, + (Block::Image(left), Block::Image(right)) => { + assert_eq!(left.asset_id, right.asset_id); + assert_eq!(left.alt_text, right.alt_text); + assert_eq!(left.caption, right.caption); + assert_eq!(left.sizing, right.sizing); + assert_eq!(left.alignment, right.alignment); + }, + _ => panic!("projected block kind changed"), + } } } } diff --git a/crates/flowstate-collab/tests/translation.rs b/crates/flowstate-collab/tests/translation.rs new file mode 100644 index 00000000..358dbc26 --- /dev/null +++ b/crates/flowstate-collab/tests/translation.rs @@ -0,0 +1,327 @@ +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; + use gpui_flowtext::{ + Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, + InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, RunStyles, block_from_input_block, capture_document_span, + delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, + mutate_runs_in_range, paragraph_text, paragraphs_mut, plain, remove_block_ids, run, split_paragraph_at, update_paragraph_block, + }; + use loro::LoroDoc; + + const MULTIBYTE: &str = "aé🌍\u{2028}x"; + + #[test] + fn delete_range_uses_utf8_byte_offsets() { + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain(MULTIBYTE)])]); + let paragraph = document.ids.paragraph_ids[0]; + let start = "a".len(); + let end = "aé🌍".len(); + + delete_range_in_paragraph(&mut document, 0, start..end); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::DeleteRange { + start_paragraph: paragraph, + start_byte: start, + end_paragraph: paragraph, + end_byte: end, + }], + ); + } + + #[test] + fn split_and_join_paragraphs_preserve_multibyte_text_and_styles() { + let semantic = RunStyles::default().with(RunStyle::Semantic(2)).with_direct_underline(); + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("aé"), run("🌍\u{2028}x", semantic)])]); + let original = document.ids.paragraph_ids[0]; + let split_byte = "aé".len(); + + split_paragraph_at(&mut document, 0, split_byte); + let split = document.ids.paragraph_ids[1]; + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SplitParagraph { + paragraph: original, + byte: split_byte, + new_paragraph: split, + }], + ); + + let first_len = paragraph_text(&document, 0).len(); + delete_cross_paragraph_range( + &mut document, + DocumentOffset { + paragraph: 0, + byte: first_len, + }..DocumentOffset { paragraph: 1, byte: 0 }, + ); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::JoinParagraphs { + first: original, + second: split, + }], + ); + } + + #[test] + fn paragraph_and_run_style_updates_match_flowtext_runs() { + let base_style = RunStyles::default().with(RunStyle::Semantic(1)).with_strikethrough(); + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![run(MULTIBYTE, base_style)])]); + let paragraph = document.ids.paragraph_ids[0]; + + paragraphs_mut(&mut document)[0].style = ParagraphStyle::Custom(6); + update_paragraph_block(&mut document, 0); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SetParagraphStyle { + paragraph, + style: ParagraphStyle::Custom(6), + }], + ); + + let unstyled = "a".len().."aé".len(); + mutate_runs_in_range( + &mut document, + DocumentOffset { + paragraph: 0, + byte: unstyled.start, + }..DocumentOffset { + paragraph: 0, + byte: unstyled.end, + }, + |styles| *styles = RunStyles::default(), + ); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SetRunStyles { + paragraph, + range: unstyled, + styles: RunStyles::default(), + }], + ); + + let highlighted = "aé".len().."aé🌍".len(); + let highlighted_styles = RunStyles::default() + .with(RunStyle::Semantic(4)) + .with(RunStyle::Highlight(3)) + .with_direct_underline() + .with_strikethrough(); + mutate_runs_in_range( + &mut document, + DocumentOffset { + paragraph: 0, + byte: highlighted.start, + }..DocumentOffset { + paragraph: 0, + byte: highlighted.end, + }, + |styles| *styles = highlighted_styles, + ); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::SetRunStyles { + paragraph, + range: highlighted, + styles: highlighted_styles, + }], + ); + } + + #[test] + fn replace_paragraph_span_refreshes_text_runs_and_styles() { + let (mut document, loro, mut binding) = collab_fixture(vec![ + paragraph_block(vec![plain(MULTIBYTE)]), + paragraph_block(vec![plain("second aé")]), + ]); + let start_paragraph = document.ids.paragraph_ids[0]; + let before = capture_document_span(&document, 0..2); + let inserted_styles = RunStyles::default().with(RunStyle::Semantic(5)).with_direct_underline(); + + insert_text_at(&mut document, 0, "a".len(), "Zé", inserted_styles); + paragraphs_mut(&mut document)[1].style = ParagraphStyle::Custom(2); + update_paragraph_block(&mut document, 1); + let after = capture_document_span(&document, 0..2); + + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::ReplaceParagraphSpan { + start_paragraph: Some(start_paragraph), + before, + after, + }], + ); + } + + #[test] + fn insert_move_replace_and_delete_object_blocks_match_flowtext_document() { + let (mut document, loro, mut binding) = collab_fixture(vec![ + paragraph_block(vec![plain("one aé")]), + paragraph_block(vec![plain("two 🌍")]), + ]); + let equation = equation_block("x = aé + 🌍"); + + let equation_id = insert_object_block(&mut document, 1, equation); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::InsertBlock { + block: equation_id, + block_ix: 1, + }], + ); + + move_block(&mut document, 1, 2); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::MoveBlock { + block: equation_id, + new_block_ix: 2, + }], + ); + + replace_object_block(&mut document, 2, equation_block("y = 🌍^2")); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::ReplaceBlock { block: Some(equation_id) }], + ); + + delete_object_block(&mut document, 2); + apply_and_assert_projection( + &document, + &loro, + &mut binding, + &[CanonicalOperation::DeleteBlock { block: equation_id }], + ); + } + + #[test] + fn replace_document_rebuilds_the_loro_projection() { + let (_, loro, mut binding) = collab_fixture(vec![ + paragraph_block(vec![plain("old")]), + equation_block("old = 1"), + ]); + let document = document_from_input_blocks( + DocumentTheme::default(), + vec![ + paragraph_block(vec![plain("new aé"), run("🌍", RunStyles::default().with(RunStyle::Highlight(1)))]), + equation_block("new = aé + 🌍"), + paragraph_block(vec![plain("tail")]), + ], + ); + + apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::ReplaceDocument]); + } + + fn collab_fixture(blocks: Vec) -> (Document, LoroDoc, DocBinding) { + let document = document_from_input_blocks(DocumentTheme::default(), blocks); + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, SessionId::from_bytes([42; 32]), "translation", &document) + .expect("populate should succeed"); + let binding = DocBinding::build(&loro, &document).expect("binding should build"); + (document, loro, binding) + } + + fn apply_and_assert_projection(document: &Document, loro: &LoroDoc, binding: &mut DocBinding, ops: &[CanonicalOperation]) { + LocalApplier { doc: loro, binding } + .apply(document, ops) + .expect("local apply should succeed"); + let projected = projection::document_from_loro(loro, DocumentTheme::default()).expect("projection should succeed"); + assert_document_projection_eq(document, &projected); + } + + fn paragraph_block(runs: Vec) -> InputBlock { + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs, + }) + } + + fn equation_block(source: &str) -> InputBlock { + InputBlock::Equation(InputEquationBlock { + source: source.to_string(), + syntax: InputEquationSyntax::Latex, + display: InputEquationDisplay::Display, + }) + } + + fn insert_object_block(document: &mut Document, block_ix: usize, input: InputBlock) -> gpui_flowtext::BlockId { + Arc::make_mut(&mut document.blocks).insert(block_ix, block_from_input_block(&input)); + insert_block_id(document, block_ix) + } + + fn move_block(document: &mut Document, from: usize, to: usize) { + let blocks = Arc::make_mut(&mut document.blocks); + let block = blocks.remove(from); + blocks.insert(to.min(blocks.len()), block); + let block_id = document.ids.block_ids.remove(from); + document.ids.block_ids.insert(to.min(document.ids.block_ids.len()), block_id); + } + + fn replace_object_block(document: &mut Document, block_ix: usize, input: InputBlock) { + Arc::make_mut(&mut document.blocks)[block_ix] = block_from_input_block(&input); + } + + fn delete_object_block(document: &mut Document, block_ix: usize) { + Arc::make_mut(&mut document.blocks).remove(block_ix); + remove_block_ids(document, block_ix..block_ix + 1); + } + + fn assert_document_projection_eq(expected: &Document, actual: &Document) { + assert_eq!(expected.blocks.len(), actual.blocks.len()); + assert_eq!(expected.paragraphs.len(), actual.paragraphs.len()); + let mut paragraph_ix = 0; + for (expected_block, actual_block) in expected.blocks.iter().zip(actual.blocks.iter()) { + match (expected_block, actual_block) { + (Block::Paragraph(expected_paragraph), Block::Paragraph(actual_paragraph)) => { + assert_eq!(paragraph_text(expected, paragraph_ix), paragraph_text(actual, paragraph_ix)); + assert_eq!(expected_paragraph.style, actual_paragraph.style); + assert_eq!(expected_paragraph.runs, actual_paragraph.runs); + }, + (Block::Equation(expected), Block::Equation(actual)) => { + assert_eq!(expected.source, actual.source); + assert_eq!(expected.syntax, actual.syntax); + assert_eq!(expected.display, actual.display); + }, + (Block::Image(expected), Block::Image(actual)) => { + assert_eq!(expected.asset_id, actual.asset_id); + assert_eq!(expected.alt_text, actual.alt_text); + assert_eq!(expected.caption, actual.caption); + assert_eq!(expected.sizing, actual.sizing); + assert_eq!(expected.alignment, actual.alignment); + }, + (Block::Table(expected), Block::Table(actual)) => { + assert_eq!(expected.rows, actual.rows); + assert_eq!(expected.column_widths, actual.column_widths); + assert_eq!(expected.style, actual.style); + }, + _ => panic!("projected block kind changed"), + } + if matches!(expected_block, Block::Paragraph(_)) { + paragraph_ix += 1; + } + } + } +} diff --git a/crates/flowstate/src/collab/asset_transfer.rs b/crates/flowstate/src/collab/asset_transfer.rs new file mode 100644 index 00000000..fd75720b --- /dev/null +++ b/crates/flowstate/src/collab/asset_transfer.rs @@ -0,0 +1,198 @@ +use std::{collections::hash_map::DefaultHasher, hash::{Hash, Hasher}, sync::Arc}; + +use anyhow::{Context as _, Result, anyhow, ensure}; +use flowstate_collab::{ + net::NetCommand, + schema::{BLOCKS, BlockPayload, DATA, KIND, KIND_IMAGE, decode_block_payload}, +}; +use gpui::{Context, SharedString}; +use loro::{Container, LoroValue, ValueOrContainer}; + +use crate::rich_text_element::{AssetId, AssetRecord, CollabPatch}; + +use super::CollabSession; + +pub(super) fn schedule_missing_assets( + session: &mut CollabSession, + preferred_peer: Option, + cx: &mut Context, +) { + let Some(editor) = session.editor.clone() else { + return; + }; + let Some(doc) = &session.doc else { + return; + }; + + let assets = match image_assets_in_loro(doc) { + Ok(assets) => assets, + Err(error) => { + eprintln!("flowstate collab asset scan failed: {error:#}"); + return; + }, + }; + if assets.is_empty() { + return; + } + + let missing_assets = { + let editor = editor.read(cx); + assets + .iter() + .filter(|meta| { + let id = AssetId(meta.asset_id); + editor + .document() + .assets + .assets + .get(&id) + .is_none_or(AssetRecord::is_loading_placeholder) + }) + .cloned() + .collect::>() + }; + if missing_assets.is_empty() { + return; + } + + let placeholders = missing_assets + .iter() + .map(|meta| CollabPatch::AssetArrived { + id: AssetId(meta.asset_id), + record: meta.placeholder_record(), + }) + .collect::>(); + session.apply_or_queue_patches(placeholders, cx); + + let candidates = session.pull_candidates(preferred_peer); + if candidates.is_empty() { + return; + } + + for meta in missing_assets { + let id = AssetId(meta.asset_id); + if !session.asset_pulls_in_flight.insert(id) { + continue; + } + start_asset_pull(session, candidates.clone(), meta, cx); + } +} + +fn start_asset_pull( + session: &mut CollabSession, + candidates: Vec, + meta: ImageAssetMeta, + cx: &mut Context, +) { + let (reply_tx, reply_rx) = async_channel::bounded(1); + let id = AssetId(meta.asset_id); + if session + .net_tx + .try_send(NetCommand::PullAsset { + session: session.session, + candidates, + asset: meta.asset_id, + reply: reply_tx, + }) + .is_err() + { + session.asset_pulls_in_flight.remove(&id); + return; + } + + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + session.asset_pulls_in_flight.remove(&id); + match result { + Ok(Ok(bytes)) => match meta.record_from_bytes(bytes.bytes) { + Ok(record) => { + session.apply_or_queue_patches(vec![CollabPatch::AssetArrived { id, record }], cx); + }, + Err(error) => eprintln!("flowstate collab rejected fetched asset {id:?}: {error:#}"), + }, + Ok(Err(error)) => eprintln!("flowstate collab asset pull failed for {id:?}: {error:#}"), + Err(error) => eprintln!("flowstate collab asset pull channel closed for {id:?}: {error}"), + } + }); + }) + .detach(); +} + +#[derive(Clone, Debug)] +struct ImageAssetMeta { + asset_id: u128, + mime: String, + original_name: Option, + content_hash: u64, + byte_len: u64, +} + +impl ImageAssetMeta { + fn placeholder_record(&self) -> AssetRecord { + AssetRecord { + id: AssetId(self.asset_id), + mime_type: SharedString::from(self.mime.clone()), + original_name: self.original_name.clone().map(SharedString::from), + content_hash: self.content_hash, + bytes: Arc::new(Vec::new()), + } + } + + fn record_from_bytes(&self, bytes: Vec) -> Result { + ensure!(bytes.len() as u64 == self.byte_len, "asset byte length mismatch"); + let mut hasher = DefaultHasher::new(); + bytes.hash(&mut hasher); + let content_hash = hasher.finish(); + ensure!(content_hash == self.content_hash, "asset content hash mismatch"); + Ok(AssetRecord { + id: AssetId(self.asset_id), + mime_type: SharedString::from(self.mime.clone()), + original_name: self.original_name.clone().map(SharedString::from), + content_hash, + bytes: Arc::new(bytes), + }) + } +} + +fn image_assets_in_loro(doc: &loro::LoroDoc) -> Result> { + let blocks = doc.get_movable_list(BLOCKS); + let mut assets = Vec::new(); + for ix in 0..blocks.len() { + let Some(ValueOrContainer::Container(Container::Map(map))) = blocks.get(ix) else { + continue; + }; + let Some(ValueOrContainer::Value(LoroValue::String(kind))) = map.get(KIND) else { + continue; + }; + if kind.as_ref() != KIND_IMAGE { + continue; + } + let Some(ValueOrContainer::Value(LoroValue::Binary(data))) = map.get(DATA) else { + return Err(anyhow!("image block {ix} is missing asset metadata")); + }; + let payload = decode_block_payload(&data).context("decoding image asset metadata failed")?; + let BlockPayload::Image { + asset_id, + mime, + original_name, + content_hash, + byte_len, + .. + } = payload + else { + continue; + }; + if byte_len == 0 { + continue; + } + assets.push(ImageAssetMeta { + asset_id, + mime, + original_name, + content_hash, + byte_len, + }); + } + Ok(assets) +} diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index 3bff0775..c041cb16 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -1,5 +1,7 @@ mod session; mod presence_view; +pub mod share_dialog; +mod shutdown; pub mod status; use std::collections::HashMap; @@ -16,7 +18,8 @@ use uuid::Uuid; use crate::rich_text_element::RichTextEditor; -pub use session::{Attachment, CollabSession, Connectivity, DetachReason, JoinStage, JoinedDocument, SessionPhase}; +pub use session::{Attachment, CollabSession, Connectivity, DetachReason, JoinStage, JoinedDocument, SessionPhase, SessionRosterEntry}; +pub use shutdown::shutdown; pub struct JoinRequest { pub session: SessionId, @@ -92,7 +95,7 @@ impl CollabManager { let commands = self.ensure_runtime(cx)?; let session = ticket.session; let inviter = ticket.inviter.id; - let collab = CollabSession::joining(session, ticket.title.clone(), commands.clone()); + let collab = CollabSession::joining(session, ticket.title.clone(), commands.clone(), vec![ticket.inviter.clone()]); let entity = cx.new(|_| collab); self.register_session(entity.clone(), cx); @@ -196,6 +199,12 @@ impl CollabManager { .map(|session| session.read(cx).phase().clone()) } + pub fn roster_for_panel(&self, panel_id: Uuid, cx: &App) -> Vec { + self + .session_for_panel(panel_id) + .map_or_else(Vec::new, |session| session.read(cx).roster()) + } + fn register_session(&mut self, session: Entity, cx: &mut Context) { let id = session.read(cx).session_id(); if let Some(panel_id) = session.read(cx).panel_id() { @@ -394,3 +403,7 @@ where pub fn phase_for_panel(panel_id: Uuid, cx: &App) -> Option { CollabManager::global(cx).phase_for_panel(panel_id, cx) } + +pub fn roster_for_panel(panel_id: Uuid, cx: &App) -> Vec { + CollabManager::global(cx).roster_for_panel(panel_id, cx) +} diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index 9a9446ef..92443787 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -1,8 +1,6 @@ use std::{ - cmp::Ordering, collections::HashSet, rc::Rc, - sync::{Arc, Mutex}, time::Instant, }; @@ -11,16 +9,14 @@ use flowstate_collab::{ SessionId, binding::DocBinding, local_apply::LocalApplier, - net::{NetCommand, PublishPayload, runtime::CommandSender, direct::{DirectServeRequest, DirectSessionHandler}}, - presence::{PresenceSelection, PresenceState, PresenceStore}, + net::{NetCommand, PeerAddr, PublishPayload, anti_entropy::AntiEntropyState, direct::{DirectServeRequest, DirectSessionHandler}, runtime::CommandSender}, + presence::PresenceStore, projection, - proto_direct::AssetBytes, proto_gossip::GossipMsg, - remote_apply::RemoteApplier, schema, }; use gpui::{Context, Entity, Subscription}; -use loro::{ExportMode, LoroDoc, Subscription as LoroSubscription, VersionVector, event::Subscriber}; +use loro::{LoroDoc, Subscription as LoroSubscription, UndoManager}; use uuid::Uuid; use crate::app_settings::load_document_theme; @@ -28,6 +24,15 @@ use crate::rich_text_element::{AssetId, CollabPatch, Document, EditorEvent, Rich use super::presence_view; +#[path = "asset_transfer.rs"] +mod asset_transfer; +#[path = "session_io.rs"] +mod session_io; +#[path = "session_presence.rs"] +mod session_presence; +#[path = "session_timers.rs"] +mod session_timers; + #[derive(Clone, Debug)] pub enum SessionPhase { Creating, @@ -69,6 +74,13 @@ pub struct JoinedDocument { pub document: Document, } +#[derive(Clone, Debug)] +pub struct SessionRosterEntry { + pub name: String, + pub color_rgb: u32, + pub is_self: bool, +} + pub struct CollabSession { session: SessionId, title: String, @@ -83,12 +95,25 @@ pub struct CollabSession { net_tx: CommandSender, direct_tx: async_channel::Sender, direct_rx: async_channel::Receiver, + undo_tx: async_channel::Sender, + undo_rx: async_channel::Receiver, editor_subscriptions: Vec, loro_subscriptions: Vec, neighbors: HashSet, - update_pull_in_flight: bool, + bootstrap_addrs: Vec, + asset_pulls_in_flight: HashSet, + anti_entropy: AntiEntropyState, + undo_manager: Option, direct_pump_started: bool, + undo_pump_started: bool, local_update_publish_attached: bool, + timers_started: bool, + endpoint_online: bool, + zero_neighbors_since: Option, + inbound_since_last_digest: bool, + quiet_digest_rounds: u8, + next_recovery_at: Option, + last_document_activity: Instant, } impl CollabSession { @@ -104,6 +129,8 @@ impl CollabSession { projection::populate_from_document(&doc, session, &title, &document)?; let binding = DocBinding::build(&doc, &document)?; let (direct_tx, direct_rx) = async_channel::unbounded(); + let (undo_tx, undo_rx) = async_channel::unbounded(); + let now = Instant::now(); Ok(Self { session, @@ -119,17 +146,32 @@ impl CollabSession { net_tx, direct_tx, direct_rx, + undo_tx, + undo_rx, editor_subscriptions: Vec::new(), loro_subscriptions: Vec::new(), neighbors: HashSet::new(), - update_pull_in_flight: false, + bootstrap_addrs: Vec::new(), + asset_pulls_in_flight: HashSet::new(), + anti_entropy: AntiEntropyState::new(session, now), + undo_manager: None, direct_pump_started: false, + undo_pump_started: false, local_update_publish_attached: false, + timers_started: false, + endpoint_online: true, + zero_neighbors_since: Some(now), + inbound_since_last_digest: false, + quiet_digest_rounds: 0, + next_recovery_at: None, + last_document_activity: now, }) } - pub fn joining(session: SessionId, title: String, net_tx: CommandSender) -> Self { + pub fn joining(session: SessionId, title: String, net_tx: CommandSender, bootstrap_addrs: Vec) -> Self { let (direct_tx, direct_rx) = async_channel::unbounded(); + let (undo_tx, undo_rx) = async_channel::unbounded(); + let now = Instant::now(); Self { session, title, @@ -144,12 +186,25 @@ impl CollabSession { net_tx, direct_tx, direct_rx, + undo_tx, + undo_rx, editor_subscriptions: Vec::new(), loro_subscriptions: Vec::new(), neighbors: HashSet::new(), - update_pull_in_flight: false, + bootstrap_addrs, + asset_pulls_in_flight: HashSet::new(), + anti_entropy: AntiEntropyState::new(session, now), + undo_manager: None, direct_pump_started: false, + undo_pump_started: false, local_update_publish_attached: false, + timers_started: false, + endpoint_online: true, + zero_neighbors_since: Some(now), + inbound_since_last_digest: false, + quiet_digest_rounds: 0, + next_recovery_at: None, + last_document_activity: now, } } @@ -169,10 +224,35 @@ impl CollabSession { &self.phase } + pub fn roster(&self) -> Vec { + let Some(presence) = &self.presence else { + return Vec::new(); + }; + let self_key = presence.self_key().to_string(); + presence + .roster() + .into_iter() + .map(|entry| SessionRosterEntry { + is_self: entry.key == self_key, + name: entry.name, + color_rgb: entry.color_rgb, + }) + .collect() + } + pub fn direct_handler(&self) -> DirectSessionHandler { DirectSessionHandler::new(self.direct_tx.clone()) } + pub(super) fn pull_candidates(&self, preferred: Option) -> Vec { + let mut candidates = Vec::with_capacity(self.neighbors.len() + usize::from(preferred.is_some())); + if let Some(peer) = preferred { + candidates.push(peer); + } + candidates.extend(self.neighbors.iter().copied().filter(|peer| Some(*peer) != preferred)); + candidates + } + pub fn set_join_stage(&mut self, stage: JoinStage, cx: &mut Context) { self.phase = SessionPhase::Joining(stage); cx.notify(); @@ -190,7 +270,7 @@ impl CollabSession { let (reply_tx, reply_rx) = async_channel::bounded(1); if let Err(error) = self.net_tx.try_send(NetCommand::PullSnapshot { session: self.session, - from: inviter, + candidates: self.pull_candidates(Some(inviter)), reply: reply_tx, }) { let detail = format!("requesting collaboration snapshot failed: {error}"); @@ -252,15 +332,19 @@ impl CollabSession { self.import_update_bytes(&update, cx)?; } self.flush_pending_remote_patches(cx); + asset_transfer::schedule_missing_assets(self, None, cx); self.publish_digest(); cx.notify(); Ok(()) } pub fn attach(&mut self, cx: &mut Context) { + self.attach_undo_manager(); self.attach_editor_hooks(cx); self.attach_loro_publish_hook(); self.attach_direct_request_pump(cx); + self.attach_undo_request_pump(cx); + self.attach_timers(cx); } pub fn establish_local_peer(&mut self, peer: &flowstate_collab::ids::PeerId, cx: &mut Context) { @@ -278,12 +362,14 @@ impl CollabSession { self.presence = Some(presence); } self.refresh_own_presence(cx); + self.endpoint_online = true; self.phase = SessionPhase::Attached(Attachment { connectivity: Connectivity::Online, peers_present: self.peers_present(), }); self.publish_presence_snapshot(); self.publish_digest(); + asset_transfer::schedule_missing_assets(self, None, cx); cx.notify(); } @@ -324,6 +410,7 @@ impl CollabSession { editor.update(cx, |editor, cx| { editor.set_collab_undo_redirect(None); editor.set_collab_capture(false); + editor.clear_collab_history(); let _ = editor.take_pending_collab_edits(); editor.set_external_carets(Vec::new(), cx); }); @@ -331,12 +418,18 @@ impl CollabSession { self.editor_subscriptions.clear(); self.loro_subscriptions.clear(); + self.undo_manager = None; self.presence = None; self.binding = None; self.doc = None; self.pending_remote_patches.clear(); self.pending_remote_updates.clear(); self.neighbors.clear(); + self.asset_pulls_in_flight.clear(); + self.zero_neighbors_since = Some(Instant::now()); + self.inbound_since_last_digest = false; + self.quiet_digest_rounds = 0; + self.next_recovery_at = None; self.local_update_publish_attached = false; self.phase = SessionPhase::Detached(reason); cx.notify(); @@ -349,7 +442,11 @@ impl CollabSession { } let document = editor.read(cx).document().clone(); - let edits = editor.update(cx, |editor, _| editor.take_pending_collab_edits()); + let edits = editor.update(cx, |editor, _| { + let edits = editor.take_pending_collab_edits(); + editor.clear_collab_history(); + edits + }); let Some(doc) = &self.doc else { return Ok(()); }; @@ -357,18 +454,26 @@ impl CollabSession { return Ok(()); }; + let mut applied = false; for edit in edits { if edit.operations.is_empty() { continue; } LocalApplier { doc, binding }.apply(&document, &edit.operations)?; + applied = true; + } + if applied { + self.last_document_activity = Instant::now(); } Ok(()) } pub fn handle_gossip(&mut self, from: flowstate_collab::ids::PeerId, msg: GossipMsg, cx: &mut Context) { + self.note_inbound_traffic(cx); let result = match msg { - GossipMsg::Update(bytes) => self.import_update_bytes(&bytes, cx), + GossipMsg::Update(bytes) => self + .import_update_bytes(&bytes, cx) + .map(|()| asset_transfer::schedule_missing_assets(self, Some(from), cx)), GossipMsg::UpdateAvailable { blob, .. } => { self.pull_blob(from, blob, cx); Ok(()) @@ -386,58 +491,41 @@ impl CollabSession { pub fn neighbor_up(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { self.neighbors.insert(peer); - if let SessionPhase::Attached(attachment) = &mut self.phase { - attachment.connectivity = Connectivity::Online; - } + self.zero_neighbors_since = None; + self.anti_entropy.on_neighbor_up(); + self.mark_online(cx); self.publish_digest(); + asset_transfer::schedule_missing_assets(self, Some(peer), cx); cx.notify(); } pub fn neighbor_down(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { self.neighbors.remove(&peer); + if self.neighbors.is_empty() && self.zero_neighbors_since.is_none() { + self.zero_neighbors_since = Some(Instant::now()); + } + self.evaluate_connectivity(cx); cx.notify(); } pub fn handle_gossip_lagged(&mut self, cx: &mut Context) { self.publish_digest(); - if let Some(peer) = self.neighbors.iter().next().copied() { - let vv = self.doc.as_ref().map(|doc| doc.oplog_vv().encode()).unwrap_or_default(); - self.pull_updates(peer, vv, cx); - } + let peer = self.neighbors.iter().next().copied(); + let vv = self.doc.as_ref().map(|doc| doc.oplog_vv().encode()).unwrap_or_default(); + let action = self.anti_entropy.on_lagged(peer, vv); + self.handle_gap_action(action, cx); } pub fn set_endpoint_online(&mut self, online: bool, cx: &mut Context) { - if let SessionPhase::Attached(attachment) = &mut self.phase { - attachment.connectivity = if online { - Connectivity::Online - } else { - Connectivity::Offline { since: Instant::now(), retries: 0 } - }; - cx.notify(); - } - } - - pub fn import_update_bytes(&mut self, bytes: &[u8], cx: &mut Context) -> Result<()> { - if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() { - self.pending_remote_updates.push(bytes.to_vec()); - return Ok(()); + self.endpoint_online = online; + if online { + if self.peers_present() == 0 || !self.neighbors.is_empty() { + self.mark_online(cx); + } + } else { + self.evaluate_connectivity(cx); } - - let editor = self.editor.clone().context("collaboration session has no editor")?; - let document = Arc::new(editor.read(cx).document().clone()); - let doc = self.doc.clone().context("collaboration session has no Loro document")?; - let binding = self.binding.take().context("collaboration session has no document binding")?; - let binding = Arc::new(Mutex::new(binding)); - let patches = Arc::new(Mutex::new(Vec::::new())); - let sub = self.diff_subscription(doc.clone(), document, binding.clone(), patches.clone()); - - let import_result = doc.import_with(bytes, "remote"); - drop(sub); - self.binding = Some(take_mutex_value(binding, "document binding")?); - let patches = take_mutex_value(patches, "remote patches")?; - import_result.context("importing collaboration update failed")?; - self.apply_or_queue_patches(patches, cx); - Ok(()) + cx.notify(); } fn attach_editor_hooks(&mut self, cx: &mut Context) { @@ -448,13 +536,15 @@ impl CollabSession { return; }; - editor.update(cx, |editor, _| { + editor.update(cx, |editor, cx| { if editor.document_path().is_none() { - editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title))); + editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title)), cx); } + editor.clear_collab_history(); editor.set_collab_capture(true); - editor.set_collab_undo_redirect(Some(Rc::new(|_redirect: UndoRedirect| { - // TODO(Flowstate P2P §8/M3): replace this guard with Loro UndoManager undo/redo routing. + let undo_tx = self.undo_tx.clone(); + editor.set_collab_undo_redirect(Some(Rc::new(move |redirect: UndoRedirect| { + let _ = undo_tx.try_send(redirect); }))); }); @@ -492,307 +582,40 @@ impl CollabSession { self.local_update_publish_attached = true; } - fn attach_direct_request_pump(&mut self, cx: &mut Context) { - if self.direct_pump_started { + fn attach_undo_manager(&mut self) { + if self.undo_manager.is_some() { return; } - self.direct_pump_started = true; - let requests = self.direct_rx.clone(); - cx.spawn(async move |session, cx| { - while let Ok(request) = requests.recv().await { - if session.update(cx, |session, cx| session.handle_direct_request(request, cx)).is_err() { - break; - } - } - }) - .detach(); - } - - fn handle_direct_request(&mut self, request: DirectServeRequest, cx: &mut Context) { - match request { - DirectServeRequest::Snapshot { reply } => { - let _ = reply.try_send(self.snapshot_bytes()); - }, - DirectServeRequest::Updates { have_vv, reply } => { - let _ = reply.try_send(self.update_bytes(&have_vv)); - }, - DirectServeRequest::Asset { asset, reply } => { - let _ = reply.try_send(self.asset_bytes(asset, cx)); - }, - } - } - - fn snapshot_bytes(&self) -> Result> { - self - .doc - .as_ref() - .context("collaboration session is not attached")? - .export(ExportMode::Snapshot) - .context("exporting collaboration snapshot failed") - } - - fn update_bytes(&self, have_vv: &[u8]) -> Result> { - let vv = VersionVector::decode(have_vv).context("decoding collaboration version vector failed")?; - self - .doc - .as_ref() - .context("collaboration session is not attached")? - .export(ExportMode::updates(&vv)) - .context("exporting collaboration updates failed") - } - - fn asset_bytes(&self, asset: u128, cx: &mut Context) -> Result { - let editor = self.editor.as_ref().context("collaboration session has no editor")?; - let bytes = editor - .read(cx) - .document() - .assets - .assets - .get(&AssetId(asset)) - .map(|record| record.bytes.as_ref().clone()) - .ok_or_else(|| anyhow!("collaboration asset {asset} is not available"))?; - Ok(AssetBytes { bytes }) - } - - fn diff_subscription( - &self, - doc: LoroDoc, - document: Arc, - binding: Arc>, - patches: Arc>>, - ) -> LoroSubscription { - let subscribed_doc = doc.clone(); - let callback: Subscriber = Arc::new(move |event| { - let produced = { - let mut binding = match binding.lock() { - Ok(binding) => binding, - Err(error) => { - eprintln!("flowstate collab binding lock failed: {error}"); - return; - }, - }; - let result = { - let mut applier = RemoteApplier { - doc: &doc, - binding: &mut binding, - }; - applier.apply_event(&document, &event) - }; - drop(binding); - result - }; - match produced { - Ok(mut produced) => { - if let Ok(mut patches) = patches.lock() { - patches.append(&mut produced); - } - }, - Err(error) => eprintln!("flowstate collab remote apply failed: {error:#}"), - } - }); - subscribed_doc.subscribe_root(callback) - } - - fn apply_or_queue_patches(&mut self, mut patches: Vec, cx: &mut Context) { - if patches.is_empty() { - return; - } - self.pending_remote_patches.append(&mut patches); - self.flush_pending_remote_patches(cx); - } - - fn flush_pending_remote_patches(&mut self, cx: &mut Context) -> bool { - let Some(editor) = self.editor.clone() else { - return false; - }; - if self.pending_remote_patches.is_empty() || editor.read(cx).collab_apply_deferred() { - return false; - } - let patches = std::mem::take(&mut self.pending_remote_patches); - editor.update(cx, |editor, cx| editor.apply_collab_patches(&patches, cx)); - self.refresh_external_carets(cx); - true - } - - fn handle_digest( - &mut self, - from: flowstate_collab::ids::PeerId, - digest_session: SessionId, - vv: &[u8], - cx: &mut Context, - ) -> Result<()> { - if digest_session != self.session { - bail!("received a collaboration digest for a different session"); - } let Some(doc) = &self.doc else { - return Ok(()); - }; - let sender_vv = VersionVector::decode(vv).context("decoding collaboration digest failed")?; - let our_vv = doc.oplog_vv(); - match sender_vv.partial_cmp(&our_vv) { - Some(Ordering::Greater) | None => self.pull_updates(from, our_vv.encode(), cx), - Some(Ordering::Equal | Ordering::Less) => {}, - } - Ok(()) - } - - fn pull_updates(&mut self, from: flowstate_collab::ids::PeerId, our_vv: Vec, cx: &mut Context) { - if self.update_pull_in_flight { return; - } - self.update_pull_in_flight = true; - let (reply_tx, reply_rx) = async_channel::bounded(1); - let send_result = self.net_tx.try_send(NetCommand::PullUpdates { - session: self.session, - from, - our_vv, - reply: reply_tx, - }); - if send_result.is_err() { - self.update_pull_in_flight = false; - return; - } - cx.spawn(async move |session, cx| { - let result = reply_rx.recv().await; - let _ = session.update(cx, |session, cx| { - session.update_pull_in_flight = false; - if let Ok(Ok(bytes)) = result - && let Err(error) = session.import_update_bytes(&bytes, cx) - { - session.detach(DetachReason::Fatal(format!("pulling collaboration updates failed: {error:#}")), cx); - } - }); - }) - .detach(); + }; + let mut undo_manager = UndoManager::new(doc); + undo_manager.set_merge_interval(500); + undo_manager.set_max_undo_steps(300); + self.undo_manager = Some(undo_manager); } - fn pull_blob(&mut self, from: flowstate_collab::ids::PeerId, blob: flowstate_collab::BlobId, cx: &mut Context) { - let (reply_tx, reply_rx) = async_channel::bounded(1); - if self - .net_tx - .try_send(NetCommand::PullBlob { - session: self.session, - from, - blob, - reply: reply_tx, - }) - .is_err() - { + fn attach_undo_request_pump(&mut self, cx: &mut Context) { + if self.undo_pump_started { return; } + self.undo_pump_started = true; + let requests = self.undo_rx.clone(); cx.spawn(async move |session, cx| { - let result = reply_rx.recv().await; - let _ = session.update(cx, |session, cx| { - if let Ok(Ok(bytes)) = result - && let Err(error) = session.import_update_bytes(&bytes, cx) + while let Ok(redirect) = requests.recv().await { + if session + .update(cx, |session, cx| { + if let Err(error) = session.apply_loro_undo_redirect(redirect, cx) { + session.detach(DetachReason::Fatal(format!("collaboration undo failed: {error:#}")), cx); + } + }) + .is_err() { - session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + break; } - }); + } }) .detach(); } - fn publish_digest(&self) { - if let Some(doc) = &self.doc { - let _ = self.net_tx.try_send(NetCommand::Publish { - session: self.session, - payload: PublishPayload::Digest { vv: doc.oplog_vv().encode() }, - }); - } - } - - fn apply_presence(&mut self, bytes: &[u8], cx: &mut Context) { - if let Some(presence) = &self.presence { - if let Err(error) = presence.apply(bytes) { - eprintln!("flowstate collab presence update failed: {error:#}"); - } - self.refresh_peer_count(); - self.refresh_external_carets(cx); - cx.notify(); - } - } - - fn refresh_own_presence(&mut self, cx: &mut Context) { - let Some(presence) = &self.presence else { - return; - }; - let state = PresenceState { - name: default_presence_name(), - selection: self.own_presence_selection(cx), - }; - if let Err(error) = presence.set_self(&state) { - eprintln!("flowstate collab presence encode failed: {error:#}"); - } - self.refresh_peer_count(); - } - - fn own_presence_selection(&self, cx: &mut Context) -> Option { - let editor = self.editor.as_ref()?.read(cx); - let binding = self.binding.as_ref()?; - presence_view::selection_for_editor(editor, binding) - } - - fn refresh_external_carets(&mut self, cx: &mut Context) { - let Some(presence) = &self.presence else { - return; - }; - let Some(doc) = &self.doc else { - return; - }; - let Some(binding) = &self.binding else { - return; - }; - let Some(editor) = self.editor.clone() else { - return; - }; - let carets = presence_view::external_carets(doc, binding, presence); - editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); - } - - fn publish_presence_snapshot(&self) { - if let Some(presence) = &self.presence { - self.publish_presence_bytes(presence.encode_all()); - } - } - - fn publish_presence_bytes(&self, bytes: Vec) { - if bytes.is_empty() { - return; - } - let _ = self.net_tx.try_send(NetCommand::Publish { - session: self.session, - payload: PublishPayload::Presence(bytes), - }); - } - - fn refresh_peer_count(&mut self) { - let peers_present = self.peers_present(); - if let SessionPhase::Attached(attachment) = &mut self.phase { - attachment.peers_present = peers_present; - } - } - - fn peers_present(&self) -> usize { - self - .presence - .as_ref() - .map_or(0, |presence| presence.roster().len().saturating_sub(1)) - } -} - -fn take_mutex_value(value: Arc>, label: &str) -> Result { - match Arc::try_unwrap(value) { - Ok(mutex) => mutex - .into_inner() - .map_err(|error| anyhow!("collaboration {label} lock was poisoned: {error}")), - Err(_) => Err(anyhow!("collaboration {label} is still referenced")), - } -} - -fn default_presence_name() -> String { - std::env::var("FLOWSTATE_COLLAB_NAME") - .or_else(|_| std::env::var("USER")) - .or_else(|_| std::env::var("USERNAME")) - .unwrap_or_else(|_| "Flowstate user".to_string()) } diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs new file mode 100644 index 00000000..09fcc0bd --- /dev/null +++ b/crates/flowstate/src/collab/session_io.rs @@ -0,0 +1,313 @@ +use std::sync::{Arc, Mutex}; + +use anyhow::{Context as _, Result, anyhow}; +use flowstate_collab::{ + SessionId, + binding::DocBinding, + net::{ + NetCommand, PublishPayload, + anti_entropy::{GapAction, VersionVectorRelation}, + direct::DirectServeRequest, + }, + proto_direct::AssetBytes, + remote_apply::RemoteApplier, +}; +use gpui::Context; +use loro::{ExportMode, LoroDoc, Subscription as LoroSubscription, VersionVector, event::Subscriber}; + +use crate::rich_text_element::{AssetId, CollabPatch, Document, UndoRedirect}; + +use super::{CollabSession, DetachReason}; + +impl CollabSession { + pub fn import_update_bytes(&mut self, bytes: &[u8], cx: &mut Context) -> Result<()> { + if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() { + self.pending_remote_updates.push(bytes.to_vec()); + return Ok(()); + } + + let editor = self.editor.clone().context("collaboration session has no editor")?; + let document = Arc::new(editor.read(cx).document().clone()); + let doc = self.doc.clone().context("collaboration session has no Loro document")?; + let binding = self.binding.take().context("collaboration session has no document binding")?; + let binding = Arc::new(Mutex::new(binding)); + let patches = Arc::new(Mutex::new(Vec::::new())); + let sub = self.diff_subscription(doc.clone(), document, binding.clone(), patches.clone()); + + let import_result = doc.import_with(bytes, "remote"); + drop(sub); + self.binding = Some(take_mutex_value(binding, "document binding")?); + let patches = take_mutex_value(patches, "remote patches")?; + import_result.context("importing collaboration update failed")?; + self.apply_or_queue_patches(patches, cx); + Ok(()) + } + + pub(super) fn attach_direct_request_pump(&mut self, cx: &mut Context) { + if self.direct_pump_started { + return; + } + self.direct_pump_started = true; + let requests = self.direct_rx.clone(); + cx.spawn(async move |session, cx| { + while let Ok(request) = requests.recv().await { + if session.update(cx, |session, cx| session.handle_direct_request(request, cx)).is_err() { + break; + } + } + }) + .detach(); + } + + pub(super) fn handle_digest( + &mut self, + from: flowstate_collab::ids::PeerId, + digest_session: SessionId, + vv: &[u8], + cx: &mut Context, + ) -> Result<()> { + let Some(doc) = &self.doc else { + return Ok(()); + }; + let sender_vv = VersionVector::decode(vv).context("decoding collaboration digest failed")?; + let our_vv = doc.oplog_vv(); + let relation = match sender_vv.partial_cmp(&our_vv) { + Some(std::cmp::Ordering::Equal) => VersionVectorRelation::Equal, + Some(std::cmp::Ordering::Greater) => VersionVectorRelation::SenderHasMissingOps, + Some(std::cmp::Ordering::Less) => VersionVectorRelation::WeHaveMissingOps, + None => VersionVectorRelation::Concurrent, + }; + let action = self + .anti_entropy + .consider_digest(from, digest_session, relation, our_vv.encode()); + self.handle_gap_action(action, cx); + Ok(()) + } + + pub(super) fn pull_blob(&mut self, from: flowstate_collab::ids::PeerId, blob: flowstate_collab::BlobId, cx: &mut Context) { + let (reply_tx, reply_rx) = async_channel::bounded(1); + if self + .net_tx + .try_send(NetCommand::PullBlob { + session: self.session, + candidates: self.pull_candidates(Some(from)), + blob, + reply: reply_tx, + }) + .is_err() + { + return; + } + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + if let Ok(Ok(bytes)) = result + && let Err(error) = session.import_update_bytes(&bytes, cx) + { + session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + } + }); + }) + .detach(); + } + + pub(super) fn publish_digest(&self) { + if let Some(doc) = &self.doc { + let _ = self.net_tx.try_send(NetCommand::Publish { + session: self.session, + payload: PublishPayload::Digest { vv: doc.oplog_vv().encode() }, + }); + } + } + + pub(super) fn flush_pending_remote_patches(&mut self, cx: &mut Context) -> bool { + let Some(editor) = self.editor.clone() else { + return false; + }; + if self.pending_remote_patches.is_empty() || editor.read(cx).collab_apply_deferred() { + return false; + } + let patches = std::mem::take(&mut self.pending_remote_patches); + editor.update(cx, |editor, cx| { + editor.clear_collab_history(); + editor.apply_collab_patches(&patches, cx); + }); + self.last_document_activity = std::time::Instant::now(); + self.refresh_external_carets(cx); + true + } + + pub(super) fn apply_loro_undo_redirect(&mut self, redirect: UndoRedirect, cx: &mut Context) -> Result<()> { + if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() || self.undo_manager.is_none() { + return Ok(()); + } + + let editor = self.editor.clone().context("collaboration session has no editor")?; + let document = Arc::new(editor.read(cx).document().clone()); + let doc = self.doc.clone().context("collaboration session has no Loro document")?; + let binding = self.binding.take().context("collaboration session has no document binding")?; + let binding = Arc::new(Mutex::new(binding)); + let patches = Arc::new(Mutex::new(Vec::::new())); + let sub = self.diff_subscription(doc, document, binding.clone(), patches.clone()); + + let undo_result = match redirect { + UndoRedirect::Undo => self + .undo_manager + .as_mut() + .context("collaboration session has no undo manager")? + .undo(), + UndoRedirect::Redo => self + .undo_manager + .as_mut() + .context("collaboration session has no undo manager")? + .redo(), + }; + drop(sub); + self.binding = Some(take_mutex_value(binding, "document binding")?); + let patches = take_mutex_value(patches, "undo patches")?; + let applied = undo_result.context("applying collaboration undo operation failed")?; + if applied { + self.apply_or_queue_patches(patches, cx); + self.publish_digest(); + } + Ok(()) + } + + pub(super) fn handle_gap_action(&mut self, action: GapAction, cx: &mut Context) { + match action { + GapAction::None => {}, + GapAction::Pull { from, our_vv } => self.start_update_pull(from, our_vv, cx), + GapAction::LineageMismatch { from, expected, got } => { + eprintln!("flowstate collab ignored mismatched digest from {from}: expected session {expected}, got {got}"); + }, + } + } + + fn handle_direct_request(&mut self, request: DirectServeRequest, cx: &mut Context) { + match request { + DirectServeRequest::Snapshot { reply } => { + let _ = reply.try_send(self.snapshot_bytes()); + }, + DirectServeRequest::Updates { have_vv, reply } => { + let _ = reply.try_send(self.update_bytes(&have_vv)); + }, + DirectServeRequest::Asset { asset, reply } => { + let _ = reply.try_send(self.asset_bytes(asset, cx)); + }, + } + } + + fn snapshot_bytes(&self) -> Result> { + self + .doc + .as_ref() + .context("collaboration session is not attached")? + .export(ExportMode::Snapshot) + .context("exporting collaboration snapshot failed") + } + + fn update_bytes(&self, have_vv: &[u8]) -> Result> { + let vv = VersionVector::decode(have_vv).context("decoding collaboration version vector failed")?; + self + .doc + .as_ref() + .context("collaboration session is not attached")? + .export(ExportMode::updates(&vv)) + .context("exporting collaboration updates failed") + } + + fn asset_bytes(&self, asset: u128, cx: &mut Context) -> Result { + let editor = self.editor.as_ref().context("collaboration session has no editor")?; + let bytes = editor + .read(cx) + .document() + .assets + .assets + .get(&AssetId(asset)) + .map(|record| record.bytes.as_ref().clone()) + .ok_or_else(|| anyhow!("collaboration asset {asset} is not available"))?; + Ok(AssetBytes { bytes }) + } + + fn diff_subscription( + &self, + doc: LoroDoc, + document: Arc, + binding: Arc>, + patches: Arc>>, + ) -> LoroSubscription { + let subscribed_doc = doc.clone(); + let callback: Subscriber = Arc::new(move |event| { + let produced = { + let mut binding = match binding.lock() { + Ok(binding) => binding, + Err(error) => { + eprintln!("flowstate collab binding lock failed: {error}"); + return; + }, + }; + let result = { + let mut applier = RemoteApplier { + doc: &doc, + binding: &mut binding, + }; + applier.apply_event(&document, &event) + }; + drop(binding); + result + }; + match produced { + Ok(mut produced) => { + if let Ok(mut patches) = patches.lock() { + patches.append(&mut produced); + } + }, + Err(error) => eprintln!("flowstate collab remote apply failed: {error:#}"), + } + }); + subscribed_doc.subscribe_root(callback) + } + + pub(super) fn apply_or_queue_patches(&mut self, mut patches: Vec, cx: &mut Context) { + if patches.is_empty() { + return; + } + self.pending_remote_patches.append(&mut patches); + self.flush_pending_remote_patches(cx); + } + + fn start_update_pull(&mut self, from: flowstate_collab::ids::PeerId, our_vv: Vec, cx: &mut Context) { + let (reply_tx, reply_rx) = async_channel::bounded(1); + let send_result = self.net_tx.try_send(NetCommand::PullUpdates { + session: self.session, + candidates: self.pull_candidates(Some(from)), + our_vv, + reply: reply_tx, + }); + if send_result.is_err() { + self.anti_entropy.finish_pull(); + return; + } + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + session.anti_entropy.finish_pull(); + if let Ok(Ok(bytes)) = result + && let Err(error) = session.import_update_bytes(&bytes, cx) + { + session.detach(DetachReason::Fatal(format!("pulling collaboration updates failed: {error:#}")), cx); + } + }); + }) + .detach(); + } +} + +fn take_mutex_value(value: Arc>, label: &str) -> Result { + match Arc::try_unwrap(value) { + Ok(mutex) => mutex + .into_inner() + .map_err(|error| anyhow!("collaboration {label} lock was poisoned: {error}")), + Err(_) => Err(anyhow!("collaboration {label} is still referenced")), + } +} diff --git a/crates/flowstate/src/collab/session_presence.rs b/crates/flowstate/src/collab/session_presence.rs new file mode 100644 index 00000000..e78010e3 --- /dev/null +++ b/crates/flowstate/src/collab/session_presence.rs @@ -0,0 +1,92 @@ +use flowstate_collab::presence::{PresenceSelection, PresenceState}; +use gpui::Context; + +use super::{CollabSession, presence_view}; + +impl CollabSession { + pub(super) fn apply_presence(&mut self, bytes: &[u8], cx: &mut Context) { + if let Some(presence) = &self.presence { + if let Err(error) = presence.apply(bytes) { + eprintln!("flowstate collab presence update failed: {error:#}"); + } + self.refresh_peer_count(); + self.evaluate_connectivity(cx); + self.refresh_external_carets(cx); + cx.notify(); + } + } + + pub(super) fn refresh_own_presence(&mut self, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let state = PresenceState { + name: default_presence_name(), + selection: self.own_presence_selection(cx), + }; + if let Err(error) = presence.set_self(&state) { + eprintln!("flowstate collab presence encode failed: {error:#}"); + } + self.refresh_peer_count(); + } + + pub(super) fn refresh_external_carets(&mut self, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let Some(doc) = &self.doc else { + return; + }; + let Some(binding) = &self.binding else { + return; + }; + let Some(editor) = self.editor.clone() else { + return; + }; + let carets = presence_view::external_carets(doc, binding, presence); + editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); + } + + pub(super) fn publish_presence_snapshot(&self) { + if let Some(presence) = &self.presence { + self.publish_presence_bytes(presence.encode_all()); + } + } + + pub(super) fn publish_presence_bytes(&self, bytes: Vec) { + if bytes.is_empty() { + return; + } + let _ = self.net_tx.try_send(flowstate_collab::net::NetCommand::Publish { + session: self.session, + payload: flowstate_collab::net::PublishPayload::Presence(bytes), + }); + } + + pub(super) fn refresh_peer_count(&mut self) { + let peers_present = self.peers_present(); + if let super::SessionPhase::Attached(attachment) = &mut self.phase { + attachment.peers_present = peers_present; + } + } + + pub(super) fn peers_present(&self) -> usize { + self + .presence + .as_ref() + .map_or(0, |presence| presence.roster().len().saturating_sub(1)) + } + + fn own_presence_selection(&self, cx: &mut Context) -> Option { + let editor = self.editor.as_ref()?.read(cx); + let binding = self.binding.as_ref()?; + presence_view::selection_for_editor(editor, binding) + } +} + +fn default_presence_name() -> String { + std::env::var("FLOWSTATE_COLLAB_NAME") + .or_else(|_| std::env::var("USER")) + .or_else(|_| std::env::var("USERNAME")) + .unwrap_or_else(|_| "Flowstate user".to_string()) +} diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs new file mode 100644 index 00000000..7c88a4b1 --- /dev/null +++ b/crates/flowstate/src/collab/session_timers.rs @@ -0,0 +1,272 @@ +use std::time::{Duration, Instant}; + +use anyhow::{Context as _, Result}; +use flowstate_collab::{ + binding::DocBinding, + net::NetCommand, + presence::PRESENCE_KEEPALIVE_SECS, + projection, self_check, +}; +use gpui::{Context, Timer}; + +use crate::{app_settings::load_document_theme, rich_text_element::Document}; + +use super::{Attachment, CollabSession, Connectivity, SessionPhase}; + +const ZERO_NEIGHBOR_OFFLINE_GRACE: Duration = Duration::from_secs(5); +const QUIET_DIGEST_ROUNDS_OFFLINE: u8 = 2; +const SELF_CHECK_INTERVAL: Duration = Duration::from_secs(30); +const SELF_CHECK_IDLE: Duration = Duration::from_secs(2); +const RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30); + +impl CollabSession { + pub(super) fn attach_timers(&mut self, cx: &mut Context) { + if self.timers_started { + return; + } + self.timers_started = true; + self.spawn_digest_timer(cx); + self.spawn_presence_timer(cx); + self.spawn_self_check_timer(cx); + } + + pub(super) fn note_inbound_traffic(&mut self, cx: &mut Context) { + self.inbound_since_last_digest = true; + self.quiet_digest_rounds = 0; + if self.mark_online(cx) { + self.anti_entropy.on_reconnect(); + self.publish_digest(); + } + } + + pub(super) fn mark_online(&mut self, cx: &mut Context) -> bool { + let mut changed = false; + if let SessionPhase::Attached(attachment) = &mut self.phase + && matches!(attachment.connectivity, Connectivity::Offline { .. }) + { + attachment.connectivity = Connectivity::Online; + self.next_recovery_at = None; + changed = true; + } + if changed { + cx.notify(); + } + changed + } + + pub(super) fn evaluate_connectivity(&mut self, cx: &mut Context) { + if !matches!(self.phase, SessionPhase::Attached(_)) { + return; + } + + self.refresh_peer_count(); + if self.peers_present() == 0 || !self.neighbors.is_empty() { + self.mark_online(cx); + return; + } + + let now = Instant::now(); + let zero_since = self.zero_neighbors_since.get_or_insert(now); + let quiet_enough = self.quiet_digest_rounds >= QUIET_DIGEST_ROUNDS_OFFLINE || !self.endpoint_online; + if quiet_enough && now.saturating_duration_since(*zero_since) >= ZERO_NEIGHBOR_OFFLINE_GRACE { + self.mark_offline(now, cx); + } + } + + fn spawn_digest_timer(&mut self, cx: &mut Context) { + cx.spawn(async move |session, cx| { + loop { + let delay = match session.update(cx, |session, _| session.next_digest_delay()) { + Ok(Some(delay)) => delay, + Ok(None) | Err(_) => break, + }; + Timer::after(delay).await; + match session.update(cx, |session, cx| session.run_digest_tick(cx)) { + Ok(true) => {}, + Ok(false) | Err(_) => break, + } + } + }) + .detach(); + } + + fn spawn_presence_timer(&mut self, cx: &mut Context) { + cx.spawn(async move |session, cx| { + loop { + Timer::after(Duration::from_secs(PRESENCE_KEEPALIVE_SECS)).await; + match session.update(cx, |session, cx| session.run_presence_tick(cx)) { + Ok(true) => {}, + Ok(false) | Err(_) => break, + } + } + }) + .detach(); + } + + fn spawn_self_check_timer(&mut self, cx: &mut Context) { + cx.spawn(async move |session, cx| { + loop { + Timer::after(SELF_CHECK_INTERVAL).await; + match session.update(cx, |session, cx| session.run_self_check_tick(cx)) { + Ok(true) => {}, + Ok(false) | Err(_) => break, + } + } + }) + .detach(); + } + + fn next_digest_delay(&self) -> Option { + self.timer_live().then(|| self.anti_entropy.duration_until_digest(Instant::now())) + } + + fn run_digest_tick(&mut self, cx: &mut Context) -> bool { + if !self.timer_live() { + return false; + } + + let now = Instant::now(); + if self.anti_entropy.digest_due(now) { + if matches!(self.phase, SessionPhase::Attached(_)) { + self.publish_digest(); + self.finish_digest_round(now, cx); + } + self.anti_entropy.mark_digest_sent(now); + } + true + } + + fn run_presence_tick(&mut self, cx: &mut Context) -> bool { + if !self.timer_live() { + return false; + } + if self.presence.is_some() { + if let Some(presence) = &self.presence { + presence.remove_outdated(); + } + self.refresh_own_presence(cx); + self.refresh_peer_count(); + self.evaluate_connectivity(cx); + self.refresh_external_carets(cx); + cx.notify(); + } + true + } + + fn run_self_check_tick(&mut self, cx: &mut Context) -> bool { + if !self.timer_live() { + return false; + } + if !matches!(self.phase, SessionPhase::Attached(_)) || self.last_document_activity.elapsed() < SELF_CHECK_IDLE { + return true; + } + if let Err(error) = self.run_self_check(cx) { + eprintln!("flowstate collab self-check failed: {error:#}"); + } + true + } + + fn finish_digest_round(&mut self, now: Instant, cx: &mut Context) { + if self.inbound_since_last_digest { + self.quiet_digest_rounds = 0; + } else { + self.quiet_digest_rounds = self.quiet_digest_rounds.saturating_add(1); + } + self.inbound_since_last_digest = false; + self.evaluate_connectivity(cx); + self.run_recovery_if_due(now, cx); + } + + fn run_recovery_if_due(&mut self, now: Instant, cx: &mut Context) { + let Some(next_recovery_at) = self.next_recovery_at else { + return; + }; + if now < next_recovery_at { + return; + } + + let mut attempted = false; + if let SessionPhase::Attached(Attachment { + connectivity: Connectivity::Offline { retries, .. }, + .. + }) = &mut self.phase + { + let delay = recovery_delay(*retries); + *retries = retries.saturating_add(1); + self.next_recovery_at = Some(now + delay); + attempted = true; + } + + if attempted { + if self.bootstrap_addrs.is_empty() { + let _ = self.net_tx.try_send(NetCommand::EnsureUp); + } else { + let _ = self.net_tx.try_send(NetCommand::JoinSession { + session: self.session, + bootstrap: self.bootstrap_addrs.clone(), + }); + } + self.publish_digest(); + cx.notify(); + } + } + + fn run_self_check(&mut self, cx: &mut Context) -> Result<()> { + let Some(doc) = &self.doc else { + return Ok(()); + }; + let Some(editor) = self.editor.clone() else { + return Ok(()); + }; + + let live_document = editor.read(cx).document().clone(); + let live_hash = self_check::projection_hash(&live_document); + let mut projected = projection::document_from_loro(doc, load_document_theme())?; + projected.assets = live_document.assets.clone(); + let projected_hash = self_check::projection_hash(&projected); + if live_hash == projected_hash { + return Ok(()); + } + + eprintln!( + "flowstate collab projection drift detected: live_hash={live_hash:016x} projected_hash={projected_hash:016x} vv_bytes={}", + doc.oplog_vv().encode().len() + ); + self.rebuild_from_projection(projected, cx) + } + + fn rebuild_from_projection(&mut self, projected: Document, cx: &mut Context) -> Result<()> { + let Some(doc) = &self.doc else { + return Ok(()); + }; + let Some(editor) = self.editor.clone() else { + return Ok(()); + }; + editor.update(cx, |editor, cx| editor.replace_document_from_collaboration(projected, cx)); + let document = editor.read(cx).document().clone(); + self.binding = Some(DocBinding::build(doc, &document).context("rebuilding collaboration binding after self-check failed")?); + self.last_document_activity = Instant::now(); + self.refresh_external_carets(cx); + Ok(()) + } + + fn mark_offline(&mut self, now: Instant, cx: &mut Context) { + if let SessionPhase::Attached(attachment) = &mut self.phase + && matches!(attachment.connectivity, Connectivity::Online) + { + attachment.connectivity = Connectivity::Offline { since: now, retries: 0 }; + self.next_recovery_at = Some(now); + cx.notify(); + } + } + + fn timer_live(&self) -> bool { + !matches!(self.phase, SessionPhase::Detached(_)) + } +} + +fn recovery_delay(retries: u32) -> Duration { + let shift = retries.min(5); + let secs = 1_u64.checked_shl(shift).unwrap_or(RECOVERY_MAX_BACKOFF.as_secs()); + Duration::from_secs(secs).min(RECOVERY_MAX_BACKOFF) +} diff --git a/crates/flowstate/src/collab/share_dialog.rs b/crates/flowstate/src/collab/share_dialog.rs new file mode 100644 index 00000000..f0008b09 --- /dev/null +++ b/crates/flowstate/src/collab/share_dialog.rs @@ -0,0 +1,530 @@ +use flowstate_collab::ticket::SessionTicket; +use gpui::{ + AnyElement, AnyWindowHandle, App, ClipboardItem, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, + KeyDownEvent, MouseButton, ParentElement, Render, SharedString, Subscription, WeakEntity, Window, div, prelude::*, px, relative, rgb, +}; +use gpui_component::{ + ActiveTheme as _, Disableable, IconName, Sizable, + button::{Button, ButtonVariants as _}, + h_flex, + input::{Input, InputEvent, InputState}, + scroll::ScrollableElement, + v_flex, +}; +use uuid::Uuid; + +use crate::workspace::Workspace; + +use super::{Connectivity, SessionPhase, status}; + +#[derive(Clone, Copy, PartialEq, Eq)] +pub enum CollabDialogMode { + Share, + Join, +} + +pub struct CollabShareDialog { + workspace: WeakEntity, + panel_id: Option, + mode: CollabDialogMode, + join_input: Entity, + ticket_text: Option, + ticket_loading: bool, + ticket_error: Option, + copy_notice: Option, + join_error: Option, + _input_subscription: Subscription, +} + +#[hotpath::measure_all] +impl CollabShareDialog { + pub fn new( + workspace: WeakEntity, + panel_id: Option, + mode: CollabDialogMode, + window: &mut Window, + cx: &mut Context, + ) -> Self { + let join_input = cx.new(|cx| InputState::new(window, cx).placeholder("Paste a Flowstate collaboration invite")); + let _input_subscription = cx.subscribe(&join_input, |dialog, _, event: &InputEvent, cx| { + if let InputEvent::Change = event { + dialog.validate_join_ticket(cx); + } + }); + + let mut dialog = Self { + workspace, + panel_id, + mode, + join_input, + ticket_text: None, + ticket_loading: false, + ticket_error: None, + copy_notice: None, + join_error: None, + _input_subscription, + }; + if mode == CollabDialogMode::Share { + dialog.refresh_ticket(None, cx); + } + dialog + } + + pub fn focus(&self, window: &mut Window, cx: &mut Context) { + if self.mode == CollabDialogMode::Join { + self.join_input.focus_handle(cx).focus(window); + } + } + + fn close(&mut self, cx: &mut Context) { + let _ = self.workspace.update(cx, |workspace, cx| workspace.close_collaboration_dialog(cx)); + } + + fn set_mode(&mut self, mode: CollabDialogMode, cx: &mut Context) { + if self.mode == mode { + return; + } + self.mode = mode; + self.join_error = None; + self.copy_notice = None; + if mode == CollabDialogMode::Share { + self.refresh_ticket(None, cx); + } + cx.notify(); + } + + fn start_session(&mut self, cx: &mut Context) { + let Some(panel_id) = self.panel_id else { + self.ticket_error = Some("Open a document before starting a session.".into()); + cx.notify(); + return; + }; + + match self + .workspace + .update(cx, |workspace, cx| workspace.start_collaboration_on_document(panel_id, cx)) + { + Ok(Some(_)) => self.refresh_ticket(None, cx), + Ok(None) => { + self.ticket_error = Some("The active document could not be shared.".into()); + cx.notify(); + }, + Err(error) => { + self.ticket_error = Some(format!("The workspace is no longer available: {error}").into()); + cx.notify(); + }, + } + } + + fn refresh_ticket(&mut self, copy_to_clipboard: Option, cx: &mut Context) { + let Some(panel_id) = self.panel_id else { + self.ticket_text = None; + self.ticket_loading = false; + return; + }; + let Some(ticket_rx) = crate::collab::request_ticket_for_panel(panel_id, cx) else { + self.ticket_text = None; + self.ticket_loading = false; + self.ticket_error = None; + return; + }; + + self.ticket_loading = true; + self.ticket_error = None; + self.copy_notice = None; + cx.notify(); + + cx.spawn(async move |dialog, cx| { + let result = ticket_rx.recv().await; + let encoded = match result { + Ok(Ok(ticket)) => Ok(ticket.encode_text()), + Ok(Err(error)) => Err(format!("Creating collaboration invite failed: {error:#}")), + Err(error) => Err(format!("Creating collaboration invite failed: {error}")), + }; + let should_copy = copy_to_clipboard.is_some(); + + if let Ok(text) = &encoded + && let Some(window_handle) = copy_to_clipboard + { + let text = text.clone(); + let _ = window_handle.update(cx, |_, _, cx| cx.write_to_clipboard(ClipboardItem::new_string(text))); + } + + let _ = dialog.update(cx, |dialog, cx| { + dialog.ticket_loading = false; + match encoded { + Ok(text) => { + dialog.ticket_text = Some(text.into()); + dialog.ticket_error = None; + if should_copy { + dialog.copy_notice = Some("Fresh invite copied to clipboard.".into()); + } + }, + Err(error) => { + dialog.ticket_error = Some(error.into()); + }, + } + cx.notify(); + }); + }) + .detach(); + } + + fn copy_invite(&mut self, window: &mut Window, cx: &mut Context) { + self.refresh_ticket(Some(window.window_handle()), cx); + } + + fn validate_join_ticket(&mut self, cx: &mut Context) { + self.join_error = match self.parse_join_ticket(cx) { + Ok(_) => None, + Err(error) => Some(error.into()), + }; + cx.notify(); + } + + fn parse_join_ticket(&self, cx: &App) -> Result, String> { + let text = self.join_input.read(cx).value().trim().to_string(); + if text.is_empty() { + return Ok(None); + } + SessionTicket::decode_text(&text) + .map(Some) + .map_err(|error| format!("Invalid invite: {error}")) + } + + fn join_session(&mut self, window: &mut Window, cx: &mut Context) { + let ticket = match self.parse_join_ticket(cx) { + Ok(Some(ticket)) => ticket, + Ok(None) => { + self.join_error = Some("Paste an invite before joining.".into()); + cx.notify(); + return; + }, + Err(error) => { + self.join_error = Some(error.into()); + cx.notify(); + return; + }, + }; + + match self + .workspace + .update(cx, |workspace, cx| workspace.join_collaboration_session(ticket, window, cx)) + { + Ok(Some(_)) => self.close(cx), + Ok(None) => { + self.join_error = Some("Joining collaboration session failed.".into()); + cx.notify(); + }, + Err(error) => { + self.join_error = Some(format!("The workspace is no longer available: {error}").into()); + cx.notify(); + }, + } + } + + fn confirm_leave(&mut self, window: &mut Window, cx: &mut Context) { + let Some(panel_id) = self.panel_id else { + return; + }; + let _ = self + .workspace + .update(cx, |workspace, cx| workspace.confirm_leave_collaboration_on_panel(panel_id, window, cx)); + } + + fn on_key_down(&mut self, event: &KeyDownEvent, window: &mut Window, cx: &mut Context) { + match event.keystroke.key.as_str() { + "escape" => { + self.close(cx); + cx.stop_propagation(); + }, + "enter" if self.mode == CollabDialogMode::Join => { + self.join_session(window, cx); + cx.stop_propagation(); + }, + _ => {}, + } + } + + fn render_share_pane(&self, phase: Option<&SessionPhase>, cx: &mut Context) -> AnyElement { + if self.panel_id.is_none() { + return v_flex() + .gap_3() + .child(section_title("Share this document", cx)) + .child(helper_text("Open a rich-text document before starting a collaboration session.", cx)) + .into_any_element(); + } + + if phase.is_none_or(|phase| matches!(phase, SessionPhase::Detached(_))) { + return v_flex() + .gap_3() + .child(section_title("Share this document", cx)) + .child(helper_text( + "Start a session to create one invite ticket. Everyone who joins can edit, invite others, and leave independently.", + cx, + )) + .child( + Button::new("start-collaboration-session") + .label("Start session") + .primary() + .on_click(cx.listener(|dialog, _, _, cx| dialog.start_session(cx))), + ) + .when_some(self.ticket_error.clone(), |this, error| this.child(error_text(error, cx))) + .into_any_element(); + } + + let Some(phase) = phase else { + return v_flex().into_any_element(); + }; + let roster = self + .panel_id + .map_or_else(Vec::new, |panel_id| crate::collab::roster_for_panel(panel_id, cx)); + v_flex() + .gap_3() + .child(h_flex().items_center().justify_between().child(section_title("Invite people", cx)).child(status::status_pill(phase, cx))) + .child(helper_text(connectivity_text(phase).as_str(), cx)) + .child(roster_list(roster, cx)) + .child(ticket_box(self.ticket_text.clone(), self.ticket_loading, cx)) + .when_some(self.ticket_error.clone(), |this, error| this.child(error_text(error, cx))) + .when_some(self.copy_notice.clone(), |this, notice| this.child(success_text(notice, cx))) + .child( + h_flex() + .gap_2() + .child( + Button::new("copy-collaboration-invite") + .label(if self.ticket_loading { "Minting..." } else { "Copy invite" }) + .primary() + .disabled(self.ticket_loading) + .on_click(cx.listener(|dialog, _, window, cx| dialog.copy_invite(window, cx))), + ) + .child( + Button::new("refresh-collaboration-invite") + .label("Refresh invite") + .outline() + .disabled(self.ticket_loading) + .on_click(cx.listener(|dialog, _, _, cx| dialog.refresh_ticket(None, cx))), + ) + .child(div().flex_1()) + .child( + Button::new("leave-collaboration-session") + .label("Leave session") + .danger() + .on_click(cx.listener(|dialog, _, window, cx| dialog.confirm_leave(window, cx))), + ), + ) + .into_any_element() + } + + fn render_join_pane(&self, cx: &mut Context) -> AnyElement { + let parsed = self.parse_join_ticket(cx).ok().flatten(); + v_flex() + .gap_3() + .child(section_title("Join a session", cx)) + .child(helper_text("Paste an invite ticket. Joining opens the shared document as a new tab and never merges into your existing files.", cx)) + .child(Input::new(&self.join_input).w_full().cleanable(true)) + .when_some(parsed.as_ref().map(|ticket| ticket.title.clone()), |this, title| { + this.child(success_text(format!("Invite for: {title}").into(), cx)) + }) + .when_some(self.join_error.clone(), |this, error| this.child(error_text(error, cx))) + .child( + h_flex() + .justify_end() + .gap_2() + .child( + Button::new("join-collaboration-session") + .label("Join") + .primary() + .on_click(cx.listener(|dialog, _, window, cx| dialog.join_session(window, cx))), + ), + ) + .into_any_element() + } +} + +#[hotpath::measure_all] +impl Focusable for CollabShareDialog { + fn focus_handle(&self, cx: &App) -> FocusHandle { + self.join_input.focus_handle(cx) + } +} + +#[hotpath::measure_all] +impl Render for CollabShareDialog { + fn render(&mut self, _: &mut Window, cx: &mut Context) -> impl IntoElement { + let phase = self.panel_id.and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)); + + div() + .absolute() + .top_0() + .right_0() + .bottom_0() + .left_0() + .bg(cx.theme().background.opacity(0.72)) + .flex() + .items_center() + .justify_center() + .occlude() + .on_key_down(cx.listener(Self::on_key_down)) + .on_mouse_down(MouseButton::Left, cx.listener(|dialog, _, _, cx| { + dialog.close(cx); + cx.stop_propagation(); + })) + .on_scroll_wheel(|_, _, cx| cx.stop_propagation()) + .child( + v_flex() + .w(px(620.0)) + .max_w_full() + .max_h(px(620.0)) + .overflow_hidden() + .rounded_lg() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().popover) + .shadow_lg() + .on_mouse_down(MouseButton::Left, |_, _, cx| cx.stop_propagation()) + .child( + h_flex() + .h(px(44.0)) + .items_center() + .justify_between() + .px_4() + .border_b_1() + .border_color(cx.theme().border) + .child(div().font_weight(gpui::FontWeight::SEMIBOLD).child("Share / Collaborate")) + .child( + Button::new("close-collaboration-dialog") + .icon(IconName::Close) + .xsmall() + .ghost() + .tooltip("Close") + .on_click(cx.listener(|dialog, _, _, cx| dialog.close(cx))), + ), + ) + .child( + h_flex() + .gap_2() + .px_4() + .pt_4() + .child(tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { + dialog.set_mode(CollabDialogMode::Share, cx); + }))) + .child(tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { + dialog.set_mode(CollabDialogMode::Join, cx); + dialog.focus(window, cx); + }))), + ) + .child( + div() + .flex_1() + .overflow_y_scrollbar() + .p_4() + .child(match self.mode { + CollabDialogMode::Share => self.render_share_pane(phase.as_ref(), cx), + CollabDialogMode::Join => self.render_join_pane(cx), + }), + ), + ) + } +} + +fn tab_button(id: &'static str, label: &'static str, selected: bool) -> Button { + let button = Button::new(id).label(label).small(); + if selected { button.primary() } else { button.outline() } +} + +fn section_title(title: &'static str, cx: &App) -> impl IntoElement { + div() + .text_lg() + .font_weight(gpui::FontWeight::SEMIBOLD) + .text_color(cx.theme().foreground) + .child(title) +} + +fn helper_text(text: &str, cx: &App) -> impl IntoElement { + div() + .text_sm() + .line_height(relative(1.45)) + .text_color(cx.theme().muted_foreground) + .child(text.to_string()) +} + +fn ticket_box(ticket: Option, loading: bool, cx: &App) -> impl IntoElement { + div() + .w_full() + .min_h(px(84.0)) + .max_h(px(132.0)) + .overflow_y_scrollbar() + .rounded_md() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().background) + .p_3() + .text_xs() + .line_height(relative(1.35)) + .text_color(cx.theme().foreground) + .child(if loading { + SharedString::from("Minting a fresh invite...") + } else { + ticket.unwrap_or_else(|| "Start sharing to mint an invite.".into()) + }) +} + +fn error_text(text: SharedString, cx: &App) -> impl IntoElement { + div() + .text_sm() + .text_color(cx.theme().danger) + .child(text) +} + +fn success_text(text: SharedString, cx: &App) -> impl IntoElement { + div() + .text_sm() + .text_color(cx.theme().success) + .child(text) +} + +fn roster_list(entries: Vec, cx: &App) -> AnyElement { + let mut list = v_flex() + .gap_2() + .rounded_md() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().background) + .p_3() + .child(div().text_xs().text_color(cx.theme().muted_foreground).child("Participants")); + + if entries.is_empty() { + return list + .child(div().text_sm().text_color(cx.theme().muted_foreground).child("Presence is starting...")) + .into_any_element(); + } + + for entry in entries { + let label = if entry.is_self { + format!("{} (you)", entry.name) + } else { + entry.name + }; + list = list.child( + h_flex() + .items_center() + .gap_2() + .child(div().size(px(8.0)).rounded_full().bg(rgb(entry.color_rgb))) + .child(div().text_sm().text_color(cx.theme().foreground).child(label)), + ); + } + list.into_any_element() +} + +fn connectivity_text(phase: &SessionPhase) -> String { + match phase { + SessionPhase::Creating => "Starting collaboration...".to_string(), + SessionPhase::Joining(_) => "Joining collaboration...".to_string(), + SessionPhase::Attached(attachment) => match &attachment.connectivity { + Connectivity::Online if attachment.peers_present == 0 => "Only you - share the invite.".to_string(), + Connectivity::Online => format!("{} people in session.", attachment.peers_present + 1), + Connectivity::Offline { .. } => "Offline - reconnecting. Changes will sync when connected.".to_string(), + }, + SessionPhase::Detached(_) => "This document is local.".to_string(), + } +} diff --git a/crates/flowstate/src/collab/shutdown.rs b/crates/flowstate/src/collab/shutdown.rs new file mode 100644 index 00000000..6db912e0 --- /dev/null +++ b/crates/flowstate/src/collab/shutdown.rs @@ -0,0 +1,23 @@ +use flowstate_collab::net::NetCommand; +use gpui::BorrowAppContext; + +use super::CollabManager; + +impl CollabManager { + pub(super) fn shutdown_runtime(&mut self) { + if let Some(runtime) = self.runtime.take() { + let _ = runtime.commands.try_send(NetCommand::Shutdown); + } + self.sessions_by_id.clear(); + self.session_by_panel.clear(); + self.event_pump_started = false; + self.endpoint_online = false; + } +} + +pub fn shutdown(cx: &mut C) +where + C: BorrowAppContext, +{ + cx.update_default_global::(|manager, _| manager.shutdown_runtime()); +} diff --git a/crates/flowstate/src/collab/status.rs b/crates/flowstate/src/collab/status.rs index 4be211ae..ea59dc14 100644 --- a/crates/flowstate/src/collab/status.rs +++ b/crates/flowstate/src/collab/status.rs @@ -30,6 +30,42 @@ pub fn status_pill(phase: &SessionPhase, cx: &App) -> AnyElement { .into_any_element() } +pub fn tab_badge(phase: &SessionPhase, cx: &App) -> Option { + let (count, color) = match phase { + SessionPhase::Attached(attachment) => { + let color = match &attachment.connectivity { + Connectivity::Online => cx.theme().success, + Connectivity::Offline { .. } => cx.theme().muted_foreground, + }; + (Some(attachment.peers_present + 1), color) + }, + SessionPhase::Creating | SessionPhase::Joining(_) => (None, cx.theme().info), + SessionPhase::Detached(_) => return None, + }; + + Some( + h_flex() + .items_center() + .gap_0p5() + .child( + div() + .size(px(6.0)) + .rounded_full() + .bg(color), + ) + .when_some(count, |this, count| { + this.child( + div() + .text_size(px(9.0)) + .font_weight(gpui::FontWeight::SEMIBOLD) + .text_color(color) + .child(count.to_string()), + ) + }) + .into_any_element(), + ) +} + fn status_label_and_color(phase: &SessionPhase, cx: &App) -> (String, gpui::Hsla) { match phase { SessionPhase::Creating => ("Starting share".to_string(), cx.theme().info), @@ -52,7 +88,7 @@ fn join_label(stage: &JoinStage) -> String { } fn attached_label_and_color(attachment: &Attachment, cx: &App) -> (String, gpui::Hsla) { - match attachment.connectivity { + match &attachment.connectivity { Connectivity::Online => { let participants = attachment.peers_present + 1; if attachment.peers_present == 0 { diff --git a/crates/flowstate/src/commands.rs b/crates/flowstate/src/commands.rs index 02e5b684..f7a8b887 100644 --- a/crates/flowstate/src/commands.rs +++ b/crates/flowstate/src/commands.rs @@ -76,6 +76,8 @@ pub enum CommandId { OpenDocument, OpenDemoDocument, CloseDocument, + ShareDocument, + JoinSession, StartCollaboration, CopyCollaborationTicket, JoinCollaborationFromClipboard, @@ -205,6 +207,8 @@ pub const COMMAND_SPECS: &[CommandSpec] = &[ CommandSpec::new(CommandId::OpenDocument, "Open Document", APP, &[]), CommandSpec::new(CommandId::OpenDemoDocument, "Open Demo Document", APP, &[]), CommandSpec::new(CommandId::CloseDocument, "Close Document", APP, &[]), + CommandSpec::new(CommandId::ShareDocument, "Share / Collaborate...", APP, &[]), + CommandSpec::new(CommandId::JoinSession, "Join Collaboration Session...", APP, &[]), CommandSpec::new(CommandId::StartCollaboration, "Start Collaboration", APP, &[]), CommandSpec::new(CommandId::CopyCollaborationTicket, "Copy Collaboration Invite", APP, &[]), CommandSpec::new(CommandId::JoinCollaborationFromClipboard, "Join Collaboration from Clipboard", APP, &[]), diff --git a/crates/flowstate/src/commands/keymap.rs b/crates/flowstate/src/commands/keymap.rs index 6ed22b1d..4e69f6ea 100644 --- a/crates/flowstate/src/commands/keymap.rs +++ b/crates/flowstate/src/commands/keymap.rs @@ -151,6 +151,8 @@ fn action_for_command(command: CommandId) -> Option> { | CommandId::OpenDocument | CommandId::OpenDemoDocument | CommandId::CloseDocument + | CommandId::ShareDocument + | CommandId::JoinSession | CommandId::StartCollaboration | CommandId::CopyCollaborationTicket | CommandId::JoinCollaborationFromClipboard diff --git a/crates/flowstate/src/rich_text_element/mod.rs b/crates/flowstate/src/rich_text_element/mod.rs index 3bece707..4bf4a4b9 100644 --- a/crates/flowstate/src/rich_text_element/mod.rs +++ b/crates/flowstate/src/rich_text_element/mod.rs @@ -64,6 +64,8 @@ pub fn flowstate_command_to_rich_text(command: CommandId) -> Option) { + self.open_collaboration_dialog_with_mode(crate::collab::share_dialog::CollabDialogMode::Share, window, cx); + } + + pub fn open_join_collaboration_dialog(&mut self, window: &mut Window, cx: &mut Context) { + self.open_collaboration_dialog_with_mode(crate::collab::share_dialog::CollabDialogMode::Join, window, cx); + } + + pub fn close_collaboration_dialog(&mut self, cx: &mut Context) { + self.collaboration_dialog = None; + cx.notify(); + } + + fn open_collaboration_dialog_with_mode( + &mut self, + mode: crate::collab::share_dialog::CollabDialogMode, + window: &mut Window, + cx: &mut Context, + ) { + let workspace = cx.entity().downgrade(); + let panel_id = self.active_editor.as_ref().and(self.active_document_id); + let dialog = cx.new(|cx| crate::collab::share_dialog::CollabShareDialog::new(workspace, panel_id, mode, window, cx)); + dialog.update(cx, |dialog, cx| dialog.focus(window, cx)); + self.collaboration_dialog = Some(dialog); + cx.notify(); + } + pub fn start_collaboration_on_active_document(&mut self, cx: &mut Context) -> Option { - let panel_id = self.active_document_id?; - let editor = self.active_editor.clone()?; + self.start_collaboration_on_document(self.active_document_id?, cx) + } + + pub fn start_collaboration_on_document(&mut self, panel_id: Uuid, cx: &mut Context) -> Option { + let panel = self + .document_panels + .iter() + .find(|panel| panel.read(cx).id() == panel_id)?; + let editor = panel.read(cx).editor(); let title = self .document_panels .iter() @@ -116,6 +150,51 @@ impl Workspace { self.join_collaboration_session(ticket, window, cx).is_some() } + pub fn confirm_leave_collaboration_on_active_document(&mut self, window: &mut Window, cx: &mut Context) -> bool { + self + .active_document_id + .is_some_and(|panel_id| self.confirm_leave_collaboration_on_panel(panel_id, window, cx)) + } + + pub fn confirm_leave_collaboration_on_panel(&mut self, panel_id: Uuid, window: &mut Window, cx: &mut Context) -> bool { + let phase = crate::collab::phase_for_panel(panel_id, cx); + if !collaboration_phase_blocks_close(phase.as_ref()) { + return false; + } + let detail = format!("{} Your copy of the document stays open.", collaboration_leave_detail(phase.as_ref())); + let answer = window.prompt( + PromptLevel::Warning, + "Leave this session?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ); + let window_handle = window.window_handle(); + cx.spawn(async move |workspace, cx| { + if !matches!(answer.await, Ok(0)) { + return; + } + let _ = workspace.update(cx, |workspace, cx| { + workspace.leave_collaboration_on_panel(panel_id, cx); + }); + let _ = window_handle.update(cx, |_, window, cx| { + std::mem::drop(window.prompt( + PromptLevel::Info, + "Left session", + Some("Left session - this copy is now local."), + &[PromptButton::ok("Ok")], + cx, + )); + }); + }) + .detach(); + true + } + + pub fn leave_collaboration_on_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> bool { + crate::collab::leave_session_for_panel(panel_id, cx) + } + pub fn join_collaboration_session( &mut self, ticket: flowstate_collab::ticket::SessionTicket, @@ -198,7 +277,7 @@ impl Workspace { pub fn leave_collaboration_on_active_document(&mut self, cx: &mut Context) -> bool { self .active_document_id - .is_some_and(|panel_id| crate::collab::leave_session_for_panel(panel_id, cx)) + .is_some_and(|panel_id| self.leave_collaboration_on_panel(panel_id, cx)) } pub fn leave_all_collaboration_sessions(&mut self, cx: &mut Context) { diff --git a/crates/flowstate/src/workspace/workspace/collab_prompts.rs b/crates/flowstate/src/workspace/workspace/collab_prompts.rs new file mode 100644 index 00000000..14960b67 --- /dev/null +++ b/crates/flowstate/src/workspace/workspace/collab_prompts.rs @@ -0,0 +1,324 @@ +#[derive(Clone)] +struct CollaborationClosePanel { + id: Uuid, + panel: PanelKind, + save_prompt: CollaborationSavePrompt, +} + +#[derive(Clone, Copy)] +enum CollaborationSavePrompt { + Clean, + Dirty, + Pathless, +} + +enum CollaborationCloseResolution { + Proceed { discard: Option }, + Cancelled, +} + +#[hotpath::measure_all] +impl Workspace { + fn close_collaboration_document_panel( + &mut self, + panel_id: Uuid, + panel_kind: PanelKind, + window: &mut Window, + cx: &mut Context, + ) -> bool { + let phase = crate::collab::phase_for_panel(panel_id, cx); + if !collaboration_phase_blocks_close(phase.as_ref()) { + return false; + } + + let detail = collaboration_leave_detail(phase.as_ref()); + let save_prompt = collaboration_save_prompt_for_panel(&panel_kind, cx); + let answer = window.prompt( + PromptLevel::Warning, + "Leave the collaboration session?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ); + let window_handle = window.window_handle(); + cx.spawn(async move |workspace, cx| { + if !matches!(answer.await, Ok(0)) { + return; + } + + let resolution = resolve_collaboration_save(panel_kind, save_prompt, window_handle, cx).await; + let CollaborationCloseResolution::Proceed { discard } = resolution else { + return; + }; + if let Some(panel) = discard { + panel.discard(cx); + } + + let _ = window_handle.update(cx, |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| { + workspace.leave_collaboration_on_panel(panel_id, cx); + workspace.remove_document_panel(panel_id, window, cx); + }); + }); + }) + .detach(); + true + } + + fn request_close_window_with_collaboration(&mut self, window: &mut Window, cx: &mut Context) -> bool { + let collaboration_panels = self.collaboration_close_panels(cx); + if collaboration_panels.is_empty() { + return false; + } + + let collaboration_ids = collaboration_panels + .iter() + .map(|panel| panel.id) + .collect::>(); + let dirty_panels = self + .dirty_panels(cx) + .into_iter() + .filter(|panel| !collaboration_ids.contains(&panel.panel_id(cx))) + .collect::>(); + + let answer = if collaboration_panels.len() == 1 { + let phase = crate::collab::phase_for_panel(collaboration_panels[0].id, cx); + window.prompt( + PromptLevel::Warning, + "Leave the collaboration session and quit?", + Some(&collaboration_leave_detail(phase.as_ref())), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + } else { + let detail = format!( + "You're in {} collaboration sessions. Leave all and quit?", + collaboration_panels.len() + ); + window.prompt( + PromptLevel::Warning, + "Leave collaboration sessions?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + }; + let window_handle = window.window_handle(); + + cx.spawn(async move |workspace, cx| { + if !matches!(answer.await, Ok(0)) { + return; + } + + let mut discards = Vec::new(); + for panel in collaboration_panels { + match resolve_collaboration_save(panel.panel, panel.save_prompt, window_handle, cx).await { + CollaborationCloseResolution::Proceed { discard } => { + if let Some(discard) = discard { + discards.push(discard); + } + }, + CollaborationCloseResolution::Cancelled => return, + } + } + + if !resolve_dirty_window_close(dirty_panels, window_handle, cx).await { + return; + } + + for panel in discards { + panel.discard(cx); + } + + let _ = window_handle.update(cx, |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); + crate::collab::shutdown(cx); + window.remove_window(); + }); + }) + .detach(); + true + } + + fn collaboration_close_panels(&self, cx: &App) -> Vec { + self + .document_panels + .iter() + .filter_map(|panel| { + let panel_state = panel.read(cx); + let id = panel_state.id(); + let phase = crate::collab::phase_for_panel(id, cx); + collaboration_phase_blocks_close(phase.as_ref()).then(|| { + let panel_kind = PanelKind::Document { + panel: panel.clone(), + editor: panel_state.editor(), + }; + CollaborationClosePanel { + id, + save_prompt: collaboration_save_prompt_for_panel(&panel_kind, cx), + panel: panel_kind, + } + }) + }) + .collect() + } +} + +#[hotpath::measure_all] +impl PanelKind { + fn panel_id(&self, cx: &App) -> Uuid { + match self { + PanelKind::Document { panel, .. } => panel.read(cx).id(), + PanelKind::Flow { panel, .. } => panel.read(cx).id(), + } + } +} + +#[hotpath::measure] +async fn resolve_collaboration_save( + panel: PanelKind, + prompt: CollaborationSavePrompt, + window_handle: AnyWindowHandle, + cx: &mut gpui::AsyncApp, +) -> CollaborationCloseResolution { + match prompt { + CollaborationSavePrompt::Clean => CollaborationCloseResolution::Proceed { discard: None }, + CollaborationSavePrompt::Dirty | CollaborationSavePrompt::Pathless => { + let answer = match window_handle.update(cx, |_, window, cx| { + let (detail, save_label) = match prompt { + CollaborationSavePrompt::Pathless => ("This shared tab has no saved file.", "Save As..."), + CollaborationSavePrompt::Dirty => ("This document has unsaved changes.", "Save"), + CollaborationSavePrompt::Clean => unreachable!(), + }; + window.prompt( + PromptLevel::Warning, + "Save changes before closing?", + Some(detail), + &[PromptButton::ok(save_label), PromptButton::new("Don't Save"), PromptButton::cancel("Cancel")], + cx, + ) + }) { + Ok(answer) => answer, + Err(error) => { + show_save_failed(window_handle, cx, format!("failed to show save prompt: {error}")); + return CollaborationCloseResolution::Cancelled; + }, + }; + + match answer.await { + Ok(0) => match panel.save(window_handle, cx).await { + PanelSaveOutcome::Saved => CollaborationCloseResolution::Proceed { discard: None }, + PanelSaveOutcome::Cancelled => CollaborationCloseResolution::Cancelled, + PanelSaveOutcome::Failed(error) => { + show_save_failed(window_handle, cx, error); + CollaborationCloseResolution::Cancelled + }, + }, + Ok(1) => CollaborationCloseResolution::Proceed { discard: Some(panel) }, + _ => CollaborationCloseResolution::Cancelled, + } + }, + } +} + +#[hotpath::measure] +async fn resolve_dirty_window_close( + dirty_panels: Vec, + window_handle: AnyWindowHandle, + cx: &mut gpui::AsyncApp, +) -> bool { + if dirty_panels.is_empty() { + return true; + } + let message = if dirty_panels.len() == 1 { + "This document has unsaved changes." + } else { + "One or more documents have unsaved changes." + }; + let answer = match window_handle.update(cx, |_, window, cx| { + window.prompt( + PromptLevel::Warning, + "Save changes before closing?", + Some(message), + &[PromptButton::ok("Save"), PromptButton::new("Don't Save"), PromptButton::cancel("Cancel")], + cx, + ) + }) { + Ok(answer) => answer, + Err(error) => { + show_save_failed(window_handle, cx, format!("failed to show save prompt: {error}")); + return false; + }, + }; + + match answer.await { + Ok(0) => { + for panel in dirty_panels { + match panel.save(window_handle, cx).await { + PanelSaveOutcome::Saved => {}, + PanelSaveOutcome::Cancelled => return false, + PanelSaveOutcome::Failed(error) => { + show_save_failed(window_handle, cx, error); + return false; + }, + } + } + true + }, + Ok(1) => { + for panel in dirty_panels { + panel.discard(cx); + } + true + }, + _ => false, + } +} + +#[hotpath::measure] +fn collaboration_save_prompt_for_panel(panel: &PanelKind, cx: &App) -> CollaborationSavePrompt { + match panel { + PanelKind::Document { editor, .. } => { + let editor = editor.read(cx); + if editor.document_path().is_none() { + CollaborationSavePrompt::Pathless + } else if editor.has_unsaved_changes() { + CollaborationSavePrompt::Dirty + } else { + CollaborationSavePrompt::Clean + } + }, + PanelKind::Flow { editor, .. } => { + let editor = editor.read(cx); + if editor.document_path().is_none() { + CollaborationSavePrompt::Pathless + } else if editor.has_unsaved_changes() { + CollaborationSavePrompt::Dirty + } else { + CollaborationSavePrompt::Clean + } + }, + } +} + +#[hotpath::measure] +fn collaboration_phase_blocks_close(phase: Option<&crate::collab::SessionPhase>) -> bool { + phase.is_some_and(|phase| !matches!(phase, crate::collab::SessionPhase::Detached(_))) +} + +#[hotpath::measure] +fn collaboration_leave_detail(phase: Option<&crate::collab::SessionPhase>) -> String { + match phase { + Some(crate::collab::SessionPhase::Attached(attachment)) if attachment.peers_present == 0 => { + "This tab is live, but no other people are present right now.".to_string() + }, + Some(crate::collab::SessionPhase::Attached(attachment)) => format!( + "This tab is live with {} other {}.", + attachment.peers_present, + if attachment.peers_present == 1 { "person" } else { "people" } + ), + Some(crate::collab::SessionPhase::Creating) => "This tab is starting a collaboration session.".to_string(), + Some(crate::collab::SessionPhase::Joining(_)) => "This tab is joining a collaboration session.".to_string(), + _ => "This tab is connected to a collaboration session.".to_string(), + } +} diff --git a/crates/flowstate/src/workspace/workspace/documents.rs b/crates/flowstate/src/workspace/workspace/documents.rs index 8e225b38..a7879e2c 100644 --- a/crates/flowstate/src/workspace/workspace/documents.rs +++ b/crates/flowstate/src/workspace/workspace/documents.rs @@ -89,6 +89,7 @@ impl Workspace { autosave_enabled: load_autosave(), autosave_document_generations: HashMap::new(), autosave_flow_in_flight: HashSet::new(), + collaboration_dialog: None, file_search_overlay: None, tub_root: None, tub_index: None, @@ -694,6 +695,9 @@ impl Workspace { else { return; }; + if self.close_collaboration_document_panel(panel_id, panel_kind.clone(), window, cx) { + return; + } if !panel_kind.is_dirty(cx) { self.remove_document_panel(panel_id, window, cx); return; @@ -734,6 +738,9 @@ impl Workspace { } fn request_close_window(&mut self, window: &mut Window, cx: &mut Context) { + if self.request_close_window_with_collaboration(window, cx) { + return; + } let dirty_panels = self.dirty_panels(cx); if dirty_panels.is_empty() { self.leave_all_collaboration_sessions(cx); diff --git a/crates/flowstate/src/workspace/workspace/keybindings.rs b/crates/flowstate/src/workspace/workspace/keybindings.rs index 9cfed3fb..9578568f 100644 --- a/crates/flowstate/src/workspace/workspace/keybindings.rs +++ b/crates/flowstate/src/workspace/workspace/keybindings.rs @@ -52,10 +52,18 @@ impl Workspace { self.close_active_document(window, cx); true }, + CommandId::ShareDocument => { + self.open_collaboration_dialog(window, cx); + true + }, + CommandId::JoinSession => { + self.open_join_collaboration_dialog(window, cx); + true + }, CommandId::StartCollaboration => self.start_collaboration_on_active_document(cx).is_some(), CommandId::CopyCollaborationTicket => self.copy_active_collaboration_ticket(window, cx), CommandId::JoinCollaborationFromClipboard => self.join_collaboration_from_clipboard(window, cx), - CommandId::LeaveCollaboration => self.leave_collaboration_on_active_document(cx), + CommandId::LeaveCollaboration => self.confirm_leave_collaboration_on_active_document(window, cx), CommandId::FindInDocument => self.open_active_document_search_bar(window, cx), CommandId::ZoomIn => { if let Some(editor) = self.active_editor.clone() { @@ -211,7 +219,7 @@ impl Workspace { } fn non_document_keybinding_surface_is_open(&self) -> bool { - self.settings_overlay.is_some() || self.file_search_overlay.is_some() + self.settings_overlay.is_some() || self.file_search_overlay.is_some() || self.collaboration_dialog.is_some() } fn focused_workspace_input_is_focused(&self, window: &Window, cx: &App) -> bool { @@ -229,6 +237,10 @@ impl Workspace { .file_search_overlay .as_ref() .is_some_and(|overlay| overlay.read(cx).focus_handle(cx).is_focused(window)) + || self + .collaboration_dialog + .as_ref() + .is_some_and(|dialog| dialog.read(cx).focus_handle(cx).is_focused(window)) || self .document_panels .iter() diff --git a/crates/flowstate/src/workspace/workspace/mod.rs b/crates/flowstate/src/workspace/workspace/mod.rs index b0399622..6b27bbbc 100644 --- a/crates/flowstate/src/workspace/workspace/mod.rs +++ b/crates/flowstate/src/workspace/workspace/mod.rs @@ -103,6 +103,7 @@ pub struct Workspace { autosave_enabled: bool, autosave_document_generations: HashMap, autosave_flow_in_flight: HashSet, + collaboration_dialog: Option>, file_search_overlay: Option>, tub_root: Option, tub_index: Option>, @@ -261,6 +262,7 @@ impl DocumentStyleSection { } include!("documents.rs"); +include!("collab_prompts.rs"); include!("collab.rs"); include!("workspace_state.rs"); include!("load.rs"); diff --git a/crates/flowstate/src/workspace/workspace/render_documents.rs b/crates/flowstate/src/workspace/workspace/render_documents.rs index 187be03c..fd0b30bb 100644 --- a/crates/flowstate/src/workspace/workspace/render_documents.rs +++ b/crates/flowstate/src/workspace/workspace/render_documents.rs @@ -75,6 +75,7 @@ impl Workspace { .children(tabs.into_iter().map(|tab| { let panel_id = tab.id; let workspace = workspace.clone(); + let collab_phase = crate::collab::phase_for_panel(panel_id, cx); let tab_prefix = h_flex() .ml(px(5.0)) .mr(px(-3.0)) @@ -104,6 +105,9 @@ impl Workspace { .border_color(cx.theme().warning.opacity(0.72)) .child(pin_label), ) + }) + .when_some(collab_phase.as_ref().and_then(|phase| crate::collab::status::tab_badge(phase, cx)), |this, badge| { + this.child(badge) }); let close_button = icon_button(("close-tab", panel_id.as_u128() as u64), AppIcon::Close) .tooltip("Close document") diff --git a/crates/flowstate/src/workspace/workspace/render_status.rs b/crates/flowstate/src/workspace/workspace/render_status.rs index 1f62927a..30f27a3e 100644 --- a/crates/flowstate/src/workspace/workspace/render_status.rs +++ b/crates/flowstate/src/workspace/workspace/render_status.rs @@ -284,7 +284,13 @@ impl Workspace { if matches!(phase, crate::collab::SessionPhase::Detached(_)) { this } else { - this.child(crate::collab::status::status_pill(&phase, cx)) + this.child( + Button::new("collaboration-status-pill") + .text() + .compact() + .child(crate::collab::status::status_pill(&phase, cx)) + .on_click(cx.listener(|workspace, _, window, cx| workspace.open_collaboration_dialog(window, cx))), + ) } }) .when_some(zoom, |this, percent| this.child(self.render_zoom_slider(percent, cx))) diff --git a/crates/flowstate/src/workspace/workspace/render_top_bar.rs b/crates/flowstate/src/workspace/workspace/render_top_bar.rs index aaf3b073..c4a50afb 100644 --- a/crates/flowstate/src/workspace/workspace/render_top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/render_top_bar.rs @@ -2,6 +2,10 @@ impl Workspace { fn render_top_bar(&mut self, _window: &Window, cx: &mut Context) -> impl IntoElement { let workspace = cx.entity().downgrade(); + let active_collaborating = self + .active_document_id + .and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)) + .is_some_and(|phase| !matches!(phase, crate::collab::SessionPhase::Detached(_))); TitleBar::new() .on_close_window(move |_, window, cx| { let _ = workspace.update(cx, |workspace, cx| workspace.request_close_window(window, cx)); @@ -23,6 +27,8 @@ impl Workspace { !self.ribbon_collapsed, !self.toolkit_collapsed, )) + .child(div().flex_1()) + .child(share_top_bar_button(cx, self.active_editor.is_some(), active_collaborating)) .child(settings_top_bar_button(cx)), ) } diff --git a/crates/flowstate/src/workspace/workspace/top_bar.rs b/crates/flowstate/src/workspace/workspace/top_bar.rs index ac3b2ed5..c24f701e 100644 --- a/crates/flowstate/src/workspace/workspace/top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/top_bar.rs @@ -122,18 +122,21 @@ fn collaboration_top_bar_button(cx: &mut Context, has_document: bool) .ghost() .dropdown_menu(move |menu, _, _| { menu - .item(file_menu_item(workspace.clone(), "Start Sharing", !has_document, |workspace, _, cx| { - let _ = workspace.start_collaboration_on_active_document(cx); + .item(file_menu_item(workspace.clone(), "Share / Collaborate...", !has_document, |workspace, window, cx| { + workspace.open_collaboration_dialog(window, cx); })) .item(file_menu_item(workspace.clone(), "Copy Invite Ticket", !has_document, |workspace, window, cx| { workspace.copy_active_collaboration_ticket(window, cx); })) .separator() + .item(file_menu_item(workspace.clone(), "Join Session...", false, |workspace, window, cx| { + workspace.open_join_collaboration_dialog(window, cx); + })) .item(file_menu_item(workspace.clone(), "Join from Clipboard", false, |workspace, window, cx| { workspace.join_collaboration_from_clipboard(window, cx); })) - .item(file_menu_item(workspace.clone(), "Leave Shared Session", !has_document, |workspace, _, cx| { - workspace.leave_collaboration_on_active_document(cx); + .item(file_menu_item(workspace.clone(), "Leave Shared Session", !has_document, |workspace, window, cx| { + workspace.confirm_leave_collaboration_on_active_document(window, cx); })) }), ) @@ -176,6 +179,13 @@ fn file_top_bar_button(has_document: bool, cx: &mut Context) -> impl workspace.save_active_as(window, cx); })) .separator() + .item(file_menu_item(workspace.clone(), "Share Document...", !has_document, |workspace, window, cx| { + workspace.open_collaboration_dialog(window, cx); + })) + .item(file_menu_item(workspace.clone(), "Join Session...", false, |workspace, window, cx| { + workspace.open_join_collaboration_dialog(window, cx); + })) + .separator() .item(file_menu_item(workspace.clone(), "Close File", !has_document, |workspace, window, cx| { workspace.close_active_document(window, cx); })) @@ -246,6 +256,38 @@ fn insert_top_bar_button(cx: &mut Context, has_document: bool) -> imp ) } +#[hotpath::measure] +fn share_top_bar_button(cx: &mut Context, has_document: bool, collaborating: bool) -> impl IntoElement { + let workspace = cx.entity().downgrade(); + let button = Button::new("top-share-document") + .label("Share") + .xsmall() + .ghost() + .disabled(!has_document) + .tooltip("Share / Collaborate") + .on_click(move |_, window, cx| { + let _ = workspace.update(cx, |workspace, cx| workspace.open_collaboration_dialog(window, cx)); + }); + + div() + .h_full() + .flex_none() + .flex() + .items_center() + .justify_center() + .on_mouse_down(MouseButton::Left, |_, _, cx| cx.stop_propagation()) + .child(if collaborating { + button.custom( + ButtonCustomVariant::new(cx) + .foreground(cx.theme().success) + .hover(cx.theme().success.opacity(0.12)) + .active(cx.theme().success.opacity(0.18)), + ) + } else { + button + }) +} + #[hotpath::measure] fn insert_image_from_top_bar(workspace: &WeakEntity, cx: &mut App) { let _ = workspace.update(cx, |workspace, cx| { diff --git a/crates/flowstate/src/workspace/workspace/traits.rs b/crates/flowstate/src/workspace/workspace/traits.rs index 4111d93b..1c61fdcc 100644 --- a/crates/flowstate/src/workspace/workspace/traits.rs +++ b/crates/flowstate/src/workspace/workspace/traits.rs @@ -45,6 +45,7 @@ impl Render for Workspace { this.child(self.render_settings_overlay(overlay, cx)) }) .when_some(self.file_search_overlay.clone(), |this, overlay| this.child(overlay)) + .when_some(self.collaboration_dialog.clone(), |this, dialog| this.child(dialog)) .when_some(self.outline_context_menu.as_ref(), |this, ctx| { let workspace = cx.entity().downgrade(); let menu = ctx.menu_view.clone(); diff --git a/crates/flowstate/src/workspace/workspace/window.rs b/crates/flowstate/src/workspace/workspace/window.rs index 663c2385..eae10e7d 100644 --- a/crates/flowstate/src/workspace/workspace/window.rs +++ b/crates/flowstate/src/workspace/workspace/window.rs @@ -9,6 +9,90 @@ pub fn install_workspace_close_prompt(workspace: Entity, window: &mut return true; } + let collaboration_panels = workspace.read(cx).collaboration_close_panels(cx); + if !collaboration_panels.is_empty() { + if prompt_open.get() { + return false; + } + prompt_open.set(true); + + let collaboration_ids = collaboration_panels + .iter() + .map(|panel| panel.id) + .collect::>(); + let dirty_panels = workspace + .read(cx) + .dirty_panels(cx) + .into_iter() + .filter(|panel| !collaboration_ids.contains(&panel.panel_id(cx))) + .collect::>(); + let answer = if collaboration_panels.len() == 1 { + let phase = crate::collab::phase_for_panel(collaboration_panels[0].id, cx); + window.prompt( + PromptLevel::Warning, + "Leave the collaboration session and quit?", + Some(&collaboration_leave_detail(phase.as_ref())), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + } else { + let detail = format!( + "You're in {} collaboration sessions. Leave all and quit?", + collaboration_panels.len() + ); + window.prompt( + PromptLevel::Warning, + "Leave collaboration sessions?", + Some(&detail), + &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], + cx, + ) + }; + let prompt_open = prompt_open.clone(); + let allow_close = allow_close.clone(); + let workspace = workspace.clone(); + + cx.spawn(async move |cx| { + let mut should_close = false; + if matches!(answer.await, Ok(0)) { + let mut discards = Vec::new(); + let mut ok = true; + for panel in collaboration_panels { + match resolve_collaboration_save(panel.panel, panel.save_prompt, window_handle, cx).await { + CollaborationCloseResolution::Proceed { discard } => { + if let Some(discard) = discard { + discards.push(discard); + } + }, + CollaborationCloseResolution::Cancelled => { + ok = false; + break; + }, + } + } + if ok && resolve_dirty_window_close(dirty_panels, window_handle, cx).await { + for panel in discards { + panel.discard(cx); + } + should_close = true; + } + } + + prompt_open.set(false); + if should_close { + allow_close.set(true); + let _ = window_handle.update(cx, |_, window, cx| { + workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); + crate::collab::shutdown(cx); + window.remove_window(); + }); + } + }) + .detach(); + + return false; + } + let dirty_panels = workspace.read(cx).dirty_panels(cx); if dirty_panels.is_empty() { workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); diff --git a/crates/gpui-flowtext/src/document/blocks.rs b/crates/gpui-flowtext/src/document/blocks.rs index de88397a..f26c0997 100644 --- a/crates/gpui-flowtext/src/document/blocks.rs +++ b/crates/gpui-flowtext/src/document/blocks.rs @@ -24,6 +24,16 @@ pub struct AssetRecord { pub bytes: Arc>, } +pub const IMAGE_LOADING_PLACEHOLDER_WIDTH_PX: f32 = 240.0; +pub const IMAGE_LOADING_PLACEHOLDER_HEIGHT_PX: f32 = 160.0; + +impl AssetRecord { + #[must_use] + pub fn is_loading_placeholder(&self) -> bool { + self.bytes.is_empty() + } +} + #[derive(Clone, Debug, PartialEq, Eq)] pub struct ImageBlock { pub asset_id: AssetId, diff --git a/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs b/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs index 09d57050..4c76e189 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/caret_movement.rs @@ -115,6 +115,7 @@ impl RichTextEditor { let _ = self.paragraph_item_sizes(window, cx); self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } @@ -199,6 +200,7 @@ impl RichTextEditor { self.goal_x = Some(used_goal_x); self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } @@ -256,6 +258,7 @@ impl RichTextEditor { self.goal_x = Some(goal_x); self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs index 417a6f67..e288ab42 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs @@ -47,22 +47,33 @@ impl RichTextEditor { CollabPatch::ReplaceObjectBlock { row, block } => { let mut blocks = collab_structural_blocks_from_document(&self.document); if *row < blocks.len() { + if selected_block_in_range(self.selected_block, *row..row.saturating_add(1)) { + self.clear_remote_object_editing_state(); + } blocks[*row] = block.clone(); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + clamp_selection_to_document(&self.document, &mut self.selection); extend_invalidation(invalidation, 0..self.document.paragraphs.len()); } }, CollabPatch::InsertBlocks { row, blocks: inserted } => { let mut blocks = collab_structural_blocks_from_document(&self.document); let row = (*row).min(blocks.len()); + if selected_block_ix(self.selected_block).is_some_and(|block_ix| block_ix >= row) { + self.clear_remote_object_editing_state(); + } blocks.splice(row..row, inserted.iter().cloned()); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); + clamp_selection_to_document(&self.document, &mut self.selection); extend_invalidation(invalidation, 0..self.document.paragraphs.len()); }, CollabPatch::DeleteBlocks { row, count } => { let mut blocks = collab_structural_blocks_from_document(&self.document); let start = (*row).min(blocks.len()); let end = start.saturating_add(*count).min(blocks.len()); + if selected_block_ix(self.selected_block).is_some_and(|block_ix| block_ix >= start) { + self.clear_remote_object_editing_state(); + } blocks.drain(start..end); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); clamp_selection_to_document(&self.document, &mut self.selection); @@ -71,6 +82,11 @@ impl RichTextEditor { CollabPatch::MoveBlock { from, to } => { let mut blocks = collab_structural_blocks_from_document(&self.document); if *from < blocks.len() { + let first = (*from).min(*to); + let last = (*from).max(*to); + if selected_block_ix(self.selected_block).is_some_and(|block_ix| (first..=last).contains(&block_ix)) { + self.clear_remote_object_editing_state(); + } let block = blocks.remove(*from); blocks.insert((*to).min(blocks.len()), block); rebuild_document_from_collab_structural_blocks(&mut self.document, blocks); @@ -83,6 +99,33 @@ impl RichTextEditor { }, } } + + fn clear_remote_object_editing_state(&mut self) { + self.selected_block = None; + self.image_resize_drag = None; + self.table_column_resize_drag = None; + self.table_cell_block_ix = 0; + self.table_cell_anchor = 0; + self.table_cell_caret = 0; + self.equation_source_anchor = 0; + self.equation_source_caret = 0; + } +} + +#[hotpath::measure] +fn selected_block_ix(selection: Option) -> Option { + match selection { + Some(BlockSelection::Image(block_ix) + | BlockSelection::Equation(block_ix) + | BlockSelection::Table(block_ix) + | BlockSelection::TableCell { block_ix, .. }) => Some(block_ix), + None => None, + } +} + +#[hotpath::measure] +fn selected_block_in_range(selection: Option, range: Range) -> bool { + selected_block_ix(selection).is_some_and(|block_ix| range.contains(&block_ix)) } #[hotpath::measure] diff --git a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs index 99bdde75..147bd3f7 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs @@ -262,8 +262,14 @@ impl RichTextEditor { self.collab_undo_redirect = hook; } - pub fn set_recovery_path(&mut self, path: Option) { + pub fn clear_collab_history(&mut self) { + self.undo_stack.clear(); + self.redo_stack.clear(); + } + + pub fn set_recovery_path(&mut self, path: Option, cx: &mut Context) { self.recovery_path = path; + self.schedule_recovery_write(cx); } pub fn collaboration_role(&self) -> Option { diff --git a/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs b/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs index d4e4c77a..cc0ef168 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs @@ -19,12 +19,17 @@ fn reserved_object_frame(document: &Document, row_size: Size, selected: fn image_object_frame(document: &Document, image: &ImageBlock, asset: &AssetRecord, row_size: Size, selected: bool) -> gpui::Div { let available_width = (row_size.width - document.theme.pageless_inset_x * 2.0).max(px(1.0)); let intrinsic = image_asset_intrinsic_size(asset); - let object_width = match image.sizing { - ImageSizing::Fixed { width_px, .. } => px(width_px as f32).min(available_width), - ImageSizing::FitWidth => available_width, - ImageSizing::Intrinsic => intrinsic - .map(|(width, _)| width.min(available_width)) - .unwrap_or(available_width), + let loading = asset.is_loading_placeholder(); + let object_width = if loading { + px(IMAGE_LOADING_PLACEHOLDER_WIDTH_PX).min(available_width) + } else { + match image.sizing { + ImageSizing::Fixed { width_px, .. } => px(width_px as f32).min(available_width), + ImageSizing::FitWidth => available_width, + ImageSizing::Intrinsic => intrinsic + .map(|(width, _)| width.min(available_width)) + .unwrap_or(available_width), + } }; let object_height = (row_size.height - document.theme.paragraph_after).max(px(1.0)); let left_margin = document.theme.pageless_inset_x @@ -41,13 +46,22 @@ fn image_object_frame(document: &Document, image: &ImageBlock, asset: &AssetReco .mr(document.theme.pageless_inset_x) .mb(document.theme.paragraph_after) .overflow_hidden() - .bg(rgb(0xffffff)) + .bg(if loading { rgb(0xf3f4f6) } else { rgb(0xffffff) }) .border_1() - .border_color(if selected { rgb(0x0969da) } else { rgb(0xffffff) }) + .border_color(if selected { + rgb(0x0969da) + } else if loading { + rgb(0xd0d7de) + } else { + rgb(0xffffff) + }) } #[hotpath::measure] fn image_asset_intrinsic_size(asset: &AssetRecord) -> Option<(Pixels, Pixels)> { + if asset.is_loading_placeholder() { + return None; + } let size = imagesize::blob_size(asset.bytes.as_ref()).ok()?; if size.width == 0 || size.height == 0 { return None; diff --git a/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs b/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs index 9faa7b9e..3fa523ea 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/render_blocks.rs @@ -13,6 +13,9 @@ fn render_image_block( .child("Missing image") .into_any_element(); }; + if asset.is_loading_placeholder() { + return render_loading_image_placeholder(document, image, asset, row_size, selected, editor, block_ix); + } let Some(format) = ImageFormat::from_mime_type(asset.mime_type.as_ref()) else { return reserved_object_frame(document, row_size, selected) .child("Unsupported image") @@ -37,6 +40,34 @@ fn render_image_block( .into_any_element() } +#[hotpath::measure] +fn render_loading_image_placeholder( + document: &Document, + image: &ImageBlock, + asset: &AssetRecord, + row_size: Size, + selected: bool, + editor: Entity, + block_ix: usize, +) -> gpui::AnyElement { + image_object_frame(document, image, asset, row_size, selected) + .child( + div() + .size_full() + .flex() + .flex_col() + .items_center() + .justify_center() + .gap_1() + .text_sm() + .text_color(rgb(0x6b7280)) + .child(div().text_xs().text_color(rgb(0x9ca3af)).child("...")) + .child("Loading image"), + ) + .when(selected, |this| this.children(image_resize_handles(editor, block_ix))) + .into_any_element() +} + #[hotpath::measure] fn image_resize_handles(editor: Entity, block_ix: usize) -> Vec { [ @@ -254,4 +285,3 @@ type EquationRenderCache = FxHashMap>, Stri static EQUATION_SVG_CACHE: OnceLock> = OnceLock::new(); static EQUATION_PNG_CACHE: OnceLock> = OnceLock::new(); - diff --git a/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs b/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs index 9d45f715..75d9183e 100644 --- a/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs +++ b/crates/gpui-flowtext/src/rich_text/layout/block_layout.rs @@ -274,6 +274,14 @@ fn structural_block_bounds(document: &Document, width: Pixels, y: Pixels, height #[hotpath::measure] fn image_placeholder_height(document: &Document, image: &ImageBlock, width: Pixels) -> Pixels { + if document + .assets + .assets + .get(&image.asset_id) + .is_some_and(AssetRecord::is_loading_placeholder) + { + return px(IMAGE_LOADING_PLACEHOLDER_HEIGHT_PX); + } let available_width = (width - document.theme.pageless_inset_x * 2.0).max(px(1.0)); let intrinsic = image_intrinsic_size(document, image); match image.sizing { @@ -295,6 +303,9 @@ pub(super) fn image_layout_height_for_test(document: &Document, image: &ImageBlo #[hotpath::measure] fn image_intrinsic_size(document: &Document, image: &ImageBlock) -> Option<(Pixels, Pixels)> { let asset = document.assets.assets.get(&image.asset_id)?; + if asset.is_loading_placeholder() { + return None; + } let size = imagesize::blob_size(asset.bytes.as_ref()).ok()?; if size.width == 0 || size.height == 0 { return None; diff --git a/helpers/docs/collab_qa.md b/helpers/docs/collab_qa.md new file mode 100644 index 00000000..3e22751b --- /dev/null +++ b/helpers/docs/collab_qa.md @@ -0,0 +1,59 @@ +# Flowstate Collaboration QA + +Manual QA for P2P `.db8` collaboration. Run with two or three app instances using separate working directories. Use fresh documents unless a scenario says otherwise. + +## Setup + +1. Start instance A and open or create a `.db8` document. +2. Use `Share / Collaborate...` to start a session and copy an invite. +3. Start instance B, use `Join Collaboration Session...`, paste the invite, and verify a new shared tab opens. +4. For three-peer cases, repeat join from instance C using a fresh invite from either A or B. + +## Core Sync + +1. Type in A and verify B/C update live. +2. Type in B and verify A/C update live. +3. Paste multi-paragraph rich text and verify convergence. +4. Exercise multibyte text with the fixture `a\u{00e9}\u{1f30d}\u{2028}x`. +5. Apply paragraph styles, run styles, highlights, underline, and strikethrough from different peers. +6. Insert, move, replace, and delete image/equation/table blocks. +7. Paste an image on A; verify B/C show a loading placeholder then the image. +8. Paste a different image on B; verify A/C can serve it after B has it. + +## Presence + +1. Move the caret with keyboard arrows in A; verify B sees the remote caret move. +2. Select text in B; verify A sees the caret position update. +3. Leave C or kill it; verify presence ages out within 30 seconds. +4. Reopen the share dialog and verify participant count/status remains sensible. + +## Undo And Conflicts + +1. Type several edits on A, then undo and redo; verify only A's edits are undone/redone and all peers converge. +2. Interleave A and B edits in the same paragraph, then undo on A and B independently. +3. Simultaneously edit the same table cell from A and B; verify the block converges with last-writer-wins behavior. +4. Simultaneously split a paragraph on A while B types later in the paragraph; verify no data loss and convergence. + +## Network And Recovery + +1. Disconnect B's network or block traffic, keep editing on B, then reconnect; verify anti-entropy catches up. +2. Let A leave while B and C remain; verify B and C continue editing and converging. +3. Force a large paste above the gossip inline limit; verify blob pull or later anti-entropy convergence. +4. Quit or crash a pathless joined tab; verify a recovery file exists under the temp `flowstate-collab-recovery` directory and opens as a local document on recovery. +5. Test a stale/dead invite; verify join fails with actionable copy. + +## Prompts And Leave + +1. Click `Leave session`; cancel, then leave. Verify the tab remains open as a local document. +2. Close an attached clean path-backed tab; verify leave prompt, then close. +3. Close an attached dirty path-backed tab; verify leave prompt, save prompt, and cancel at each step keeps the session attached. +4. Close a pathless attached tab; verify leave prompt, `Save As...` / `Don't Save` / `Cancel` behavior. +5. Quit with multiple attached tabs; verify the combined leave prompt, then per-tab save prompts. Cancel anywhere must abort quit. + +## Compatibility And Error Cases + +1. Paste a malformed invite; verify inline parse error and no network dial. +2. Use an unsupported-version invite if available; verify clear rejection. +3. Try joining a session already open in the same app; verify it is rejected. +4. Kill the inviter before join; verify join fails without modifying existing documents. +5. Confirm saved files are plain `.db8` snapshots and do not contain CRDT history. From 3a8bc262d538900f4d1fba078775d532f43a49a4 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Sat, 13 Jun 2026 18:31:37 -0500 Subject: [PATCH 08/45] run cargo fmt --- crates/flowstate-collab/src/ids.rs | 4 +- crates/flowstate-collab/src/net/mod.rs | 61 ++++++++-- crates/flowstate-collab/src/net/runtime.rs | 27 ++++- crates/flowstate-collab/src/net/swarm.rs | 7 +- crates/flowstate-collab/src/presence.rs | 12 +- crates/flowstate-collab/src/remote_apply.rs | 15 +-- crates/flowstate-collab/src/schema.rs | 5 +- crates/flowstate-collab/tests/anti_entropy.rs | 18 +-- crates/flowstate-collab/tests/convergence.rs | 24 +++- crates/flowstate-collab/tests/local_apply.rs | 29 +++-- crates/flowstate-collab/tests/projection.rs | 21 +++- crates/flowstate-collab/tests/translation.rs | 47 ++++---- crates/flowstate/src/collab/asset_transfer.rs | 6 +- crates/flowstate/src/collab/mod.rs | 98 ++++++++------- crates/flowstate/src/collab/presence_view.rs | 28 ++--- crates/flowstate/src/collab/session.rs | 99 +++++++++------- crates/flowstate/src/collab/session_io.rs | 40 +++++-- .../flowstate/src/collab/session_presence.rs | 10 +- crates/flowstate/src/collab/session_timers.rs | 15 ++- crates/flowstate/src/collab/share_dialog.rs | 112 +++++++++++------- crates/flowstate/src/collab/status.rs | 23 +--- 21 files changed, 408 insertions(+), 293 deletions(-) diff --git a/crates/flowstate-collab/src/ids.rs b/crates/flowstate-collab/src/ids.rs index df1df075..5eb2225f 100644 --- a/crates/flowstate-collab/src/ids.rs +++ b/crates/flowstate-collab/src/ids.rs @@ -6,9 +6,7 @@ pub type PeerId = iroh::EndpointId; pub const SESSION_ID_LEN: usize = 32; -pub const PALETTE: [u32; 8] = [ - 0x3b82f6, 0xef4444, 0x22c55e, 0xf59e0b, 0x8b5cf6, 0x06b6d4, 0xec4899, 0x84cc16, -]; +pub const PALETTE: [u32; 8] = [0x3b82f6, 0xef4444, 0x22c55e, 0xf59e0b, 0x8b5cf6, 0x06b6d4, 0xec4899, 0x84cc16]; #[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize, Deserialize)] pub struct SessionId([u8; SESSION_ID_LEN]); diff --git a/crates/flowstate-collab/src/net/mod.rs b/crates/flowstate-collab/src/net/mod.rs index 9cef1b23..dcd0aff0 100644 --- a/crates/flowstate-collab/src/net/mod.rs +++ b/crates/flowstate-collab/src/net/mod.rs @@ -8,7 +8,11 @@ use anyhow::Result; use async_channel::Sender; use iroh::{EndpointAddr, EndpointId}; -use crate::{ids::{BlobId, SessionId}, proto_direct::AssetBytes, proto_gossip::GossipMsg}; +use crate::{ + ids::{BlobId, SessionId}, + proto_direct::AssetBytes, + proto_gossip::GossipMsg, +}; use self::direct::DirectSessionHandler; @@ -25,16 +29,51 @@ pub enum PublishPayload { #[derive(Debug)] pub enum NetCommand { EnsureUp, - RegisterDirectHandler { session: SessionId, handler: DirectSessionHandler }, - CreateSession { session: SessionId, reply: Reply> }, - JoinSession { session: SessionId, bootstrap: Vec }, - LeaveSession { session: SessionId }, - Publish { session: SessionId, payload: PublishPayload }, - PullUpdates { session: SessionId, candidates: Vec, our_vv: Vec, reply: Reply>> }, - PullSnapshot { session: SessionId, candidates: Vec, reply: Reply>> }, - PullBlob { session: SessionId, candidates: Vec, blob: BlobId, reply: Reply>> }, - PullAsset { session: SessionId, candidates: Vec, asset: u128, reply: Reply> }, - MintTicketAddr { reply: Reply }, + RegisterDirectHandler { + session: SessionId, + handler: DirectSessionHandler, + }, + CreateSession { + session: SessionId, + reply: Reply>, + }, + JoinSession { + session: SessionId, + bootstrap: Vec, + }, + LeaveSession { + session: SessionId, + }, + Publish { + session: SessionId, + payload: PublishPayload, + }, + PullUpdates { + session: SessionId, + candidates: Vec, + our_vv: Vec, + reply: Reply>>, + }, + PullSnapshot { + session: SessionId, + candidates: Vec, + reply: Reply>>, + }, + PullBlob { + session: SessionId, + candidates: Vec, + blob: BlobId, + reply: Reply>>, + }, + PullAsset { + session: SessionId, + candidates: Vec, + asset: u128, + reply: Reply>, + }, + MintTicketAddr { + reply: Reply, + }, Shutdown, } diff --git a/crates/flowstate-collab/src/net/runtime.rs b/crates/flowstate-collab/src/net/runtime.rs index f09ca235..5c574b2b 100644 --- a/crates/flowstate-collab/src/net/runtime.rs +++ b/crates/flowstate-collab/src/net/runtime.rs @@ -31,9 +31,7 @@ pub fn start() -> Result<(CommandSender, EventReceiver)> { let runtime = spawn_runtime()?; let set_failed = RUNTIME.set(runtime.clone()).is_err(); - if set_failed - && let Some(runtime) = RUNTIME.get() - { + if set_failed && let Some(runtime) = RUNTIME.get() { return Ok((runtime.commands.clone(), runtime.events.clone())); } Ok((runtime.commands, runtime.events)) @@ -102,7 +100,14 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha direct_state.attach_session(session).await; replace_swarm( &mut swarms, - SwarmHandle::spawn(endpoint.clone(), gossip.clone(), direct_state.clone(), session, Vec::new(), evt_tx.clone())?, + SwarmHandle::spawn( + endpoint.clone(), + gossip.clone(), + direct_state.clone(), + session, + Vec::new(), + evt_tx.clone(), + )?, ) .await; let _ = reply @@ -143,11 +148,21 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullBlob { session, candidates, blob, reply } => { + NetCommand::PullBlob { + session, + candidates, + blob, + reply, + } => { let result = direct::pull_with_fallback(DirectRequest::Blob { session, blob }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullAsset { session, candidates, asset, reply } => { + NetCommand::PullAsset { + session, + candidates, + asset, + reply, + } => { let result = direct::pull_with_fallback(DirectRequest::Asset { session, asset }, candidates, DIRECT_PULL_TIMEOUT) .await .map(|bytes| AssetBytes { bytes }); diff --git a/crates/flowstate-collab/src/net/swarm.rs b/crates/flowstate-collab/src/net/swarm.rs index 6cd10bb9..84a1a427 100644 --- a/crates/flowstate-collab/src/net/swarm.rs +++ b/crates/flowstate-collab/src/net/swarm.rs @@ -97,12 +97,7 @@ async fn run_session( Ok(()) } -async fn publish( - sender: &GossipSender, - direct_state: &DirectServeState, - session: SessionId, - payload: PublishPayload, -) -> Result<()> { +async fn publish(sender: &GossipSender, direct_state: &DirectServeState, session: SessionId, payload: PublishPayload) -> Result<()> { let (message, neighbors_only) = match payload { PublishPayload::Update(bytes) => (update_message(direct_state, session, bytes).await?, false), PublishPayload::Presence(bytes) => (GossipMsg::Presence(bytes), false), diff --git a/crates/flowstate-collab/src/presence.rs b/crates/flowstate-collab/src/presence.rs index b3cdce47..9f1ff442 100644 --- a/crates/flowstate-collab/src/presence.rs +++ b/crates/flowstate-collab/src/presence.rs @@ -1,7 +1,10 @@ //! Presence state and roster derivation. use anyhow::{Context as _, Result}; -use loro::{LoroValue, Subscription, awareness::{EphemeralStore, EphemeralStoreEvent}}; +use loro::{ + LoroValue, Subscription, + awareness::{EphemeralStore, EphemeralStoreEvent}, +}; use serde::{Deserialize, Serialize}; use crate::ids::{PALETTE, PeerId, SessionId}; @@ -97,7 +100,12 @@ impl PresenceStore { .into_iter() .filter_map(|(key, value)| roster_entry_from_value(key, value)) .collect::>(); - entries.sort_by(|left, right| left.name.cmp(&right.name).then_with(|| left.key.cmp(&right.key))); + entries.sort_by(|left, right| { + left + .name + .cmp(&right.name) + .then_with(|| left.key.cmp(&right.key)) + }); entries } diff --git a/crates/flowstate-collab/src/remote_apply.rs b/crates/flowstate-collab/src/remote_apply.rs index 24bdc4d6..e29e3ac2 100644 --- a/crates/flowstate-collab/src/remote_apply.rs +++ b/crates/flowstate-collab/src/remote_apply.rs @@ -43,13 +43,7 @@ impl RemoteApplier<'_> { Ok(patches) } - fn apply_text_diff( - &self, - document: &Document, - target: &ContainerID, - delta: &[TextDelta], - patches: &mut Vec, - ) -> Result<()> { + fn apply_text_diff(&self, document: &Document, target: &ContainerID, delta: &[TextDelta], patches: &mut Vec) -> Result<()> { let Some(row_ix) = self.binding.by_container.get(target).copied() else { return Ok(()); }; @@ -235,12 +229,7 @@ fn structural_block_for_insert(input: &InputBlock) -> CollabStructuralBlock { } } -fn binding_row_from_insert( - map: LoroMap, - input: &InputBlock, - block_id: BlockId, - paragraph_id: Option, -) -> Result { +fn binding_row_from_insert(map: LoroMap, input: &InputBlock, block_id: BlockId, paragraph_id: Option) -> Result { let text = if matches!(input, InputBlock::Paragraph(_)) { Some(text_child(&map)?) } else { diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs index 925e1290..2b4675e8 100644 --- a/crates/flowstate-collab/src/schema.rs +++ b/crates/flowstate-collab/src/schema.rs @@ -202,7 +202,10 @@ pub fn input_runs_from_delta(delta: &[TextDelta]) -> Vec { last.text.push_str(insert); continue; } - runs.push(InputRun { text: insert.clone(), styles }); + runs.push(InputRun { + text: insert.clone(), + styles, + }); } runs } diff --git a/crates/flowstate-collab/tests/anti_entropy.rs b/crates/flowstate-collab/tests/anti_entropy.rs index 932b678d..81f58232 100644 --- a/crates/flowstate-collab/tests/anti_entropy.rs +++ b/crates/flowstate-collab/tests/anti_entropy.rs @@ -29,10 +29,7 @@ mod tests { state.finish_pull(); assert_eq!( state.consider_digest(peer, session, VersionVectorRelation::Concurrent, vec![6]), - GapAction::Pull { - from: peer, - our_vv: vec![6], - } + GapAction::Pull { from: peer, our_vv: vec![6] } ); } @@ -69,10 +66,7 @@ mod tests { ); assert_eq!( state.consider_digest(peer, session, VersionVectorRelation::SenderHasMissingOps, vec![2]), - GapAction::Pull { - from: peer, - our_vv: vec![2], - } + GapAction::Pull { from: peer, our_vv: vec![2] } ); } @@ -83,13 +77,7 @@ mod tests { let mut state = AntiEntropyState::new(session, Instant::now()); assert_eq!(state.on_lagged(None, vec![1]), GapAction::None); - assert_eq!( - state.on_lagged(Some(peer), vec![2]), - GapAction::Pull { - from: peer, - our_vv: vec![2], - } - ); + assert_eq!(state.on_lagged(Some(peer), vec![2]), GapAction::Pull { from: peer, our_vv: vec![2] }); assert_eq!(state.on_lagged(Some(peer), vec![3]), GapAction::None); } diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs index 3e6cf790..f39a5b17 100644 --- a/crates/flowstate-collab/tests/convergence.rs +++ b/crates/flowstate-collab/tests/convergence.rs @@ -35,7 +35,9 @@ mod tests { peer_a.set_run_styles( 0, "a".len().."a".len() + inserted, - RunStyles::default().with(RunStyle::Highlight(4)).with_direct_underline(), + RunStyles::default() + .with(RunStyle::Highlight(4)) + .with_direct_underline(), ); peer_b.insert_text(1, "second ".len(), "Bé", RunStyles::default().with_strikethrough()); @@ -72,7 +74,9 @@ mod tests { fn from_snapshot(session: SessionId, snapshot: &[u8]) -> Self { let loro = schema::new_configured_doc(); - loro.import(snapshot).expect("snapshot import should succeed"); + loro + .import(snapshot) + .expect("snapshot import should succeed"); projection::verify_lineage(&loro, session).expect("lineage should match"); let document = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); let binding = DocBinding::build(&loro, &document).expect("binding should build"); @@ -83,7 +87,10 @@ mod tests { let updates = Arc::new(Mutex::new(Vec::new())); let captured = updates.clone(); let subscription = loro.subscribe_local_update(Box::new(move |bytes| { - captured.lock().expect("update capture lock should not be poisoned").push(bytes.clone()); + captured + .lock() + .expect("update capture lock should not be poisoned") + .push(bytes.clone()); true })); Self { @@ -96,11 +103,18 @@ mod tests { } fn snapshot(&self) -> Vec { - self.loro.export(ExportMode::Snapshot).expect("snapshot export should succeed") + self + .loro + .export(ExportMode::Snapshot) + .expect("snapshot export should succeed") } fn updates(&self) -> Vec> { - self.updates.lock().expect("update capture lock should not be poisoned").clone() + self + .updates + .lock() + .expect("update capture lock should not be poisoned") + .clone() } fn insert_text(&mut self, paragraph_ix: usize, byte: usize, text: &str, styles: RunStyles) { diff --git a/crates/flowstate-collab/tests/local_apply.rs b/crates/flowstate-collab/tests/local_apply.rs index 27d02ea2..ad066346 100644 --- a/crates/flowstate-collab/tests/local_apply.rs +++ b/crates/flowstate-collab/tests/local_apply.rs @@ -19,20 +19,25 @@ mod tests { projection::populate_from_document(&loro, SessionId::from_bytes([9; 32]), "local-apply", &document).expect("populate should succeed"); let mut binding = DocBinding::build(&loro, &document).expect("binding should build"); let paragraph = document.ids.paragraph_ids[0]; - let styles = RunStyles::default().with(RunStyle::Semantic(4)).with_strikethrough(); + let styles = RunStyles::default() + .with(RunStyle::Semantic(4)) + .with_strikethrough(); insert_text_at(&mut document, 0, "a".len(), "Z", styles); - LocalApplier { doc: &loro, binding: &mut binding } - .apply( - &document, - &[CanonicalOperation::InsertText { - paragraph, - byte: "a".len(), - text: "Z".to_string(), - styles, - }], - ) - .expect("local apply should succeed"); + LocalApplier { + doc: &loro, + binding: &mut binding, + } + .apply( + &document, + &[CanonicalOperation::InsertText { + paragraph, + byte: "a".len(), + text: "Z".to_string(), + styles, + }], + ) + .expect("local apply should succeed"); let projected = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); assert_document_text_and_runs_eq(&document, &projected); diff --git a/crates/flowstate-collab/tests/projection.rs b/crates/flowstate-collab/tests/projection.rs index 4acee34c..f0675052 100644 --- a/crates/flowstate-collab/tests/projection.rs +++ b/crates/flowstate-collab/tests/projection.rs @@ -17,7 +17,12 @@ mod tests { style: ParagraphStyle::Custom(2), runs: vec![ plain("aé"), - run("🌍\u{2028}x", RunStyles::default().with(RunStyle::Semantic(3)).with_direct_underline()), + run( + "🌍\u{2028}x", + RunStyles::default() + .with(RunStyle::Semantic(3)) + .with_direct_underline(), + ), ], }), InputBlock::Equation(InputEquationBlock { @@ -65,8 +70,18 @@ mod tests { let loro = schema::new_configured_doc(); projection::populate_from_document(&loro, SessionId::from_bytes([8; 32]), "offsets", &original).expect("populate should succeed"); let binding = DocBinding::build(&loro, &original).expect("binding should build"); - let text = binding.rows[0].text.as_ref().expect("first row should be paragraph text"); - let byte_to_unicode = [(0, 0), ("a".len(), 1), ("aé".len(), 2), ("aé🌍".len(), 3), ("aé🌍\u{2028}".len(), 4), (MULTIBYTE.len(), 5)]; + let text = binding.rows[0] + .text + .as_ref() + .expect("first row should be paragraph text"); + let byte_to_unicode = [ + (0, 0), + ("a".len(), 1), + ("aé".len(), 2), + ("aé🌍".len(), 3), + ("aé🌍\u{2028}".len(), 4), + (MULTIBYTE.len(), 5), + ]; assert_eq!(text.len_utf8(), MULTIBYTE.len()); for (byte, unicode) in byte_to_unicode { diff --git a/crates/flowstate-collab/tests/translation.rs b/crates/flowstate-collab/tests/translation.rs index 358dbc26..86afb991 100644 --- a/crates/flowstate-collab/tests/translation.rs +++ b/crates/flowstate-collab/tests/translation.rs @@ -6,8 +6,8 @@ mod tests { use gpui_flowtext::{ Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, RunStyles, block_from_input_block, capture_document_span, - delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, - mutate_runs_in_range, paragraph_text, paragraphs_mut, plain, remove_block_ids, run, split_paragraph_at, update_paragraph_block, + delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, mutate_runs_in_range, + paragraph_text, paragraphs_mut, plain, remove_block_ids, run, split_paragraph_at, update_paragraph_block, }; use loro::LoroDoc; @@ -36,7 +36,9 @@ mod tests { #[test] fn split_and_join_paragraphs_preserve_multibyte_text_and_styles() { - let semantic = RunStyles::default().with(RunStyle::Semantic(2)).with_direct_underline(); + let semantic = RunStyles::default() + .with(RunStyle::Semantic(2)) + .with_direct_underline(); let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("aé"), run("🌍\u{2028}x", semantic)])]); let original = document.ids.paragraph_ids[0]; let split_byte = "aé".len(); @@ -75,7 +77,9 @@ mod tests { #[test] fn paragraph_and_run_style_updates_match_flowtext_runs() { - let base_style = RunStyles::default().with(RunStyle::Semantic(1)).with_strikethrough(); + let base_style = RunStyles::default() + .with(RunStyle::Semantic(1)) + .with_strikethrough(); let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![run(MULTIBYTE, base_style)])]); let paragraph = document.ids.paragraph_ids[0]; @@ -145,13 +149,13 @@ mod tests { #[test] fn replace_paragraph_span_refreshes_text_runs_and_styles() { - let (mut document, loro, mut binding) = collab_fixture(vec![ - paragraph_block(vec![plain(MULTIBYTE)]), - paragraph_block(vec![plain("second aé")]), - ]); + let (mut document, loro, mut binding) = + collab_fixture(vec![paragraph_block(vec![plain(MULTIBYTE)]), paragraph_block(vec![plain("second aé")])]); let start_paragraph = document.ids.paragraph_ids[0]; let before = capture_document_span(&document, 0..2); - let inserted_styles = RunStyles::default().with(RunStyle::Semantic(5)).with_direct_underline(); + let inserted_styles = RunStyles::default() + .with(RunStyle::Semantic(5)) + .with_direct_underline(); insert_text_at(&mut document, 0, "a".len(), "Zé", inserted_styles); paragraphs_mut(&mut document)[1].style = ParagraphStyle::Custom(2); @@ -172,10 +176,7 @@ mod tests { #[test] fn insert_move_replace_and_delete_object_blocks_match_flowtext_document() { - let (mut document, loro, mut binding) = collab_fixture(vec![ - paragraph_block(vec![plain("one aé")]), - paragraph_block(vec![plain("two 🌍")]), - ]); + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("one aé")]), paragraph_block(vec![plain("two 🌍")])]); let equation = equation_block("x = aé + 🌍"); let equation_id = insert_object_block(&mut document, 1, equation); @@ -209,20 +210,12 @@ mod tests { ); delete_object_block(&mut document, 2); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::DeleteBlock { block: equation_id }], - ); + apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::DeleteBlock { block: equation_id }]); } #[test] fn replace_document_rebuilds_the_loro_projection() { - let (_, loro, mut binding) = collab_fixture(vec![ - paragraph_block(vec![plain("old")]), - equation_block("old = 1"), - ]); + let (_, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("old")]), equation_block("old = 1")]); let document = document_from_input_blocks( DocumentTheme::default(), vec![ @@ -238,8 +231,7 @@ mod tests { fn collab_fixture(blocks: Vec) -> (Document, LoroDoc, DocBinding) { let document = document_from_input_blocks(DocumentTheme::default(), blocks); let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, SessionId::from_bytes([42; 32]), "translation", &document) - .expect("populate should succeed"); + projection::populate_from_document(&loro, SessionId::from_bytes([42; 32]), "translation", &document).expect("populate should succeed"); let binding = DocBinding::build(&loro, &document).expect("binding should build"); (document, loro, binding) } @@ -277,7 +269,10 @@ mod tests { let block = blocks.remove(from); blocks.insert(to.min(blocks.len()), block); let block_id = document.ids.block_ids.remove(from); - document.ids.block_ids.insert(to.min(document.ids.block_ids.len()), block_id); + document + .ids + .block_ids + .insert(to.min(document.ids.block_ids.len()), block_id); } fn replace_object_block(document: &mut Document, block_ix: usize, input: InputBlock) { diff --git a/crates/flowstate/src/collab/asset_transfer.rs b/crates/flowstate/src/collab/asset_transfer.rs index fd75720b..24502cca 100644 --- a/crates/flowstate/src/collab/asset_transfer.rs +++ b/crates/flowstate/src/collab/asset_transfer.rs @@ -1,4 +1,8 @@ -use std::{collections::hash_map::DefaultHasher, hash::{Hash, Hasher}, sync::Arc}; +use std::{ + collections::hash_map::DefaultHasher, + hash::{Hash, Hasher}, + sync::Arc, +}; use anyhow::{Context as _, Result, anyhow, ensure}; use flowstate_collab::{ diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index c041cb16..52dffdea 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -1,5 +1,5 @@ -mod session; mod presence_view; +mod session; pub mod share_dialog; mod shutdown; pub mod status; @@ -10,7 +10,10 @@ use anyhow::{Context as _, Result, anyhow, ensure}; use async_channel::Receiver; use flowstate_collab::{ SessionId, - net::{NetCommand, NetEvent, TicketSeed, runtime::{self, CommandSender}}, + net::{ + NetCommand, NetEvent, TicketSeed, + runtime::{self, CommandSender}, + }, ticket::SessionTicket, }; use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global, ReadGlobal}; @@ -74,7 +77,10 @@ impl CollabManager { self.register_session(entity.clone(), cx); commands - .try_send(NetCommand::RegisterDirectHandler { session, handler: direct_handler }) + .try_send(NetCommand::RegisterDirectHandler { + session, + handler: direct_handler, + }) .context("registering collaboration direct handler failed")?; let (reply_tx, reply_rx) = async_channel::bounded(1); @@ -90,7 +96,10 @@ impl CollabManager { T: 'static, { ensure!(ticket.is_supported_version(), "unsupported collaboration protocol version"); - ensure!(!self.sessions_by_id.contains_key(&ticket.session), "collaboration session is already open"); + ensure!( + !self.sessions_by_id.contains_key(&ticket.session), + "collaboration session is already open" + ); let commands = self.ensure_runtime(cx)?; let session = ticket.session; @@ -104,7 +113,10 @@ impl CollabManager { bootstrap: vec![ticket.inviter], }) { entity.update(cx, |session, cx| { - session.detach(DetachReason::JoinFailed(format!("joining collaboration network session failed: {error}")), cx); + session.detach( + DetachReason::JoinFailed(format!("joining collaboration network session failed: {error}")), + cx, + ); }); self.unregister_session(session); return Err(error).context("joining collaboration network session failed"); @@ -172,7 +184,9 @@ impl CollabManager { let commands = self.ensure_runtime(cx).ok()?; let (ticket_tx, ticket_rx) = async_channel::bounded(1); let (reply_tx, reply_rx) = async_channel::bounded(1); - commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }).ok()?; + commands + .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) + .ok()?; cx.spawn(async move |_, _| { let ticket = match reply_rx.recv().await { @@ -227,9 +241,7 @@ impl CollabManager { } let (commands, events) = runtime::start()?; - self.runtime = Some(CollabRuntime { - commands: commands.clone(), - }); + self.runtime = Some(CollabRuntime { commands: commands.clone() }); self.start_event_pump(events, cx); Ok(commands) } @@ -276,9 +288,7 @@ impl CollabManager { self.update_session(session, cx, |session, cx| session.handle_gossip_lagged(cx)); }, NetEvent::SubscribeFailed { session, error } => { - let detached = self.update_session(session, cx, |session, cx| { - session.detach(DetachReason::Fatal(error), cx) - }); + let detached = self.update_session(session, cx, |session, cx| session.detach(DetachReason::Fatal(error), cx)); if detached.unwrap_or(false) { self.unregister_session(session); } @@ -303,22 +313,20 @@ impl CollabManager { where T: 'static, { - cx.spawn(async move |_, cx| { - match reply_rx.recv().await { - Ok(Ok(seed)) => { - let _ = session.update(cx, |session, cx| session.establish_local_peer(&seed.inviter.id, cx)); - }, - Ok(Err(error)) => { - let _ = session.update(cx, |session, cx| { - session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error:#}")), cx); - }); - }, - Err(error) => { - let _ = session.update(cx, |session, cx| { - session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error}")), cx); - }); - }, - } + cx.spawn(async move |_, cx| match reply_rx.recv().await { + Ok(Ok(seed)) => { + let _ = session.update(cx, |session, cx| session.establish_local_peer(&seed.inviter.id, cx)); + }, + Ok(Err(error)) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error:#}")), cx); + }); + }, + Err(error) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error}")), cx); + }); + }, }) .detach(); } @@ -331,17 +339,15 @@ impl CollabManager { commands .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) .context("requesting collaboration endpoint address failed")?; - cx.spawn(async move |_, cx| { - match reply_rx.recv().await { - Ok(addr) => { - let _ = session.update(cx, |session, cx| session.establish_local_peer(&addr.id, cx)); - }, - Err(error) => { - let _ = session.update(cx, |session, cx| { - session.detach(DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error}")), cx); - }); - }, - } + cx.spawn(async move |_, cx| match reply_rx.recv().await { + Ok(addr) => { + let _ = session.update(cx, |session, cx| session.establish_local_peer(&addr.id, cx)); + }, + Err(error) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error}")), cx); + }); + }, }) .detach(); Ok(()) @@ -355,12 +361,7 @@ where CollabManager::init(cx); } -pub fn start_session_for_panel( - panel_id: Uuid, - editor: Entity, - title: String, - cx: &mut Context, -) -> Result +pub fn start_session_for_panel(panel_id: Uuid, editor: Entity, title: String, cx: &mut Context) -> Result where T: 'static, { @@ -374,12 +375,7 @@ where cx.update_default_global::(|manager, cx| manager.join_session(ticket, cx)) } -pub fn attach_joined_session( - session_id: SessionId, - panel_id: Uuid, - editor: Entity, - cx: &mut Context, -) -> Result<()> +pub fn attach_joined_session(session_id: SessionId, panel_id: Uuid, editor: Entity, cx: &mut Context) -> Result<()> where T: 'static, { diff --git a/crates/flowstate/src/collab/presence_view.rs b/crates/flowstate/src/collab/presence_view.rs index 387d5ba8..66492dc6 100644 --- a/crates/flowstate/src/collab/presence_view.rs +++ b/crates/flowstate/src/collab/presence_view.rs @@ -1,7 +1,14 @@ use std::path::PathBuf; -use flowstate_collab::{SessionId, binding::DocBinding, presence::{PresenceSelection, PresenceStore}}; -use loro::{LoroDoc, cursor::{Cursor, PosType, Side}}; +use flowstate_collab::{ + SessionId, + binding::DocBinding, + presence::{PresenceSelection, PresenceStore}, +}; +use loro::{ + LoroDoc, + cursor::{Cursor, PosType, Side}, +}; use crate::rich_text_element::{DocumentOffset, ExternalCaret, ParagraphId, RichTextEditor}; @@ -44,15 +51,12 @@ fn cursor_bytes_for_offset(binding: &DocBinding, paragraph: ParagraphId, byte: u let text = row.text.as_ref()?; let byte = byte.min(text.len_utf8()); let pos = text.convert_pos(byte, PosType::Bytes, PosType::Unicode)?; - text.get_cursor(pos, Side::Middle).map(|cursor| cursor.encode()) + text + .get_cursor(pos, Side::Middle) + .map(|cursor| cursor.encode()) } -fn external_caret_for_presence( - doc: &LoroDoc, - binding: &DocBinding, - selection: &PresenceSelection, - color_rgb: u32, -) -> Option { +fn external_caret_for_presence(doc: &LoroDoc, binding: &DocBinding, selection: &PresenceSelection, color_rgb: u32) -> Option { let cursor = Cursor::decode(&selection.head).ok()?; let row_ix = binding.by_container.get(&cursor.container).copied()?; let row = binding.rows.get(row_ix)?; @@ -92,9 +96,5 @@ fn sanitized_recovery_title(title: &str) -> String { } } let trimmed = out.trim_matches(['_', '.']).to_string(); - if trimmed.is_empty() { - "shared-document".to_string() - } else { - trimmed - } + if trimmed.is_empty() { "shared-document".to_string() } else { trimmed } } diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index 92443787..94ee7175 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -1,15 +1,16 @@ -use std::{ - collections::HashSet, - rc::Rc, - time::Instant, -}; +use std::{collections::HashSet, rc::Rc, time::Instant}; use anyhow::{Context as _, Result, anyhow, bail}; use flowstate_collab::{ SessionId, binding::DocBinding, local_apply::LocalApplier, - net::{NetCommand, PeerAddr, PublishPayload, anti_entropy::AntiEntropyState, direct::{DirectServeRequest, DirectSessionHandler}, runtime::CommandSender}, + net::{ + NetCommand, PeerAddr, PublishPayload, + anti_entropy::AntiEntropyState, + direct::{DirectServeRequest, DirectSessionHandler}, + runtime::CommandSender, + }, presence::PresenceStore, projection, proto_gossip::GossipMsg, @@ -249,7 +250,13 @@ impl CollabSession { if let Some(peer) = preferred { candidates.push(peer); } - candidates.extend(self.neighbors.iter().copied().filter(|peer| Some(*peer) != preferred)); + candidates.extend( + self + .neighbors + .iter() + .copied() + .filter(|peer| Some(*peer) != preferred), + ); candidates } @@ -313,12 +320,7 @@ impl CollabSession { result_rx } - pub fn attach_joined_editor( - &mut self, - panel_id: Uuid, - editor: Entity, - cx: &mut Context, - ) -> Result<()> { + pub fn attach_joined_editor(&mut self, panel_id: Uuid, editor: Entity, cx: &mut Context) -> Result<()> { if self.doc.is_none() || self.binding.is_none() { bail!("collaboration snapshot has not finished loading"); } @@ -352,13 +354,15 @@ impl CollabSession { let presence = PresenceStore::new(peer); let session = self.session; let net_tx = self.net_tx.clone(); - self.loro_subscriptions.push(presence.subscribe_local_updates(move |bytes| { - let _ = net_tx.try_send(NetCommand::Publish { - session, - payload: PublishPayload::Presence(bytes.clone()), - }); - true - })); + self + .loro_subscriptions + .push(presence.subscribe_local_updates(move |bytes| { + let _ = net_tx.try_send(NetCommand::Publish { + session, + payload: PublishPayload::Presence(bytes.clone()), + }); + true + })); self.presence = Some(presence); } self.refresh_own_presence(cx); @@ -403,7 +407,9 @@ impl CollabSession { presence.delete_self(); self.publish_presence_bytes(presence.encode_self()); } - let _ = self.net_tx.try_send(NetCommand::LeaveSession { session: self.session }); + let _ = self + .net_tx + .try_send(NetCommand::LeaveSession { session: self.session }); self.flush_pending_remote_patches(cx); if let Some(editor) = self.editor.clone() { @@ -511,7 +517,11 @@ impl CollabSession { pub fn handle_gossip_lagged(&mut self, cx: &mut Context) { self.publish_digest(); let peer = self.neighbors.iter().next().copied(); - let vv = self.doc.as_ref().map(|doc| doc.oplog_vv().encode()).unwrap_or_default(); + let vv = self + .doc + .as_ref() + .map(|doc| doc.oplog_vv().encode()) + .unwrap_or_default(); let action = self.anti_entropy.on_lagged(peer, vv); self.handle_gap_action(action, cx); } @@ -548,19 +558,23 @@ impl CollabSession { }))); }); - self.editor_subscriptions.push(cx.observe(&editor, |session, editor, cx| { - if let Err(error) = session.flush_local_edits(editor.clone(), cx) { - session.detach(DetachReason::Fatal(format!("capturing local collaboration edit failed: {error:#}")), cx); - return; - } - session.flush_pending_remote_patches(cx); - })); + self + .editor_subscriptions + .push(cx.observe(&editor, |session, editor, cx| { + if let Err(error) = session.flush_local_edits(editor.clone(), cx) { + session.detach(DetachReason::Fatal(format!("capturing local collaboration edit failed: {error:#}")), cx); + return; + } + session.flush_pending_remote_patches(cx); + })); - self.editor_subscriptions.push(cx.subscribe(&editor, |session, _, event: &EditorEvent, cx| { - if matches!(event, EditorEvent::SelectionChanged { .. }) { - session.refresh_own_presence(cx); - } - })); + self + .editor_subscriptions + .push(cx.subscribe(&editor, |session, _, event: &EditorEvent, cx| { + if matches!(event, EditorEvent::SelectionChanged { .. }) { + session.refresh_own_presence(cx); + } + })); } fn attach_loro_publish_hook(&mut self) { @@ -572,13 +586,15 @@ impl CollabSession { }; let session = self.session; let net_tx = self.net_tx.clone(); - self.loro_subscriptions.push(doc.subscribe_local_update(Box::new(move |bytes| { - let _ = net_tx.try_send(NetCommand::Publish { - session, - payload: PublishPayload::Update(bytes.clone()), - }); - true - }))); + self + .loro_subscriptions + .push(doc.subscribe_local_update(Box::new(move |bytes| { + let _ = net_tx.try_send(NetCommand::Publish { + session, + payload: PublishPayload::Update(bytes.clone()), + }); + true + }))); self.local_update_publish_attached = true; } @@ -617,5 +633,4 @@ impl CollabSession { }) .detach(); } - } diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs index 09fcc0bd..2323b517 100644 --- a/crates/flowstate/src/collab/session_io.rs +++ b/crates/flowstate/src/collab/session_io.rs @@ -26,10 +26,19 @@ impl CollabSession { return Ok(()); } - let editor = self.editor.clone().context("collaboration session has no editor")?; + let editor = self + .editor + .clone() + .context("collaboration session has no editor")?; let document = Arc::new(editor.read(cx).document().clone()); - let doc = self.doc.clone().context("collaboration session has no Loro document")?; - let binding = self.binding.take().context("collaboration session has no document binding")?; + let doc = self + .doc + .clone() + .context("collaboration session has no Loro document")?; + let binding = self + .binding + .take() + .context("collaboration session has no document binding")?; let binding = Arc::new(Mutex::new(binding)); let patches = Arc::new(Mutex::new(Vec::::new())); let sub = self.diff_subscription(doc.clone(), document, binding.clone(), patches.clone()); @@ -51,7 +60,10 @@ impl CollabSession { let requests = self.direct_rx.clone(); cx.spawn(async move |session, cx| { while let Ok(request) = requests.recv().await { - if session.update(cx, |session, cx| session.handle_direct_request(request, cx)).is_err() { + if session + .update(cx, |session, cx| session.handle_direct_request(request, cx)) + .is_err() + { break; } } @@ -142,10 +154,19 @@ impl CollabSession { return Ok(()); } - let editor = self.editor.clone().context("collaboration session has no editor")?; + let editor = self + .editor + .clone() + .context("collaboration session has no editor")?; let document = Arc::new(editor.read(cx).document().clone()); - let doc = self.doc.clone().context("collaboration session has no Loro document")?; - let binding = self.binding.take().context("collaboration session has no document binding")?; + let doc = self + .doc + .clone() + .context("collaboration session has no Loro document")?; + let binding = self + .binding + .take() + .context("collaboration session has no document binding")?; let binding = Arc::new(Mutex::new(binding)); let patches = Arc::new(Mutex::new(Vec::::new())); let sub = self.diff_subscription(doc, document, binding.clone(), patches.clone()); @@ -217,7 +238,10 @@ impl CollabSession { } fn asset_bytes(&self, asset: u128, cx: &mut Context) -> Result { - let editor = self.editor.as_ref().context("collaboration session has no editor")?; + let editor = self + .editor + .as_ref() + .context("collaboration session has no editor")?; let bytes = editor .read(cx) .document() diff --git a/crates/flowstate/src/collab/session_presence.rs b/crates/flowstate/src/collab/session_presence.rs index e78010e3..b9e69d29 100644 --- a/crates/flowstate/src/collab/session_presence.rs +++ b/crates/flowstate/src/collab/session_presence.rs @@ -57,10 +57,12 @@ impl CollabSession { if bytes.is_empty() { return; } - let _ = self.net_tx.try_send(flowstate_collab::net::NetCommand::Publish { - session: self.session, - payload: flowstate_collab::net::PublishPayload::Presence(bytes), - }); + let _ = self + .net_tx + .try_send(flowstate_collab::net::NetCommand::Publish { + session: self.session, + payload: flowstate_collab::net::PublishPayload::Presence(bytes), + }); } pub(super) fn refresh_peer_count(&mut self) { diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs index 7c88a4b1..17d69be9 100644 --- a/crates/flowstate/src/collab/session_timers.rs +++ b/crates/flowstate/src/collab/session_timers.rs @@ -1,12 +1,7 @@ use std::time::{Duration, Instant}; use anyhow::{Context as _, Result}; -use flowstate_collab::{ - binding::DocBinding, - net::NetCommand, - presence::PRESENCE_KEEPALIVE_SECS, - projection, self_check, -}; +use flowstate_collab::{binding::DocBinding, net::NetCommand, presence::PRESENCE_KEEPALIVE_SECS, projection, self_check}; use gpui::{Context, Timer}; use crate::{app_settings::load_document_theme, rich_text_element::Document}; @@ -117,7 +112,9 @@ impl CollabSession { } fn next_digest_delay(&self) -> Option { - self.timer_live().then(|| self.anti_entropy.duration_until_digest(Instant::now())) + self + .timer_live() + .then(|| self.anti_entropy.duration_until_digest(Instant::now())) } fn run_digest_tick(&mut self, cx: &mut Context) -> bool { @@ -267,6 +264,8 @@ impl CollabSession { fn recovery_delay(retries: u32) -> Duration { let shift = retries.min(5); - let secs = 1_u64.checked_shl(shift).unwrap_or(RECOVERY_MAX_BACKOFF.as_secs()); + let secs = 1_u64 + .checked_shl(shift) + .unwrap_or(RECOVERY_MAX_BACKOFF.as_secs()); Duration::from_secs(secs).min(RECOVERY_MAX_BACKOFF) } diff --git a/crates/flowstate/src/collab/share_dialog.rs b/crates/flowstate/src/collab/share_dialog.rs index f0008b09..7a92e6c1 100644 --- a/crates/flowstate/src/collab/share_dialog.rs +++ b/crates/flowstate/src/collab/share_dialog.rs @@ -1,7 +1,7 @@ use flowstate_collab::ticket::SessionTicket; use gpui::{ - AnyElement, AnyWindowHandle, App, ClipboardItem, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, - KeyDownEvent, MouseButton, ParentElement, Render, SharedString, Subscription, WeakEntity, Window, div, prelude::*, px, relative, rgb, + AnyElement, AnyWindowHandle, App, ClipboardItem, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, KeyDownEvent, + MouseButton, ParentElement, Render, SharedString, Subscription, WeakEntity, Window, div, prelude::*, px, relative, rgb, }; use gpui_component::{ ActiveTheme as _, Disableable, IconName, Sizable, @@ -77,7 +77,9 @@ impl CollabShareDialog { } fn close(&mut self, cx: &mut Context) { - let _ = self.workspace.update(cx, |workspace, cx| workspace.close_collaboration_dialog(cx)); + let _ = self + .workspace + .update(cx, |workspace, cx| workspace.close_collaboration_dialog(cx)); } fn set_mode(&mut self, mode: CollabDialogMode, cx: &mut Context) { @@ -281,7 +283,13 @@ impl CollabShareDialog { .map_or_else(Vec::new, |panel_id| crate::collab::roster_for_panel(panel_id, cx)); v_flex() .gap_3() - .child(h_flex().items_center().justify_between().child(section_title("Invite people", cx)).child(status::status_pill(phase, cx))) + .child( + h_flex() + .items_center() + .justify_between() + .child(section_title("Invite people", cx)) + .child(status::status_pill(phase, cx)), + ) .child(helper_text(connectivity_text(phase).as_str(), cx)) .child(roster_list(roster, cx)) .child(ticket_box(self.ticket_text.clone(), self.ticket_loading, cx)) @@ -320,22 +328,22 @@ impl CollabShareDialog { v_flex() .gap_3() .child(section_title("Join a session", cx)) - .child(helper_text("Paste an invite ticket. Joining opens the shared document as a new tab and never merges into your existing files.", cx)) + .child(helper_text( + "Paste an invite ticket. Joining opens the shared document as a new tab and never merges into your existing files.", + cx, + )) .child(Input::new(&self.join_input).w_full().cleanable(true)) .when_some(parsed.as_ref().map(|ticket| ticket.title.clone()), |this, title| { this.child(success_text(format!("Invite for: {title}").into(), cx)) }) .when_some(self.join_error.clone(), |this, error| this.child(error_text(error, cx))) .child( - h_flex() - .justify_end() - .gap_2() - .child( - Button::new("join-collaboration-session") - .label("Join") - .primary() - .on_click(cx.listener(|dialog, _, window, cx| dialog.join_session(window, cx))), - ), + h_flex().justify_end().gap_2().child( + Button::new("join-collaboration-session") + .label("Join") + .primary() + .on_click(cx.listener(|dialog, _, window, cx| dialog.join_session(window, cx))), + ), ) .into_any_element() } @@ -351,7 +359,9 @@ impl Focusable for CollabShareDialog { #[hotpath::measure_all] impl Render for CollabShareDialog { fn render(&mut self, _: &mut Window, cx: &mut Context) -> impl IntoElement { - let phase = self.panel_id.and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)); + let phase = self + .panel_id + .and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)); div() .absolute() @@ -365,10 +375,13 @@ impl Render for CollabShareDialog { .justify_center() .occlude() .on_key_down(cx.listener(Self::on_key_down)) - .on_mouse_down(MouseButton::Left, cx.listener(|dialog, _, _, cx| { - dialog.close(cx); - cx.stop_propagation(); - })) + .on_mouse_down( + MouseButton::Left, + cx.listener(|dialog, _, _, cx| { + dialog.close(cx); + cx.stop_propagation(); + }), + ) .on_scroll_wheel(|_, _, cx| cx.stop_propagation()) .child( v_flex() @@ -390,7 +403,11 @@ impl Render for CollabShareDialog { .px_4() .border_b_1() .border_color(cx.theme().border) - .child(div().font_weight(gpui::FontWeight::SEMIBOLD).child("Share / Collaborate")) + .child( + div() + .font_weight(gpui::FontWeight::SEMIBOLD) + .child("Share / Collaborate"), + ) .child( Button::new("close-collaboration-dialog") .icon(IconName::Close) @@ -405,13 +422,17 @@ impl Render for CollabShareDialog { .gap_2() .px_4() .pt_4() - .child(tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { - dialog.set_mode(CollabDialogMode::Share, cx); - }))) - .child(tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { - dialog.set_mode(CollabDialogMode::Join, cx); - dialog.focus(window, cx); - }))), + .child( + tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { + dialog.set_mode(CollabDialogMode::Share, cx); + })), + ) + .child( + tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { + dialog.set_mode(CollabDialogMode::Join, cx); + dialog.focus(window, cx); + })), + ), ) .child( div() @@ -470,17 +491,11 @@ fn ticket_box(ticket: Option, loading: bool, cx: &App) -> impl Int } fn error_text(text: SharedString, cx: &App) -> impl IntoElement { - div() - .text_sm() - .text_color(cx.theme().danger) - .child(text) + div().text_sm().text_color(cx.theme().danger).child(text) } fn success_text(text: SharedString, cx: &App) -> impl IntoElement { - div() - .text_sm() - .text_color(cx.theme().success) - .child(text) + div().text_sm().text_color(cx.theme().success).child(text) } fn roster_list(entries: Vec, cx: &App) -> AnyElement { @@ -491,26 +506,37 @@ fn roster_list(entries: Vec, cx: &App) -> Any .border_color(cx.theme().border) .bg(cx.theme().background) .p_3() - .child(div().text_xs().text_color(cx.theme().muted_foreground).child("Participants")); + .child( + div() + .text_xs() + .text_color(cx.theme().muted_foreground) + .child("Participants"), + ); if entries.is_empty() { return list - .child(div().text_sm().text_color(cx.theme().muted_foreground).child("Presence is starting...")) + .child( + div() + .text_sm() + .text_color(cx.theme().muted_foreground) + .child("Presence is starting..."), + ) .into_any_element(); } for entry in entries { - let label = if entry.is_self { - format!("{} (you)", entry.name) - } else { - entry.name - }; + let label = if entry.is_self { format!("{} (you)", entry.name) } else { entry.name }; list = list.child( h_flex() .items_center() .gap_2() .child(div().size(px(8.0)).rounded_full().bg(rgb(entry.color_rgb))) - .child(div().text_sm().text_color(cx.theme().foreground).child(label)), + .child( + div() + .text_sm() + .text_color(cx.theme().foreground) + .child(label), + ), ); } list.into_any_element() diff --git a/crates/flowstate/src/collab/status.rs b/crates/flowstate/src/collab/status.rs index ea59dc14..7a6201ee 100644 --- a/crates/flowstate/src/collab/status.rs +++ b/crates/flowstate/src/collab/status.rs @@ -1,4 +1,4 @@ -use gpui::{AnyElement, App, IntoElement, div, px, prelude::*}; +use gpui::{AnyElement, App, IntoElement, div, prelude::*, px}; use gpui_component::{ActiveTheme as _, h_flex}; use super::{Attachment, Connectivity, JoinStage, SessionPhase}; @@ -15,18 +15,8 @@ pub fn status_pill(phase: &SessionPhase, cx: &App) -> AnyElement { .border_1() .border_color(color.opacity(0.38)) .bg(color.opacity(0.10)) - .child( - div() - .size(px(6.0)) - .rounded_full() - .bg(color), - ) - .child( - div() - .text_size(px(10.0)) - .text_color(color) - .child(label), - ) + .child(div().size(px(6.0)).rounded_full().bg(color)) + .child(div().text_size(px(10.0)).text_color(color).child(label)) .into_any_element() } @@ -47,12 +37,7 @@ pub fn tab_badge(phase: &SessionPhase, cx: &App) -> Option { h_flex() .items_center() .gap_0p5() - .child( - div() - .size(px(6.0)) - .rounded_full() - .bg(color), - ) + .child(div().size(px(6.0)).rounded_full().bg(color)) .when_some(count, |this, count| { this.child( div() From 51fa2b9a2f5d67481423eeb992a0ea6d45f731b3 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Sat, 13 Jun 2026 18:58:47 -0500 Subject: [PATCH 09/45] add fix plan --- fix.md | 341 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 341 insertions(+) create mode 100644 fix.md diff --git a/fix.md b/fix.md new file mode 100644 index 00000000..08dfda9c --- /dev/null +++ b/fix.md @@ -0,0 +1,341 @@ +# Flowstate Collaboration — Remediation Plan (`fix.md`) + +Closes the gaps found auditing the `fable-collab` branch against `plan.md`. The +implementation is ~85% faithful and builds/clippy-clean; this doc enumerates the +deltas needed to reach the plan's intent. Each task lists **files**, **what to do**, +and an **acceptance check**. Audience: an Opus-tier implementer. Keep `plan.md` as +the source of truth for behavior; section refs (e.g. §13.2) point there. + +Conventions (from repo `AGENTS.md` / `plan.md`): 2-space indent, files <1000 LOC, +prefer `gpui-component` widgets, `cargo clippy` clean per task, never bare +`unwrap()` on `get`, every `#[allow]`/`unsafe` needs a `reason`. Don't relitigate +§20 decisions. + +--- + +## Execution plan (parallelism) + +Five tasks split across crate/file boundaries run in **two waves**. Within a wave, +spawn one subagent per task — they touch disjoint files so they won't collide. + +``` +WAVE 1 (3 agents, fully parallel — different crates/files): + ├─ Agent E "editor" → crates/gpui-flowtext/** (Tasks E1,E2) + ├─ Agent D "docsync" → flowstate-collab/{schema,local_apply,remote_apply, + │ self_check,patch_apply(new)} + tests/{convergence, + │ remote_apply} (Tasks D1,D2,D3) + └─ Agent N "net" → flowstate-collab/net/**, ticket.rs, runtime.rs, + tests/swarm_loopback(new) (Tasks N1,N2,N3) + +WAVE 2 (2 agents, parallel — app crate, partitioned by file): + ├─ Agent S "session" → flowstate/src/collab/{mod,session,session_io, + │ session_timers,asset_transfer}.rs, collab.rs, + │ flowstate-collab/net/mod.rs (Tasks S1..S6) + └─ Agent U "ui" → flowstate/src/collab/{share_dialog,status, + presence_view,notify(new)}.rs, render_status.rs + (Tasks U1..U4) + +WAVE 3 (1 agent, sequential — after 1+2 merge): + └─ Verification + docs + LOC splits (Tasks V1,V2) +``` + +**Coordination points (only two):** +1. **Notifications contract.** Agent S emits notices; Agent U renders them. Land + the shared event enum first: Agent S adds `pub enum SessionNotice {...}` + + `cx.emit(...)` in `session.rs`; Agent U subscribes and renders via + `notify.rs`. Define the enum (below in S5) before either starts so both compile. +2. **`NetEvent::IncompatibleVersion`.** Agent N adds the variant + emits it + (N2); Agent S routes it in the `mod.rs` event pump; Agent U renders it. Agree + the variant name up front. + +Severity: **P0** = correctness / explicit plan gate · **P1** = robustness/UX · +**P2** = fidelity/cleanup. + +--- + +## WAVE 1 + +### Agent D — Document-sync test gate (the big one) + +#### D1 (P0) — Convergence proptest fuzz + RemoteApplier coverage +`plan.md` §21.2 calls this "the M2 gate; nothing integrates until this is green." +Current `tests/convergence.rs` is a single hardcoded 2-peer `#[test]` that never +exercises `RemoteApplier`. Replace it with the real fuzz. + +- **New file `flowstate-collab/src/patch_apply.rs`** (GPUI-free, ≤400 LOC): add + `pub fn apply_patches(document: &mut Document, binding: &mut DocBinding, doc: &LoroDoc, patches: &[CollabPatch]) -> Result<()>`. + Mutate the `Document` from each `CollabPatch` using the **exported `edit_ops` + primitives** (`insert_text_at`, `mutate_runs_in_range`, span replace, block + splice — already re-exported from `gpui-flowtext`, see existing imports in + `tests/convergence.rs`). This is the headless twin of the editor's + `apply_collab_patches`; it lets tests close the loop without GPUI. Register in + `lib.rs`. + - For `ParagraphText` regenerate the paragraph from `new`; for structural + patches splice `blocks`/`paragraphs`/`ids` using the **patch-provided** + `block_id`/`paragraph_id` (do not invent ids); `AssetArrived` is a no-op here. +- **Rewrite `tests/convergence.rs`** as a `proptest!`: + - Peer = `(Document, LoroDoc, DocBinding)` + a diff-event subscription that + collects `DiffEvent`s. + - Generate random per-peer op programs, weighted **70% text ins/del, 10% + split/join, 10% styles, 10% block ops**, run through the real `edit_ops` + + `LocalApplier`. N ∈ {2, 3}. + - Virtual network exchanging `subscribe_local_update` bytes with random + **delay, reorder, duplication, and per-peer offline windows** (buffer → + flush). On import, drain collected `DiffEvent`s → `RemoteApplier::apply_event` + → `apply_patches` into that peer's `Document`. + - After quiescence + full exchange: assert **all peers' `Document`s + byte-identical** (texts, run vectors, styles, block kinds, payload bytes) AND + equal to a fresh `projection::document_from_loro` of any peer. Shrunk failures + should print the op program. + - Every text fixture must include `"aé🌍\u{2028}x"`. +- **Acceptance:** `cargo test -p flowstate-collab` green with the new proptest + (≥256 cases); failure shrinks print the program. + +#### D2 (P0) — RemoteApplier golden tests + `ReplaceBlock{None}` +`tests/translation.rs` covers LocalApplier only; the remote path is untested. +- **New `tests/remote_apply.rs`**: for each `CollabPatch`-producing diff (text, + style, data/rev, list insert/delete/move), drive a second `LoroDoc` by importing + peer-1 updates, run `RemoteApplier::apply_event`, assert the produced + `Vec` shape + (via `apply_patches`) byte-equality with the source + `Document`. +- Add the missing `ReplaceBlock { block: None }` LocalApplier path test + (`local_apply.rs` `single_changed_object_row`) to `tests/translation.rs`. +- **Acceptance:** new tests green; every `CanonicalOperation` and every + `CollabPatch` variant now has coverage. + +#### D3 (P2) — Offset/self-check discipline +- **`self_check.rs`**: reorder `projection_hash` to match §7.2 literal order + (styles → run `(len,styles)` → text bytes → kinds → payload). Harmless but spec. +- **`schema.rs` + `local_apply.rs`**: add the §5.5 debug assertion + `debug_assert_eq!(text.len_utf8(), paragraph_text_len(&document.paragraphs[ix]))` + after each paragraph mutation in the LocalApplier (both directions). +- **Acceptance:** tests still green in debug; assertion present at mutation sites. + +--- + +### Agent N — Networking robustness + +#### N1 (P2) — Await first reachable address before minting tickets +§10.1 / §23-V1. Tickets currently embed `endpoint.addr()` immediately +(`net/runtime.rs:156-158,177-179`), so a cold install can embed an unreachable +address. +- In `net/runtime.rs`, after `bind()`, await the endpoint's first reachable + address (watcher/online API — verify exact name on docs.rs for iroh 0.98 via the + `docs` subagent). Gate `CreateSession` / `MintTicketAddr` replies on it (short + timeout, then best-effort). +- **Acceptance:** minted `EndpointAddr` contains a relay URL and/or direct addrs + in a from-cold run. + +#### N2 (P1) — Surface protocol-version skew +§11 / F5. Today a version mismatch is only `eprintln!` in `swarm.rs:149`. +- Add `NetEvent::IncompatibleVersion { session, peer }` (in + `flowstate-collab/src/net/mod.rs`). Emit it from the gossip decode-error path in + `swarm.rs` when the failure is a version mismatch (distinguish from generic + decode error). Keep the message-ignore behavior. +- (Agent S routes it; Agent U renders a one-time-per-peer toast.) +- **Acceptance:** decoding a wrong-`PROTOCOL_VERSION` frame emits the event once. + +#### N3 (P2) — Swarm loopback integration test +§21.3 / M1 deliverable — currently absent. +- **New `tests/swarm_loopback.rs`** (`#[ignore]` if CI blocks UDP): real iroh + endpoints + real gossip on localhost, `RelayMode::Disabled`, addresses + registered manually. 3 peers: A creates, B & C join via A's ticket; assert + snapshot fetch, A→B→C update propagation, presence roster on all three, **A + leaves → B & C keep editing & converge** (the no-starter property), a forced + >2 KiB blob path, and a kill-subscription→resubscribe→digest-pull heal. +- **Acceptance:** `cargo test -p flowstate-collab -- --ignored swarm_loopback` + passes locally. + +--- + +### Agent E — Editor fidelity (`gpui-flowtext`) + +#### E1 (P1) — Emit `SelectionChanged` on all selection-mutation paths +§15.7. Presence carets are driven off `EditorEvent::SelectionChanged`, but it's +not emitted on mouse paths, so mouse-only caret moves don't update presence. +- Add `emit_selection_changed(cx)` (mirror existing usage in `caret_movement.rs`) + to the selection-mutating paths that currently lack it: + `mouse.rs` (click/drag set-selection at `:38,153,163,196,233,254,274`), + `hit_testing.rs` caret placement, `select_all` (`commands.rs:308`), search-result + nav (`search_highlights.rs:32`), outline `scroll_to_paragraph` (`commands.rs:162`). +- Guard against spurious emits (only when the selection actually changed). +- **Acceptance:** clicking to move the caret fires `SelectionChanged` (add a small + editor test or assert via existing test harness in `rich_text/tests`). + +#### E2 (P1) — IME deferral + table/equation offset remap +§7.3 / §7-item-4. +- **IME:** `editor/platform.rs` exposes no marked-text state. Add a query + `pub fn ime_composition_active(&self) -> bool` (track marked-text begin/end in + the platform input handler), and add it to the `||` chain in + `collab_apply.rs:23` `collab_apply_deferred`. +- **Offset remap:** in `collab_apply.rs` `ParagraphText` handling, also remap + `table_cell_*` / `equation_source_*` selection offsets through `delta_utf8` + (same retain/insert/delete walk used for `self.selection`), not just the main + selection. +- **Acceptance:** remote patch during an active IME composition is queued (not + applied); a remote edit to a paragraph while a table cell is being edited remaps + the cell caret instead of corrupting it. + +--- + +## WAVE 2 (app crate) + +### Agent S — Session lifecycle robustness + +#### S1 (P0) — Self-join detection +§13.3 step 1 / F16. `join_session` (`collab/mod.rs:94-129`) lacks the own-invite +guard. +- After parsing the ticket, if `ticket.inviter.id == ` (get it + via the runtime/`CollabManager`), fail fast with friendly copy ("That's your own + invite — open the Share dialog instead."). +- **Acceptance:** pasting one's own ticket shows the friendly error, never dials. + +#### S2 (P1) — Connectivity detection per §13.2 +`session_timers.rs:52-69` uses OR (`quiet_rounds>=2 || !endpoint_online`) and has +no probe dial; plan requires **all of**: zero neighbors >5 s **AND** 2 quiet +digest rounds **AND** (`EndpointOnline(false)` OR probe-dial-to-last-peer-fails). +- Change the boolean to the three-way AND. Add a probe-dial attempt to the most + recent known peer as the third conjunct's OR-branch (best-effort; reuse + `pull_with_fallback`-style dial or a lightweight `NetCommand`). +- In the offline recovery loop (`run_recovery_if_due`), **retain and re-register + last-presence-seen peer addresses** (not just the original inviter) and **await + first `NeighborUp` (10 s)** before flipping back to `Online`. +- **Acceptance:** "alone but networked" stays `Online, peers_present==0`; a real + network drop flips to `Offline` only when all three conditions hold; reconnect + restores `Online` after a `NeighborUp`. + +#### S3 (P1) — Join timeouts + budget + `EnsureUp` +§13.3 / F2 / F4. +- Issue `NetCommand::EnsureUp` at the start of the join path. +- After `JoinSession`, **await first `NeighborUp` with a 15 s timeout** → + `Detached(JoinFailed("couldn't reach anyone in this session…"))`. +- Wrap the snapshot pull in a **30 s overall budget** (distinct from the 10 s + per-peer `DIRECT_PULL_TIMEOUT`). +- **Acceptance:** ticket for a dead session fails after ~15 s with actionable copy. + +#### S4 (P1) — Stream join progress +§13.3 step 4. `FetchingSnapshot { got, total }` is frozen at `0/None` +(`session.rs:274`). +- Thread byte progress from the direct-pull (the `DirectResponseHeader::Ok{total_len}` + + chunk loop already knows `got`/`total`) up to the session phase so the join + dialog shows real progress. May need a progress channel on `PullSnapshot`. +- **Acceptance:** join dialog advances `got/total` during a large snapshot fetch. + +#### S5 (P1) — "(shared)" tab title + emit `SessionNotice` +- §13.3 step 6: title the joined tab `format!("{} (shared)", meta.title)` + (`collab.rs:271` currently passes the raw ticket title). +- **Notifications contract:** add to `session.rs`: + ```rust + pub enum SessionNotice { + PeerJoined(String), PeerLeft(String), LeftSession, + ViewRebuilt, IncompatibleVersion(String), + } + impl EventEmitter for CollabSession {} + ``` + Emit `PeerJoined`/`PeerLeft` from the presence-roster diff, `LeftSession` from + `detach(UserLeft)`, `ViewRebuilt` from the self-heal rebuild + (`session_timers.rs:228`, replacing the `eprintln!`), `IncompatibleVersion` from + the routed `NetEvent::IncompatibleVersion` (S6). Replace existing `eprintln!` + notice sites. +- **Acceptance:** roster changes/self-heal/leave each `cx.emit` a `SessionNotice`. + +#### S6 (P1) — Route `NetEvent::IncompatibleVersion` +- In the `mod.rs` NetEvent pump, route the new variant (N2) to the session → + emit `SessionNotice::IncompatibleVersion`, deduped per peer. + +#### S7 (P2) — 50 ms presence debounce +§9 / §13.5.3. `refresh_own_presence` fires synchronously on every +`SelectionChanged` (`session.rs:571-577`). +- Debounce the presence refresh by ~50 ms (coalesce rapid selection changes via a + timer/`cx.spawn` guarded flag). +- **Acceptance:** rapid caret movement produces ≤1 presence broadcast per ~50 ms. + +--- + +### Agent U — UI & notifications + +#### U1 (P1) — Notifications (the §17.6 gap) +Today notices are `eprintln!`/modal prompts; gpui-component's notification API +(`vendor/gpui-component/src/notification.rs`) is unused. +- **New `collab/notify.rs`**: thin helper wrapping the vendored notification + widget (look up exact push API via the `docs` subagent if unclear). +- Subscribe to `CollabSession`'s `SessionNotice` (from S5) in the workspace render + path and render toasts: peer joined ("Maya joined"), peer left, "Left session — + this copy is now local", view-rebuilt (debug only), + incompatible-version (once per peer). +- Replace the modal `window.prompt` "Left session" feedback (`collab.rs:180-188`) + with a transient toast. +- **Acceptance:** a peer joining/leaving shows a toast; leaving shows the "now + local" toast (no modal). + +#### U2 (P1) — Share dialog → gpui-component `Dialog` +§17.3 + repo UI rule. The dialog is a hand-rolled `div` overlay +(`share_dialog.rs:366-447`), no avatar, and join progress isn't surfaced. +- Rebuild on gpui-component `Dialog` (+ `Input`, `Button`, clipboard copy button, + `avatar` for the roster). Preserve all current states: not-attached explainer + + `[Start session]`; attached view (freshly-minted ticket + Copy + roster + + connectivity line + `[Leave session]` danger); join tab (paste + inline + validation showing title + `[Join]` + inline error). +- **Surface `JoinStage` progress** inside the join tab (Resolving / Subscribing / + Fetching got·total / Building), using S4's data. +- Keep: **no role switch, no "End for everyone"** (§20.2/§20.4). +- **Acceptance:** dialog renders via `Dialog`; roster shows avatars; join shows + staged progress. + +#### U3 (P2) — Roster color hash consistency +§9. Rendering hashes the hex key (`presence.rs:157-162`) while `self_color` +hashes raw `EndpointId` bytes — two functions. Pick one (`hash(EndpointId) % +PALETTE.len()` per plan) and use it everywhere; drop the vestigial one. +- **Acceptance:** a peer's dot color matches between roster and caret, derived from + `EndpointId`. + +#### U4 (P2) — LOC split for `share_dialog.rs` +After U2, if `share_dialog.rs` exceeds ~500 LOC, split (e.g. `share_dialog.rs` +view + `share_dialog_state.rs` logic). Keep <1000 hard cap; aim for the §3.1 caps. + +--- + +## WAVE 3 — Verification & docs (sequential, after merge) + +#### V1 — Cleanup + final gate +- Split `collab/mod.rs` (currently 409 LOC > §3.1 cap 350) into `mod.rs` + + e.g. `manager.rs`/`pump.rs` if it didn't shrink during Wave 2. +- (Optional P2) `SessionId::new()` → use `OsRng` explicitly per §12.2. +- (Optional P2) Add `head_container`/`anchor_container` to `PresenceSelection` + (§9) if container-from-cursor recovery proves fragile; otherwise leave a + `// reason:` comment that it's intentionally derived. +- Run `cargo clippy` (whole workspace) — must be clean; `cargo test` (workspace, + GPUI-free + editor) green; `swarm_loopback` green locally. + +#### V2 — Docs (§22 M6) +- `helpers/docs/collaboration.md` — user how-to + maintainer architecture notes. +- `helpers/docs/collab_qa.md` — the §21.4 manual QA script (all 21 F-rows + + same-paragraph typing, Enter-split race, same-cell race, image paste, undo + tug-of-war, leave/rejoin, offline-lid resync, close-prompt cancellation, 30-min + soak). + +--- + +## Quick traceability (gap → task) + +| Plan ref | Gap | Task | +|---|---|---| +| §21.2 | Convergence fuzz is a 2-peer stub; RemoteApplier untested | D1, D2 | +| §13.3 / F16 | No self-join detection | S1 | +| §13.2 | Offline logic OR-not-AND; no probe dial; addrs not retained | S2 | +| §13.3 / F2,F4 | No 15 s neighbor / 30 s snapshot timeout; no EnsureUp | S3 | +| §13.3 | Join progress frozen 0/None | S4, U2 | +| §13.3 | "(shared)" title missing | S5 | +| §17.6 | Notifications unimplemented (eprintln/modal) | N2, S5, S6, U1 | +| §15.7 | SelectionChanged not emitted on mouse paths | E1 | +| §7.3 / §7 | IME deferral + table/equation remap missing | E2 | +| §17.3 | Custom overlay not gpui-component Dialog; no avatar | U2 | +| §9 | 50 ms debounce missing; dual color hash | S7, U3 | +| §10.1 | No await-first-reachable-addr | N1 | +| §21.3 | swarm_loopback test absent | N3 | +| §7.2 / §5.5 | hash order; len_utf8 assert | D3 | +| §3.1 | LOC caps: mod.rs 409, share_dialog.rs 530 | U4, V1 | + +**Recommended order if serializing:** D1 first (it's the gate that surfaces latent +remote-path bugs), then everything else; D1/D2/D3, E1/E2, N1/N2/N3 are safe to run +concurrently in Wave 1. From 2dd32492081e15f09b50c7947a00361ed6c1a578 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Sat, 13 Jun 2026 19:05:54 -0500 Subject: [PATCH 10/45] more fix --- fix.md | 293 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 293 insertions(+) diff --git a/fix.md b/fix.md index 08dfda9c..4165f773 100644 --- a/fix.md +++ b/fix.md @@ -336,6 +336,299 @@ view + `share_dialog_state.rs` logic). Keep <1000 hard cap; aim for the §3.1 ca | §7.2 / §5.5 | hash order; len_utf8 assert | D3 | | §3.1 | LOC caps: mod.rs 409, share_dialog.rs 530 | U4, V1 | +In addition to things that are left to implement, there are bugs found in existing implementation - fix these after the above implementations if they still exist. Before fixing - verify that they actually are bugs (is what is being said true or not, and then fix them. + +Verify each finding against current code. Fix only still-valid issues, skip the +rest with a brief reason, keep changes minimal, and validate. + +Inline comments: +In `@crates/flowstate-collab/src/ids.rs`: +- Around line 18-20: `SessionId::new` currently uses `rand::random()`, but +session IDs need a stronger entropy source; update the constructor in +`crates/flowstate-collab/src/ids.rs` to generate the ID with `rand::rngs::OsRng` +(or `SysRng`) instead of `rand::random()`, and keep the change localized to the +`SessionId`/ID generation path so all session IDs come from a cryptographically +secure source. + +In `@crates/flowstate-collab/src/local_apply.rs`: +- Around line 148-164: In local_apply.rs inside the split/row insertion logic, +validate the new row metadata first before mutating the source LoroText: fetch +the inserted block id and version (the lookups around block_ids.get(insert_ix) +and blocks.get(insert_ix)) before calling delete_utf8 on the original paragraph, +and only truncate text once those checks succeed so a failed split cannot leave +the document partially edited. + +In `@crates/flowstate-collab/src/net/blobs.rs`: +- Around line 45-57: In `BlobStore::insert_with_id`, reinserting an existing +`BlobId` currently leaves the old payload in `entries`, and `BlobStore::get` +returns that stale first match. Update `insert_with_id` to remove or overwrite +any existing entry with the same `BlobId` before pushing the new bytes, while +keeping `total_bytes` consistent with the replacement logic. + +In `@crates/flowstate-collab/src/net/direct.rs`: +- Around line 138-146: The semaphore permit is being acquired after reading the +request frame, which allows unbounded task spawning and request buffering before +the concurrency limit is checked. Move the +self.permits.clone().try_acquire_owned() check to occur before the read_frame +call in handle_stream() to ensure the concurrency limit gates network work. This +issue applies at two locations: the primary site in handle_stream (lines +138-146) and a sibling location (lines 152-160). Additionally, consider gating +the permit acquisition even earlier, before tokio::spawn invokes handle_stream +in the accept() method, to limit task creation itself. + +In `@crates/flowstate-collab/src/net/runtime.rs`: +- Around line 18-40: `start()` leaves `RUNTIME` pointing at dead channels after +`net_main()` shuts down, so future calls reuse a stopped bridge; change +`RuntimeBridge`/`start()` so a failed or exited runtime can be detected and +replaced with a fresh `spawn_runtime()` result instead of returning the cached +pair. Also update the `direct.rs` endpoint cache (the `OnceLock` around the +client endpoint) so it is invalidated or refreshed when the runtime restarts, +otherwise the new bridge will still use a stale endpoint. +- Around line 101-107: Contain per-command failures inside net_main so a bad +join/publish does not exit the runtime or skip responses. In +NetCommand::CreateSession, catch SwarmHandle::spawn errors locally, reply to the +caller with the failure, and keep the loop running; in the publish path around +handle.publish(payload).await, handle the error locally instead of propagating +it. Apply the same error containment at +crates/flowstate-collab/src/net/runtime.rs:101-107, +crates/flowstate-collab/src/net/runtime.rs:116-120, and +crates/flowstate-collab/src/net/runtime.rs:128-130, using the existing net_main +/ replace_swarm / handle.publish flow as the anchor points. + +In `@crates/flowstate-collab/src/projection.rs`: +- Around line 169-175: `verify_lineage` in +`crates/flowstate-collab/src/projection.rs` only validates `META_SESSION`, so +add a `META_SCHEMA` check against `SCHEMA_VERSION` before returning success. +Update the `verify_lineage` path to fail fast with an error when the stored +schema is missing or does not match the current version, alongside the existing +session check, so snapshot projection never reaches `document_from_loro` with an +incompatible layout. + +In `@crates/flowstate-collab/src/remote_apply.rs`: +- Around line 34-37: In remote_apply.rs, the Diff::Map handling in +remote_apply::apply should emit only one object-replacement patch per map diff +instead of calling apply_map_diff once per updated key; update the logic so +ReplaceObjectBlock is generated once per affected row even when replace_block_at +changes KIND, DATA, and REV together. Use the apply_map_diff path and its +related map-diff dispatch to dedupe by row/target before applying patches, and +ensure the same fix covers the other map-diff site referenced by the comment. + +In `@crates/flowstate/src/collab/asset_transfer.rs`: +- Around line 142-147: In record_from_bytes, replace the use of DefaultHasher +with a stable, deterministic wire hash for cross-peer verification so identical +bytes always produce the same content_hash across platforms and Rust versions. +Update the content_hash computation to use a portable algorithm such as SHA-256 +or xxHash, and keep the existing byte_len and content_hash validation logic in +AssetRecord unchanged except for comparing against the new stable hash value. + +In `@crates/flowstate/src/collab/mod.rs`: +- Around line 119-146: The `session_by_panel` mapping is being inserted on line +137 before two fallible operations execute: the `try_send` call for +RegisterDirectHandler and the `establish_joined_peer` call. If either operation +fails, the function returns an error but leaves the mapping in place, causing +subsequent attach attempts to incorrectly think the panel is already attached. +Move the `self.session_by_panel.insert(panel_id, session_id)` statement to after +the `Self::establish_joined_peer(session, commands, cx)?` call succeeds, +ensuring the mapping is only recorded when the entire attachment operation +completes successfully. +- Around line 53-86: The `start_session_for_panel` function registers a session +at line 75 via `self.register_session(entity.clone(), cx)`, but if either the +`RegisterDirectHandler` command send (line 77) or the `CreateSession` command +send (line 82) fails, the function returns an error without cleaning up the +registered session. This creates a resource leak where the session ID remains +registered but the network layer never received the commands. To fix this, when +either `try_send` call fails, detach the session from its context and unregister +it before returning the error. Reference the correct cleanup pattern used in the +`join_session` method at lines 106-110, which properly calls detach and +unregister before returning errors, and apply the same approach to both error +paths in `start_session_for_panel`. + +In `@crates/flowstate/src/collab/presence_view.rs`: +- Around line 33-39: The directory creation in collaboration_recovery_path +currently ignores failures from create_dir_all, so update this function to +handle that error instead of discarding it. Either change +collaboration_recovery_path to return a Result and propagate the +std::fs::create_dir_all error to callers, or at minimum log/return a failure +when the temp recovery directory cannot be created; keep the rest of the +path-building logic with SessionId and sanitized_recovery_title unchanged. + +In `@crates/flowstate/src/collab/session_timers.rs`: +- Around line 238-249: When `rebuild_from_projection` rebuilds the document +after a self-check by calling `replace_document_from_collaboration()`, any +deferred remote patches stored in `session_io.rs` become stale since they +describe edits already present in the rebuilt projection. These stale patches +would then be incorrectly re-applied on the next flush. Clear the +`pending_remote_patches` from `session_io.rs` within the +`rebuild_from_projection` method after the document has been rebuilt to prevent +this re-application of already-synchronized edits. + +In `@crates/flowstate/src/collab/session.rs`: +- Around line 131-132: Switch the session request channels in `Session` setup +from unbounded to bounded to apply backpressure and avoid memory growth; update +the `direct_tx/direct_rx` and `undo_tx/undo_rx` creation in `collab/session.rs` +to use `async_channel::bounded` with a justified capacity, and make sure the +downstream direct request pump and undo request pump code paths still handle +send/receive behavior correctly. If keeping unbounded is intentional, document +that decision near the channel setup. + +In `@crates/flowstate/src/workspace/workspace/collab.rs`: +- Around line 204-208: In join_collaboration_from_clipboard’s synchronous error +path around crate::collab::join_session, don’t just log and return None; surface +a user-visible error prompt before exiting the branch. Update the error handling +in workspace/collab.rs so the failure is reported through the existing UI/error +notification mechanism used by this flow, while still returning None after the +prompt is shown. + +In `@crates/flowstate/src/workspace/workspace/top_bar.rs`: +- Around line 109-143: Update collaboration_top_bar_button in +workspace/top_bar.rs so the “Leave Shared Session” menu item is disabled unless +the active document actually has an attached collaboration session, not just +when has_document is true. Use the same workspace/session state used by +confirm_leave_collaboration_on_active_document to compute the enabled flag, and +pass that into file_menu_item for the “Leave Shared Session” entry so local tabs +don’t show a no-op action. + +In `@crates/flowstate/src/workspace/workspace/window.rs`: +- Around line 84-87: `window.rs` is shutting down the global collaboration +runtime too early, which breaks other open windows; change the +`window_handle.update` close path so +`workspace.leave_all_collaboration_sessions(cx)` only affects the current +workspace and `crate::collab::shutdown(cx)` is deferred until the last +workspace/window is gone or behind a shared reference count. Also audit the +collaboration close/leave flow in `collab_prompts.rs` so it does not trigger a +global shutdown from a single workspace path, and route any such teardown +through the same last-window guard. + +In `@crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs`: +- Around line 79-80: Update prepare_block_insertion_index and +insert_ordered_block_fragment_after_caret so the placeholder caret paragraph is +only removed when the insertion does not include InputBlock::Paragraph; preserve +the lone empty paragraph for paragraph-block inserts to keep +self.document.paragraphs and self.document.blocks aligned. Use the existing +block-rebuild logic in insert_ordered_block_fragment_after_caret and the caret +cleanup in prepare_block_insertion_index to gate the removal instead of +unconditionally stripping the paragraph. + +--- + +Outside diff comments: +In `@crates/gpui-flowtext/src/rich_text/editor/mouse.rs`: +- Around line 278-304: Build the `ReplaceParagraphSpan` canonical op from the +pre-edit state in `mouse.rs` so it reflects the drag/drop before any mutation is +applied; capture the source-side deleted paragraph span as part of the same +canonical operation, not just the inserted drop range. Update the +`EditRecord`/`mark_document_changed_with_ops` path around `canonical_operations` +so the payload uses the pre-delete document contents and includes both source +and destination paragraph ranges when the drag crosses paragraphs. + +In `@crates/gpui-flowtext/src/rich_text/editor/object_selection.rs`: +- Around line 492-500: The block identifier is being resolved after the block +and id vectors have been mutated, causing the lookup to fail and fall back to a +fabricated BlockId(0), which can delete the wrong remote block in collaboration +mode. Capture the block identifier by calling +self.identity_map.block_id(block_ix) BEFORE removing the block from the vectors +using blocks.remove() and remove_block_ids(). Use the captured block id in the +CanonicalOperation for DeleteBlock. If the block id cannot be captured (when the +lookup fails), replace the DeleteBlock operation with a broader operation like +ReplaceDocument instead of using a fake BlockId(0). Apply this same fix to all +locations where blocks are deleted and canonical operations are created. + +--- + +Nitpick comments: +In `@crates/flowstate-collab/src/binding.rs`: +- Around line 148-167: In the rebuild_indexes method, remove the unnecessary +clone operation on line 153 where each BindingRow is being cloned. Since the +index_row method only requires a reference to the BindingRow (as seen in its +signature), pass a direct reference to self.rows[ix] to index_row instead of +creating a clone first. This eliminates wasteful O(n) allocations that occur +every time indexes are rebuilt after insert/remove/move operations. + +In `@crates/flowstate-collab/src/net/swarm.rs`: +- Around line 137-162: Replace the eprintln! call in the handle_event function's +error handling branch (where proto_gossip::decode fails) with a structured +logging call using tracing::warn!. This will allow production deployments to +configure log levels and get structured output instead of relying on stderr +output, enabling better observability and log aggregation. + +In `@crates/flowstate-collab/src/proto_direct.rs`: +- Around line 31-38: Simplify encode_frame in proto_direct.rs by removing the +redundant u32::try_from(payload.len())? conversion after the MAX_FRAME_LEN +check; since payload.len() is already bounded below u32::MAX, compute the length +directly as a u32 and keep the rest of the frame assembly unchanged. + +In `@crates/flowstate-collab/src/proto_gossip.rs`: +- Line 7: Move the DIRECT_ALPN constant out of proto_gossip.rs and into +proto_direct.rs so it lives with the other direct-protocol constants like +MAX_FRAME_LEN and MAX_PAYLOAD_CHUNK_LEN. Update any direct-protocol code that +references DIRECT_ALPN, especially net/direct.rs and net/runtime.rs, to import +it from proto_direct.rs instead of proto_gossip.rs, and leave proto_gossip.rs +containing only gossip-related constants such as PROTOCOL_VERSION and +GOSSIP_INLINE_LIMIT. +- Around line 30-32: The encoded_len function allocates the full message buffer +by calling encode(msg)? just to measure its length, which is inefficient even +though the practical impact is minimal with the current 2KB size bound. Either +implement a size-counting approach that avoids the full buffer allocation (such +as using a custom serializer that only counts bytes without storing them), or if +accepting the allocation as reasonable, add a comment in the function explaining +why the current approach is acceptable given the bounded message size and single +call site. + +In `@crates/flowstate-collab/src/ticket.rs`: +- Around line 46-48: The encode_bytes method in the Ticket trait implementation +uses expect() which can panic on serialization failure. First, check the Ticket +trait definition to see if the return type can be changed from Vec to a +Result type. If the trait signature is flexible, update encode_bytes to return +Result, Box> and use the ? operator instead of +expect(). If the trait signature is fixed and requires Vec, then keep the +current approach but add a doc comment above the method explaining why postcard +serialization of SessionTicket cannot fail and the panic is justified as a last +resort safeguard. + +In `@crates/flowstate-collab/tests/convergence.rs`: +- Around line 14-55: The test +`two_peers_converge_with_reordered_and_duplicated_update_imports` currently +validates convergence for paragraph text and style operations but does not cover +structural block operations. Add new convergence test functions that exercise +insert, move, replace, and delete operations on structural blocks (such as +images, equations, and tables) between two peers with reordered and duplicated +imports, following the same test pattern used for the existing paragraph-focused +test to ensure these operations also converge correctly. + +In `@crates/flowstate/src/collab/session_presence.rs`: +- Around line 9-11: The code currently uses eprintln! macro for logging error +messages when presence updates fail. Replace the eprintln! calls with a +structured logging framework such as tracing or log to enable better +observability, filtering, and production-grade log management. Update all +occurrences where eprintln! is used for the flowstate collab presence update +error message (in the error handling block for presence.apply() and any similar +error logging) to use the appropriate logging macro from your chosen framework +instead, ensuring consistent structured logging throughout the module. + +In `@crates/flowstate/src/collab/session.rs`: +- Around line 592-595: The UndoManager setup in session.rs uses magic numbers +for merge_interval and max_undo_steps; extract the 500 and 300 values into named +constants or a session/undo configuration near UndoManager::new, +set_merge_interval, and set_max_undo_steps so the defaults are easy to find and +adjust. + +In `@crates/flowstate/src/collab/share_dialog.rs`: +- Around line 146-151: The `window_handle.update()` call that writes to the +clipboard is silently discarding any errors with the `let _ =` pattern. Replace +this pattern by capturing the Result returned from the update call and logging +any failures using an appropriate logger (such as error or warn level). Keep the +operation as best-effort by not propagating or panicking on the error, but +ensure that clipboard write failures are logged to help with debugging +clipboard-related issues. + +In `@crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs`: +- Around line 265-268: The method `clear_collab_history` has a misleading name +since it clears both the `undo_stack` and `redo_stack` entirely, affecting all +editor history rather than just collaboration-specific history. Rename this +method to `clear_undo_redo_stacks` or `clear_all_history` to accurately reflect +that it clears all undo and redo history, not just collaboration-related +history. Be sure to update all call sites that reference this method name. + **Recommended order if serializing:** D1 first (it's the gate that surfaces latent remote-path bugs), then everything else; D1/D2/D3, E1/E2, N1/N2/N3 are safe to run concurrently in Wave 1. From 0f065e757640fde1d657c75a4f72f8ef8fb639bc Mon Sep 17 00:00:00 2001 From: Adhitya Thirumala Date: Sun, 14 Jun 2026 00:37:04 -0500 Subject: [PATCH 11/45] fix.md implemented and coderabbit as well --- Cargo.lock | 2 + Cargo.toml | 1 + crates/flowstate-collab/Cargo.toml | 1 + crates/flowstate-collab/src/binding.rs | 25 +- crates/flowstate-collab/src/ids.rs | 13 +- crates/flowstate-collab/src/lib.rs | 4 +- crates/flowstate-collab/src/local_apply.rs | 102 ++- crates/flowstate-collab/src/net/blobs.rs | 29 + crates/flowstate-collab/src/net/direct.rs | 84 +- crates/flowstate-collab/src/net/mod.rs | 8 + crates/flowstate-collab/src/net/runtime.rs | 170 +++- crates/flowstate-collab/src/net/swarm.rs | 58 +- crates/flowstate-collab/src/patch_apply.rs | 182 ++++ crates/flowstate-collab/src/presence.rs | 18 +- crates/flowstate-collab/src/projection.rs | 4 + crates/flowstate-collab/src/proto_direct.rs | 3 +- crates/flowstate-collab/src/proto_gossip.rs | 27 +- crates/flowstate-collab/src/remote_apply.rs | 242 +++++- crates/flowstate-collab/src/schema.rs | 10 +- crates/flowstate-collab/src/self_check.rs | 4 +- crates/flowstate-collab/src/ticket.rs | 2 + crates/flowstate-collab/tests/convergence.rs | 817 +++++++++++++++--- crates/flowstate-collab/tests/remote_apply.rs | 316 +++++++ .../flowstate-collab/tests/swarm_loopback.rs | 260 ++++++ crates/flowstate-collab/tests/translation.rs | 19 + crates/flowstate/Cargo.toml | 1 + crates/flowstate/src/collab/asset_transfer.rs | 18 +- crates/flowstate/src/collab/manager.rs | 343 ++++++++ crates/flowstate/src/collab/mod.rs | 365 +------- crates/flowstate/src/collab/notify.rs | 40 + crates/flowstate/src/collab/presence_view.rs | 4 +- crates/flowstate/src/collab/pump.rs | 78 ++ crates/flowstate/src/collab/session.rs | 228 ++++- crates/flowstate/src/collab/session_io.rs | 9 +- .../flowstate/src/collab/session_presence.rs | 69 +- crates/flowstate/src/collab/session_timers.rs | 96 +- crates/flowstate/src/collab/share_dialog.rs | 320 ++----- .../flowstate/src/collab/share_dialog_view.rs | 158 ++++ crates/flowstate/src/collab/shutdown.rs | 13 - crates/flowstate/src/collab/status.rs | 4 +- crates/flowstate/src/workspace/icons.rs | 4 +- .../src/workspace/workspace/collab.rs | 41 +- .../src/workspace/workspace/collab_prompts.rs | 1 - .../src/workspace/workspace/documents.rs | 2 + .../flowstate/src/workspace/workspace/mod.rs | 4 +- .../src/workspace/workspace/render_status.rs | 39 + .../src/workspace/workspace/render_top_bar.rs | 2 +- .../src/workspace/workspace/top_bar.rs | 4 +- .../src/workspace/workspace/traits.rs | 3 +- .../src/workspace/workspace/window.rs | 1 - crates/gpui-flowtext/src/document/blocks.rs | 10 + .../gpui-flowtext/src/persistence/records.rs | 5 + .../src/persistence/validation.rs | 11 +- .../src/rich_text/editor/block_insertion.rs | 12 +- .../src/rich_text/editor/collab_apply.rs | 21 +- .../src/rich_text/editor/commands.rs | 5 + .../src/rich_text/editor/hit_testing.rs | 1 + .../src/rich_text/editor/lifecycle.rs | 4 +- .../gpui-flowtext/src/rich_text/editor/mod.rs | 9 +- .../src/rich_text/editor/mouse.rs | 50 +- .../src/rich_text/editor/object_assets.rs | 10 +- .../src/rich_text/editor/object_selection.rs | 8 +- .../src/rich_text/editor/platform.rs | 190 ++++ .../src/rich_text/editor/search_highlights.rs | 4 + crates/gpui-flowtext/src/rich_text/element.rs | 6 +- .../src/rich_text/tests/collab_capture.rs | 47 + .../src/rich_text/tests/edit_layout.rs | 4 - .../src/rich_text/tests/persistence_blocks.rs | 5 +- helpers/docs/collab_qa.md | 38 + helpers/docs/collaboration.md | 58 ++ 70 files changed, 3741 insertions(+), 1005 deletions(-) create mode 100644 crates/flowstate-collab/src/patch_apply.rs create mode 100644 crates/flowstate-collab/tests/remote_apply.rs create mode 100644 crates/flowstate-collab/tests/swarm_loopback.rs create mode 100644 crates/flowstate/src/collab/manager.rs create mode 100644 crates/flowstate/src/collab/notify.rs create mode 100644 crates/flowstate/src/collab/pump.rs create mode 100644 crates/flowstate/src/collab/share_dialog_view.rs create mode 100644 helpers/docs/collaboration.md diff --git a/Cargo.lock b/Cargo.lock index 2460d5df..63e3a161 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2996,6 +2996,7 @@ dependencies = [ "serde_json", "tempfile", "toml 1.1.2+spec-1.1.0", + "tracing", "uuid", ] @@ -3016,6 +3017,7 @@ dependencies = [ "rand 0.9.4", "serde", "tokio", + "tracing", "twox-hash", "uuid", ] diff --git a/Cargo.toml b/Cargo.toml index 3ab5cbee..3ded0683 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -15,6 +15,7 @@ anyhow = "1.0.102" async-channel = "2" clap = { version = "4.5", features = ["derive"] } cosmic-text = { version = "0.14.2", default-features = false, features = [ + "std", "shape-run-cache", ] } crop = "0.4.3" diff --git a/crates/flowstate-collab/Cargo.toml b/crates/flowstate-collab/Cargo.toml index 62a7b68d..715a9d85 100644 --- a/crates/flowstate-collab/Cargo.toml +++ b/crates/flowstate-collab/Cargo.toml @@ -17,6 +17,7 @@ postcard = { workspace = true } rand = { workspace = true } serde = { workspace = true } tokio = { workspace = true } +tracing = "0.1.44" twox-hash = { workspace = true } uuid = { workspace = true } diff --git a/crates/flowstate-collab/src/binding.rs b/crates/flowstate-collab/src/binding.rs index 2ed3923c..9eac1bd9 100644 --- a/crates/flowstate-collab/src/binding.rs +++ b/crates/flowstate-collab/src/binding.rs @@ -97,8 +97,8 @@ impl DocBinding { debug_assert!(false, "binding row has no matching document block"); continue; }; - debug_assert_eq!(row.kind, BlockKind::from_block(block)); - debug_assert_eq!(document.ids.block_ids.get(row_ix).copied(), Some(row.block_id)); + debug_assert_eq!(row.kind, BlockKind::from_block(block), "binding row {row_ix} kind does not match document block"); + debug_assert_eq!(document.ids.block_ids.get(row_ix).copied(), Some(row.block_id), "binding row {row_ix} block id does not match document block id"); debug_assert_eq!(row.version, block_version(block)); if matches!(row.kind, BlockKind::Paragraph) { debug_assert_eq!(document.ids.paragraph_ids.get(paragraph_ix).copied(), row.paragraph_id); @@ -146,13 +146,22 @@ impl DocBinding { } pub fn rebuild_indexes(&mut self) { - self.by_paragraph.clear(); - self.by_block.clear(); - self.by_container.clear(); - for ix in 0..self.rows.len() { - let row = self.rows[ix].clone(); - self.index_row(ix, &row); + let mut by_paragraph = HashMap::new(); + let mut by_block = HashMap::new(); + let mut by_container = HashMap::new(); + for (ix, row) in self.rows.iter().enumerate() { + by_block.insert(row.block_id, ix); + by_container.insert(row.map.id(), ix); + if let Some(text) = &row.text { + by_container.insert(text.id(), ix); + } + if let Some(paragraph_id) = row.paragraph_id { + by_paragraph.insert(paragraph_id, ix); + } } + self.by_paragraph = by_paragraph; + self.by_block = by_block; + self.by_container = by_container; } fn index_row(&mut self, ix: usize, row: &BindingRow) { diff --git a/crates/flowstate-collab/src/ids.rs b/crates/flowstate-collab/src/ids.rs index 5eb2225f..9db48fd9 100644 --- a/crates/flowstate-collab/src/ids.rs +++ b/crates/flowstate-collab/src/ids.rs @@ -1,5 +1,6 @@ use std::{fmt, str::FromStr}; +use rand::{TryRngCore as _, rngs::OsRng}; use serde::{Deserialize, Serialize}; pub type PeerId = iroh::EndpointId; @@ -14,7 +15,11 @@ pub struct SessionId([u8; SESSION_ID_LEN]); impl SessionId { #[must_use] pub fn new() -> Self { - Self(rand::random()) + let mut bytes = [0; SESSION_ID_LEN]; + OsRng + .try_fill_bytes(&mut bytes) + .expect("OS randomness must be available to create collaboration session IDs"); + Self(bytes) } #[must_use] @@ -29,7 +34,11 @@ impl SessionId { #[must_use] pub fn color_index_for_peer(peer: &PeerId) -> usize { - let bytes = peer.as_bytes(); + Self::color_index_for_peer_bytes(peer.as_bytes()) + } + + #[must_use] + pub fn color_index_for_peer_bytes(bytes: &[u8]) -> usize { let mut hash = 0usize; for byte in bytes.iter().take(8) { hash = hash.wrapping_mul(257).wrapping_add(usize::from(*byte)); diff --git a/crates/flowstate-collab/src/lib.rs b/crates/flowstate-collab/src/lib.rs index d930ae3f..b6bae492 100644 --- a/crates/flowstate-collab/src/lib.rs +++ b/crates/flowstate-collab/src/lib.rs @@ -8,6 +8,7 @@ pub mod binding; pub mod ids; pub mod local_apply; pub mod net; +pub mod patch_apply; pub mod presence; pub mod projection; pub mod proto_direct; @@ -18,4 +19,5 @@ pub mod self_check; pub mod ticket; pub use ids::{BlobId, SessionId}; -pub use proto_gossip::{DIRECT_ALPN, PROTOCOL_VERSION}; +pub use proto_direct::DIRECT_ALPN; +pub use proto_gossip::PROTOCOL_VERSION; diff --git a/crates/flowstate-collab/src/local_apply.rs b/crates/flowstate-collab/src/local_apply.rs index e7b11fc1..2fec20f0 100644 --- a/crates/flowstate-collab/src/local_apply.rs +++ b/crates/flowstate-collab/src/local_apply.rs @@ -11,8 +11,8 @@ use crate::{ projection::{insert_object_container, insert_paragraph_container, replace_blocks_from_document}, schema::{ BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_TABLE, MARK_HIGHLIGHT, MARK_SEMANTIC, MARK_STRIKE, MARK_UNDERLINE, REV, - STYLE, apply_mark_intervals, decode_paragraph_style, encode_paragraph_style, mark_intervals_from_runs, payload_from_block, - run_styles_from_attrs, set_run_styles_utf8, unmark_utf8, + STYLE, apply_mark_intervals, debug_assert_paragraph_text_len, decode_paragraph_style, encode_paragraph_style, mark_intervals_from_runs, + payload_from_block, run_styles_from_attrs, set_run_styles_utf8, unmark_utf8, }, }; @@ -43,19 +43,19 @@ impl LocalApplier<'_> { byte, text, styles, - } => self.insert_text(*paragraph, *byte, text, *styles), + } => self.insert_text(document, *paragraph, *byte, text, *styles), CanonicalOperation::DeleteRange { start_paragraph, start_byte, end_paragraph, end_byte, - } => self.delete_range(*start_paragraph, *start_byte, *end_paragraph, *end_byte), + } => self.delete_range(document, *start_paragraph, *start_byte, *end_paragraph, *end_byte), CanonicalOperation::SplitParagraph { paragraph, byte, new_paragraph, } => self.split_paragraph(document, *paragraph, *byte, *new_paragraph), - CanonicalOperation::JoinParagraphs { first, second } => self.join_paragraphs(*first, *second), + CanonicalOperation::JoinParagraphs { first, second } => self.join_paragraphs(document, *first, *second), CanonicalOperation::SetParagraphStyle { paragraph, style } => { let row = self.row_for_paragraph(*paragraph)?; row.map.insert(STYLE, encode_paragraph_style(*style))?; @@ -68,6 +68,7 @@ impl LocalApplier<'_> { .as_ref() .context("paragraph row is missing its LoroText")?; set_run_styles_utf8(text, range.clone(), *styles)?; + self.debug_assert_row_text_len(document, self.row_ix_for_paragraph(*paragraph)?); Ok(()) }, CanonicalOperation::InsertBlock { block, block_ix } => self.insert_block(document, *block, *block_ix), @@ -83,24 +84,37 @@ impl LocalApplier<'_> { } } - fn insert_text(&mut self, paragraph: ParagraphId, byte: usize, value: &str, styles: RunStyles) -> Result<()> { - let row = self.row_for_paragraph(paragraph)?; + fn insert_text(&mut self, document: &Document, paragraph: ParagraphId, byte: usize, value: &str, styles: RunStyles) -> Result<()> { + let row_ix = self.row_ix_for_paragraph(paragraph)?; + let row = self + .binding + .rows + .get(row_ix) + .context("DocBinding paragraph index is out of bounds")?; let text = row .text .as_ref() .context("paragraph row is missing its LoroText")?; text.insert_utf8(byte, value)?; - if styles != RunStyles::default() { - set_run_styles_utf8(text, byte..byte + value.len(), styles)?; - } + set_run_styles_utf8(text, byte..byte + value.len(), styles)?; + self.debug_assert_row_text_len(document, row_ix); Ok(()) } - fn delete_range(&mut self, start_paragraph: ParagraphId, start: usize, end_paragraph: ParagraphId, end: usize) -> Result<()> { + fn delete_range( + &mut self, + document: &Document, + start_paragraph: ParagraphId, + start: usize, + end_paragraph: ParagraphId, + end: usize, + ) -> Result<()> { let start_ix = self.row_ix_for_paragraph(start_paragraph)?; let end_ix = self.row_ix_for_paragraph(end_paragraph)?; if start_ix == end_ix { - return self.delete_text_at_row(start_ix, start, end); + self.delete_text_at_row(start_ix, start, end)?; + self.debug_assert_row_text_len(document, start_ix); + return Ok(()); } if start_ix > end_ix { bail!("cross-paragraph delete start row is after end row"); @@ -129,11 +143,24 @@ impl LocalApplier<'_> { .binding .remove_row(start_ix + 1) .context("DocBinding end row disappeared during cross-paragraph delete")?; + self.debug_assert_row_text_len(document, start_ix); Ok(()) } fn split_paragraph(&mut self, document: &Document, paragraph: ParagraphId, byte: usize, new_paragraph: ParagraphId) -> Result<()> { let row_ix = self.row_ix_for_paragraph(paragraph)?; + let insert_ix = row_ix + 1; + let block_id = document + .ids + .block_ids + .get(insert_ix) + .copied() + .context("post-split document is missing the new block id")?; + let version = document + .blocks + .get(insert_ix) + .map(block_version) + .context("post-split document is missing the new block")?; let row = self .binding .rows @@ -149,19 +176,8 @@ impl LocalApplier<'_> { if byte < text.len_utf8() { text.delete_utf8(byte, text.len_utf8() - byte)?; } + self.debug_assert_row_text_len(document, row_ix); - let insert_ix = row_ix + 1; - let block_id = document - .ids - .block_ids - .get(insert_ix) - .copied() - .context("post-split document is missing the new block id")?; - let version = document - .blocks - .get(insert_ix) - .map(block_version) - .context("post-split document is missing the new block")?; let blocks = self.blocks(); let (map, new_text) = insert_paragraph_container(&blocks, insert_ix, style, &[], "")?; insert_delta_at(&new_text, 0, &tail)?; @@ -176,13 +192,14 @@ impl LocalApplier<'_> { version, }, ); + self.debug_assert_row_text_len(document, insert_ix); Ok(()) } - fn join_paragraphs(&mut self, first: ParagraphId, second: ParagraphId) -> Result<()> { + fn join_paragraphs(&mut self, document: &Document, first: ParagraphId, second: ParagraphId) -> Result<()> { let first_ix = self.row_ix_for_paragraph(first)?; let first_len = self.paragraph_text_at_row(first_ix)?.len_utf8(); - self.delete_range(first, first_len, second, 0) + self.delete_range(document, first, first_len, second, 0) } fn insert_block(&mut self, document: &Document, block_id: BlockId, block_ix: usize) -> Result<()> { @@ -208,7 +225,9 @@ impl LocalApplier<'_> { ¶graph.runs, &text, block_version(block), - ) + )?; + self.debug_assert_row_text_len(document, block_ix); + Ok(()) }, Block::Image(_) | Block::Equation(_) | Block::Table(_) => { let blocks = self.blocks(); @@ -261,6 +280,7 @@ impl LocalApplier<'_> { for ix in 0..matched { self.update_matched_paragraph( + document, old_rows[ix], &before.paragraphs[ix], &before_texts[ix], @@ -309,6 +329,7 @@ impl LocalApplier<'_> { text, block_version(&document.blocks[block_ix]), )?; + self.debug_assert_row_text_len(document, insert_ix); } } @@ -347,7 +368,15 @@ impl LocalApplier<'_> { Ok(()) } - fn update_matched_paragraph(&self, row_ix: usize, before: &Paragraph, before_text: &str, after: &Paragraph, after_text: &str) -> Result<()> { + fn update_matched_paragraph( + &self, + document: &Document, + row_ix: usize, + before: &Paragraph, + before_text: &str, + after: &Paragraph, + after_text: &str, + ) -> Result<()> { let row = self .binding .rows @@ -366,6 +395,7 @@ impl LocalApplier<'_> { if before.style != after.style { row.map.insert(STYLE, encode_paragraph_style(after.style))?; } + self.debug_assert_row_text_len(document, row_ix); Ok(()) } @@ -462,6 +492,18 @@ impl LocalApplier<'_> { .context("DocBinding paragraph index is out of bounds") } + fn debug_assert_row_text_len(&self, document: &Document, row_ix: usize) { + let Some(row) = self.binding.rows.get(row_ix) else { + return; + }; + let Some(text) = row.text.as_ref() else { + return; + }; + if let Some(paragraph_ix) = self.paragraph_ordinal_for_row(row_ix) { + debug_assert_paragraph_text_len(text, document, paragraph_ix); + } + } + fn row_ix_for_paragraph(&self, paragraph: ParagraphId) -> Result { self .binding @@ -548,9 +590,7 @@ fn insert_delta_at(text: &LoroText, byte: usize, delta: &[TextDelta]) -> Result< } text.insert_utf8(cursor, insert)?; let styles = run_styles_from_attrs(attributes.as_ref()); - if styles != RunStyles::default() { - set_run_styles_utf8(text, cursor..cursor + insert.len(), styles)?; - } + set_run_styles_utf8(text, cursor..cursor + insert.len(), styles)?; cursor += insert.len(); } Ok(()) diff --git a/crates/flowstate-collab/src/net/blobs.rs b/crates/flowstate-collab/src/net/blobs.rs index 9a579d64..a4c9c1e3 100644 --- a/crates/flowstate-collab/src/net/blobs.rs +++ b/crates/flowstate-collab/src/net/blobs.rs @@ -43,6 +43,13 @@ impl BlobOutbox { } pub fn insert_with_id(&mut self, id: BlobId, bytes: Vec) { + if let Some((_, existing)) = self.entries.iter_mut().find(|(candidate, _)| *candidate == id) { + self.total_bytes = self.total_bytes.saturating_sub(existing.len()).saturating_add(bytes.len()); + *existing = bytes; + self.trim(); + return; + } + self.total_bytes = self.total_bytes.saturating_add(bytes.len()); self.entries.push_back((id, bytes)); self.trim(); @@ -81,3 +88,25 @@ impl BlobOutbox { } } } + +#[cfg(test)] +mod tests { + use std::num::NonZeroUsize; + + use crate::BlobId; + + use super::BlobOutbox; + + #[test] + fn insert_with_id_replaces_existing_payload_and_total() { + let mut outbox = BlobOutbox::new(NonZeroUsize::new(4).expect("non-zero max blobs"), 1024); + let id = BlobId(7); + + outbox.insert_with_id(id, vec![1, 2, 3]); + outbox.insert_with_id(id, vec![4, 5]); + + assert_eq!(outbox.len(), 1); + assert_eq!(outbox.total_bytes(), 2); + assert_eq!(outbox.get(id), Some([4, 5].as_slice())); + } +} diff --git a/crates/flowstate-collab/src/net/direct.rs b/crates/flowstate-collab/src/net/direct.rs index 429f5f0b..57a2c115 100644 --- a/crates/flowstate-collab/src/net/direct.rs +++ b/crates/flowstate-collab/src/net/direct.rs @@ -1,6 +1,6 @@ use std::{ collections::{HashMap, HashSet}, - sync::{Arc, OnceLock}, + sync::{Arc, OnceLock, RwLock as StdRwLock}, time::Duration, }; @@ -12,20 +12,22 @@ use iroh::{ protocol::{AcceptError, ProtocolHandler}, }; use tokio::{ - sync::{RwLock, Semaphore}, + sync::{OwnedSemaphorePermit, RwLock, Semaphore}, time::timeout, }; use crate::{ ids::{BlobId, SessionId}, - proto_direct::{AssetBytes, DirectRequest, DirectResponseHeader, MAX_FRAME_LEN, MAX_PAYLOAD_CHUNK_LEN, decode_frame, encode_frame}, - proto_gossip::DIRECT_ALPN, + proto_direct::{ + AssetBytes, DIRECT_ALPN, DirectRequest, DirectResponseHeader, MAX_FRAME_LEN, MAX_PAYLOAD_CHUNK_LEN, decode_frame, + encode_frame, + }, }; -use super::blobs::BlobOutbox; +use super::{PullProgress, blobs::BlobOutbox}; const DIRECT_SERVE_CONCURRENCY: usize = 4; -static CLIENT_ENDPOINT: OnceLock = OnceLock::new(); +static CLIENT_ENDPOINT: OnceLock>> = OnceLock::new(); #[derive(Clone, Debug)] pub struct DirectSessionHandler { @@ -135,16 +137,10 @@ impl DirectProto { } } - async fn handle_stream(&self, mut send: SendStream, mut recv: RecvStream) -> Result<()> { + async fn handle_stream(&self, mut send: SendStream, mut recv: RecvStream, _permit: OwnedSemaphorePermit) -> Result<()> { let request = read_frame::(&mut recv).await?; - let Ok(permit) = self.permits.clone().try_acquire_owned() else { - write_response(&mut send, ServeOutcome::Header(DirectResponseHeader::Busy)).await?; - return Ok(()); - }; - let outcome = self.state.serve(request).await; write_response(&mut send, outcome).await?; - drop(permit); Ok(()) } } @@ -152,10 +148,17 @@ impl DirectProto { impl ProtocolHandler for DirectProto { async fn accept(&self, connection: Connection) -> Result<(), AcceptError> { while let Ok((send, recv)) = connection.accept_bi().await { + let Ok(permit) = self.permits.clone().try_acquire_owned() else { + let mut send = send; + if let Err(error) = write_response(&mut send, ServeOutcome::Header(DirectResponseHeader::Busy)).await { + tracing::warn!("flowstate collab direct busy response failed: {error:#}"); + } + continue; + }; let proto = self.clone(); tokio::spawn(async move { - if let Err(error) = proto.handle_stream(send, recv).await { - eprintln!("flowstate collab direct stream failed: {error:#}"); + if let Err(error) = proto.handle_stream(send, recv, permit).await { + tracing::warn!("flowstate collab direct stream failed: {error:#}"); } }); } @@ -164,15 +167,28 @@ impl ProtocolHandler for DirectProto { } pub(crate) fn install_endpoint(endpoint: Endpoint) { - let _ = CLIENT_ENDPOINT.set(endpoint); + let cache = CLIENT_ENDPOINT.get_or_init(|| StdRwLock::new(None)); + if let Ok(mut cached) = cache.write() { + *cached = Some(endpoint); + } } pub async fn pull_with_fallback(req: DirectRequest, candidates: Vec, per_peer_timeout: Duration) -> Result> { + pull_with_fallback_progress(req, candidates, per_peer_timeout, None).await +} + +pub async fn pull_with_fallback_progress( + req: DirectRequest, + candidates: Vec, + per_peer_timeout: Duration, + progress: Option>, +) -> Result> { let endpoint = CLIENT_ENDPOINT .get() - .cloned() + .and_then(|cache| cache.read().ok().and_then(|cached| cached.clone())) .ok_or_else(|| anyhow!("collaboration direct client endpoint is not running"))?; - pull_with_endpoint(&endpoint, req, candidates, per_peer_timeout).await + ensure!(!endpoint.is_closed(), "collaboration direct client endpoint is not running"); + pull_with_endpoint_progress(&endpoint, req, candidates, per_peer_timeout, progress).await } pub async fn pull_with_endpoint( @@ -180,12 +196,22 @@ pub async fn pull_with_endpoint( req: DirectRequest, candidates: Vec, per_peer_timeout: Duration, +) -> Result> { + pull_with_endpoint_progress(endpoint, req, candidates, per_peer_timeout, None).await +} + +pub async fn pull_with_endpoint_progress( + endpoint: &Endpoint, + req: DirectRequest, + candidates: Vec, + per_peer_timeout: Duration, + progress: Option>, ) -> Result> { ensure!(!candidates.is_empty(), "direct pull has no candidate peers"); let mut errors = Vec::new(); for peer in candidates { - match timeout(per_peer_timeout, pull_once(endpoint, peer, req.clone())).await { + match timeout(per_peer_timeout, pull_once(endpoint, peer, req.clone(), progress.as_ref())).await { Ok(Ok(bytes)) => return Ok(bytes), Ok(Err(error)) => errors.push(format!("{peer}: {error:#}")), Err(_) => errors.push(format!("{peer}: timed out after {per_peer_timeout:?}")), @@ -195,7 +221,7 @@ pub async fn pull_with_endpoint( Err(anyhow!("direct pull failed for all candidates: {}", errors.join("; "))) } -async fn pull_once(endpoint: &Endpoint, peer: EndpointId, req: DirectRequest) -> Result> { +async fn pull_once(endpoint: &Endpoint, peer: EndpointId, req: DirectRequest, progress: Option<&Sender>) -> Result> { let connection = endpoint .connect(peer, DIRECT_ALPN) .await @@ -209,7 +235,7 @@ async fn pull_once(endpoint: &Endpoint, peer: EndpointId, req: DirectRequest) -> let header = read_frame::(&mut recv).await?; match header { - DirectResponseHeader::Ok { total_len } => read_payload(&mut recv, total_len).await, + DirectResponseHeader::Ok { total_len } => read_payload(&mut recv, total_len, progress).await, DirectResponseHeader::NotAttached => Err(anyhow!("peer is not attached to this session")), DirectResponseHeader::NotFound => Err(anyhow!("peer does not have the requested collaboration data")), DirectResponseHeader::Busy => Err(anyhow!("peer is busy serving collaboration data")), @@ -286,14 +312,20 @@ async fn write_payload(send: &mut SendStream, payload: &[u8]) -> Result<()> { Ok(()) } -async fn read_payload(recv: &mut RecvStream, total_len: u64) -> Result> { - let total_len = usize::try_from(total_len).context("direct payload is too large for this platform")?; - let mut payload = vec![0; total_len]; +async fn read_payload(recv: &mut RecvStream, total_len: u64, progress: Option<&Sender>) -> Result> { + let total_len_usize = usize::try_from(total_len).context("direct payload is too large for this platform")?; + let mut payload = vec![0; total_len_usize]; let mut offset = 0; - while offset < total_len { - let next = (offset + MAX_PAYLOAD_CHUNK_LEN).min(total_len); + if let Some(progress) = progress { + let _ = progress.try_send(PullProgress { got: 0, total: total_len }); + } + while offset < total_len_usize { + let next = (offset + MAX_PAYLOAD_CHUNK_LEN).min(total_len_usize); recv.read_exact(&mut payload[offset..next]).await?; offset = next; + if let Some(progress) = progress { + let _ = progress.try_send(PullProgress { got: offset as u64, total: total_len }); + } } Ok(payload) } diff --git a/crates/flowstate-collab/src/net/mod.rs b/crates/flowstate-collab/src/net/mod.rs index dcd0aff0..fe954cba 100644 --- a/crates/flowstate-collab/src/net/mod.rs +++ b/crates/flowstate-collab/src/net/mod.rs @@ -57,6 +57,7 @@ pub enum NetCommand { PullSnapshot { session: SessionId, candidates: Vec, + progress: Option>, reply: Reply>>, }, PullBlob { @@ -77,6 +78,12 @@ pub enum NetCommand { Shutdown, } +#[derive(Clone, Copy, Debug)] +pub struct PullProgress { + pub got: u64, + pub total: u64, +} + #[derive(Clone, Debug)] pub struct TicketSeed { pub inviter: EndpointAddr, @@ -85,6 +92,7 @@ pub struct TicketSeed { #[derive(Clone, Debug)] pub enum NetEvent { Gossip { session: SessionId, from: EndpointId, msg: GossipMsg }, + IncompatibleVersion { session: SessionId, peer: EndpointId }, NeighborUp { session: SessionId, peer: EndpointId }, NeighborDown { session: SessionId, peer: EndpointId }, GossipLagged { session: SessionId }, diff --git a/crates/flowstate-collab/src/net/runtime.rs b/crates/flowstate-collab/src/net/runtime.rs index 5c574b2b..2b77869c 100644 --- a/crates/flowstate-collab/src/net/runtime.rs +++ b/crates/flowstate-collab/src/net/runtime.rs @@ -1,10 +1,20 @@ -use std::{collections::HashMap, sync::OnceLock, thread, time::Duration}; +use std::{ + collections::HashMap, + sync::{Mutex, OnceLock}, + thread, + time::Duration, +}; use anyhow::{Context as _, Result}; -use iroh::{Endpoint, EndpointAddr, endpoint::presets}; +use iroh::{Endpoint, EndpointAddr, Watcher as _, endpoint::presets}; use iroh_gossip::net::Gossip; +use tokio::time::timeout; +use tracing::warn; -use crate::proto_direct::{AssetBytes, DirectRequest}; +use crate::{ + SessionId, + proto_direct::{AssetBytes, DirectRequest}, +}; use super::{ NetCommand, NetEvent, TicketSeed, @@ -16,7 +26,8 @@ pub type CommandSender = async_channel::Sender; pub type EventReceiver = async_channel::Receiver; const DIRECT_PULL_TIMEOUT: Duration = Duration::from_secs(10); -static RUNTIME: OnceLock = OnceLock::new(); +const ENDPOINT_ADDR_READY_TIMEOUT: Duration = Duration::from_secs(2); +static RUNTIME: OnceLock>> = OnceLock::new(); #[derive(Clone)] struct RuntimeBridge { @@ -24,17 +35,28 @@ struct RuntimeBridge { events: EventReceiver, } -pub fn start() -> Result<(CommandSender, EventReceiver)> { - if let Some(runtime) = RUNTIME.get() { - return Ok((runtime.commands.clone(), runtime.events.clone())); +impl RuntimeBridge { + fn is_alive(&self) -> bool { + !self.commands.is_closed() } - let runtime = spawn_runtime()?; - let set_failed = RUNTIME.set(runtime.clone()).is_err(); - if set_failed && let Some(runtime) = RUNTIME.get() { - return Ok((runtime.commands.clone(), runtime.events.clone())); + fn handles(&self) -> (CommandSender, EventReceiver) { + (self.commands.clone(), self.events.clone()) + } +} + +pub fn start() -> Result<(CommandSender, EventReceiver)> { + let runtime = RUNTIME.get_or_init(|| Mutex::new(None)); + let mut runtime = runtime.lock().expect("collaboration runtime lock poisoned"); + if let Some(runtime) = runtime.as_ref().filter(|runtime| runtime.is_alive()) { + return Ok(runtime.handles()); } - Ok((runtime.commands, runtime.events)) + + let next_runtime = spawn_runtime()?; + let handles = next_runtime.handles(); + *runtime = Some(next_runtime); + drop(runtime); + Ok(handles) } fn spawn_runtime() -> Result { @@ -53,14 +75,14 @@ fn spawn_runtime() -> Result { Ok(runtime) => runtime, Err(error) => { let _ = thread_evt_tx.try_send(NetEvent::EndpointOnline(false)); - eprintln!("flowstate collaboration tokio runtime failed: {error:#}"); + tracing::warn!("flowstate collaboration tokio runtime failed: {error:#}"); return; }, }; if let Err(error) = runtime.block_on(net_main(cmd_rx, evt_tx)) { let _ = thread_evt_tx.try_send(NetEvent::EndpointOnline(false)); - eprintln!("flowstate collaboration networking stopped: {error:#}"); + tracing::warn!("flowstate collaboration networking stopped: {error:#}"); } }) .context("spawning collaboration networking thread failed")?; @@ -97,32 +119,22 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha direct_state.register_handler(session, handler).await; }, NetCommand::CreateSession { session, reply } => { - direct_state.attach_session(session).await; - replace_swarm( - &mut swarms, - SwarmHandle::spawn( - endpoint.clone(), - gossip.clone(), - direct_state.clone(), - session, - Vec::new(), - evt_tx.clone(), - )?, - ) - .await; - let _ = reply - .send(Ok(TicketSeed { - inviter: endpoint_addr(&endpoint), - })) - .await; + let result = create_session(&mut swarms, &endpoint, &gossip, &direct_state, &evt_tx, session).await; + if let Err(error) = &result { + warn!("flowstate collaboration create session failed: {error:#}"); + } + let _ = reply.send(result).await; }, NetCommand::JoinSession { session, bootstrap } => { - direct_state.attach_session(session).await; - replace_swarm( - &mut swarms, - SwarmHandle::spawn(endpoint.clone(), gossip.clone(), direct_state.clone(), session, bootstrap, evt_tx.clone())?, - ) - .await; + if let Err(error) = join_session(&mut swarms, &endpoint, &gossip, &direct_state, &evt_tx, session, bootstrap).await { + warn!("flowstate collaboration join session failed: {error:#}"); + let _ = evt_tx + .send(NetEvent::SubscribeFailed { + session, + error: format!("{error:#}"), + }) + .await; + } }, NetCommand::LeaveSession { session } => { if let Some(handle) = swarms.remove(&session) { @@ -131,8 +143,10 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha direct_state.detach_session(session).await; }, NetCommand::Publish { session, payload } => { - if let Some(handle) = swarms.get(&session) { - handle.publish(payload).await?; + if let Some(handle) = swarms.get(&session) + && let Err(error) = handle.publish(payload).await + { + warn!("flowstate collaboration publish failed: {error:#}"); } }, NetCommand::PullUpdates { @@ -144,8 +158,13 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha let result = direct::pull_with_fallback(DirectRequest::Updates { session, have_vv: our_vv }, candidates, DIRECT_PULL_TIMEOUT).await; let _ = reply.send(result).await; }, - NetCommand::PullSnapshot { session, candidates, reply } => { - let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, candidates, DIRECT_PULL_TIMEOUT).await; + NetCommand::PullSnapshot { + session, + candidates, + progress, + reply, + } => { + let result = direct::pull_with_fallback_progress(DirectRequest::Snapshot { session }, candidates, DIRECT_PULL_TIMEOUT, progress).await; let _ = reply.send(result).await; }, NetCommand::PullBlob { @@ -169,7 +188,7 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha let _ = reply.send(result).await; }, NetCommand::MintTicketAddr { reply } => { - let _ = reply.send(endpoint_addr(&endpoint)).await; + let _ = reply.send(reachable_endpoint_addr(&endpoint).await).await; }, NetCommand::Shutdown => break, } @@ -182,6 +201,50 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha Ok(()) } +async fn create_session( + swarms: &mut HashMap, + endpoint: &Endpoint, + gossip: &Gossip, + direct_state: &DirectServeState, + evt_tx: &async_channel::Sender, + session: SessionId, +) -> Result { + direct_state.attach_session(session).await; + replace_swarm( + swarms, + SwarmHandle::spawn( + endpoint.clone(), + gossip.clone(), + direct_state.clone(), + session, + Vec::new(), + evt_tx.clone(), + )?, + ) + .await; + Ok(TicketSeed { + inviter: reachable_endpoint_addr(endpoint).await, + }) +} + +async fn join_session( + swarms: &mut HashMap, + endpoint: &Endpoint, + gossip: &Gossip, + direct_state: &DirectServeState, + evt_tx: &async_channel::Sender, + session: SessionId, + bootstrap: Vec, +) -> Result<()> { + direct_state.attach_session(session).await; + replace_swarm( + swarms, + SwarmHandle::spawn(endpoint.clone(), gossip.clone(), direct_state.clone(), session, bootstrap, evt_tx.clone())?, + ) + .await; + Ok(()) +} + async fn replace_swarm(swarms: &mut HashMap, handle: SwarmHandle) { let session = handle.session; if let Some(previous) = swarms.insert(session, handle) { @@ -192,3 +255,24 @@ async fn replace_swarm(swarms: &mut HashMap, hand fn endpoint_addr(endpoint: &Endpoint) -> EndpointAddr { endpoint.addr() } + +async fn reachable_endpoint_addr(endpoint: &Endpoint) -> EndpointAddr { + let mut addr_watcher = endpoint.watch_addr(); + let wait = async { + loop { + if !addr_watcher.get().is_empty() { + return; + } + tokio::select! { + () = endpoint.online() => return, + updated = addr_watcher.updated() => { + if updated.is_err() { + return; + } + }, + } + } + }; + let _ = timeout(ENDPOINT_ADDR_READY_TIMEOUT, wait).await; + endpoint_addr(endpoint) +} diff --git a/crates/flowstate-collab/src/net/swarm.rs b/crates/flowstate-collab/src/net/swarm.rs index 84a1a427..8ebf63e4 100644 --- a/crates/flowstate-collab/src/net/swarm.rs +++ b/crates/flowstate-collab/src/net/swarm.rs @@ -7,6 +7,7 @@ use iroh_gossip::{ proto::TopicId, }; use n0_future::StreamExt as _; +use tracing::warn; use crate::{ ids::SessionId, @@ -141,7 +142,15 @@ async fn handle_event(event: Event, session: SessionId, evt_tx: &Sender eprintln!("flowstate collab ignored gossip frame: {error:#}"), + Err(error) if proto_gossip::is_protocol_version_mismatch(&error) => { + let _ = evt_tx + .send(NetEvent::IncompatibleVersion { + session, + peer: message.delivered_from, + }) + .await; + }, + Err(error) => warn!("flowstate collab ignored gossip frame: {error:#}"), }, Event::NeighborUp(peer) => { let _ = evt_tx.send(NetEvent::NeighborUp { session, peer }).await; @@ -173,3 +182,50 @@ fn endpoint_id(addr: &EndpointAddr) -> EndpointId { fn topic_id(session: SessionId) -> TopicId { TopicId::from_bytes(*session.as_bytes()) } + +#[cfg(test)] +mod tests { + use iroh::SecretKey; + use iroh_gossip::{ + api::{Event, Message}, + proto::DeliveryScope, + }; + + use crate::{ + net::NetEvent, + proto_gossip::{self, GossipMsg}, + SessionId, + }; + + use super::handle_event; + + #[tokio::test] + async fn version_mismatch_emits_incompatible_version_once() { + let session = SessionId::new(); + let peer = SecretKey::generate().public(); + let mut frame = proto_gossip::encode(&GossipMsg::Presence(Vec::new())).expect("valid gossip frame"); + frame[0] = frame[0].wrapping_add(1); + + let (evt_tx, evt_rx) = async_channel::unbounded(); + handle_event( + Event::Received(Message { + content: frame.into(), + scope: DeliveryScope::Neighbors, + delivered_from: peer, + }), + session, + &evt_tx, + ) + .await + .expect("gossip event handling should ignore mismatches"); + + match evt_rx.recv().await.expect("incompatible-version event") { + NetEvent::IncompatibleVersion { session: event_session, peer: event_peer } => { + assert_eq!(event_session, session); + assert_eq!(event_peer, peer); + }, + event => panic!("unexpected event: {event:?}"), + } + assert!(evt_rx.try_recv().is_err()); + } +} diff --git a/crates/flowstate-collab/src/patch_apply.rs b/crates/flowstate-collab/src/patch_apply.rs new file mode 100644 index 00000000..5afa2162 --- /dev/null +++ b/crates/flowstate-collab/src/patch_apply.rs @@ -0,0 +1,182 @@ +//! Headless application of collaboration patches to flowtext documents. + +use anyhow::{Context as _, Result}; +use gpui_flowtext::{ + Block, CollabPatch, CollabStructuralBlock, Document, InputBlock, InputParagraph, InputRun, document_from_input_blocks, + input_block_from_block, paragraph_text, paragraphs_mut, rebuild_document_sections, update_paragraph_block, +}; +use loro::LoroDoc; + +use crate::binding::DocBinding; + +pub fn apply_patches(document: &mut Document, binding: &mut DocBinding, doc: &LoroDoc, patches: &[CollabPatch]) -> Result<()> { + for patch in patches { + apply_one(document, patch)?; + } + binding.refresh_versions(document); + binding.assert_consistent(doc, document); + Ok(()) +} + +fn apply_one(document: &mut Document, patch: &CollabPatch) -> Result<()> { + match patch { + CollabPatch::ParagraphText { row, new, .. } => { + replace_paragraph_block(document, *row, new)?; + }, + CollabPatch::ParagraphStyle { row, style } => { + if let Some(paragraph_ix) = paragraph_ix_for_block(document, *row) + && let Some(paragraph) = paragraphs_mut(document).get_mut(paragraph_ix) + { + paragraph.style = *style; + gpui_flowtext::bump_paragraph_version(paragraph); + update_paragraph_block(document, paragraph_ix); + rebuild_document_sections(document); + } + }, + CollabPatch::ReplaceObjectBlock { row, block } => { + let mut blocks = structural_blocks_from_document(document)?; + if *row < blocks.len() { + blocks[*row] = block.clone(); + rebuild_document_from_structural_blocks(document, blocks)?; + } + }, + CollabPatch::InsertBlocks { row, blocks: inserted } => { + let mut blocks = structural_blocks_from_document(document)?; + let row = (*row).min(blocks.len()); + blocks.splice(row..row, inserted.iter().cloned()); + rebuild_document_from_structural_blocks(document, blocks)?; + }, + CollabPatch::DeleteBlocks { row, count } => { + let mut blocks = structural_blocks_from_document(document)?; + let start = (*row).min(blocks.len()); + let end = start.saturating_add(*count).min(blocks.len()); + blocks.drain(start..end); + rebuild_document_from_structural_blocks(document, blocks)?; + }, + CollabPatch::MoveBlock { from, to } => { + let mut blocks = structural_blocks_from_document(document)?; + if *from < blocks.len() { + let block = blocks.remove(*from); + blocks.insert((*to).min(blocks.len()), block); + rebuild_document_from_structural_blocks(document, blocks)?; + } + }, + CollabPatch::AssetArrived { .. } => {}, + } + Ok(()) +} + +fn replace_paragraph_block(document: &mut Document, row: usize, paragraph: &InputParagraph) -> Result<()> { + let mut blocks = structural_blocks_from_document(document)?; + let Some(block) = blocks.get_mut(row) else { + return Ok(()); + }; + if !matches!(block.block, InputBlock::Paragraph(_)) { + return Ok(()); + } + block.block = InputBlock::Paragraph(paragraph.clone()); + rebuild_document_from_structural_blocks(document, blocks) +} + +fn structural_blocks_from_document(document: &Document) -> Result> { + let mut paragraph_ix = 0; + document + .blocks + .iter() + .enumerate() + .map(|(block_ix, block)| { + let block_id = document + .ids + .block_ids + .get(block_ix) + .copied() + .context("document block is missing its collaboration id")?; + match block { + Block::Paragraph(paragraph) => { + let paragraph_id = document + .ids + .paragraph_ids + .get(paragraph_ix) + .copied() + .context("document paragraph is missing its collaboration id")?; + let input = input_paragraph_from_document_paragraph(document, paragraph_ix, paragraph); + paragraph_ix += 1; + Ok(CollabStructuralBlock { + block_id, + paragraph_id: Some(paragraph_id), + block: InputBlock::Paragraph(input), + }) + }, + Block::Image(_) | Block::Equation(_) | Block::Table(_) => Ok(CollabStructuralBlock { + block_id, + paragraph_id: None, + block: input_block_from_block(block), + }), + } + }) + .collect() +} + +fn input_paragraph_from_document_paragraph(document: &Document, paragraph_ix: usize, paragraph: &gpui_flowtext::Paragraph) -> InputParagraph { + let text = paragraph_text(document, paragraph_ix); + let mut byte = 0usize; + InputParagraph { + style: paragraph.style, + runs: paragraph + .runs + .iter() + .map(|run| { + let start = byte; + let end = start.saturating_add(run.len).min(text.len()); + byte = end; + InputRun { + text: text.get(start..end).unwrap_or_default().to_string(), + styles: run.styles, + } + }) + .collect(), + } +} + +fn rebuild_document_from_structural_blocks(document: &mut Document, blocks: Vec) -> Result<()> { + let assets = document.assets.clone(); + let theme = document.theme.clone(); + let document_id = document.ids.document_id; + let input_blocks = blocks + .iter() + .map(|block| block.block.clone()) + .collect::>(); + let block_ids = blocks.iter().map(|block| block.block_id).collect::>(); + let paragraph_ids = blocks + .iter() + .filter_map(|block| match &block.block { + InputBlock::Paragraph(_) => Some(block.paragraph_id.context("structural paragraph patch is missing a paragraph id")), + InputBlock::Image(_) | InputBlock::Equation(_) | InputBlock::Table(_) => None, + }) + .collect::>>()?; + + let mut rebuilt = document_from_input_blocks(theme, input_blocks); + rebuilt.assets = assets; + rebuilt.ids.document_id = document_id; + rebuilt.ids.block_ids = block_ids; + rebuilt.ids.paragraph_ids = paragraph_ids; + rebuild_document_sections(&mut rebuilt); + debug_assert_eq!(rebuilt.ids.block_ids.len(), rebuilt.blocks.len()); + debug_assert_eq!(rebuilt.ids.paragraph_ids.len(), rebuilt.paragraphs.len()); + *document = rebuilt; + Ok(()) +} + +fn paragraph_ix_for_block(document: &Document, target_row: usize) -> Option { + let mut paragraph_ix = 0; + for (block_ix, block) in document.blocks.iter().enumerate() { + if !matches!(block, Block::Paragraph(_)) { + continue; + } + if block_ix == target_row { + return Some(paragraph_ix); + } + paragraph_ix += 1; + } + None +} diff --git a/crates/flowstate-collab/src/presence.rs b/crates/flowstate-collab/src/presence.rs index 9f1ff442..23f9c477 100644 --- a/crates/flowstate-collab/src/presence.rs +++ b/crates/flowstate-collab/src/presence.rs @@ -146,18 +146,24 @@ fn roster_entry_from_value(key: String, value: LoroValue) -> Option return None; }; let state = decode_state(bytes.as_ref()).ok()?; + let color_rgb = color_for_peer_key(&key)?; Some(RosterEntry { - color_rgb: color_for_key(&key), + color_rgb, key, name: state.name, selection: state.selection, }) } -fn color_for_key(key: &str) -> u32 { - let mut hash = 0usize; - for byte in key.as_bytes().iter().take(16) { - hash = hash.wrapping_mul(257).wrapping_add(usize::from(*byte)); +fn color_for_peer_key(key: &str) -> Option { + let mut bytes = Vec::with_capacity(key.len() / 2); + let mut chunks = key.as_bytes().chunks_exact(2); + if !chunks.remainder().is_empty() { + return None; + } + for chunk in &mut chunks { + let text = std::str::from_utf8(chunk).ok()?; + bytes.push(u8::from_str_radix(text, 16).ok()?); } - PALETTE[hash % PALETTE.len()] + Some(PALETTE[SessionId::color_index_for_peer_bytes(&bytes)]) } diff --git a/crates/flowstate-collab/src/projection.rs b/crates/flowstate-collab/src/projection.rs index 961fc657..3facd7e6 100644 --- a/crates/flowstate-collab/src/projection.rs +++ b/crates/flowstate-collab/src/projection.rs @@ -168,6 +168,10 @@ fn map_binary(map: &LoroMap, key: &str) -> Result> { pub fn verify_lineage(doc: &LoroDoc, session: SessionId) -> Result<()> { let meta = doc.get_map(META); + let schema = map_i64(&meta, META_SCHEMA).context("collaboration document has no schema version")?; + if schema != SCHEMA_VERSION { + bail!("collaboration document schema version {schema} is not supported") + } let stored = map_string(&meta, META_SESSION).context("collaboration document has no session lineage")?; if stored != session.to_string() { bail!("collaboration document lineage does not match the ticket session") diff --git a/crates/flowstate-collab/src/proto_direct.rs b/crates/flowstate-collab/src/proto_direct.rs index c1a12a48..721b3cfe 100644 --- a/crates/flowstate-collab/src/proto_direct.rs +++ b/crates/flowstate-collab/src/proto_direct.rs @@ -4,6 +4,7 @@ use serde::{Deserialize, Serialize}; use crate::ids::{BlobId, SessionId}; use crate::proto_gossip::PROTOCOL_VERSION; +pub const DIRECT_ALPN: &[u8] = b"flowstate/collab-direct/0"; pub const MAX_FRAME_LEN: usize = 2 * 1024 * 1024; pub const MAX_PAYLOAD_CHUNK_LEN: usize = 256 * 1024; @@ -31,7 +32,7 @@ pub struct AssetBytes { pub fn encode_frame(value: &T) -> Result> { let payload = postcard::to_stdvec(&(PROTOCOL_VERSION, value))?; ensure!(payload.len() <= MAX_FRAME_LEN, "direct frame exceeds {MAX_FRAME_LEN} bytes"); - let len = u32::try_from(payload.len())?; + let len = payload.len() as u32; let mut frame = len.to_le_bytes().to_vec(); frame.extend(payload); Ok(frame) diff --git a/crates/flowstate-collab/src/proto_gossip.rs b/crates/flowstate-collab/src/proto_gossip.rs index 54aa6e71..0df8de4b 100644 --- a/crates/flowstate-collab/src/proto_gossip.rs +++ b/crates/flowstate-collab/src/proto_gossip.rs @@ -1,12 +1,31 @@ -use anyhow::{Result, anyhow, ensure}; +use std::fmt; + +use anyhow::{Result, ensure}; use serde::{Deserialize, Serialize}; use crate::ids::{BlobId, SessionId}; pub const PROTOCOL_VERSION: u16 = 1; -pub const DIRECT_ALPN: &[u8] = b"flowstate/collab-direct/0"; pub const GOSSIP_INLINE_LIMIT: usize = 2 * 1024; +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct ProtocolVersionMismatch { + pub version: u16, +} + +impl fmt::Display for ProtocolVersionMismatch { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "unsupported gossip protocol version {}", self.version) + } +} + +impl std::error::Error for ProtocolVersionMismatch {} + +#[must_use] +pub fn is_protocol_version_mismatch(error: &anyhow::Error) -> bool { + error.is::() +} + #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] pub enum GossipMsg { Update(Vec), @@ -28,6 +47,8 @@ pub fn encode_inline(msg: &GossipMsg) -> Result> { } pub fn encoded_len(msg: &GossipMsg) -> Result { + // This intentionally uses the encoder: postcard has no stable counting + // serializer here, and callers only need a small inline/blob decision. Ok(encode(msg)?.len()) } @@ -35,7 +56,7 @@ pub fn decode(bytes: &[u8]) -> Result { ensure!(bytes.len() >= 2, "gossip frame is missing protocol version"); let version = u16::from_le_bytes([bytes[0], bytes[1]]); if version != PROTOCOL_VERSION { - return Err(anyhow!("unsupported gossip protocol version {version}")); + return Err(ProtocolVersionMismatch { version }.into()); } Ok(postcard::from_bytes(&bytes[2..])?) } diff --git a/crates/flowstate-collab/src/remote_apply.rs b/crates/flowstate-collab/src/remote_apply.rs index e29e3ac2..2d9488cd 100644 --- a/crates/flowstate-collab/src/remote_apply.rs +++ b/crates/flowstate-collab/src/remote_apply.rs @@ -6,7 +6,7 @@ use gpui_flowtext::{ paragraph_text, }; use loro::{ - Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroText, LoroValue, TextDelta, ValueOrContainer, + Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroText, LoroValue, TextDelta, ValueOrContainer, event::{Diff, DiffEvent, ListDiffItem}, }; @@ -27,15 +27,13 @@ impl RemoteApplier<'_> { return Ok(Vec::new()); } + let pre_event_binding = self.binding.clone(); + let inserted_containers = inserted_container_ids(self.doc, event)?; let mut patches = Vec::new(); for diff in &event.events { match &diff.diff { - Diff::Text(delta) => self.apply_text_diff(document, diff.target, delta, &mut patches)?, - Diff::Map(delta) => { - for key in delta.updated.keys() { - self.apply_map_diff(diff.target, key.as_ref(), &mut patches)?; - } - }, + Diff::Text(delta) => self.apply_text_diff(document, &pre_event_binding, &inserted_containers, diff.target, delta, &mut patches)?, + Diff::Map(delta) => self.apply_map_diff(diff.target, delta.updated.keys().map(|key| key.as_ref()), &inserted_containers, &mut patches)?, Diff::List(delta) => self.apply_list_diff(diff.target, delta, &mut patches)?, Diff::Tree(_) | Diff::Counter(_) | Diff::Unknown => {}, } @@ -43,7 +41,18 @@ impl RemoteApplier<'_> { Ok(patches) } - fn apply_text_diff(&self, document: &Document, target: &ContainerID, delta: &[TextDelta], patches: &mut Vec) -> Result<()> { + fn apply_text_diff( + &self, + document: &Document, + pre_event_binding: &DocBinding, + inserted_containers: &[ContainerID], + target: &ContainerID, + delta: &[TextDelta], + patches: &mut Vec, + ) -> Result<()> { + if inserted_containers.contains(target) { + return Ok(()); + } let Some(row_ix) = self.binding.by_container.get(target).copied() else { return Ok(()); }; @@ -60,8 +69,18 @@ impl RemoteApplier<'_> { .as_ref() .context("paragraph diff row is missing its LoroText")?; let style = decode_paragraph_style(map_i64(&row.map, STYLE)?); - let paragraph_ix = paragraph_ordinal_for_row(self.binding, row_ix).context("paragraph diff row has no paragraph ordinal")?; - let old_text = paragraph_text(document, paragraph_ix); + let old_paragraph_ix = pre_event_binding + .by_container + .get(target) + .copied() + .and_then(|old_row_ix| paragraph_ordinal_for_row(pre_event_binding, old_row_ix)); + let old_text = if let Some(paragraph_ix) = old_paragraph_ix + && paragraph_ix < document.paragraphs.len() + { + paragraph_text(document, paragraph_ix) + } else { + String::new() + }; patches.push(CollabPatch::ParagraphText { row: row_ix, new: crate::schema::input_paragraph_from_text(text, style), @@ -70,38 +89,53 @@ impl RemoteApplier<'_> { Ok(()) } - fn apply_map_diff(&mut self, target: &ContainerID, key: &str, patches: &mut Vec) -> Result<()> { + fn apply_map_diff<'a>( + &mut self, + target: &ContainerID, + keys: impl IntoIterator, + inserted_containers: &[ContainerID], + patches: &mut Vec, + ) -> Result<()> { + if inserted_containers.contains(target) { + return Ok(()); + } let Some(row_ix) = self.binding.by_container.get(target).copied() else { return Ok(()); }; + let mut style_changed = false; + let mut object_changed = false; + for key in keys { + match key { + STYLE => style_changed = true, + DATA | REV => object_changed = true, + _ => {}, + } + } let row = self .binding .rows .get_mut(row_ix) .context("map diff row is outside DocBinding")?; - match key { - STYLE if matches!(row.kind, BlockKind::Paragraph) => { - patches.push(CollabPatch::ParagraphStyle { - row: row_ix, - style: decode_paragraph_style(map_i64(&row.map, STYLE)?), - }); - }, - DATA | REV => { - let input = input_block_from_container(&row.map)?; - row.kind = block_kind_for_input(&input); - row.text = None; - row.paragraph_id = None; - row.version = 0; - patches.push(CollabPatch::ReplaceObjectBlock { - row: row_ix, - block: CollabStructuralBlock { - block_id: row.block_id, - paragraph_id: None, - block: input, - }, - }); - }, - _ => {}, + if style_changed && matches!(row.kind, BlockKind::Paragraph) { + patches.push(CollabPatch::ParagraphStyle { + row: row_ix, + style: decode_paragraph_style(map_i64(&row.map, STYLE)?), + }); + } + if object_changed { + let input = input_block_from_container(&row.map)?; + row.kind = block_kind_for_input(&input); + row.text = None; + row.paragraph_id = None; + row.version = 0; + patches.push(CollabPatch::ReplaceObjectBlock { + row: row_ix, + block: CollabStructuralBlock { + block_id: row.block_id, + paragraph_id: None, + block: input, + }, + }); } self.binding.rebuild_indexes(); Ok(()) @@ -114,28 +148,43 @@ impl RemoteApplier<'_> { } let mut row_ix = 0usize; + let final_block_ids = block_container_ids(&blocks)?; for item in delta { match item { ListDiffItem::Retain { retain } => row_ix = row_ix.saturating_add(*retain), ListDiffItem::Delete { delete } => { - let count = (*delete).min(self.binding.rows.len().saturating_sub(row_ix)); - if count == 0 { - continue; - } - for _ in 0..count { + let mut delete_start = None; + let mut delete_count = 0usize; + for _ in 0..*delete { + let Some(row) = self.binding.rows.get(row_ix) else { + break; + }; + if final_block_ids.contains(&row.map.id()) { + if let Some(start) = delete_start.take() { + patches.push(CollabPatch::DeleteBlocks { row: start, count: delete_count }); + delete_count = 0; + } + row_ix += 1; + continue; + } + delete_start.get_or_insert(row_ix); self .binding .remove_row(row_ix) .context("DocBinding row disappeared during remote block delete")?; + delete_count += 1; + } + if let Some(start) = delete_start { + patches.push(CollabPatch::DeleteBlocks { row: start, count: delete_count }); } - patches.push(CollabPatch::DeleteBlocks { row: row_ix, count }); }, ListDiffItem::Insert { insert, is_move } => { for value in insert { let map = map_from_insert(value)?; if *is_move && let Some(from) = self.binding.by_container.get(&map.id()).copied() { - patches.push(CollabPatch::MoveBlock { from, to: row_ix }); - self.binding.move_row(from, row_ix); + let to = if from < row_ix { row_ix - 1 } else { row_ix }; + patches.push(CollabPatch::MoveBlock { from, to }); + self.binding.move_row(from, to); row_ix += 1; continue; } @@ -153,6 +202,7 @@ impl RemoteApplier<'_> { }, } } + reconcile_binding_to_match_blocks(self.binding, &blocks, patches)?; Ok(()) } } @@ -221,6 +271,114 @@ fn map_from_insert(value: &ValueOrContainer) -> Result { } } +fn inserted_container_ids(doc: &LoroDoc, event: &DiffEvent<'_>) -> Result> { + let blocks_id = doc.get_movable_list(BLOCKS).id(); + let mut ids = Vec::new(); + for diff in &event.events { + let Diff::List(delta) = &diff.diff else { + continue; + }; + if *diff.target != blocks_id { + continue; + } + for item in delta { + let ListDiffItem::Insert { insert, is_move } = item else { + continue; + }; + if *is_move { + continue; + } + for value in insert { + let map = map_from_insert(value)?; + ids.push(map.id()); + if let Some(ValueOrContainer::Container(Container::Text(text))) = map.get(TEXT) { + ids.push(text.id()); + } + } + } + } + Ok(ids) +} + +fn block_container_ids(blocks: &LoroMovableList) -> Result> { + let mut ids = Vec::with_capacity(blocks.len()); + for ix in 0..blocks.len() { + ids.push(map_from_list(blocks, ix)?.id()); + } + Ok(ids) +} + +fn reconcile_binding_to_match_blocks(binding: &mut DocBinding, blocks: &LoroMovableList, patches: &mut Vec) -> Result<()> { + let final_ids = block_container_ids(blocks)?; + remove_stale_binding_rows(binding, &final_ids, patches); + insert_missing_binding_rows(binding, blocks, &final_ids, patches)?; + for to in (0..final_ids.len()).rev() { + if binding + .rows + .get(to) + .is_some_and(|row| row.map.id() == final_ids[to]) + { + continue; + } + let from = binding + .by_container + .get(&final_ids[to]) + .copied() + .context("final Loro block row is missing from DocBinding")?; + patches.push(CollabPatch::MoveBlock { from, to }); + binding.move_row(from, to); + } + Ok(()) +} + +fn remove_stale_binding_rows(binding: &mut DocBinding, final_ids: &[ContainerID], patches: &mut Vec) { + let mut retained_ids = Vec::new(); + let mut delete_rows = Vec::new(); + for (row_ix, row) in binding.rows.iter().enumerate() { + let id = row.map.id(); + if !final_ids.contains(&id) || retained_ids.contains(&id) { + delete_rows.push(row_ix); + } else { + retained_ids.push(id); + } + } + + for row_ix in delete_rows.into_iter().rev() { + let _ = binding.remove_row(row_ix); + patches.push(CollabPatch::DeleteBlocks { row: row_ix, count: 1 }); + } +} + +fn insert_missing_binding_rows( + binding: &mut DocBinding, + blocks: &LoroMovableList, + final_ids: &[ContainerID], + patches: &mut Vec, +) -> Result<()> { + for (row_ix, id) in final_ids.iter().enumerate() { + if binding.by_container.contains_key(id) { + continue; + } + let map = map_from_list(blocks, row_ix)?; + let input = input_block_from_container(&map)?; + let structural = structural_block_for_insert(&input); + let row = binding_row_from_insert(map, &input, structural.block_id, structural.paragraph_id)?; + binding.insert_row(row_ix, row); + patches.push(CollabPatch::InsertBlocks { + row: row_ix, + blocks: vec![structural], + }); + } + Ok(()) +} + +fn map_from_list(blocks: &LoroMovableList, ix: usize) -> Result { + match blocks.get(ix) { + Some(ValueOrContainer::Container(Container::Map(map))) => Ok(map), + Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => bail!("remote block row {ix} is not a map container"), + } +} + fn structural_block_for_insert(input: &InputBlock) -> CollabStructuralBlock { CollabStructuralBlock { block_id: new_block_id(), diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs index 2b4675e8..d3a8f79d 100644 --- a/crates/flowstate-collab/src/schema.rs +++ b/crates/flowstate-collab/src/schema.rs @@ -9,8 +9,8 @@ use std::{collections::HashMap, hash::BuildHasher, ops::Range}; use anyhow::Result; use gpui_flowtext::{ - AssetStore, Block, HighlightStyle, InputBlock, InputBlockAlignment, InputEquationDisplay, InputImageSizing, InputParagraph, InputRun, - InputTableBlock, RunSemanticStyle, RunStyles, TextRun, input_block_from_block, + AssetStore, Block, Document, HighlightStyle, InputBlock, InputBlockAlignment, InputEquationDisplay, InputImageSizing, InputParagraph, + InputRun, InputTableBlock, RunSemanticStyle, RunStyles, TextRun, input_block_from_block, paragraph_text_len, }; use loro::{ExpandType, LoroDoc, LoroResult, LoroText, LoroValue, StyleConfig, StyleConfigMap, TextDelta, cursor::PosType}; use serde::{Deserialize, Serialize}; @@ -277,6 +277,12 @@ pub fn apply_mark_intervals(text: &LoroText, intervals: &MarkIntervals) -> Resul Ok(()) } +pub fn debug_assert_paragraph_text_len(text: &LoroText, document: &Document, paragraph_ix: usize) { + if let Some(paragraph) = document.paragraphs.get(paragraph_ix) { + debug_assert_eq!(text.len_utf8(), paragraph_text_len(paragraph)); + } +} + #[must_use] pub const fn encode_paragraph_style(style: gpui_flowtext::ParagraphStyle) -> i64 { match style { diff --git a/crates/flowstate-collab/src/self_check.rs b/crates/flowstate-collab/src/self_check.rs index 86d25c17..48f3a0d7 100644 --- a/crates/flowstate-collab/src/self_check.rs +++ b/crates/flowstate-collab/src/self_check.rs @@ -14,13 +14,13 @@ pub fn projection_hash(document: &Document) -> u64 { for block in document.blocks.iter() { match block { Block::Paragraph(paragraph) => { - "p".hash(&mut hasher); paragraph.style.hash(&mut hasher); - paragraph_text(document, paragraph_ix).hash(&mut hasher); for run in ¶graph.runs { run.len.hash(&mut hasher); run.styles.hash(&mut hasher); } + paragraph_text(document, paragraph_ix).as_bytes().hash(&mut hasher); + "p".hash(&mut hasher); paragraph_ix += 1; }, Block::Image(_) | Block::Equation(_) | Block::Table(_) => { diff --git a/crates/flowstate-collab/src/ticket.rs b/crates/flowstate-collab/src/ticket.rs index 9d10092c..e0ed76e6 100644 --- a/crates/flowstate-collab/src/ticket.rs +++ b/crates/flowstate-collab/src/ticket.rs @@ -43,6 +43,8 @@ impl SessionTicket { impl Ticket for SessionTicket { const KIND: &'static str = TICKET_KIND; + /// `iroh_tickets::Ticket::encode_bytes` cannot return a `Result`; `SessionTicket` + /// only contains postcard-serializable fields, so serialization failure would be a programmer error. fn encode_bytes(&self) -> Vec { postcard::to_stdvec(self).expect("SessionTicket serialization should not fail") } diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs index f39a5b17..a5533292 100644 --- a/crates/flowstate-collab/tests/convergence.rs +++ b/crates/flowstate-collab/tests/convergence.rs @@ -1,59 +1,270 @@ #[cfg(test)] mod tests { - use std::sync::{Arc, Mutex}; + use std::{fmt::Write as _, ops::Range, sync::{Arc, Mutex}}; - use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; + use anyhow::{Context as _, Result, anyhow, bail}; + use flowstate_collab::{ + SessionId, + binding::DocBinding, + local_apply::LocalApplier, + patch_apply::apply_patches, + projection, schema, + }; use gpui_flowtext::{ - CanonicalOperation, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyle, RunStyles, document_from_input_blocks, - insert_text_at, mutate_runs_in_range, paragraph_text, plain, + Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, HighlightStyle, InputBlock, InputEquationBlock, InputEquationDisplay, + InputEquationSyntax, InputParagraph, ParagraphStyle, RunSemanticStyle, RunStyle, RunStyles, block_from_input_block, + delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, mutate_runs_in_range, + paragraph_text, plain, remove_block_ids, split_paragraph_at, }; - use loro::{ExportMode, LoroDoc, Subscription}; + use loro::{ExportMode, LoroDoc, Subscription, event::Subscriber}; + use proptest::{prelude::*, test_runner::TestCaseError}; const MULTIBYTE: &str = "aé🌍\u{2028}x"; + proptest! { + #![proptest_config(ProptestConfig { cases: 256, max_shrink_iters: 2048, ..ProptestConfig::default() })] + + #[test] + fn peers_converge_with_reordered_duplicate_and_buffered_updates( + peer_count in 2usize..=3, + delivery_seed in any::(), + ops in prop::collection::vec(fuzz_op(), 1..28), + ) { + run_program(peer_count, delivery_seed, &ops) + .map_err(|error| TestCaseError::fail(format!("{error:#}\nprogram: {ops:#?}")))?; + } + } + + #[test] + fn style_then_join_converges_when_updates_reorder() { + let ops = [ + FuzzOp::Style { + peer: 152, + paragraph: 0, + start: 87, + len: 53, + style: 3, + }, + FuzzOp::Join { peer: 162, paragraph: 0 }, + ]; + + run_program(2, 0, &ops).expect("style then join should converge"); + } + #[test] - fn two_peers_converge_with_reordered_and_duplicated_update_imports() { + fn insert_object_then_split_converges() { + let ops = [ + FuzzOp::InsertObject { peer: 6, row: 70, source: 0 }, + FuzzOp::Split { + peer: 0, + paragraph: 188, + byte: 0, + }, + ]; + + run_program(2, 0, &ops).expect("insert object then split should converge"); + } + + #[test] + fn insert_object_at_start_then_split_converges_across_three_peers() { + let ops = [ + FuzzOp::InsertObject { peer: 96, row: 129, source: 0 }, + FuzzOp::Split { + peer: 108, + paragraph: 91, + byte: 0, + }, + ]; + + run_program(3, 0, &ops).expect("insert object at start then split should converge"); + } + + + #[test] + fn insert_object_between_paragraphs_does_not_create_invalid_join() { + let ops = [ + FuzzOp::InsertObject { peer: 6, row: 70, source: 0 }, + FuzzOp::Join { peer: 162, paragraph: 0 }, + ]; + + run_program(2, 0, &ops).expect("non-adjacent paragraph join should be skipped"); + } + + #[test] + fn insert_object_between_paragraphs_then_text_edit_preserves_order() { + let ops = [ + FuzzOp::InsertObject { peer: 7, row: 106, source: 0 }, + FuzzOp::DeleteText { + peer: 19, + paragraph: 48, + start: 0, + len: 0, + }, + ]; + + run_program(2, 0, &ops).expect("text edit after object insert should preserve block order"); + } + + #[test] + fn concurrent_object_insert_move_interleaving_converges() { + let ops = [ + FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, + FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, + FuzzOp::Join { peer: 86, paragraph: 76 }, + FuzzOp::Style { peer: 144, paragraph: 122, start: 123, len: 147, style: 171 }, + FuzzOp::DeleteObject { peer: 212, row: 116 }, + FuzzOp::Split { peer: 44, paragraph: 85, byte: 123 }, + FuzzOp::InsertText { peer: 47, paragraph: 61, byte: 160, text: 91, style: 192 }, + FuzzOp::DeleteObject { peer: 216, row: 243 }, + FuzzOp::DeleteText { peer: 86, paragraph: 45, start: 114, len: 95 }, + FuzzOp::InsertObject { peer: 122, row: 21, source: 177 }, + FuzzOp::InsertText { peer: 44, paragraph: 23, byte: 38, text: 111, style: 124 }, + FuzzOp::MoveObject { peer: 1, row: 223, to: 58 }, + FuzzOp::InsertObject { peer: 167, row: 184, source: 207 }, + FuzzOp::InsertText { peer: 199, paragraph: 128, byte: 125, text: 41, style: 165 }, + FuzzOp::MoveObject { peer: 211, row: 103, to: 135 }, + ]; + + run_program(2, 9640017939277306283, &ops).expect("concurrent object move interleaving should converge"); + } + + #[test] + fn buffered_object_split_join_move_program_converges() { + let ops = [ + FuzzOp::Style { peer: 34, paragraph: 235, start: 63, len: 151, style: 35 }, + FuzzOp::DeleteText { peer: 10, paragraph: 205, start: 201, len: 24 }, + FuzzOp::Style { peer: 244, paragraph: 167, start: 13, len: 60, style: 35 }, + FuzzOp::InsertText { peer: 159, paragraph: 175, byte: 242, text: 64, style: 7 }, + FuzzOp::Style { peer: 86, paragraph: 113, start: 237, len: 135, style: 240 }, + FuzzOp::DeleteText { peer: 186, paragraph: 150, start: 45, len: 253 }, + FuzzOp::InsertText { peer: 236, paragraph: 70, byte: 124, text: 228, style: 238 }, + FuzzOp::Style { peer: 212, paragraph: 57, start: 248, len: 168, style: 47 }, + FuzzOp::Style { peer: 128, paragraph: 69, start: 13, len: 201, style: 209 }, + FuzzOp::InsertObject { peer: 208, row: 159, source: 38 }, + FuzzOp::DeleteText { peer: 70, paragraph: 2, start: 247, len: 227 }, + FuzzOp::DeleteText { peer: 45, paragraph: 59, start: 175, len: 74 }, + FuzzOp::InsertText { peer: 234, paragraph: 209, byte: 216, text: 230, style: 207 }, + FuzzOp::DeleteText { peer: 197, paragraph: 38, start: 169, len: 78 }, + FuzzOp::InsertText { peer: 118, paragraph: 146, byte: 65, text: 27, style: 176 }, + FuzzOp::Split { peer: 32, paragraph: 124, byte: 11 }, + FuzzOp::Join { peer: 19, paragraph: 41 }, + FuzzOp::MoveObject { peer: 94, row: 196, to: 197 }, + FuzzOp::DeleteText { peer: 56, paragraph: 171, start: 218, len: 154 }, + FuzzOp::DeleteObject { peer: 222, row: 98 }, + ]; + + run_program(3, 6385737403235304726, &ops).expect("minimized buffered update program should converge"); + } + + #[derive(Clone, Debug)] + enum FuzzOp { + InsertText { peer: u8, paragraph: u8, byte: u8, text: u8, style: u8 }, + DeleteText { peer: u8, paragraph: u8, start: u8, len: u8 }, + Split { peer: u8, paragraph: u8, byte: u8 }, + Join { peer: u8, paragraph: u8 }, + Style { peer: u8, paragraph: u8, start: u8, len: u8, style: u8 }, + InsertObject { peer: u8, row: u8, source: u8 }, + DeleteObject { peer: u8, row: u8 }, + MoveObject { peer: u8, row: u8, to: u8 }, + } + + impl FuzzOp { + fn peer(&self) -> u8 { + match self { + Self::InsertText { peer, .. } + | Self::DeleteText { peer, .. } + | Self::Split { peer, .. } + | Self::Join { peer, .. } + | Self::Style { peer, .. } + | Self::InsertObject { peer, .. } + | Self::DeleteObject { peer, .. } + | Self::MoveObject { peer, .. } => *peer, + } + } + } + + fn fuzz_op() -> impl Strategy { + prop_oneof![ + 7 => (any::(), any::(), any::(), any::(), any::()).prop_map(|(peer, paragraph, byte, text, style)| FuzzOp::InsertText { peer, paragraph, byte, text, style }), + 7 => (any::(), any::(), any::(), any::()).prop_map(|(peer, paragraph, start, len)| FuzzOp::DeleteText { peer, paragraph, start, len }), + 1 => (any::(), any::(), any::()).prop_map(|(peer, paragraph, byte)| FuzzOp::Split { peer, paragraph, byte }), + 1 => (any::(), any::()).prop_map(|(peer, paragraph)| FuzzOp::Join { peer, paragraph }), + 2 => (any::(), any::(), any::(), any::(), any::()).prop_map(|(peer, paragraph, start, len, style)| FuzzOp::Style { peer, paragraph, start, len, style }), + 1 => (any::(), any::(), any::()).prop_map(|(peer, row, source)| FuzzOp::InsertObject { peer, row, source }), + 1 => (any::(), any::()).prop_map(|(peer, row)| FuzzOp::DeleteObject { peer, row }), + 1 => (any::(), any::(), any::()).prop_map(|(peer, row, to)| FuzzOp::MoveObject { peer, row, to }), + ] + } + + fn run_program(peer_count: usize, delivery_seed: u64, ops: &[FuzzOp]) -> Result<()> { let session = SessionId::from_bytes([77; 32]); let initial = document_from_input_blocks( DocumentTheme::default(), vec![ - InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain(MULTIBYTE)], - }), - InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain("second aé🌍")], - }), + paragraph_block(MULTIBYTE), + paragraph_block("second aé🌍"), ], ); - let mut peer_a = Peer::from_initial(session, initial); - let mut peer_b = Peer::from_snapshot(session, &peer_a.snapshot()); - - peer_a.insert_text(0, "a".len(), "A🌍", RunStyles::default().with(RunStyle::Semantic(2))); - let inserted = "A🌍".len(); - peer_a.set_run_styles( - 0, - "a".len().."a".len() + inserted, - RunStyles::default() - .with(RunStyle::Highlight(4)) - .with_direct_underline(), - ); - peer_b.insert_text(1, "second ".len(), "Bé", RunStyles::default().with_strikethrough()); + let mut peers = Vec::with_capacity(peer_count); + let first = Peer::from_initial(session, initial)?; + let snapshot = first.snapshot()?; + peers.push(first); + for _ in 1..peer_count { + peers.push(Peer::from_snapshot(session, &snapshot)?); + } + + let mut network = Vec::new(); + for (step, op) in ops.iter().enumerate() { + let source = usize::from(op.peer()) % peer_count; + if !peers[source].apply_op(op)? { + continue; + } + for update in peers[source].drain_updates() { + for target in 0..peer_count { + if target == source { + continue; + } + let copies = 1 + usize::from(pseudo(delivery_seed, step, source, target, 0).is_multiple_of(5)); + for copy in 0..copies { + let buffered = pseudo(delivery_seed, step, source, target, copy).is_multiple_of(7); + let key = pseudo(delivery_seed, step, source, target, copy + 17) + .saturating_add(if buffered { 10_000 + step as u64 } else { 0 }); + network.push(Message { key, target, bytes: update.clone() }); + } + } + } + } - let updates_a = peer_a.updates(); - let updates_b = peer_b.updates(); - assert_eq!(updates_a.len(), 2); - assert_eq!(updates_b.len(), 1); + network.sort_by_key(|message| message.key); + for message in network { + peers[message.target].import_update(&message.bytes)?; + } - // Import A's second commit before its first commit to exercise Loro's pending-import path, - // then replay duplicates to assert idempotency. - import_sequence(&peer_b.loro, [&updates_a[1], &updates_a[1], &updates_a[0], &updates_a[0]]); - import_sequence(&peer_a.loro, [&updates_b[0], &updates_b[0]]); + let projected = projection::document_from_loro(&peers[0].loro, DocumentTheme::default())?; + let expected = canonical_document_bytes(&projected)?; + for (peer_ix, peer) in peers.iter().enumerate() { + let peer_projection = projection::document_from_loro(&peer.loro, DocumentTheme::default())?; + let projection_bytes = canonical_document_bytes(&peer_projection)?; + let document_bytes = canonical_document_bytes(&peer.document)?; + if projection_bytes != expected { + bail!("peer Loro projection diverged"); + } + if document_bytes != expected { + bail!( + "peer {peer_ix} document diverged from Loro projection\nexpected: {}\nactual: {}\npeer projection: {}", + describe_document(&projected), + describe_document(&peer.document), + describe_document(&peer_projection), + ); + } + } + Ok(()) + } - let projected_a = projection::document_from_loro(&peer_a.loro, DocumentTheme::default()).expect("projection A should succeed"); - let projected_b = projection::document_from_loro(&peer_b.loro, DocumentTheme::default()).expect("projection B should succeed"); - assert_documents_eq(&projected_a, &projected_b); + struct Message { + key: u64, + target: usize, + bytes: Vec, } struct Peer { @@ -65,108 +276,484 @@ mod tests { } impl Peer { - fn from_initial(session: SessionId, document: Document) -> Self { + fn from_initial(session: SessionId, document: Document) -> Result { let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, session, "convergence", &document).expect("populate should succeed"); - let binding = DocBinding::build(&loro, &document).expect("binding should build"); - Self::with_subscription(document, loro, binding) + projection::populate_from_document(&loro, session, "convergence", &document)?; + let binding = DocBinding::build(&loro, &document)?; + Ok(Self::with_subscription(document, loro, binding)) } - fn from_snapshot(session: SessionId, snapshot: &[u8]) -> Self { + fn from_snapshot(session: SessionId, snapshot: &[u8]) -> Result { let loro = schema::new_configured_doc(); - loro - .import(snapshot) - .expect("snapshot import should succeed"); - projection::verify_lineage(&loro, session).expect("lineage should match"); - let document = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); - let binding = DocBinding::build(&loro, &document).expect("binding should build"); - Self::with_subscription(document, loro, binding) + loro.import(snapshot)?; + projection::verify_lineage(&loro, session)?; + let document = projection::document_from_loro(&loro, DocumentTheme::default())?; + let binding = DocBinding::build(&loro, &document)?; + Ok(Self::with_subscription(document, loro, binding)) } fn with_subscription(document: Document, loro: LoroDoc, binding: DocBinding) -> Self { let updates = Arc::new(Mutex::new(Vec::new())); let captured = updates.clone(); let subscription = loro.subscribe_local_update(Box::new(move |bytes| { - captured - .lock() - .expect("update capture lock should not be poisoned") - .push(bytes.clone()); + captured.lock().expect("update capture lock should not be poisoned").push(bytes.clone()); true })); - Self { - document, - loro, - binding, - updates, - _subscription: subscription, + Self { document, loro, binding, updates, _subscription: subscription } + } + + fn snapshot(&self) -> Result> { + Ok(self.loro.export(ExportMode::Snapshot)?) + } + + fn drain_updates(&self) -> Vec> { + let mut updates = self.updates.lock().expect("update capture lock should not be poisoned"); + std::mem::take(&mut *updates) + } + + fn apply_op(&mut self, op: &FuzzOp) -> Result { + let Some(ops) = self.mutate_document(op)? else { + return Ok(false); + }; + LocalApplier { doc: &self.loro, binding: &mut self.binding } + .apply(&self.document, &ops)?; + Ok(true) + } + + fn mutate_document(&mut self, op: &FuzzOp) -> Result>> { + match *op { + FuzzOp::InsertText { paragraph, byte, text, style, .. } => { + if self.document.paragraphs.is_empty() { + return Ok(None); + } + let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); + let value = text_choice(text); + let styles = style_choice(style); + let offset = byte_boundary(¶graph_text(&self.document, paragraph_ix), byte); + let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; + insert_text_preserving_blocks(&mut self.document, paragraph_ix, offset, value, styles); + Ok(Some(vec![CanonicalOperation::InsertText { + paragraph: paragraph_id, + byte: offset, + text: value.to_string(), + styles, + }])) + }, + FuzzOp::DeleteText { paragraph, start, len, .. } => { + if self.document.paragraphs.is_empty() { + return Ok(None); + } + let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); + let text = paragraph_text(&self.document, paragraph_ix); + let Some(range) = non_empty_range(&text, start, len) else { + return Ok(None); + }; + let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; + delete_text_preserving_blocks(&mut self.document, paragraph_ix, range.clone()); + Ok(Some(vec![CanonicalOperation::DeleteRange { + start_paragraph: paragraph_id, + start_byte: range.start, + end_paragraph: paragraph_id, + end_byte: range.end, + }])) + }, + FuzzOp::Split { paragraph, byte, .. } => { + if self.document.paragraphs.len() >= 6 || self.document.paragraphs.is_empty() { + return Ok(None); + } + let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); + let split_byte = byte_boundary(¶graph_text(&self.document, paragraph_ix), byte); + let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; + let Some(new_paragraph) = split_paragraph_preserving_blocks(&mut self.document, paragraph_ix, split_byte) else { + return Ok(None); + }; + Ok(Some(vec![CanonicalOperation::SplitParagraph { + paragraph: paragraph_id, + byte: split_byte, + new_paragraph, + }])) + }, + FuzzOp::Join { paragraph, .. } => { + let pairs = adjacent_paragraph_pairs(&self.document); + if pairs.is_empty() { + return Ok(None); + } + let first_ix = pairs[usize::from(paragraph) % pairs.len()]; + let first = self.document.ids.paragraph_ids[first_ix]; + let second = self.document.ids.paragraph_ids[first_ix + 1]; + let first_len = paragraph_text(&self.document, first_ix).len(); + delete_cross_paragraph_range( + &mut self.document, + DocumentOffset { paragraph: first_ix, byte: first_len }..DocumentOffset { paragraph: first_ix + 1, byte: 0 }, + ); + Ok(Some(vec![CanonicalOperation::JoinParagraphs { first, second }])) + }, + FuzzOp::Style { paragraph, start, len, style, .. } => { + if self.document.paragraphs.is_empty() { + return Ok(None); + } + let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); + let text = paragraph_text(&self.document, paragraph_ix); + let Some(range) = non_empty_range(&text, start, len) else { + return Ok(None); + }; + let styles = style_choice(style); + let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; + mutate_runs_in_range( + &mut self.document, + DocumentOffset { paragraph: paragraph_ix, byte: range.start }..DocumentOffset { paragraph: paragraph_ix, byte: range.end }, + |run_styles| *run_styles = styles, + ); + Ok(Some(vec![CanonicalOperation::SetRunStyles { paragraph: paragraph_id, range, styles }])) + }, + FuzzOp::InsertObject { row, source, .. } => { + if self.document.blocks.len() >= 8 { + return Ok(None); + } + let row = usize::from(row) % (self.document.blocks.len() + 1); + let input = equation_block(equation_source(source)); + Arc::make_mut(&mut self.document.blocks).insert(row, block_from_input_block(&input)); + let block = insert_block_id(&mut self.document, row); + Ok(Some(vec![CanonicalOperation::InsertBlock { block, block_ix: row }])) + }, + FuzzOp::DeleteObject { row, .. } => { + let objects = object_rows(&self.document); + if objects.is_empty() { + return Ok(None); + } + let row = objects[usize::from(row) % objects.len()]; + let block = self.document.ids.block_ids[row]; + Arc::make_mut(&mut self.document.blocks).remove(row); + remove_block_ids(&mut self.document, row..row + 1); + Ok(Some(vec![CanonicalOperation::DeleteBlock { block }])) + }, + FuzzOp::MoveObject { row, to, .. } => { + let objects = object_rows(&self.document); + if objects.is_empty() || self.document.blocks.len() < 2 { + return Ok(None); + } + let from = objects[usize::from(row) % objects.len()]; + let to = usize::from(to) % self.document.blocks.len(); + if from == to { + return Ok(None); + } + let block = self.document.ids.block_ids[from]; + let moved_block = Arc::make_mut(&mut self.document.blocks).remove(from); + let insert_ix = to.min(self.document.blocks.len()); + Arc::make_mut(&mut self.document.blocks).insert(insert_ix, moved_block); + let block_id = self.document.ids.block_ids.remove(from); + self.document.ids.block_ids.insert(insert_ix.min(self.document.ids.block_ids.len()), block_id); + Ok(Some(vec![CanonicalOperation::MoveBlock { block, new_block_ix: to }])) + }, } } - fn snapshot(&self) -> Vec { - self - .loro - .export(ExportMode::Snapshot) - .expect("snapshot export should succeed") - } - - fn updates(&self) -> Vec> { - self - .updates - .lock() - .expect("update capture lock should not be poisoned") - .clone() - } - - fn insert_text(&mut self, paragraph_ix: usize, byte: usize, text: &str, styles: RunStyles) { - let paragraph = self.document.ids.paragraph_ids[paragraph_ix]; - insert_text_at(&mut self.document, paragraph_ix, byte, text, styles); - self.apply(&[CanonicalOperation::InsertText { - paragraph, - byte, - text: text.to_string(), - styles, - }]); - } - - fn set_run_styles(&mut self, paragraph_ix: usize, range: std::ops::Range, styles: RunStyles) { - let paragraph = self.document.ids.paragraph_ids[paragraph_ix]; - mutate_runs_in_range( - &mut self.document, - gpui_flowtext::DocumentOffset { - paragraph: paragraph_ix, - byte: range.start, - }..gpui_flowtext::DocumentOffset { - paragraph: paragraph_ix, - byte: range.end, + fn import_update(&mut self, update: &[u8]) -> Result<()> { + let document = Arc::new(self.document.clone()); + let binding = Arc::new(Mutex::new(std::mem::take(&mut self.binding))); + let patches = Arc::new(Mutex::new(Vec::new())); + let error = Arc::new(Mutex::new(None::)); + let doc = self.loro.clone(); + let callback: Subscriber = Arc::new({ + let binding = binding.clone(); + let patches = patches.clone(); + let error = error.clone(); + move |event| { + let result = binding + .lock() + .map_err(|lock_error| anyhow!("binding lock poisoned: {lock_error}")) + .and_then(|mut binding| { + flowstate_collab::remote_apply::RemoteApplier { doc: &doc, binding: &mut binding } + .apply_event(&document, &event) + }); + match result { + Ok(mut produced) => patches.lock().expect("patch lock should not be poisoned").append(&mut produced), + Err(apply_error) => *error.lock().expect("error lock should not be poisoned") = Some(format!("{apply_error:#}")), + } + } + }); + let subscription = self.loro.subscribe_root(callback); + let import_result = self.loro.import_with(update, "remote"); + drop(subscription); + self.binding = Arc::try_unwrap(binding) + .map_err(|_| anyhow!("binding callback outlived import"))? + .into_inner() + .map_err(|error| anyhow!("binding lock poisoned: {error}"))?; + import_result.context("importing remote update failed")?; + let remote_error = error.lock().expect("error lock should not be poisoned").take(); + if let Some(error) = remote_error { + bail!("remote apply failed: {error}"); + } + let patches = Arc::try_unwrap(patches) + .map_err(|_| anyhow!("patch callback outlived import"))? + .into_inner() + .map_err(|error| anyhow!("patch lock poisoned: {error}"))?; + apply_patches(&mut self.document, &mut self.binding, &self.loro, &patches)?; + Ok(()) + } + } + + fn paragraph_block(text: &str) -> InputBlock { + InputBlock::Paragraph(InputParagraph { style: ParagraphStyle::Normal, runs: vec![plain(text)] }) + } + + fn equation_block(source: &str) -> InputBlock { + InputBlock::Equation(InputEquationBlock { + source: source.to_string(), + syntax: InputEquationSyntax::Latex, + display: InputEquationDisplay::Display, + }) + } + + fn object_rows(document: &Document) -> Vec { + document + .blocks + .iter() + .enumerate() + .filter_map(|(ix, block)| (!matches!(block, Block::Paragraph(_))).then_some(ix)) + .collect() + } + + fn adjacent_paragraph_pairs(document: &Document) -> Vec { + let mut pairs = Vec::new(); + let mut paragraph_ix = 0; + let mut previous_paragraph_ix = None; + for block in document.blocks.iter() { + if matches!(block, Block::Paragraph(_)) { + if let Some(previous) = previous_paragraph_ix { + pairs.push(previous); + } + previous_paragraph_ix = Some(paragraph_ix); + paragraph_ix += 1; + } else { + previous_paragraph_ix = None; + } + } + pairs + } + + fn insert_text_preserving_blocks(document: &mut Document, paragraph_ix: usize, byte: usize, text: &str, styles: RunStyles) { + let Some(snapshot) = block_order_snapshot(document, paragraph_ix) else { + return; + }; + insert_text_at(document, paragraph_ix, byte, text, styles); + restore_block_order(document, paragraph_ix, snapshot); + } + + fn delete_text_preserving_blocks(document: &mut Document, paragraph_ix: usize, range: Range) { + let Some(snapshot) = block_order_snapshot(document, paragraph_ix) else { + return; + }; + delete_range_in_paragraph(document, paragraph_ix, range); + restore_block_order(document, paragraph_ix, snapshot); + } + + fn split_paragraph_preserving_blocks(document: &mut Document, paragraph_ix: usize, byte: usize) -> Option { + let block_ix = block_ix_for_paragraph_ix(document, paragraph_ix)?; + let old_blocks = document.blocks.as_ref().clone(); + let old_block_ids = document.ids.block_ids.clone(); + + split_paragraph_at(document, paragraph_ix, byte); + + let new_paragraph_id = *document.ids.paragraph_ids.get(paragraph_ix + 1)?; + let new_block_id = document + .ids + .block_ids + .iter() + .copied() + .find(|id| !old_block_ids.contains(id))?; + let left = document.paragraphs.get(paragraph_ix).cloned()?; + let right = document.paragraphs.get(paragraph_ix + 1).cloned()?; + + let mut blocks = old_blocks; + *blocks.get_mut(block_ix)? = Block::Paragraph(left); + blocks.insert(block_ix + 1, Block::Paragraph(right)); + document.blocks = Arc::new(blocks); + + let mut block_ids = old_block_ids; + block_ids.insert(block_ix + 1, new_block_id); + document.ids.block_ids = block_ids; + gpui_flowtext::rebuild_document_sections(document); + + Some(new_paragraph_id) + } + + fn block_order_snapshot(document: &Document, paragraph_ix: usize) -> Option<(usize, Vec, Vec)> { + Some((block_ix_for_paragraph_ix(document, paragraph_ix)?, document.blocks.as_ref().clone(), document.ids.block_ids.clone())) + } + + fn restore_block_order(document: &mut Document, paragraph_ix: usize, (block_ix, mut blocks, block_ids): (usize, Vec, Vec)) { + if let Some(paragraph) = document.paragraphs.get(paragraph_ix).cloned() + && let Some(block) = blocks.get_mut(block_ix) + { + *block = Block::Paragraph(paragraph); + } + document.blocks = Arc::new(blocks); + document.ids.block_ids = block_ids; + gpui_flowtext::rebuild_document_sections(document); + } + + fn block_ix_for_paragraph_ix(document: &Document, target_paragraph_ix: usize) -> Option { + let mut paragraph_ix = 0; + for (block_ix, block) in document.blocks.iter().enumerate() { + if matches!(block, Block::Paragraph(_)) { + if paragraph_ix == target_paragraph_ix { + return Some(block_ix); + } + paragraph_ix += 1; + } + } + None + } + + fn text_choice(ix: u8) -> &'static str { + match ix % 6 { + 0 => "a", + 1 => "é", + 2 => "🌍", + 3 => "\u{2028}", + 4 => "x", + _ => MULTIBYTE, + } + } + + fn equation_source(ix: u8) -> &'static str { + match ix % 4 { + 0 => "x = aé", + 1 => "y = 🌍^2", + 2 => "z = a + \u{2028}x", + _ => MULTIBYTE, + } + } + + fn style_choice(ix: u8) -> RunStyles { + match ix % 5 { + 0 => RunStyles::default(), + 1 => RunStyles::default().with(RunStyle::Semantic(1)), + 2 => RunStyles::default().with(RunStyle::Highlight(2)), + 3 => RunStyles::default().with_direct_underline(), + _ => RunStyles::default().with(RunStyle::Semantic(3)).with_strikethrough(), + } + } + + fn byte_boundary(text: &str, pick: u8) -> usize { + let boundaries = byte_boundaries(text); + boundaries[usize::from(pick) % boundaries.len()] + } + + fn non_empty_range(text: &str, start_pick: u8, len_pick: u8) -> Option> { + let boundaries = byte_boundaries(text); + if boundaries.len() < 2 { + return None; + } + let start_ix = usize::from(start_pick) % (boundaries.len() - 1); + let remaining = boundaries.len() - start_ix - 1; + let end_ix = start_ix + 1 + usize::from(len_pick) % remaining; + Some(boundaries[start_ix]..boundaries[end_ix]) + } + + fn byte_boundaries(text: &str) -> Vec { + text + .char_indices() + .map(|(byte, _)| byte) + .chain(std::iter::once(text.len())) + .collect() + } + + fn pseudo(seed: u64, step: usize, source: usize, target: usize, copy: usize) -> u64 { + let mut value = seed + ^ ((step as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15)) + ^ ((source as u64) << 17) + ^ ((target as u64) << 33) + ^ ((copy as u64) << 49); + value ^= value >> 30; + value = value.wrapping_mul(0xBF58_476D_1CE4_E5B9); + value ^= value >> 27; + value.wrapping_mul(0x94D0_49BB_1331_11EB) ^ (value >> 31) + } + + fn canonical_document_bytes(document: &Document) -> Result> { + let mut bytes = Vec::new(); + let mut paragraph_ix = 0; + for block in document.blocks.iter() { + match block { + Block::Paragraph(paragraph) => { + bytes.push(b'p'); + push_paragraph_style(&mut bytes, paragraph.style); + for run in ¶graph.runs { + push_usize(&mut bytes, run.len); + push_run_styles(&mut bytes, run.styles); + } + let text = paragraph_text(document, paragraph_ix); + push_usize(&mut bytes, text.len()); + bytes.extend_from_slice(text.as_bytes()); + paragraph_ix += 1; }, - |run_styles| *run_styles = styles, - ); - self.apply(&[CanonicalOperation::SetRunStyles { paragraph, range, styles }]); + Block::Image(_) | Block::Equation(_) | Block::Table(_) => { + bytes.push(match block { + Block::Image(_) => b'i', + Block::Equation(_) => b'e', + Block::Table(_) => b't', + Block::Paragraph(_) => unreachable!(), + }); + let payload = schema::payload_from_block(block, &document.assets).context("object block should have a payload")?; + let payload = postcard::to_stdvec(&payload)?; + push_usize(&mut bytes, payload.len()); + bytes.extend_from_slice(&payload); + }, + } } + Ok(bytes) + } - fn apply(&mut self, ops: &[CanonicalOperation]) { - LocalApplier { - doc: &self.loro, - binding: &mut self.binding, + fn describe_document(document: &Document) -> String { + let mut output = String::new(); + let mut paragraph_ix = 0; + for (block_ix, block) in document.blocks.iter().enumerate() { + match block { + Block::Paragraph(paragraph) => { + let text = paragraph_text(document, paragraph_ix); + let _ = write!(output, "#{block_ix}:p:{text:?}:"); + for run in ¶graph.runs { + let _ = write!(output, "{}@{:?};", run.len, run.styles); + } + paragraph_ix += 1; + }, + Block::Image(_) => { + let _ = write!(output, "#{block_ix}:image;"); + }, + Block::Equation(_) => { + let _ = write!(output, "#{block_ix}:equation;"); + }, + Block::Table(_) => { + let _ = write!(output, "#{block_ix}:table;"); + }, } - .apply(&self.document, ops) - .expect("local apply should succeed"); } + output } - fn import_sequence<'a>(doc: &LoroDoc, updates: impl IntoIterator>) { - for update in updates { - doc.import(update).expect("update import should succeed"); + fn push_paragraph_style(bytes: &mut Vec, style: ParagraphStyle) { + match style { + ParagraphStyle::Normal => bytes.extend_from_slice(&[0, 0]), + ParagraphStyle::Custom(slot) => bytes.extend_from_slice(&[1, slot]), } } - fn assert_documents_eq(left: &Document, right: &Document) { - assert_eq!(left.blocks.len(), right.blocks.len()); - assert_eq!(left.paragraphs.len(), right.paragraphs.len()); - for ix in 0..left.paragraphs.len() { - assert_eq!(paragraph_text(left, ix), paragraph_text(right, ix)); - assert_eq!(left.paragraphs[ix].style, right.paragraphs[ix].style); - assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); + fn push_run_styles(bytes: &mut Vec, styles: RunStyles) { + match styles.semantic { + RunSemanticStyle::Plain => bytes.extend_from_slice(&[0, 0]), + RunSemanticStyle::Custom(slot) => bytes.extend_from_slice(&[1, slot]), + } + bytes.push(u8::from(styles.direct_underline)); + bytes.push(u8::from(styles.strikethrough)); + match styles.highlight { + Some(HighlightStyle::Custom(slot)) => bytes.extend_from_slice(&[1, slot]), + None => bytes.extend_from_slice(&[0, 0]), } } + + fn push_usize(bytes: &mut Vec, value: usize) { + bytes.extend_from_slice(&(value as u64).to_le_bytes()); + } } diff --git a/crates/flowstate-collab/tests/remote_apply.rs b/crates/flowstate-collab/tests/remote_apply.rs new file mode 100644 index 00000000..d9488674 --- /dev/null +++ b/crates/flowstate-collab/tests/remote_apply.rs @@ -0,0 +1,316 @@ +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + + use flowstate_collab::{ + SessionId, + binding::DocBinding, + local_apply::LocalApplier, + patch_apply::apply_patches, + projection, schema, + }; + use gpui_flowtext::{ + Block, CanonicalOperation, CollabPatch, Document, DocumentOffset, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, + InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, RunStyles, block_from_input_block, document_from_input_blocks, + insert_block_id, insert_text_at, mutate_runs_in_range, paragraph_text, paragraphs_mut, plain, remove_block_ids, update_paragraph_block, + }; + use loro::{ExportMode, LoroDoc, Subscription, event::Subscriber}; + + const MULTIBYTE: &str = "aé🌍\u{2028}x"; + + #[test] + fn remote_text_insert_produces_paragraph_text_patch() { + let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE)]); + let paragraph = source.document.ids.paragraph_ids[0]; + let styles = RunStyles::default().with(RunStyle::Semantic(3)); + + insert_text_at(&mut source.document, 0, "a".len(), "Zé", styles); + let updates = source.apply(&[CanonicalOperation::InsertText { + paragraph, + byte: "a".len(), + text: "Zé".to_string(), + styles, + }]); + let patches = target.import_updates(&updates); + + assert!(matches!(patches.as_slice(), [CollabPatch::ParagraphText { row: 0, .. }])); + assert_documents_match(&source.document, &target.document, &target.loro); + } + + #[test] + fn remote_run_and_paragraph_styles_produce_style_patches() { + let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE)]); + let paragraph = source.document.ids.paragraph_ids[0]; + + paragraphs_mut(&mut source.document)[0].style = ParagraphStyle::Custom(7); + update_paragraph_block(&mut source.document, 0); + let paragraph_style_updates = source.apply(&[CanonicalOperation::SetParagraphStyle { + paragraph, + style: ParagraphStyle::Custom(7), + }]); + let paragraph_style_patches = target.import_updates(¶graph_style_updates); + assert!(matches!(paragraph_style_patches.as_slice(), [CollabPatch::ParagraphStyle { row: 0, style: ParagraphStyle::Custom(7) }])); + + let range = "a".len().."aé🌍".len(); + let styles = RunStyles::default() + .with(RunStyle::Highlight(2)) + .with_direct_underline(); + mutate_runs_in_range( + &mut source.document, + DocumentOffset { paragraph: 0, byte: range.start }..DocumentOffset { paragraph: 0, byte: range.end }, + |run_styles| *run_styles = styles, + ); + let run_style_updates = source.apply(&[CanonicalOperation::SetRunStyles { paragraph, range, styles }]); + let run_style_patches = target.import_updates(&run_style_updates); + assert!(matches!(run_style_patches.as_slice(), [CollabPatch::ParagraphText { row: 0, .. }])); + assert_documents_match(&source.document, &target.document, &target.loro); + } + + #[test] + fn remote_object_data_rev_replacement_emits_one_patch() { + let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE), equation_block("x = 1")]); + replace_equation(&mut source.document, 1, "x = aé + 🌍", 1); + + let updates = source.apply(&[CanonicalOperation::ReplaceBlock { + block: Some(source.document.ids.block_ids[1]), + }]); + let patches = target.import_updates(&updates); + + assert!(matches!(patches.as_slice(), [CollabPatch::ReplaceObjectBlock { row: 1, .. }])); + assert_documents_match(&source.document, &target.document, &target.loro); + } + + #[test] + fn remote_list_insert_delete_and_move_emit_structural_patches() { + let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE), paragraph_block("tail")]); + + insert_object_block(&mut source.document, 1, equation_block("insert aé")); + let equation_id = source.document.ids.block_ids[1]; + let insert_updates = source.apply(&[CanonicalOperation::InsertBlock { + block: equation_id, + block_ix: 1, + }]); + let insert_patches = target.import_updates(&insert_updates); + assert!(matches!(insert_patches.as_slice(), [CollabPatch::InsertBlocks { row: 1, blocks } ] if blocks.len() == 1)); + assert_documents_match(&source.document, &target.document, &target.loro); + + move_block(&mut source.document, 1, 2); + let move_updates = source.apply(&[CanonicalOperation::MoveBlock { + block: equation_id, + new_block_ix: 2, + }]); + let move_patches = target.import_updates(&move_updates); + assert!(move_patches.iter().any(|patch| matches!(patch, CollabPatch::MoveBlock { from: 1, to: 2 }))); + assert_documents_match(&source.document, &target.document, &target.loro); + + delete_object_block(&mut source.document, 2); + let delete_updates = source.apply(&[CanonicalOperation::DeleteBlock { block: equation_id }]); + let delete_patches = target.import_updates(&delete_updates); + assert!(matches!(delete_patches.as_slice(), [CollabPatch::DeleteBlocks { row: 2, count: 1 }])); + assert_documents_match(&source.document, &target.document, &target.loro); + } + + struct SourcePeer { + document: Document, + loro: LoroDoc, + binding: DocBinding, + updates: Arc>>>, + _subscription: Subscription, + } + + struct TargetPeer { + document: Document, + loro: LoroDoc, + binding: DocBinding, + } + + impl SourcePeer { + fn apply(&mut self, ops: &[CanonicalOperation]) -> Vec> { + LocalApplier { + doc: &self.loro, + binding: &mut self.binding, + } + .apply(&self.document, ops) + .expect("local apply should succeed"); + let mut updates = self.updates.lock().expect("update lock should not be poisoned"); + std::mem::take(&mut *updates) + } + } + + impl TargetPeer { + fn import_updates(&mut self, updates: &[Vec]) -> Vec { + let mut all_patches = Vec::new(); + for update in updates { + all_patches.extend(self.import_update(update)); + } + all_patches + } + + fn import_update(&mut self, update: &[u8]) -> Vec { + let document = Arc::new(self.document.clone()); + let binding = Arc::new(Mutex::new(std::mem::take(&mut self.binding))); + let patches = Arc::new(Mutex::new(Vec::new())); + let error = Arc::new(Mutex::new(None::)); + let doc = self.loro.clone(); + let callback: Subscriber = Arc::new({ + let binding = binding.clone(); + let patches = patches.clone(); + let error = error.clone(); + move |event| { + let result = binding + .lock() + .map_err(|lock_error| anyhow::anyhow!("binding lock poisoned: {lock_error}")) + .and_then(|mut binding| { + flowstate_collab::remote_apply::RemoteApplier { + doc: &doc, + binding: &mut binding, + } + .apply_event(&document, &event) + }); + match result { + Ok(mut produced) => patches.lock().expect("patch lock should not be poisoned").append(&mut produced), + Err(apply_error) => *error.lock().expect("error lock should not be poisoned") = Some(format!("{apply_error:#}")), + } + } + }); + let subscription = self.loro.subscribe_root(callback); + let import_result = self.loro.import_with(update, "remote"); + drop(subscription); + self.binding = Arc::try_unwrap(binding) + .expect("binding callback should be dropped") + .into_inner() + .expect("binding lock should not be poisoned"); + import_result.expect("remote import should succeed"); + let remote_error = error.lock().expect("error lock should not be poisoned").take(); + if let Some(error) = remote_error { + panic!("remote apply failed: {error}"); + } + let patches = Arc::try_unwrap(patches) + .expect("patch callback should be dropped") + .into_inner() + .expect("patch lock should not be poisoned"); + apply_patches(&mut self.document, &mut self.binding, &self.loro, &patches).expect("patch apply should succeed"); + patches + } + } + + fn pair(blocks: Vec) -> (SourcePeer, TargetPeer) { + let session = SessionId::from_bytes([19; 32]); + let document = document_from_input_blocks(DocumentTheme::default(), blocks); + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, session, "remote-apply", &document).expect("populate should succeed"); + let binding = DocBinding::build(&loro, &document).expect("binding should build"); + let updates = Arc::new(Mutex::new(Vec::new())); + let captured = updates.clone(); + let subscription = loro.subscribe_local_update(Box::new(move |bytes| { + captured.lock().expect("update lock should not be poisoned").push(bytes.clone()); + true + })); + let snapshot = loro.export(ExportMode::Snapshot).expect("snapshot export should succeed"); + + let target_loro = schema::new_configured_doc(); + target_loro.import(&snapshot).expect("snapshot import should succeed"); + projection::verify_lineage(&target_loro, session).expect("lineage should match"); + let target_document = projection::document_from_loro(&target_loro, DocumentTheme::default()).expect("projection should succeed"); + let target_binding = DocBinding::build(&target_loro, &target_document).expect("binding should build"); + + ( + SourcePeer { + document, + loro, + binding, + updates, + _subscription: subscription, + }, + TargetPeer { + document: target_document, + loro: target_loro, + binding: target_binding, + }, + ) + } + + fn paragraph_block(text: &str) -> InputBlock { + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(text)], + }) + } + + fn equation_block(source: &str) -> InputBlock { + InputBlock::Equation(InputEquationBlock { + source: source.to_string(), + syntax: InputEquationSyntax::Latex, + display: InputEquationDisplay::Display, + }) + } + + fn insert_object_block(document: &mut Document, row: usize, block: InputBlock) { + Arc::make_mut(&mut document.blocks).insert(row, block_from_input_block(&block)); + insert_block_id(document, row); + } + + fn replace_equation(document: &mut Document, row: usize, source: &str, version: u64) { + let mut block = block_from_input_block(&equation_block(source)); + if let Block::Equation(equation) = &mut block { + equation.version = version; + } + Arc::make_mut(&mut document.blocks)[row] = block; + } + + fn move_block(document: &mut Document, from: usize, to: usize) { + let block = Arc::make_mut(&mut document.blocks).remove(from); + let insert_ix = to.min(document.blocks.len()); + Arc::make_mut(&mut document.blocks).insert(insert_ix, block); + let block_id = document.ids.block_ids.remove(from); + document.ids.block_ids.insert(to.min(document.ids.block_ids.len()), block_id); + } + + fn delete_object_block(document: &mut Document, row: usize) { + Arc::make_mut(&mut document.blocks).remove(row); + remove_block_ids(document, row..row + 1); + } + + fn assert_documents_match(expected: &Document, actual: &Document, loro: &LoroDoc) { + let projected = projection::document_from_loro(loro, DocumentTheme::default()).expect("projection should succeed"); + assert_document_shape_eq(expected, actual); + assert_document_shape_eq(expected, &projected); + } + + fn assert_document_shape_eq(left: &Document, right: &Document) { + assert_eq!(left.blocks.len(), right.blocks.len()); + assert_eq!(left.paragraphs.len(), right.paragraphs.len()); + let mut paragraph_ix = 0; + for (left_block, right_block) in left.blocks.iter().zip(right.blocks.iter()) { + match (left_block, right_block) { + (Block::Paragraph(left_paragraph), Block::Paragraph(right_paragraph)) => { + assert_eq!(paragraph_text(left, paragraph_ix), paragraph_text(right, paragraph_ix)); + assert_eq!(left_paragraph.style, right_paragraph.style); + assert_eq!(left_paragraph.runs, right_paragraph.runs); + }, + (Block::Equation(left), Block::Equation(right)) => { + assert_eq!(left.source, right.source); + assert_eq!(left.syntax, right.syntax); + assert_eq!(left.display, right.display); + }, + (Block::Image(left), Block::Image(right)) => { + assert_eq!(left.asset_id, right.asset_id); + assert_eq!(left.alt_text, right.alt_text); + assert_eq!(left.caption, right.caption); + assert_eq!(left.sizing, right.sizing); + assert_eq!(left.alignment, right.alignment); + }, + (Block::Table(left), Block::Table(right)) => { + assert_eq!(left.rows, right.rows); + assert_eq!(left.column_widths, right.column_widths); + assert_eq!(left.style, right.style); + }, + _ => panic!("document block kind changed"), + } + if matches!(left_block, Block::Paragraph(_)) { + paragraph_ix += 1; + } + } + } + +} diff --git a/crates/flowstate-collab/tests/swarm_loopback.rs b/crates/flowstate-collab/tests/swarm_loopback.rs new file mode 100644 index 00000000..439bbfb0 --- /dev/null +++ b/crates/flowstate-collab/tests/swarm_loopback.rs @@ -0,0 +1,260 @@ +#![cfg(test)] + +use std::time::Duration; + +use anyhow::{Context as _, Result, bail, ensure}; +use async_channel::Receiver; +use flowstate_collab::{ + DIRECT_ALPN, SessionId, + net::{ + NetEvent, PublishPayload, + direct::{self, DirectProto, DirectServeRequest, DirectServeState, DirectSessionHandler}, + swarm::SwarmHandle, + }, + proto_direct::DirectRequest, + proto_gossip::{GOSSIP_INLINE_LIMIT, GossipMsg}, +}; +use iroh::{Endpoint, EndpointAddr, Watcher as _, address_lookup::memory::MemoryLookup, endpoint::presets, protocol::Router}; +use iroh_gossip::net::Gossip; +use tokio::time::timeout; + +struct Peer { + endpoint: Endpoint, + gossip: Gossip, + direct_state: DirectServeState, + router: Router, + events: Receiver, + swarm: Option, +} + +impl Peer { + async fn spawn(lookup: MemoryLookup) -> Result { + let endpoint = Endpoint::builder(presets::Minimal) + .address_lookup(lookup) + .clear_ip_transports() + .bind_addr("127.0.0.1:0")? + .bind() + .await + .context("binding loopback endpoint failed")?; + let gossip = Gossip::builder().spawn(endpoint.clone()); + let direct_state = DirectServeState::default(); + let router = Router::builder(endpoint.clone()) + .accept(iroh_gossip::ALPN, gossip.clone()) + .accept(DIRECT_ALPN, DirectProto::new(direct_state.clone())) + .spawn(); + let (_, events) = async_channel::unbounded(); + + Ok(Self { + endpoint, + gossip, + direct_state, + router, + events, + swarm: None, + }) + } + + fn start_swarm(&mut self, session: SessionId, bootstrap: Vec) -> Result<()> { + let (evt_tx, evt_rx) = async_channel::unbounded(); + self.events = evt_rx; + self.swarm = Some(SwarmHandle::spawn( + self.endpoint.clone(), + self.gossip.clone(), + self.direct_state.clone(), + session, + bootstrap, + evt_tx, + )?); + Ok(()) + } + + async fn publish(&self, payload: PublishPayload) -> Result<()> { + let Some(swarm) = &self.swarm else { + bail!("peer swarm is not running"); + }; + swarm.publish(payload).await + } + + async fn stop_swarm(&mut self) { + if let Some(swarm) = self.swarm.take() { + swarm.stop().await; + } + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires local UDP loopback and real iroh gossip timing"] +async fn swarm_loopback_net_subset() -> Result<()> { + let session = SessionId::new(); + let lookup = MemoryLookup::new(); + let mut a = Peer::spawn(lookup.clone()).await?; + let mut b = Peer::spawn(lookup.clone()).await?; + let mut c = Peer::spawn(lookup.clone()).await?; + + let a_addr = wait_addr(&a.endpoint).await; + let b_addr = wait_addr(&b.endpoint).await; + let c_addr = wait_addr(&c.endpoint).await; + ensure!(!a_addr.is_empty(), "A endpoint did not expose loopback addresses"); + ensure!(!b_addr.is_empty(), "B endpoint did not expose loopback addresses"); + ensure!(!c_addr.is_empty(), "C endpoint did not expose loopback addresses"); + let addrs = [a_addr.clone(), b_addr.clone(), c_addr.clone()]; + for addr in addrs.iter().cloned() { + lookup.add_endpoint_info(addr); + } + for endpoint in [&a.endpoint, &b.endpoint, &c.endpoint] { + endpoint + .address_lookup()? + .add(MemoryLookup::from_endpoint_info(addrs.iter().cloned())); + } + + install_snapshot_handler(&a, session, b"snapshot-from-a".to_vec()).await; + let snapshot = direct::pull_with_endpoint( + &b.endpoint, + DirectRequest::Snapshot { session }, + vec![a.endpoint.id()], + Duration::from_secs(5), + ) + .await?; + assert_eq!(snapshot, b"snapshot-from-a".to_vec()); + + a.start_swarm(session, Vec::new())?; + b.start_swarm(session, vec![a_addr.clone()])?; + c.start_swarm(session, vec![b_addr.clone()])?; + recv_neighbor(&a.events).await?; + recv_neighbor(&b.events).await?; + recv_neighbor(&c.events).await?; + + let small_update = b"a-to-b-to-c".to_vec(); + a.publish(PublishPayload::Update(small_update.clone())).await?; + recv_update(&b.events, &small_update).await?; + recv_update(&c.events, &small_update).await?; + + a.publish(PublishPayload::Presence(b"presence-a".to_vec())).await?; + b.publish(PublishPayload::Presence(b"presence-b".to_vec())).await?; + c.publish(PublishPayload::Presence(b"presence-c".to_vec())).await?; + recv_presence(&a.events).await?; + recv_presence(&b.events).await?; + recv_presence(&c.events).await?; + + let large_update = vec![7; GOSSIP_INLINE_LIMIT + 1]; + a.publish(PublishPayload::Update(large_update.clone())).await?; + let blob = recv_update_available(&b.events, large_update.len() as u64).await?; + let blob_bytes = direct::pull_with_endpoint( + &b.endpoint, + DirectRequest::Blob { session, blob }, + vec![a.endpoint.id()], + Duration::from_secs(5), + ) + .await?; + assert_eq!(blob_bytes, large_update); + + a.stop_swarm().await; + let b_update = b"b-after-a-left".to_vec(); + b.publish(PublishPayload::Update(b_update.clone())).await?; + recv_update(&c.events, &b_update).await?; + let c_update = b"c-after-a-left".to_vec(); + c.publish(PublishPayload::Update(c_update.clone())).await?; + recv_update(&b.events, &c_update).await?; + + c.stop_swarm().await; + c.start_swarm(session, vec![b_addr])?; + b.publish(PublishPayload::Digest { vv: b"digest-after-resubscribe".to_vec() }).await?; + recv_digest(&c.events).await?; + + a.router.shutdown().await.context("shutting down A router failed")?; + b.router.shutdown().await.context("shutting down B router failed")?; + c.router.shutdown().await.context("shutting down C router failed")?; + Ok(()) +} + +async fn install_snapshot_handler(peer: &Peer, session: SessionId, snapshot: Vec) { + let (request_tx, request_rx) = async_channel::unbounded(); + peer + .direct_state + .register_handler(session, DirectSessionHandler::new(request_tx)) + .await; + tokio::spawn(async move { + while let Ok(request) = request_rx.recv().await { + if let DirectServeRequest::Snapshot { reply } = request { + let _ = reply.send(Ok(snapshot.clone())).await; + } + } + }); +} + +async fn wait_addr(endpoint: &Endpoint) -> EndpointAddr { + let mut watcher = endpoint.watch_addr(); + let wait = async { + loop { + let addr = watcher.get(); + if !addr.is_empty() { + return addr; + } + if watcher.updated().await.is_err() { + return endpoint.addr(); + } + } + }; + timeout(Duration::from_secs(3), wait).await.unwrap_or_else(|_| endpoint.addr()) +} + +async fn recv_update(events: &Receiver, expected: &[u8]) -> Result<()> { + recv_matching(events, "update", |event| { + matches!(event, NetEvent::Gossip { msg: GossipMsg::Update(bytes), .. } if bytes == expected) + }) + .await + .map(|_| ()) +} + +async fn recv_presence(events: &Receiver) -> Result<()> { + recv_matching(events, "presence", |event| { + matches!(event, NetEvent::Gossip { msg: GossipMsg::Presence(_), .. }) + }) + .await + .map(|_| ()) +} + +async fn recv_neighbor(events: &Receiver) -> Result<()> { + recv_matching(events, "neighbor", |event| matches!(event, NetEvent::NeighborUp { .. })) + .await + .map(|_| ()) +} + +async fn recv_update_available(events: &Receiver, expected_len: u64) -> Result { + let event = recv_matching(events, "update-available", |event| { + matches!(event, NetEvent::Gossip { msg: GossipMsg::UpdateAvailable { len, .. }, .. } if *len == expected_len) + }) + .await?; + let NetEvent::Gossip { + msg: GossipMsg::UpdateAvailable { blob, .. }, + .. + } = event + else { + bail!("matched event was not update-available"); + }; + Ok(blob) +} + +async fn recv_digest(events: &Receiver) -> Result<()> { + recv_matching(events, "digest", |event| { + matches!(event, NetEvent::Gossip { msg: GossipMsg::Digest { .. }, .. }) + }) + .await + .map(|_| ()) +} + +async fn recv_matching(events: &Receiver, label: &str, matches: F) -> Result +where + F: Fn(&NetEvent) -> bool, +{ + timeout(Duration::from_secs(10), async { + loop { + let event = events.recv().await.context("event channel closed")?; + if matches(&event) { + return Ok(event); + } + } + }) + .await + .with_context(|| format!("timed out waiting for {label}"))? +} diff --git a/crates/flowstate-collab/tests/translation.rs b/crates/flowstate-collab/tests/translation.rs index 86afb991..acd7222f 100644 --- a/crates/flowstate-collab/tests/translation.rs +++ b/crates/flowstate-collab/tests/translation.rs @@ -213,6 +213,14 @@ mod tests { apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::DeleteBlock { block: equation_id }]); } + #[test] + fn replace_block_none_updates_single_changed_object_row() { + let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain(MULTIBYTE)]), equation_block("x = 1")]); + + replace_object_block_with_version(&mut document, 1, equation_block("x = aé + 🌍"), 1); + apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::ReplaceBlock { block: None }]); + } + #[test] fn replace_document_rebuilds_the_loro_projection() { let (_, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("old")]), equation_block("old = 1")]); @@ -279,6 +287,17 @@ mod tests { Arc::make_mut(&mut document.blocks)[block_ix] = block_from_input_block(&input); } + fn replace_object_block_with_version(document: &mut Document, block_ix: usize, input: InputBlock, version: u64) { + let mut block = block_from_input_block(&input); + match &mut block { + Block::Paragraph(_) => {}, + Block::Image(image) => image.version = version, + Block::Equation(equation) => equation.version = version, + Block::Table(table) => table.version = version, + } + Arc::make_mut(&mut document.blocks)[block_ix] = block; + } + fn delete_object_block(document: &mut Document, block_ix: usize) { Arc::make_mut(&mut document.blocks).remove(block_ix); remove_block_ids(document, block_ix..block_ix + 1); diff --git a/crates/flowstate/Cargo.toml b/crates/flowstate/Cargo.toml index a06fe97f..5d6ea6e8 100644 --- a/crates/flowstate/Cargo.toml +++ b/crates/flowstate/Cargo.toml @@ -32,6 +32,7 @@ serde_json = { workspace = true } tempfile = { workspace = true } toml = { workspace = true } uuid = { workspace = true } +tracing = "0.1.44" [features] hotpath = [ diff --git a/crates/flowstate/src/collab/asset_transfer.rs b/crates/flowstate/src/collab/asset_transfer.rs index 24502cca..0435cf2b 100644 --- a/crates/flowstate/src/collab/asset_transfer.rs +++ b/crates/flowstate/src/collab/asset_transfer.rs @@ -1,8 +1,4 @@ -use std::{ - collections::hash_map::DefaultHasher, - hash::{Hash, Hasher}, - sync::Arc, -}; +use std::sync::Arc; use anyhow::{Context as _, Result, anyhow, ensure}; use flowstate_collab::{ @@ -31,7 +27,7 @@ pub(super) fn schedule_missing_assets( let assets = match image_assets_in_loro(doc) { Ok(assets) => assets, Err(error) => { - eprintln!("flowstate collab asset scan failed: {error:#}"); + tracing::warn!("flowstate collab asset scan failed: {error:#}"); return; }, }; @@ -113,10 +109,10 @@ fn start_asset_pull( Ok(record) => { session.apply_or_queue_patches(vec![CollabPatch::AssetArrived { id, record }], cx); }, - Err(error) => eprintln!("flowstate collab rejected fetched asset {id:?}: {error:#}"), + Err(error) => tracing::warn!("flowstate collab rejected fetched asset {id:?}: {error:#}"), }, - Ok(Err(error)) => eprintln!("flowstate collab asset pull failed for {id:?}: {error:#}"), - Err(error) => eprintln!("flowstate collab asset pull channel closed for {id:?}: {error}"), + Ok(Err(error)) => tracing::warn!("flowstate collab asset pull failed for {id:?}: {error:#}"), + Err(error) => tracing::warn!("flowstate collab asset pull channel closed for {id:?}: {error}"), } }); }) @@ -145,9 +141,7 @@ impl ImageAssetMeta { fn record_from_bytes(&self, bytes: Vec) -> Result { ensure!(bytes.len() as u64 == self.byte_len, "asset byte length mismatch"); - let mut hasher = DefaultHasher::new(); - bytes.hash(&mut hasher); - let content_hash = hasher.finish(); + let content_hash = AssetRecord::stable_content_hash(&bytes); ensure!(content_hash == self.content_hash, "asset content hash mismatch"); Ok(AssetRecord { id: AssetId(self.asset_id), diff --git a/crates/flowstate/src/collab/manager.rs b/crates/flowstate/src/collab/manager.rs new file mode 100644 index 00000000..7280233c --- /dev/null +++ b/crates/flowstate/src/collab/manager.rs @@ -0,0 +1,343 @@ +use std::collections::HashMap; + +use anyhow::{Context as _, Result, anyhow, ensure}; +use async_channel::Receiver; +use flowstate_collab::{ + SessionId, + ids::PeerId, + net::{ + NetCommand, TicketSeed, + runtime::{self, CommandSender}, + }, + ticket::SessionTicket, +}; +use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global}; +use uuid::Uuid; + +use crate::rich_text_element::RichTextEditor; + +use super::session::{CollabSession, DetachReason, JoinedDocument, SessionPhase, SessionRosterEntry}; + +pub struct JoinRequest { + pub session: SessionId, + pub completed: Receiver>, +} + +#[derive(Default)] +pub struct CollabManager { + runtime: Option, + pub(super) sessions_by_id: HashMap>, + session_by_panel: HashMap, + own_endpoint_id: Option, + pub(super) endpoint_online: bool, + pub(super) event_pump_started: bool, +} + +impl Global for CollabManager {} + +#[derive(Clone)] +struct CollabRuntime { + commands: CommandSender, +} + +impl CollabManager { + pub fn init(cx: &mut C) + where + C: BorrowAppContext, + { + cx.update_default_global::(|_, _| {}); + } + + pub(super) fn shutdown_runtime(&mut self) { + if let Some(runtime) = self.runtime.take() { + let _ = runtime.commands.try_send(NetCommand::Shutdown); + } + self.sessions_by_id.clear(); + self.session_by_panel.clear(); + self.own_endpoint_id = None; + self.event_pump_started = false; + self.endpoint_online = false; + } + + pub fn start_session_for_panel( + &mut self, + panel_id: Uuid, + editor: Entity, + title: String, + cx: &mut Context, + ) -> Result + where + T: 'static, + { + if let Some(session) = self.session_by_panel.get(&panel_id).copied() { + return Ok(session); + } + + let commands = self.ensure_runtime(cx)?; + let session = SessionId::new(); + let document = editor.read(cx).document().clone(); + let collab = CollabSession::from_local_document(session, panel_id, editor, title, document, commands.clone())?; + let direct_handler = collab.direct_handler(); + let entity = cx.new(|_| collab); + entity.update(cx, |session, cx| session.attach(cx)); + + self.register_session(entity.clone(), cx); + if let Err(error) = commands.try_send(NetCommand::RegisterDirectHandler { + session, + handler: direct_handler, + }) { + Self::detach_entity(&entity, DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), cx); + self.unregister_session(session); + return Err(error).context("registering collaboration direct handler failed"); + } + + let (reply_tx, reply_rx) = async_channel::bounded(1); + if let Err(error) = commands.try_send(NetCommand::CreateSession { session, reply: reply_tx }) { + Self::detach_entity(&entity, DetachReason::Fatal(format!("creating collaboration network session failed: {error}")), cx); + self.unregister_session(session); + return Err(error).context("creating collaboration network session failed"); + } + Self::finish_create_session(entity, reply_rx, cx); + Ok(session) + } + + pub fn join_session(&mut self, ticket: SessionTicket, cx: &mut Context) -> Result + where + T: 'static, + { + ensure!(ticket.is_supported_version(), "unsupported collaboration protocol version"); + let commands = self.ensure_runtime(cx)?; + commands + .try_send(NetCommand::EnsureUp) + .context("starting collaboration networking failed")?; + ensure!( + self.own_endpoint_id != Some(ticket.inviter.id), + "That's your own invite — open the Share dialog instead." + ); + ensure!( + !self.sessions_by_id.contains_key(&ticket.session), + "collaboration session is already open" + ); + + let session = ticket.session; + let inviter = ticket.inviter.id; + let collab = CollabSession::joining(session, ticket.title.clone(), commands.clone(), vec![ticket.inviter.clone()]); + let entity = cx.new(|_| collab); + self.register_session(entity.clone(), cx); + let neighbor_rx = entity.update(cx, |session, cx| session.prepare_join_neighbor_wait(cx)); + + if let Err(error) = commands.try_send(NetCommand::JoinSession { + session, + bootstrap: vec![ticket.inviter], + }) { + entity.update(cx, |session, cx| { + session.detach( + DetachReason::JoinFailed(format!("joining collaboration network session failed: {error}")), + cx, + ); + }); + self.unregister_session(session); + return Err(error).context("joining collaboration network session failed"); + } + let completed = entity.update(cx, |session, cx| session.begin_join_bootstrap(inviter, neighbor_rx, cx)); + Ok(JoinRequest { session, completed }) + } + + pub fn attach_joined_session( + &mut self, + session_id: SessionId, + panel_id: Uuid, + editor: Entity, + cx: &mut Context, + ) -> Result<()> + where + T: 'static, + { + let commands = self.ensure_runtime(cx)?; + let session = self + .sessions_by_id + .get(&session_id) + .cloned() + .ok_or_else(|| anyhow!("collaboration session is not registered"))?; + let direct_handler = session.read(cx).direct_handler(); + if let Err(error) = commands.try_send(NetCommand::RegisterDirectHandler { + session: session_id, + handler: direct_handler, + }) { + Self::detach_entity(&session, DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), cx); + self.unregister_session(session_id); + return Err(error).context("registering collaboration direct handler failed"); + } + if let Err(error) = Self::establish_joined_peer(session.clone(), commands, cx) { + Self::detach_entity(&session, DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error:#}")), cx); + self.unregister_session(session_id); + return Err(error); + } + if let Err(error) = session.update(cx, |session, cx| session.attach_joined_editor(panel_id, editor, cx)) { + Self::detach_entity(&session, DetachReason::Fatal(format!("attaching joined collaboration editor failed: {error:#}")), cx); + self.unregister_session(session_id); + return Err(error); + } + self.session_by_panel.insert(panel_id, session_id); + Ok(()) + } + + pub fn leave_session_for_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> bool + where + T: 'static, + { + let Some(session_id) = self.session_by_panel.get(&panel_id).copied() else { + return false; + }; + let Some(session) = self.sessions_by_id.get(&session_id).cloned() else { + self.unregister_session(session_id); + return false; + }; + + let _ = session.update(cx, |session, cx| session.detach(DetachReason::UserLeft, cx)); + self.unregister_session(session_id); + true + } + + pub fn request_ticket_for_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> Option>> + where + T: 'static, + { + let session_id = self.session_by_panel.get(&panel_id).copied()?; + let session = self.sessions_by_id.get(&session_id)?.clone(); + let title = session.read(cx).title().to_string(); + let commands = self.ensure_runtime(cx).ok()?; + let (ticket_tx, ticket_rx) = async_channel::bounded(1); + let (reply_tx, reply_rx) = async_channel::bounded(1); + commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }).ok()?; + + cx.spawn(async move |_, cx| { + let ticket = match reply_rx.recv().await { + Ok(addr) => { + let own_endpoint_id = addr.id; + let _ = cx.update_global::(|manager, _| manager.own_endpoint_id = Some(own_endpoint_id)); + Ok(SessionTicket::new(session_id, addr, title)) + }, + Err(error) => Err(anyhow!("collaboration endpoint address unavailable: {error}")), + }; + let _ = ticket_tx.send(ticket).await; + }) + .detach(); + Some(ticket_rx) + } + + pub fn session_for_panel(&self, panel_id: Uuid) -> Option> { + self + .session_by_panel + .get(&panel_id) + .and_then(|session| self.sessions_by_id.get(session)) + .cloned() + } + + pub fn phase_for_panel(&self, panel_id: Uuid, cx: &App) -> Option { + self + .session_for_panel(panel_id) + .map(|session| session.read(cx).phase().clone()) + } + + pub fn phase_for_session(&self, session_id: SessionId, cx: &App) -> Option { + self + .sessions_by_id + .get(&session_id) + .map(|session| session.read(cx).phase().clone()) + } + + pub fn session_for_id(&self, session_id: SessionId) -> Option> { + self.sessions_by_id.get(&session_id).cloned() + } + + pub fn roster_for_panel(&self, panel_id: Uuid, cx: &App) -> Vec { + self + .session_for_panel(panel_id) + .map_or_else(Vec::new, |session| session.read(cx).roster()) + } + + fn register_session(&mut self, session: Entity, cx: &mut Context) { + let id = session.read(cx).session_id(); + if let Some(panel_id) = session.read(cx).panel_id() { + self.session_by_panel.insert(panel_id, id); + } + self.sessions_by_id.insert(id, session); + } + + pub(super) fn unregister_session(&mut self, session: SessionId) { + self.sessions_by_id.remove(&session); + self.session_by_panel.retain(|_, active| *active != session); + } + + fn ensure_runtime(&mut self, cx: &mut Context) -> Result + where + T: 'static, + { + if let Some(runtime) = &self.runtime { + return Ok(runtime.commands.clone()); + } + + let (commands, events) = runtime::start()?; + self.runtime = Some(CollabRuntime { commands: commands.clone() }); + self.start_event_pump(events, cx); + Ok(commands) + } + + fn finish_create_session(session: Entity, reply_rx: Receiver>, cx: &mut Context) + where + T: 'static, + { + cx.spawn(async move |_, cx| match reply_rx.recv().await { + Ok(Ok(seed)) => { + let own_endpoint_id = seed.inviter.id; + let _ = cx.update_global::(|manager, _| manager.own_endpoint_id = Some(own_endpoint_id)); + let _ = session.update(cx, |session, cx| session.establish_local_peer(&seed.inviter.id, cx)); + }, + Ok(Err(error)) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error:#}")), cx); + }); + }, + Err(error) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error}")), cx); + }); + }, + }) + .detach(); + } + + fn establish_joined_peer(session: Entity, commands: CommandSender, cx: &mut Context) -> Result<()> + where + T: 'static, + { + let (reply_tx, reply_rx) = async_channel::bounded(1); + commands + .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) + .context("requesting collaboration endpoint address failed")?; + cx.spawn(async move |_, cx| match reply_rx.recv().await { + Ok(addr) => { + let own_endpoint_id = addr.id; + let _ = cx.update_global::(|manager, _| manager.own_endpoint_id = Some(own_endpoint_id)); + let _ = session.update(cx, |session, cx| session.establish_local_peer(&addr.id, cx)); + }, + Err(error) => { + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error}")), cx); + }); + }, + }) + .detach(); + Ok(()) + } + + fn detach_entity(entity: &Entity, reason: DetachReason, cx: &mut Context) + where + T: 'static, + { + entity.update(cx, |session, cx| { + session.detach(reason, cx); + }); + } +} diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index 52dffdea..3ef3cf06 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -1,359 +1,26 @@ +mod manager; +mod pump; mod presence_view; +pub(crate) mod notify; mod session; pub mod share_dialog; mod shutdown; pub mod status; -use std::collections::HashMap; - -use anyhow::{Context as _, Result, anyhow, ensure}; +use anyhow::Result; use async_channel::Receiver; -use flowstate_collab::{ - SessionId, - net::{ - NetCommand, NetEvent, TicketSeed, - runtime::{self, CommandSender}, - }, - ticket::SessionTicket, -}; -use gpui::{App, AppContext, BorrowAppContext, Context, Entity, Global, ReadGlobal}; +use flowstate_collab::{SessionId, ticket::SessionTicket}; +use gpui::{App, BorrowAppContext, Context, Entity, ReadGlobal}; use uuid::Uuid; use crate::rich_text_element::RichTextEditor; -pub use session::{Attachment, CollabSession, Connectivity, DetachReason, JoinStage, JoinedDocument, SessionPhase, SessionRosterEntry}; +pub use manager::{CollabManager, JoinRequest}; +pub use session::{ + Attachment, CollabSession, Connectivity, DetachReason, JoinStage, JoinedDocument, SessionNotice, SessionPhase, SessionRosterEntry, +}; pub use shutdown::shutdown; -pub struct JoinRequest { - pub session: SessionId, - pub completed: Receiver>, -} - -#[derive(Default)] -pub struct CollabManager { - runtime: Option, - sessions_by_id: HashMap>, - session_by_panel: HashMap, - endpoint_online: bool, - event_pump_started: bool, -} - -impl Global for CollabManager {} - -#[derive(Clone)] -struct CollabRuntime { - commands: CommandSender, -} - -impl CollabManager { - pub fn init(cx: &mut C) - where - C: BorrowAppContext, - { - cx.update_default_global::(|_, _| {}); - } - - pub fn start_session_for_panel( - &mut self, - panel_id: Uuid, - editor: Entity, - title: String, - cx: &mut Context, - ) -> Result - where - T: 'static, - { - if let Some(session) = self.session_by_panel.get(&panel_id).copied() { - return Ok(session); - } - - let commands = self.ensure_runtime(cx)?; - let session = SessionId::new(); - let document = editor.read(cx).document().clone(); - let collab = CollabSession::from_local_document(session, panel_id, editor, title, document, commands.clone())?; - let direct_handler = collab.direct_handler(); - let entity = cx.new(|_| collab); - entity.update(cx, |session, cx| session.attach(cx)); - - self.register_session(entity.clone(), cx); - commands - .try_send(NetCommand::RegisterDirectHandler { - session, - handler: direct_handler, - }) - .context("registering collaboration direct handler failed")?; - - let (reply_tx, reply_rx) = async_channel::bounded(1); - commands - .try_send(NetCommand::CreateSession { session, reply: reply_tx }) - .context("creating collaboration network session failed")?; - Self::finish_create_session(entity, reply_rx, cx); - Ok(session) - } - - pub fn join_session(&mut self, ticket: SessionTicket, cx: &mut Context) -> Result - where - T: 'static, - { - ensure!(ticket.is_supported_version(), "unsupported collaboration protocol version"); - ensure!( - !self.sessions_by_id.contains_key(&ticket.session), - "collaboration session is already open" - ); - - let commands = self.ensure_runtime(cx)?; - let session = ticket.session; - let inviter = ticket.inviter.id; - let collab = CollabSession::joining(session, ticket.title.clone(), commands.clone(), vec![ticket.inviter.clone()]); - let entity = cx.new(|_| collab); - self.register_session(entity.clone(), cx); - - if let Err(error) = commands.try_send(NetCommand::JoinSession { - session, - bootstrap: vec![ticket.inviter], - }) { - entity.update(cx, |session, cx| { - session.detach( - DetachReason::JoinFailed(format!("joining collaboration network session failed: {error}")), - cx, - ); - }); - self.unregister_session(session); - return Err(error).context("joining collaboration network session failed"); - } - let completed = entity.update(cx, |session, cx| { - session.set_join_stage(JoinStage::Subscribing, cx); - session.begin_join_bootstrap(inviter, cx) - }); - Ok(JoinRequest { session, completed }) - } - - pub fn attach_joined_session( - &mut self, - session_id: SessionId, - panel_id: Uuid, - editor: Entity, - cx: &mut Context, - ) -> Result<()> - where - T: 'static, - { - let commands = self.ensure_runtime(cx)?; - let session = self - .sessions_by_id - .get(&session_id) - .cloned() - .ok_or_else(|| anyhow!("collaboration session is not registered"))?; - let direct_handler = session.read(cx).direct_handler(); - session.update(cx, |session, cx| session.attach_joined_editor(panel_id, editor, cx))?; - self.session_by_panel.insert(panel_id, session_id); - commands - .try_send(NetCommand::RegisterDirectHandler { - session: session_id, - handler: direct_handler, - }) - .context("registering collaboration direct handler failed")?; - Self::establish_joined_peer(session, commands, cx)?; - Ok(()) - } - - pub fn leave_session_for_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> bool - where - T: 'static, - { - let Some(session_id) = self.session_by_panel.get(&panel_id).copied() else { - return false; - }; - let Some(session) = self.sessions_by_id.get(&session_id).cloned() else { - self.unregister_session(session_id); - return false; - }; - - let _ = session.update(cx, |session, cx| session.detach(DetachReason::UserLeft, cx)); - self.unregister_session(session_id); - true - } - - pub fn request_ticket_for_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> Option>> - where - T: 'static, - { - let session_id = self.session_by_panel.get(&panel_id).copied()?; - let session = self.sessions_by_id.get(&session_id)?.clone(); - let title = session.read(cx).title().to_string(); - let commands = self.ensure_runtime(cx).ok()?; - let (ticket_tx, ticket_rx) = async_channel::bounded(1); - let (reply_tx, reply_rx) = async_channel::bounded(1); - commands - .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) - .ok()?; - - cx.spawn(async move |_, _| { - let ticket = match reply_rx.recv().await { - Ok(addr) => Ok(SessionTicket::new(session_id, addr, title)), - Err(error) => Err(anyhow!("collaboration endpoint address unavailable: {error}")), - }; - let _ = ticket_tx.send(ticket).await; - }) - .detach(); - Some(ticket_rx) - } - - pub fn session_for_panel(&self, panel_id: Uuid) -> Option> { - self - .session_by_panel - .get(&panel_id) - .and_then(|session| self.sessions_by_id.get(session)) - .cloned() - } - - pub fn phase_for_panel(&self, panel_id: Uuid, cx: &App) -> Option { - self - .session_for_panel(panel_id) - .map(|session| session.read(cx).phase().clone()) - } - - pub fn roster_for_panel(&self, panel_id: Uuid, cx: &App) -> Vec { - self - .session_for_panel(panel_id) - .map_or_else(Vec::new, |session| session.read(cx).roster()) - } - - fn register_session(&mut self, session: Entity, cx: &mut Context) { - let id = session.read(cx).session_id(); - if let Some(panel_id) = session.read(cx).panel_id() { - self.session_by_panel.insert(panel_id, id); - } - self.sessions_by_id.insert(id, session); - } - - fn unregister_session(&mut self, session: SessionId) { - self.sessions_by_id.remove(&session); - self.session_by_panel.retain(|_, active| *active != session); - } - - fn ensure_runtime(&mut self, cx: &mut Context) -> Result - where - T: 'static, - { - if let Some(runtime) = &self.runtime { - return Ok(runtime.commands.clone()); - } - - let (commands, events) = runtime::start()?; - self.runtime = Some(CollabRuntime { commands: commands.clone() }); - self.start_event_pump(events, cx); - Ok(commands) - } - - fn start_event_pump(&mut self, events: runtime::EventReceiver, cx: &mut Context) - where - T: 'static, - { - if self.event_pump_started { - return; - } - self.event_pump_started = true; - cx.spawn(async move |_, cx| { - while let Ok(event) = events.recv().await { - if cx - .update_global::(|manager, cx| manager.handle_net_event(event, cx)) - .is_err() - { - break; - } - } - }) - .detach(); - } - - fn handle_net_event(&mut self, event: NetEvent, cx: &mut App) { - match event { - NetEvent::EndpointOnline(online) => { - self.endpoint_online = online; - for session in self.sessions_by_id.values() { - session.update(cx, |session, cx| session.set_endpoint_online(online, cx)); - } - }, - NetEvent::Gossip { session, from, msg } => { - self.update_session(session, cx, |session, cx| session.handle_gossip(from, msg, cx)); - }, - NetEvent::NeighborUp { session, peer } => { - self.update_session(session, cx, |session, cx| session.neighbor_up(peer, cx)); - }, - NetEvent::NeighborDown { session, peer } => { - self.update_session(session, cx, |session, cx| session.neighbor_down(peer, cx)); - }, - NetEvent::GossipLagged { session } => { - self.update_session(session, cx, |session, cx| session.handle_gossip_lagged(cx)); - }, - NetEvent::SubscribeFailed { session, error } => { - let detached = self.update_session(session, cx, |session, cx| session.detach(DetachReason::Fatal(error), cx)); - if detached.unwrap_or(false) { - self.unregister_session(session); - } - }, - } - } - - fn update_session( - &mut self, - session: SessionId, - cx: &mut App, - update: impl FnOnce(&mut CollabSession, &mut Context) -> R, - ) -> Option { - self - .sessions_by_id - .get(&session) - .cloned() - .map(|session| session.update(cx, update)) - } - - fn finish_create_session(session: Entity, reply_rx: Receiver>, cx: &mut Context) - where - T: 'static, - { - cx.spawn(async move |_, cx| match reply_rx.recv().await { - Ok(Ok(seed)) => { - let _ = session.update(cx, |session, cx| session.establish_local_peer(&seed.inviter.id, cx)); - }, - Ok(Err(error)) => { - let _ = session.update(cx, |session, cx| { - session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error:#}")), cx); - }); - }, - Err(error) => { - let _ = session.update(cx, |session, cx| { - session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error}")), cx); - }); - }, - }) - .detach(); - } - - fn establish_joined_peer(session: Entity, commands: CommandSender, cx: &mut Context) -> Result<()> - where - T: 'static, - { - let (reply_tx, reply_rx) = async_channel::bounded(1); - commands - .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) - .context("requesting collaboration endpoint address failed")?; - cx.spawn(async move |_, cx| match reply_rx.recv().await { - Ok(addr) => { - let _ = session.update(cx, |session, cx| session.establish_local_peer(&addr.id, cx)); - }, - Err(error) => { - let _ = session.update(cx, |session, cx| { - session.detach(DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error}")), cx); - }); - }, - }) - .detach(); - Ok(()) - } -} - pub fn init(cx: &mut C) where C: BorrowAppContext, @@ -400,6 +67,18 @@ pub fn phase_for_panel(panel_id: Uuid, cx: &App) -> Option { CollabManager::global(cx).phase_for_panel(panel_id, cx) } +pub fn phase_for_session(session_id: SessionId, cx: &App) -> Option { + CollabManager::global(cx).phase_for_session(session_id, cx) +} + +pub fn session_for_panel(panel_id: Uuid, cx: &App) -> Option> { + CollabManager::global(cx).session_for_panel(panel_id) +} + +pub fn session_for_id(session_id: SessionId, cx: &App) -> Option> { + CollabManager::global(cx).session_for_id(session_id) +} + pub fn roster_for_panel(panel_id: Uuid, cx: &App) -> Vec { CollabManager::global(cx).roster_for_panel(panel_id, cx) } diff --git a/crates/flowstate/src/collab/notify.rs b/crates/flowstate/src/collab/notify.rs new file mode 100644 index 00000000..e0892a21 --- /dev/null +++ b/crates/flowstate/src/collab/notify.rs @@ -0,0 +1,40 @@ +use gpui::{App, Window}; +use gpui_component::{ + WindowExt as _, + notification::Notification, +}; + +use super::SessionNotice; + +pub fn show_session_notice(notice: &SessionNotice, window: &mut Window, cx: &mut App) { + match notice { + SessionNotice::PeerJoined(name) => push_info(collaborator_message(name, "joined"), window, cx), + SessionNotice::PeerLeft(name) => push_info(collaborator_message(name, "left"), window, cx), + SessionNotice::LeftSession => push_info("Left session - this copy is now local.", window, cx), + SessionNotice::ViewRebuilt => { + #[cfg(debug_assertions)] + push_info("Collaboration view rebuilt from remote state.", window, cx); + }, + SessionNotice::IncompatibleVersion(peer) => push_warning( + format!("{peer} is using an incompatible collaboration version."), + window, + cx, + ), + } +} + +fn push_info(message: impl Into, window: &mut Window, cx: &mut App) { + window.push_notification(Notification::info(message).title("Collaboration"), cx); +} + +fn push_warning(message: impl Into, window: &mut Window, cx: &mut App) { + window.push_notification(Notification::warning(message).title("Collaboration"), cx); +} + +fn collaborator_message(name: &str, action: &str) -> String { + if name.trim().is_empty() { + format!("A collaborator {action}") + } else { + format!("{name} {action}") + } +} diff --git a/crates/flowstate/src/collab/presence_view.rs b/crates/flowstate/src/collab/presence_view.rs index 66492dc6..d77099cc 100644 --- a/crates/flowstate/src/collab/presence_view.rs +++ b/crates/flowstate/src/collab/presence_view.rs @@ -39,7 +39,9 @@ pub fn external_carets(doc: &LoroDoc, binding: &DocBinding, presence: &PresenceS pub fn collaboration_recovery_path(session: SessionId, title: &str) -> PathBuf { let dir = std::env::temp_dir().join("flowstate-collab-recovery"); - let _ = std::fs::create_dir_all(&dir); + if let Err(error) = std::fs::create_dir_all(&dir) { + tracing::warn!("creating collaboration recovery directory failed ({}): {error}", dir.display()); + } let session_hex = session.to_string(); let prefix = session_hex.get(..16).unwrap_or(&session_hex); dir.join(format!("{prefix}-{}.db8", sanitized_recovery_title(title))) diff --git a/crates/flowstate/src/collab/pump.rs b/crates/flowstate/src/collab/pump.rs new file mode 100644 index 00000000..0c001bed --- /dev/null +++ b/crates/flowstate/src/collab/pump.rs @@ -0,0 +1,78 @@ +use flowstate_collab::{ + SessionId, + net::{NetEvent, runtime}, +}; +use gpui::{App, Context}; + +use super::{ + manager::CollabManager, + session::{CollabSession, DetachReason}, +}; + +impl CollabManager { + pub(super) fn start_event_pump(&mut self, events: runtime::EventReceiver, cx: &mut Context) + where + T: 'static, + { + if self.event_pump_started { + return; + } + self.event_pump_started = true; + cx.spawn(async move |_, cx| { + while let Ok(event) = events.recv().await { + if cx + .update_global::(|manager, cx| manager.handle_net_event(event, cx)) + .is_err() + { + break; + } + } + }) + .detach(); + } + + fn handle_net_event(&mut self, event: NetEvent, cx: &mut App) { + match event { + NetEvent::EndpointOnline(online) => { + self.endpoint_online = online; + for session in self.sessions_by_id.values() { + session.update(cx, |session, cx| session.set_endpoint_online(online, cx)); + } + }, + NetEvent::Gossip { session, from, msg } => { + self.update_session(session, cx, |session, cx| session.handle_gossip(from, msg, cx)); + }, + NetEvent::IncompatibleVersion { session, peer } => { + self.update_session(session, cx, |session, cx| session.handle_incompatible_version(peer, cx)); + }, + NetEvent::NeighborUp { session, peer } => { + self.update_session(session, cx, |session, cx| session.neighbor_up(peer, cx)); + }, + NetEvent::NeighborDown { session, peer } => { + self.update_session(session, cx, |session, cx| session.neighbor_down(peer, cx)); + }, + NetEvent::GossipLagged { session } => { + self.update_session(session, cx, |session, cx| session.handle_gossip_lagged(cx)); + }, + NetEvent::SubscribeFailed { session, error } => { + let detached = self.update_session(session, cx, |session, cx| session.detach(DetachReason::Fatal(error), cx)); + if detached.unwrap_or(false) { + self.unregister_session(session); + } + }, + } + } + + fn update_session( + &mut self, + session: SessionId, + cx: &mut App, + update: impl FnOnce(&mut CollabSession, &mut Context) -> R, + ) -> Option { + self + .sessions_by_id + .get(&session) + .cloned() + .map(|session| session.update(cx, update)) + } +} diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index 94ee7175..49c5ec67 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -1,9 +1,10 @@ -use std::{collections::HashSet, rc::Rc, time::Instant}; +use std::{collections::HashSet, rc::Rc, time::{Duration, Instant}}; use anyhow::{Context as _, Result, anyhow, bail}; use flowstate_collab::{ SessionId, binding::DocBinding, + ids::PeerId, local_apply::LocalApplier, net::{ NetCommand, PeerAddr, PublishPayload, @@ -16,7 +17,7 @@ use flowstate_collab::{ proto_gossip::GossipMsg, schema, }; -use gpui::{Context, Entity, Subscription}; +use gpui::{Context, Entity, EventEmitter, Subscription, Timer}; use loro::{LoroDoc, Subscription as LoroSubscription, UndoManager}; use uuid::Uuid; @@ -69,6 +70,22 @@ pub enum DetachReason { Fatal(String), } +#[derive(Clone, Debug)] +pub enum SessionNotice { + PeerJoined(String), + PeerLeft(String), + LeftSession, + ViewRebuilt, + IncompatibleVersion(String), +} + +impl EventEmitter for CollabSession {} + +pub(super) enum JoinNeighborSignal { + NeighborUp, + TimedOut, +} + pub struct JoinedDocument { pub session: SessionId, pub title: String, @@ -107,6 +124,7 @@ pub struct CollabSession { undo_manager: Option, direct_pump_started: bool, undo_pump_started: bool, + presence_refresh_pending: bool, local_update_publish_attached: bool, timers_started: bool, endpoint_online: bool, @@ -114,9 +132,22 @@ pub struct CollabSession { inbound_since_last_digest: bool, quiet_digest_rounds: u8, next_recovery_at: Option, + awaiting_recovery_neighbor_until: Option, + known_peers: HashSet, + probe_pending: bool, + last_probe_failed: bool, + join_neighbor_tx: Option>, + incompatible_version_peers: HashSet, last_document_activity: Instant, } +const DIRECT_REQUEST_CHANNEL_CAPACITY: usize = 32; +const UNDO_REQUEST_CHANNEL_CAPACITY: usize = 128; +const UNDO_MERGE_INTERVAL_MS: i64 = 500; +const UNDO_MAX_STEPS: usize = 300; +const JOIN_FIRST_NEIGHBOR_TIMEOUT: Duration = Duration::from_secs(15); +const JOIN_SNAPSHOT_TIMEOUT: Duration = Duration::from_secs(30); + impl CollabSession { pub fn from_local_document( session: SessionId, @@ -129,8 +160,10 @@ impl CollabSession { let doc = schema::new_configured_doc(); projection::populate_from_document(&doc, session, &title, &document)?; let binding = DocBinding::build(&doc, &document)?; - let (direct_tx, direct_rx) = async_channel::unbounded(); - let (undo_tx, undo_rx) = async_channel::unbounded(); + // Direct serving is network-bound and should backpressure rather than grow without limit. + let (direct_tx, direct_rx) = async_channel::bounded(DIRECT_REQUEST_CHANNEL_CAPACITY); + // Undo redirects can burst under key-repeat, but a stuck pump should not retain unbounded UI events. + let (undo_tx, undo_rx) = async_channel::bounded(UNDO_REQUEST_CHANNEL_CAPACITY); let now = Instant::now(); Ok(Self { @@ -158,6 +191,7 @@ impl CollabSession { undo_manager: None, direct_pump_started: false, undo_pump_started: false, + presence_refresh_pending: false, local_update_publish_attached: false, timers_started: false, endpoint_online: true, @@ -165,13 +199,19 @@ impl CollabSession { inbound_since_last_digest: false, quiet_digest_rounds: 0, next_recovery_at: None, + awaiting_recovery_neighbor_until: None, + known_peers: HashSet::new(), + probe_pending: false, + last_probe_failed: false, + join_neighbor_tx: None, + incompatible_version_peers: HashSet::new(), last_document_activity: now, }) } pub fn joining(session: SessionId, title: String, net_tx: CommandSender, bootstrap_addrs: Vec) -> Self { - let (direct_tx, direct_rx) = async_channel::unbounded(); - let (undo_tx, undo_rx) = async_channel::unbounded(); + let (direct_tx, direct_rx) = async_channel::bounded(DIRECT_REQUEST_CHANNEL_CAPACITY); + let (undo_tx, undo_rx) = async_channel::bounded(UNDO_REQUEST_CHANNEL_CAPACITY); let now = Instant::now(); Self { session, @@ -198,6 +238,7 @@ impl CollabSession { undo_manager: None, direct_pump_started: false, undo_pump_started: false, + presence_refresh_pending: false, local_update_publish_attached: false, timers_started: false, endpoint_online: true, @@ -205,6 +246,12 @@ impl CollabSession { inbound_since_last_digest: false, quiet_digest_rounds: 0, next_recovery_at: None, + awaiting_recovery_neighbor_until: None, + known_peers: HashSet::new(), + probe_pending: false, + last_probe_failed: false, + join_neighbor_tx: None, + incompatible_version_peers: HashSet::new(), last_document_activity: now, } } @@ -265,28 +312,50 @@ impl CollabSession { cx.notify(); } - pub fn begin_join_bootstrap( + pub(super) fn prepare_join_neighbor_wait(&mut self, cx: &mut Context) -> async_channel::Receiver { + let (neighbor_tx, neighbor_rx) = async_channel::bounded(1); + self.join_neighbor_tx = Some(neighbor_tx.clone()); + self.phase = SessionPhase::Joining(JoinStage::Subscribing); + cx.spawn(async move |_, _| { + Timer::after(JOIN_FIRST_NEIGHBOR_TIMEOUT).await; + let _ = neighbor_tx.try_send(JoinNeighborSignal::TimedOut); + }) + .detach(); + cx.notify(); + neighbor_rx + } + + pub(super) fn begin_join_bootstrap( &mut self, - inviter: flowstate_collab::ids::PeerId, + inviter: PeerId, + neighbor_rx: async_channel::Receiver, cx: &mut Context, ) -> async_channel::Receiver> { let (result_tx, result_rx) = async_channel::bounded(1); - self.phase = SessionPhase::Joining(JoinStage::FetchingSnapshot { got: 0, total: None }); - cx.notify(); - - let (reply_tx, reply_rx) = async_channel::bounded(1); - if let Err(error) = self.net_tx.try_send(NetCommand::PullSnapshot { - session: self.session, - candidates: self.pull_candidates(Some(inviter)), - reply: reply_tx, - }) { - let detail = format!("requesting collaboration snapshot failed: {error}"); - self.detach(DetachReason::JoinFailed(detail.clone()), cx); - let _ = result_tx.try_send(Err(anyhow!(detail))); - return result_rx; - } cx.spawn(async move |session, cx| { + match neighbor_rx.recv().await { + Ok(JoinNeighborSignal::NeighborUp) => {}, + Ok(JoinNeighborSignal::TimedOut) | Err(_) => { + let detail = "Couldn't reach anyone in this session. Make sure the inviter is online and the invite is current, then try again.".to_string(); + let _ = session.update(cx, |session, cx| { + session.detach(DetachReason::JoinFailed(detail.clone()), cx); + }); + let _ = result_tx.try_send(Err(anyhow!(detail))); + return; + }, + } + + let reply_rx = match session.update(cx, |session, cx| session.start_join_snapshot_pull(inviter, result_tx.clone(), cx)) { + Ok(Ok(reply_rx)) => reply_rx, + Ok(Err(_)) => return, + Err(error) => { + let detail = format!("collaboration join session disappeared: {error}"); + let _ = result_tx.try_send(Err(anyhow!(detail))); + return; + }, + }; + let joined = match reply_rx.recv().await { Ok(Ok(bytes)) => session .update(cx, |session, cx| match session.finish_join_snapshot(&bytes, cx) { @@ -313,13 +382,79 @@ impl CollabSession { Err(anyhow!(detail)) }, }; - let _ = result_tx.send(joined).await; + let _ = result_tx.try_send(joined); }) .detach(); result_rx } + fn start_join_snapshot_pull( + &mut self, + inviter: PeerId, + result_tx: async_channel::Sender>, + cx: &mut Context, + ) -> Result>>> { + self.join_neighbor_tx = None; + self.phase = SessionPhase::Joining(JoinStage::FetchingSnapshot { got: 0, total: None }); + cx.notify(); + + let (reply_tx, reply_rx) = async_channel::bounded(1); + let (progress_tx, progress_rx) = async_channel::bounded(8); + if let Err(error) = self.net_tx.try_send(NetCommand::PullSnapshot { + session: self.session, + candidates: self.pull_candidates(Some(inviter)), + progress: Some(progress_tx), + reply: reply_tx, + }) { + let detail = format!("requesting collaboration snapshot failed: {error}"); + self.detach(DetachReason::JoinFailed(detail.clone()), cx); + let _ = result_tx.try_send(Err(anyhow!(detail.clone()))); + bail!(detail); + } + + cx.spawn(async move |session, cx| { + while let Ok(progress) = progress_rx.recv().await { + if session + .update(cx, |session, cx| { + if matches!(session.phase, SessionPhase::Joining(JoinStage::FetchingSnapshot { .. })) { + session.phase = SessionPhase::Joining(JoinStage::FetchingSnapshot { + got: progress.got, + total: Some(progress.total), + }); + cx.notify(); + } + }) + .is_err() + { + break; + } + } + }) + .detach(); + + cx.spawn(async move |session, cx| { + Timer::after(JOIN_SNAPSHOT_TIMEOUT).await; + let detail = "Joining this session timed out while fetching the document snapshot. Check your connection and try again.".to_string(); + let timed_out = session + .update(cx, |session, cx| { + if matches!(session.phase, SessionPhase::Joining(JoinStage::FetchingSnapshot { .. })) { + session.detach(DetachReason::JoinFailed(detail.clone()), cx); + true + } else { + false + } + }) + .unwrap_or(false); + if timed_out { + let _ = result_tx.try_send(Err(anyhow!(detail))); + } + }) + .detach(); + + Ok(reply_rx) + } + pub fn attach_joined_editor(&mut self, panel_id: Uuid, editor: Entity, cx: &mut Context) -> Result<()> { if self.doc.is_none() || self.binding.is_none() { bail!("collaboration snapshot has not finished loading"); @@ -378,6 +513,15 @@ impl CollabSession { } fn finish_join_snapshot(&mut self, snapshot: &[u8], cx: &mut Context) -> Result { + if matches!(self.phase, SessionPhase::Detached(_)) { + bail!("collaboration join is no longer active"); + } + let total = snapshot.len() as u64; + self.phase = SessionPhase::Joining(JoinStage::FetchingSnapshot { + got: total, + total: Some(total), + }); + cx.notify(); self.phase = SessionPhase::Joining(JoinStage::Building); cx.notify(); @@ -393,7 +537,7 @@ impl CollabSession { self.binding = Some(binding); Ok(JoinedDocument { session: self.session, - title: self.title.clone(), + title: format!("{} (shared)", self.title), document, }) } @@ -403,6 +547,7 @@ impl CollabSession { return false; } + let user_left = matches!(reason, DetachReason::UserLeft); if let Some(presence) = &self.presence { presence.delete_self(); self.publish_presence_bytes(presence.encode_self()); @@ -416,7 +561,7 @@ impl CollabSession { editor.update(cx, |editor, cx| { editor.set_collab_undo_redirect(None); editor.set_collab_capture(false); - editor.clear_collab_history(); + editor.clear_undo_redo_stacks(); let _ = editor.take_pending_collab_edits(); editor.set_external_carets(Vec::new(), cx); }); @@ -436,8 +581,16 @@ impl CollabSession { self.inbound_since_last_digest = false; self.quiet_digest_rounds = 0; self.next_recovery_at = None; + self.awaiting_recovery_neighbor_until = None; + self.probe_pending = false; + self.last_probe_failed = false; + self.join_neighbor_tx = None; + self.presence_refresh_pending = false; self.local_update_publish_attached = false; self.phase = SessionPhase::Detached(reason); + if user_left { + cx.emit(SessionNotice::LeftSession); + } cx.notify(); true } @@ -450,7 +603,7 @@ impl CollabSession { let document = editor.read(cx).document().clone(); let edits = editor.update(cx, |editor, _| { let edits = editor.take_pending_collab_edits(); - editor.clear_collab_history(); + editor.clear_undo_redo_stacks(); edits }); let Some(doc) = &self.doc else { @@ -475,6 +628,7 @@ impl CollabSession { } pub fn handle_gossip(&mut self, from: flowstate_collab::ids::PeerId, msg: GossipMsg, cx: &mut Context) { + self.known_peers.insert(from); self.note_inbound_traffic(cx); let result = match msg { GossipMsg::Update(bytes) => self @@ -497,7 +651,12 @@ impl CollabSession { pub fn neighbor_up(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { self.neighbors.insert(peer); + self.known_peers.insert(peer); self.zero_neighbors_since = None; + self.last_probe_failed = false; + if let Some(join_neighbor_tx) = self.join_neighbor_tx.take() { + let _ = join_neighbor_tx.try_send(JoinNeighborSignal::NeighborUp); + } self.anti_entropy.on_neighbor_up(); self.mark_online(cx); self.publish_digest(); @@ -528,6 +687,9 @@ impl CollabSession { pub fn set_endpoint_online(&mut self, online: bool, cx: &mut Context) { self.endpoint_online = online; + if online { + self.last_probe_failed = false; + } if online { if self.peers_present() == 0 || !self.neighbors.is_empty() { self.mark_online(cx); @@ -538,6 +700,12 @@ impl CollabSession { cx.notify(); } + pub fn handle_incompatible_version(&mut self, peer: PeerId, cx: &mut Context) { + if self.incompatible_version_peers.insert(peer) { + cx.emit(SessionNotice::IncompatibleVersion(peer.to_string())); + } + } + fn attach_editor_hooks(&mut self, cx: &mut Context) { if !self.editor_subscriptions.is_empty() { return; @@ -550,7 +718,7 @@ impl CollabSession { if editor.document_path().is_none() { editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title)), cx); } - editor.clear_collab_history(); + editor.clear_undo_redo_stacks(); editor.set_collab_capture(true); let undo_tx = self.undo_tx.clone(); editor.set_collab_undo_redirect(Some(Rc::new(move |redirect: UndoRedirect| { @@ -572,7 +740,7 @@ impl CollabSession { .editor_subscriptions .push(cx.subscribe(&editor, |session, _, event: &EditorEvent, cx| { if matches!(event, EditorEvent::SelectionChanged { .. }) { - session.refresh_own_presence(cx); + session.schedule_own_presence_refresh(cx); } })); } @@ -606,8 +774,8 @@ impl CollabSession { return; }; let mut undo_manager = UndoManager::new(doc); - undo_manager.set_merge_interval(500); - undo_manager.set_max_undo_steps(300); + undo_manager.set_merge_interval(UNDO_MERGE_INTERVAL_MS); + undo_manager.set_max_undo_steps(UNDO_MAX_STEPS); self.undo_manager = Some(undo_manager); } diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs index 2323b517..209d4768 100644 --- a/crates/flowstate/src/collab/session_io.rs +++ b/crates/flowstate/src/collab/session_io.rs @@ -14,6 +14,7 @@ use flowstate_collab::{ }; use gpui::Context; use loro::{ExportMode, LoroDoc, Subscription as LoroSubscription, VersionVector, event::Subscriber}; +use tracing::warn; use crate::rich_text_element::{AssetId, CollabPatch, Document, UndoRedirect}; @@ -141,7 +142,7 @@ impl CollabSession { } let patches = std::mem::take(&mut self.pending_remote_patches); editor.update(cx, |editor, cx| { - editor.clear_collab_history(); + editor.clear_undo_redo_stacks(); editor.apply_collab_patches(&patches, cx); }); self.last_document_activity = std::time::Instant::now(); @@ -199,7 +200,7 @@ impl CollabSession { GapAction::None => {}, GapAction::Pull { from, our_vv } => self.start_update_pull(from, our_vv, cx), GapAction::LineageMismatch { from, expected, got } => { - eprintln!("flowstate collab ignored mismatched digest from {from}: expected session {expected}, got {got}"); + warn!("flowstate collab ignored mismatched digest from {from}: expected session {expected}, got {got}"); }, } } @@ -266,7 +267,7 @@ impl CollabSession { let mut binding = match binding.lock() { Ok(binding) => binding, Err(error) => { - eprintln!("flowstate collab binding lock failed: {error}"); + tracing::warn!("flowstate collab binding lock failed: {error}"); return; }, }; @@ -286,7 +287,7 @@ impl CollabSession { patches.append(&mut produced); } }, - Err(error) => eprintln!("flowstate collab remote apply failed: {error:#}"), + Err(error) => tracing::warn!("flowstate collab remote apply failed: {error:#}"), } }); subscribed_doc.subscribe_root(callback) diff --git a/crates/flowstate/src/collab/session_presence.rs b/crates/flowstate/src/collab/session_presence.rs index b9e69d29..150ad7cd 100644 --- a/crates/flowstate/src/collab/session_presence.rs +++ b/crates/flowstate/src/collab/session_presence.rs @@ -1,14 +1,22 @@ -use flowstate_collab::presence::{PresenceSelection, PresenceState}; -use gpui::Context; +use std::{collections::HashMap, time::Duration}; -use super::{CollabSession, presence_view}; +use flowstate_collab::presence::{PresenceSelection, PresenceState, PresenceStore}; +use gpui::{Context, Timer}; +use tracing::warn; + +use super::{CollabSession, SessionNotice, presence_view}; + +const PRESENCE_REFRESH_DEBOUNCE: Duration = Duration::from_millis(50); impl CollabSession { pub(super) fn apply_presence(&mut self, bytes: &[u8], cx: &mut Context) { if let Some(presence) = &self.presence { + let before = remote_roster(presence); if let Err(error) = presence.apply(bytes) { - eprintln!("flowstate collab presence update failed: {error:#}"); + warn!(error = %error, "flowstate collab presence update failed"); + return; } + self.emit_presence_roster_diff(before, cx); self.refresh_peer_count(); self.evaluate_connectivity(cx); self.refresh_external_carets(cx); @@ -25,11 +33,35 @@ impl CollabSession { selection: self.own_presence_selection(cx), }; if let Err(error) = presence.set_self(&state) { - eprintln!("flowstate collab presence encode failed: {error:#}"); + warn!(error = %error, "flowstate collab presence encode failed"); } self.refresh_peer_count(); } + pub(super) fn schedule_own_presence_refresh(&mut self, cx: &mut Context) { + if self.presence.is_none() || self.presence_refresh_pending { + return; + } + self.presence_refresh_pending = true; + cx.spawn(async move |session, cx| { + Timer::after(PRESENCE_REFRESH_DEBOUNCE).await; + let _ = session.update(cx, |session, cx| { + session.presence_refresh_pending = false; + session.refresh_own_presence(cx); + }); + }) + .detach(); + } + + pub(super) fn remove_outdated_presence(&mut self, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let before = remote_roster(presence); + presence.remove_outdated(); + self.emit_presence_roster_diff(before, cx); + } + pub(super) fn refresh_external_carets(&mut self, cx: &mut Context) { let Some(presence) = &self.presence else { return; @@ -84,6 +116,33 @@ impl CollabSession { let binding = self.binding.as_ref()?; presence_view::selection_for_editor(editor, binding) } + + fn emit_presence_roster_diff(&mut self, before: HashMap, cx: &mut Context) { + let Some(presence) = &self.presence else { + return; + }; + let after = remote_roster(presence); + for (key, name) in &after { + if !before.contains_key(key) { + cx.emit(SessionNotice::PeerJoined(name.clone())); + } + } + for (key, name) in before { + if !after.contains_key(&key) { + cx.emit(SessionNotice::PeerLeft(name)); + } + } + } +} + +fn remote_roster(presence: &PresenceStore) -> HashMap { + let self_key = presence.self_key().to_string(); + presence + .roster() + .into_iter() + .filter(|entry| entry.key != self_key) + .map(|entry| (entry.key, entry.name)) + .collect() } fn default_presence_name() -> String { diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs index 17d69be9..f39d026b 100644 --- a/crates/flowstate/src/collab/session_timers.rs +++ b/crates/flowstate/src/collab/session_timers.rs @@ -3,16 +3,18 @@ use std::time::{Duration, Instant}; use anyhow::{Context as _, Result}; use flowstate_collab::{binding::DocBinding, net::NetCommand, presence::PRESENCE_KEEPALIVE_SECS, projection, self_check}; use gpui::{Context, Timer}; +use tracing::warn; use crate::{app_settings::load_document_theme, rich_text_element::Document}; -use super::{Attachment, CollabSession, Connectivity, SessionPhase}; +use super::{Attachment, CollabSession, Connectivity, DetachReason, SessionNotice, SessionPhase}; const ZERO_NEIGHBOR_OFFLINE_GRACE: Duration = Duration::from_secs(5); const QUIET_DIGEST_ROUNDS_OFFLINE: u8 = 2; const SELF_CHECK_INTERVAL: Duration = Duration::from_secs(30); const SELF_CHECK_IDLE: Duration = Duration::from_secs(2); const RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30); +const RECOVERY_NEIGHBOR_WAIT: Duration = Duration::from_secs(10); impl CollabSession { pub(super) fn attach_timers(&mut self, cx: &mut Context) { @@ -35,12 +37,22 @@ impl CollabSession { } pub(super) fn mark_online(&mut self, cx: &mut Context) -> bool { + if self + .awaiting_recovery_neighbor_until + .is_some_and(|deadline| self.peers_present() > 0 && self.neighbors.is_empty() && Instant::now() < deadline) + { + return false; + } + let mut changed = false; if let SessionPhase::Attached(attachment) = &mut self.phase && matches!(attachment.connectivity, Connectivity::Offline { .. }) { attachment.connectivity = Connectivity::Online; self.next_recovery_at = None; + self.awaiting_recovery_neighbor_until = None; + self.probe_pending = false; + self.last_probe_failed = false; changed = true; } if changed { @@ -62,8 +74,12 @@ impl CollabSession { let now = Instant::now(); let zero_since = self.zero_neighbors_since.get_or_insert(now); - let quiet_enough = self.quiet_digest_rounds >= QUIET_DIGEST_ROUNDS_OFFLINE || !self.endpoint_online; - if quiet_enough && now.saturating_duration_since(*zero_since) >= ZERO_NEIGHBOR_OFFLINE_GRACE { + let zero_long_enough = now.saturating_duration_since(*zero_since) >= ZERO_NEIGHBOR_OFFLINE_GRACE; + let quiet_enough = self.quiet_digest_rounds >= QUIET_DIGEST_ROUNDS_OFFLINE; + if zero_long_enough + && quiet_enough + && (!self.endpoint_online || self.last_probe_failed || !self.start_connectivity_probe(cx)) + { self.mark_offline(now, cx); } } @@ -138,9 +154,7 @@ impl CollabSession { return false; } if self.presence.is_some() { - if let Some(presence) = &self.presence { - presence.remove_outdated(); - } + self.remove_outdated_presence(cx); self.refresh_own_presence(cx); self.refresh_peer_count(); self.evaluate_connectivity(cx); @@ -158,7 +172,7 @@ impl CollabSession { return true; } if let Err(error) = self.run_self_check(cx) { - eprintln!("flowstate collab self-check failed: {error:#}"); + warn!(error = %error, "flowstate collab self-check failed"); } true } @@ -195,6 +209,7 @@ impl CollabSession { } if attempted { + self.awaiting_recovery_neighbor_until = Some(now + RECOVERY_NEIGHBOR_WAIT); if self.bootstrap_addrs.is_empty() { let _ = self.net_tx.try_send(NetCommand::EnsureUp); } else { @@ -225,28 +240,85 @@ impl CollabSession { return Ok(()); } - eprintln!( - "flowstate collab projection drift detected: live_hash={live_hash:016x} projected_hash={projected_hash:016x} vv_bytes={}", - doc.oplog_vv().encode().len() + warn!( + live_hash, + projected_hash, + vv_bytes = doc.oplog_vv().encode().len(), + "flowstate collab projection drift detected" ); self.rebuild_from_projection(projected, cx) } fn rebuild_from_projection(&mut self, projected: Document, cx: &mut Context) -> Result<()> { - let Some(doc) = &self.doc else { + let Some(doc) = self.doc.clone() else { return Ok(()); }; let Some(editor) = self.editor.clone() else { return Ok(()); }; editor.update(cx, |editor, cx| editor.replace_document_from_collaboration(projected, cx)); + self.pending_remote_patches.clear(); let document = editor.read(cx).document().clone(); - self.binding = Some(DocBinding::build(doc, &document).context("rebuilding collaboration binding after self-check failed")?); + self.binding = Some(DocBinding::build(&doc, &document).context("rebuilding collaboration binding after self-check failed")?); self.last_document_activity = Instant::now(); self.refresh_external_carets(cx); + cx.emit(SessionNotice::ViewRebuilt); Ok(()) } + fn start_connectivity_probe(&mut self, cx: &mut Context) -> bool { + if self.probe_pending { + return true; + } + let Some(doc) = &self.doc else { + return false; + }; + let candidates = self.known_peers.iter().copied().collect::>(); + if candidates.is_empty() { + return false; + } + let our_vv = doc.oplog_vv().encode(); + let (reply_tx, reply_rx) = async_channel::bounded(1); + if self + .net_tx + .try_send(NetCommand::PullUpdates { + session: self.session, + candidates, + our_vv, + reply: reply_tx, + }) + .is_err() + { + return false; + } + + self.probe_pending = true; + cx.spawn(async move |session, cx| { + let result = reply_rx.recv().await; + let _ = session.update(cx, |session, cx| { + session.probe_pending = false; + match result { + Ok(Ok(bytes)) => { + session.last_probe_failed = false; + if !bytes.is_empty() + && let Err(error) = session.import_update_bytes(&bytes, cx) + { + session.detach(DetachReason::Fatal(format!("probing collaboration peer failed to apply updates: {error:#}")), cx); + return; + } + session.note_inbound_traffic(cx); + }, + Ok(Err(_)) | Err(_) => { + session.last_probe_failed = true; + session.evaluate_connectivity(cx); + }, + } + }); + }) + .detach(); + true + } + fn mark_offline(&mut self, now: Instant, cx: &mut Context) { if let SessionPhase::Attached(attachment) = &mut self.phase && matches!(attachment.connectivity, Connectivity::Online) diff --git a/crates/flowstate/src/collab/share_dialog.rs b/crates/flowstate/src/collab/share_dialog.rs index 7a92e6c1..3031946f 100644 --- a/crates/flowstate/src/collab/share_dialog.rs +++ b/crates/flowstate/src/collab/share_dialog.rs @@ -1,10 +1,10 @@ -use flowstate_collab::ticket::SessionTicket; +use flowstate_collab::{SessionId, ticket::SessionTicket}; use gpui::{ - AnyElement, AnyWindowHandle, App, ClipboardItem, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, KeyDownEvent, - MouseButton, ParentElement, Render, SharedString, Subscription, WeakEntity, Window, div, prelude::*, px, relative, rgb, + AnyElement, App, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, KeyDownEvent, ParentElement, Render, + SharedString, Subscription, WeakEntity, Window, div, prelude::*, px, }; use gpui_component::{ - ActiveTheme as _, Disableable, IconName, Sizable, + Disableable, WindowExt as _, button::{Button, ButtonVariants as _}, h_flex, input::{Input, InputEvent, InputState}, @@ -15,7 +15,11 @@ use uuid::Uuid; use crate::workspace::Workspace; -use super::{Connectivity, SessionPhase, status}; +use super::{DetachReason, SessionPhase, status}; + +#[path = "share_dialog_view.rs"] +mod share_dialog_view; +use share_dialog_view::*; #[derive(Clone, Copy, PartialEq, Eq)] pub enum CollabDialogMode { @@ -28,12 +32,14 @@ pub struct CollabShareDialog { panel_id: Option, mode: CollabDialogMode, join_input: Entity, + joining_session: Option, ticket_text: Option, ticket_loading: bool, ticket_error: Option, copy_notice: Option, join_error: Option, _input_subscription: Subscription, + _join_subscription: Option, } #[hotpath::measure_all] @@ -46,8 +52,10 @@ impl CollabShareDialog { cx: &mut Context, ) -> Self { let join_input = cx.new(|cx| InputState::new(window, cx).placeholder("Paste a Flowstate collaboration invite")); - let _input_subscription = cx.subscribe(&join_input, |dialog, _, event: &InputEvent, cx| { + let input_subscription = cx.subscribe(&join_input, |dialog, _, event: &InputEvent, cx| { if let InputEvent::Change = event { + dialog.joining_session = None; + dialog._join_subscription = None; dialog.validate_join_ticket(cx); } }); @@ -57,15 +65,17 @@ impl CollabShareDialog { panel_id, mode, join_input, + joining_session: None, ticket_text: None, ticket_loading: false, ticket_error: None, copy_notice: None, join_error: None, - _input_subscription, + _input_subscription: input_subscription, + _join_subscription: None, }; if mode == CollabDialogMode::Share { - dialog.refresh_ticket(None, cx); + dialog.refresh_ticket(cx); } dialog } @@ -76,10 +86,11 @@ impl CollabShareDialog { } } - fn close(&mut self, cx: &mut Context) { + fn close(&mut self, window: &mut Window, cx: &mut Context) { let _ = self .workspace .update(cx, |workspace, cx| workspace.close_collaboration_dialog(cx)); + window.close_dialog(cx); } fn set_mode(&mut self, mode: CollabDialogMode, cx: &mut Context) { @@ -90,7 +101,7 @@ impl CollabShareDialog { self.join_error = None; self.copy_notice = None; if mode == CollabDialogMode::Share { - self.refresh_ticket(None, cx); + self.refresh_ticket(cx); } cx.notify(); } @@ -106,7 +117,7 @@ impl CollabShareDialog { .workspace .update(cx, |workspace, cx| workspace.start_collaboration_on_document(panel_id, cx)) { - Ok(Some(_)) => self.refresh_ticket(None, cx), + Ok(Some(_)) => self.refresh_ticket(cx), Ok(None) => { self.ticket_error = Some("The active document could not be shared.".into()); cx.notify(); @@ -118,7 +129,7 @@ impl CollabShareDialog { } } - fn refresh_ticket(&mut self, copy_to_clipboard: Option, cx: &mut Context) { + fn refresh_ticket(&mut self, cx: &mut Context) { let Some(panel_id) = self.panel_id else { self.ticket_text = None; self.ticket_loading = false; @@ -143,14 +154,6 @@ impl CollabShareDialog { Ok(Err(error)) => Err(format!("Creating collaboration invite failed: {error:#}")), Err(error) => Err(format!("Creating collaboration invite failed: {error}")), }; - let should_copy = copy_to_clipboard.is_some(); - - if let Ok(text) = &encoded - && let Some(window_handle) = copy_to_clipboard - { - let text = text.clone(); - let _ = window_handle.update(cx, |_, _, cx| cx.write_to_clipboard(ClipboardItem::new_string(text))); - } let _ = dialog.update(cx, |dialog, cx| { dialog.ticket_loading = false; @@ -158,9 +161,6 @@ impl CollabShareDialog { Ok(text) => { dialog.ticket_text = Some(text.into()); dialog.ticket_error = None; - if should_copy { - dialog.copy_notice = Some("Fresh invite copied to clipboard.".into()); - } }, Err(error) => { dialog.ticket_error = Some(error.into()); @@ -172,10 +172,6 @@ impl CollabShareDialog { .detach(); } - fn copy_invite(&mut self, window: &mut Window, cx: &mut Context) { - self.refresh_ticket(Some(window.window_handle()), cx); - } - fn validate_join_ticket(&mut self, cx: &mut Context) { self.join_error = match self.parse_join_ticket(cx) { Ok(_) => None, @@ -195,6 +191,9 @@ impl CollabShareDialog { } fn join_session(&mut self, window: &mut Window, cx: &mut Context) { + if self.joining_session.is_some() { + return; + } let ticket = match self.parse_join_ticket(cx) { Ok(Some(ticket)) => ticket, Ok(None) => { @@ -213,7 +212,12 @@ impl CollabShareDialog { .workspace .update(cx, |workspace, cx| workspace.join_collaboration_session(ticket, window, cx)) { - Ok(Some(_)) => self.close(cx), + Ok(Some(session)) => { + self.join_error = None; + self.joining_session = Some(session); + self.subscribe_join_session(session, cx); + cx.notify(); + }, Ok(None) => { self.join_error = Some("Joining collaboration session failed.".into()); cx.notify(); @@ -225,6 +229,10 @@ impl CollabShareDialog { } } + fn subscribe_join_session(&mut self, session_id: SessionId, cx: &mut Context) { + self._join_subscription = crate::collab::session_for_id(session_id, cx).map(|session| cx.observe(&session, |_, _, cx| cx.notify())); + } + fn confirm_leave(&mut self, window: &mut Window, cx: &mut Context) { let Some(panel_id) = self.panel_id else { return; @@ -237,7 +245,7 @@ impl CollabShareDialog { fn on_key_down(&mut self, event: &KeyDownEvent, window: &mut Window, cx: &mut Context) { match event.keystroke.key.as_str() { "escape" => { - self.close(cx); + self.close(window, cx); cx.stop_propagation(); }, "enter" if self.mode == CollabDialogMode::Join => { @@ -296,35 +304,19 @@ impl CollabShareDialog { .when_some(self.ticket_error.clone(), |this, error| this.child(error_text(error, cx))) .when_some(self.copy_notice.clone(), |this, notice| this.child(success_text(notice, cx))) .child( - h_flex() - .gap_2() - .child( - Button::new("copy-collaboration-invite") - .label(if self.ticket_loading { "Minting..." } else { "Copy invite" }) - .primary() - .disabled(self.ticket_loading) - .on_click(cx.listener(|dialog, _, window, cx| dialog.copy_invite(window, cx))), - ) - .child( - Button::new("refresh-collaboration-invite") - .label("Refresh invite") - .outline() - .disabled(self.ticket_loading) - .on_click(cx.listener(|dialog, _, _, cx| dialog.refresh_ticket(None, cx))), - ) - .child(div().flex_1()) - .child( - Button::new("leave-collaboration-session") - .label("Leave session") - .danger() - .on_click(cx.listener(|dialog, _, window, cx| dialog.confirm_leave(window, cx))), - ), + h_flex().justify_end().child( + Button::new("leave-collaboration-session") + .label("Leave session") + .danger() + .on_click(cx.listener(|dialog, _, window, cx| dialog.confirm_leave(window, cx))), + ), ) .into_any_element() } fn render_join_pane(&self, cx: &mut Context) -> AnyElement { let parsed = self.parse_join_ticket(cx).ok().flatten(); + let progress = self.join_progress(cx); v_flex() .gap_3() .child(section_title("Join a session", cx)) @@ -336,17 +328,36 @@ impl CollabShareDialog { .when_some(parsed.as_ref().map(|ticket| ticket.title.clone()), |this, title| { this.child(success_text(format!("Invite for: {title}").into(), cx)) }) + .when_some(progress, |this, (is_error, text)| { + if is_error { + this.child(error_text(text, cx)) + } else { + this.child(progress_text(text, cx)) + } + }) .when_some(self.join_error.clone(), |this, error| this.child(error_text(error, cx))) .child( - h_flex().justify_end().gap_2().child( + h_flex().justify_end().child( Button::new("join-collaboration-session") - .label("Join") + .label(if self.joining_session.is_some() { "Joining..." } else { "Join" }) .primary() + .disabled(self.joining_session.is_some()) .on_click(cx.listener(|dialog, _, window, cx| dialog.join_session(window, cx))), ), ) .into_any_element() } + + fn join_progress(&self, cx: &App) -> Option<(bool, SharedString)> { + let phase = crate::collab::phase_for_session(self.joining_session?, cx)?; + match phase { + SessionPhase::Creating => Some((false, "Starting collaboration...".into())), + SessionPhase::Joining(stage) => Some((false, status::join_stage_label(&stage).into())), + SessionPhase::Attached(_) => Some((false, "Joined. Opening shared document...".into())), + SessionPhase::Detached(DetachReason::JoinFailed(error)) => Some((true, error.into())), + SessionPhase::Detached(_) => None, + } + } } #[hotpath::measure_all] @@ -363,194 +374,29 @@ impl Render for CollabShareDialog { .panel_id .and_then(|panel_id| crate::collab::phase_for_panel(panel_id, cx)); - div() - .absolute() - .top_0() - .right_0() - .bottom_0() - .left_0() - .bg(cx.theme().background.opacity(0.72)) - .flex() - .items_center() - .justify_center() - .occlude() + v_flex() + .max_h(px(560.0)) + .gap_4() .on_key_down(cx.listener(Self::on_key_down)) - .on_mouse_down( - MouseButton::Left, - cx.listener(|dialog, _, _, cx| { - dialog.close(cx); - cx.stop_propagation(); - }), - ) - .on_scroll_wheel(|_, _, cx| cx.stop_propagation()) .child( - v_flex() - .w(px(620.0)) - .max_w_full() - .max_h(px(620.0)) - .overflow_hidden() - .rounded_lg() - .border_1() - .border_color(cx.theme().border) - .bg(cx.theme().popover) - .shadow_lg() - .on_mouse_down(MouseButton::Left, |_, _, cx| cx.stop_propagation()) - .child( - h_flex() - .h(px(44.0)) - .items_center() - .justify_between() - .px_4() - .border_b_1() - .border_color(cx.theme().border) - .child( - div() - .font_weight(gpui::FontWeight::SEMIBOLD) - .child("Share / Collaborate"), - ) - .child( - Button::new("close-collaboration-dialog") - .icon(IconName::Close) - .xsmall() - .ghost() - .tooltip("Close") - .on_click(cx.listener(|dialog, _, _, cx| dialog.close(cx))), - ), - ) - .child( - h_flex() - .gap_2() - .px_4() - .pt_4() - .child( - tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { - dialog.set_mode(CollabDialogMode::Share, cx); - })), - ) - .child( - tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { - dialog.set_mode(CollabDialogMode::Join, cx); - dialog.focus(window, cx); - })), - ), - ) - .child( - div() - .flex_1() - .overflow_y_scrollbar() - .p_4() - .child(match self.mode { - CollabDialogMode::Share => self.render_share_pane(phase.as_ref(), cx), - CollabDialogMode::Join => self.render_join_pane(cx), - }), - ), + h_flex() + .gap_2() + .child(tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { + dialog.set_mode(CollabDialogMode::Share, cx); + }))) + .child(tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { + dialog.set_mode(CollabDialogMode::Join, cx); + dialog.focus(window, cx); + }))), ) - } -} - -fn tab_button(id: &'static str, label: &'static str, selected: bool) -> Button { - let button = Button::new(id).label(label).small(); - if selected { button.primary() } else { button.outline() } -} - -fn section_title(title: &'static str, cx: &App) -> impl IntoElement { - div() - .text_lg() - .font_weight(gpui::FontWeight::SEMIBOLD) - .text_color(cx.theme().foreground) - .child(title) -} - -fn helper_text(text: &str, cx: &App) -> impl IntoElement { - div() - .text_sm() - .line_height(relative(1.45)) - .text_color(cx.theme().muted_foreground) - .child(text.to_string()) -} - -fn ticket_box(ticket: Option, loading: bool, cx: &App) -> impl IntoElement { - div() - .w_full() - .min_h(px(84.0)) - .max_h(px(132.0)) - .overflow_y_scrollbar() - .rounded_md() - .border_1() - .border_color(cx.theme().border) - .bg(cx.theme().background) - .p_3() - .text_xs() - .line_height(relative(1.35)) - .text_color(cx.theme().foreground) - .child(if loading { - SharedString::from("Minting a fresh invite...") - } else { - ticket.unwrap_or_else(|| "Start sharing to mint an invite.".into()) - }) -} - -fn error_text(text: SharedString, cx: &App) -> impl IntoElement { - div().text_sm().text_color(cx.theme().danger).child(text) -} - -fn success_text(text: SharedString, cx: &App) -> impl IntoElement { - div().text_sm().text_color(cx.theme().success).child(text) -} - -fn roster_list(entries: Vec, cx: &App) -> AnyElement { - let mut list = v_flex() - .gap_2() - .rounded_md() - .border_1() - .border_color(cx.theme().border) - .bg(cx.theme().background) - .p_3() - .child( - div() - .text_xs() - .text_color(cx.theme().muted_foreground) - .child("Participants"), - ); - - if entries.is_empty() { - return list .child( div() - .text_sm() - .text_color(cx.theme().muted_foreground) - .child("Presence is starting..."), + .flex_1() + .overflow_y_scrollbar() + .child(match self.mode { + CollabDialogMode::Share => self.render_share_pane(phase.as_ref(), cx), + CollabDialogMode::Join => self.render_join_pane(cx), + }), ) - .into_any_element(); - } - - for entry in entries { - let label = if entry.is_self { format!("{} (you)", entry.name) } else { entry.name }; - list = list.child( - h_flex() - .items_center() - .gap_2() - .child(div().size(px(8.0)).rounded_full().bg(rgb(entry.color_rgb))) - .child( - div() - .text_sm() - .text_color(cx.theme().foreground) - .child(label), - ), - ); - } - list.into_any_element() -} - -fn connectivity_text(phase: &SessionPhase) -> String { - match phase { - SessionPhase::Creating => "Starting collaboration...".to_string(), - SessionPhase::Joining(_) => "Joining collaboration...".to_string(), - SessionPhase::Attached(attachment) => match &attachment.connectivity { - Connectivity::Online if attachment.peers_present == 0 => "Only you - share the invite.".to_string(), - Connectivity::Online => format!("{} people in session.", attachment.peers_present + 1), - Connectivity::Offline { .. } => "Offline - reconnecting. Changes will sync when connected.".to_string(), - }, - SessionPhase::Detached(_) => "This document is local.".to_string(), } } diff --git a/crates/flowstate/src/collab/share_dialog_view.rs b/crates/flowstate/src/collab/share_dialog_view.rs new file mode 100644 index 00000000..4852d24d --- /dev/null +++ b/crates/flowstate/src/collab/share_dialog_view.rs @@ -0,0 +1,158 @@ +use gpui::{AnyElement, App, Context, IntoElement, ParentElement, SharedString, div, prelude::*, px, relative, rgb}; +use gpui_component::{ + ActiveTheme as _, Disableable, Sizable, + avatar::Avatar, + button::{Button, ButtonVariants as _}, + clipboard::Clipboard, + h_flex, + scroll::ScrollableElement, + v_flex, +}; + +use super::CollabShareDialog; +use crate::collab::{Connectivity, SessionPhase}; + +pub(super) fn tab_button(id: &'static str, label: &'static str, selected: bool) -> Button { + let button = Button::new(id).label(label).small(); + if selected { button.primary() } else { button.outline() } +} + +pub(super) fn section_title(title: &'static str, cx: &App) -> impl IntoElement { + div() + .text_lg() + .font_weight(gpui::FontWeight::SEMIBOLD) + .text_color(cx.theme().foreground) + .child(title) +} + +pub(super) fn helper_text(text: &str, cx: &App) -> impl IntoElement { + div() + .text_sm() + .line_height(relative(1.45)) + .text_color(cx.theme().muted_foreground) + .child(text.to_string()) +} + +pub(super) fn ticket_box(ticket: Option, loading: bool, cx: &mut Context) -> AnyElement { + h_flex() + .items_start() + .gap_2() + .rounded_md() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().background) + .p_3() + .child( + div() + .flex_1() + .min_h(px(84.0)) + .max_h(px(132.0)) + .overflow_y_scrollbar() + .text_xs() + .line_height(relative(1.35)) + .text_color(cx.theme().foreground) + .child(if loading { + SharedString::from("Minting a fresh invite...") + } else { + ticket.clone().unwrap_or_else(|| "Start sharing to mint an invite.".into()) + }), + ) + .child(copy_invite_button(ticket, loading, cx)) + .into_any_element() +} + +pub(super) fn error_text(text: SharedString, cx: &App) -> impl IntoElement { + div().text_sm().text_color(cx.theme().danger).child(text) +} + +pub(super) fn progress_text(text: SharedString, cx: &App) -> impl IntoElement { + div().text_sm().text_color(cx.theme().info).child(text) +} + +pub(super) fn success_text(text: SharedString, cx: &App) -> impl IntoElement { + div().text_sm().text_color(cx.theme().success).child(text) +} + +pub(super) fn roster_list(entries: Vec, cx: &App) -> AnyElement { + let mut list = v_flex() + .gap_2() + .rounded_md() + .border_1() + .border_color(cx.theme().border) + .bg(cx.theme().background) + .p_3() + .child( + div() + .text_xs() + .text_color(cx.theme().muted_foreground) + .child("Participants"), + ); + + if entries.is_empty() { + return list + .child( + div() + .text_sm() + .text_color(cx.theme().muted_foreground) + .child("Presence is starting..."), + ) + .into_any_element(); + } + + for entry in entries { + let name = entry.name; + let label = if entry.is_self { format!("{name} (you)") } else { name.clone() }; + let color = gpui::Hsla::from(rgb(entry.color_rgb)); + list = list.child( + h_flex() + .items_center() + .gap_2() + .child( + Avatar::new() + .name(name) + .small() + .bg(color.opacity(0.14)) + .text_color(color) + .border_color(color.opacity(0.32)), + ) + .child(div().size(px(8.0)).rounded_full().bg(color)) + .child(div().text_sm().text_color(cx.theme().foreground).child(label)), + ); + } + list.into_any_element() +} + +pub(super) fn connectivity_text(phase: &SessionPhase) -> String { + match phase { + SessionPhase::Creating => "Starting collaboration...".to_string(), + SessionPhase::Joining(_) => "Joining collaboration...".to_string(), + SessionPhase::Attached(attachment) => match &attachment.connectivity { + Connectivity::Online if attachment.peers_present == 0 => "Only you - share the invite.".to_string(), + Connectivity::Online => format!("{} people in session.", attachment.peers_present + 1), + Connectivity::Offline { .. } => "Offline - reconnecting. Changes will sync when connected.".to_string(), + }, + SessionPhase::Detached(_) => "This document is local.".to_string(), + } +} + +fn copy_invite_button(ticket: Option, loading: bool, cx: &mut Context) -> AnyElement { + let Some(ticket) = ticket.filter(|_| !loading) else { + return Button::new("copy-collaboration-invite-disabled") + .label("Copy") + .small() + .disabled(true) + .into_any_element(); + }; + let dialog = cx.entity().downgrade(); + h_flex() + .items_center() + .gap_1() + .child(Clipboard::new("copy-collaboration-invite").value(ticket).on_copied(move |_, _, cx| { + let _ = dialog.update(cx, |dialog, cx| { + dialog.copy_notice = Some("Invite copied to clipboard.".into()); + cx.notify(); + }); + })) + .child(div().text_xs().text_color(cx.theme().muted_foreground).child("Copy")) + .into_any_element() +} diff --git a/crates/flowstate/src/collab/shutdown.rs b/crates/flowstate/src/collab/shutdown.rs index 6db912e0..8dd1923f 100644 --- a/crates/flowstate/src/collab/shutdown.rs +++ b/crates/flowstate/src/collab/shutdown.rs @@ -1,20 +1,7 @@ -use flowstate_collab::net::NetCommand; use gpui::BorrowAppContext; use super::CollabManager; -impl CollabManager { - pub(super) fn shutdown_runtime(&mut self) { - if let Some(runtime) = self.runtime.take() { - let _ = runtime.commands.try_send(NetCommand::Shutdown); - } - self.sessions_by_id.clear(); - self.session_by_panel.clear(); - self.event_pump_started = false; - self.endpoint_online = false; - } -} - pub fn shutdown(cx: &mut C) where C: BorrowAppContext, diff --git a/crates/flowstate/src/collab/status.rs b/crates/flowstate/src/collab/status.rs index 7a6201ee..b02d2a8a 100644 --- a/crates/flowstate/src/collab/status.rs +++ b/crates/flowstate/src/collab/status.rs @@ -54,13 +54,13 @@ pub fn tab_badge(phase: &SessionPhase, cx: &App) -> Option { fn status_label_and_color(phase: &SessionPhase, cx: &App) -> (String, gpui::Hsla) { match phase { SessionPhase::Creating => ("Starting share".to_string(), cx.theme().info), - SessionPhase::Joining(stage) => (join_label(stage), cx.theme().info), + SessionPhase::Joining(stage) => (join_stage_label(stage), cx.theme().info), SessionPhase::Attached(attachment) => attached_label_and_color(attachment, cx), SessionPhase::Detached(_) => ("Not shared".to_string(), cx.theme().muted_foreground), } } -fn join_label(stage: &JoinStage) -> String { +pub fn join_stage_label(stage: &JoinStage) -> String { match stage { JoinStage::Resolving => "Resolving invite".to_string(), JoinStage::Subscribing => "Joining session".to_string(), diff --git a/crates/flowstate/src/workspace/icons.rs b/crates/flowstate/src/workspace/icons.rs index 49fa963d..4ca37537 100644 --- a/crates/flowstate/src/workspace/icons.rs +++ b/crates/flowstate/src/workspace/icons.rs @@ -2,7 +2,9 @@ use gpui::ParentElement; use gpui_component::button::Button; use gpui_component::button::ButtonVariants as _; -use gpui_component::{Icon, IconName, Sizable as _}; +use gpui_component::{Icon, Sizable as _}; +#[cfg(not(target_os = "macos"))] +use gpui_component::IconName; #[derive(Clone, Copy)] pub enum AppIcon { diff --git a/crates/flowstate/src/workspace/workspace/collab.rs b/crates/flowstate/src/workspace/workspace/collab.rs index e288eda0..1bcae9ce 100644 --- a/crates/flowstate/src/workspace/workspace/collab.rs +++ b/crates/flowstate/src/workspace/workspace/collab.rs @@ -19,9 +19,26 @@ impl Workspace { window: &mut Window, cx: &mut Context, ) { + if self.collaboration_dialog.is_some() { + window.close_dialog(cx); + self.collaboration_dialog = None; + } let workspace = cx.entity().downgrade(); let panel_id = self.active_editor.as_ref().and(self.active_document_id); let dialog = cx.new(|cx| crate::collab::share_dialog::CollabShareDialog::new(workspace, panel_id, mode, window, cx)); + let dialog_for_render = dialog.clone(); + let workspace_for_close = cx.entity().downgrade(); + window.open_dialog(cx, move |component_dialog, _, _| { + let workspace_for_close = workspace_for_close.clone(); + component_dialog + .title("Share / Collaborate") + .w(px(620.0)) + .max_w(px(620.0)) + .on_close(move |_, _, cx| { + let _ = workspace_for_close.update(cx, |workspace, cx| workspace.close_collaboration_dialog(cx)); + }) + .child(dialog_for_render.clone()) + }); dialog.update(cx, |dialog, cx| dialog.focus(window, cx)); self.collaboration_dialog = Some(dialog); cx.notify(); @@ -47,7 +64,7 @@ impl Workspace { match crate::collab::start_session_for_panel(panel_id, editor, title, cx) { Ok(session) => Some(session), Err(error) => { - eprintln!("starting collaboration session failed: {error:#}"); + tracing::warn!("starting collaboration session failed: {error:#}"); None }, } @@ -169,7 +186,6 @@ impl Workspace { &[PromptButton::ok("Leave"), PromptButton::cancel("Cancel")], cx, ); - let window_handle = window.window_handle(); cx.spawn(async move |workspace, cx| { if !matches!(answer.await, Ok(0)) { return; @@ -177,15 +193,6 @@ impl Workspace { let _ = workspace.update(cx, |workspace, cx| { workspace.leave_collaboration_on_panel(panel_id, cx); }); - let _ = window_handle.update(cx, |_, window, cx| { - std::mem::drop(window.prompt( - PromptLevel::Info, - "Left session", - Some("Left session - this copy is now local."), - &[PromptButton::ok("Ok")], - cx, - )); - }); }) .detach(); true @@ -204,7 +211,14 @@ impl Workspace { let request = match crate::collab::join_session(ticket, cx) { Ok(request) => request, Err(error) => { - eprintln!("joining collaboration session failed: {error:#}"); + let detail = format!("Joining collaboration session failed: {error:#}"); + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Join failed", + Some(&detail), + &[PromptButton::ok("Ok")], + cx, + )); return None; }, }; @@ -230,6 +244,9 @@ impl Workspace { &[PromptButton::ok("Ok")], cx, )); + } else if workspace.collaboration_dialog.is_some() { + workspace.close_collaboration_dialog(cx); + window.close_dialog(cx); } }, Ok(Err(error)) => { diff --git a/crates/flowstate/src/workspace/workspace/collab_prompts.rs b/crates/flowstate/src/workspace/workspace/collab_prompts.rs index 14960b67..b1dd6b2f 100644 --- a/crates/flowstate/src/workspace/workspace/collab_prompts.rs +++ b/crates/flowstate/src/workspace/workspace/collab_prompts.rs @@ -132,7 +132,6 @@ impl Workspace { let _ = window_handle.update(cx, |_, window, cx| { let _ = workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); - crate::collab::shutdown(cx); window.remove_window(); }); }) diff --git a/crates/flowstate/src/workspace/workspace/documents.rs b/crates/flowstate/src/workspace/workspace/documents.rs index a7879e2c..a27d3cc0 100644 --- a/crates/flowstate/src/workspace/workspace/documents.rs +++ b/crates/flowstate/src/workspace/workspace/documents.rs @@ -90,6 +90,8 @@ impl Workspace { autosave_document_generations: HashMap::new(), autosave_flow_in_flight: HashSet::new(), collaboration_dialog: None, + collab_notice_subscriptions: HashMap::new(), + collab_incompatible_version_notices: HashSet::new(), file_search_overlay: None, tub_root: None, tub_index: None, diff --git a/crates/flowstate/src/workspace/workspace/mod.rs b/crates/flowstate/src/workspace/workspace/mod.rs index 6b27bbbc..0d916083 100644 --- a/crates/flowstate/src/workspace/workspace/mod.rs +++ b/crates/flowstate/src/workspace/workspace/mod.rs @@ -31,7 +31,7 @@ use gpui_component::tab::{Tab, TabBar}; use gpui_component::tree::{TreeItem, TreeState, tree}; use gpui_component::{ ActiveTheme as _, Colorize as _, Disableable, Icon, IconName, PixelsExt, Root, Selectable, Sizable, Theme, ThemeRegistry, TitleBar, - VirtualListScrollHandle, h_flex, v_flex, + VirtualListScrollHandle, WindowExt as _, h_flex, v_flex, }; use uuid::Uuid; @@ -104,6 +104,8 @@ pub struct Workspace { autosave_document_generations: HashMap, autosave_flow_in_flight: HashSet, collaboration_dialog: Option>, + collab_notice_subscriptions: HashMap, + collab_incompatible_version_notices: HashSet, file_search_overlay: Option>, tub_root: Option, tub_index: Option>, diff --git a/crates/flowstate/src/workspace/workspace/render_status.rs b/crates/flowstate/src/workspace/workspace/render_status.rs index 30f27a3e..9a444819 100644 --- a/crates/flowstate/src/workspace/workspace/render_status.rs +++ b/crates/flowstate/src/workspace/workspace/render_status.rs @@ -232,6 +232,8 @@ impl Workspace { } fn render_status_bar(&mut self, window: &mut Window, cx: &mut Context) -> impl IntoElement { + self.sync_collab_notice_subscriptions(window, cx); + let speech_word_count = if let (Some(document_id), Some(editor)) = (self.active_document_id, self.active_editor.as_ref()) { let editor = editor.read(cx); let generation = editor.edit_generation(); @@ -305,6 +307,43 @@ impl Workspace { ) }) } + + fn sync_collab_notice_subscriptions(&mut self, window: &mut Window, cx: &mut Context) { + let mut active_sessions = HashSet::new(); + let window_handle = window.window_handle(); + + for panel in &self.document_panels { + let panel_id = panel.read(cx).id(); + let Some(session) = crate::collab::session_for_panel(panel_id, cx) else { + continue; + }; + let session_id = session.read(cx).session_id(); + active_sessions.insert(session_id); + if self.collab_notice_subscriptions.contains_key(&session_id) { + continue; + } + + let notice_window_handle = window_handle; + let subscription = cx.subscribe(&session, move |workspace, _, notice: &crate::collab::SessionNotice, cx| { + let notice = notice.clone(); + if let crate::collab::SessionNotice::IncompatibleVersion(peer) = ¬ice + && !workspace.collab_incompatible_version_notices.insert(peer.clone()) + { + return; + } + if let Err(error) = notice_window_handle.update(cx, |_, window, cx| { + crate::collab::notify::show_session_notice(¬ice, window, cx); + }) { + tracing::warn!("showing collaboration notification failed: {error}"); + } + }); + self.collab_notice_subscriptions.insert(session_id, subscription); + } + + self + .collab_notice_subscriptions + .retain(|session_id, _| active_sessions.contains(session_id)); + } } fn speech_word_count(document: &Document) -> usize { diff --git a/crates/flowstate/src/workspace/workspace/render_top_bar.rs b/crates/flowstate/src/workspace/workspace/render_top_bar.rs index c4a50afb..a39caaff 100644 --- a/crates/flowstate/src/workspace/workspace/render_top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/render_top_bar.rs @@ -20,7 +20,7 @@ impl Workspace { .child(file_top_bar_button(self.active_document_id.is_some(), cx)) .child(insert_top_bar_button(cx, self.active_editor.is_some())) .child(document_top_bar_button(cx)) - .child(collaboration_top_bar_button(cx, self.active_document_id.is_some())) + .child(collaboration_top_bar_button(cx, self.active_document_id.is_some(), active_collaborating)) .child(view_top_bar_button( cx, !self.outline_collapsed, diff --git a/crates/flowstate/src/workspace/workspace/top_bar.rs b/crates/flowstate/src/workspace/workspace/top_bar.rs index c24f701e..c189aabb 100644 --- a/crates/flowstate/src/workspace/workspace/top_bar.rs +++ b/crates/flowstate/src/workspace/workspace/top_bar.rs @@ -106,7 +106,7 @@ fn document_top_bar_button(cx: &mut Context) -> impl IntoElement { ) } -fn collaboration_top_bar_button(cx: &mut Context, has_document: bool) -> impl IntoElement { +fn collaboration_top_bar_button(cx: &mut Context, has_document: bool, active_collaborating: bool) -> impl IntoElement { let workspace = cx.entity().downgrade(); div() .h_full() @@ -135,7 +135,7 @@ fn collaboration_top_bar_button(cx: &mut Context, has_document: bool) .item(file_menu_item(workspace.clone(), "Join from Clipboard", false, |workspace, window, cx| { workspace.join_collaboration_from_clipboard(window, cx); })) - .item(file_menu_item(workspace.clone(), "Leave Shared Session", !has_document, |workspace, window, cx| { + .item(file_menu_item(workspace.clone(), "Leave Shared Session", !active_collaborating, |workspace, window, cx| { workspace.confirm_leave_collaboration_on_active_document(window, cx); })) }), diff --git a/crates/flowstate/src/workspace/workspace/traits.rs b/crates/flowstate/src/workspace/workspace/traits.rs index 1c61fdcc..62b1a21a 100644 --- a/crates/flowstate/src/workspace/workspace/traits.rs +++ b/crates/flowstate/src/workspace/workspace/traits.rs @@ -45,7 +45,6 @@ impl Render for Workspace { this.child(self.render_settings_overlay(overlay, cx)) }) .when_some(self.file_search_overlay.clone(), |this, overlay| this.child(overlay)) - .when_some(self.collaboration_dialog.clone(), |this, dialog| this.child(dialog)) .when_some(self.outline_context_menu.as_ref(), |this, ctx| { let workspace = cx.entity().downgrade(); let menu = ctx.menu_view.clone(); @@ -75,5 +74,7 @@ impl Render for Workspace { .with_priority(1), ) }) + .when_some(Root::render_dialog_layer(window, cx), |this, layer| this.child(layer)) + .when_some(Root::render_notification_layer(window, cx), |this, layer| this.child(layer)) } } diff --git a/crates/flowstate/src/workspace/workspace/window.rs b/crates/flowstate/src/workspace/workspace/window.rs index eae10e7d..af19175a 100644 --- a/crates/flowstate/src/workspace/workspace/window.rs +++ b/crates/flowstate/src/workspace/workspace/window.rs @@ -83,7 +83,6 @@ pub fn install_workspace_close_prompt(workspace: Entity, window: &mut allow_close.set(true); let _ = window_handle.update(cx, |_, window, cx| { workspace.update(cx, |workspace, cx| workspace.leave_all_collaboration_sessions(cx)); - crate::collab::shutdown(cx); window.remove_window(); }); } diff --git a/crates/gpui-flowtext/src/document/blocks.rs b/crates/gpui-flowtext/src/document/blocks.rs index f26c0997..1a67c8f3 100644 --- a/crates/gpui-flowtext/src/document/blocks.rs +++ b/crates/gpui-flowtext/src/document/blocks.rs @@ -28,6 +28,16 @@ pub const IMAGE_LOADING_PLACEHOLDER_WIDTH_PX: f32 = 240.0; pub const IMAGE_LOADING_PLACEHOLDER_HEIGHT_PX: f32 = 160.0; impl AssetRecord { + #[must_use] + pub fn stable_content_hash(bytes: &[u8]) -> u64 { + const FNV_OFFSET_BASIS: u64 = 0xcbf2_9ce4_8422_2325; + const FNV_PRIME: u64 = 0x0000_0100_0000_01b3; + + bytes.iter().fold(FNV_OFFSET_BASIS, |hash, byte| { + (hash ^ u64::from(*byte)).wrapping_mul(FNV_PRIME) + }) + } + #[must_use] pub fn is_loading_placeholder(&self) -> bool { self.bytes.is_empty() diff --git a/crates/gpui-flowtext/src/persistence/records.rs b/crates/gpui-flowtext/src/persistence/records.rs index a4c0df79..de392ba8 100644 --- a/crates/gpui-flowtext/src/persistence/records.rs +++ b/crates/gpui-flowtext/src/persistence/records.rs @@ -674,6 +674,11 @@ fn read_asset_record(cursor: &mut Cursor<&[u8]>) -> io::Result { let content_hash = read_u64(cursor)?; let byte_len = read_len(cursor, "native document asset byte length")?; let bytes = read_bytes(cursor, byte_len, "native document asset bytes")?.to_vec(); + let content_hash = if bytes.is_empty() { + content_hash + } else { + AssetRecord::stable_content_hash(&bytes) + }; Ok(AssetRecord { id, mime_type, diff --git a/crates/gpui-flowtext/src/persistence/validation.rs b/crates/gpui-flowtext/src/persistence/validation.rs index 60a91db1..b5b7cbdb 100644 --- a/crates/gpui-flowtext/src/persistence/validation.rs +++ b/crates/gpui-flowtext/src/persistence/validation.rs @@ -161,12 +161,11 @@ fn validate_image_payload(image: &ImageBlock, document: &Document) -> io::Result { return Err(io::Error::new(io::ErrorKind::InvalidData, "image caption run lengths are invalid")); } - if let Some(asset) = document.assets.assets.get(&image.asset_id) { - let mut hasher = DefaultHasher::new(); - asset.bytes.hash(&mut hasher); - if asset.content_hash != hasher.finish() { - return Err(io::Error::new(io::ErrorKind::InvalidData, "image asset content hash mismatch")); - } + if let Some(asset) = document.assets.assets.get(&image.asset_id) + && !asset.is_loading_placeholder() + && asset.content_hash != AssetRecord::stable_content_hash(asset.bytes.as_ref()) + { + return Err(io::Error::new(io::ErrorKind::InvalidData, "image asset content hash mismatch")); } Ok(()) } diff --git a/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs b/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs index 0f4823af..543c3c3f 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs @@ -77,7 +77,8 @@ impl RichTextEditor { } fn insert_ordered_block_fragment_after_caret(&mut self, input_blocks: &[InputBlock], cx: &mut Context) { - let insert_ix = self.prepare_block_insertion_index(cx); + let remove_empty_caret_paragraph = !input_blocks.iter().any(|block| matches!(block, InputBlock::Paragraph(_))); + let insert_ix = self.prepare_block_insertion_index(remove_empty_caret_paragraph, cx); let insert_paragraph_ix = self .document .blocks @@ -164,7 +165,8 @@ impl RichTextEditor { if blocks.is_empty() { return; } - let insert_ix = self.prepare_block_insertion_index(cx); + let remove_empty_caret_paragraph = !blocks.iter().any(|block| matches!(block, Block::Paragraph(_))); + let insert_ix = self.prepare_block_insertion_index(remove_empty_caret_paragraph, cx); let inserted_count = blocks.len(); Arc::make_mut(&mut self.document.blocks).splice(insert_ix..insert_ix, blocks); for relative_ix in 0..inserted_count { @@ -178,7 +180,7 @@ impl RichTextEditor { self.paragraph_height_cache_revision = self.paragraph_height_cache_revision.wrapping_add(1); } - fn prepare_block_insertion_index(&mut self, cx: &mut Context) -> usize { + fn prepare_block_insertion_index(&mut self, remove_empty_caret_paragraph: bool, cx: &mut Context) -> usize { if let Some( BlockSelection::Image(block_ix) | BlockSelection::Equation(block_ix) @@ -189,7 +191,9 @@ impl RichTextEditor { return (block_ix + 1).min(self.document.blocks.len()); } - if let Some(insert_ix) = self.remove_empty_caret_paragraph_for_block_insertion(cx) { + if remove_empty_caret_paragraph + && let Some(insert_ix) = self.remove_empty_caret_paragraph_for_block_insertion(cx) + { return insert_ix; } diff --git a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs index e288ab42..885f25e3 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs @@ -21,12 +21,17 @@ impl RichTextEditor { } pub fn collab_apply_deferred(&self) -> bool { - self.selecting || self.active_text_drag.is_some() || self.image_resize_drag.is_some() || self.table_column_resize_drag.is_some() + self.selecting + || self.active_text_drag.is_some() + || self.image_resize_drag.is_some() + || self.table_column_resize_drag.is_some() + || self.ime_composition_active() } fn apply_one_collab_patch(&mut self, patch: &CollabPatch, invalidation: &mut Option>) { match patch { CollabPatch::ParagraphText { row, new, delta_utf8 } => { + self.remap_object_text_selection_for_delta(*row, delta_utf8); if let Some(paragraph_ix) = self.paragraph_ix_for_block(*row) { remap_selection_for_text_delta(&mut self.selection, paragraph_ix, delta_utf8); replace_paragraph_content(&mut self.document, paragraph_ix, new); @@ -110,6 +115,20 @@ impl RichTextEditor { self.equation_source_anchor = 0; self.equation_source_caret = 0; } + + fn remap_object_text_selection_for_delta(&mut self, row: usize, delta: &[CollabTextDelta]) { + match self.selected_block { + Some(BlockSelection::TableCell { block_ix, .. }) if block_ix == row => { + self.table_cell_anchor = remap_byte(self.table_cell_anchor, delta); + self.table_cell_caret = remap_byte(self.table_cell_caret, delta); + }, + Some(BlockSelection::Equation(block_ix)) if block_ix == row => { + self.equation_source_anchor = remap_byte(self.equation_source_anchor, delta); + self.equation_source_caret = remap_byte(self.equation_source_caret, delta); + }, + Some(BlockSelection::Image(_) | BlockSelection::Equation(_) | BlockSelection::Table(_) | BlockSelection::TableCell { .. }) | None => {}, + } + } } #[hotpath::measure] diff --git a/crates/gpui-flowtext/src/rich_text/editor/commands.rs b/crates/gpui-flowtext/src/rich_text/editor/commands.rs index a2f70730..68ecb2b2 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/commands.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/commands.rs @@ -159,6 +159,7 @@ impl RichTextEditor { if paragraph_ix < self.document.paragraphs.len() { // Outline navigation should place the insertion caret at the start of // the target paragraph, matching what the user just selected in the nav. + let before_selection = self.selection.clone(); self.selection = EditorSelection { anchor: DocumentOffset { paragraph: paragraph_ix, @@ -171,6 +172,9 @@ impl RichTextEditor { }; self.goal_x = None; self.reset_caret_blink(cx); + if self.selection != before_selection { + self.emit_selection_changed(cx); + } let width = self.current_layout_width(); let start = paragraph_ix.saturating_sub(2); @@ -312,6 +316,7 @@ impl RichTextEditor { self.selection = selection; self.goal_x = None; self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/hit_testing.rs b/crates/gpui-flowtext/src/rich_text/editor/hit_testing.rs index f14b9fdf..a748ecef 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/hit_testing.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/hit_testing.rs @@ -110,6 +110,7 @@ impl RichTextEditor { self.goal_x = None; self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs index 147bd3f7..99ae0f6f 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs @@ -75,6 +75,7 @@ impl RichTextEditor { search_highlights: Vec::new(), active_search_highlight: None, last_text_input_at: None, + ime_marked_range: None, pending_typing_prefetch_resume: false, resume_chunk_prefetch_after_typing: false, paragraph_chunk_layout_cache: vec![None; paragraph_count], @@ -190,6 +191,7 @@ impl RichTextEditor { self.caret_blink_active = false; self.external_carets.clear(); self.last_text_input_at = None; + self.ime_marked_range = None; self.pending_typing_prefetch_resume = false; self.resume_chunk_prefetch_after_typing = false; self.paragraph_chunk_layout_cache = Vec::new(); @@ -262,7 +264,7 @@ impl RichTextEditor { self.collab_undo_redirect = hook; } - pub fn clear_collab_history(&mut self) { + pub fn clear_undo_redo_stacks(&mut self) { self.undo_stack.clear(); self.redo_stack.clear(); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/mod.rs b/crates/gpui-flowtext/src/rich_text/editor/mod.rs index 7a05e431..2174d5c3 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mod.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mod.rs @@ -12,9 +12,10 @@ use std::{ use crop::Rope; use gpui::{ - App, Bounds, ClipboardEntry, ClipboardItem, Context, CursorStyle, DragMoveEvent, Entity, ExternalPaths, FocusHandle, Focusable, Image, - ImageFormat, InteractiveElement, IntoElement, KeyDownEvent, MouseButton, MouseDownEvent, MouseMoveEvent, MouseUpEvent, PathPromptOptions, - Pixels, Point, Render, SharedString, Size, Subscription, Task, Timer, Window, actions, div, img, point, prelude::*, px, relative, rgb, size, + App, Bounds, ClipboardEntry, ClipboardItem, Context, CursorStyle, DragMoveEvent, Entity, EntityInputHandler, ExternalPaths, FocusHandle, + Focusable, Image, ImageFormat, InteractiveElement, IntoElement, KeyDownEvent, MouseButton, MouseDownEvent, MouseMoveEvent, MouseUpEvent, + PathPromptOptions, Pixels, Point, Render, SharedString, Size, Subscription, Task, Timer, UTF16Selection, Window, actions, div, img, point, + prelude::*, px, relative, rgb, size, }; use gpui_component::ActiveTheme as _; use gpui_component::scroll::{Scrollbar, ScrollbarHandle, ScrollbarShow}; @@ -907,7 +908,7 @@ pub struct RichTextEditor { autoscroll_active: bool, pub(super) caret_visible: bool, caret_blink_active: bool, - last_text_input_at: Option, + last_text_input_at: Option, ime_marked_range: Option>, external_carets: Vec, pub(super) search_highlights: Vec>, pub(super) active_search_highlight: Option, diff --git a/crates/gpui-flowtext/src/rich_text/editor/mouse.rs b/crates/gpui-flowtext/src/rich_text/editor/mouse.rs index 0e553a7a..7ae56606 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mouse.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mouse.rs @@ -35,6 +35,7 @@ impl RichTextEditor { 2 => SelectionGranularity::Word, _ => SelectionGranularity::Paragraph, }; + let before_selection = self.selection.clone(); self.selection = match self.drag_granularity { SelectionGranularity::Character if event.modifiers.shift => EditorSelection { anchor: self.selection.anchor, @@ -49,6 +50,9 @@ impl RichTextEditor { }; self.drag_anchor = Some(self.selection.anchor); self.reset_caret_blink(cx); + if self.selection != before_selection { + self.emit_selection_changed(cx); + } cx.notify(); } @@ -150,7 +154,11 @@ impl RichTextEditor { source_range: source_range.clone(), fragment: selected_rich_fragment(&self.document, source_range), }); + let before_selection = self.selection.clone(); self.selection = pending_drag.source_selection; + if self.selection != before_selection { + self.emit_selection_changed(cx); + } self.pending_text_drag = None; } if self.active_text_drag.is_some() { @@ -163,6 +171,7 @@ impl RichTextEditor { self.selection = selection; self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); } cx.notify(); return; @@ -196,6 +205,7 @@ impl RichTextEditor { self.selection = selection; self.scroll_head_into_view(); self.reset_caret_blink(cx); + self.emit_selection_changed(cx); cx.notify(); } else { cx.notify(); @@ -230,9 +240,13 @@ impl RichTextEditor { } else if self.pending_text_drag.take().is_some() { self.clear_drop_preview(); let caret = self.hit_test_document_position(event.position, window, cx); + let before_selection = self.selection.clone(); self.selection = EditorSelection { anchor: caret, head: caret }; self.scroll_head_into_view(); self.reset_caret_blink(cx); + if self.selection != before_selection { + self.emit_selection_changed(cx); + } cx.notify(); } if self.selecting { @@ -251,13 +265,18 @@ impl RichTextEditor { fn move_rich_text_fragment(&mut self, drag: ActiveTextDrag, drop: DocumentOffset, cx: &mut Context) { if offset_in_range(drop, drag.source_range.clone()) { self.clear_drop_preview(); + let before_selection = self.selection.clone(); self.selection = EditorSelection { anchor: drag.source_range.start, head: drag.source_range.end, }; + if self.selection != before_selection { + self.emit_selection_changed(cx); + } cx.notify(); return; } + let before_document = self.document.clone(); let before_selection = EditorSelection { anchor: drag.source_range.start, head: drag.source_range.end, @@ -267,6 +286,10 @@ impl RichTextEditor { self.next_edit_generation = self.next_edit_generation.wrapping_add(1); let source_range = drag.source_range.clone(); let adjusted_drop = adjust_drop_after_source_delete(drop, source_range.clone()); + let capture_start = source_range.start.paragraph.min(drop.paragraph); + let capture_end = source_range.end.paragraph.max(drop.paragraph).saturating_add(1); + let before_span = capture_document_span(&before_document, capture_start..capture_end); + let start_paragraph = self.identity_map.paragraph_id(before_span.start_paragraph); self.selection = before_selection.clone(); self.delete_selection_internal(); let inserted_start = adjusted_drop; @@ -275,16 +298,24 @@ impl RichTextEditor { anchor: inserted_end, head: inserted_end, }; + self.emit_selection_changed(cx); + let paragraph_delta = self.document.paragraphs.len() as isize - before_document.paragraphs.len() as isize; + let after_count = before_span + .paragraphs + .len() + .saturating_add_signed(paragraph_delta) + .min( + self + .document + .paragraphs + .len() + .saturating_sub(before_span.start_paragraph), + ); + let after_span = capture_document_span(&self.document, before_span.start_paragraph..before_span.start_paragraph + after_count); let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: self.identity_map.paragraph_id(inserted_start.paragraph), - before: capture_document_span( - &self.document, - inserted_start.paragraph..(inserted_start.paragraph + 1).min(self.document.paragraphs.len()), - ), - after: capture_document_span( - &self.document, - inserted_start.paragraph..(inserted_end.paragraph + 1).min(self.document.paragraphs.len()), - ), + start_paragraph, + before: before_span, + after: after_span, }]; self.undo_stack.push(EditRecord { before_selection, @@ -434,6 +465,7 @@ impl RichTextEditor { && editor.selection.head != head { editor.selection.head = head; + editor.emit_selection_changed(cx); } cx.notify(); true diff --git a/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs b/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs index cc0ef168..a32b114e 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/object_assets.rs @@ -82,14 +82,13 @@ fn image_asset_from_path(path: &Path) -> Option<(AssetRecord, SharedString)> { .file_name() .map(|name| name.to_string_lossy().to_string()); let alt_text: SharedString = original_name.clone().unwrap_or_default().into(); - let mut hasher = DefaultHasher::new(); - bytes.hash(&mut hasher); + let content_hash = AssetRecord::stable_content_hash(&bytes); Some(( AssetRecord { id: AssetId(uuid::Uuid::new_v4().as_u128()), mime_type: format.mime_type().into(), original_name: original_name.map(Into::into), - content_hash: hasher.finish(), + content_hash, bytes: Arc::new(bytes), }, alt_text, @@ -99,14 +98,13 @@ fn image_asset_from_path(path: &Path) -> Option<(AssetRecord, SharedString)> { #[hotpath::measure] fn image_asset_from_image(image: Image) -> (AssetRecord, SharedString) { let asset_id = AssetId(uuid::Uuid::new_v4().as_u128()); - let mut hasher = DefaultHasher::new(); - image.bytes.hash(&mut hasher); + let content_hash = AssetRecord::stable_content_hash(&image.bytes); ( AssetRecord { id: asset_id, mime_type: image.format.mime_type().into(), original_name: None, - content_hash: hasher.finish(), + content_hash, bytes: Arc::new(image.bytes), }, "Pasted image".into(), diff --git a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs index d29c9dd9..17793784 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs @@ -485,6 +485,7 @@ impl RichTextEditor { if block_ix >= self.document.blocks.len() { return false; } + let block_id = self.identity_map.block_id(block_ix); let blocks = Arc::make_mut(&mut self.document.blocks); if matches!(blocks.get(block_ix), Some(Block::Paragraph(_))) { return false; @@ -495,9 +496,10 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::DeleteBlock { - block: self.identity_map.block_id(block_ix).unwrap_or(BlockId(0)), - }]; + let canonical_operations = block_id.map_or_else( + || vec![CanonicalOperation::ReplaceDocument], + |block| vec![CanonicalOperation::DeleteBlock { block }], + ); self.undo_stack.push(EditRecord { before_selection: before_selection.clone(), before_generation, diff --git a/crates/gpui-flowtext/src/rich_text/editor/platform.rs b/crates/gpui-flowtext/src/rich_text/editor/platform.rs index e83c2827..4d33cf68 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/platform.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/platform.rs @@ -19,3 +19,193 @@ fn windows_apply_capslock(text: &str) -> String { } } +#[hotpath::measure_all] +impl RichTextEditor { + pub fn ime_composition_active(&self) -> bool { + self.ime_marked_range.is_some() + } + + fn selection_utf16_range(&self) -> Range { + let text = full_document_text(&self.document); + let anchor = byte_to_utf16( + &text, + global_byte(&self.document, clamp_offset_to_document(&self.document, self.selection.anchor)), + ); + let head = byte_to_utf16( + &text, + global_byte(&self.document, clamp_offset_to_document(&self.document, self.selection.head)), + ); + anchor.min(head)..anchor.max(head) + } + + fn utf16_range_to_document_offsets(&self, range: Range) -> Range { + let text = full_document_text(&self.document); + let start = utf16_to_byte(&text, range.start); + let end = utf16_to_byte(&text, range.end).max(start); + global_to_document_offset(&self.document, start)..global_to_document_offset(&self.document, end) + } + + fn replace_text_for_utf16_range(&mut self, range: Range, text: &str, cx: &mut Context) { + let offsets = self.utf16_range_to_document_offsets(range); + let selection = EditorSelection { + anchor: offsets.start, + head: offsets.end, + }; + let before_selection = self.selection.clone(); + self.selection = selection; + if text.is_empty() { + if self.selection.is_caret() { + if self.selection != before_selection { + self.emit_selection_changed(cx); + self.reset_caret_blink(cx); + cx.notify(); + } + return; + } + self.apply_document_edit(cx, |editor, cx| { + editor.delete_selection_internal(); + editor.after_text_mutation(cx); + }); + } else { + self.insert_text_command(text, cx); + } + if self.selection != before_selection { + self.emit_selection_changed(cx); + } + } +} + +#[hotpath::measure_all] +impl EntityInputHandler for RichTextEditor { + fn text_for_range( + &mut self, + range: Range, + adjusted_range: &mut Option>, + _: &mut Window, + _: &mut Context, + ) -> Option { + let text = full_document_text(&self.document); + let start = utf16_to_byte(&text, range.start); + let end = utf16_to_byte(&text, range.end).max(start); + *adjusted_range = Some(byte_range_to_utf16(&text, start..end)); + Some(document_text_slice(&self.document, start..end)) + } + + fn selected_text_range(&mut self, _: bool, _: &mut Window, _: &mut Context) -> Option { + Some(UTF16Selection { + range: self.selection_utf16_range(), + reversed: self.selection.anchor > self.selection.head, + }) + } + + fn marked_text_range(&self, _: &mut Window, _: &mut Context) -> Option> { + self.ime_marked_range.clone() + } + + fn unmark_text(&mut self, _: &mut Window, cx: &mut Context) { + if self.ime_marked_range.take().is_some() { + cx.notify(); + } + } + + fn replace_text_in_range(&mut self, range: Option>, text: &str, _: &mut Window, cx: &mut Context) { + let range = range + .or_else(|| self.ime_marked_range.clone()) + .unwrap_or_else(|| self.selection_utf16_range()); + self.ime_marked_range = None; + self.replace_text_for_utf16_range(range, text, cx); + } + + fn replace_and_mark_text_in_range( + &mut self, + range: Option>, + new_text: &str, + new_selected_range: Option>, + _: &mut Window, + cx: &mut Context, + ) { + let range = range + .or_else(|| self.ime_marked_range.clone()) + .unwrap_or_else(|| self.selection_utf16_range()); + let mark_start = range.start; + self.replace_text_for_utf16_range(range, new_text, cx); + if new_text.is_empty() { + self.ime_marked_range = None; + return; + } + let marked_range = mark_start..mark_start + new_text.encode_utf16().count(); + self.ime_marked_range = Some(marked_range.clone()); + let selection_range = new_selected_range + .map(|range| marked_range.start + range.start..marked_range.start + range.end) + .unwrap_or_else(|| marked_range.end..marked_range.end); + let offsets = self.utf16_range_to_document_offsets(selection_range); + let selection = EditorSelection { + anchor: offsets.start, + head: offsets.end, + }; + if self.selection != selection { + self.selection = selection; + self.emit_selection_changed(cx); + self.reset_caret_blink(cx); + cx.notify(); + } + } + + fn bounds_for_range( + &mut self, + range_utf16: Range, + _: Bounds, + _: &mut Window, + _: &mut Context, + ) -> Option> { + let offsets = self.utf16_range_to_document_offsets(range_utf16); + let layout = self.layout_for_offset(offsets.start)?; + let origin = layout.bounds?.origin; + caret_bounds(&layout, offsets.start, origin) + } + + fn character_index_for_point(&mut self, point: Point, window: &mut Window, cx: &mut Context) -> Option { + let offset = self.hit_test_document_position(point, window, cx); + let text = full_document_text(&self.document); + Some(byte_to_utf16( + &text, + global_byte(&self.document, clamp_offset_to_document(&self.document, offset)), + )) + } +} + +#[hotpath::measure] +fn byte_range_to_utf16(text: &str, range: Range) -> Range { + byte_to_utf16(text, range.start)..byte_to_utf16(text, range.end) +} + +#[hotpath::measure] +fn byte_to_utf16(text: &str, byte: usize) -> usize { + let byte = previous_char_boundary(text, byte); + text[..byte].encode_utf16().count() +} + +#[hotpath::measure] +fn utf16_to_byte(text: &str, target: usize) -> usize { + let mut utf16 = 0usize; + for (byte, ch) in text.char_indices() { + if target <= utf16 { + return byte; + } + let next = utf16 + ch.len_utf16(); + if target < next { + return byte; + } + utf16 = next; + } + text.len() +} + +#[hotpath::measure] +fn previous_char_boundary(text: &str, byte: usize) -> usize { + let mut byte = byte.min(text.len()); + while !text.is_char_boundary(byte) { + byte = byte.saturating_sub(1); + } + byte +} diff --git a/crates/gpui-flowtext/src/rich_text/editor/search_highlights.rs b/crates/gpui-flowtext/src/rich_text/editor/search_highlights.rs index fd26b9b3..128b9df0 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/search_highlights.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/search_highlights.rs @@ -29,10 +29,14 @@ impl RichTextEditor { return false; }; + let before_selection = self.selection.clone(); self.selection = EditorSelection { anchor: range.start, head: range.end, }; + if self.selection != before_selection { + self.emit_selection_changed(cx); + } self.apply_document_edit(cx, |editor, cx| { editor.insert_text(replacement, cx); }); diff --git a/crates/gpui-flowtext/src/rich_text/element.rs b/crates/gpui-flowtext/src/rich_text/element.rs index c719585b..d60b3e99 100644 --- a/crates/gpui-flowtext/src/rich_text/element.rs +++ b/crates/gpui-flowtext/src/rich_text/element.rs @@ -1,8 +1,8 @@ use std::{cell::RefCell, rc::Rc}; use gpui::{ - App, AvailableSpace, Background, Bounds, Element, ElementId, Entity, GlobalElementId, InspectorElementId, IntoElement, LayoutId, Pixels, - Style, Window, fill, px, relative, + App, AvailableSpace, Background, Bounds, Element, ElementId, ElementInputHandler, Entity, GlobalElementId, InspectorElementId, IntoElement, + LayoutId, Pixels, Style, Window, fill, px, relative, }; use super::*; @@ -259,6 +259,8 @@ impl Element for VirtualParagraphChunkElement { ) }; if let Some((layout, bounds)) = self.layout.positioned() { + let focus_handle = self.editor.read(cx).focus_handle.clone(); + window.handle_input(&focus_handle, ElementInputHandler::new(bounds, self.editor.clone()), cx); if self.chunk_ix == 0 { let collapse_state = self .editor diff --git a/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs index eb44cefb..55985187 100644 --- a/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs +++ b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs @@ -1,5 +1,12 @@ +use std::{cell::RefCell, rc::Rc}; + use gpui::AppContext as _; +struct SelectionEventRecorder { + selections: Rc>>, + _subscription: gpui::Subscription, +} + #[gpui::test] fn collab_capture_fast_path_emits_single_grapheme_deltas(cx: &mut gpui::TestAppContext) { let editor = cx.update(|cx| cx.new(|cx| RichTextEditor::new_with_path(blank_document(), None, cx))); @@ -57,3 +64,43 @@ fn collab_capture_fast_path_emits_single_grapheme_deltas(cx: &mut gpui::TestAppC }); assert!(edits_after_paste_undo.is_empty()); } + +#[gpui::test] +fn select_all_emits_selection_changed_once(cx: &mut gpui::TestAppContext) { + let document = document_from_input( + DocumentTheme::default(), + vec![InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain("alpha beta")], + }], + ); + let editor = cx.update(|cx| cx.new(|cx| RichTextEditor::new_with_path(document, None, cx))); + let selections = Rc::new(RefCell::new(Vec::new())); + let recorder_selections = selections.clone(); + let _recorder = cx.update(|cx| { + let editor = editor.clone(); + cx.new(|cx| SelectionEventRecorder { + selections: recorder_selections, + _subscription: cx.subscribe(&editor, |recorder: &mut SelectionEventRecorder, _, event: &EditorEvent, _| { + if let EditorEvent::SelectionChanged { selection } = event { + recorder.selections.borrow_mut().push(selection.clone()); + } + }), + }) + }); + + cx.update(|cx| editor.update(cx, |editor, cx| editor.select_all(cx))); + let first_events = selections.borrow(); + assert_eq!(first_events.len(), 1); + assert_eq!( + first_events[0].normalized(), + DocumentOffset { paragraph: 0, byte: 0 }..DocumentOffset { + paragraph: 0, + byte: "alpha beta".len(), + } + ); + drop(first_events); + + cx.update(|cx| editor.update(cx, |editor, cx| editor.select_all(cx))); + assert_eq!(selections.borrow().len(), 1); +} diff --git a/crates/gpui-flowtext/src/rich_text/tests/edit_layout.rs b/crates/gpui-flowtext/src/rich_text/tests/edit_layout.rs index ba997a0e..d999d47b 100644 --- a/crates/gpui-flowtext/src/rich_text/tests/edit_layout.rs +++ b/crates/gpui-flowtext/src/rich_text/tests/edit_layout.rs @@ -1,9 +1,5 @@ use super::*; use gpui::{Bounds, black, point, px, size}; -use std::{ - collections::hash_map::DefaultHasher, - hash::{Hash, Hasher}, -}; #[test] #[hotpath::measure] diff --git a/crates/gpui-flowtext/src/rich_text/tests/persistence_blocks.rs b/crates/gpui-flowtext/src/rich_text/tests/persistence_blocks.rs index eacb1739..fe0879f7 100644 --- a/crates/gpui-flowtext/src/rich_text/tests/persistence_blocks.rs +++ b/crates/gpui-flowtext/src/rich_text/tests/persistence_blocks.rs @@ -133,15 +133,14 @@ fn document_v4_round_trip_preserves_mixed_block_order_and_assets() { ); let asset_id = AssetId(42); let asset_bytes = vec![1, 2, 3, 4]; - let mut hasher = DefaultHasher::new(); - asset_bytes.hash(&mut hasher); + let content_hash = AssetRecord::stable_content_hash(&asset_bytes); document.assets.assets.insert( asset_id, AssetRecord { id: asset_id, mime_type: "image/png".into(), original_name: Some("figure.png".into()), - content_hash: hasher.finish(), + content_hash, bytes: std::sync::Arc::new(asset_bytes), }, ); diff --git a/helpers/docs/collab_qa.md b/helpers/docs/collab_qa.md index 3e22751b..427b9e93 100644 --- a/helpers/docs/collab_qa.md +++ b/helpers/docs/collab_qa.md @@ -57,3 +57,41 @@ Manual QA for P2P `.db8` collaboration. Run with two or three app instances usin 3. Try joining a session already open in the same app; verify it is rejected. 4. Kill the inviter before join; verify join fails without modifying existing documents. 5. Confirm saved files are plain `.db8` snapshots and do not contain CRDT history. + +## Failure Checklist + +| Row | Scenario | Expected Result | +|---|---|---| +| F1 | Garbage or truncated ticket | Inline parse error; no network dial. | +| F2 | Inviter unreachable at join | Join times out with actionable copy. | +| F3 | Inviter reachable but snapshot pull fails | Fallback candidates are tried; all failures end in `JoinFailed`. | +| F4 | Old dead-session ticket | Join times out and copy indicates the session may be over. | +| F5 | Protocol version skew | Join rejects fast or in-session gossip is ignored with a one-time notice. | +| F6 | Lineage violation | Join meta check fails or digest mismatch is ignored; no merge. | +| F7 | Peer crash or kill | Presence ages out within 30 seconds and roster updates. | +| F8 | Transient network loss | Offline pill appears; edits accumulate and resync after reconnect. | +| F9 | Long offline window | Reconnect merges local and remote edits without data loss. | +| F10 | All other peers leave | Session remains attached as `Only you`; no auto-detach. | +| F11 | Gossip receiver lag | Digest pull runs and peers converge. | +| F12 | Oversized blob pull fails | Later anti-entropy recovers the missing update. | +| F13 | Projection drift | Self-check rebuilds from projection and clamps selection. | +| F14 | Remote patch during drag or IME | Patch is deferred, then applied safely. | +| F15 | Asset fetch fails everywhere | Placeholder persists; text sync remains unaffected. | +| F16 | Self-join with own ticket | Friendly error; no dial. | +| F17 | Share on already-attached tab | Attached view opens and can mint a fresh invite. | +| F18 | Close tab or quit while attached | Transactional prompts appear; Cancel aborts the close or quit. | +| F19 | Crash with attached pathless tab | Recovery file opens as a local document on next launch. | +| F20 | Same table cell edited concurrently | Cell converges with documented last-writer-wins behavior. | +| F21 | Peer clock skew | Presence remains sensible; document sync does not depend on wall clock. | + +## Additional Soak Checklist + +1. Simultaneously type in the same paragraph from two peers. +2. Race Enter-split on one peer against typing later in the same paragraph on another peer. +3. Race edits in the same table cell and observe last-writer-wins convergence. +4. Paste an image on each peer and verify asset placeholders resolve. +5. Undo and redo interleaved edits from multiple peers. +6. Leave and rejoin using a fresh ticket. +7. Close a laptop lid or disconnect a peer for 5 minutes, then resync. +8. Cancel close-tab prompts at each step and verify the session stays attached. +9. Run a 30-minute soak with autosave enabled on a path-backed peer. diff --git a/helpers/docs/collaboration.md b/helpers/docs/collaboration.md new file mode 100644 index 00000000..880993bc --- /dev/null +++ b/helpers/docs/collaboration.md @@ -0,0 +1,58 @@ +# Flowstate Collaboration + +Flowstate collaboration lets multiple app instances edit the same `.db8` document over a peer-to-peer session. Saved files remain normal `.db8` files; collaboration history is session state, not file state. + +## User How-To + +1. Open the document you want to share. +2. Choose `Share / Collaborate...`. +3. Select `Start session`, then copy the invite. +4. Send the invite to another Flowstate user. +5. The other user chooses `Join Collaboration Session...`, pastes the invite, and joins into a new tab titled with `(shared)`. +6. Use the Share dialog to copy fresh invites, inspect participants, or leave the session. + +Status labels: + +| Status | Meaning | +|---|---| +| Online | Connected and publishing local edits. | +| Only you | Session is still active, but no other peers are currently present. | +| Offline | Local edits continue and sync should resume after reconnection. | +| Fetching | Joining peer is downloading the current document snapshot. | +| Building | Snapshot download finished and the local shared document is being built. | + +Leaving a session keeps the current tab as a local document. Closing or quitting while attached should prompt before leaving. Pathless attached tabs write recovery snapshots under the temp `flowstate-collab-recovery` directory if the app exits unexpectedly. + +## Maintainer Notes + +Collaboration is split between the GPUI app crate and the GPUI-free collaboration crate: + +| Area | Responsibility | +|---|---| +| `crates/flowstate/src/collab` | Session lifecycle, UI-facing phases, presence, asset pulls, direct request serving, and editor patch application. | +| `crates/flowstate-collab` | Loro schema/projection, local and remote appliers, anti-entropy, tickets, gossip, direct pulls, and convergence tests. | +| `crates/gpui-flowtext` | Document model, editor canonical operations, block assets, persistence, and collab patch application. | + +Data flow: + +1. Local editor mutations emit canonical operations. +2. `LocalApplier` writes those operations into the session `LoroDoc`. +3. Local Loro updates publish over gossip when small, or by blob/direct pull when large. +4. Remote update imports produce `CollabPatch` values via `RemoteApplier`. +5. App sessions apply patches immediately unless the editor is in a deferred state such as drag or IME composition. +6. Digest anti-entropy compares version vectors and pulls missing updates when gossip lags. +7. Image assets travel separately by asset ID and stable content hash; missing assets render placeholders until fetched. + +Operational notes: + +1. Invite tickets contain the session ID, title, protocol version, and inviter endpoint address. +2. Joins first subscribe to the session, then direct-pull a snapshot with byte progress, then build the editor document. +3. Lineage checks reject mismatched session metadata before applying snapshots or digest pulls. +4. Presence uses ephemeral state and should not drive document correctness. +5. Saved `.db8` files intentionally exclude CRDT session history. + +Verification references: + +1. `helpers/docs/collab_qa.md` has the manual QA matrix. +2. `cargo test -p flowstate-collab` covers GPUI-free sync behavior. +3. `cargo test -p flowstate-collab --test convergence` runs the convergence fuzz gate. From 4c244f93a439157d521ef7eed273f349012e11a0 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Mon, 15 Jun 2026 21:54:16 -0500 Subject: [PATCH 12/45] added logging Implemented collaboration-only tracing with non-blocking file logging. What changed: - Added tracing, tracing-subscriber, and tracing-appender. - Added crates/flowstate/src/logging.rs using tracing_appender::non_blocking, so log writing runs on its own worker thread. - Logging is initialized from main and the WorkerGuard is held for app lifetime. - Added structured collaboration logs for: - session create, join, attach, leave, shutdown - gossip subscribe/publish/receive/lag/neighbors - direct snapshot/update/blob/asset pulls and serves - text sync import/export, local edit capture, remote patches - presence publish/apply and peer counts - anti-entropy digest decisions and recovery pulls - asset transfer and validation - offline/reconnect/recovery/self-check drift - relevant join/share workspace errors Defaults: - Logs go to flowstate_data_dir()/logs as daily rolling flowstate.log... files. - Override with FLOWSTATE_LOG_DIR=/path/to/logs. - Default filter is collaboration-only: - flowstate::collab=trace - flowstate::workspace::workspace=trace - flowstate_collab=trace - Override with FLOWSTATE_LOG or RUST_LOG. Validation: - cargo clippy -p flowstate -p flowstate-collab passes. - git diff --check passes. --- Cargo.lock | 4 + Cargo.toml | 3 + crates/flowstate-collab/Cargo.toml | 1 + .../flowstate-collab/src/net/anti_entropy.rs | 28 ++- crates/flowstate-collab/src/net/direct.rs | 187 +++++++++++++-- crates/flowstate-collab/src/net/mod.rs | 39 ++++ crates/flowstate-collab/src/net/runtime.rs | 65 +++++- crates/flowstate-collab/src/net/swarm.rs | 53 ++++- crates/flowstate-collab/src/proto_gossip.rs | 21 ++ crates/flowstate/Cargo.toml | 3 + crates/flowstate/src/collab/asset_transfer.rs | 25 +- crates/flowstate/src/collab/mod.rs | 105 +++++++-- crates/flowstate/src/collab/session.rs | 148 ++++++++++-- crates/flowstate/src/collab/session_io.rs | 220 +++++++++++++++--- .../flowstate/src/collab/session_presence.rs | 39 +++- crates/flowstate/src/collab/session_timers.rs | 65 +++++- crates/flowstate/src/collab/shutdown.rs | 8 +- crates/flowstate/src/lib.rs | 1 + crates/flowstate/src/logging.rs | 65 ++++++ crates/flowstate/src/main.rs | 9 + .../src/workspace/workspace/collab.rs | 26 ++- 21 files changed, 984 insertions(+), 131 deletions(-) create mode 100644 crates/flowstate/src/logging.rs diff --git a/Cargo.lock b/Cargo.lock index 2460d5df..748de41c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2996,6 +2996,9 @@ dependencies = [ "serde_json", "tempfile", "toml 1.1.2+spec-1.1.0", + "tracing", + "tracing-appender", + "tracing-subscriber", "uuid", ] @@ -3016,6 +3019,7 @@ dependencies = [ "rand 0.9.4", "serde", "tokio", + "tracing", "twox-hash", "uuid", ] diff --git a/Cargo.toml b/Cargo.toml index 3ab5cbee..a11e02c2 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -60,6 +60,9 @@ tantivy = "0.26.1" tempfile = "3" tokio = { version = "1", features = ["rt-multi-thread", "macros", "time", "sync"] } toml = "1.1.2" +tracing = "0.1" +tracing-appender = "0.2" +tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] } twox-hash = "2" typst = "0.14" typst-assets = "0.14" diff --git a/crates/flowstate-collab/Cargo.toml b/crates/flowstate-collab/Cargo.toml index 62a7b68d..22261202 100644 --- a/crates/flowstate-collab/Cargo.toml +++ b/crates/flowstate-collab/Cargo.toml @@ -17,6 +17,7 @@ postcard = { workspace = true } rand = { workspace = true } serde = { workspace = true } tokio = { workspace = true } +tracing = { workspace = true } twox-hash = { workspace = true } uuid = { workspace = true } diff --git a/crates/flowstate-collab/src/net/anti_entropy.rs b/crates/flowstate-collab/src/net/anti_entropy.rs index ecb80adf..6f9b7db7 100644 --- a/crates/flowstate-collab/src/net/anti_entropy.rs +++ b/crates/flowstate-collab/src/net/anti_entropy.rs @@ -56,24 +56,30 @@ impl AntiEntropyState { pub fn mark_digest_sent(&mut self, now: Instant) { self.next_digest = next_jittered_deadline(now); + tracing::trace!(session = %self.session, next_digest_in = ?self.next_digest.saturating_duration_since(now), "collaboration anti-entropy digest sent"); } pub fn schedule_immediate_digest(&mut self) { self.next_digest = Instant::now(); + tracing::trace!(session = %self.session, "collaboration anti-entropy digest scheduled immediately"); } pub fn on_neighbor_up(&mut self) { + tracing::debug!(session = %self.session, "collaboration anti-entropy neighbor-up recovery triggered"); self.schedule_immediate_digest(); } pub fn on_reconnect(&mut self) { + tracing::debug!(session = %self.session, "collaboration anti-entropy reconnect recovery triggered"); self.schedule_immediate_digest(); } #[must_use] pub fn on_lagged(&mut self, fallback_peer: Option, our_vv: Vec) -> GapAction { + tracing::warn!(session = %self.session, fallback_peer = ?fallback_peer, our_vv_bytes = our_vv.len(), "collaboration anti-entropy gossip lagged"); self.schedule_immediate_digest(); let Some(from) = fallback_peer else { + tracing::warn!(session = %self.session, "collaboration anti-entropy lagged without fallback peer"); return GapAction::None; }; self.begin_pull(from, our_vv) @@ -81,7 +87,21 @@ impl AntiEntropyState { #[must_use] pub fn consider_digest(&mut self, from: EndpointId, digest_session: SessionId, relation: VersionVectorRelation, our_vv: Vec) -> GapAction { + tracing::trace!( + session = %self.session, + from = %from, + digest_session = %digest_session, + ?relation, + our_vv_bytes = our_vv.len(), + "collaboration anti-entropy digest considered", + ); if digest_session != self.session { + tracing::warn!( + session = %self.session, + from = %from, + digest_session = %digest_session, + "collaboration anti-entropy digest lineage mismatch", + ); return GapAction::LineageMismatch { from, expected: self.session, @@ -91,19 +111,25 @@ impl AntiEntropyState { match relation { VersionVectorRelation::SenderHasMissingOps | VersionVectorRelation::Concurrent => self.begin_pull(from, our_vv), - VersionVectorRelation::Equal | VersionVectorRelation::WeHaveMissingOps => GapAction::None, + VersionVectorRelation::Equal | VersionVectorRelation::WeHaveMissingOps => { + tracing::trace!(session = %self.session, from = %from, ?relation, "collaboration anti-entropy digest needs no pull"); + GapAction::None + }, } } pub fn finish_pull(&mut self) { + tracing::debug!(session = %self.session, was_in_flight = self.pull_in_flight, "collaboration anti-entropy pull finished"); self.pull_in_flight = false; } fn begin_pull(&mut self, from: EndpointId, our_vv: Vec) -> GapAction { if self.pull_in_flight { + tracing::debug!(session = %self.session, from = %from, "collaboration anti-entropy skipped pull because one is already in flight"); return GapAction::None; } self.pull_in_flight = true; + tracing::debug!(session = %self.session, from = %from, our_vv_bytes = our_vv.len(), "collaboration anti-entropy pull started"); GapAction::Pull { from, our_vv } } } diff --git a/crates/flowstate-collab/src/net/direct.rs b/crates/flowstate-collab/src/net/direct.rs index 429f5f0b..a6e9a691 100644 --- a/crates/flowstate-collab/src/net/direct.rs +++ b/crates/flowstate-collab/src/net/direct.rs @@ -60,33 +60,58 @@ struct DirectServeInner { impl DirectServeState { pub async fn attach_session(&self, session: SessionId) { - self.inner.write().await.attached.insert(session); + let mut inner = self.inner.write().await; + let inserted = inner.attached.insert(session); + tracing::debug!(%session, inserted, attached_sessions = inner.attached.len(), "attached collaboration direct session"); } pub async fn detach_session(&self, session: SessionId) { let mut inner = self.inner.write().await; - inner.attached.remove(&session); - inner.blobs.remove(&session); - inner.handlers.remove(&session); + let removed = inner.attached.remove(&session); + let blob_count = inner.blobs.remove(&session).map_or(0, |outbox| outbox.len()); + let handler_removed = inner.handlers.remove(&session).is_some(); + tracing::debug!(%session, removed, blob_count, handler_removed, attached_sessions = inner.attached.len(), "detached collaboration direct session"); } pub async fn register_handler(&self, session: SessionId, handler: DirectSessionHandler) { let mut inner = self.inner.write().await; inner.attached.insert(session); - inner.handlers.insert(session, handler); + let replaced = inner.handlers.insert(session, handler).is_some(); + tracing::debug!(%session, replaced, handler_count = inner.handlers.len(), "registered collaboration direct session handler"); } pub async fn insert_blob(&self, session: SessionId, bytes: Vec) -> BlobId { - let mut inner = self.inner.write().await; - inner.attached.insert(session); - inner.blobs.entry(session).or_default().insert(bytes) + let byte_len = bytes.len(); + let (blob, outbox_len, outbox_bytes) = { + let mut inner = self.inner.write().await; + inner.attached.insert(session); + let outbox = inner.blobs.entry(session).or_default(); + let blob = outbox.insert(bytes); + let outbox_len = outbox.len(); + let outbox_bytes = outbox.total_bytes(); + drop(inner); + (blob, outbox_len, outbox_bytes) + }; + tracing::debug!( + %session, + ?blob, + byte_len, + outbox_len, + outbox_bytes, + "stored collaboration direct blob for peer pull", + ); + blob } async fn serve(&self, request: DirectRequest) -> ServeOutcome { let session = request.session(); + let request_kind = direct_request_kind(&request); + let request_detail_bytes = direct_request_detail_bytes(&request); + tracing::trace!(%session, request_kind, request_detail_bytes, "serving collaboration direct request"); let handler = { let inner = self.inner.read().await; if !inner.attached.contains(&session) { + tracing::warn!(%session, request_kind, "collaboration direct request rejected because session is not attached"); return ServeOutcome::Header(DirectResponseHeader::NotAttached); } if let DirectRequest::Blob { blob, .. } = request { @@ -94,7 +119,11 @@ impl DirectServeState { .blobs .get(&session) .and_then(|outbox| outbox.get(blob)) - .map_or(ServeOutcome::Header(DirectResponseHeader::NotFound), |bytes| { + .map_or_else(|| { + tracing::warn!(%session, ?blob, "collaboration direct blob request missed outbox"); + ServeOutcome::Header(DirectResponseHeader::NotFound) + }, |bytes| { + tracing::debug!(%session, ?blob, bytes = bytes.len(), "collaboration direct blob request hit outbox"); ServeOutcome::Payload(bytes.to_vec()) }); } @@ -102,13 +131,17 @@ impl DirectServeState { }; let Some(handler) = handler else { + tracing::warn!(%session, request_kind, "collaboration direct request rejected because handler is missing"); return ServeOutcome::Header(DirectResponseHeader::NotFound); }; match request { - DirectRequest::Snapshot { .. } => request_payload(handler.requests, |reply| DirectServeRequest::Snapshot { reply }).await, - DirectRequest::Updates { have_vv, .. } => request_payload(handler.requests, |reply| DirectServeRequest::Updates { have_vv, reply }).await, - DirectRequest::Asset { asset, .. } => request_asset(handler.requests, asset).await, + DirectRequest::Snapshot { .. } => request_payload(handler.requests, session, request_kind, |reply| DirectServeRequest::Snapshot { reply }).await, + DirectRequest::Updates { have_vv, .. } => { + tracing::trace!(%session, have_vv_bytes = have_vv.len(), "forwarding collaboration direct updates request to session"); + request_payload(handler.requests, session, request_kind, |reply| DirectServeRequest::Updates { have_vv, reply }).await + }, + DirectRequest::Asset { asset, .. } => request_asset(handler.requests, session, asset).await, DirectRequest::Blob { .. } => ServeOutcome::Header(DirectResponseHeader::NotFound), } } @@ -120,6 +153,22 @@ enum ServeOutcome { Payload(Vec), } +impl ServeOutcome { + fn kind(&self) -> &'static str { + match self { + Self::Header(header) => direct_response_header_kind(header), + Self::Payload(_) => "ok", + } + } + + fn payload_len(&self) -> usize { + match self { + Self::Header(_) => 0, + Self::Payload(payload) => payload.len(), + } + } +} + #[derive(Clone, Debug)] pub struct DirectProto { state: DirectServeState, @@ -137,12 +186,17 @@ impl DirectProto { async fn handle_stream(&self, mut send: SendStream, mut recv: RecvStream) -> Result<()> { let request = read_frame::(&mut recv).await?; + let session = request.session(); + let request_kind = direct_request_kind(&request); + tracing::debug!(%session, request_kind, "received collaboration direct stream request"); let Ok(permit) = self.permits.clone().try_acquire_owned() else { + tracing::warn!(%session, request_kind, "collaboration direct server is busy"); write_response(&mut send, ServeOutcome::Header(DirectResponseHeader::Busy)).await?; return Ok(()); }; let outcome = self.state.serve(request).await; + tracing::debug!(%session, request_kind, outcome = outcome.kind(), payload_bytes = outcome.payload_len(), "collaboration direct request served"); write_response(&mut send, outcome).await?; drop(permit); Ok(()) @@ -151,20 +205,23 @@ impl DirectProto { impl ProtocolHandler for DirectProto { async fn accept(&self, connection: Connection) -> Result<(), AcceptError> { + tracing::trace!("accepted collaboration direct connection"); while let Ok((send, recv)) = connection.accept_bi().await { let proto = self.clone(); tokio::spawn(async move { if let Err(error) = proto.handle_stream(send, recv).await { - eprintln!("flowstate collab direct stream failed: {error:#}"); + tracing::error!(error = %format_args!("{error:#}"), "collaboration direct stream failed"); } }); } + tracing::trace!("collaboration direct connection closed"); Ok(()) } } pub(crate) fn install_endpoint(endpoint: Endpoint) { - let _ = CLIENT_ENDPOINT.set(endpoint); + let installed = CLIENT_ENDPOINT.set(endpoint).is_ok(); + tracing::debug!(installed, "installed collaboration direct client endpoint"); } pub async fn pull_with_fallback(req: DirectRequest, candidates: Vec, per_peer_timeout: Duration) -> Result> { @@ -181,67 +238,118 @@ pub async fn pull_with_endpoint( candidates: Vec, per_peer_timeout: Duration, ) -> Result> { + let session = req.session(); + let request_kind = direct_request_kind(&req); + let candidate_count = candidates.len(); + if candidates.is_empty() { + tracing::warn!(%session, request_kind, "collaboration direct pull has no candidate peers"); + } ensure!(!candidates.is_empty(), "direct pull has no candidate peers"); + tracing::debug!(%session, request_kind, candidate_count, ?per_peer_timeout, "starting collaboration direct pull"); let mut errors = Vec::new(); for peer in candidates { + tracing::trace!(%session, request_kind, peer = %peer, "attempting collaboration direct pull peer"); match timeout(per_peer_timeout, pull_once(endpoint, peer, req.clone())).await { - Ok(Ok(bytes)) => return Ok(bytes), - Ok(Err(error)) => errors.push(format!("{peer}: {error:#}")), - Err(_) => errors.push(format!("{peer}: timed out after {per_peer_timeout:?}")), + Ok(Ok(bytes)) => { + tracing::debug!(%session, request_kind, peer = %peer, bytes = bytes.len(), "collaboration direct pull peer succeeded"); + return Ok(bytes); + }, + Ok(Err(error)) => { + tracing::warn!(%session, request_kind, peer = %peer, error = %format_args!("{error:#}"), "collaboration direct pull peer failed"); + errors.push(format!("{peer}: {error:#}")); + }, + Err(_) => { + tracing::warn!(%session, request_kind, peer = %peer, ?per_peer_timeout, "collaboration direct pull peer timed out"); + errors.push(format!("{peer}: timed out after {per_peer_timeout:?}")); + }, } } + tracing::error!(%session, request_kind, candidate_count, "collaboration direct pull failed for all candidates"); Err(anyhow!("direct pull failed for all candidates: {}", errors.join("; "))) } async fn pull_once(endpoint: &Endpoint, peer: EndpointId, req: DirectRequest) -> Result> { + let session = req.session(); + let request_kind = direct_request_kind(&req); + tracing::trace!(%session, request_kind, peer = %peer, "dialing collaboration direct peer"); let connection = endpoint .connect(peer, DIRECT_ALPN) .await .context("direct dial failed")?; + tracing::trace!(%session, request_kind, peer = %peer, "opening collaboration direct stream"); let (mut send, mut recv) = connection .open_bi() .await .context("opening direct request stream failed")?; - write_frame(&mut send, &encode_frame(&req)?).await?; + let frame = encode_frame(&req)?; + tracing::trace!(%session, request_kind, peer = %peer, frame_bytes = frame.len(), "sending collaboration direct request frame"); + write_frame(&mut send, &frame).await?; send.finish()?; let header = read_frame::(&mut recv).await?; + tracing::trace!(%session, request_kind, peer = %peer, response = direct_response_header_kind(&header), "received collaboration direct response header"); match header { - DirectResponseHeader::Ok { total_len } => read_payload(&mut recv, total_len).await, + DirectResponseHeader::Ok { total_len } => { + let payload = read_payload(&mut recv, total_len).await?; + tracing::trace!(%session, request_kind, peer = %peer, bytes = payload.len(), "read collaboration direct response payload"); + Ok(payload) + }, DirectResponseHeader::NotAttached => Err(anyhow!("peer is not attached to this session")), DirectResponseHeader::NotFound => Err(anyhow!("peer does not have the requested collaboration data")), DirectResponseHeader::Busy => Err(anyhow!("peer is busy serving collaboration data")), } } -async fn request_payload(requests: Sender, make_request: F) -> ServeOutcome +async fn request_payload(requests: Sender, session: SessionId, request_kind: &'static str, make_request: F) -> ServeOutcome where F: FnOnce(Sender>>) -> DirectServeRequest, { let (reply_tx, reply_rx) = async_channel::bounded(1); if requests.send(make_request(reply_tx)).await.is_err() { + tracing::warn!(%session, request_kind, "collaboration direct session request channel closed"); return ServeOutcome::Header(DirectResponseHeader::NotAttached); } match reply_rx.recv().await { - Ok(Ok(bytes)) => ServeOutcome::Payload(bytes), - Ok(Err(_)) | Err(_) => ServeOutcome::Header(DirectResponseHeader::NotFound), + Ok(Ok(bytes)) => { + tracing::trace!(%session, request_kind, bytes = bytes.len(), "collaboration direct session returned payload"); + ServeOutcome::Payload(bytes) + }, + Ok(Err(error)) => { + tracing::warn!(%session, request_kind, error = %format_args!("{error:#}"), "collaboration direct session failed to produce payload"); + ServeOutcome::Header(DirectResponseHeader::NotFound) + }, + Err(error) => { + tracing::warn!(%session, request_kind, error = %error, "collaboration direct session payload reply channel closed"); + ServeOutcome::Header(DirectResponseHeader::NotFound) + }, } } -async fn request_asset(requests: Sender, asset: u128) -> ServeOutcome { +async fn request_asset(requests: Sender, session: SessionId, asset: u128) -> ServeOutcome { let (reply_tx, reply_rx) = async_channel::bounded(1); if requests .send(DirectServeRequest::Asset { asset, reply: reply_tx }) .await .is_err() { + tracing::warn!(%session, asset, "collaboration direct asset request channel closed"); return ServeOutcome::Header(DirectResponseHeader::NotAttached); } match reply_rx.recv().await { - Ok(Ok(asset)) => ServeOutcome::Payload(asset.bytes), - Ok(Err(_)) | Err(_) => ServeOutcome::Header(DirectResponseHeader::NotFound), + Ok(Ok(asset_bytes)) => { + tracing::trace!(%session, asset, bytes = asset_bytes.bytes.len(), "collaboration direct session returned asset"); + ServeOutcome::Payload(asset_bytes.bytes) + }, + Ok(Err(error)) => { + tracing::warn!(%session, asset, error = %format_args!("{error:#}"), "collaboration direct session failed to produce asset"); + ServeOutcome::Header(DirectResponseHeader::NotFound) + }, + Err(error) => { + tracing::warn!(%session, asset, error = %error, "collaboration direct asset reply channel closed"); + ServeOutcome::Header(DirectResponseHeader::NotFound) + }, } } @@ -260,6 +368,7 @@ where } async fn write_response(send: &mut SendStream, outcome: ServeOutcome) -> Result<()> { + tracing::trace!(outcome = outcome.kind(), payload_bytes = outcome.payload_len(), "writing collaboration direct response"); match outcome { ServeOutcome::Header(header) => write_frame(send, &encode_frame(&header)?).await?, ServeOutcome::Payload(payload) => { @@ -275,11 +384,13 @@ async fn write_response(send: &mut SendStream, outcome: ServeOutcome) -> Result< } async fn write_frame(send: &mut SendStream, frame: &[u8]) -> Result<()> { + tracing::trace!(frame_bytes = frame.len(), "writing collaboration direct frame"); send.write_all(frame).await?; Ok(()) } async fn write_payload(send: &mut SendStream, payload: &[u8]) -> Result<()> { + tracing::trace!(payload_bytes = payload.len(), chunk_bytes = MAX_PAYLOAD_CHUNK_LEN, "writing collaboration direct payload"); for chunk in payload.chunks(MAX_PAYLOAD_CHUNK_LEN) { send.write_all(chunk).await?; } @@ -288,6 +399,7 @@ async fn write_payload(send: &mut SendStream, payload: &[u8]) -> Result<()> { async fn read_payload(recv: &mut RecvStream, total_len: u64) -> Result> { let total_len = usize::try_from(total_len).context("direct payload is too large for this platform")?; + tracing::trace!(payload_bytes = total_len, chunk_bytes = MAX_PAYLOAD_CHUNK_LEN, "reading collaboration direct payload"); let mut payload = vec![0; total_len]; let mut offset = 0; while offset < total_len { @@ -298,6 +410,31 @@ async fn read_payload(recv: &mut RecvStream, total_len: u64) -> Result> Ok(payload) } +fn direct_request_kind(request: &DirectRequest) -> &'static str { + match request { + DirectRequest::Snapshot { .. } => "snapshot", + DirectRequest::Updates { .. } => "updates", + DirectRequest::Blob { .. } => "blob", + DirectRequest::Asset { .. } => "asset", + } +} + +fn direct_request_detail_bytes(request: &DirectRequest) -> usize { + match request { + DirectRequest::Updates { have_vv, .. } => have_vv.len(), + DirectRequest::Snapshot { .. } | DirectRequest::Blob { .. } | DirectRequest::Asset { .. } => 0, + } +} + +fn direct_response_header_kind(header: &DirectResponseHeader) -> &'static str { + match header { + DirectResponseHeader::Ok { .. } => "ok", + DirectResponseHeader::NotAttached => "not_attached", + DirectResponseHeader::NotFound => "not_found", + DirectResponseHeader::Busy => "busy", + } +} + impl DirectRequest { fn session(&self) -> SessionId { match self { diff --git a/crates/flowstate-collab/src/net/mod.rs b/crates/flowstate-collab/src/net/mod.rs index dcd0aff0..18869a86 100644 --- a/crates/flowstate-collab/src/net/mod.rs +++ b/crates/flowstate-collab/src/net/mod.rs @@ -26,6 +26,25 @@ pub enum PublishPayload { Digest { vv: Vec }, } +impl PublishPayload { + #[must_use] + pub fn kind(&self) -> &'static str { + match self { + Self::Update(_) => "update", + Self::Presence(_) => "presence", + Self::Digest { .. } => "digest", + } + } + + #[must_use] + pub fn byte_len(&self) -> usize { + match self { + Self::Update(bytes) | Self::Presence(bytes) => bytes.len(), + Self::Digest { vv } => vv.len(), + } + } +} + #[derive(Debug)] pub enum NetCommand { EnsureUp, @@ -77,6 +96,26 @@ pub enum NetCommand { Shutdown, } +impl NetCommand { + #[must_use] + pub fn kind(&self) -> &'static str { + match self { + Self::EnsureUp => "ensure_up", + Self::RegisterDirectHandler { .. } => "register_direct_handler", + Self::CreateSession { .. } => "create_session", + Self::JoinSession { .. } => "join_session", + Self::LeaveSession { .. } => "leave_session", + Self::Publish { .. } => "publish", + Self::PullUpdates { .. } => "pull_updates", + Self::PullSnapshot { .. } => "pull_snapshot", + Self::PullBlob { .. } => "pull_blob", + Self::PullAsset { .. } => "pull_asset", + Self::MintTicketAddr { .. } => "mint_ticket_addr", + Self::Shutdown => "shutdown", + } + } +} + #[derive(Clone, Debug)] pub struct TicketSeed { pub inviter: EndpointAddr, diff --git a/crates/flowstate-collab/src/net/runtime.rs b/crates/flowstate-collab/src/net/runtime.rs index 5c574b2b..4cde22fc 100644 --- a/crates/flowstate-collab/src/net/runtime.rs +++ b/crates/flowstate-collab/src/net/runtime.rs @@ -26,12 +26,15 @@ struct RuntimeBridge { pub fn start() -> Result<(CommandSender, EventReceiver)> { if let Some(runtime) = RUNTIME.get() { + tracing::debug!("reusing existing collaboration network runtime"); return Ok((runtime.commands.clone(), runtime.events.clone())); } + tracing::info!("starting collaboration network runtime"); let runtime = spawn_runtime()?; let set_failed = RUNTIME.set(runtime.clone()).is_err(); if set_failed && let Some(runtime) = RUNTIME.get() { + tracing::debug!("another caller initialized collaboration network runtime first"); return Ok((runtime.commands.clone(), runtime.events.clone())); } Ok((runtime.commands, runtime.events)) @@ -45,6 +48,7 @@ fn spawn_runtime() -> Result { thread::Builder::new() .name("flowstate-collab-net".into()) .spawn(move || { + tracing::debug!("collaboration network thread started"); let runtime = match tokio::runtime::Builder::new_multi_thread() .worker_threads(2) .enable_all() @@ -53,15 +57,16 @@ fn spawn_runtime() -> Result { Ok(runtime) => runtime, Err(error) => { let _ = thread_evt_tx.try_send(NetEvent::EndpointOnline(false)); - eprintln!("flowstate collaboration tokio runtime failed: {error:#}"); + tracing::error!(error = %format_args!("{error:#}"), "collaboration tokio runtime failed"); return; }, }; if let Err(error) = runtime.block_on(net_main(cmd_rx, evt_tx)) { let _ = thread_evt_tx.try_send(NetEvent::EndpointOnline(false)); - eprintln!("flowstate collaboration networking stopped: {error:#}"); + tracing::error!(error = %format_args!("{error:#}"), "collaboration networking stopped unexpectedly"); } + tracing::debug!("collaboration network thread stopped"); }) .context("spawning collaboration networking thread failed")?; @@ -72,10 +77,13 @@ fn spawn_runtime() -> Result { } async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_channel::Sender) -> Result<()> { + tracing::info!("binding collaboration endpoint"); let endpoint = Endpoint::builder(presets::N0) .bind() .await .context("binding collaboration endpoint failed")?; + let bound_addr = endpoint_addr(&endpoint); + tracing::info!(peer = %bound_addr.id, "collaboration endpoint bound"); direct::install_endpoint(endpoint.clone()); let gossip = Gossip::builder().spawn(endpoint.clone()); @@ -84,19 +92,24 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha .accept(iroh_gossip::ALPN, gossip.clone()) .accept(crate::DIRECT_ALPN, DirectProto::new(direct_state.clone())) .spawn(); + tracing::info!(peer = %bound_addr.id, "collaboration direct and gossip protocols installed"); let _ = evt_tx.send(NetEvent::EndpointOnline(true)).await; let mut swarms = HashMap::new(); while let Ok(command) = cmd_rx.recv().await { + tracing::trace!(command = command.kind(), "collaboration network command received"); match command { NetCommand::EnsureUp => { + tracing::debug!("collaboration endpoint ensure-up requested"); let _ = evt_tx.send(NetEvent::EndpointOnline(true)).await; }, NetCommand::RegisterDirectHandler { session, handler } => { + tracing::debug!(%session, "registering collaboration direct handler"); direct_state.register_handler(session, handler).await; }, NetCommand::CreateSession { session, reply } => { + tracing::info!(%session, "creating collaboration network session"); direct_state.attach_session(session).await; replace_swarm( &mut swarms, @@ -110,29 +123,41 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha )?, ) .await; + tracing::debug!(%session, peer = %bound_addr.id, "collaboration network session created"); let _ = reply .send(Ok(TicketSeed { - inviter: endpoint_addr(&endpoint), + inviter: bound_addr.clone(), })) .await; }, NetCommand::JoinSession { session, bootstrap } => { + let bootstrap_count = bootstrap.len(); + tracing::info!(%session, bootstrap_count, "joining collaboration network session"); direct_state.attach_session(session).await; replace_swarm( &mut swarms, SwarmHandle::spawn(endpoint.clone(), gossip.clone(), direct_state.clone(), session, bootstrap, evt_tx.clone())?, ) .await; + tracing::debug!(%session, bootstrap_count, "collaboration network join subscribed"); }, NetCommand::LeaveSession { session } => { + tracing::info!(%session, "leaving collaboration network session"); if let Some(handle) = swarms.remove(&session) { handle.stop().await; + } else { + tracing::debug!(%session, "leave requested for missing collaboration swarm"); } direct_state.detach_session(session).await; }, NetCommand::Publish { session, payload } => { + let payload_kind = payload.kind(); + let payload_bytes = payload.byte_len(); if let Some(handle) = swarms.get(&session) { + tracing::trace!(%session, payload_kind, payload_bytes, "publishing collaboration payload"); handle.publish(payload).await?; + } else { + tracing::warn!(%session, payload_kind, payload_bytes, "dropped collaboration publish for missing swarm"); } }, NetCommand::PullUpdates { @@ -141,11 +166,18 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha our_vv, reply, } => { + let candidate_count = candidates.len(); + let vv_bytes = our_vv.len(); + tracing::debug!(%session, candidate_count, vv_bytes, "pulling collaboration updates"); let result = direct::pull_with_fallback(DirectRequest::Updates { session, have_vv: our_vv }, candidates, DIRECT_PULL_TIMEOUT).await; + log_direct_pull_result("updates", session, candidate_count, &result); let _ = reply.send(result).await; }, NetCommand::PullSnapshot { session, candidates, reply } => { + let candidate_count = candidates.len(); + tracing::debug!(%session, candidate_count, "pulling collaboration snapshot"); let result = direct::pull_with_fallback(DirectRequest::Snapshot { session }, candidates, DIRECT_PULL_TIMEOUT).await; + log_direct_pull_result("snapshot", session, candidate_count, &result); let _ = reply.send(result).await; }, NetCommand::PullBlob { @@ -154,7 +186,10 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha blob, reply, } => { + let candidate_count = candidates.len(); + tracing::debug!(%session, ?blob, candidate_count, "pulling collaboration blob"); let result = direct::pull_with_fallback(DirectRequest::Blob { session, blob }, candidates, DIRECT_PULL_TIMEOUT).await; + log_direct_pull_result("blob", session, candidate_count, &result); let _ = reply.send(result).await; }, NetCommand::PullAsset { @@ -163,28 +198,41 @@ async fn net_main(cmd_rx: async_channel::Receiver, evt_tx: async_cha asset, reply, } => { + let candidate_count = candidates.len(); + tracing::debug!(%session, asset, candidate_count, "pulling collaboration asset"); let result = direct::pull_with_fallback(DirectRequest::Asset { session, asset }, candidates, DIRECT_PULL_TIMEOUT) .await .map(|bytes| AssetBytes { bytes }); + match &result { + Ok(bytes) => tracing::debug!(%session, asset, candidate_count, bytes = bytes.bytes.len(), "collaboration asset pull succeeded"), + Err(error) => tracing::warn!(%session, asset, candidate_count, error = %format_args!("{error:#}"), "collaboration asset pull failed"), + } let _ = reply.send(result).await; }, NetCommand::MintTicketAddr { reply } => { - let _ = reply.send(endpoint_addr(&endpoint)).await; + tracing::debug!(peer = %bound_addr.id, "minting collaboration ticket address"); + let _ = reply.send(bound_addr.clone()).await; + }, + NetCommand::Shutdown => { + tracing::info!(open_sessions = swarms.len(), "shutting down collaboration network runtime"); + break; }, - NetCommand::Shutdown => break, } } for (_, handle) in swarms { + tracing::debug!(session = %handle.session, "stopping collaboration swarm during shutdown"); handle.stop().await; } endpoint.close().await; + tracing::info!(peer = %bound_addr.id, "collaboration endpoint closed"); Ok(()) } async fn replace_swarm(swarms: &mut HashMap, handle: SwarmHandle) { let session = handle.session; if let Some(previous) = swarms.insert(session, handle) { + tracing::debug!(%session, "replacing existing collaboration swarm"); previous.stop().await; } } @@ -192,3 +240,10 @@ async fn replace_swarm(swarms: &mut HashMap, hand fn endpoint_addr(endpoint: &Endpoint) -> EndpointAddr { endpoint.addr() } + +fn log_direct_pull_result(kind: &'static str, session: crate::SessionId, candidate_count: usize, result: &Result>) { + match result { + Ok(bytes) => tracing::debug!(%session, kind, candidate_count, bytes = bytes.len(), "collaboration direct pull succeeded"), + Err(error) => tracing::warn!(%session, kind, candidate_count, error = %format_args!("{error:#}"), "collaboration direct pull failed"), + } +} diff --git a/crates/flowstate-collab/src/net/swarm.rs b/crates/flowstate-collab/src/net/swarm.rs index 84a1a427..5ab67910 100644 --- a/crates/flowstate-collab/src/net/swarm.rs +++ b/crates/flowstate-collab/src/net/swarm.rs @@ -36,11 +36,13 @@ impl SwarmHandle { bootstrap: Vec, evt_tx: Sender, ) -> Result { + tracing::info!(%session, bootstrap_count = bootstrap.len(), "spawning collaboration gossip swarm"); register_bootstrap_addrs(&endpoint, &bootstrap)?; let bootstrap_ids = bootstrap.iter().map(endpoint_id).collect::>(); let (tx, rx) = async_channel::unbounded(); tokio::spawn(async move { if let Err(error) = run_session(gossip, direct_state, session, bootstrap_ids, rx, evt_tx.clone()).await { + tracing::error!(%session, error = %format_args!("{error:#}"), "collaboration gossip swarm failed"); let _ = evt_tx .send(NetEvent::SubscribeFailed { session, @@ -53,6 +55,7 @@ impl SwarmHandle { } pub async fn publish(&self, payload: PublishPayload) -> Result<()> { + tracing::trace!(session = %self.session, payload_kind = payload.kind(), payload_bytes = payload.byte_len(), "queueing collaboration gossip publish"); self .tx .send(SwarmCommand::Publish(payload)) @@ -61,6 +64,7 @@ impl SwarmHandle { } pub async fn stop(&self) { + tracing::debug!(session = %self.session, "stopping collaboration gossip swarm"); let _ = self.tx.send(SwarmCommand::Stop).await; } } @@ -73,10 +77,12 @@ async fn run_session( rx: Receiver, evt_tx: Sender, ) -> Result<()> { + tracing::info!(%session, bootstrap_count = bootstrap.len(), "subscribing to collaboration gossip topic"); let topic = gossip .subscribe(topic_id(session), bootstrap) .await .context("subscribing to collaboration gossip topic failed")?; + tracing::info!(%session, "subscribed to collaboration gossip topic"); let (sender, mut receiver) = topic.split(); loop { @@ -84,7 +90,14 @@ async fn run_session( command = rx.recv() => { match command { Ok(SwarmCommand::Publish(payload)) => publish(&sender, &direct_state, session, payload).await?, - Ok(SwarmCommand::Stop) | Err(_) => break, + Ok(SwarmCommand::Stop) => { + tracing::debug!(%session, "collaboration gossip swarm stop received"); + break; + }, + Err(error) => { + tracing::debug!(%session, error = %error, "collaboration gossip swarm command channel closed"); + break; + }, } }, event = receiver.next() => { @@ -94,10 +107,13 @@ async fn run_session( } } + tracing::info!(%session, "collaboration gossip swarm stopped"); Ok(()) } async fn publish(sender: &GossipSender, direct_state: &DirectServeState, session: SessionId, payload: PublishPayload) -> Result<()> { + let payload_kind = payload.kind(); + let payload_bytes = payload.byte_len(); let (message, neighbors_only) = match payload { PublishPayload::Update(bytes) => (update_message(direct_state, session, bytes).await?, false), PublishPayload::Presence(bytes) => (GossipMsg::Presence(bytes), false), @@ -109,23 +125,38 @@ async fn publish(sender: &GossipSender, direct_state: &DirectServeState, session } else { proto_gossip::encode_inline(&message)? }; + let frame_bytes = frame.len(); + tracing::trace!( + %session, + payload_kind, + payload_bytes, + gossip_kind = message.kind(), + gossip_payload_bytes = message.payload_len(), + frame_bytes, + neighbors_only, + "broadcasting collaboration gossip frame", + ); if neighbors_only { sender.broadcast_neighbors(frame.into()).await?; } else { sender.broadcast(frame.into()).await?; } + tracing::trace!(%session, payload_kind, frame_bytes, neighbors_only, "collaboration gossip frame broadcast complete"); Ok(()) } async fn update_message(direct_state: &DirectServeState, session: SessionId, bytes: Vec) -> Result { + let update_bytes = bytes.len(); let inline = GossipMsg::Update(bytes.clone()); if proto_gossip::encoded_len(&inline)? <= GOSSIP_INLINE_LIMIT { + tracing::trace!(%session, update_bytes, "collaboration update will be sent inline over gossip"); return Ok(inline); } let len = bytes.len() as u64; let blob = direct_state.insert_blob(session, bytes).await; + tracing::debug!(%session, ?blob, update_bytes, "collaboration update exceeded gossip limit; publishing blob announcement"); Ok(GossipMsg::UpdateAvailable { blob, len }) } @@ -133,6 +164,14 @@ async fn handle_event(event: Event, session: SessionId, evt_tx: &Sender match proto_gossip::decode(&message.content) { Ok(msg) => { + tracing::trace!( + %session, + from = %message.delivered_from, + gossip_kind = msg.kind(), + gossip_payload_bytes = msg.payload_len(), + frame_bytes = message.content.len(), + "received collaboration gossip frame", + ); let _ = evt_tx .send(NetEvent::Gossip { session, @@ -141,15 +180,24 @@ async fn handle_event(event: Event, session: SessionId, evt_tx: &Sender eprintln!("flowstate collab ignored gossip frame: {error:#}"), + Err(error) => tracing::warn!( + %session, + from = %message.delivered_from, + frame_bytes = message.content.len(), + error = %format_args!("{error:#}"), + "ignored malformed collaboration gossip frame", + ), }, Event::NeighborUp(peer) => { + tracing::info!(%session, peer = %peer, "collaboration gossip neighbor up"); let _ = evt_tx.send(NetEvent::NeighborUp { session, peer }).await; }, Event::NeighborDown(peer) => { + tracing::info!(%session, peer = %peer, "collaboration gossip neighbor down"); let _ = evt_tx.send(NetEvent::NeighborDown { session, peer }).await; }, Event::Lagged => { + tracing::warn!(%session, "collaboration gossip receiver lagged"); let _ = evt_tx.send(NetEvent::GossipLagged { session }).await; }, } @@ -160,6 +208,7 @@ fn register_bootstrap_addrs(endpoint: &Endpoint, bootstrap: &[EndpointAddr]) -> if bootstrap.is_empty() { return Ok(()); } + tracing::debug!(bootstrap_count = bootstrap.len(), "registering collaboration bootstrap addresses"); endpoint .address_lookup()? .add(MemoryLookup::from_endpoint_info(bootstrap.iter().cloned())); diff --git a/crates/flowstate-collab/src/proto_gossip.rs b/crates/flowstate-collab/src/proto_gossip.rs index 54aa6e71..4eee9ec8 100644 --- a/crates/flowstate-collab/src/proto_gossip.rs +++ b/crates/flowstate-collab/src/proto_gossip.rs @@ -15,6 +15,27 @@ pub enum GossipMsg { Digest { session: SessionId, vv: Vec }, } +impl GossipMsg { + #[must_use] + pub fn kind(&self) -> &'static str { + match self { + Self::Update(_) => "update", + Self::UpdateAvailable { .. } => "update_available", + Self::Presence(_) => "presence", + Self::Digest { .. } => "digest", + } + } + + #[must_use] + pub fn payload_len(&self) -> u64 { + match self { + Self::Update(bytes) | Self::Presence(bytes) => bytes.len() as u64, + Self::UpdateAvailable { len, .. } => *len, + Self::Digest { vv, .. } => vv.len() as u64, + } + } +} + pub fn encode(msg: &GossipMsg) -> Result> { let mut bytes = PROTOCOL_VERSION.to_le_bytes().to_vec(); bytes.extend(postcard::to_stdvec(msg)?); diff --git a/crates/flowstate/Cargo.toml b/crates/flowstate/Cargo.toml index a06fe97f..69395949 100644 --- a/crates/flowstate/Cargo.toml +++ b/crates/flowstate/Cargo.toml @@ -32,6 +32,9 @@ serde_json = { workspace = true } tempfile = { workspace = true } toml = { workspace = true } uuid = { workspace = true } +tracing = { workspace = true } +tracing-appender = { workspace = true } +tracing-subscriber = { workspace = true } [features] hotpath = [ diff --git a/crates/flowstate/src/collab/asset_transfer.rs b/crates/flowstate/src/collab/asset_transfer.rs index 24502cca..eb85b902 100644 --- a/crates/flowstate/src/collab/asset_transfer.rs +++ b/crates/flowstate/src/collab/asset_transfer.rs @@ -22,19 +22,22 @@ pub(super) fn schedule_missing_assets( cx: &mut Context, ) { let Some(editor) = session.editor.clone() else { + tracing::trace!(session = %session.session, preferred_peer = ?preferred_peer, "skipping collaboration asset scan because editor is missing"); return; }; let Some(doc) = &session.doc else { + tracing::trace!(session = %session.session, preferred_peer = ?preferred_peer, "skipping collaboration asset scan because Loro doc is missing"); return; }; let assets = match image_assets_in_loro(doc) { Ok(assets) => assets, Err(error) => { - eprintln!("flowstate collab asset scan failed: {error:#}"); + tracing::warn!(session = %session.session, error = %format_args!("{error:#}"), "collaboration asset scan failed"); return; }, }; + tracing::trace!(session = %session.session, assets = assets.len(), "scanned collaboration image assets"); if assets.is_empty() { return; } @@ -55,6 +58,7 @@ pub(super) fn schedule_missing_assets( .cloned() .collect::>() }; + tracing::debug!(session = %session.session, assets = assets.len(), missing_assets = missing_assets.len(), "checked collaboration image assets for missing bytes"); if missing_assets.is_empty() { return; } @@ -66,18 +70,22 @@ pub(super) fn schedule_missing_assets( record: meta.placeholder_record(), }) .collect::>(); + tracing::debug!(session = %session.session, placeholders = placeholders.len(), "queueing collaboration asset placeholders"); session.apply_or_queue_patches(placeholders, cx); let candidates = session.pull_candidates(preferred_peer); if candidates.is_empty() { + tracing::warn!(session = %session.session, missing_assets = missing_assets.len(), "cannot pull collaboration assets because no candidate peers are available"); return; } for meta in missing_assets { let id = AssetId(meta.asset_id); if !session.asset_pulls_in_flight.insert(id) { + tracing::trace!(session = %session.session, ?id, "collaboration asset pull already in flight"); continue; } + tracing::debug!(session = %session.session, ?id, bytes = meta.byte_len, candidate_count = candidates.len(), "scheduling collaboration asset pull"); start_asset_pull(session, candidates.clone(), meta, cx); } } @@ -90,7 +98,9 @@ fn start_asset_pull( ) { let (reply_tx, reply_rx) = async_channel::bounded(1); let id = AssetId(meta.asset_id); - if session + let session_id = session.session; + let candidate_count = candidates.len(); + if let Err(error) = session .net_tx .try_send(NetCommand::PullAsset { session: session.session, @@ -98,11 +108,12 @@ fn start_asset_pull( asset: meta.asset_id, reply: reply_tx, }) - .is_err() { + tracing::warn!(session = %session_id, ?id, candidate_count, error = %error, "queueing collaboration asset pull failed"); session.asset_pulls_in_flight.remove(&id); return; } + tracing::debug!(session = %session_id, ?id, candidate_count, expected_bytes = meta.byte_len, "queued collaboration asset pull"); cx.spawn(async move |session, cx| { let result = reply_rx.recv().await; @@ -111,12 +122,13 @@ fn start_asset_pull( match result { Ok(Ok(bytes)) => match meta.record_from_bytes(bytes.bytes) { Ok(record) => { + tracing::debug!(session = %session_id, ?id, bytes = record.bytes.len(), "collaboration asset pull succeeded"); session.apply_or_queue_patches(vec![CollabPatch::AssetArrived { id, record }], cx); }, - Err(error) => eprintln!("flowstate collab rejected fetched asset {id:?}: {error:#}"), + Err(error) => tracing::warn!(session = %session_id, ?id, error = %format_args!("{error:#}"), "rejected fetched collaboration asset"), }, - Ok(Err(error)) => eprintln!("flowstate collab asset pull failed for {id:?}: {error:#}"), - Err(error) => eprintln!("flowstate collab asset pull channel closed for {id:?}: {error}"), + Ok(Err(error)) => tracing::warn!(session = %session_id, ?id, error = %format_args!("{error:#}"), "collaboration asset pull failed"), + Err(error) => tracing::warn!(session = %session_id, ?id, error = %error, "collaboration asset pull channel closed"), } }); }) @@ -144,6 +156,7 @@ impl ImageAssetMeta { } fn record_from_bytes(&self, bytes: Vec) -> Result { + tracing::trace!(asset = self.asset_id, expected_bytes = self.byte_len, received_bytes = bytes.len(), "validating fetched collaboration asset bytes"); ensure!(bytes.len() as u64 == self.byte_len, "asset byte length mismatch"); let mut hasher = DefaultHasher::new(); bytes.hash(&mut hasher); diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index 52dffdea..66da30b6 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -64,29 +64,43 @@ impl CollabManager { T: 'static, { if let Some(session) = self.session_by_panel.get(&panel_id).copied() { + tracing::debug!(%panel_id, %session, "collaboration session already exists for panel"); return Ok(session); } let commands = self.ensure_runtime(cx)?; let session = SessionId::new(); let document = editor.read(cx).document().clone(); + tracing::info!( + %panel_id, + %session, + title = %title, + paragraphs = document.paragraphs.len(), + blocks = document.blocks.len(), + assets = document.assets.assets.len(), + "starting local collaboration session", + ); let collab = CollabSession::from_local_document(session, panel_id, editor, title, document, commands.clone())?; let direct_handler = collab.direct_handler(); let entity = cx.new(|_| collab); entity.update(cx, |session, cx| session.attach(cx)); self.register_session(entity.clone(), cx); - commands - .try_send(NetCommand::RegisterDirectHandler { + if let Err(error) = commands.try_send(NetCommand::RegisterDirectHandler { session, handler: direct_handler, - }) - .context("registering collaboration direct handler failed")?; + }) { + tracing::error!(%panel_id, %session, error = %error, "registering collaboration direct handler failed"); + return Err(error).context("registering collaboration direct handler failed"); + } + tracing::debug!(%panel_id, %session, "registered collaboration direct handler for local session"); let (reply_tx, reply_rx) = async_channel::bounded(1); - commands - .try_send(NetCommand::CreateSession { session, reply: reply_tx }) - .context("creating collaboration network session failed")?; + if let Err(error) = commands.try_send(NetCommand::CreateSession { session, reply: reply_tx }) { + tracing::error!(%panel_id, %session, error = %error, "queueing collaboration create-session command failed"); + return Err(error).context("creating collaboration network session failed"); + } + tracing::debug!(%panel_id, %session, "queued collaboration create-session command"); Self::finish_create_session(entity, reply_rx, cx); Ok(session) } @@ -95,7 +109,14 @@ impl CollabManager { where T: 'static, { + tracing::info!(session = %ticket.session, inviter = %ticket.inviter.id, title = %ticket.title, "joining collaboration session from ticket"); + if !ticket.is_supported_version() { + tracing::warn!(session = %ticket.session, "unsupported collaboration protocol version in ticket"); + } ensure!(ticket.is_supported_version(), "unsupported collaboration protocol version"); + if self.sessions_by_id.contains_key(&ticket.session) { + tracing::warn!(session = %ticket.session, "collaboration session is already open"); + } ensure!( !self.sessions_by_id.contains_key(&ticket.session), "collaboration session is already open" @@ -112,6 +133,7 @@ impl CollabManager { session, bootstrap: vec![ticket.inviter], }) { + tracing::error!(%session, error = %error, "queueing collaboration join-session command failed"); entity.update(cx, |session, cx| { session.detach( DetachReason::JoinFailed(format!("joining collaboration network session failed: {error}")), @@ -121,6 +143,7 @@ impl CollabManager { self.unregister_session(session); return Err(error).context("joining collaboration network session failed"); } + tracing::debug!(%session, inviter = %inviter, "queued collaboration join-session command"); let completed = entity.update(cx, |session, cx| { session.set_join_stage(JoinStage::Subscribing, cx); session.begin_join_bootstrap(inviter, cx) @@ -138,6 +161,7 @@ impl CollabManager { where T: 'static, { + tracing::info!(%session_id, %panel_id, "attaching joined collaboration session to editor"); let commands = self.ensure_runtime(cx)?; let session = self .sessions_by_id @@ -147,12 +171,14 @@ impl CollabManager { let direct_handler = session.read(cx).direct_handler(); session.update(cx, |session, cx| session.attach_joined_editor(panel_id, editor, cx))?; self.session_by_panel.insert(panel_id, session_id); - commands - .try_send(NetCommand::RegisterDirectHandler { + if let Err(error) = commands.try_send(NetCommand::RegisterDirectHandler { session: session_id, handler: direct_handler, - }) - .context("registering collaboration direct handler failed")?; + }) { + tracing::error!(%session_id, %panel_id, error = %error, "registering collaboration direct handler for joined session failed"); + return Err(error).context("registering collaboration direct handler failed"); + } + tracing::debug!(%session_id, %panel_id, "registered collaboration direct handler for joined session"); Self::establish_joined_peer(session, commands, cx)?; Ok(()) } @@ -162,13 +188,16 @@ impl CollabManager { T: 'static, { let Some(session_id) = self.session_by_panel.get(&panel_id).copied() else { + tracing::debug!(%panel_id, "leave requested for panel without collaboration session"); return false; }; let Some(session) = self.sessions_by_id.get(&session_id).cloned() else { + tracing::warn!(%panel_id, %session_id, "leave requested for missing registered collaboration session"); self.unregister_session(session_id); return false; }; + tracing::info!(%panel_id, %session_id, "leaving collaboration session for panel"); let _ = session.update(cx, |session, cx| session.detach(DetachReason::UserLeft, cx)); self.unregister_session(session_id); true @@ -184,14 +213,22 @@ impl CollabManager { let commands = self.ensure_runtime(cx).ok()?; let (ticket_tx, ticket_rx) = async_channel::bounded(1); let (reply_tx, reply_rx) = async_channel::bounded(1); - commands - .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) - .ok()?; + if let Err(error) = commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }) { + tracing::error!(%panel_id, %session_id, error = %error, "queueing collaboration ticket address command failed"); + return None; + } + tracing::debug!(%panel_id, %session_id, "requested collaboration ticket address"); cx.spawn(async move |_, _| { let ticket = match reply_rx.recv().await { - Ok(addr) => Ok(SessionTicket::new(session_id, addr, title)), - Err(error) => Err(anyhow!("collaboration endpoint address unavailable: {error}")), + Ok(addr) => { + tracing::info!(%session_id, inviter = %addr.id, "created collaboration share ticket"); + Ok(SessionTicket::new(session_id, addr, title)) + }, + Err(error) => { + tracing::error!(%session_id, error = %error, "collaboration endpoint address unavailable for ticket"); + Err(anyhow!("collaboration endpoint address unavailable: {error}")) + }, }; let _ = ticket_tx.send(ticket).await; }) @@ -223,13 +260,16 @@ impl CollabManager { let id = session.read(cx).session_id(); if let Some(panel_id) = session.read(cx).panel_id() { self.session_by_panel.insert(panel_id, id); + tracing::debug!(%id, %panel_id, "registered collaboration session panel mapping"); } self.sessions_by_id.insert(id, session); + tracing::debug!(%id, open_sessions = self.sessions_by_id.len(), "registered collaboration session"); } fn unregister_session(&mut self, session: SessionId) { - self.sessions_by_id.remove(&session); + let removed = self.sessions_by_id.remove(&session).is_some(); self.session_by_panel.retain(|_, active| *active != session); + tracing::debug!(%session, removed, open_sessions = self.sessions_by_id.len(), "unregistered collaboration session"); } fn ensure_runtime(&mut self, cx: &mut Context) -> Result @@ -237,9 +277,11 @@ impl CollabManager { T: 'static, { if let Some(runtime) = &self.runtime { + tracing::trace!("collaboration runtime already available"); return Ok(runtime.commands.clone()); } + tracing::info!("starting collaboration runtime for manager"); let (commands, events) = runtime::start()?; self.runtime = Some(CollabRuntime { commands: commands.clone() }); self.start_event_pump(events, cx); @@ -251,18 +293,22 @@ impl CollabManager { T: 'static, { if self.event_pump_started { + tracing::trace!("collaboration network event pump already started"); return; } self.event_pump_started = true; + tracing::debug!("starting collaboration network event pump"); cx.spawn(async move |_, cx| { while let Ok(event) = events.recv().await { if cx .update_global::(|manager, cx| manager.handle_net_event(event, cx)) .is_err() { + tracing::warn!("collaboration network event pump could not update manager; stopping"); break; } } + tracing::debug!("collaboration network event pump stopped"); }) .detach(); } @@ -270,24 +316,30 @@ impl CollabManager { fn handle_net_event(&mut self, event: NetEvent, cx: &mut App) { match event { NetEvent::EndpointOnline(online) => { + tracing::info!(online, "collaboration endpoint status changed"); self.endpoint_online = online; for session in self.sessions_by_id.values() { session.update(cx, |session, cx| session.set_endpoint_online(online, cx)); } }, NetEvent::Gossip { session, from, msg } => { + tracing::trace!(%session, from = %from, gossip_kind = msg.kind(), gossip_payload_bytes = msg.payload_len(), "routing collaboration gossip event"); self.update_session(session, cx, |session, cx| session.handle_gossip(from, msg, cx)); }, NetEvent::NeighborUp { session, peer } => { + tracing::debug!(%session, peer = %peer, "routing collaboration neighbor-up event"); self.update_session(session, cx, |session, cx| session.neighbor_up(peer, cx)); }, NetEvent::NeighborDown { session, peer } => { + tracing::debug!(%session, peer = %peer, "routing collaboration neighbor-down event"); self.update_session(session, cx, |session, cx| session.neighbor_down(peer, cx)); }, NetEvent::GossipLagged { session } => { + tracing::warn!(%session, "routing collaboration gossip-lagged event"); self.update_session(session, cx, |session, cx| session.handle_gossip_lagged(cx)); }, NetEvent::SubscribeFailed { session, error } => { + tracing::error!(%session, error = %error, "collaboration gossip subscription failed"); let detached = self.update_session(session, cx, |session, cx| session.detach(DetachReason::Fatal(error), cx)); if detached.unwrap_or(false) { self.unregister_session(session); @@ -307,22 +359,30 @@ impl CollabManager { .get(&session) .cloned() .map(|session| session.update(cx, update)) + .or_else(|| { + tracing::warn!(%session, "collaboration event targeted missing session"); + None + }) } fn finish_create_session(session: Entity, reply_rx: Receiver>, cx: &mut Context) where T: 'static, { + let session_id = session.read(cx).session_id(); cx.spawn(async move |_, cx| match reply_rx.recv().await { Ok(Ok(seed)) => { + tracing::info!(%session_id, inviter = %seed.inviter.id, "collaboration create-session completed"); let _ = session.update(cx, |session, cx| session.establish_local_peer(&seed.inviter.id, cx)); }, Ok(Err(error)) => { + tracing::error!(%session_id, error = %format_args!("{error:#}"), "collaboration create-session failed"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error:#}")), cx); }); }, Err(error) => { + tracing::error!(%session_id, error = %error, "collaboration create-session reply channel closed"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error}")), cx); }); @@ -335,15 +395,20 @@ impl CollabManager { where T: 'static, { + let session_id = session.read(cx).session_id(); let (reply_tx, reply_rx) = async_channel::bounded(1); - commands - .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) - .context("requesting collaboration endpoint address failed")?; + if let Err(error) = commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }) { + tracing::error!(%session_id, error = %error, "queueing collaboration endpoint address command failed"); + return Err(error).context("requesting collaboration endpoint address failed"); + } + tracing::debug!(%session_id, "requested collaboration endpoint address for joined session"); cx.spawn(async move |_, cx| match reply_rx.recv().await { Ok(addr) => { + tracing::info!(%session_id, peer = %addr.id, "collaboration joined session local peer established"); let _ = session.update(cx, |session, cx| session.establish_local_peer(&addr.id, cx)); }, Err(error) => { + tracing::error!(%session_id, error = %error, "collaboration endpoint address unavailable for joined session"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error}")), cx); }); diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index 94ee7175..39a90765 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -132,6 +132,15 @@ impl CollabSession { let (direct_tx, direct_rx) = async_channel::unbounded(); let (undo_tx, undo_rx) = async_channel::unbounded(); let now = Instant::now(); + tracing::info!( + %session, + %panel_id, + title = %title, + paragraphs = document.paragraphs.len(), + blocks = document.blocks.len(), + assets = document.assets.assets.len(), + "built local collaboration document projection", + ); Ok(Self { session, @@ -173,6 +182,7 @@ impl CollabSession { let (direct_tx, direct_rx) = async_channel::unbounded(); let (undo_tx, undo_rx) = async_channel::unbounded(); let now = Instant::now(); + tracing::info!(%session, title = %title, bootstrap_count = bootstrap_addrs.len(), "created joining collaboration session state"); Self { session, title, @@ -261,6 +271,7 @@ impl CollabSession { } pub fn set_join_stage(&mut self, stage: JoinStage, cx: &mut Context) { + tracing::debug!(session = %self.session, ?stage, "collaboration join stage changed"); self.phase = SessionPhase::Joining(stage); cx.notify(); } @@ -275,11 +286,15 @@ impl CollabSession { cx.notify(); let (reply_tx, reply_rx) = async_channel::bounded(1); + let candidates = self.pull_candidates(Some(inviter)); + let session_id = self.session; + tracing::info!(session = %self.session, inviter = %inviter, candidate_count = candidates.len(), "requesting collaboration join snapshot"); if let Err(error) = self.net_tx.try_send(NetCommand::PullSnapshot { session: self.session, - candidates: self.pull_candidates(Some(inviter)), + candidates, reply: reply_tx, }) { + tracing::error!(session = %self.session, inviter = %inviter, error = %error, "queueing collaboration join snapshot pull failed"); let detail = format!("requesting collaboration snapshot failed: {error}"); self.detach(DetachReason::JoinFailed(detail.clone()), cx); let _ = result_tx.try_send(Err(anyhow!(detail))); @@ -288,17 +303,25 @@ impl CollabSession { cx.spawn(async move |session, cx| { let joined = match reply_rx.recv().await { - Ok(Ok(bytes)) => session - .update(cx, |session, cx| match session.finish_join_snapshot(&bytes, cx) { - Ok(joined) => Ok(joined), - Err(error) => { - let detail = format!("building collaboration document failed: {error:#}"); - session.detach(DetachReason::JoinFailed(detail), cx); - Err(error.context("building collaboration document failed")) - }, - }) - .unwrap_or_else(|error| Err(anyhow!("collaboration join session disappeared: {error}"))), + Ok(Ok(bytes)) => { + tracing::info!(session = %session_id, snapshot_bytes = bytes.len(), "collaboration join snapshot pulled"); + session + .update(cx, |session, cx| match session.finish_join_snapshot(&bytes, cx) { + Ok(joined) => Ok(joined), + Err(error) => { + tracing::error!(session = %session.session, error = %format_args!("{error:#}"), "building collaboration document from snapshot failed"); + let detail = format!("building collaboration document failed: {error:#}"); + session.detach(DetachReason::JoinFailed(detail), cx); + Err(error.context("building collaboration document failed")) + }, + }) + .unwrap_or_else(|error| { + tracing::error!(error = %error, "collaboration join session disappeared while building snapshot"); + Err(anyhow!("collaboration join session disappeared: {error}")) + }) + }, Ok(Err(error)) => { + tracing::error!(session = %session_id, error = %format_args!("{error:#}"), "pulling collaboration join snapshot failed"); let detail = format!("pulling collaboration snapshot failed: {error:#}"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::JoinFailed(detail.clone()), cx); @@ -306,6 +329,7 @@ impl CollabSession { Err(anyhow!(detail)) }, Err(error) => { + tracing::error!(session = %session_id, error = %error, "collaboration join snapshot reply channel closed"); let detail = format!("collaboration snapshot reply channel closed: {error}"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::JoinFailed(detail.clone()), cx); @@ -322,9 +346,17 @@ impl CollabSession { pub fn attach_joined_editor(&mut self, panel_id: Uuid, editor: Entity, cx: &mut Context) -> Result<()> { if self.doc.is_none() || self.binding.is_none() { + tracing::warn!(session = %self.session, %panel_id, "cannot attach joined editor before snapshot load finishes"); bail!("collaboration snapshot has not finished loading"); } + tracing::info!( + session = %self.session, + %panel_id, + pending_remote_updates = self.pending_remote_updates.len(), + pending_remote_patches = self.pending_remote_patches.len(), + "attaching joined collaboration editor", + ); self.panel_id = Some(panel_id); self.editor = Some(editor); self.attach(cx); @@ -337,10 +369,12 @@ impl CollabSession { asset_transfer::schedule_missing_assets(self, None, cx); self.publish_digest(); cx.notify(); + tracing::info!(session = %self.session, %panel_id, "joined collaboration editor attached"); Ok(()) } pub fn attach(&mut self, cx: &mut Context) { + tracing::debug!(session = %self.session, phase = ?self.phase, "attaching collaboration session hooks"); self.attach_undo_manager(); self.attach_editor_hooks(cx); self.attach_loro_publish_hook(); @@ -351,19 +385,27 @@ impl CollabSession { pub fn establish_local_peer(&mut self, peer: &flowstate_collab::ids::PeerId, cx: &mut Context) { if self.presence.is_none() { + tracing::info!(session = %self.session, peer = %peer, "establishing local collaboration peer presence"); let presence = PresenceStore::new(peer); let session = self.session; let net_tx = self.net_tx.clone(); self .loro_subscriptions .push(presence.subscribe_local_updates(move |bytes| { - let _ = net_tx.try_send(NetCommand::Publish { + let bytes_len = bytes.len(); + if let Err(error) = net_tx.try_send(NetCommand::Publish { session, payload: PublishPayload::Presence(bytes.clone()), - }); + }) { + tracing::warn!(%session, bytes = bytes_len, error = %error, "queueing collaboration presence publish failed"); + } else { + tracing::trace!(%session, bytes = bytes_len, "queued collaboration presence publish from local update"); + } true })); self.presence = Some(presence); + } else { + tracing::debug!(session = %self.session, peer = %peer, "local collaboration peer presence already established"); } self.refresh_own_presence(cx); self.endpoint_online = true; @@ -375,9 +417,11 @@ impl CollabSession { self.publish_digest(); asset_transfer::schedule_missing_assets(self, None, cx); cx.notify(); + tracing::info!(session = %self.session, peer = %peer, peers_present = self.peers_present(), "local collaboration peer established"); } fn finish_join_snapshot(&mut self, snapshot: &[u8], cx: &mut Context) -> Result { + tracing::info!(session = %self.session, snapshot_bytes = snapshot.len(), "building collaboration document from join snapshot"); self.phase = SessionPhase::Joining(JoinStage::Building); cx.notify(); @@ -388,6 +432,13 @@ impl CollabSession { projection::verify_lineage(&doc, self.session)?; let document = projection::document_from_loro(&doc, load_document_theme())?; let binding = DocBinding::build(&doc, &document)?; + tracing::info!( + session = %self.session, + paragraphs = document.paragraphs.len(), + blocks = document.blocks.len(), + assets = document.assets.assets.len(), + "built collaboration document from join snapshot", + ); self.doc = Some(doc); self.binding = Some(binding); @@ -400,16 +451,21 @@ impl CollabSession { pub fn detach(&mut self, reason: DetachReason, cx: &mut Context) -> bool { if matches!(self.phase, SessionPhase::Detached(_)) { + tracing::debug!(session = %self.session, ?reason, "collaboration session already detached"); return false; } + tracing::warn!(session = %self.session, ?reason, phase = ?self.phase, "detaching collaboration session"); if let Some(presence) = &self.presence { presence.delete_self(); self.publish_presence_bytes(presence.encode_self()); } - let _ = self + if let Err(error) = self .net_tx - .try_send(NetCommand::LeaveSession { session: self.session }); + .try_send(NetCommand::LeaveSession { session: self.session }) + { + tracing::warn!(session = %self.session, error = %error, "queueing collaboration leave-session command failed during detach"); + } self.flush_pending_remote_patches(cx); if let Some(editor) = self.editor.clone() { @@ -439,11 +495,13 @@ impl CollabSession { self.local_update_publish_attached = false; self.phase = SessionPhase::Detached(reason); cx.notify(); + tracing::info!(session = %self.session, "collaboration session detached and cleaned up"); true } pub fn flush_local_edits(&mut self, editor: Entity, cx: &mut Context) -> Result<()> { if matches!(self.phase, SessionPhase::Detached(_) | SessionPhase::Joining(_)) { + tracing::trace!(session = %self.session, phase = ?self.phase, "skipping local collaboration edit flush for inactive phase"); return Ok(()); } @@ -453,10 +511,19 @@ impl CollabSession { editor.clear_collab_history(); edits }); + let edit_count = edits.len(); + let operation_count = edits.iter().map(|edit| edit.operations.len()).sum::(); + if edit_count == 0 || operation_count == 0 { + tracing::trace!(session = %self.session, edit_count, operation_count, "no local collaboration edits to flush"); + } else { + tracing::debug!(session = %self.session, edit_count, operation_count, "flushing local collaboration edits into Loro"); + } let Some(doc) = &self.doc else { + tracing::warn!(session = %self.session, edit_count, operation_count, "cannot flush local collaboration edits because Loro doc is missing"); return Ok(()); }; let Some(binding) = &mut self.binding else { + tracing::warn!(session = %self.session, edit_count, operation_count, "cannot flush local collaboration edits because binding is missing"); return Ok(()); }; @@ -465,22 +532,32 @@ impl CollabSession { if edit.operations.is_empty() { continue; } - LocalApplier { doc, binding }.apply(&document, &edit.operations)?; + let operation_count = edit.operations.len(); + if let Err(error) = (LocalApplier { doc, binding }).apply(&document, &edit.operations) { + tracing::error!(session = %self.session, operation_count, error = %format_args!("{error:#}"), "applying local collaboration edit failed"); + return Err(error); + } + tracing::trace!(session = %self.session, operation_count, "applied local collaboration edit to Loro"); applied = true; } if applied { self.last_document_activity = Instant::now(); + tracing::debug!(session = %self.session, "local collaboration edits flushed"); } Ok(()) } pub fn handle_gossip(&mut self, from: flowstate_collab::ids::PeerId, msg: GossipMsg, cx: &mut Context) { + let gossip_kind = msg.kind(); + let gossip_payload_bytes = msg.payload_len(); + tracing::trace!(session = %self.session, from = %from, gossip_kind, gossip_payload_bytes, "handling collaboration gossip message"); self.note_inbound_traffic(cx); let result = match msg { GossipMsg::Update(bytes) => self .import_update_bytes(&bytes, cx) .map(|()| asset_transfer::schedule_missing_assets(self, Some(from), cx)), - GossipMsg::UpdateAvailable { blob, .. } => { + GossipMsg::UpdateAvailable { blob, len } => { + tracing::debug!(session = %self.session, from = %from, ?blob, bytes = len, "collaboration update available via direct blob pull"); self.pull_blob(from, blob, cx); Ok(()) }, @@ -491,30 +568,34 @@ impl CollabSession { GossipMsg::Digest { session, vv } => self.handle_digest(from, session, &vv, cx), }; if let Err(error) = result { + tracing::error!(session = %self.session, from = %from, gossip_kind, error = %format_args!("{error:#}"), "collaboration gossip handling failed"); self.detach(DetachReason::Fatal(format!("collaboration update failed: {error:#}")), cx); } } pub fn neighbor_up(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { - self.neighbors.insert(peer); + let inserted = self.neighbors.insert(peer); self.zero_neighbors_since = None; self.anti_entropy.on_neighbor_up(); self.mark_online(cx); self.publish_digest(); asset_transfer::schedule_missing_assets(self, Some(peer), cx); cx.notify(); + tracing::info!(session = %self.session, peer = %peer, inserted, neighbors = self.neighbors.len(), "collaboration neighbor up"); } pub fn neighbor_down(&mut self, peer: flowstate_collab::ids::PeerId, cx: &mut Context) { - self.neighbors.remove(&peer); + let removed = self.neighbors.remove(&peer); if self.neighbors.is_empty() && self.zero_neighbors_since.is_none() { self.zero_neighbors_since = Some(Instant::now()); } self.evaluate_connectivity(cx); cx.notify(); + tracing::info!(session = %self.session, peer = %peer, removed, neighbors = self.neighbors.len(), "collaboration neighbor down"); } pub fn handle_gossip_lagged(&mut self, cx: &mut Context) { + tracing::warn!(session = %self.session, neighbors = self.neighbors.len(), "collaboration gossip lagged; scheduling recovery"); self.publish_digest(); let peer = self.neighbors.iter().next().copied(); let vv = self @@ -527,6 +608,7 @@ impl CollabSession { } pub fn set_endpoint_online(&mut self, online: bool, cx: &mut Context) { + tracing::info!(session = %self.session, online, previous_online = self.endpoint_online, "collaboration endpoint online state applied to session"); self.endpoint_online = online; if online { if self.peers_present() == 0 || !self.neighbors.is_empty() { @@ -540,15 +622,19 @@ impl CollabSession { fn attach_editor_hooks(&mut self, cx: &mut Context) { if !self.editor_subscriptions.is_empty() { + tracing::trace!(session = %self.session, "collaboration editor hooks already attached"); return; } let Some(editor) = self.editor.clone() else { + tracing::warn!(session = %self.session, "cannot attach collaboration editor hooks because editor is missing"); return; }; + tracing::debug!(session = %self.session, "attaching collaboration editor hooks"); editor.update(cx, |editor, cx| { if editor.document_path().is_none() { editor.set_recovery_path(Some(presence_view::collaboration_recovery_path(self.session, &self.title)), cx); + tracing::debug!(session = %self.session, "set collaboration recovery path for untitled document"); } editor.clear_collab_history(); editor.set_collab_capture(true); @@ -562,6 +648,7 @@ impl CollabSession { .editor_subscriptions .push(cx.observe(&editor, |session, editor, cx| { if let Err(error) = session.flush_local_edits(editor.clone(), cx) { + tracing::error!(session = %session.session, error = %format_args!("{error:#}"), "capturing local collaboration edit failed"); session.detach(DetachReason::Fatal(format!("capturing local collaboration edit failed: {error:#}")), cx); return; } @@ -572,13 +659,16 @@ impl CollabSession { .editor_subscriptions .push(cx.subscribe(&editor, |session, _, event: &EditorEvent, cx| { if matches!(event, EditorEvent::SelectionChanged { .. }) { + tracing::trace!(session = %session.session, "collaboration local selection changed; refreshing presence"); session.refresh_own_presence(cx); } })); + tracing::debug!(session = %self.session, subscriptions = self.editor_subscriptions.len(), "collaboration editor hooks attached"); } fn attach_loro_publish_hook(&mut self) { if self.doc.is_none() || self.local_update_publish_attached { + tracing::trace!(session = %self.session, has_doc = self.doc.is_some(), attached = self.local_update_publish_attached, "skipping collaboration Loro publish hook attach"); return; } let Some(doc) = &self.doc else { @@ -586,13 +676,19 @@ impl CollabSession { }; let session = self.session; let net_tx = self.net_tx.clone(); + tracing::debug!(%session, "attaching collaboration Loro local-update publish hook"); self .loro_subscriptions .push(doc.subscribe_local_update(Box::new(move |bytes| { - let _ = net_tx.try_send(NetCommand::Publish { + let bytes_len = bytes.len(); + if let Err(error) = net_tx.try_send(NetCommand::Publish { session, payload: PublishPayload::Update(bytes.clone()), - }); + }) { + tracing::warn!(%session, bytes = bytes_len, error = %error, "queueing collaboration update publish failed"); + } else { + tracing::trace!(%session, bytes = bytes_len, "queued collaboration update publish from local Loro update"); + } true }))); self.local_update_publish_attached = true; @@ -600,36 +696,46 @@ impl CollabSession { fn attach_undo_manager(&mut self) { if self.undo_manager.is_some() { + tracing::trace!(session = %self.session, "collaboration undo manager already attached"); return; } let Some(doc) = &self.doc else { + tracing::warn!(session = %self.session, "cannot attach collaboration undo manager because Loro doc is missing"); return; }; let mut undo_manager = UndoManager::new(doc); undo_manager.set_merge_interval(500); undo_manager.set_max_undo_steps(300); self.undo_manager = Some(undo_manager); + tracing::debug!(session = %self.session, merge_interval_ms = 500, max_undo_steps = 300, "collaboration undo manager attached"); } fn attach_undo_request_pump(&mut self, cx: &mut Context) { if self.undo_pump_started { + tracing::trace!(session = %self.session, "collaboration undo request pump already started"); return; } self.undo_pump_started = true; let requests = self.undo_rx.clone(); + let session_id = self.session; + tracing::debug!(session = %session_id, "starting collaboration undo request pump"); cx.spawn(async move |session, cx| { while let Ok(redirect) = requests.recv().await { + tracing::debug!(session = %session_id, ?redirect, "received collaboration undo redirect"); if session .update(cx, |session, cx| { if let Err(error) = session.apply_loro_undo_redirect(redirect, cx) { + tracing::error!(session = %session.session, error = %format_args!("{error:#}"), "collaboration undo redirect failed"); session.detach(DetachReason::Fatal(format!("collaboration undo failed: {error:#}")), cx); } }) .is_err() { + tracing::debug!(session = %session_id, "collaboration undo request pump session disappeared"); break; } } + tracing::debug!(session = %session_id, "collaboration undo request pump stopped"); }) .detach(); } diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs index 2323b517..9ac0c5c7 100644 --- a/crates/flowstate/src/collab/session_io.rs +++ b/crates/flowstate/src/collab/session_io.rs @@ -22,10 +22,20 @@ use super::{CollabSession, DetachReason}; impl CollabSession { pub fn import_update_bytes(&mut self, bytes: &[u8], cx: &mut Context) -> Result<()> { if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() { + tracing::debug!( + session = %self.session, + bytes = bytes.len(), + queued_updates = self.pending_remote_updates.len() + 1, + has_doc = self.doc.is_some(), + has_binding = self.binding.is_some(), + has_editor = self.editor.is_some(), + "queueing remote collaboration update until session is attached", + ); self.pending_remote_updates.push(bytes.to_vec()); return Ok(()); } + tracing::debug!(session = %self.session, bytes = bytes.len(), "importing remote collaboration update"); let editor = self .editor .clone() @@ -47,26 +57,36 @@ impl CollabSession { drop(sub); self.binding = Some(take_mutex_value(binding, "document binding")?); let patches = take_mutex_value(patches, "remote patches")?; - import_result.context("importing collaboration update failed")?; + if let Err(error) = import_result.context("importing collaboration update failed") { + tracing::error!(session = %self.session, bytes = bytes.len(), error = %format_args!("{error:#}"), "remote collaboration update import failed"); + return Err(error); + } + tracing::debug!(session = %self.session, bytes = bytes.len(), patches = patches.len(), "remote collaboration update imported"); self.apply_or_queue_patches(patches, cx); Ok(()) } pub(super) fn attach_direct_request_pump(&mut self, cx: &mut Context) { if self.direct_pump_started { + tracing::trace!(session = %self.session, "collaboration direct request pump already started"); return; } self.direct_pump_started = true; let requests = self.direct_rx.clone(); + let session_id = self.session; + tracing::debug!(session = %session_id, "starting collaboration direct request pump"); cx.spawn(async move |session, cx| { while let Ok(request) = requests.recv().await { + tracing::trace!(session = %session_id, request_kind = direct_serve_request_kind(&request), "received collaboration direct serve request"); if session .update(cx, |session, cx| session.handle_direct_request(request, cx)) .is_err() { + tracing::debug!(session = %session_id, "collaboration direct request pump session disappeared"); break; } } + tracing::debug!(session = %session_id, "collaboration direct request pump stopped"); }) .detach(); } @@ -79,9 +99,16 @@ impl CollabSession { cx: &mut Context, ) -> Result<()> { let Some(doc) = &self.doc else { + tracing::debug!(session = %self.session, from = %from, digest_session = %digest_session, vv_bytes = vv.len(), "ignored collaboration digest because Loro doc is missing"); return Ok(()); }; - let sender_vv = VersionVector::decode(vv).context("decoding collaboration digest failed")?; + let sender_vv = match VersionVector::decode(vv).context("decoding collaboration digest failed") { + Ok(sender_vv) => sender_vv, + Err(error) => { + tracing::warn!(session = %self.session, from = %from, digest_session = %digest_session, vv_bytes = vv.len(), error = %format_args!("{error:#}"), "decoding collaboration digest failed"); + return Err(error); + }, + }; let our_vv = doc.oplog_vv(); let relation = match sender_vv.partial_cmp(&our_vv) { Some(std::cmp::Ordering::Equal) => VersionVectorRelation::Equal, @@ -92,31 +119,49 @@ impl CollabSession { let action = self .anti_entropy .consider_digest(from, digest_session, relation, our_vv.encode()); + tracing::trace!( + session = %self.session, + from = %from, + digest_session = %digest_session, + vv_bytes = vv.len(), + ?relation, + action = gap_action_kind(&action), + "handled collaboration digest", + ); self.handle_gap_action(action, cx); Ok(()) } pub(super) fn pull_blob(&mut self, from: flowstate_collab::ids::PeerId, blob: flowstate_collab::BlobId, cx: &mut Context) { let (reply_tx, reply_rx) = async_channel::bounded(1); - if self + let candidates = self.pull_candidates(Some(from)); + tracing::debug!(session = %self.session, from = %from, ?blob, candidate_count = candidates.len(), "requesting collaboration update blob pull"); + if let Err(error) = self .net_tx .try_send(NetCommand::PullBlob { session: self.session, - candidates: self.pull_candidates(Some(from)), + candidates, blob, reply: reply_tx, }) - .is_err() { + tracing::warn!(session = %self.session, from = %from, ?blob, error = %error, "queueing collaboration blob pull failed"); return; } + let session_id = self.session; cx.spawn(async move |session, cx| { let result = reply_rx.recv().await; let _ = session.update(cx, |session, cx| { - if let Ok(Ok(bytes)) = result - && let Err(error) = session.import_update_bytes(&bytes, cx) - { - session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + match result { + Ok(Ok(bytes)) => { + tracing::debug!(session = %session_id, ?blob, bytes = bytes.len(), "collaboration blob pull succeeded"); + if let Err(error) = session.import_update_bytes(&bytes, cx) { + tracing::error!(session = %session_id, ?blob, error = %format_args!("{error:#}"), "importing pulled collaboration blob failed"); + session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + } + }, + Ok(Err(error)) => tracing::warn!(session = %session_id, ?blob, error = %format_args!("{error:#}"), "collaboration blob pull failed"), + Err(error) => tracing::warn!(session = %session_id, ?blob, error = %error, "collaboration blob pull reply channel closed"), } }); }) @@ -125,35 +170,62 @@ impl CollabSession { pub(super) fn publish_digest(&self) { if let Some(doc) = &self.doc { - let _ = self.net_tx.try_send(NetCommand::Publish { + let vv = doc.oplog_vv().encode(); + let vv_bytes = vv.len(); + if let Err(error) = self.net_tx.try_send(NetCommand::Publish { session: self.session, - payload: PublishPayload::Digest { vv: doc.oplog_vv().encode() }, - }); + payload: PublishPayload::Digest { vv }, + }) { + tracing::warn!(session = %self.session, vv_bytes, error = %error, "queueing collaboration digest publish failed"); + } else { + tracing::trace!(session = %self.session, vv_bytes, "queued collaboration digest publish"); + } + } else { + tracing::trace!(session = %self.session, "skipping collaboration digest publish because Loro doc is missing"); } } pub(super) fn flush_pending_remote_patches(&mut self, cx: &mut Context) -> bool { let Some(editor) = self.editor.clone() else { + tracing::trace!(session = %self.session, pending_patches = self.pending_remote_patches.len(), "cannot flush remote collaboration patches because editor is missing"); return false; }; if self.pending_remote_patches.is_empty() || editor.read(cx).collab_apply_deferred() { + tracing::trace!( + session = %self.session, + pending_patches = self.pending_remote_patches.len(), + deferred = editor.read(cx).collab_apply_deferred(), + "remote collaboration patch flush skipped", + ); return false; } let patches = std::mem::take(&mut self.pending_remote_patches); + tracing::debug!(session = %self.session, patches = patches.len(), "flushing remote collaboration patches to editor"); editor.update(cx, |editor, cx| { editor.clear_collab_history(); editor.apply_collab_patches(&patches, cx); }); self.last_document_activity = std::time::Instant::now(); self.refresh_external_carets(cx); + tracing::debug!(session = %self.session, patches = patches.len(), "remote collaboration patches flushed to editor"); true } pub(super) fn apply_loro_undo_redirect(&mut self, redirect: UndoRedirect, cx: &mut Context) -> Result<()> { if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() || self.undo_manager.is_none() { + tracing::warn!( + session = %self.session, + ?redirect, + has_doc = self.doc.is_some(), + has_binding = self.binding.is_some(), + has_editor = self.editor.is_some(), + has_undo_manager = self.undo_manager.is_some(), + "cannot apply collaboration undo redirect because session state is incomplete", + ); return Ok(()); } + tracing::debug!(session = %self.session, ?redirect, "applying collaboration undo redirect"); let editor = self .editor .clone() @@ -187,6 +259,7 @@ impl CollabSession { self.binding = Some(take_mutex_value(binding, "document binding")?); let patches = take_mutex_value(patches, "undo patches")?; let applied = undo_result.context("applying collaboration undo operation failed")?; + tracing::debug!(session = %self.session, ?redirect, applied, patches = patches.len(), "collaboration undo redirect applied"); if applied { self.apply_or_queue_patches(patches, cx); self.publish_digest(); @@ -197,44 +270,62 @@ impl CollabSession { pub(super) fn handle_gap_action(&mut self, action: GapAction, cx: &mut Context) { match action { GapAction::None => {}, - GapAction::Pull { from, our_vv } => self.start_update_pull(from, our_vv, cx), + GapAction::Pull { from, our_vv } => { + tracing::debug!(session = %self.session, from = %from, our_vv_bytes = our_vv.len(), "collaboration gap action requested update pull"); + self.start_update_pull(from, our_vv, cx); + }, GapAction::LineageMismatch { from, expected, got } => { - eprintln!("flowstate collab ignored mismatched digest from {from}: expected session {expected}, got {got}"); + tracing::warn!(session = %self.session, from = %from, expected = %expected, got = %got, "ignored mismatched collaboration digest"); }, } } fn handle_direct_request(&mut self, request: DirectServeRequest, cx: &mut Context) { + tracing::trace!(session = %self.session, request_kind = direct_serve_request_kind(&request), "serving collaboration direct request from session"); match request { DirectServeRequest::Snapshot { reply } => { - let _ = reply.try_send(self.snapshot_bytes()); + let result = self.snapshot_bytes(); + log_direct_serve_result(self.session, "snapshot", &result); + let _ = reply.try_send(result); }, DirectServeRequest::Updates { have_vv, reply } => { - let _ = reply.try_send(self.update_bytes(&have_vv)); + tracing::trace!(session = %self.session, have_vv_bytes = have_vv.len(), "serving collaboration updates request"); + let result = self.update_bytes(&have_vv); + log_direct_serve_result(self.session, "updates", &result); + let _ = reply.try_send(result); }, DirectServeRequest::Asset { asset, reply } => { - let _ = reply.try_send(self.asset_bytes(asset, cx)); + let result = self.asset_bytes(asset, cx); + match &result { + Ok(bytes) => tracing::debug!(session = %self.session, asset, bytes = bytes.bytes.len(), "served collaboration asset direct request"), + Err(error) => tracing::warn!(session = %self.session, asset, error = %format_args!("{error:#}"), "serving collaboration asset direct request failed"), + } + let _ = reply.try_send(result); }, } } fn snapshot_bytes(&self) -> Result> { - self + let bytes = self .doc .as_ref() .context("collaboration session is not attached")? .export(ExportMode::Snapshot) - .context("exporting collaboration snapshot failed") + .context("exporting collaboration snapshot failed")?; + tracing::debug!(session = %self.session, bytes = bytes.len(), "exported collaboration snapshot"); + Ok(bytes) } fn update_bytes(&self, have_vv: &[u8]) -> Result> { let vv = VersionVector::decode(have_vv).context("decoding collaboration version vector failed")?; - self + let bytes = self .doc .as_ref() .context("collaboration session is not attached")? .export(ExportMode::updates(&vv)) - .context("exporting collaboration updates failed") + .context("exporting collaboration updates failed")?; + tracing::debug!(session = %self.session, have_vv_bytes = have_vv.len(), bytes = bytes.len(), "exported collaboration updates"); + Ok(bytes) } fn asset_bytes(&self, asset: u128, cx: &mut Context) -> Result { @@ -250,6 +341,7 @@ impl CollabSession { .get(&AssetId(asset)) .map(|record| record.bytes.as_ref().clone()) .ok_or_else(|| anyhow!("collaboration asset {asset} is not available"))?; + tracing::debug!(session = %self.session, asset, bytes = bytes.len(), "exported collaboration asset bytes"); Ok(AssetBytes { bytes }) } @@ -266,7 +358,7 @@ impl CollabSession { let mut binding = match binding.lock() { Ok(binding) => binding, Err(error) => { - eprintln!("flowstate collab binding lock failed: {error}"); + tracing::error!(error = %error, "collaboration binding lock failed during remote apply"); return; }, }; @@ -282,11 +374,14 @@ impl CollabSession { }; match produced { Ok(mut produced) => { + tracing::trace!(patches = produced.len(), "collaboration remote apply produced patches"); if let Ok(mut patches) = patches.lock() { patches.append(&mut produced); + } else { + tracing::error!("collaboration remote patch lock failed"); } }, - Err(error) => eprintln!("flowstate collab remote apply failed: {error:#}"), + Err(error) => tracing::error!(error = %format_args!("{error:#}"), "collaboration remote apply failed"), } }); subscribed_doc.subscribe_root(callback) @@ -294,32 +389,48 @@ impl CollabSession { pub(super) fn apply_or_queue_patches(&mut self, mut patches: Vec, cx: &mut Context) { if patches.is_empty() { + tracing::trace!(session = %self.session, "no remote collaboration patches to queue"); return; } + for patch in &patches { + trace_collab_patch(self.session, patch); + } + tracing::debug!(session = %self.session, patches = patches.len(), pending_before = self.pending_remote_patches.len(), "queueing remote collaboration patches"); self.pending_remote_patches.append(&mut patches); - self.flush_pending_remote_patches(cx); + let flushed = self.flush_pending_remote_patches(cx); + tracing::trace!(session = %self.session, pending_after = self.pending_remote_patches.len(), flushed, "remote collaboration patch queue updated"); } fn start_update_pull(&mut self, from: flowstate_collab::ids::PeerId, our_vv: Vec, cx: &mut Context) { let (reply_tx, reply_rx) = async_channel::bounded(1); + let candidates = self.pull_candidates(Some(from)); + tracing::debug!(session = %self.session, from = %from, our_vv_bytes = our_vv.len(), candidate_count = candidates.len(), "requesting collaboration update pull"); let send_result = self.net_tx.try_send(NetCommand::PullUpdates { session: self.session, - candidates: self.pull_candidates(Some(from)), + candidates, our_vv, reply: reply_tx, }); - if send_result.is_err() { + if let Err(error) = send_result { + tracing::warn!(session = %self.session, from = %from, error = %error, "queueing collaboration update pull failed"); self.anti_entropy.finish_pull(); return; } + let session_id = self.session; cx.spawn(async move |session, cx| { let result = reply_rx.recv().await; let _ = session.update(cx, |session, cx| { session.anti_entropy.finish_pull(); - if let Ok(Ok(bytes)) = result - && let Err(error) = session.import_update_bytes(&bytes, cx) - { - session.detach(DetachReason::Fatal(format!("pulling collaboration updates failed: {error:#}")), cx); + match result { + Ok(Ok(bytes)) => { + tracing::debug!(session = %session_id, from = %from, bytes = bytes.len(), "collaboration update pull succeeded"); + if let Err(error) = session.import_update_bytes(&bytes, cx) { + tracing::error!(session = %session_id, from = %from, error = %format_args!("{error:#}"), "importing pulled collaboration updates failed"); + session.detach(DetachReason::Fatal(format!("pulling collaboration updates failed: {error:#}")), cx); + } + }, + Ok(Err(error)) => tracing::warn!(session = %session_id, from = %from, error = %format_args!("{error:#}"), "collaboration update pull failed"), + Err(error) => tracing::warn!(session = %session_id, from = %from, error = %error, "collaboration update pull reply channel closed"), } }); }) @@ -335,3 +446,52 @@ fn take_mutex_value(value: Arc>, label: &str) -> Result { Err(_) => Err(anyhow!("collaboration {label} is still referenced")), } } + +fn direct_serve_request_kind(request: &DirectServeRequest) -> &'static str { + match request { + DirectServeRequest::Snapshot { .. } => "snapshot", + DirectServeRequest::Updates { .. } => "updates", + DirectServeRequest::Asset { .. } => "asset", + } +} + +fn gap_action_kind(action: &GapAction) -> &'static str { + match action { + GapAction::None => "none", + GapAction::Pull { .. } => "pull", + GapAction::LineageMismatch { .. } => "lineage_mismatch", + } +} + +fn log_direct_serve_result(session: SessionId, kind: &'static str, result: &Result>) { + match result { + Ok(bytes) => tracing::debug!(%session, kind, bytes = bytes.len(), "served collaboration direct payload"), + Err(error) => tracing::warn!(%session, kind, error = %format_args!("{error:#}"), "serving collaboration direct payload failed"), + } +} + +fn trace_collab_patch(session: SessionId, patch: &CollabPatch) { + match patch { + CollabPatch::ParagraphText { row, delta_utf8, .. } => { + tracing::trace!(%session, patch_kind = "paragraph_text", row, deltas = delta_utf8.len(), "queued collaboration patch"); + }, + CollabPatch::ParagraphStyle { row, style } => { + tracing::trace!(%session, patch_kind = "paragraph_style", row, ?style, "queued collaboration patch"); + }, + CollabPatch::ReplaceObjectBlock { row, block } => { + tracing::trace!(%session, patch_kind = "replace_object_block", row, block_id = ?block.block_id, "queued collaboration patch"); + }, + CollabPatch::InsertBlocks { row, blocks } => { + tracing::trace!(%session, patch_kind = "insert_blocks", row, blocks = blocks.len(), "queued collaboration patch"); + }, + CollabPatch::DeleteBlocks { row, count } => { + tracing::trace!(%session, patch_kind = "delete_blocks", row, count, "queued collaboration patch"); + }, + CollabPatch::MoveBlock { from, to } => { + tracing::trace!(%session, patch_kind = "move_block", from, to, "queued collaboration patch"); + }, + CollabPatch::AssetArrived { id, record } => { + tracing::trace!(%session, patch_kind = "asset_arrived", ?id, bytes = record.bytes.len(), "queued collaboration patch"); + }, + } +} diff --git a/crates/flowstate/src/collab/session_presence.rs b/crates/flowstate/src/collab/session_presence.rs index b9e69d29..8b0413d7 100644 --- a/crates/flowstate/src/collab/session_presence.rs +++ b/crates/flowstate/src/collab/session_presence.rs @@ -6,69 +6,96 @@ use super::{CollabSession, presence_view}; impl CollabSession { pub(super) fn apply_presence(&mut self, bytes: &[u8], cx: &mut Context) { if let Some(presence) = &self.presence { + tracing::trace!(session = %self.session, bytes = bytes.len(), "applying remote collaboration presence update"); if let Err(error) = presence.apply(bytes) { - eprintln!("flowstate collab presence update failed: {error:#}"); + tracing::warn!(session = %self.session, bytes = bytes.len(), error = %format_args!("{error:#}"), "collaboration presence update failed"); } self.refresh_peer_count(); self.evaluate_connectivity(cx); self.refresh_external_carets(cx); cx.notify(); + } else { + tracing::debug!(session = %self.session, bytes = bytes.len(), "ignored collaboration presence update before local presence was established"); } } pub(super) fn refresh_own_presence(&mut self, cx: &mut Context) { let Some(presence) = &self.presence else { + tracing::trace!(session = %self.session, "skipping own collaboration presence refresh because store is missing"); return; }; + let selection = self.own_presence_selection(cx); let state = PresenceState { name: default_presence_name(), - selection: self.own_presence_selection(cx), + selection, }; + tracing::trace!(session = %self.session, has_selection = state.selection.is_some(), "refreshing own collaboration presence"); if let Err(error) = presence.set_self(&state) { - eprintln!("flowstate collab presence encode failed: {error:#}"); + tracing::warn!(session = %self.session, error = %format_args!("{error:#}"), "collaboration presence encode failed"); } self.refresh_peer_count(); } pub(super) fn refresh_external_carets(&mut self, cx: &mut Context) { let Some(presence) = &self.presence else { + tracing::trace!(session = %self.session, "skipping external caret refresh because presence is missing"); return; }; let Some(doc) = &self.doc else { + tracing::trace!(session = %self.session, "skipping external caret refresh because Loro doc is missing"); return; }; let Some(binding) = &self.binding else { + tracing::trace!(session = %self.session, "skipping external caret refresh because binding is missing"); return; }; let Some(editor) = self.editor.clone() else { + tracing::trace!(session = %self.session, "skipping external caret refresh because editor is missing"); return; }; let carets = presence_view::external_carets(doc, binding, presence); + tracing::trace!(session = %self.session, carets = carets.len(), "refreshing collaboration external carets"); editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); } pub(super) fn publish_presence_snapshot(&self) { if let Some(presence) = &self.presence { - self.publish_presence_bytes(presence.encode_all()); + let bytes = presence.encode_all(); + tracing::debug!(session = %self.session, bytes = bytes.len(), "publishing collaboration presence snapshot"); + self.publish_presence_bytes(bytes); + } else { + tracing::trace!(session = %self.session, "skipping collaboration presence snapshot because store is missing"); } } pub(super) fn publish_presence_bytes(&self, bytes: Vec) { if bytes.is_empty() { + tracing::trace!(session = %self.session, "skipping empty collaboration presence publish"); return; } - let _ = self + let byte_len = bytes.len(); + if let Err(error) = self .net_tx .try_send(flowstate_collab::net::NetCommand::Publish { session: self.session, payload: flowstate_collab::net::PublishPayload::Presence(bytes), - }); + }) + { + tracing::warn!(session = %self.session, bytes = byte_len, error = %error, "queueing collaboration presence publish failed"); + } else { + tracing::trace!(session = %self.session, bytes = byte_len, "queued collaboration presence publish"); + } } pub(super) fn refresh_peer_count(&mut self) { let peers_present = self.peers_present(); + let session_id = self.session; if let super::SessionPhase::Attached(attachment) = &mut self.phase { + let previous = attachment.peers_present; attachment.peers_present = peers_present; + if previous != peers_present { + tracing::info!(session = %session_id, previous, peers_present, "collaboration peer presence count changed"); + } } } diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs index 17d69be9..13ccdfdc 100644 --- a/crates/flowstate/src/collab/session_timers.rs +++ b/crates/flowstate/src/collab/session_timers.rs @@ -17,9 +17,11 @@ const RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30); impl CollabSession { pub(super) fn attach_timers(&mut self, cx: &mut Context) { if self.timers_started { + tracing::trace!(session = %self.session, "collaboration timers already started"); return; } self.timers_started = true; + tracing::debug!(session = %self.session, "starting collaboration timers"); self.spawn_digest_timer(cx); self.spawn_presence_timer(cx); self.spawn_self_check_timer(cx); @@ -28,6 +30,7 @@ impl CollabSession { pub(super) fn note_inbound_traffic(&mut self, cx: &mut Context) { self.inbound_since_last_digest = true; self.quiet_digest_rounds = 0; + tracing::trace!(session = %self.session, "noted inbound collaboration network traffic"); if self.mark_online(cx) { self.anti_entropy.on_reconnect(); self.publish_digest(); @@ -44,6 +47,7 @@ impl CollabSession { changed = true; } if changed { + tracing::info!(session = %self.session, "collaboration session marked online"); cx.notify(); } changed @@ -51,24 +55,39 @@ impl CollabSession { pub(super) fn evaluate_connectivity(&mut self, cx: &mut Context) { if !matches!(self.phase, SessionPhase::Attached(_)) { + tracing::trace!(session = %self.session, phase = ?self.phase, "skipping collaboration connectivity evaluation for inactive phase"); return; } self.refresh_peer_count(); if self.peers_present() == 0 || !self.neighbors.is_empty() { + tracing::trace!(session = %self.session, peers_present = self.peers_present(), neighbors = self.neighbors.len(), "collaboration connectivity considered online"); self.mark_online(cx); return; } let now = Instant::now(); - let zero_since = self.zero_neighbors_since.get_or_insert(now); + let peers_present = self.peers_present(); + let neighbors = self.neighbors.len(); + let zero_since = *self.zero_neighbors_since.get_or_insert(now); let quiet_enough = self.quiet_digest_rounds >= QUIET_DIGEST_ROUNDS_OFFLINE || !self.endpoint_online; - if quiet_enough && now.saturating_duration_since(*zero_since) >= ZERO_NEIGHBOR_OFFLINE_GRACE { + tracing::trace!( + session = %self.session, + peers_present, + neighbors, + quiet_rounds = self.quiet_digest_rounds, + quiet_enough, + endpoint_online = self.endpoint_online, + "evaluating collaboration connectivity", + ); + if quiet_enough && now.saturating_duration_since(zero_since) >= ZERO_NEIGHBOR_OFFLINE_GRACE { self.mark_offline(now, cx); } } fn spawn_digest_timer(&mut self, cx: &mut Context) { + let session_id = self.session; + tracing::debug!(session = %session_id, "starting collaboration digest timer"); cx.spawn(async move |session, cx| { loop { let delay = match session.update(cx, |session, _| session.next_digest_delay()) { @@ -81,11 +100,14 @@ impl CollabSession { Ok(false) | Err(_) => break, } } + tracing::debug!(session = %session_id, "collaboration digest timer stopped"); }) .detach(); } fn spawn_presence_timer(&mut self, cx: &mut Context) { + let session_id = self.session; + tracing::debug!(session = %session_id, "starting collaboration presence timer"); cx.spawn(async move |session, cx| { loop { Timer::after(Duration::from_secs(PRESENCE_KEEPALIVE_SECS)).await; @@ -94,11 +116,14 @@ impl CollabSession { Ok(false) | Err(_) => break, } } + tracing::debug!(session = %session_id, "collaboration presence timer stopped"); }) .detach(); } fn spawn_self_check_timer(&mut self, cx: &mut Context) { + let session_id = self.session; + tracing::debug!(session = %session_id, "starting collaboration self-check timer"); cx.spawn(async move |session, cx| { loop { Timer::after(SELF_CHECK_INTERVAL).await; @@ -107,6 +132,7 @@ impl CollabSession { Ok(false) | Err(_) => break, } } + tracing::debug!(session = %session_id, "collaboration self-check timer stopped"); }) .detach(); } @@ -125,6 +151,7 @@ impl CollabSession { let now = Instant::now(); if self.anti_entropy.digest_due(now) { if matches!(self.phase, SessionPhase::Attached(_)) { + tracing::trace!(session = %self.session, quiet_rounds = self.quiet_digest_rounds, inbound_since_last_digest = self.inbound_since_last_digest, "collaboration digest timer fired"); self.publish_digest(); self.finish_digest_round(now, cx); } @@ -138,6 +165,7 @@ impl CollabSession { return false; } if self.presence.is_some() { + tracing::trace!(session = %self.session, "collaboration presence timer fired"); if let Some(presence) = &self.presence { presence.remove_outdated(); } @@ -155,10 +183,12 @@ impl CollabSession { return false; } if !matches!(self.phase, SessionPhase::Attached(_)) || self.last_document_activity.elapsed() < SELF_CHECK_IDLE { + tracing::trace!(session = %self.session, phase = ?self.phase, idle_for = ?self.last_document_activity.elapsed(), "skipping collaboration self-check tick"); return true; } + tracing::trace!(session = %self.session, idle_for = ?self.last_document_activity.elapsed(), "running collaboration self-check tick"); if let Err(error) = self.run_self_check(cx) { - eprintln!("flowstate collab self-check failed: {error:#}"); + tracing::error!(session = %self.session, error = %format_args!("{error:#}"), "collaboration self-check failed"); } true } @@ -170,6 +200,7 @@ impl CollabSession { self.quiet_digest_rounds = self.quiet_digest_rounds.saturating_add(1); } self.inbound_since_last_digest = false; + tracing::trace!(session = %self.session, quiet_rounds = self.quiet_digest_rounds, "finished collaboration digest round"); self.evaluate_connectivity(cx); self.run_recovery_if_due(now, cx); } @@ -195,13 +226,18 @@ impl CollabSession { } if attempted { + tracing::warn!(session = %self.session, bootstrap_count = self.bootstrap_addrs.len(), "attempting collaboration connectivity recovery"); if self.bootstrap_addrs.is_empty() { - let _ = self.net_tx.try_send(NetCommand::EnsureUp); + if let Err(error) = self.net_tx.try_send(NetCommand::EnsureUp) { + tracing::warn!(session = %self.session, error = %error, "queueing collaboration ensure-up recovery failed"); + } } else { - let _ = self.net_tx.try_send(NetCommand::JoinSession { + if let Err(error) = self.net_tx.try_send(NetCommand::JoinSession { session: self.session, bootstrap: self.bootstrap_addrs.clone(), - }); + }) { + tracing::warn!(session = %self.session, error = %error, "queueing collaboration join-session recovery failed"); + } } self.publish_digest(); cx.notify(); @@ -222,12 +258,16 @@ impl CollabSession { projected.assets = live_document.assets.clone(); let projected_hash = self_check::projection_hash(&projected); if live_hash == projected_hash { + tracing::trace!(session = %self.session, live_hash = %format_args!("{live_hash:016x}"), "collaboration projection self-check passed"); return Ok(()); } - eprintln!( - "flowstate collab projection drift detected: live_hash={live_hash:016x} projected_hash={projected_hash:016x} vv_bytes={}", - doc.oplog_vv().encode().len() + tracing::error!( + session = %self.session, + live_hash = %format_args!("{live_hash:016x}"), + projected_hash = %format_args!("{projected_hash:016x}"), + vv_bytes = doc.oplog_vv().encode().len(), + "collaboration projection drift detected", ); self.rebuild_from_projection(projected, cx) } @@ -239,20 +279,27 @@ impl CollabSession { let Some(editor) = self.editor.clone() else { return Ok(()); }; + tracing::warn!(session = %self.session, paragraphs = projected.paragraphs.len(), blocks = projected.blocks.len(), "rebuilding editor document from collaboration projection"); editor.update(cx, |editor, cx| editor.replace_document_from_collaboration(projected, cx)); let document = editor.read(cx).document().clone(); self.binding = Some(DocBinding::build(doc, &document).context("rebuilding collaboration binding after self-check failed")?); self.last_document_activity = Instant::now(); self.refresh_external_carets(cx); + tracing::info!(session = %self.session, "rebuilt editor document from collaboration projection"); Ok(()) } fn mark_offline(&mut self, now: Instant, cx: &mut Context) { + let session_id = self.session; + let quiet_rounds = self.quiet_digest_rounds; + let neighbors = self.neighbors.len(); if let SessionPhase::Attached(attachment) = &mut self.phase && matches!(attachment.connectivity, Connectivity::Online) { + let peers_present = attachment.peers_present; attachment.connectivity = Connectivity::Offline { since: now, retries: 0 }; self.next_recovery_at = Some(now); + tracing::warn!(session = %session_id, quiet_rounds, neighbors, peers_present, "collaboration session marked offline"); cx.notify(); } } diff --git a/crates/flowstate/src/collab/shutdown.rs b/crates/flowstate/src/collab/shutdown.rs index 6db912e0..2c2d3e7e 100644 --- a/crates/flowstate/src/collab/shutdown.rs +++ b/crates/flowstate/src/collab/shutdown.rs @@ -5,13 +5,17 @@ use super::CollabManager; impl CollabManager { pub(super) fn shutdown_runtime(&mut self) { - if let Some(runtime) = self.runtime.take() { - let _ = runtime.commands.try_send(NetCommand::Shutdown); + tracing::info!(open_sessions = self.sessions_by_id.len(), runtime_running = self.runtime.is_some(), "shutting down collaboration runtime"); + if let Some(runtime) = self.runtime.take() + && let Err(error) = runtime.commands.try_send(NetCommand::Shutdown) + { + tracing::warn!(error = %error, "queueing collaboration runtime shutdown failed"); } self.sessions_by_id.clear(); self.session_by_panel.clear(); self.event_pump_started = false; self.endpoint_online = false; + tracing::debug!("collaboration runtime shutdown state cleared"); } } diff --git a/crates/flowstate/src/lib.rs b/crates/flowstate/src/lib.rs index b65baa3c..c422958d 100644 --- a/crates/flowstate/src/lib.rs +++ b/crates/flowstate/src/lib.rs @@ -11,6 +11,7 @@ pub mod commands; pub mod docx_conversion; pub mod file_search; pub mod flow; +pub mod logging; pub mod ribbon; pub mod rich_text_element; pub mod workspace; diff --git a/crates/flowstate/src/logging.rs b/crates/flowstate/src/logging.rs new file mode 100644 index 00000000..887fa11d --- /dev/null +++ b/crates/flowstate/src/logging.rs @@ -0,0 +1,65 @@ +use std::{env, path::{Path, PathBuf}}; + +use anyhow::{Context as _, Result}; +use tracing_appender::non_blocking::WorkerGuard; +use tracing_subscriber::{EnvFilter, layer::SubscriberExt as _, util::SubscriberInitExt as _}; + +use crate::app_settings::flowstate_data_dir; + +const DEFAULT_LOG_FILTER: &str = "flowstate::collab=trace,flowstate::workspace::workspace=trace,flowstate_collab=trace"; + +pub struct LoggingGuard { + _guard: WorkerGuard, + directory: PathBuf, +} + +impl LoggingGuard { + pub fn directory(&self) -> &Path { + &self.directory + } +} + +pub fn init() -> Result { + let directory = log_directory(); + std::fs::create_dir_all(&directory) + .with_context(|| format!("creating log directory {} failed", directory.display()))?; + let file_appender = tracing_appender::rolling::daily(&directory, "flowstate.log"); + let (writer, guard) = tracing_appender::non_blocking(file_appender); + + tracing_subscriber::registry() + .with(env_filter()) + .with( + tracing_subscriber::fmt::layer() + .with_writer(writer) + .with_ansi(false) + .with_file(true) + .with_line_number(true) + .with_target(true) + .with_thread_ids(true) + .with_thread_names(true), + ) + .try_init() + .context("initializing flowstate logging failed")?; + + Ok(LoggingGuard { + _guard: guard, + directory, + }) +} + +fn env_filter() -> EnvFilter { + let filter = env::var("FLOWSTATE_LOG") + .or_else(|_| env::var("RUST_LOG")) + .unwrap_or_else(|_| DEFAULT_LOG_FILTER.to_string()); + + EnvFilter::try_new(&filter).unwrap_or_else(|error| { + eprintln!("invalid FLOWSTATE_LOG/RUST_LOG filter {filter:?}: {error}; falling back to {DEFAULT_LOG_FILTER:?}"); + EnvFilter::new(DEFAULT_LOG_FILTER) + }) +} + +fn log_directory() -> PathBuf { + env::var_os("FLOWSTATE_LOG_DIR") + .map(PathBuf::from) + .unwrap_or_else(|| flowstate_data_dir().join("logs")) +} diff --git a/crates/flowstate/src/main.rs b/crates/flowstate/src/main.rs index 2af3b150..d6a545ba 100644 --- a/crates/flowstate/src/main.rs +++ b/crates/flowstate/src/main.rs @@ -7,6 +7,7 @@ use clap::{Parser, Subcommand}; use flowstate::{ docx_conversion::{convert_db8_to_docx, convert_db8_to_pdf, convert_docx_to_pdf, convert_pdf_to_db8}, + logging, run_standalone, write_demo_document, }; @@ -94,6 +95,14 @@ enum CliCommand { #[hotpath::main(allocator = FlowstateAllocator)] fn main() { + let _logging_guard = match logging::init() { + Ok(guard) => Some(guard), + Err(error) => { + eprintln!("flowstate logging initialization failed: {error:#}"); + None + }, + }; + let cli = Cli::parse(); if cli.write_demo_db8 { diff --git a/crates/flowstate/src/workspace/workspace/collab.rs b/crates/flowstate/src/workspace/workspace/collab.rs index e288eda0..d5dae7ea 100644 --- a/crates/flowstate/src/workspace/workspace/collab.rs +++ b/crates/flowstate/src/workspace/workspace/collab.rs @@ -44,10 +44,14 @@ impl Workspace { .map(|panel| panel.read(cx).title_text().to_string()) .unwrap_or_else(|| "Shared document".to_string()); + tracing::info!(%panel_id, title = %title, "workspace starting collaboration on document"); match crate::collab::start_session_for_panel(panel_id, editor, title, cx) { - Ok(session) => Some(session), + Ok(session) => { + tracing::info!(%panel_id, %session, "workspace started collaboration on document"); + Some(session) + }, Err(error) => { - eprintln!("starting collaboration session failed: {error:#}"); + tracing::error!(%panel_id, error = %format_args!("{error:#}"), "starting collaboration session failed"); None }, } @@ -95,6 +99,7 @@ impl Workspace { let result = ticket_rx.recv().await; let _ = window_handle.update(cx, |_, window, cx| match result { Ok(Ok(ticket)) => { + tracing::info!(session = %ticket.session, inviter = %ticket.inviter.id, "copied collaboration invite ticket to clipboard"); cx.write_to_clipboard(gpui::ClipboardItem::new_string(ticket.encode_text())); std::mem::drop(window.prompt( PromptLevel::Info, @@ -105,10 +110,12 @@ impl Workspace { )); }, Ok(Err(error)) => { + tracing::error!(error = %format_args!("{error:#}"), "creating collaboration invite failed"); let detail = format!("Creating collaboration invite failed: {error:#}"); std::mem::drop(window.prompt(PromptLevel::Critical, "Share failed", Some(&detail), &[PromptButton::ok("Ok")], cx)); }, Err(error) => { + tracing::error!(error = %error, "collaboration invite receiver closed"); let detail = format!("Creating collaboration invite failed: {error}"); std::mem::drop(window.prompt(PromptLevel::Critical, "Share failed", Some(&detail), &[PromptButton::ok("Ok")], cx)); }, @@ -120,6 +127,7 @@ impl Workspace { pub fn join_collaboration_from_clipboard(&mut self, window: &mut Window, cx: &mut Context) -> bool { let Some(item) = cx.read_from_clipboard() else { + tracing::warn!("join collaboration from clipboard failed because clipboard is empty"); std::mem::drop(window.prompt( PromptLevel::Critical, "Join failed", @@ -130,6 +138,7 @@ impl Workspace { return true; }; let Some(text) = item.text() else { + tracing::warn!("join collaboration from clipboard failed because clipboard has no text"); std::mem::drop(window.prompt( PromptLevel::Critical, "Join failed", @@ -142,11 +151,13 @@ impl Workspace { let ticket = match flowstate_collab::ticket::SessionTicket::decode_text(&text) { Ok(ticket) => ticket, Err(error) => { + tracing::warn!(bytes = text.len(), error = %format_args!("{error:#}"), "clipboard collaboration invite decode failed"); let detail = format!("The clipboard text is not a valid collaboration invite: {error:#}"); std::mem::drop(window.prompt(PromptLevel::Critical, "Join failed", Some(&detail), &[PromptButton::ok("Ok")], cx)); return true; }, }; + tracing::info!(session = %ticket.session, inviter = %ticket.inviter.id, "joining collaboration session from clipboard"); self.join_collaboration_session(ticket, window, cx).is_some() } @@ -192,7 +203,9 @@ impl Workspace { } pub fn leave_collaboration_on_panel(&mut self, panel_id: Uuid, cx: &mut Context) -> bool { - crate::collab::leave_session_for_panel(panel_id, cx) + let left = crate::collab::leave_session_for_panel(panel_id, cx); + tracing::info!(%panel_id, left, "workspace leave collaboration requested"); + left } pub fn join_collaboration_session( @@ -201,10 +214,11 @@ impl Workspace { window: &mut Window, cx: &mut Context, ) -> Option { + tracing::info!(session = %ticket.session, inviter = %ticket.inviter.id, "workspace joining collaboration session"); let request = match crate::collab::join_session(ticket, cx) { Ok(request) => request, Err(error) => { - eprintln!("joining collaboration session failed: {error:#}"); + tracing::error!(error = %format_args!("{error:#}"), "joining collaboration session failed"); return None; }, }; @@ -218,10 +232,12 @@ impl Workspace { let _ = workspace.update(cx, |workspace, cx| { match result { Ok(Ok(joined)) => { + tracing::info!(session = %joined.session, title = %joined.title, "collaboration join completed; opening joined document"); let panel = workspace.add_joined_collaboration_panel(joined.document, joined.title, window, cx); let panel_id = panel.read(cx).id(); let editor = panel.read(cx).editor(); if let Err(error) = crate::collab::attach_joined_session(joined.session, panel_id, editor, cx) { + tracing::error!(session = %joined.session, %panel_id, error = %format_args!("{error:#}"), "collaboration joined document attachment failed"); let detail = format!("Joined document opened, but collaboration attachment failed: {error:#}"); std::mem::drop(window.prompt( PromptLevel::Critical, @@ -233,6 +249,7 @@ impl Workspace { } }, Ok(Err(error)) => { + tracing::error!(%session, error = %format_args!("{error:#}"), "joining collaboration session failed"); let detail = format!("Joining collaboration session failed: {error:#}"); std::mem::drop(window.prompt( PromptLevel::Critical, @@ -243,6 +260,7 @@ impl Workspace { )); }, Err(error) => { + tracing::error!(%session, error = %error, "collaboration join completion channel closed"); let detail = format!("Joining collaboration session failed: {error}"); std::mem::drop(window.prompt( PromptLevel::Critical, From c8375bd74afcdf9ad429f3292deaf577cf602414 Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Mon, 15 Jun 2026 22:29:06 -0500 Subject: [PATCH 13/45] new thing --- crates/flowstate/src/collab/manager.rs | 71 +++++++++++++++++++++++--- crates/flowstate/src/collab/pump.rs | 15 ++++++ 2 files changed, 78 insertions(+), 8 deletions(-) diff --git a/crates/flowstate/src/collab/manager.rs b/crates/flowstate/src/collab/manager.rs index 7280233c..4cc9de87 100644 --- a/crates/flowstate/src/collab/manager.rs +++ b/crates/flowstate/src/collab/manager.rs @@ -49,14 +49,18 @@ impl CollabManager { } pub(super) fn shutdown_runtime(&mut self) { - if let Some(runtime) = self.runtime.take() { - let _ = runtime.commands.try_send(NetCommand::Shutdown); + tracing::info!(open_sessions = self.sessions_by_id.len(), runtime_running = self.runtime.is_some(), "shutting down collaboration runtime"); + if let Some(runtime) = self.runtime.take() + && let Err(error) = runtime.commands.try_send(NetCommand::Shutdown) + { + tracing::warn!(error = %error, "queueing collaboration runtime shutdown failed"); } self.sessions_by_id.clear(); self.session_by_panel.clear(); self.own_endpoint_id = None; self.event_pump_started = false; self.endpoint_online = false; + tracing::debug!("collaboration runtime shutdown state cleared"); } pub fn start_session_for_panel( @@ -70,12 +74,22 @@ impl CollabManager { T: 'static, { if let Some(session) = self.session_by_panel.get(&panel_id).copied() { + tracing::debug!(%panel_id, %session, "collaboration session already exists for panel"); return Ok(session); } let commands = self.ensure_runtime(cx)?; let session = SessionId::new(); let document = editor.read(cx).document().clone(); + tracing::info!( + %panel_id, + %session, + title = %title, + paragraphs = document.paragraphs.len(), + blocks = document.blocks.len(), + assets = document.assets.assets.len(), + "starting local collaboration session", + ); let collab = CollabSession::from_local_document(session, panel_id, editor, title, document, commands.clone())?; let direct_handler = collab.direct_handler(); let entity = cx.new(|_| collab); @@ -86,17 +100,21 @@ impl CollabManager { session, handler: direct_handler, }) { + tracing::error!(%panel_id, %session, error = %error, "registering collaboration direct handler failed"); Self::detach_entity(&entity, DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), cx); self.unregister_session(session); return Err(error).context("registering collaboration direct handler failed"); } + tracing::debug!(%panel_id, %session, "registered collaboration direct handler for local session"); let (reply_tx, reply_rx) = async_channel::bounded(1); if let Err(error) = commands.try_send(NetCommand::CreateSession { session, reply: reply_tx }) { + tracing::error!(%panel_id, %session, error = %error, "queueing collaboration create-session command failed"); Self::detach_entity(&entity, DetachReason::Fatal(format!("creating collaboration network session failed: {error}")), cx); self.unregister_session(session); return Err(error).context("creating collaboration network session failed"); } + tracing::debug!(%panel_id, %session, "queued collaboration create-session command"); Self::finish_create_session(entity, reply_rx, cx); Ok(session) } @@ -105,6 +123,10 @@ impl CollabManager { where T: 'static, { + tracing::info!(session = %ticket.session, inviter = %ticket.inviter.id, title = %ticket.title, "joining collaboration session from ticket"); + if !ticket.is_supported_version() { + tracing::warn!(session = %ticket.session, "unsupported collaboration protocol version in ticket"); + } ensure!(ticket.is_supported_version(), "unsupported collaboration protocol version"); let commands = self.ensure_runtime(cx)?; commands @@ -114,6 +136,9 @@ impl CollabManager { self.own_endpoint_id != Some(ticket.inviter.id), "That's your own invite — open the Share dialog instead." ); + if self.sessions_by_id.contains_key(&ticket.session) { + tracing::warn!(session = %ticket.session, "collaboration session is already open"); + } ensure!( !self.sessions_by_id.contains_key(&ticket.session), "collaboration session is already open" @@ -130,6 +155,7 @@ impl CollabManager { session, bootstrap: vec![ticket.inviter], }) { + tracing::error!(%session, error = %error, "queueing collaboration join-session command failed"); entity.update(cx, |session, cx| { session.detach( DetachReason::JoinFailed(format!("joining collaboration network session failed: {error}")), @@ -139,6 +165,7 @@ impl CollabManager { self.unregister_session(session); return Err(error).context("joining collaboration network session failed"); } + tracing::debug!(%session, inviter = %inviter, "queued collaboration join-session command"); let completed = entity.update(cx, |session, cx| session.begin_join_bootstrap(inviter, neighbor_rx, cx)); Ok(JoinRequest { session, completed }) } @@ -153,6 +180,7 @@ impl CollabManager { where T: 'static, { + tracing::info!(%session_id, %panel_id, "attaching joined collaboration session to editor"); let commands = self.ensure_runtime(cx)?; let session = self .sessions_by_id @@ -164,10 +192,12 @@ impl CollabManager { session: session_id, handler: direct_handler, }) { + tracing::error!(%session_id, %panel_id, error = %error, "registering collaboration direct handler for joined session failed"); Self::detach_entity(&session, DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), cx); self.unregister_session(session_id); return Err(error).context("registering collaboration direct handler failed"); } + tracing::debug!(%session_id, %panel_id, "registered collaboration direct handler for joined session"); if let Err(error) = Self::establish_joined_peer(session.clone(), commands, cx) { Self::detach_entity(&session, DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error:#}")), cx); self.unregister_session(session_id); @@ -187,13 +217,16 @@ impl CollabManager { T: 'static, { let Some(session_id) = self.session_by_panel.get(&panel_id).copied() else { + tracing::debug!(%panel_id, "leave requested for panel without collaboration session"); return false; }; let Some(session) = self.sessions_by_id.get(&session_id).cloned() else { + tracing::warn!(%panel_id, %session_id, "leave requested for missing registered collaboration session"); self.unregister_session(session_id); return false; }; + tracing::info!(%panel_id, %session_id, "leaving collaboration session for panel"); let _ = session.update(cx, |session, cx| session.detach(DetachReason::UserLeft, cx)); self.unregister_session(session_id); true @@ -209,16 +242,24 @@ impl CollabManager { let commands = self.ensure_runtime(cx).ok()?; let (ticket_tx, ticket_rx) = async_channel::bounded(1); let (reply_tx, reply_rx) = async_channel::bounded(1); - commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }).ok()?; + if let Err(error) = commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }) { + tracing::error!(%panel_id, %session_id, error = %error, "queueing collaboration ticket address command failed"); + return None; + } + tracing::debug!(%panel_id, %session_id, "requested collaboration ticket address"); cx.spawn(async move |_, cx| { let ticket = match reply_rx.recv().await { Ok(addr) => { + tracing::info!(%session_id, inviter = %addr.id, "created collaboration share ticket"); let own_endpoint_id = addr.id; let _ = cx.update_global::(|manager, _| manager.own_endpoint_id = Some(own_endpoint_id)); Ok(SessionTicket::new(session_id, addr, title)) }, - Err(error) => Err(anyhow!("collaboration endpoint address unavailable: {error}")), + Err(error) => { + tracing::error!(%session_id, error = %error, "collaboration endpoint address unavailable for ticket"); + Err(anyhow!("collaboration endpoint address unavailable: {error}")) + }, }; let _ = ticket_tx.send(ticket).await; }) @@ -261,13 +302,16 @@ impl CollabManager { let id = session.read(cx).session_id(); if let Some(panel_id) = session.read(cx).panel_id() { self.session_by_panel.insert(panel_id, id); + tracing::debug!(%id, %panel_id, "registered collaboration session panel mapping"); } self.sessions_by_id.insert(id, session); + tracing::debug!(%id, open_sessions = self.sessions_by_id.len(), "registered collaboration session"); } pub(super) fn unregister_session(&mut self, session: SessionId) { - self.sessions_by_id.remove(&session); + let removed = self.sessions_by_id.remove(&session).is_some(); self.session_by_panel.retain(|_, active| *active != session); + tracing::debug!(%session, removed, open_sessions = self.sessions_by_id.len(), "unregistered collaboration session"); } fn ensure_runtime(&mut self, cx: &mut Context) -> Result @@ -275,9 +319,11 @@ impl CollabManager { T: 'static, { if let Some(runtime) = &self.runtime { + tracing::trace!("collaboration runtime already available"); return Ok(runtime.commands.clone()); } + tracing::info!("starting collaboration runtime for manager"); let (commands, events) = runtime::start()?; self.runtime = Some(CollabRuntime { commands: commands.clone() }); self.start_event_pump(events, cx); @@ -288,18 +334,22 @@ impl CollabManager { where T: 'static, { + let session_id = session.read(cx).session_id(); cx.spawn(async move |_, cx| match reply_rx.recv().await { Ok(Ok(seed)) => { + tracing::info!(%session_id, inviter = %seed.inviter.id, "collaboration create-session completed"); let own_endpoint_id = seed.inviter.id; let _ = cx.update_global::(|manager, _| manager.own_endpoint_id = Some(own_endpoint_id)); let _ = session.update(cx, |session, cx| session.establish_local_peer(&seed.inviter.id, cx)); }, Ok(Err(error)) => { + tracing::error!(%session_id, error = %format_args!("{error:#}"), "collaboration create-session failed"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error:#}")), cx); }); }, Err(error) => { + tracing::error!(%session_id, error = %error, "collaboration create-session reply channel closed"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::Fatal(format!("creating collaboration session failed: {error}")), cx); }); @@ -312,17 +362,22 @@ impl CollabManager { where T: 'static, { + let session_id = session.read(cx).session_id(); let (reply_tx, reply_rx) = async_channel::bounded(1); - commands - .try_send(NetCommand::MintTicketAddr { reply: reply_tx }) - .context("requesting collaboration endpoint address failed")?; + if let Err(error) = commands.try_send(NetCommand::MintTicketAddr { reply: reply_tx }) { + tracing::error!(%session_id, error = %error, "queueing collaboration endpoint address command failed"); + return Err(error).context("requesting collaboration endpoint address failed"); + } + tracing::debug!(%session_id, "requested collaboration endpoint address for joined session"); cx.spawn(async move |_, cx| match reply_rx.recv().await { Ok(addr) => { + tracing::info!(%session_id, peer = %addr.id, "collaboration joined session local peer established"); let own_endpoint_id = addr.id; let _ = cx.update_global::(|manager, _| manager.own_endpoint_id = Some(own_endpoint_id)); let _ = session.update(cx, |session, cx| session.establish_local_peer(&addr.id, cx)); }, Err(error) => { + tracing::error!(%session_id, error = %error, "collaboration endpoint address unavailable for joined session"); let _ = session.update(cx, |session, cx| { session.detach(DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error}")), cx); }); diff --git a/crates/flowstate/src/collab/pump.rs b/crates/flowstate/src/collab/pump.rs index 0c001bed..29dbaf14 100644 --- a/crates/flowstate/src/collab/pump.rs +++ b/crates/flowstate/src/collab/pump.rs @@ -15,18 +15,22 @@ impl CollabManager { T: 'static, { if self.event_pump_started { + tracing::trace!("collaboration network event pump already started"); return; } self.event_pump_started = true; + tracing::debug!("starting collaboration network event pump"); cx.spawn(async move |_, cx| { while let Ok(event) = events.recv().await { if cx .update_global::(|manager, cx| manager.handle_net_event(event, cx)) .is_err() { + tracing::warn!("collaboration network event pump could not update manager; stopping"); break; } } + tracing::debug!("collaboration network event pump stopped"); }) .detach(); } @@ -34,27 +38,34 @@ impl CollabManager { fn handle_net_event(&mut self, event: NetEvent, cx: &mut App) { match event { NetEvent::EndpointOnline(online) => { + tracing::info!(online, "collaboration endpoint status changed"); self.endpoint_online = online; for session in self.sessions_by_id.values() { session.update(cx, |session, cx| session.set_endpoint_online(online, cx)); } }, NetEvent::Gossip { session, from, msg } => { + tracing::trace!(%session, from = %from, gossip_kind = msg.kind(), gossip_payload_bytes = msg.payload_len(), "routing collaboration gossip event"); self.update_session(session, cx, |session, cx| session.handle_gossip(from, msg, cx)); }, NetEvent::IncompatibleVersion { session, peer } => { + tracing::warn!(%session, peer = %peer, "routing collaboration incompatible-version event"); self.update_session(session, cx, |session, cx| session.handle_incompatible_version(peer, cx)); }, NetEvent::NeighborUp { session, peer } => { + tracing::debug!(%session, peer = %peer, "routing collaboration neighbor-up event"); self.update_session(session, cx, |session, cx| session.neighbor_up(peer, cx)); }, NetEvent::NeighborDown { session, peer } => { + tracing::debug!(%session, peer = %peer, "routing collaboration neighbor-down event"); self.update_session(session, cx, |session, cx| session.neighbor_down(peer, cx)); }, NetEvent::GossipLagged { session } => { + tracing::warn!(%session, "routing collaboration gossip-lagged event"); self.update_session(session, cx, |session, cx| session.handle_gossip_lagged(cx)); }, NetEvent::SubscribeFailed { session, error } => { + tracing::error!(%session, error = %error, "collaboration gossip subscription failed"); let detached = self.update_session(session, cx, |session, cx| session.detach(DetachReason::Fatal(error), cx)); if detached.unwrap_or(false) { self.unregister_session(session); @@ -74,5 +85,9 @@ impl CollabManager { .get(&session) .cloned() .map(|session| session.update(cx, update)) + .or_else(|| { + tracing::warn!(%session, "collaboration event targeted missing session"); + None + }) } } From 1388ae7eca419eb7f0ad1d70e6cafe5e6bff204b Mon Sep 17 00:00:00 2001 From: Chinmay Khaladkar Date: Tue, 16 Jun 2026 19:33:36 -0700 Subject: [PATCH 14/45] testing xhigh fixes --- crates/flowstate-collab/src/binding.rs | 24 +- crates/flowstate-collab/src/local_apply.rs | 421 ++++++------------ crates/flowstate-collab/src/net/swarm.rs | 41 +- crates/flowstate-collab/src/patch_apply.rs | 10 + crates/flowstate-collab/src/projection.rs | 93 ++-- crates/flowstate-collab/src/remote_apply.rs | 252 +++++++---- crates/flowstate-collab/src/schema.rs | 110 ++++- crates/flowstate-collab/tests/convergence.rs | 81 +++- crates/flowstate-collab/tests/projection.rs | 11 +- crates/flowstate-collab/tests/remote_apply.rs | 2 +- crates/flowstate/src/collab/manager.rs | 27 +- crates/flowstate/src/collab/notify.rs | 11 +- crates/flowstate/src/collab/presence_view.rs | 54 +-- crates/flowstate/src/collab/session.rs | 17 +- crates/flowstate/src/collab/session_io.rs | 3 + .../flowstate/src/collab/session_presence.rs | 34 +- crates/flowstate/src/collab/session_timers.rs | 8 +- crates/gpui-flowtext/src/collaboration.rs | 5 + .../src/rich_text/benchmarks/layout.rs | 3 +- .../src/rich_text/editor/collab_apply.rs | 18 +- .../src/rich_text/editor/commands.rs | 3 + .../src/rich_text/editor/lifecycle.rs | 29 ++ .../gpui-flowtext/src/rich_text/editor/mod.rs | 7 +- .../src/rich_text/editor/object_selection.rs | 5 + .../src/rich_text/editor/platform.rs | 8 + .../editor/table_equation_editing.rs | 10 + crates/gpui-flowtext/src/rich_text/element.rs | 6 +- crates/gpui-flowtext/src/rich_text/paint.rs | 6 +- .../src/rich_text/tests/collab_capture.rs | 82 ++++ 29 files changed, 851 insertions(+), 530 deletions(-) diff --git a/crates/flowstate-collab/src/binding.rs b/crates/flowstate-collab/src/binding.rs index 9eac1bd9..8696e39a 100644 --- a/crates/flowstate-collab/src/binding.rs +++ b/crates/flowstate-collab/src/binding.rs @@ -4,7 +4,7 @@ use std::collections::HashMap; use anyhow::{Context as _, Result, bail}; use gpui_flowtext::{Block, BlockId, Document, ParagraphId}; -use loro::{Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroText, ValueOrContainer}; +use loro::{Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, ValueOrContainer}; use crate::schema::BLOCKS; @@ -19,7 +19,6 @@ pub enum BlockKind { #[derive(Clone, Debug)] pub struct BindingRow { pub map: LoroMap, - pub text: Option, pub kind: BlockKind, pub block_id: BlockId, pub paragraph_id: Option, @@ -68,14 +67,8 @@ impl DocBinding { .get(row_ix) .copied() .context("document is missing a block id for a block row")?; - let text = if matches!(kind, BlockKind::Paragraph) { - Some(text_child(&map)?) - } else { - None - }; binding.push_row(BindingRow { map, - text, kind, block_id, paragraph_id, @@ -152,9 +145,6 @@ impl DocBinding { for (ix, row) in self.rows.iter().enumerate() { by_block.insert(row.block_id, ix); by_container.insert(row.map.id(), ix); - if let Some(text) = &row.text { - by_container.insert(text.id(), ix); - } if let Some(paragraph_id) = row.paragraph_id { by_paragraph.insert(paragraph_id, ix); } @@ -167,9 +157,6 @@ impl DocBinding { fn index_row(&mut self, ix: usize, row: &BindingRow) { self.by_block.insert(row.block_id, ix); self.by_container.insert(row.map.id(), ix); - if let Some(text) = &row.text { - self.by_container.insert(text.id(), ix); - } if let Some(paragraph_id) = row.paragraph_id { self.by_paragraph.insert(paragraph_id, ix); } @@ -206,12 +193,3 @@ fn map_at(blocks: &LoroMovableList, ix: usize) -> Result { }, } } - -fn text_child(map: &LoroMap) -> Result { - match map.get(crate::schema::TEXT) { - Some(ValueOrContainer::Container(Container::Text(text))) => Ok(text), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("paragraph block map is missing its text container") - }, - } -} diff --git a/crates/flowstate-collab/src/local_apply.rs b/crates/flowstate-collab/src/local_apply.rs index 2fec20f0..a1b64dc5 100644 --- a/crates/flowstate-collab/src/local_apply.rs +++ b/crates/flowstate-collab/src/local_apply.rs @@ -1,18 +1,17 @@ //! Translation from flowtext canonical operations into Loro operations. use anyhow::{Context as _, Result, bail}; -use gpui_flowtext::{ - Block, BlockId, CanonicalOperation, Document, DocumentSpan, Paragraph, ParagraphId, RunStyles, TextRun, paragraph_text, paragraph_text_len, -}; -use loro::{CommitOptions, LoroDoc, LoroMap, LoroMovableList, LoroText, LoroValue, TextDelta, UpdateOptions, ValueOrContainer, cursor::PosType}; +use std::ops::Range; + +use gpui_flowtext::{Block, BlockId, CanonicalOperation, Document, DocumentSpan, ParagraphId, RunStyles, full_document_text}; +use loro::{CommitOptions, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer}; use crate::{ binding::{BindingRow, BlockKind, DocBinding, block_version}, - projection::{insert_object_container, insert_paragraph_container, replace_blocks_from_document}, + projection::{insert_object_container, insert_paragraph_container, replace_blocks_from_document, replace_body_from_document}, schema::{ - BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_TABLE, MARK_HIGHLIGHT, MARK_SEMANTIC, MARK_STRIKE, MARK_UNDERLINE, REV, - STYLE, apply_mark_intervals, debug_assert_paragraph_text_len, decode_paragraph_style, encode_paragraph_style, mark_intervals_from_runs, - payload_from_block, run_styles_from_attrs, set_run_styles_utf8, unmark_utf8, + BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_TABLE, REV, STYLE, body_text, decode_paragraph_style, + encode_paragraph_style, payload_from_block, set_paragraph_style_utf8, set_run_styles_utf8, text_content, }, }; @@ -59,21 +58,19 @@ impl LocalApplier<'_> { CanonicalOperation::SetParagraphStyle { paragraph, style } => { let row = self.row_for_paragraph(*paragraph)?; row.map.insert(STYLE, encode_paragraph_style(*style))?; + let range = self.body_range_for_paragraph(*paragraph)?; + set_paragraph_style_utf8(&body_text(self.doc), range, *style)?; Ok(()) }, CanonicalOperation::SetRunStyles { paragraph, range, styles } => { - let row = self.row_for_paragraph(*paragraph)?; - let text = row - .text - .as_ref() - .context("paragraph row is missing its LoroText")?; - set_run_styles_utf8(text, range.clone(), *styles)?; - self.debug_assert_row_text_len(document, self.row_ix_for_paragraph(*paragraph)?); + let start = self.body_byte_for_paragraph_byte(*paragraph, range.start)?; + let end = self.body_byte_for_paragraph_byte(*paragraph, range.end)?; + set_run_styles_utf8(&body_text(self.doc), start..end, *styles)?; Ok(()) }, CanonicalOperation::InsertBlock { block, block_ix } => self.insert_block(document, *block, *block_ix), - CanonicalOperation::DeleteBlock { block } => self.delete_block(*block), - CanonicalOperation::MoveBlock { block, new_block_ix } => self.move_block(*block, *new_block_ix), + CanonicalOperation::DeleteBlock { block } => self.delete_block(document, *block), + CanonicalOperation::MoveBlock { block, new_block_ix } => self.move_block(document, *block, *new_block_ix), CanonicalOperation::ReplaceParagraphSpan { start_paragraph, before, @@ -85,19 +82,11 @@ impl LocalApplier<'_> { } fn insert_text(&mut self, document: &Document, paragraph: ParagraphId, byte: usize, value: &str, styles: RunStyles) -> Result<()> { - let row_ix = self.row_ix_for_paragraph(paragraph)?; - let row = self - .binding - .rows - .get(row_ix) - .context("DocBinding paragraph index is out of bounds")?; - let text = row - .text - .as_ref() - .context("paragraph row is missing its LoroText")?; - text.insert_utf8(byte, value)?; - set_run_styles_utf8(text, byte..byte + value.len(), styles)?; - self.debug_assert_row_text_len(document, row_ix); + let byte = self.body_byte_for_paragraph_byte(paragraph, byte)?; + let body = body_text(self.doc); + body.insert_utf8(byte, value)?; + set_run_styles_utf8(&body, byte..byte + value.len(), styles)?; + self.debug_assert_body_matches_document(document); Ok(()) } @@ -111,39 +100,37 @@ impl LocalApplier<'_> { ) -> Result<()> { let start_ix = self.row_ix_for_paragraph(start_paragraph)?; let end_ix = self.row_ix_for_paragraph(end_paragraph)?; + let start_byte = self.body_byte_for_paragraph_byte(start_paragraph, start)?; + let end_byte = self.body_byte_for_paragraph_byte(end_paragraph, end)?; + if end_byte > start_byte { + body_text(self.doc).delete_utf8(start_byte, end_byte - start_byte)?; + } if start_ix == end_ix { - self.delete_text_at_row(start_ix, start, end)?; - self.debug_assert_row_text_len(document, start_ix); + self.debug_assert_body_matches_document(document); return Ok(()); } if start_ix > end_ix { bail!("cross-paragraph delete start row is after end row"); } - let start_text = self.paragraph_text_at_row(start_ix)?; - let end_text = self.paragraph_text_at_row(end_ix)?; - let tail = end_text.slice_delta(end, end_text.len_utf8(), PosType::Bytes)?; - let start_len = start_text.len_utf8(); - if start < start_len { - start_text.delete_utf8(start, start_len - start)?; - } - - let blocks = self.blocks(); - for row_ix in ((start_ix + 1)..end_ix).rev() { - blocks.delete(row_ix, 1)?; - self - .binding - .remove_row(row_ix) - .context("DocBinding row disappeared during cross-paragraph delete")?; + let start_paragraph_ix = self + .paragraph_ordinal_for_row(start_ix) + .context("delete start row is not a paragraph")?; + let end_paragraph_ix = self + .paragraph_ordinal_for_row(end_ix) + .context("delete end row is not a paragraph")?; + if end_paragraph_ix > start_paragraph_ix { + let deleted_rows = self.paragraph_row_indices(start_paragraph_ix + 1, end_paragraph_ix - start_paragraph_ix)?; + let blocks = self.blocks(); + for row_ix in deleted_rows.into_iter().rev() { + blocks.delete(row_ix, 1)?; + self + .binding + .remove_row(row_ix) + .context("DocBinding paragraph row disappeared during cross-paragraph delete")?; + } } - - insert_delta_at(&start_text, start, &tail)?; - blocks.delete(start_ix + 1, 1)?; - self - .binding - .remove_row(start_ix + 1) - .context("DocBinding end row disappeared during cross-paragraph delete")?; - self.debug_assert_row_text_len(document, start_ix); + self.debug_assert_body_matches_document(document); Ok(()) } @@ -161,44 +148,52 @@ impl LocalApplier<'_> { .get(insert_ix) .map(block_version) .context("post-split document is missing the new block")?; - let row = self - .binding - .rows - .get(row_ix) - .context("DocBinding paragraph index is out of bounds")?; - let style = decode_paragraph_style(map_i64(&row.map, STYLE)?); - let text = row - .text - .as_ref() - .context("paragraph row is missing its LoroText")? - .clone(); - let tail = text.slice_delta(byte, text.len_utf8(), PosType::Bytes)?; - if byte < text.len_utf8() { - text.delete_utf8(byte, text.len_utf8() - byte)?; - } - self.debug_assert_row_text_len(document, row_ix); + let body_byte = self.body_byte_for_paragraph_byte(paragraph, byte)?; + body_text(self.doc).insert_utf8(body_byte, "\n")?; let blocks = self.blocks(); - let (map, new_text) = insert_paragraph_container(&blocks, insert_ix, style, &[], "")?; - insert_delta_at(&new_text, 0, &tail)?; + let paragraph_ix = self + .paragraph_ordinal_for_row(row_ix) + .context("split row is not a paragraph")? + .saturating_add(1); + let style = document + .paragraphs + .get(paragraph_ix) + .map_or_else( + || { + self + .binding + .rows + .get(row_ix) + .and_then(|row| map_i64(&row.map, STYLE).ok()) + .map(decode_paragraph_style) + .unwrap_or(gpui_flowtext::ParagraphStyle::Normal) + }, + |paragraph| paragraph.style, + ); + let map = insert_paragraph_container(&blocks, insert_ix, style)?; self.binding.insert_row( insert_ix, BindingRow { map, - text: Some(new_text), kind: BlockKind::Paragraph, block_id, paragraph_id: Some(new_paragraph), version, }, ); - self.debug_assert_row_text_len(document, insert_ix); + self.debug_assert_body_matches_document(document); Ok(()) } fn join_paragraphs(&mut self, document: &Document, first: ParagraphId, second: ParagraphId) -> Result<()> { let first_ix = self.row_ix_for_paragraph(first)?; - let first_len = self.paragraph_text_at_row(first_ix)?.len_utf8(); + let first_paragraph_ix = self + .paragraph_ordinal_for_row(first_ix) + .context("join first row is not a paragraph")?; + let first_len = self + .body_paragraph_range(first_paragraph_ix)? + .len(); self.delete_range(document, first, first_len, second, 0) } @@ -209,6 +204,7 @@ impl LocalApplier<'_> { .context("inserted block index is out of bounds in the post-edit document")?; match block { Block::Paragraph(paragraph) => { + replace_body_from_document(self.doc, document)?; let paragraph_ix = document_paragraph_ix_for_block(document, block_ix)?; let paragraph_id = document .ids @@ -216,17 +212,14 @@ impl LocalApplier<'_> { .get(paragraph_ix) .copied() .context("inserted paragraph block is missing a paragraph id")?; - let text = paragraph_text(document, paragraph_ix); self.insert_paragraph_row( block_ix, block_id, paragraph_id, paragraph.style, - ¶graph.runs, - &text, block_version(block), )?; - self.debug_assert_row_text_len(document, block_ix); + self.debug_assert_body_matches_document(document); Ok(()) }, Block::Image(_) | Block::Equation(_) | Block::Table(_) => { @@ -236,7 +229,6 @@ impl LocalApplier<'_> { block_ix, BindingRow { map, - text: None, kind: BlockKind::from_block(block), block_id, paragraph_id: None, @@ -248,20 +240,36 @@ impl LocalApplier<'_> { } } - fn delete_block(&mut self, block: BlockId) -> Result<()> { + fn delete_block(&mut self, document: &Document, block: BlockId) -> Result<()> { let row_ix = self.row_ix_for_block(block)?; + let was_paragraph = matches!( + self.binding.rows.get(row_ix).map(|row| row.kind), + Some(BlockKind::Paragraph) + ); self.blocks().delete(row_ix, 1)?; self .binding .remove_row(row_ix) .context("DocBinding block row disappeared during block delete")?; + if was_paragraph { + replace_body_from_document(self.doc, document)?; + self.debug_assert_body_matches_document(document); + } Ok(()) } - fn move_block(&mut self, block: BlockId, new_block_ix: usize) -> Result<()> { + fn move_block(&mut self, document: &Document, block: BlockId, new_block_ix: usize) -> Result<()> { let old_ix = self.row_ix_for_block(block)?; + let was_paragraph = matches!( + self.binding.rows.get(old_ix).map(|row| row.kind), + Some(BlockKind::Paragraph) + ); self.blocks().mov(old_ix, new_block_ix)?; self.binding.move_row(old_ix, new_block_ix); + if was_paragraph { + replace_body_from_document(self.doc, document)?; + self.debug_assert_body_matches_document(document); + } Ok(()) } @@ -272,81 +280,8 @@ impl LocalApplier<'_> { before: &DocumentSpan, after: &DocumentSpan, ) -> Result<()> { - let start = self.replace_span_start_ordinal(start_paragraph, before); - let old_rows = self.paragraph_row_indices(start, before.paragraphs.len())?; - let before_texts = span_paragraph_texts(before)?; - let after_texts = span_paragraph_texts(after)?; - let matched = before.paragraphs.len().min(after.paragraphs.len()); - - for ix in 0..matched { - self.update_matched_paragraph( - document, - old_rows[ix], - &before.paragraphs[ix], - &before_texts[ix], - &after.paragraphs[ix], - &after_texts[ix], - )?; - } - - if after.paragraphs.len() > matched { - let base_insert_ix = if matched > 0 { - old_rows[matched - 1] + 1 - } else { - old_rows - .first() - .copied() - .unwrap_or_else(|| self.row_ix_for_paragraph_insert_ordinal(start)) - }; - for (ix, (paragraph, text)) in after - .paragraphs - .iter() - .zip(after_texts.iter()) - .enumerate() - .skip(matched) - { - let insert_ix = base_insert_ix + ix - matched; - let paragraph_ix = start + ix; - let block_ix = document_block_ix_for_paragraph(document, paragraph_ix)?; - let block_id = document - .ids - .block_ids - .get(block_ix) - .copied() - .context("inserted paragraph span block is missing a block id")?; - let paragraph_id = document - .ids - .paragraph_ids - .get(paragraph_ix) - .copied() - .context("inserted paragraph span block is missing a paragraph id")?; - self.insert_paragraph_row( - insert_ix, - block_id, - paragraph_id, - paragraph.style, - ¶graph.runs, - text, - block_version(&document.blocks[block_ix]), - )?; - self.debug_assert_row_text_len(document, insert_ix); - } - } - - if before.paragraphs.len() > matched { - let blocks = self.blocks(); - for row_ix in old_rows[matched..].iter().rev().copied() { - blocks.delete(row_ix, 1)?; - self - .binding - .remove_row(row_ix) - .context("DocBinding paragraph span row disappeared during delete")?; - } - } - - self.binding.refresh_versions(document); - self.binding.assert_consistent(self.doc, document); - Ok(()) + let _ = (start_paragraph, before, after); + self.replace_document(document) } fn replace_block(&mut self, document: &Document, block: Option) -> Result<()> { @@ -368,37 +303,6 @@ impl LocalApplier<'_> { Ok(()) } - fn update_matched_paragraph( - &self, - document: &Document, - row_ix: usize, - before: &Paragraph, - before_text: &str, - after: &Paragraph, - after_text: &str, - ) -> Result<()> { - let row = self - .binding - .rows - .get(row_ix) - .context("matched paragraph row is out of bounds")?; - let text = row - .text - .as_ref() - .context("matched paragraph row is missing its LoroText")?; - if before_text != after_text { - text.update(after_text, UpdateOptions::default())?; - } - if before.runs != after.runs { - refresh_marks(text, &after.runs)?; - } - if before.style != after.style { - row.map.insert(STYLE, encode_paragraph_style(after.style))?; - } - self.debug_assert_row_text_len(document, row_ix); - Ok(()) - } - fn replace_block_at(&mut self, document: &Document, row_ix: usize) -> Result { let Some(block) = document.blocks.get(row_ix) else { return Ok(false); @@ -419,7 +323,6 @@ impl LocalApplier<'_> { .map .insert(REV, map_i64_or(&row.map, REV, 0)?.saturating_add(1))?; row.kind = BlockKind::from_block(block); - row.text = None; row.paragraph_id = None; row.version = block_version(block); self.binding.rebuild_indexes(); @@ -432,17 +335,14 @@ impl LocalApplier<'_> { block_id: BlockId, paragraph_id: ParagraphId, style: gpui_flowtext::ParagraphStyle, - runs: &[TextRun], - text: &str, version: u64, ) -> Result<()> { let blocks = self.blocks(); - let (map, loro_text) = insert_paragraph_container(&blocks, row_ix, style, runs, text)?; + let map = insert_paragraph_container(&blocks, row_ix, style)?; self.binding.insert_row( row_ix, BindingRow { map, - text: Some(loro_text), kind: BlockKind::Paragraph, block_id, paragraph_id: Some(paragraph_id), @@ -466,23 +366,6 @@ impl LocalApplier<'_> { changed } - fn delete_text_at_row(&self, row_ix: usize, start: usize, end: usize) -> Result<()> { - let text = self.paragraph_text_at_row(row_ix)?; - if end > start { - text.delete_utf8(start, end - start)?; - } - Ok(()) - } - - fn paragraph_text_at_row(&self, row_ix: usize) -> Result { - self - .binding - .rows - .get(row_ix) - .and_then(|row| row.text.clone()) - .context("paragraph row is missing its LoroText") - } - fn row_for_paragraph(&self, paragraph: ParagraphId) -> Result<&BindingRow> { let ix = self.row_ix_for_paragraph(paragraph)?; self @@ -492,16 +375,8 @@ impl LocalApplier<'_> { .context("DocBinding paragraph index is out of bounds") } - fn debug_assert_row_text_len(&self, document: &Document, row_ix: usize) { - let Some(row) = self.binding.rows.get(row_ix) else { - return; - }; - let Some(text) = row.text.as_ref() else { - return; - }; - if let Some(paragraph_ix) = self.paragraph_ordinal_for_row(row_ix) { - debug_assert_paragraph_text_len(text, document, paragraph_ix); - } + fn debug_assert_body_matches_document(&self, document: &Document) { + debug_assert_eq!(text_content(&body_text(self.doc)), full_document_text(document)); } fn row_ix_for_paragraph(&self, paragraph: ParagraphId) -> Result { @@ -522,13 +397,6 @@ impl LocalApplier<'_> { .context("block id is not present in DocBinding") } - fn replace_span_start_ordinal(&self, start_paragraph: Option, before: &DocumentSpan) -> usize { - start_paragraph - .and_then(|paragraph| self.binding.by_paragraph.get(¶graph).copied()) - .and_then(|row_ix| self.paragraph_ordinal_for_row(row_ix)) - .unwrap_or(before.start_paragraph) - } - fn paragraph_ordinal_for_row(&self, target_row: usize) -> Option { let mut paragraph_ix = 0; for (row_ix, row) in self.binding.rows.iter().enumerate() { @@ -560,68 +428,31 @@ impl LocalApplier<'_> { Ok(rows) } - fn row_ix_for_paragraph_insert_ordinal(&self, target: usize) -> usize { - let mut paragraph_ix = 0; - for (row_ix, row) in self.binding.rows.iter().enumerate() { - if row.paragraph_id.is_none() { - continue; - } - if paragraph_ix == target { - return row_ix; - } - paragraph_ix += 1; - } - self.binding.rows.len() - } - fn blocks(&self) -> LoroMovableList { self.doc.get_movable_list(BLOCKS) } -} -fn insert_delta_at(text: &LoroText, byte: usize, delta: &[TextDelta]) -> Result<()> { - let mut cursor = byte; - for item in delta { - let TextDelta::Insert { insert, attributes } = item else { - continue; - }; - if insert.is_empty() { - continue; - } - text.insert_utf8(cursor, insert)?; - let styles = run_styles_from_attrs(attributes.as_ref()); - set_run_styles_utf8(text, cursor..cursor + insert.len(), styles)?; - cursor += insert.len(); + fn body_byte_for_paragraph_byte(&self, paragraph: ParagraphId, byte: usize) -> Result { + let row_ix = self.row_ix_for_paragraph(paragraph)?; + let paragraph_ix = self + .paragraph_ordinal_for_row(row_ix) + .context("paragraph row has no paragraph ordinal")?; + let range = self.body_paragraph_range(paragraph_ix)?; + Ok(range.start + byte.min(range.len())) } - Ok(()) -} -fn refresh_marks(text: &LoroText, runs: &[TextRun]) -> Result<()> { - let len = text.len_utf8(); - if len == 0 { - return Ok(()); - } - for key in [MARK_SEMANTIC, MARK_UNDERLINE, MARK_STRIKE, MARK_HIGHLIGHT] { - unmark_utf8(text, 0..len, key)?; + fn body_range_for_paragraph(&self, paragraph: ParagraphId) -> Result> { + let row_ix = self.row_ix_for_paragraph(paragraph)?; + let paragraph_ix = self + .paragraph_ordinal_for_row(row_ix) + .context("paragraph row has no paragraph ordinal")?; + self.body_paragraph_range(paragraph_ix) } - apply_mark_intervals(text, &mark_intervals_from_runs(runs)) -} -fn span_paragraph_texts(span: &DocumentSpan) -> Result> { - let mut texts = Vec::with_capacity(span.paragraphs.len()); - let mut byte = 0; - for (ix, paragraph) in span.paragraphs.iter().enumerate() { - let end = byte + paragraph_text_len(paragraph); - texts.push( - span - .text - .get(byte..end) - .context("DocumentSpan text is shorter than its paragraph runs")? - .to_string(), - ); - byte = end + usize::from(ix + 1 < span.paragraphs.len()); + fn body_paragraph_range(&self, target_paragraph_ix: usize) -> Result> { + paragraph_range_in_body_text(&text_content(&body_text(self.doc)), target_paragraph_ix) + .context("paragraph ordinal is outside the body text") } - Ok(texts) } fn document_paragraph_ix_for_block(document: &Document, target_block_ix: usize) -> Result { @@ -639,18 +470,6 @@ fn document_paragraph_ix_for_block(document: &Document, target_block_ix: usize) bail!("target block index is out of bounds") } -fn document_block_ix_for_paragraph(document: &Document, target_paragraph_ix: usize) -> Result { - let mut paragraph_ix = 0; - for (block_ix, block) in document.blocks.iter().enumerate() { - match block { - Block::Paragraph(_) if paragraph_ix == target_paragraph_ix => return Ok(block_ix), - Block::Paragraph(_) => paragraph_ix += 1, - Block::Image(_) | Block::Equation(_) | Block::Table(_) => {}, - } - } - bail!("target paragraph index is out of bounds") -} - fn kind_for_payload(payload: &BlockPayload) -> &'static str { match payload { BlockPayload::Image { .. } => KIND_IMAGE, @@ -659,6 +478,22 @@ fn kind_for_payload(payload: &BlockPayload) -> &'static str { } } +fn paragraph_range_in_body_text(text: &str, target_paragraph_ix: usize) -> Option> { + let mut paragraph_ix = 0usize; + let mut start = 0usize; + for (byte, ch) in text.char_indices() { + if ch != '\n' { + continue; + } + if paragraph_ix == target_paragraph_ix { + return Some(start..byte); + } + paragraph_ix += 1; + start = byte + ch.len_utf8(); + } + (paragraph_ix == target_paragraph_ix).then_some(start..text.len()) +} + fn map_i64(map: &LoroMap, key: &str) -> Result { match map.get(key) { Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), diff --git a/crates/flowstate-collab/src/net/swarm.rs b/crates/flowstate-collab/src/net/swarm.rs index 2f63b64a..7b2ab7dc 100644 --- a/crates/flowstate-collab/src/net/swarm.rs +++ b/crates/flowstate-collab/src/net/swarm.rs @@ -39,9 +39,10 @@ impl SwarmHandle { tracing::info!(%session, bootstrap_count = bootstrap.len(), "spawning collaboration gossip swarm"); register_bootstrap_addrs(&endpoint, &bootstrap)?; let bootstrap_ids = bootstrap.iter().map(endpoint_id).collect::>(); + let local_peer = endpoint_id(&endpoint.addr()); let (tx, rx) = async_channel::unbounded(); tokio::spawn(async move { - if let Err(error) = run_session(gossip, direct_state, session, bootstrap_ids, rx, evt_tx.clone()).await { + if let Err(error) = run_session(gossip, direct_state, session, local_peer, bootstrap_ids, rx, evt_tx.clone()).await { tracing::error!(%session, error = %format_args!("{error:#}"), "collaboration gossip swarm failed"); let _ = evt_tx .send(NetEvent::SubscribeFailed { @@ -73,6 +74,7 @@ async fn run_session( gossip: Gossip, direct_state: DirectServeState, session: SessionId, + local_peer: EndpointId, bootstrap: Vec, rx: Receiver, evt_tx: Sender, @@ -102,7 +104,7 @@ async fn run_session( }, event = receiver.next() => { let event = event.context("collaboration gossip receiver ended")??; - handle_event(event, session, &evt_tx).await?; + handle_event(event, session, local_peer, &evt_tx).await?; }, } } @@ -160,8 +162,16 @@ async fn update_message(direct_state: &DirectServeState, session: SessionId, byt Ok(GossipMsg::UpdateAvailable { blob, len }) } -async fn handle_event(event: Event, session: SessionId, evt_tx: &Sender) -> Result<()> { +async fn handle_event(event: Event, session: SessionId, local_peer: EndpointId, evt_tx: &Sender) -> Result<()> { match event { + Event::Received(message) if message.delivered_from == local_peer => { + tracing::trace!( + %session, + from = %message.delivered_from, + frame_bytes = message.content.len(), + "ignored self-delivered collaboration gossip frame", + ); + }, Event::Received(message) => match proto_gossip::decode(&message.content) { Ok(msg) => { tracing::trace!( @@ -250,6 +260,7 @@ mod tests { #[tokio::test] async fn version_mismatch_emits_incompatible_version_once() { let session = SessionId::new(); + let local_peer = SecretKey::generate().public(); let peer = SecretKey::generate().public(); let mut frame = proto_gossip::encode(&GossipMsg::Presence(Vec::new())).expect("valid gossip frame"); frame[0] = frame[0].wrapping_add(1); @@ -262,6 +273,7 @@ mod tests { delivered_from: peer, }), session, + local_peer, &evt_tx, ) .await @@ -276,4 +288,27 @@ mod tests { } assert!(evt_rx.try_recv().is_err()); } + + #[tokio::test] + async fn self_delivered_gossip_frame_is_ignored() { + let session = SessionId::new(); + let local_peer = SecretKey::generate().public(); + let frame = proto_gossip::encode(&GossipMsg::Update(vec![1, 2, 3])).expect("valid gossip frame"); + + let (evt_tx, evt_rx) = async_channel::unbounded(); + handle_event( + Event::Received(Message { + content: frame.into(), + scope: DeliveryScope::Neighbors, + delivered_from: local_peer, + }), + session, + local_peer, + &evt_tx, + ) + .await + .expect("self-delivered gossip frames should be ignored"); + + assert!(evt_rx.try_recv().is_err()); + } } diff --git a/crates/flowstate-collab/src/patch_apply.rs b/crates/flowstate-collab/src/patch_apply.rs index 5afa2162..bf77c1cc 100644 --- a/crates/flowstate-collab/src/patch_apply.rs +++ b/crates/flowstate-collab/src/patch_apply.rs @@ -33,6 +33,16 @@ fn apply_one(document: &mut Document, patch: &CollabPatch) -> Result<()> { rebuild_document_sections(document); } }, + CollabPatch::ParagraphRuns { row, runs } => { + if let Some(paragraph_ix) = paragraph_ix_for_block(document, *row) + && let Some(paragraph) = paragraphs_mut(document).get_mut(paragraph_ix) + { + paragraph.runs.clone_from(runs); + gpui_flowtext::bump_paragraph_version(paragraph); + update_paragraph_block(document, paragraph_ix); + rebuild_document_sections(document); + } + }, CollabPatch::ReplaceObjectBlock { row, block } => { let mut blocks = structural_blocks_from_document(document)?; if *row < blocks.len() { diff --git a/crates/flowstate-collab/src/projection.rs b/crates/flowstate-collab/src/projection.rs index 3facd7e6..fd0f7bd0 100644 --- a/crates/flowstate-collab/src/projection.rs +++ b/crates/flowstate-collab/src/projection.rs @@ -1,15 +1,17 @@ //! Projection between flowtext documents and the shared Loro document. use anyhow::{Context as _, Result, bail}; -use gpui_flowtext::{AssetStore, Block, Document, DocumentTheme, InputBlock, document_from_input_blocks, paragraph_text}; -use loro::{Container, LoroDoc, LoroMap, LoroMovableList, LoroText, LoroValue, ValueOrContainer}; +use gpui_flowtext::{ + AssetStore, Block, Document, DocumentTheme, InputBlock, document_from_input_blocks, full_document_text, paragraph_byte_range, +}; +use loro::{Container, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer}; use crate::{ SessionId, schema::{ - BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_PARAGRAPH, KIND_TABLE, META, META_SCHEMA, META_SESSION, META_TITLE, REV, - SCHEMA_VERSION, STYLE, TEXT, apply_mark_intervals, block_from_payload, configure_text_styles, decode_paragraph_style, - encode_paragraph_style, input_paragraph_from_text, mark_intervals_from_runs, payload_from_block, + BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_PARAGRAPH, KIND_TABLE, META, META_SCHEMA, META_SESSION, META_TITLE, + MarkIntervals, REV, SCHEMA_VERSION, STYLE, apply_mark_intervals, block_from_payload, body_text, configure_text_styles, + decode_paragraph_style, encode_paragraph_style, input_paragraphs_from_body_text, mark_intervals_from_runs, payload_from_block, }, }; @@ -23,16 +25,14 @@ pub fn populate_from_document(doc: &LoroDoc, session: SessionId, title: &str, do pub fn replace_blocks_from_document(doc: &LoroDoc, document: &Document) -> Result<()> { configure_text_styles(doc); + replace_body_from_document(doc, document)?; let blocks = doc.get_movable_list(BLOCKS); clear_blocks(&blocks)?; - let mut paragraph_ix = 0; for block in document.blocks.iter() { match block { Block::Paragraph(paragraph) => { - let text = paragraph_text(document, paragraph_ix); - insert_paragraph_container(&blocks, blocks.len(), paragraph.style, ¶graph.runs, &text)?; - paragraph_ix += 1; + insert_paragraph_container(&blocks, blocks.len(), paragraph.style)?; }, Block::Image(_) | Block::Equation(_) | Block::Table(_) => { insert_object_container(&blocks, blocks.len(), block, document)?; @@ -46,10 +46,15 @@ pub fn replace_blocks_from_document(doc: &LoroDoc, document: &Document) -> Resul pub fn input_blocks_from_loro(doc: &LoroDoc) -> Result> { configure_text_styles(doc); let blocks = doc.get_movable_list(BLOCKS); + let body_paragraphs = input_paragraphs_from_body_text(&body_text(doc)); + let mut paragraph_ix = 0usize; let mut input_blocks = Vec::with_capacity(blocks.len()); for ix in 0..blocks.len() { let map = map_at(&blocks, ix)?; - input_blocks.push(input_block_from_container(&map)?); + input_blocks.push(input_block_from_container(&map, body_paragraphs.get(paragraph_ix))?); + if map_string(&map, KIND)? == KIND_PARAGRAPH { + paragraph_ix += 1; + } } Ok(input_blocks) } @@ -58,12 +63,18 @@ pub fn document_from_loro(doc: &LoroDoc, theme: DocumentTheme) -> Result Result { +pub fn input_block_from_container(map: &LoroMap, body_paragraph: Option<&gpui_flowtext::InputParagraph>) -> Result { match map_string(map, KIND)?.as_str() { KIND_PARAGRAPH => { - let text = text_at(map, TEXT)?; let style = decode_paragraph_style(map_i64(map, STYLE)?); - Ok(InputBlock::Paragraph(input_paragraph_from_text(&text, style))) + let mut paragraph = body_paragraph.cloned().unwrap_or(gpui_flowtext::InputParagraph { + style, + runs: Vec::new(), + }); + if body_paragraph.is_none() { + paragraph.style = style; + } + Ok(InputBlock::Paragraph(paragraph)) }, KIND_IMAGE | KIND_EQUATION | KIND_TABLE => { let payload = map_binary(map, DATA)?; @@ -93,16 +104,11 @@ pub fn insert_paragraph_container( blocks: &LoroMovableList, ix: usize, style: gpui_flowtext::ParagraphStyle, - runs: &[gpui_flowtext::TextRun], - paragraph_text: &str, -) -> Result<(LoroMap, LoroText)> { +) -> Result { let map = blocks.insert_container(ix, LoroMap::new())?; map.insert(KIND, KIND_PARAGRAPH)?; map.insert(STYLE, encode_paragraph_style(style))?; - let text = map.insert_container(TEXT, LoroText::new())?; - text.insert_utf8(0, paragraph_text)?; - apply_mark_intervals(&text, &mark_intervals_from_runs(runs))?; - Ok((map, text)) + Ok(map) } pub fn insert_object_container(blocks: &LoroMovableList, ix: usize, block: &Block, document: &Document) -> Result { @@ -130,15 +136,6 @@ fn map_at(blocks: &LoroMovableList, ix: usize) -> Result { } } -fn text_at(map: &LoroMap, key: &str) -> Result { - match map.get(key) { - Some(ValueOrContainer::Container(Container::Text(text))) => Ok(text), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("collaboration map key {key} is not a text container") - }, - } -} - fn map_string(map: &LoroMap, key: &str) -> Result { match map.get(key) { Some(ValueOrContainer::Value(LoroValue::String(value))) => Ok(value.to_string()), @@ -148,6 +145,44 @@ fn map_string(map: &LoroMap, key: &str) -> Result { } } +pub fn replace_body_from_document(doc: &LoroDoc, document: &Document) -> Result<()> { + let body = body_text(doc); + let len = body.len_utf8(); + if len > 0 { + body.delete_utf8(0, len)?; + } + let text = full_document_text(document); + if !text.is_empty() { + body.insert_utf8(0, &text)?; + apply_mark_intervals(&body, &document_mark_intervals(document))?; + } + Ok(()) +} + +fn document_mark_intervals(document: &Document) -> MarkIntervals { + let mut intervals: MarkIntervals = [Vec::new(), Vec::new(), Vec::new(), Vec::new(), Vec::new()]; + for (paragraph_ix, paragraph) in document.paragraphs.iter().enumerate() { + let paragraph_start = paragraph_byte_range(document, paragraph_ix).start; + for (target, source) in intervals + .iter_mut() + .zip(mark_intervals_from_runs(¶graph.runs)) + { + target.extend( + source + .into_iter() + .map(|(range, value)| (paragraph_start + range.start..paragraph_start + range.end, value)), + ); + } + if paragraph.style != gpui_flowtext::ParagraphStyle::Normal { + let range = paragraph_byte_range(document, paragraph_ix); + if !range.is_empty() { + intervals[4].push((range, LoroValue::I64(encode_paragraph_style(paragraph.style)))); + } + } + } + intervals +} + fn map_i64(map: &LoroMap, key: &str) -> Result { match map.get(key) { Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), diff --git a/crates/flowstate-collab/src/remote_apply.rs b/crates/flowstate-collab/src/remote_apply.rs index 2d9488cd..5cf333b8 100644 --- a/crates/flowstate-collab/src/remote_apply.rs +++ b/crates/flowstate-collab/src/remote_apply.rs @@ -2,18 +2,18 @@ use anyhow::{Context as _, Result, bail}; use gpui_flowtext::{ - BlockId, CollabPatch, CollabStructuralBlock, CollabTextDelta, Document, InputBlock, ParagraphId, new_block_id, new_paragraph_id, - paragraph_text, + BlockId, CollabPatch, CollabStructuralBlock, CollabTextDelta, Document, InputBlock, InputParagraph, InputRun, ParagraphId, new_block_id, + new_paragraph_id, paragraph_text, TextRun, }; use loro::{ - Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroText, LoroValue, TextDelta, ValueOrContainer, + Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer, event::{Diff, DiffEvent, ListDiffItem}, }; use crate::{ binding::{BindingRow, BlockKind, DocBinding}, - projection::input_block_from_container, - schema::{BLOCKS, DATA, REV, STYLE, TEXT, decode_paragraph_style}, + projection::{input_block_from_container, input_blocks_from_loro}, + schema::{BLOCKS, DATA, REV, STYLE, body_text, decode_paragraph_style}, }; pub struct RemoteApplier<'s> { @@ -28,64 +28,73 @@ impl RemoteApplier<'_> { } let pre_event_binding = self.binding.clone(); + let body_id = body_text(self.doc).id(); + let mut body_changed = false; let inserted_containers = inserted_container_ids(self.doc, event)?; let mut patches = Vec::new(); for diff in &event.events { match &diff.diff { - Diff::Text(delta) => self.apply_text_diff(document, &pre_event_binding, &inserted_containers, diff.target, delta, &mut patches)?, + Diff::Text(_) if *diff.target == body_id => body_changed = true, + Diff::Text(_) => {}, Diff::Map(delta) => self.apply_map_diff(diff.target, delta.updated.keys().map(|key| key.as_ref()), &inserted_containers, &mut patches)?, Diff::List(delta) => self.apply_list_diff(diff.target, delta, &mut patches)?, Diff::Tree(_) | Diff::Counter(_) | Diff::Unknown => {}, } } + if body_changed { + self.reconcile_body_text(document, &pre_event_binding, &mut patches)?; + } Ok(patches) } - fn apply_text_diff( - &self, - document: &Document, - pre_event_binding: &DocBinding, - inserted_containers: &[ContainerID], - target: &ContainerID, - delta: &[TextDelta], - patches: &mut Vec, - ) -> Result<()> { - if inserted_containers.contains(target) { - return Ok(()); - } - let Some(row_ix) = self.binding.by_container.get(target).copied() else { - return Ok(()); - }; - let row = self - .binding - .rows - .get(row_ix) - .context("text diff row is outside DocBinding")?; - if !matches!(row.kind, BlockKind::Paragraph) { - return Ok(()); + fn reconcile_body_text(&self, document: &Document, pre_event_binding: &DocBinding, patches: &mut Vec) -> Result<()> { + let projected_blocks = input_blocks_from_loro(self.doc)?; + for (row_ix, row) in self.binding.rows.iter().enumerate() { + if !matches!(row.kind, BlockKind::Paragraph) { + continue; + } + let Some(InputBlock::Paragraph(new)) = projected_blocks.get(row_ix) else { + continue; + }; + let old = pre_event_binding + .by_container + .get(&row.map.id()) + .and_then(|old_row_ix| paragraph_ordinal_for_row(pre_event_binding, *old_row_ix)) + .and_then(|old_paragraph_ix| input_paragraph_from_document(document, old_paragraph_ix)); + let Some(old) = old else { + patches.push(CollabPatch::ParagraphText { + row: row_ix, + new: new.clone(), + delta_utf8: text_delta_for_replacement("", &input_paragraph_text(new)), + }); + continue; + }; + if input_paragraphs_equal(&old, new) { + continue; + } + let old_text = input_paragraph_text(&old); + let new_text = input_paragraph_text(new); + if old_text == new_text { + if old.style != new.style && !has_paragraph_style_patch(patches, row_ix) { + patches.push(CollabPatch::ParagraphStyle { + row: row_ix, + style: new.style, + }); + } + if !input_runs_equal(&old.runs, &new.runs) { + patches.push(CollabPatch::ParagraphRuns { + row: row_ix, + runs: text_runs_from_input(new), + }); + } + continue; + } + patches.push(CollabPatch::ParagraphText { + row: row_ix, + new: new.clone(), + delta_utf8: text_delta_for_replacement(&old_text, &new_text), + }); } - let text = row - .text - .as_ref() - .context("paragraph diff row is missing its LoroText")?; - let style = decode_paragraph_style(map_i64(&row.map, STYLE)?); - let old_paragraph_ix = pre_event_binding - .by_container - .get(target) - .copied() - .and_then(|old_row_ix| paragraph_ordinal_for_row(pre_event_binding, old_row_ix)); - let old_text = if let Some(paragraph_ix) = old_paragraph_ix - && paragraph_ix < document.paragraphs.len() - { - paragraph_text(document, paragraph_ix) - } else { - String::new() - }; - patches.push(CollabPatch::ParagraphText { - row: row_ix, - new: crate::schema::input_paragraph_from_text(text, style), - delta_utf8: text_delta_to_collab_utf8(&old_text, delta), - }); Ok(()) } @@ -123,9 +132,8 @@ impl RemoteApplier<'_> { }); } if object_changed { - let input = input_block_from_container(&row.map)?; + let input = input_block_from_container(&row.map, None)?; row.kind = block_kind_for_input(&input); - row.text = None; row.paragraph_id = None; row.version = 0; patches.push(CollabPatch::ReplaceObjectBlock { @@ -189,7 +197,7 @@ impl RemoteApplier<'_> { continue; } - let input = input_block_from_container(&map)?; + let input = input_block_from_container(&map, None)?; let structural = structural_block_for_insert(&input); let row = binding_row_from_insert(map, &input, structural.block_id, structural.paragraph_id)?; self.binding.insert_row(row_ix, row); @@ -212,24 +220,16 @@ pub fn should_process_event(event: &DiffEvent<'_>) -> bool { event.triggered_by.is_import() || event.triggered_by.is_checkout() } -fn text_delta_to_collab_utf8(old_text: &str, delta: &[TextDelta]) -> Vec { - let mut output = Vec::with_capacity(delta.len()); - let mut old_chars = 0usize; - for item in delta { - match item { - TextDelta::Retain { retain, .. } => { - let len = utf8_len_for_char_span(old_text, old_chars, *retain); - push_text_delta(&mut output, CollabTextDelta::Retain(len)); - old_chars = old_chars.saturating_add(*retain); - }, - TextDelta::Insert { insert, .. } => push_text_delta(&mut output, CollabTextDelta::Insert(insert.len())), - TextDelta::Delete { delete } => { - let len = utf8_len_for_char_span(old_text, old_chars, *delete); - push_text_delta(&mut output, CollabTextDelta::Delete(len)); - old_chars = old_chars.saturating_add(*delete); - }, - } - } +fn text_delta_for_replacement(old_text: &str, new_text: &str) -> Vec { + let prefix = common_prefix_bytes(old_text, new_text); + let suffix = common_suffix_bytes(&old_text[prefix..], &new_text[prefix..]); + let old_middle = old_text.len().saturating_sub(prefix + suffix); + let new_middle = new_text.len().saturating_sub(prefix + suffix); + let mut output = Vec::with_capacity(4); + push_text_delta(&mut output, CollabTextDelta::Retain(prefix)); + push_text_delta(&mut output, CollabTextDelta::Delete(old_middle)); + push_text_delta(&mut output, CollabTextDelta::Insert(new_middle)); + push_text_delta(&mut output, CollabTextDelta::Retain(suffix)); output } @@ -251,17 +251,91 @@ fn push_text_delta(output: &mut Vec, delta: CollabTextDelta) { } } -fn utf8_len_for_char_span(text: &str, start_char: usize, char_len: usize) -> usize { - let start = byte_for_char(text, start_char); - let end = byte_for_char(text, start_char.saturating_add(char_len)); - end.saturating_sub(start) +fn common_prefix_bytes(left: &str, right: &str) -> usize { + let mut prefix = 0usize; + for (left_ch, right_ch) in left.chars().zip(right.chars()) { + if left_ch != right_ch { + break; + } + prefix += left_ch.len_utf8(); + } + prefix +} + +fn common_suffix_bytes(left: &str, right: &str) -> usize { + let mut suffix = 0usize; + for (left_ch, right_ch) in left.chars().rev().zip(right.chars().rev()) { + if left_ch != right_ch { + break; + } + let len = left_ch.len_utf8(); + if suffix + len > left.len() || suffix + len > right.len() { + break; + } + suffix += len; + } + suffix } -fn byte_for_char(text: &str, target_char: usize) -> usize { +fn input_paragraph_from_document(document: &Document, paragraph_ix: usize) -> Option { + let paragraph = document.paragraphs.get(paragraph_ix)?; + let text = paragraph_text(document, paragraph_ix); + let mut byte = 0usize; + let runs = paragraph + .runs + .iter() + .map(|run| { + let start = byte; + let end = start.saturating_add(run.len).min(text.len()); + byte = end; + InputRun { + text: text.get(start..end).unwrap_or_default().to_string(), + styles: run.styles, + } + }) + .collect(); + Some(InputParagraph { + style: paragraph.style, + runs, + }) +} + +fn input_paragraph_text(paragraph: &InputParagraph) -> String { + let mut text = String::new(); + for run in ¶graph.runs { + text.push_str(&run.text); + } text - .char_indices() - .nth(target_char) - .map_or(text.len(), |(byte, _)| byte) +} + +fn input_paragraphs_equal(left: &InputParagraph, right: &InputParagraph) -> bool { + left.style == right.style + && input_runs_equal(&left.runs, &right.runs) +} + +fn has_paragraph_style_patch(patches: &[CollabPatch], target_row: usize) -> bool { + patches + .iter() + .any(|patch| matches!(patch, CollabPatch::ParagraphStyle { row, .. } if *row == target_row)) +} + +fn input_runs_equal(left: &[InputRun], right: &[InputRun]) -> bool { + left.len() == right.len() + && left + .iter() + .zip(right) + .all(|(left, right)| left.text == right.text && left.styles == right.styles) +} + +fn text_runs_from_input(paragraph: &InputParagraph) -> Vec { + paragraph + .runs + .iter() + .map(|run| TextRun { + len: run.text.len(), + styles: run.styles, + }) + .collect() } fn map_from_insert(value: &ValueOrContainer) -> Result { @@ -291,9 +365,6 @@ fn inserted_container_ids(doc: &LoroDoc, event: &DiffEvent<'_>) -> Result CollabStructuralBlock { } fn binding_row_from_insert(map: LoroMap, input: &InputBlock, block_id: BlockId, paragraph_id: Option) -> Result { - let text = if matches!(input, InputBlock::Paragraph(_)) { - Some(text_child(&map)?) - } else { - None - }; Ok(BindingRow { map, - text, kind: block_kind_for_input(input), block_id, paragraph_id, @@ -413,7 +478,7 @@ fn block_kind_for_input(input: &InputBlock) -> BlockKind { } fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option { - let mut paragraph_ix = 0; + let mut paragraph_ix = 0usize; for (row_ix, row) in binding.rows.iter().enumerate() { if row_ix == target_row { return row.paragraph_id.map(|_| paragraph_ix); @@ -425,13 +490,6 @@ fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option< None } -fn text_child(map: &LoroMap) -> Result { - match map.get(TEXT) { - Some(ValueOrContainer::Container(Container::Text(text))) => Ok(text), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => bail!("paragraph block map is missing its text container"), - } -} - fn map_i64(map: &LoroMap, key: &str) -> Result { match map.get(key) { Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs index d3a8f79d..3e6ea637 100644 --- a/crates/flowstate-collab/src/schema.rs +++ b/crates/flowstate-collab/src/schema.rs @@ -1,9 +1,8 @@ //! Shared Loro container schema for Flowstate rich-text documents. //! -//! The schema uses one `LoroText` per paragraph inside a movable block list. -//! Concurrent edits after a split can therefore converge into the first half of -//! the split paragraph; this is the accepted per-paragraph CRDT trade-off from -//! the implementation plan. +//! The schema stores document text in one root `LoroText`. Paragraph rows in +//! the movable block list carry block identity and metadata only; paragraph +//! breaks are newline bytes in the root text. use std::{collections::HashMap, hash::BuildHasher, ops::Range}; @@ -16,18 +15,18 @@ use loro::{ExpandType, LoroDoc, LoroResult, LoroText, LoroValue, StyleConfig, St use serde::{Deserialize, Serialize}; pub const META: &str = "meta"; +pub const BODY: &str = "body"; pub const BLOCKS: &str = "blocks"; pub const META_SCHEMA: &str = "schema"; pub const META_SESSION: &str = "session"; pub const META_TITLE: &str = "title"; -pub const SCHEMA_VERSION: i64 = 1; +pub const SCHEMA_VERSION: i64 = 2; pub const KIND: &str = "kind"; pub const KIND_PARAGRAPH: &str = "p"; pub const KIND_IMAGE: &str = "image"; pub const KIND_EQUATION: &str = "equation"; pub const KIND_TABLE: &str = "table"; -pub const TEXT: &str = "text"; pub const STYLE: &str = "style"; pub const DATA: &str = "data"; pub const REV: &str = "rev"; @@ -36,8 +35,9 @@ pub const MARK_SEMANTIC: &str = "sem"; pub const MARK_UNDERLINE: &str = "ul"; pub const MARK_STRIKE: &str = "strike"; pub const MARK_HIGHLIGHT: &str = "hl"; +pub const MARK_PARAGRAPH_STYLE: &str = "pstyle"; -pub type MarkIntervals = [Vec<(Range, LoroValue)>; 4]; +pub type MarkIntervals = [Vec<(Range, LoroValue)>; 5]; #[derive(Clone, Debug, Serialize, Deserialize)] pub enum BlockPayload { @@ -132,9 +132,27 @@ pub fn configure_text_styles(doc: &LoroDoc) { styles.insert(MARK_UNDERLINE.into(), no_expand); styles.insert(MARK_STRIKE.into(), no_expand); styles.insert(MARK_HIGHLIGHT.into(), no_expand); + styles.insert(MARK_PARAGRAPH_STYLE.into(), StyleConfig { expand: ExpandType::Both }); doc.config_text_style(styles); } +#[must_use] +pub fn body_text(doc: &LoroDoc) -> LoroText { + doc.get_text(BODY) +} + +#[must_use] +pub fn text_content(text: &LoroText) -> String { + let mut content = String::new(); + for item in text.to_delta() { + let TextDelta::Insert { insert, .. } = item else { + continue; + }; + content.push_str(&insert); + } + content +} + #[must_use] pub fn loro_pos(text: &LoroText, utf8_byte: usize) -> usize { text @@ -176,6 +194,20 @@ pub fn set_run_styles_utf8(text: &LoroText, range: Range, styles: RunStyl Ok(()) } +pub fn set_paragraph_style_utf8( + text: &LoroText, + range: Range, + style: gpui_flowtext::ParagraphStyle, +) -> LoroResult<()> { + if range.is_empty() { + return Ok(()); + } + match style { + gpui_flowtext::ParagraphStyle::Normal => unmark_utf8(text, range, MARK_PARAGRAPH_STYLE), + gpui_flowtext::ParagraphStyle::Custom(slot) => text.mark_utf8(range, MARK_PARAGRAPH_STYLE, LoroValue::I64(i64::from(slot))), + } +} + pub fn unmark_utf8(text: &LoroText, range: Range, key: &str) -> LoroResult<()> { if range.is_empty() { return Ok(()); @@ -218,6 +250,66 @@ pub fn input_paragraph_from_text(text: &LoroText, style: gpui_flowtext::Paragrap } } +#[must_use] +pub fn input_paragraphs_from_body_text(text: &LoroText) -> Vec { + let mut paragraphs = vec![InputParagraph { + style: gpui_flowtext::ParagraphStyle::Normal, + runs: Vec::new(), + }]; + for item in text.to_delta() { + let TextDelta::Insert { insert, attributes } = item else { + continue; + }; + if insert.is_empty() { + continue; + } + let styles = run_styles_from_attrs(attributes.as_ref()); + let paragraph_style = paragraph_style_from_attrs(attributes.as_ref()); + for (ix, segment) in insert.split('\n').enumerate() { + if ix > 0 { + if let Some(style) = paragraph_style { + paragraphs.last_mut().expect("paragraph accumulator is never empty").style = style; + } + paragraphs.push(InputParagraph { + style: paragraph_style.unwrap_or(gpui_flowtext::ParagraphStyle::Normal), + runs: Vec::new(), + }); + } + if !segment.is_empty() { + if let Some(style) = paragraph_style { + paragraphs.last_mut().expect("paragraph accumulator is never empty").style = style; + } + push_input_run(paragraphs.last_mut().expect("paragraph accumulator is never empty"), segment, styles); + } + } + } + paragraphs +} + +#[must_use] +pub fn paragraph_style_from_attrs(attrs: Option<&HashMap>) -> Option +where + S: BuildHasher, +{ + attrs + .and_then(|attrs| attrs.get(MARK_PARAGRAPH_STYLE)) + .and_then(loro_i64) + .map(decode_paragraph_style) +} + +pub fn push_input_run(paragraph: &mut InputParagraph, text: &str, styles: RunStyles) { + if let Some(last) = paragraph.runs.last_mut() + && last.styles == styles + { + last.text.push_str(text); + return; + } + paragraph.runs.push(InputRun { + text: text.to_string(), + styles, + }); +} + #[must_use] pub fn run_styles_from_attrs(attrs: Option<&HashMap>) -> RunStyles where @@ -245,7 +337,7 @@ where #[must_use] pub fn mark_intervals_from_runs(runs: &[TextRun]) -> MarkIntervals { - let mut intervals: MarkIntervals = [Vec::new(), Vec::new(), Vec::new(), Vec::new()]; + let mut intervals: MarkIntervals = [Vec::new(), Vec::new(), Vec::new(), Vec::new(), Vec::new()]; let mut byte = 0; for run in runs { let range = byte..byte + run.len; @@ -268,7 +360,7 @@ pub fn mark_intervals_from_runs(runs: &[TextRun]) -> MarkIntervals { } pub fn apply_mark_intervals(text: &LoroText, intervals: &MarkIntervals) -> Result<()> { - let keys = [MARK_SEMANTIC, MARK_UNDERLINE, MARK_STRIKE, MARK_HIGHLIGHT]; + let keys = [MARK_SEMANTIC, MARK_UNDERLINE, MARK_STRIKE, MARK_HIGHLIGHT, MARK_PARAGRAPH_STYLE]; for (key, intervals) in keys.into_iter().zip(intervals) { for (range, value) in intervals { text.mark_utf8(range.clone(), key, value.clone())?; diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs index a5533292..0d4ffdac 100644 --- a/crates/flowstate-collab/tests/convergence.rs +++ b/crates/flowstate-collab/tests/convergence.rs @@ -14,7 +14,7 @@ mod tests { Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, HighlightStyle, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, ParagraphStyle, RunSemanticStyle, RunStyle, RunStyles, block_from_input_block, delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, mutate_runs_in_range, - paragraph_text, plain, remove_block_ids, split_paragraph_at, + paragraph_text, paragraphs_mut, plain, remove_block_ids, split_paragraph_at, update_paragraph_block, }; use loro::{ExportMode, LoroDoc, Subscription, event::Subscriber}; use proptest::{prelude::*, test_runner::TestCaseError}; @@ -105,6 +105,85 @@ mod tests { run_program(2, 0, &ops).expect("text edit after object insert should preserve block order"); } + #[test] + fn concurrent_insert_after_split_point_stays_in_second_half() -> Result<()> { + let session = SessionId::from_bytes([88; 32]); + let initial = document_from_input_blocks(DocumentTheme::default(), vec![paragraph_block("abcd")]); + let mut amy = Peer::from_initial(session, initial)?; + let snapshot = amy.snapshot()?; + let mut bob = Peer::from_snapshot(session, &snapshot)?; + + amy.apply_op(&FuzzOp::Split { + peer: 0, + paragraph: 0, + byte: 2, + })?; + let amy_updates = amy.drain_updates(); + + bob.apply_op(&FuzzOp::InsertText { + peer: 1, + paragraph: 0, + byte: 3, + text: 4, + style: 0, + })?; + let bob_updates = bob.drain_updates(); + + for update in &bob_updates { + amy.import_update(update)?; + } + for update in &amy_updates { + bob.import_update(update)?; + } + + assert_eq!(paragraph_text(&amy.document, 0), "ab"); + assert_eq!(paragraph_text(&amy.document, 1), "cxd"); + assert_eq!(paragraph_text(&bob.document, 0), "ab"); + assert_eq!(paragraph_text(&bob.document, 1), "cxd"); + Ok(()) + } + + #[test] + fn concurrent_paragraph_style_applies_to_both_split_halves() -> Result<()> { + let session = SessionId::from_bytes([89; 32]); + let initial = document_from_input_blocks(DocumentTheme::default(), vec![paragraph_block("abcd")]); + let mut amy = Peer::from_initial(session, initial)?; + let snapshot = amy.snapshot()?; + let mut bob = Peer::from_snapshot(session, &snapshot)?; + + amy.apply_op(&FuzzOp::Split { + peer: 0, + paragraph: 0, + byte: 2, + })?; + let amy_updates = amy.drain_updates(); + + let paragraph = bob.document.ids.paragraph_ids[0]; + paragraphs_mut(&mut bob.document)[0].style = ParagraphStyle::Custom(3); + update_paragraph_block(&mut bob.document, 0); + LocalApplier { doc: &bob.loro, binding: &mut bob.binding } + .apply(&bob.document, &[CanonicalOperation::SetParagraphStyle { + paragraph, + style: ParagraphStyle::Custom(3), + }])?; + let bob_updates = bob.drain_updates(); + + for update in &bob_updates { + amy.import_update(update)?; + } + for update in &amy_updates { + bob.import_update(update)?; + } + + for peer in [&amy, &bob] { + assert_eq!(paragraph_text(&peer.document, 0), "ab"); + assert_eq!(paragraph_text(&peer.document, 1), "cd"); + assert_eq!(peer.document.paragraphs[0].style, ParagraphStyle::Custom(3)); + assert_eq!(peer.document.paragraphs[1].style, ParagraphStyle::Custom(3)); + } + Ok(()) + } + #[test] fn concurrent_object_insert_move_interleaving_converges() { let ops = [ diff --git a/crates/flowstate-collab/tests/projection.rs b/crates/flowstate-collab/tests/projection.rs index f0675052..96c192e3 100644 --- a/crates/flowstate-collab/tests/projection.rs +++ b/crates/flowstate-collab/tests/projection.rs @@ -69,11 +69,8 @@ mod tests { ); let loro = schema::new_configured_doc(); projection::populate_from_document(&loro, SessionId::from_bytes([8; 32]), "offsets", &original).expect("populate should succeed"); - let binding = DocBinding::build(&loro, &original).expect("binding should build"); - let text = binding.rows[0] - .text - .as_ref() - .expect("first row should be paragraph text"); + let _binding = DocBinding::build(&loro, &original).expect("binding should build"); + let text = schema::body_text(&loro); let byte_to_unicode = [ (0, 0), ("a".len(), 1), @@ -85,8 +82,8 @@ mod tests { assert_eq!(text.len_utf8(), MULTIBYTE.len()); for (byte, unicode) in byte_to_unicode { - assert_eq!(schema::loro_pos(text, byte), unicode); - assert_eq!(schema::utf8_byte(text, unicode), byte); + assert_eq!(schema::loro_pos(&text, byte), unicode); + assert_eq!(schema::utf8_byte(&text, unicode), byte); } } diff --git a/crates/flowstate-collab/tests/remote_apply.rs b/crates/flowstate-collab/tests/remote_apply.rs index d9488674..f4093963 100644 --- a/crates/flowstate-collab/tests/remote_apply.rs +++ b/crates/flowstate-collab/tests/remote_apply.rs @@ -62,7 +62,7 @@ mod tests { ); let run_style_updates = source.apply(&[CanonicalOperation::SetRunStyles { paragraph, range, styles }]); let run_style_patches = target.import_updates(&run_style_updates); - assert!(matches!(run_style_patches.as_slice(), [CollabPatch::ParagraphText { row: 0, .. }])); + assert!(matches!(run_style_patches.as_slice(), [CollabPatch::ParagraphRuns { row: 0, .. }])); assert_documents_match(&source.document, &target.document, &target.loro); } diff --git a/crates/flowstate/src/collab/manager.rs b/crates/flowstate/src/collab/manager.rs index 4cc9de87..402be768 100644 --- a/crates/flowstate/src/collab/manager.rs +++ b/crates/flowstate/src/collab/manager.rs @@ -136,13 +136,14 @@ impl CollabManager { self.own_endpoint_id != Some(ticket.inviter.id), "That's your own invite — open the Share dialog instead." ); - if self.sessions_by_id.contains_key(&ticket.session) { - tracing::warn!(session = %ticket.session, "collaboration session is already open"); + if self.unregister_detached_session(ticket.session, cx) { + tracing::info!(session = %ticket.session, "discarded detached collaboration session before rejoin"); + } + if let Some(existing) = self.sessions_by_id.get(&ticket.session) { + let phase = existing.read(cx).phase().clone(); + tracing::warn!(session = %ticket.session, ?phase, "collaboration session is already open"); + return Err(anyhow!("collaboration session is already open")); } - ensure!( - !self.sessions_by_id.contains_key(&ticket.session), - "collaboration session is already open" - ); let session = ticket.session; let inviter = ticket.inviter.id; @@ -314,6 +315,20 @@ impl CollabManager { tracing::debug!(%session, removed, open_sessions = self.sessions_by_id.len(), "unregistered collaboration session"); } + fn unregister_detached_session(&mut self, session: SessionId, cx: &mut Context) -> bool + where + T: 'static, + { + let detached = self + .sessions_by_id + .get(&session) + .is_some_and(|existing| matches!(existing.read(cx).phase(), SessionPhase::Detached(_))); + if detached { + self.unregister_session(session); + } + detached + } + fn ensure_runtime(&mut self, cx: &mut Context) -> Result where T: 'static, diff --git a/crates/flowstate/src/collab/notify.rs b/crates/flowstate/src/collab/notify.rs index e0892a21..c7ef9c32 100644 --- a/crates/flowstate/src/collab/notify.rs +++ b/crates/flowstate/src/collab/notify.rs @@ -1,4 +1,4 @@ -use gpui::{App, Window}; +use gpui::{App, PromptButton, PromptLevel, Window}; use gpui_component::{ WindowExt as _, notification::Notification, @@ -11,6 +11,15 @@ pub fn show_session_notice(notice: &SessionNotice, window: &mut Window, cx: &mut SessionNotice::PeerJoined(name) => push_info(collaborator_message(name, "joined"), window, cx), SessionNotice::PeerLeft(name) => push_info(collaborator_message(name, "left"), window, cx), SessionNotice::LeftSession => push_info("Left session - this copy is now local.", window, cx), + SessionNotice::Disconnected(detail) => { + std::mem::drop(window.prompt( + PromptLevel::Critical, + "Collaboration disconnected", + Some(detail.as_str()), + &[PromptButton::ok("Ok")], + cx, + )); + }, SessionNotice::ViewRebuilt => { #[cfg(debug_assertions)] push_info("Collaboration view rebuilt from remote state.", window, cx); diff --git a/crates/flowstate/src/collab/presence_view.rs b/crates/flowstate/src/collab/presence_view.rs index d77099cc..0a47fa3a 100644 --- a/crates/flowstate/src/collab/presence_view.rs +++ b/crates/flowstate/src/collab/presence_view.rs @@ -4,25 +4,25 @@ use flowstate_collab::{ SessionId, binding::DocBinding, presence::{PresenceSelection, PresenceStore}, + schema::body_text, }; use loro::{ + ContainerTrait as _, LoroDoc, cursor::{Cursor, PosType, Side}, }; -use crate::rich_text_element::{DocumentOffset, ExternalCaret, ParagraphId, RichTextEditor}; +use crate::rich_text_element::{Document, DocumentOffset, ExternalCaret, RichTextEditor, global_byte, global_to_document_offset}; -pub fn selection_for_editor(editor: &RichTextEditor, binding: &DocBinding) -> Option { +pub fn selection_for_editor(doc: &LoroDoc, editor: &RichTextEditor, _binding: &DocBinding) -> Option { let selection = editor.selection().clone(); - let anchor_paragraph = editor.paragraph_id(selection.anchor.paragraph)?; - let head_paragraph = editor.paragraph_id(selection.head.paragraph)?; Some(PresenceSelection { - anchor: cursor_bytes_for_offset(binding, anchor_paragraph, selection.anchor.byte)?, - head: cursor_bytes_for_offset(binding, head_paragraph, selection.head.byte)?, + anchor: cursor_bytes_for_offset(doc, editor.document(), selection.anchor)?, + head: cursor_bytes_for_offset(doc, editor.document(), selection.head)?, }) } -pub fn external_carets(doc: &LoroDoc, binding: &DocBinding, presence: &PresenceStore) -> Vec { +pub fn external_carets(doc: &LoroDoc, document: &Document, presence: &PresenceStore) -> Vec { let self_key = presence.self_key(); presence .roster() @@ -32,7 +32,7 @@ pub fn external_carets(doc: &LoroDoc, binding: &DocBinding, presence: &PresenceS entry .selection .as_ref() - .and_then(|selection| external_caret_for_presence(doc, binding, selection, entry.color_rgb)) + .and_then(|selection| external_caret_for_presence(doc, document, selection, entry.color_rgb)) }) .collect() } @@ -47,44 +47,34 @@ pub fn collaboration_recovery_path(session: SessionId, title: &str) -> PathBuf { dir.join(format!("{prefix}-{}.db8", sanitized_recovery_title(title))) } -fn cursor_bytes_for_offset(binding: &DocBinding, paragraph: ParagraphId, byte: usize) -> Option> { - let row_ix = binding.by_paragraph.get(¶graph).copied()?; - let row = binding.rows.get(row_ix)?; - let text = row.text.as_ref()?; - let byte = byte.min(text.len_utf8()); +fn cursor_bytes_for_offset(doc: &LoroDoc, document: &Document, offset: DocumentOffset) -> Option> { + let text = body_text(doc); + let byte = global_byte(document, offset).min(text.len_utf8()); let pos = text.convert_pos(byte, PosType::Bytes, PosType::Unicode)?; text .get_cursor(pos, Side::Middle) .map(|cursor| cursor.encode()) } -fn external_caret_for_presence(doc: &LoroDoc, binding: &DocBinding, selection: &PresenceSelection, color_rgb: u32) -> Option { +fn external_caret_for_presence( + doc: &LoroDoc, + document: &Document, + selection: &PresenceSelection, + color_rgb: u32, +) -> Option { let cursor = Cursor::decode(&selection.head).ok()?; - let row_ix = binding.by_container.get(&cursor.container).copied()?; - let row = binding.rows.get(row_ix)?; - let text = row.text.as_ref()?; + let text = body_text(doc); + if cursor.container != text.id() { + return None; + } let resolved = doc.get_cursor_pos(&cursor).ok()?; let byte = text.convert_pos(resolved.current.pos, PosType::Unicode, PosType::Bytes)?; - let paragraph = paragraph_ordinal_for_row(binding, row_ix)?; Some(ExternalCaret { - offset: DocumentOffset { paragraph, byte }, + offset: global_to_document_offset(document, byte), color_rgb, }) } -fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option { - let mut paragraph = 0; - for (row_ix, row) in binding.rows.iter().enumerate() { - if row_ix == target_row { - return row.paragraph_id.map(|_| paragraph); - } - if row.paragraph_id.is_some() { - paragraph += 1; - } - } - None -} - fn sanitized_recovery_title(title: &str) -> String { let mut out = String::new(); for ch in title.chars() { diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index 25fe648b..a263ae0c 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -75,6 +75,7 @@ pub enum SessionNotice { PeerJoined(String), PeerLeft(String), LeftSession, + Disconnected(String), ViewRebuilt, IncompatibleVersion(String), } @@ -542,6 +543,9 @@ impl CollabSession { } else { tracing::debug!(session = %self.session, peer = %peer, "local collaboration peer presence already established"); } + if let (Some(editor), Some(presence)) = (self.editor.clone(), self.presence.as_ref()) { + editor.update(cx, |editor, cx| editor.set_own_collaboration_caret_color(Some(presence.self_color()), cx)); + } self.refresh_own_presence(cx); self.endpoint_online = true; self.phase = SessionPhase::Attached(Attachment { @@ -601,6 +605,10 @@ impl CollabSession { tracing::warn!(session = %self.session, ?reason, phase = ?self.phase, "detaching collaboration session"); let user_left = matches!(reason, DetachReason::UserLeft); + let fatal_detail = match &reason { + DetachReason::Fatal(detail) => Some(detail.clone()), + DetachReason::UserLeft | DetachReason::JoinFailed(_) => None, + }; if let Some(presence) = &self.presence { presence.delete_self(); self.publish_presence_bytes(presence.encode_self()); @@ -617,6 +625,7 @@ impl CollabSession { editor.update(cx, |editor, cx| { editor.set_collab_undo_redirect(None); editor.set_collab_capture(false); + editor.set_own_collaboration_caret_color(None, cx); editor.clear_undo_redo_stacks(); let _ = editor.take_pending_collab_edits(); editor.set_external_carets(Vec::new(), cx); @@ -646,6 +655,8 @@ impl CollabSession { self.phase = SessionPhase::Detached(reason); if user_left { cx.emit(SessionNotice::LeftSession); + } else if let Some(detail) = fatal_detail { + cx.emit(SessionNotice::Disconnected(detail)); } cx.notify(); tracing::info!(session = %self.session, "collaboration session detached and cleaned up"); @@ -658,7 +669,6 @@ impl CollabSession { return Ok(()); } - let document = editor.read(cx).document().clone(); let edits = editor.update(cx, |editor, _| { let edits = editor.take_pending_collab_edits(); editor.clear_undo_redo_stacks(); @@ -668,9 +678,10 @@ impl CollabSession { let operation_count = edits.iter().map(|edit| edit.operations.len()).sum::(); if edit_count == 0 || operation_count == 0 { tracing::trace!(session = %self.session, edit_count, operation_count, "no local collaboration edits to flush"); - } else { - tracing::debug!(session = %self.session, edit_count, operation_count, "flushing local collaboration edits into Loro"); + return Ok(()); } + tracing::debug!(session = %self.session, edit_count, operation_count, "flushing local collaboration edits into Loro"); + let document = editor.read(cx).document().clone(); let Some(doc) = &self.doc else { tracing::warn!(session = %self.session, edit_count, operation_count, "cannot flush local collaboration edits because Loro doc is missing"); return Ok(()); diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs index dd0cf079..1f4efbe2 100644 --- a/crates/flowstate/src/collab/session_io.rs +++ b/crates/flowstate/src/collab/session_io.rs @@ -478,6 +478,9 @@ fn trace_collab_patch(session: SessionId, patch: &CollabPatch) { CollabPatch::ParagraphStyle { row, style } => { tracing::trace!(%session, patch_kind = "paragraph_style", row, ?style, "queued collaboration patch"); }, + CollabPatch::ParagraphRuns { row, runs } => { + tracing::trace!(%session, patch_kind = "paragraph_runs", row, runs = runs.len(), "queued collaboration patch"); + }, CollabPatch::ReplaceObjectBlock { row, block } => { tracing::trace!(%session, patch_kind = "replace_object_block", row, block_id = ?block.block_id, "queued collaboration patch"); }, diff --git a/crates/flowstate/src/collab/session_presence.rs b/crates/flowstate/src/collab/session_presence.rs index 80b3896b..9fef7235 100644 --- a/crates/flowstate/src/collab/session_presence.rs +++ b/crates/flowstate/src/collab/session_presence.rs @@ -16,11 +16,13 @@ impl CollabSession { tracing::warn!(session = %self.session, bytes = bytes.len(), error = %format_args!("{error:#}"), "collaboration presence update failed"); return; } - self.emit_presence_roster_diff(before, cx); - self.refresh_peer_count(); + let roster_changed = self.emit_presence_roster_diff(before, cx); + let peer_count_changed = self.refresh_peer_count(); self.evaluate_connectivity(cx); self.refresh_external_carets(cx); - cx.notify(); + if roster_changed || peer_count_changed { + cx.notify(); + } } else { tracing::debug!(session = %self.session, bytes = bytes.len(), "ignored collaboration presence update before local presence was established"); } @@ -58,13 +60,13 @@ impl CollabSession { .detach(); } - pub(super) fn remove_outdated_presence(&mut self, cx: &mut Context) { + pub(super) fn remove_outdated_presence(&mut self, cx: &mut Context) -> bool { let Some(presence) = &self.presence else { - return; + return false; }; let before = remote_roster(presence); presence.remove_outdated(); - self.emit_presence_roster_diff(before, cx); + self.emit_presence_roster_diff(before, cx) } pub(super) fn refresh_external_carets(&mut self, cx: &mut Context) { @@ -76,15 +78,12 @@ impl CollabSession { tracing::trace!(session = %self.session, "skipping external caret refresh because Loro doc is missing"); return; }; - let Some(binding) = &self.binding else { - tracing::trace!(session = %self.session, "skipping external caret refresh because binding is missing"); - return; - }; let Some(editor) = self.editor.clone() else { tracing::trace!(session = %self.session, "skipping external caret refresh because editor is missing"); return; }; - let carets = presence_view::external_carets(doc, binding, presence); + let document = editor.read(cx).document().clone(); + let carets = presence_view::external_carets(doc, &document, presence); tracing::trace!(session = %self.session, carets = carets.len(), "refreshing collaboration external carets"); editor.update(cx, |editor, cx| editor.set_external_carets(carets, cx)); } @@ -118,7 +117,7 @@ impl CollabSession { } } - pub(super) fn refresh_peer_count(&mut self) { + pub(super) fn refresh_peer_count(&mut self) -> bool { let peers_present = self.peers_present(); let session_id = self.session; if let super::SessionPhase::Attached(attachment) = &mut self.phase { @@ -126,8 +125,10 @@ impl CollabSession { attachment.peers_present = peers_present; if previous != peers_present { tracing::info!(session = %session_id, previous, peers_present, "collaboration peer presence count changed"); + return true; } } + false } pub(super) fn peers_present(&self) -> usize { @@ -138,16 +139,18 @@ impl CollabSession { } fn own_presence_selection(&self, cx: &mut Context) -> Option { + let doc = self.doc.as_ref()?; let editor = self.editor.as_ref()?.read(cx); let binding = self.binding.as_ref()?; - presence_view::selection_for_editor(editor, binding) + presence_view::selection_for_editor(doc, editor, binding) } - fn emit_presence_roster_diff(&mut self, before: HashMap, cx: &mut Context) { + fn emit_presence_roster_diff(&mut self, before: HashMap, cx: &mut Context) -> bool { let Some(presence) = &self.presence else { - return; + return false; }; let after = remote_roster(presence); + let changed = before != after; for (key, name) in &after { if !before.contains_key(key) { cx.emit(SessionNotice::PeerJoined(name.clone())); @@ -158,6 +161,7 @@ impl CollabSession { cx.emit(SessionNotice::PeerLeft(name)); } } + changed } } diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs index b7ae3d53..4cb9011e 100644 --- a/crates/flowstate/src/collab/session_timers.rs +++ b/crates/flowstate/src/collab/session_timers.rs @@ -181,12 +181,14 @@ impl CollabSession { } if self.presence.is_some() { tracing::trace!(session = %self.session, "collaboration presence timer fired"); - self.remove_outdated_presence(cx); + let roster_changed = self.remove_outdated_presence(cx); self.refresh_own_presence(cx); - self.refresh_peer_count(); + let peer_count_changed = self.refresh_peer_count(); self.evaluate_connectivity(cx); self.refresh_external_carets(cx); - cx.notify(); + if roster_changed || peer_count_changed { + cx.notify(); + } } true } diff --git a/crates/gpui-flowtext/src/collaboration.rs b/crates/gpui-flowtext/src/collaboration.rs index 48950216..1673a55c 100644 --- a/crates/gpui-flowtext/src/collaboration.rs +++ b/crates/gpui-flowtext/src/collaboration.rs @@ -4,6 +4,7 @@ use serde::{Deserialize, Serialize}; use super::{ AssetId, AssetRecord, Block, BlockId, Document, DocumentSpan, InputBlock, InputParagraph, ParagraphId, ParagraphStyle, RunStyles, + TextRun, new_block_id, new_paragraph_id, }; @@ -332,6 +333,10 @@ pub enum CollabPatch { row: usize, style: ParagraphStyle, }, + ParagraphRuns { + row: usize, + runs: Vec, + }, ReplaceObjectBlock { row: usize, block: CollabStructuralBlock, diff --git a/crates/gpui-flowtext/src/rich_text/benchmarks/layout.rs b/crates/gpui-flowtext/src/rich_text/benchmarks/layout.rs index 762613d3..ab82d923 100644 --- a/crates/gpui-flowtext/src/rich_text/benchmarks/layout.rs +++ b/crates/gpui-flowtext/src/rich_text/benchmarks/layout.rs @@ -40,7 +40,7 @@ fn benchmark_layout_paths( let paint_bounds = Bounds::new(point(px(0.0), px(0.0)), size(width_px, layout.size.height)); let paint_plain = include_paint.then(|| { repeated(iterations, || { - paint_layout(&layout, paint_bounds, None, None, false, px(1.0), &[], &[], None, window, cx); + paint_layout(&layout, paint_bounds, None, None, false, px(1.0), None, &[], &[], None, window, cx); }) }); let selection = top_selection(document); @@ -53,6 +53,7 @@ fn benchmark_layout_paths( None, false, px(1.0), + None, &[], &[], None, diff --git a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs index 885f25e3..d1b0c294 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs @@ -16,7 +16,12 @@ impl RichTextEditor { self.identity_map.reconcile(&self.document); self.last_collaboration_edit = None; self.layout_invalidation_hint = invalidation; - self.after_text_mutation(cx); + let generation = self.next_edit_generation; + self.next_edit_generation = self.next_edit_generation.wrapping_add(1); + self.mark_document_changed_with_ops(generation, false, None, cx); + // Remote collaboration patches should update this editor in place, but + // should not scroll the viewport as if the local user typed the change. + self.after_formatting_mutation(cx); self.layout_invalidation_hint = None; } @@ -49,6 +54,17 @@ impl RichTextEditor { extend_invalidation(invalidation, paragraph_ix..paragraph_ix + 1); } }, + CollabPatch::ParagraphRuns { row, runs } => { + if let Some(paragraph_ix) = self.paragraph_ix_for_block(*row) + && let Some(paragraph) = paragraphs_mut(&mut self.document).get_mut(paragraph_ix) + { + paragraph.runs.clone_from(runs); + bump_paragraph_version(paragraph); + update_paragraph_block(&mut self.document, paragraph_ix); + rebuild_document_sections(&mut self.document); + extend_invalidation(invalidation, paragraph_ix..paragraph_ix + 1); + } + }, CollabPatch::ReplaceObjectBlock { row, block } => { let mut blocks = collab_structural_blocks_from_document(&self.document); if *row < blocks.len() { diff --git a/crates/gpui-flowtext/src/rich_text/editor/commands.rs b/crates/gpui-flowtext/src/rich_text/editor/commands.rs index 68ecb2b2..c68854a4 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/commands.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/commands.rs @@ -583,6 +583,9 @@ impl RichTextEditor { if text.trim().is_empty() { return; } + if self.insert_text_into_selected_equation(text, cx) { + return; + } if self.insert_plain_text_into_selected_table_cell(text, cx) { return; } diff --git a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs index 99ae0f6f..1f919817 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs @@ -43,6 +43,7 @@ impl RichTextEditor { suppress_collab_capture: 0, collab_undo_redirect: None, collaboration_role: None, + own_collaboration_caret_color_rgb: None, recovery_write_in_progress: false, recovery_write_pending: false, last_recovery_generation: 0, @@ -162,6 +163,7 @@ impl RichTextEditor { self.suppress_collab_capture = 0; self.collab_undo_redirect = None; self.collaboration_role = None; + self.own_collaboration_caret_color_rgb = None; self.recovery_write_in_progress = false; self.recovery_write_pending = false; self.paste_cache = None; @@ -260,6 +262,11 @@ impl RichTextEditor { } } + #[cfg(test)] + pub(crate) fn pending_scroll_head_after_layout_for_test(&self) -> bool { + self.pending_scroll_head_after_layout + } + pub fn set_collab_undo_redirect(&mut self, hook: Option>) { self.collab_undo_redirect = hook; } @@ -502,6 +509,28 @@ impl RichTextEditor { cx.notify(); } + pub fn set_show_own_collaboration_caret_color(&mut self, enabled: bool, cx: &mut Context) { + if self.config.show_own_collaboration_caret_color != enabled { + self.config.show_own_collaboration_caret_color = enabled; + cx.notify(); + } + } + + pub fn set_own_collaboration_caret_color(&mut self, color_rgb: Option, cx: &mut Context) { + if self.own_collaboration_caret_color_rgb != color_rgb { + self.own_collaboration_caret_color_rgb = color_rgb; + cx.notify(); + } + } + + pub(super) fn local_caret_color_rgb(&self) -> Option { + self + .config + .show_own_collaboration_caret_color + .then_some(self.own_collaboration_caret_color_rgb) + .flatten() + } + pub fn save_status(&self) -> &SaveStatus { &self.save_status } diff --git a/crates/gpui-flowtext/src/rich_text/editor/mod.rs b/crates/gpui-flowtext/src/rich_text/editor/mod.rs index 2174d5c3..050e43d3 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mod.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mod.rs @@ -455,12 +455,16 @@ pub struct RichTextEditorStyleState { #[derive(Clone, Debug, Eq, PartialEq)] pub struct RichTextEditorConfig { pub smart_word_selection: bool, + pub show_own_collaboration_caret_color: bool, } #[hotpath::measure_all] impl Default for RichTextEditorConfig { fn default() -> Self { - Self { smart_word_selection: true } + Self { + smart_word_selection: true, + show_own_collaboration_caret_color: true, + } } } @@ -880,6 +884,7 @@ pub struct RichTextEditor { suppress_collab_capture: u32, collab_undo_redirect: Option>, collaboration_role: Option, + own_collaboration_caret_color_rgb: Option, recovery_write_in_progress: bool, recovery_write_pending: bool, last_recovery_generation: u64, diff --git a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs index 17793784..800c8e77 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs @@ -27,6 +27,11 @@ impl RichTextEditor { cx.notify(); } + #[cfg(test)] + pub(super) fn select_equation_block_for_test(&mut self, block_ix: usize, cx: &mut Context) { + self.select_block(BlockSelection::Equation(block_ix), cx); + } + fn select_block_from_click( &mut self, block_ix: usize, diff --git a/crates/gpui-flowtext/src/rich_text/editor/platform.rs b/crates/gpui-flowtext/src/rich_text/editor/platform.rs index 4d33cf68..562127b3 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/platform.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/platform.rs @@ -46,6 +46,9 @@ impl RichTextEditor { } fn replace_text_for_utf16_range(&mut self, range: Range, text: &str, cx: &mut Context) { + if self.insert_text_into_selected_object_text(text, cx) { + return; + } let offsets = self.utf16_range_to_document_offsets(range); let selection = EditorSelection { anchor: offsets.start, @@ -73,6 +76,11 @@ impl RichTextEditor { self.emit_selection_changed(cx); } } + + #[cfg(test)] + pub(super) fn replace_selected_text_from_platform_for_test(&mut self, text: &str, cx: &mut Context) { + self.replace_text_for_utf16_range(self.selection_utf16_range(), text, cx); + } } #[hotpath::measure_all] diff --git a/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs b/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs index 87d9386a..86ae28d6 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs @@ -1,5 +1,15 @@ #[hotpath::measure_all] impl RichTextEditor { + fn insert_text_into_selected_object_text(&mut self, text: &str, cx: &mut Context) -> bool { + if self.insert_text_into_selected_table_cell(text, cx) { + return true; + } + if self.insert_text_into_selected_equation(text, cx) { + return true; + } + false + } + fn insert_text_into_selected_table_cell(&mut self, text: &str, cx: &mut Context) -> bool { let Some(BlockSelection::TableCell { block_ix, row_ix, cell_ix }) = self.selected_block else { return false; diff --git a/crates/gpui-flowtext/src/rich_text/element.rs b/crates/gpui-flowtext/src/rich_text/element.rs index d60b3e99..ba19611b 100644 --- a/crates/gpui-flowtext/src/rich_text/element.rs +++ b/crates/gpui-flowtext/src/rich_text/element.rs @@ -161,7 +161,7 @@ impl Element for RichTextDocumentElement { cx: &mut App, ) { if let Some((layout, bounds)) = self.layout.positioned() { - paint_layout(layout.as_ref(), bounds, None, None, false, px(1.0), &[], &[], None, window, cx); + paint_layout(layout.as_ref(), bounds, None, None, false, px(1.0), None, &[], &[], None, window, cx); } } } @@ -239,7 +239,7 @@ impl Element for VirtualParagraphChunkElement { window: &mut Window, cx: &mut App, ) { - let (selection, drag_selection, caret_offset, caret_width, external_carets, search_highlights, active_search_highlight) = { + let (selection, drag_selection, caret_offset, caret_width, caret_color_rgb, external_carets, search_highlights, active_search_highlight) = { let editor = self.editor.read(cx); let drag_selection = editor.drag_source_selection(); let external_carets = editor.external_carets_for_paragraph(self.paragraph_ix); @@ -253,6 +253,7 @@ impl Element for VirtualParagraphChunkElement { && editor.focus_handle.is_focused(window)) .then_some(editor.selection.head), editor.caret_paint_width(), + editor.local_caret_color_rgb(), external_carets, editor.search_highlights.clone(), editor.active_search_highlight, @@ -300,6 +301,7 @@ impl Element for VirtualParagraphChunkElement { drag_selection.as_ref(), show_caret, caret_width, + caret_color_rgb, &external_carets, &search_highlights, active_search_highlight, diff --git a/crates/gpui-flowtext/src/rich_text/paint.rs b/crates/gpui-flowtext/src/rich_text/paint.rs index 3a295249..2254c271 100644 --- a/crates/gpui-flowtext/src/rich_text/paint.rs +++ b/crates/gpui-flowtext/src/rich_text/paint.rs @@ -12,6 +12,7 @@ pub(super) fn paint_layout( drag_selection: Option<&EditorSelection>, show_caret: bool, caret_width: Pixels, + caret_color_rgb: Option, external_carets: &[ExternalCaret], search_highlights: &[Range], active_search_highlight: Option, @@ -103,13 +104,14 @@ pub(super) fn paint_layout( && caret.intersects(&content_mask) { caret.size.width = caret_width; - window.paint_quad(fill(snap_vertical_rule_to_device_pixels(caret, window), black())); + let caret_color = caret_color_rgb.map_or_else(|| Background::from(black()), |color_rgb| Background::from(rgb(color_rgb))); + window.paint_quad(fill(snap_vertical_rule_to_device_pixels(caret, window), caret_color)); } for external_caret in external_carets { if let Some(mut caret) = caret_bounds(layout, external_caret.offset, bounds.origin) && caret.intersects(&content_mask) { - caret.size.width = px(2.0); + caret.size.width = caret_width; window.paint_quad(fill( snap_vertical_rule_to_device_pixels(caret, window), Background::from(rgb(external_caret.color_rgb)), diff --git a/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs index 55985187..ab8ae91e 100644 --- a/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs +++ b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs @@ -65,6 +65,88 @@ fn collab_capture_fast_path_emits_single_grapheme_deltas(cx: &mut gpui::TestAppC assert!(edits_after_paste_undo.is_empty()); } +#[gpui::test] +fn applying_collab_patches_does_not_arm_local_caret_scroll(cx: &mut gpui::TestAppContext) { + let editor = cx.update(|cx| cx.new(|cx| RichTextEditor::new_with_path(blank_document(), None, cx))); + + cx.update(|cx| { + editor.update(cx, |editor, cx| { + assert!(!editor.pending_scroll_head_after_layout_for_test()); + let before_generation = editor.edit_generation(); + editor.apply_collab_patches( + &[CollabPatch::ParagraphText { + row: 0, + new: InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain("remote")], + }, + delta_utf8: vec![CollabTextDelta::Insert("remote".len())], + }], + cx, + ); + assert!(!editor.pending_scroll_head_after_layout_for_test()); + assert!(editor.edit_generation() > before_generation); + }); + }); +} + +#[gpui::test] +fn own_collaboration_caret_color_can_be_toggled_off(cx: &mut gpui::TestAppContext) { + let editor = cx.update(|cx| cx.new(|cx| RichTextEditor::new_with_path(blank_document(), None, cx))); + + cx.update(|cx| { + editor.update(cx, |editor, cx| { + editor.set_own_collaboration_caret_color(Some(0x3b82f6), cx); + assert_eq!(editor.local_caret_color_rgb(), Some(0x3b82f6)); + editor.set_show_own_collaboration_caret_color(false, cx); + assert_eq!(editor.local_caret_color_rgb(), None); + editor.set_show_own_collaboration_caret_color(true, cx); + assert_eq!(editor.local_caret_color_rgb(), Some(0x3b82f6)); + }); + }); +} + +#[gpui::test] +fn text_entry_in_selected_equation_updates_equation_only(cx: &mut gpui::TestAppContext) { + let mut document = document_from_input( + DocumentTheme::default(), + vec![InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain("body")], + }], + ); + document.blocks = std::sync::Arc::new(vec![ + Block::Paragraph(document.paragraphs[0].clone()), + Block::Equation(EquationBlock { + source: "x".into(), + syntax: EquationSyntax::Latex, + display: EquationDisplay::Display, + version: 0, + }), + ]); + let editor = cx.update(|cx| cx.new(|cx| RichTextEditor::new_with_path(document, None, cx))); + + let (document, edits) = cx.update(|cx| { + editor.update(cx, |editor, cx| { + editor.set_collab_capture(true); + editor.select_equation_block_for_test(1, cx); + editor.insert_plain_text_from_toolkit("+1", cx); + editor.replace_selected_text_from_platform_for_test("+2", cx); + (editor.document().clone(), editor.take_pending_collab_edits()) + }) + }); + + assert_eq!(paragraph_text(&document, 0), "body"); + let Block::Equation(equation) = &document.blocks[1] else { + panic!("expected equation block after toolkit text insert"); + }; + assert_eq!(equation.source.as_ref(), "x+1+2"); + assert_eq!(edits.len(), 2); + for edit in edits { + assert!(matches!(edit.operations.as_slice(), [CanonicalOperation::ReplaceBlock { .. }])); + } +} + #[gpui::test] fn select_all_emits_selection_changed_once(cx: &mut gpui::TestAppContext) { let document = document_from_input( From 526d0cacfad5ef831b0f91b36f675f54106e260d Mon Sep 17 00:00:00 2001 From: "Adam D. Humphrey" Date: Tue, 16 Jun 2026 22:22:38 -0500 Subject: [PATCH 15/45] Fixed double insert --- .../gpui-flowtext/src/rich_text/editor/action_handlers.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/crates/gpui-flowtext/src/rich_text/editor/action_handlers.rs b/crates/gpui-flowtext/src/rich_text/editor/action_handlers.rs index 267e82ec..dd4f6039 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/action_handlers.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/action_handlers.rs @@ -233,24 +233,30 @@ impl RichTextEditor { key_char.to_string() }; if self.insert_text_into_selected_table_cell(&key_char, cx) { + cx.stop_propagation(); return; } if self.insert_text_into_selected_equation(&key_char, cx) { + cx.stop_propagation(); return; } self.insert_text_command(&key_char, cx); + cx.stop_propagation(); } #[cfg(not(target_os = "windows"))] { let _ = window; if self.insert_text_into_selected_table_cell(key_char, cx) { + cx.stop_propagation(); return; } if self.insert_text_into_selected_equation(key_char, cx) { + cx.stop_propagation(); return; } self.insert_text_command(key_char, cx); + cx.stop_propagation(); } } From 1a9ff7cdc8f5a2f642a84584f29de49c559e6492 Mon Sep 17 00:00:00 2001 From: Chinmay Khaladkar Date: Tue, 16 Jun 2026 20:31:00 -0700 Subject: [PATCH 16/45] newline stuff --- crates/flowstate/src/collab/session.rs | 2 ++ crates/gpui-flowtext/src/collaboration.rs | 10 ---------- .../src/rich_text/editor/edit_pipeline.rs | 6 ++---- 3 files changed, 4 insertions(+), 14 deletions(-) diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index a263ae0c..ab020762 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -623,6 +623,8 @@ impl CollabSession { if let Some(editor) = self.editor.clone() { editor.update(cx, |editor, cx| { + editor.set_recovery_path(None, cx); + editor.set_collaboration_role(None, cx); editor.set_collab_undo_redirect(None); editor.set_collab_capture(false); editor.set_own_collaboration_caret_color(None, cx); diff --git a/crates/gpui-flowtext/src/collaboration.rs b/crates/gpui-flowtext/src/collaboration.rs index 1673a55c..b1edfb22 100644 --- a/crates/gpui-flowtext/src/collaboration.rs +++ b/crates/gpui-flowtext/src/collaboration.rs @@ -5,7 +5,6 @@ use serde::{Deserialize, Serialize}; use super::{ AssetId, AssetRecord, Block, BlockId, Document, DocumentSpan, InputBlock, InputParagraph, ParagraphId, ParagraphStyle, RunStyles, TextRun, - new_block_id, new_paragraph_id, }; #[derive(Clone, Copy, Debug, Eq, PartialEq, Hash, Serialize, Deserialize)] @@ -48,15 +47,6 @@ impl DocumentIdentityMap { } } - pub fn insert_split_paragraph(&mut self, paragraph_ix: usize, block_ix: usize) { - self - .paragraph_ids - .insert((paragraph_ix + 1).min(self.paragraph_ids.len()), new_paragraph_id()); - let block_insert_ix = (block_ix + 1).min(self.block_ids.len()); - self.block_ids.insert(block_insert_ix, new_block_id()); - self.table_cell_ids.insert(block_insert_ix, Vec::new()); - } - #[must_use] pub fn paragraph_id(&self, paragraph_ix: usize) -> Option { self.paragraph_ids.get(paragraph_ix).copied() diff --git a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs index c4416d04..b67c2f4d 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs @@ -198,7 +198,7 @@ impl RichTextEditor { self.mark_document_changed_with_ops(after_generation, identity_shape_changed, Some(&collab_operations), cx); } - fn insert_paragraph_break_at_caret(&mut self, caret: DocumentOffset, block_ix: usize, cx: &mut Context) { + fn insert_paragraph_break_at_caret(&mut self, caret: DocumentOffset, _block_ix: usize, cx: &mut Context) { let before_selection = self.selection.clone(); let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; @@ -209,9 +209,7 @@ impl RichTextEditor { self.insert_paragraph_break(cx); self.suppress_mutation_notify = self.suppress_mutation_notify.saturating_sub(1); self.layout_invalidation_hint = None; - self - .identity_map - .insert_split_paragraph(caret.paragraph, block_ix); + self.identity_map.reconcile(&self.document); let after_span = capture_document_span(&self.document, caret.paragraph..caret.paragraph + 2); if before_span == after_span && before_selection == self.selection { From ae8f0cde9d73bb1358139e87c248c4177f85501e Mon Sep 17 00:00:00 2001 From: Adhi Thirumala Date: Tue, 16 Jun 2026 23:34:26 -0500 Subject: [PATCH 17/45] loro root --- FIX_LORO_ROOT.md | 614 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 614 insertions(+) create mode 100644 FIX_LORO_ROOT.md diff --git a/FIX_LORO_ROOT.md b/FIX_LORO_ROOT.md new file mode 100644 index 00000000..c7f32044 --- /dev/null +++ b/FIX_LORO_ROOT.md @@ -0,0 +1,614 @@ +# FIX_LORO_ROOT — Make the single-root LoroText model fast and correct + +> Goal: eliminate expensive full-document rewrites/reprojections while preserving +> the single-root `LoroText` design and CRDT merge correctness. This document is +> the execution plan: scoped tasks, exact code locations, dependencies, and a +> parallelization matrix so multiple subagents can work without colliding. + +--- + +## 0. TL;DR + +The collab core already migrated to a **single-root `LoroText`** design (one root +text container `"body"`, paragraphs delimited by `\n`; the `"blocks"` movable list +holds only metadata maps). The migration is **functionally complete but performance- +and merge-incomplete**: the supporting infrastructure that makes single-root cheap +(a maintained paragraph→byte offset index, incremental span splices, delta-driven +remote reconcile) was never added. As a result, **almost every edit that is not a +single-character keystroke collapses into a whole-document rewrite**, and every +remote text change re-parses and re-compares the entire document. + +There is **no second text layout in the code** (no leftover per-paragraph text +containers). The conflicts are: + +1. **Performance/merge debt** that fights the single-root model (the main work). +2. **Stale design docs** (`plan.md`, `fix.md`) that still describe the *abandoned* + per-paragraph-`LoroText` schema (v1). These are misleading but harmless to code. +3. **Dead pre-CRDT code paths** (op-replay remote apply + a postcard wire format) + left behind by the migration. +4. **One redundant source of truth**: paragraph style is stored twice + (block-map `STYLE` + body `pstyle` mark). + +The fix is organized into three independent groups (collab core, editor model, +cleanup) that can run in parallel. + +--- + +## 1. The invariant (read before changing anything) + +The single-root model must hold these invariants after **every** operation: + +- **INV-1 (text):** `body_text(doc)` UTF-8 content == `full_document_text(document)` + (all paragraphs joined by `'\n'`). A debug helper already checks this: + `debug_assert_body_matches_document` (`crates/flowstate-collab/src/local_apply.rs`). +- **INV-2 (marks):** for every paragraph, the body marks (`sem`/`ul`/`strike`/`hl` + inline, `pstyle` paragraph) over that paragraph's byte range match the paragraph's + runs/style. Whole-body rewrite is correct today *because* it reapplies all marks + (`replace_body_from_document` → `apply_mark_intervals`, `projection.rs:148-160`); + any incremental replacement **must** carry marks for the spliced range. +- **INV-3 (binding):** `DocBinding.rows` is 1:1 and in-order with `document.blocks` + (kind, block id, paragraph id, version). Checked by + `DocBinding::assert_consistent` (`binding.rs`). +- **INV-4 (convergence):** concurrent edits converge to identical state. Minimal, + intention-preserving Loro deltas are required — a full body `delete(0,len)+insert` + destroys concurrent peers' insertion anchoring even when text is unchanged. + +Every task below must keep INV-1..4. When in doubt, fall back to the existing +whole-document path (`replace_document`) rather than risk divergence. + +--- + +## 2. Current full-rewrite / full-reprojection map (the debt) + +Verified locations (line numbers approximate; anchor on symbol names): + +| Path | Location | Triggered by | Cost | +|---|---|---|---| +| `replace_paragraph_span` discards span, calls `replace_document` | `flowstate-collab/src/local_apply.rs:276-285` | **Almost every non-typing edit**: delete/backspace, selection-replace, IME/multi-char, Enter, paste, drag-move | full block-list + body rewrite + `DocBinding::build` | +| `replace_document` (`replace_blocks_from_document` + `DocBinding::build`) | `local_apply.rs:300-304` | `ReplaceParagraphSpan`, `ReplaceDocument`, `ReplaceBlock` fallback | O(doc) | +| `replace_body_from_document` (full body `delete+insert`+remark) | `projection.rs:148-160`, called at `local_apply.rs:207,255,270` | paragraph block insert/delete/move | O(text) CRDT churn | +| `body_paragraph_range` stringifies whole body + linear newline scan | `local_apply.rs:452-455` (`text_content`) + `paragraph_range_in_body_text:481-495` | **every** `InsertText`(fast-path keystroke), `DeleteRange`, `SetRunStyles`, `SetParagraphStyle`, `split_paragraph` | O(n) per call | +| `reconcile_body_text` reprojects ALL blocks + compares EVERY paragraph; Loro `TextDelta` discarded | `remote_apply.rs:37` (`Diff::Text(_)`) + `:50-99` | every remote body change | O(doc) per remote keystroke | +| Self-check timer runs full `document_from_loro` + 2 hash passes | `flowstate/src/collab/session_timers.rs:263-289` | every 30s while attached & idle ≥2s | recurring O(doc) | +| Editor: `update_paragraph_offsets_after_len_change` → tail `to_vec` + `replace_paragraph_blocks` + `rebuild_document_sections` | `gpui-flowtext/src/edit_ops/offsets.rs:59-76` | **every keystroke/backspace** | O(n) | +| Editor: `rebuild_document_sections` (walks all paragraphs) | `gpui-flowtext/src/document/core.rs:335-378` | ~20 call sites incl. per edit | O(n) | +| Editor: `DocumentIdentityMap::reconcile` clones id vecs + walks all blocks; `paragraph_index` linear scan | `gpui-flowtext/src/collaboration.rs:29-77` | every shape-changing edit + remote batch | O(n) | +| Editor: collab structural patches → `rebuild_document_from_collab_structural_blocks` (full rebuild) | `gpui-flowtext/src/rich_text/editor/collab_apply.rs:68-117,240-265`; headless `patch_apply.rs:151-178` | remote block insert/delete/move/object-replace | O(doc) | + +**Key fact:** the granular ops `DeleteRange`/`SplitParagraph`/`JoinParagraphs` +are fully implemented in `local_apply.rs:46-57,93-198` and tested +(`tests/translation.rs`, `tests/convergence.rs`) **but the editor never emits them** +— it emits `ReplaceParagraphSpan` instead. The incremental machinery exists; it is +just bypassed. + +--- + +## 3. Conflicts / leftovers vs. single-root design (inventory) + +- **No leftover per-paragraph text containers.** The only `get_text` is the root + body (`schema.rs:140-142`); `insert_container` only ever inserts maps + (`projection.rs:108,123`). There is **no `TEXT` key**. Schema keys are + `meta/body/blocks`, `schema/session/title`, `kind/style/data/rev`, + `sem/ul/strike/hl/pstyle` (`schema.rs:17-38`). +- **Schema is v2, no migration path.** `SCHEMA_VERSION = 2` (`schema.rs:23`); + `verify_lineage` *rejects* any other version (`projection.rs:204-209`). The `2` + implies a prior v1 (the per-paragraph design in `plan.md`). No upgrade code — old + snapshots are refused, not migrated. (Acceptable; just document it.) +- **Stale docs:** `plan.md` (§5, §5.1, §5.2, §6.1) and `fix.md` describe the + per-paragraph-`LoroText` schema and a `BindingRow{ text: Option }` that + no longer exist, and claim "no global offset arithmetic" — the opposite of the + implemented single-root design. → Task **T14**. +- **Dead pre-CRDT code:** editor-side op-replay remote apply + `apply_remote_operations`/`apply_canonical_operation` + (`gpui-flowtext/src/rich_text/editor/lifecycle.rs:314-480`, no callers) and the + postcard wire format `WireCanonicalOperation` + + `encode_canonical_operations`/`decode_canonical_operations` + (`gpui-flowtext/src/collaboration.rs:144-294`) + `last_collaboration_*` accessors + (no live callers). → Task **T12**. +- **Dual source of truth for paragraph style:** written to block-map `STYLE` + (`projection.rs:110`, `local_apply.rs:60`) *and* as a body `pstyle` mark + (`projection.rs:176-180`, `local_apply.rs:61-62`). On read, the map value is + shadowed by the body mark whenever a body paragraph exists + (`projection.rs:66-78`). → Task **T13**. + +--- + +## 4. Fix architecture + +Four pillars, mapped to tasks: + +1. **Foundation — maintained paragraph offset index (T1).** Store each paragraph's + body byte range in `DocBinding` (prefix-sum / Fenwick, mirroring the editor's + existing `ParagraphOffsetIndex`). Replaces O(n) body stringify+scan and provides + the offset math that incremental splices need. *Everything else builds on this.* +2. **Incremental local apply (T2, T3).** Replace the `ReplaceParagraphSpan`→ + `replace_document` shortcut and the three `replace_body_from_document` block-op + calls with span-scoped body splices + targeted block-row reconcile, with minimal + sub-paragraph text deltas and mark carry-over. Keep `replace_document` as a + safety fallback. +3. **Delta-driven remote reconcile (T4).** Consume the Loro `TextDelta` instead of + discarding it; reconcile only the affected paragraph ordinals (+ neighbors on + newline changes) instead of reprojecting and comparing the whole document. +4. **Editor-model incrementalization (T6–T11).** Make sections/offset/identity/ + structural-patch updates range-scoped instead of O(n) per edit. + +Plus cleanup (T12–T15). + +--- + +## 5. Execution plan (parallelization) + +### File-ownership matrix (avoid edit collisions) + +| Owner / Agent | Crate | Files exclusively owned | +|---|---|---| +| **A — collab offset** | flowstate-collab | `binding.rs`, **new** `body_index.rs`, the 3 range helpers in `local_apply.rs` | +| **B — collab local** | flowstate-collab | `local_apply.rs` op handlers (after A lands) | +| **C — collab remote** | flowstate-collab | `remote_apply.rs` | +| **D — editor model** | gpui-flowtext | `document/core.rs`, `edit_ops/offsets.rs`, `edit_ops/insert_delete.rs`, `edit_ops/text.rs`, `edit_ops/split_delete.rs` | +| **E — editor identity** | gpui-flowtext | `collaboration.rs` (identity map) | +| **F — editor patches** | gpui-flowtext | `rich_text/editor/collab_apply.rs`; flowstate-collab `patch_apply.rs` | +| **G — cleanup** | both | `rich_text/editor/lifecycle.rs` (dead-path removal), `plan.md`, `fix.md` | +| **H — editor emission** | gpui-flowtext | `rich_text/editor/edit_pipeline.rs`, `hit_testing.rs` (after B stable) | + +> Note: Agents **A** and **B** both touch `local_apply.rs`. Run A → B sequentially +> on the **same** worktree/agent, or land A first. Agents **C**, **D**, **E**, **F**, +> **G** are file-disjoint and can run fully in parallel. + +### Waves + +**Wave 1 (parallel):** +- A: **T1** (offset index) — *unblocks B and C.* +- D: **T6** (incremental `rebuild_document_sections`) — *unblocks T7/T8/T9.* +- E: **T10** (identity map diff + hashmap). +- F: **T11** (in-place structural patches). +- G: **T12** (remove dead pre-CRDT paths) + **T14** (retire stale docs). + +**Wave 2 (parallel, after Wave 1 deps):** +- B: **T2** (incremental `ReplaceParagraphSpan`) → then **T3** (block body splices). +- C: **T4** (delta-driven remote reconcile). +- D: **T7** → **T8** → **T9** (editor model splices). +- G: **T13** (style single-source-of-truth) + **T15** (self-check cost). + +**Wave 3 (after B stable):** +- H: **T5** (granular local op emission for delete/Enter). +- All: integration + convergence + clippy pass. + +--- + +## 6. Tasks + +Each task: **Priority · Depends · Files · Problem · Target · Steps · Acceptance · +Tests · Risk/Fallback.** + +--- + +### T1 — Maintained paragraph byte-offset index in `DocBinding` *(FOUNDATION)* + +- **Priority:** Critical · **Depends:** none · **Owner:** A +- **Files:** `crates/flowstate-collab/src/binding.rs`; **new** + `crates/flowstate-collab/src/body_index.rs` (+ register `mod body_index;` in + `crates/flowstate-collab/src/lib.rs`); the three range helpers in `local_apply.rs` + (`body_byte_for_paragraph_byte:435`, `body_range_for_paragraph:444`, + `body_paragraph_range:452`). + +**Problem.** `body_paragraph_range` calls `text_content(&body_text(doc))` (full +`to_delta` → `String`) then `paragraph_range_in_body_text` linear-scans for the Nth +`\n` — O(n) per call, on the keystroke hot path (`local_apply.rs:85,103-104,151`, +`66-67,61`). `DocBinding` (`binding.rs:28-34`) stores no offsets. + +**Target.** O(log n) (or amortized O(1)) paragraph→body-byte mapping with no +whole-body stringification on edits. + +**Steps.** +1. Add a prefix-sum/Fenwick index of paragraph byte widths to `DocBinding` + (mirror `gpui-flowtext` `ParagraphOffsetIndex`, `document/core.rs:401-470`). + Width of paragraph *k* = its UTF-8 byte length; body start of ordinal *k* = + prefix sum + *k* (for the *k* preceding `\n` separators). +2. In `body_index.rs` put: the index type, `paragraph_start(ordinal)`, + `paragraph_range(ordinal)`, `paragraph_ordinal_for_body_byte(byte)`, and a + ranged body parser `input_paragraphs_in_body_range(text, byte_range)` (a scoped + version of `schema.rs:254` `input_paragraphs_from_body_text`) for T4. Also move + the prefix/suffix diff (`remote_apply.rs:223-234` `text_delta_for_replacement`, + `common_prefix_bytes`, `common_suffix_bytes`) here so B and C can reuse it. +3. Maintain the index incrementally: add `DocBinding` methods to update one + paragraph's width, insert/remove paragraph entries, and shift. Call them from the + binding mutators (`push_row`/`insert_row`/`remove_row`/`move_row`) and add a + `refresh_paragraph_widths(doc)` used by `build`. +4. Rewrite the three `local_apply.rs` range helpers to read from the index instead + of `text_content`. Build the index in `DocBinding::build` (`binding.rs:37`). + +**Acceptance.** No `text_content(&body_text(..))` call remains on a per-edit path; +range helpers are O(log n); INV-3 holds; `assert_consistent` extended to verify the +index matches the body. + +**Tests.** `cargo test -p flowstate-collab` (esp. `tests/translation.rs`, +`tests/convergence.rs`); add a unit test: random inserts/deletes keep +`index.paragraph_range(k)` equal to a fresh scan. + +**Risk.** Index drift. Mitigate with a debug assert comparing index vs. fresh scan +after each op (compiled out in release). + +--- + +### T2 — Incremental `ReplaceParagraphSpan` in `LocalApplier` *(highest impact)* + +- **Priority:** Critical · **Depends:** T1 · **Owner:** B +- **Files:** `crates/flowstate-collab/src/local_apply.rs` + (`replace_paragraph_span:276-285`, helpers). + +**Problem.** `replace_paragraph_span` discards `before`/`after`/`start_paragraph` +and calls `replace_document` (full rewrite). This is the dominant production edit +path (delete, selection-replace, IME, Enter, paste, drag). + +**Target.** Span-scoped body splice + targeted block-row reconcile; minimal text +delta; marks carried; fall back to `replace_document` only when ids/ranges can't be +reconciled. + +**Inputs.** `document: &Document` (post-edit), `start_paragraph: Option`, +`before: &DocumentSpan`, `after: &DocumentSpan` where +`DocumentSpan { start_paragraph: usize, paragraphs: Vec, text: String }` +(`gpui-flowtext/src/document/text.rs:74-79`). New paragraph/block ids are read from +`document.ids` (as `split_paragraph` already does, `local_apply.rs:140-150`). + +**Steps.** +1. Resolve start ordinal: from `start_paragraph` id → row → ordinal (via T1); else + `before.start_paragraph`. If unresolved → fallback `replace_document`. +2. `before_n = before.paragraphs.len()`, `after_n = after.paragraphs.len()`. + Compute body range to replace: `start = index.paragraph_start(start_ord)`, + `end = index.paragraph_range(start_ord + before_n - 1).end`. +3. **Text splice (minimal):** common-prefix/suffix-diff `before.text` vs + `after.text` (shared helper from T1); `body.delete_utf8` + `body.insert_utf8` + only the differing middle within `[start..end]`. +4. **Marks:** re-apply run styles (`set_run_styles_utf8`) and `pstyle` + (`set_paragraph_style_utf8`) across the affected paragraphs' new byte ranges so + INV-2 holds even for style-only changes (compute intervals like + `projection.rs:162` `document_mark_intervals`, but only for the span). +5. **Block rows:** `delta = after_n - before_n`. If `>0` insert `delta` paragraph + rows (`insert_paragraph_container` + `BindingRow`) at the right block index with + ids from `document.ids`; if `<0` delete `-delta` rows; for overlapping rows + update `STYLE`/version/`paragraph_id` if changed. (Reuse the same helpers T3 uses.) +6. Update the T1 index for the changed range. `debug_assert_body_matches_document`. +7. Wrap in a result; on any `Err`/inconsistency → `self.replace_document(document)`. + +**Acceptance.** A single-char backspace produces a Loro delta touching only that +paragraph's changed bytes (verify via `subscribe_local_update` byte size or a delta +assert), not the whole body. Paragraph count changes update only the affected rows. + +**Tests.** Extend `tests/convergence.rs` with backspace, selection-replace across +2–3 paragraphs, paste of multiple paragraphs, Enter; assert convergence vs. a peer +and vs. the `replace_document` fallback (differential test: incremental result == +fallback result for the same edit). + +**Risk.** Off-by-one on `\n` boundaries; id assignment. Differential test against +the fallback is the safety net. + +--- + +### T3 — Incremental body splices for paragraph block insert/delete/move + +- **Priority:** Medium (currently dead-in-prod; needed for T5 & headless correctness) + · **Depends:** T1 (and shares helpers with T2) · **Owner:** B +- **Files:** `local_apply.rs` `insert_block:200`, `delete_block:243`, + `move_block:261` (remove the `replace_body_from_document` calls at 207/255/270). + +**Problem.** Paragraph block insert/delete/move rewrite the entire body even though +text content barely changes (INV-4 violation under concurrency). + +**Target.** Splice only the affected slice + separator newline; carry marks. + +**Steps.** +- **insert (paragraph):** insert `text + "\n"` (or `"\n" + text` for last position) + at `index.paragraph_start(ord)`; apply marks for the inserted text; then existing + row insert. Empty paragraph → insert just `"\n"`. +- **delete (paragraph):** delete the paragraph's body range **plus one adjacent + separator newline** (handle first/last paragraph asymmetry); then existing row + delete. +- **move (paragraph):** capture slice text+marks, delete from old position + (+separator), insert at target (+separator), re-apply marks. +- Update T1 index. Keep `replace_body_from_document` only as fallback. + +**Acceptance/Tests.** `tests/convergence.rs` block insert/delete/move converge; +INV-1/2/3 asserts pass. Differential vs. `replace_body_from_document`. + +**Risk.** Newline-adjacency edge cases (first/last/empty doc). Cover with unit tests. + +--- + +### T4 — Delta-driven remote reconciliation + +- **Priority:** High · **Depends:** T1 (offset mapping + ranged parse) · **Owner:** C +- **Files:** `crates/flowstate-collab/src/remote_apply.rs` + (`apply_event:25-48`, `reconcile_body_text:50-99`). + +**Problem.** `apply_event` matches `Diff::Text(_)` and throws the delta away +(`:37`), then `reconcile_body_text` calls `input_blocks_from_loro` (full reproject) +and compares **every** paragraph row. The emitted delta is even recomputed locally +(`text_delta_for_replacement:223-234`) instead of using Loro's. + +**Target.** Use the Loro `TextDelta` to compute affected body ranges → affected +paragraph ordinals (+ neighbors when `\n` is inserted/deleted), and reconcile only +those. + +**Steps.** +1. In `apply_event`, capture the body `TextDelta` (don't discard). Keep the + `body_changed` fast-exit when empty. +2. Walk the delta with a body-offset cursor; collect (a) changed byte ranges and + (b) net `\n` insertions/deletions (paragraph count delta and positions). +3. Map ranges → paragraph ordinals (T1 helpers on the post-import body). Build the + minimal affected ordinal set; expand by ±1 where a newline boundary changed. +4. Re-derive only those paragraphs via `input_paragraphs_in_body_range` (T1) and + emit `ParagraphText`/`ParagraphStyle`/`ParagraphRuns` patches for them; for + newline insert/delete, emit structural `InsertBlocks`/`DeleteBlocks` + reconcile + the binding rows for just that region (reuse `reconcile_binding_to_match_blocks`). +5. Prefer Loro's delta offsets for `delta_utf8` in `ParagraphText` (still fine to + recompute per-paragraph prefix/suffix for the editor's selection remap). +6. **Fallback:** if the delta can't be cleanly mapped (overlapping/ambiguous), + call the existing `reconcile_body_text` (keep it) for that event. + +**Acceptance.** A remote single-char insert in an N-paragraph doc parses/compares +O(1) paragraphs, not N. Convergence unchanged. + +**Tests.** `tests/remote_apply.rs`, `tests/convergence.rs`: remote single-char +insert/delete, remote split/join (newline insert/delete), concurrent edits to +different paragraphs (must not touch each other's patches), style-only change. + +**Risk.** Highest-complexity task. The whole-compare fallback guarantees +correctness; optimize incrementally and gate behind the fallback. + +--- + +### T5 — Emit granular local ops for delete / Enter (optional, complements T2) + +- **Priority:** Low/Medium · **Depends:** T2 stable · **Owner:** H +- **Files:** `gpui-flowtext/src/rich_text/editor/edit_pipeline.rs`, + `hit_testing.rs`. + +**Problem.** Backspace/Delete/Enter go through the generic span path. The exact ops +`DeleteRange`/`JoinParagraphs`/`SplitParagraph` already exist and are tested in +`local_apply.rs` but are never emitted, so deltas are larger than necessary. + +**Target.** Add fast paths analogous to the single-grapheme insert fast path +(`edit_pipeline.rs:3-113`) that emit `DeleteRange` (in-paragraph delete), +`JoinParagraphs` (boundary backspace/delete), `SplitParagraph` (Enter), keeping +`ReplaceParagraphSpan` (now incremental via T2) as the fallback for compound edits. + +**Acceptance/Tests.** `tests/collab_capture.rs`-style assertions that backspace +emits `DeleteRange`, Enter emits `SplitParagraph`; convergence unchanged. + +**Risk.** Low (handlers already tested); purely shrinks deltas. + +--- + +### T6 — Incremental `rebuild_document_sections` *(editor foundation)* + +- **Priority:** High · **Depends:** none · **Owner:** D +- **Files:** `gpui-flowtext/src/document/core.rs:335-378`. + +**Problem.** Walks all paragraphs to rebuild the heading outline; called from ~20 +sites including transitively per keystroke (via `replace_paragraph_blocks`). + +**Target.** Dirty-range/incremental section recomputation (only rebuild sections +intersecting the changed paragraph range; reuse stable `SectionId`s). + +**Acceptance.** Single-paragraph edit does not re-walk all paragraphs for sections. +**Tests.** `cargo test -p gpui-flowtext` (section/outline tests); add a test that +editing paragraph *k* leaves untouched sections' `SectionId`s stable. +**Risk.** Section nesting/stack recomputation at range edges — include boundary tests. + +--- + +### T7 — In-place `replace_paragraph_blocks` for single-paragraph updates + +- **Priority:** High · **Depends:** T6 · **Owner:** D +- **Files:** `gpui-flowtext/src/document/core.rs:158-215`. + +**Problem.** Full block walk + unconditional `rebuild_document_sections` even when a +single paragraph's content changed in place (paragraph-only docs have +`blocks.len()==paragraphs.len()`, so `update_paragraph_block` alone suffices). + +**Target.** Fast path: when count is unchanged and only content changed, update the +single block + the affected section range (T6) without the full walk. + +**Acceptance/Tests/Risk.** As T6; differential vs. current output for mixed +paragraph/object docs. + +--- + +### T8 — Make `update_paragraph_offsets_after_len_change` O(log n) + +- **Priority:** High · **Depends:** T6, T7 · **Owner:** D +- **Files:** `gpui-flowtext/src/edit_ops/offsets.rs:59-76` (callers + `insert_delete.rs:84,125`). + +**Problem.** Per keystroke it does: O(log n) Fenwick update (good) **then** tail +`refresh_paragraph_ranges_from` + tail `.to_vec()` clone + full +`replace_paragraph_blocks` + full `rebuild_document_sections` — net O(n). + +**Target.** Keep the Fenwick update; drop the tail clone and full block/section +rebuild for the in-place single-paragraph case (use T7's in-place update and +incremental `byte_range` shift). + +**Acceptance.** Typing/backspace in a large doc is O(log n) at the model layer. +**Tests.** Offset-index/byte-range correctness after random edits; existing editor +tests. **Risk.** `byte_range` cache staleness for the tail — shift offsets via the +Fenwick prefix instead of rewriting each. + +--- + +### T9 — Scope rebuilds in span replacement / paragraph-content apply + +- **Priority:** Medium · **Depends:** T6 · **Owner:** D +- **Files:** `gpui-flowtext/src/edit_ops/text.rs:75-97` + (`apply_document_span_replacement`); `rich_text/editor/collab_apply.rs:166-181` + (`replace_paragraph_content`). + +**Problem.** Both unconditionally call `rebuild_document_offset_index` (O(n)) and +`rebuild_document_sections` (O(n)) after a span/paragraph-scoped change. + +**Target.** Use the incremental offset update (T8) and dirty-range sections (T6); +scope to the touched paragraph range. + +**Acceptance/Tests.** Collab `ParagraphText`/`ParagraphStyle`/`ParagraphRuns` and +local span edits update only the touched range; editor tests pass. + +--- + +### T10 — Diff-based `DocumentIdentityMap::reconcile` + hashmap lookup + +- **Priority:** Medium · **Depends:** none · **Owner:** E +- **Files:** `gpui-flowtext/src/collaboration.rs:13-77`. + +**Problem.** `reconcile` clones both id vectors and walks all blocks every call +(`:29-77`); `paragraph_index` is an O(n) linear scan (`:70-76`) used per remote/ +canonical op. + +**Target.** Maintain a `FxHashMap` (and block id map); update +incrementally; make `reconcile` diff only changed ranges. + +**Acceptance/Tests.** `paragraph_index` is O(1); identity tests pass; remote apply +of many small ops doesn't show O(n²) behavior. +**Risk.** Keeping the map in sync across all edit paths — centralize updates. + +--- + +### T11 — In-place collab structural patches (no full document rebuild) + +- **Priority:** Medium · **Depends:** none (coordinate with T4 output shape) · **Owner:** F +- **Files:** `gpui-flowtext/src/rich_text/editor/collab_apply.rs:68-117,240-265`; + headless `crates/flowstate-collab/src/patch_apply.rs:46-77,151-178`. + +**Problem.** `InsertBlocks`/`DeleteBlocks`/`MoveBlock`/`ReplaceObjectBlock` call +`rebuild_document_from_collab_structural_blocks` / `rebuild_document_from_structural_blocks` +— a full `Document` reconstruction (+ whole-doc layout invalidation). + +**Target.** Splice blocks/paragraphs in place (insert/remove/move the affected rows, +shift offsets via T8/T6 incremental APIs), invalidate only the affected range. + +**Acceptance/Tests.** Remote block insert/delete/move updates only affected rows; +`patch_apply` headless tests + convergence pass. +**Risk.** Keeping `ids`, `offset_index`, `sections` consistent during in-place +splice — assert equality vs. the rebuild path in tests. + +--- + +### T12 — Remove dead pre-CRDT code paths + +- **Priority:** Medium (low risk) · **Depends:** none · **Owner:** G +- **Files:** `gpui-flowtext/src/rich_text/editor/lifecycle.rs:314-480` + (`apply_remote_operations`, `apply_canonical_operation`), + `gpui-flowtext/src/collaboration.rs:144-294` + (`WireCanonicalOperation`, `encode_canonical_operations`, + `decode_canonical_operations`), and `last_collaboration_*` accessors + (`lifecycle.rs:232-248`). + +**Problem.** Orphaned op-replay remote path and postcard wire format from before the +Loro-update-bytes sync; no live callers (remote apply now flows through +`apply_collab_patches`). + +**Steps.** Confirm no callers (`rg` for each symbol across `crates/`, excluding +tests/defs), remove, `cargo check`/`cargo clippy`. **Do NOT** remove the +`CanonicalOperation` enum or the `local_apply.rs` handlers for +`DeleteRange`/`SplitParagraph`/`JoinParagraphs` — those are the incremental local +apply machinery (kept/used by T2/T3/T5). + +**Acceptance.** Build + clippy clean; tests pass; dead symbols gone. + +--- + +### T13 — Single source of truth for paragraph style + +- **Priority:** Medium · **Depends:** none (coordinate with T2/T4) · **Owner:** G +- **Files:** `schema.rs` (`pstyle` mark fns), `projection.rs:66-78,110,176-180`, + `local_apply.rs:58-63`, `remote_apply.rs` (style diff path). + +**Problem.** Style stored twice: block-map `STYLE` (write-only on the normal read +path) and body `pstyle` mark (authoritative when a body paragraph exists). Redundant +work + drift risk. + +**Decision (recommended):** make the **block-map `STYLE`** the single source of +truth and remove the `pstyle` body mark. Rationale: the block-map row is the stable +per-paragraph identity handle in the single-root model; a body mark must be +re-maintained across every text splice (extra work for T2/T3) and is `ExpandType::Both` +(expansion hazards). Style changes already flow through the map diff path on remote +(`remote_apply.rs:118` `STYLE → style_changed`). + +**Steps.** Drop `MARK_PARAGRAPH_STYLE` config + writes (`schema.rs:135`, +`set_paragraph_style_utf8`, `document_mark_intervals` index 4); ensure +`input_block_from_container` always reads style from the map; ensure remote style +changes are emitted from the map diff only. Bump `SCHEMA_VERSION` to 3 (breaking +layout change) and update `verify_lineage`. +*Alternative* (if mark-based is preferred for some reason): keep `pstyle`, stop +writing/reading map `STYLE`. Pick one; do not keep both. + +**Acceptance/Tests.** Style set/clear + concurrent style edits converge; no `pstyle` +references remain (chosen option); convergence tests updated. +**Risk.** Behavioral; must update convergence tests and snapshot compatibility note. + +--- + +### T14 — Retire / update stale design docs + +- **Priority:** Low · **Depends:** none · **Owner:** G +- **Files:** `plan.md`, `fix.md`. + +**Problem.** Both describe the abandoned per-paragraph-`LoroText` schema (v1): +`plan.md:184-241` (schema, §5.1/§5.2 "per-paragraph texts… no global offset +arithmetic"), `plan.md:266-293` (`BindingRow{ text }`), `plan.md:331-391` (op algos +creating `"text"`); `fix.md` audits against `plan.md` as "source of truth." + +**Steps.** Either (a) add a prominent "SUPERSEDED — see FIX_LORO_ROOT.md; the +implementation uses a single-root `LoroText`" banner at the top of both, or (b) +rewrite §5/§6 to match single-root. Recommend (a) now, (b) later. + +--- + +### T15 — Reduce self-check timer cost (optional) + +- **Priority:** Low · **Depends:** none · **Owner:** G +- **Files:** `crates/flowstate/src/collab/session_timers.rs:263-308`, + `self_check.rs`. + +**Problem.** `run_self_check` calls full `document_from_loro` + two `projection_hash` +passes every 30s while attached/idle — recurring O(doc). + +**Target.** Cheaper drift detection: maintain a rolling projection hash updated by +the incremental apply paths, or compare Loro version vectors / a cached structural +hash instead of a full reprojection; keep full reprojection only when the cheap +check disagrees. + +**Acceptance/Tests.** No full reprojection on the no-drift path; drift still +detected (inject a divergence test). + +--- + +## 7. Global acceptance criteria + +- `cargo test -p flowstate-collab` and `cargo test -p gpui-flowtext` green + (esp. `tests/convergence.rs`, `tests/remote_apply.rs`, `tests/translation.rs`, + `tests/collab_capture.rs`). +- `cargo clippy` clean for touched crates (per `AGENTS.md`; fix non-false-positive + lints). Avoid `cargo build`/`fmt`/`run` unless testing a binary. +- **Differential safety:** for T2/T3/T4/T11, add tests asserting the incremental + result equals the corresponding full-rewrite/full-rebuild result for identical + edits (the fallback paths stay in the codebase as oracles). +- **Delta-size assertions:** for T2/T4/T5, assert local/remote Loro deltas are + proportional to the edit, not the document (e.g., via `subscribe_local_update` + byte length on a large doc). +- INV-1..4 hold (debug asserts compiled in for tests). + +## 8. Suggested validation harness (add once, reused by all) + +- A "large doc" fixture (e.g. 5k paragraphs) + helper to (a) apply an edit, (b) + measure emitted Loro update bytes and reconcile patch count, (c) assert two peers + converge. Put in `crates/flowstate-collab/tests/perf_smoke.rs` (logic assertions, + not wall-clock). This makes the "fast as possible" goal testable and guards + against regressions. + +## 9. Out of scope / notes + +- **No disk persistence exists** (snapshots are in-memory, peer-to-peer); no schema + migration is required for old files. If T13 bumps `SCHEMA_VERSION`, in-flight v2 + sessions simply won't interop with v3 — acceptable pre-release; note it. +- Keep `replace_document` / `reconcile_body_text` / `rebuild_document_from_*` as + fallbacks; this plan makes them rare, not absent. From 9b9e283e52eeabf633ab163e578c367df0a037c2 Mon Sep 17 00:00:00 2001 From: Adhitya Thirumala Date: Wed, 17 Jun 2026 01:28:00 -0500 Subject: [PATCH 18/45] Make single-root LoroText incremental (FIX_LORO_ROOT core) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eliminates the whole-document rewrite/reprojection on every non-trivial edit while preserving the single-root LoroText design and CRDT convergence. - T1: maintained paragraph->byte Fenwick offset index in DocBinding (new body_index.rs); range helpers no longer stringify the whole body. - T2: incremental ReplaceParagraphSpan — span-scoped minimal body splice, mark carry-over, targeted block-row reconcile; replace_document fallback. - T4: delta-driven remote reconcile — single-paragraph fast path via the Loro TextDelta + slice_delta; falls back to full reprojection on mark changes / newline changes / multi-region edits. - T6-T9: editor keystroke-path incrementalization (skip section rebuild on content edits, in-place block update, O(log n)+shift offset update). - T10: O(1) DocumentIdentityMap::paragraph_index via FxHashMap. - T12: remove dead pre-CRDT op-replay path + postcard wire format. - T15: cheap self-check via cached version-vector/projection hash. - T14: mark plan.md/fix.md superseded by FIX_LORO_ROOT.md. Adds a deterministic convergence regression and tests/perf_smoke.rs proving edit-proportional local deltas and O(1) remote paragraph reconcile. T3/T5 (dead-in-prod/optional) and T11/T13 (breaking/higher-risk) deferred. --- crates/flowstate-collab/src/binding.rs | 41 ++- crates/flowstate-collab/src/body_index.rs | 295 ++++++++++++++++++ crates/flowstate-collab/src/lib.rs | 1 + crates/flowstate-collab/src/local_apply.rs | 266 +++++++++++++--- crates/flowstate-collab/src/patch_apply.rs | 1 + crates/flowstate-collab/src/remote_apply.rs | 179 ++++++++++- crates/flowstate-collab/tests/convergence.rs | 16 + crates/flowstate-collab/tests/perf_smoke.rs | 169 ++++++++++ crates/flowstate/src/collab/session.rs | 3 + crates/flowstate/src/collab/session_timers.rs | 20 +- crates/gpui-flowtext/src/collaboration.rs | 163 +--------- crates/gpui-flowtext/src/document/core.rs | 47 ++- crates/gpui-flowtext/src/edit_ops/offsets.rs | 58 +++- crates/gpui-flowtext/src/edit_ops/text.rs | 3 +- .../src/rich_text/editor/collab_apply.rs | 13 +- .../src/rich_text/editor/edit_pipeline.rs | 1 - .../src/rich_text/editor/lifecycle.rs | 193 +----------- .../gpui-flowtext/src/rich_text/editor/mod.rs | 1 - fix.md | 12 + plan.md | 12 + 20 files changed, 1068 insertions(+), 426 deletions(-) create mode 100644 crates/flowstate-collab/src/body_index.rs create mode 100644 crates/flowstate-collab/tests/perf_smoke.rs diff --git a/crates/flowstate-collab/src/binding.rs b/crates/flowstate-collab/src/binding.rs index 8696e39a..83ac0fee 100644 --- a/crates/flowstate-collab/src/binding.rs +++ b/crates/flowstate-collab/src/binding.rs @@ -3,10 +3,13 @@ use std::collections::HashMap; use anyhow::{Context as _, Result, bail}; -use gpui_flowtext::{Block, BlockId, Document, ParagraphId}; +use gpui_flowtext::{Block, BlockId, Document, ParagraphId, paragraph_text_len}; use loro::{Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, ValueOrContainer}; -use crate::schema::BLOCKS; +use crate::{ + body_index::{BodyParagraphIndex, paragraph_lens_from_body_text}, + schema::{BLOCKS, body_text}, +}; #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub enum BlockKind { @@ -31,6 +34,9 @@ pub struct DocBinding { pub by_paragraph: HashMap, pub by_block: HashMap, pub by_container: HashMap, + /// Paragraph ordinal -> body byte-range index, kept in sync with the body text + /// so paragraph offset lookups never stringify the whole body. + pub body_index: BodyParagraphIndex, } impl DocBinding { @@ -75,10 +81,26 @@ impl DocBinding { version: block_version(block), }); } + binding.refresh_body_index(document); binding.assert_consistent(doc, document); Ok(binding) } + /// Rebuilds the paragraph byte-offset index from the document's paragraphs. + /// Used on build and on the O(n) structural paths (block insert/delete/move) + /// that already rewrite the whole body; the per-keystroke paths update the + /// index incrementally instead. + pub fn refresh_body_index(&mut self, document: &Document) { + self.body_index = BodyParagraphIndex::from_lens(document.paragraphs.iter().map(paragraph_text_len).collect()); + } + + /// Rebuilds the paragraph byte-offset index from the live Loro body text. + /// Used by the remote applier after a full body reprojection so the index + /// keeps matching the freshly imported body (O(body length)). + pub fn refresh_body_index_from_loro(&mut self, doc: &LoroDoc) { + self.body_index = BodyParagraphIndex::from_lens(paragraph_lens_from_body_text(&body_text(doc))); + } + pub fn assert_consistent(&self, doc: &LoroDoc, document: &Document) { let blocks = doc.get_movable_list(BLOCKS); debug_assert_eq!(self.rows.len(), blocks.len()); @@ -100,6 +122,21 @@ impl DocBinding { debug_assert!(row.paragraph_id.is_none()); } } + + debug_assert_eq!(self.body_index.len(), paragraph_ix, "body index paragraph count does not match paragraph rows"); + #[cfg(debug_assertions)] + { + let body_lens = paragraph_lens_from_body_text(&body_text(doc)); + debug_assert_eq!(self.body_index.len(), body_lens.len(), "body index paragraph count does not match the body text"); + for (ordinal, expected) in body_lens.iter().enumerate() { + debug_assert_eq!(self.body_index.paragraph_len(ordinal), *expected, "body index byte length mismatch vs body at paragraph {ordinal}"); + debug_assert_eq!( + self.body_index.paragraph_len(ordinal), + document.paragraphs.get(ordinal).map_or(0, paragraph_text_len), + "body index byte length mismatch vs document at paragraph {ordinal}" + ); + } + } } pub fn push_row(&mut self, row: BindingRow) { diff --git a/crates/flowstate-collab/src/body_index.rs b/crates/flowstate-collab/src/body_index.rs new file mode 100644 index 00000000..b6d522b0 --- /dev/null +++ b/crates/flowstate-collab/src/body_index.rs @@ -0,0 +1,295 @@ +//! Maintained paragraph -> body byte-offset index for the single-root body text. +//! +//! In the single-root model the whole document lives in one `LoroText` ("body") +//! whose paragraphs are separated by `'\n'`. Mapping a paragraph ordinal to its +//! byte range in the body used to require stringifying the entire body and +//! linear-scanning for the Nth newline on every edit. [`BodyParagraphIndex`] +//! replaces that with a Fenwick (binary indexed) tree over per-paragraph body +//! widths, mirroring `gpui_flowtext::ParagraphOffsetIndex`, so paragraph -> byte +//! lookups are O(log n) and content edits update in O(log n). + +use std::ops::Range; + +use loro::LoroText; + +use crate::schema::text_content; + +/// Fenwick tree over per-paragraph body byte widths plus the raw paragraph byte +/// lengths. +/// +/// `width(k) = len(k) + (1 if k is not the last paragraph else 0)` — the trailing +/// `'\n'` separator is charged to every paragraph except the last, so the prefix +/// sum of widths over the paragraphs before `k` is exactly `k`'s start byte in the +/// body. +#[derive(Clone, Debug, Default)] +pub struct BodyParagraphIndex { + /// UTF-8 byte length of each paragraph's text (no separator newline). + lens: Vec, + /// 1-indexed Fenwick tree over paragraph widths. + tree: Vec, +} + +impl BodyParagraphIndex { + /// Builds an index from per-paragraph UTF-8 byte lengths (in body order). + #[must_use] + pub fn from_lens(lens: Vec) -> Self { + let mut index = Self { + tree: vec![0; lens.len() + 1], + lens, + }; + for ordinal in 0..index.lens.len() { + let width = index.width(ordinal); + index.add(ordinal, width as isize); + } + index + } + + /// Number of paragraphs tracked by the index. + #[must_use] + pub fn len(&self) -> usize { + self.lens.len() + } + + #[must_use] + pub fn is_empty(&self) -> bool { + self.lens.is_empty() + } + + /// Byte offset where paragraph `ordinal` starts in the body text. + #[must_use] + pub fn paragraph_start(&self, ordinal: usize) -> usize { + self.prefix_sum(ordinal) + } + + /// UTF-8 byte length of paragraph `ordinal` (excluding the separator newline). + #[must_use] + pub fn paragraph_len(&self, ordinal: usize) -> usize { + self.lens.get(ordinal).copied().unwrap_or(0) + } + + /// Byte range `[start, start + len)` of paragraph `ordinal` in the body text. + #[must_use] + pub fn paragraph_range(&self, ordinal: usize) -> Range { + let start = self.paragraph_start(ordinal); + start..start + self.paragraph_len(ordinal) + } + + /// Total body length implied by the index (sum of widths). + #[must_use] + pub fn total_len(&self) -> usize { + self.prefix_sum(self.lens.len()) + } + + /// Returns the ordinal of the paragraph that contains `byte` — i.e. the largest + /// ordinal whose start is `<= byte`. A byte sitting on a separator newline maps + /// to the paragraph it terminates. + #[must_use] + pub fn paragraph_ordinal_for_body_byte(&self, byte: usize) -> usize { + if self.lens.is_empty() { + return 0; + } + let mut low = 0usize; + let mut high = self.lens.len() - 1; + while low < high { + let mid = low + (high - low).div_ceil(2); + if self.paragraph_start(mid) <= byte { + low = mid; + } else { + high = mid - 1; + } + } + low + } + + /// Updates one paragraph's byte length in O(log n). The paragraph count must be + /// unchanged (use [`Self::insert_paragraph`] / [`Self::remove_paragraph`] for + /// structural changes). + pub fn set_paragraph_len(&mut self, ordinal: usize, new_len: usize) { + let Some(old) = self.lens.get(ordinal).copied() else { + return; + }; + if old == new_len { + return; + } + self.lens[ordinal] = new_len; + self.add(ordinal, new_len as isize - old as isize); + } + + /// Inserts a new paragraph of length `len` at `ordinal`, shifting later + /// paragraphs right. O(n) (rebuilds the tree, matching the editor's structural + /// path which is already O(n)). + pub fn insert_paragraph(&mut self, ordinal: usize, len: usize) { + let ordinal = ordinal.min(self.lens.len()); + self.lens.insert(ordinal, len); + self.rebuild(); + } + + /// Removes the paragraph at `ordinal`. O(n). + pub fn remove_paragraph(&mut self, ordinal: usize) { + if ordinal >= self.lens.len() { + return; + } + self.lens.remove(ordinal); + self.rebuild(); + } + + /// Removes `count` paragraphs starting at `ordinal` in one rebuild. O(n). + pub fn remove_paragraph_range(&mut self, ordinal: usize, count: usize) { + let start = ordinal.min(self.lens.len()); + let end = start.saturating_add(count).min(self.lens.len()); + if start >= end { + return; + } + self.lens.drain(start..end); + self.rebuild(); + } + + fn rebuild(&mut self) { + let lens = std::mem::take(&mut self.lens); + *self = Self::from_lens(lens); + } + + fn width(&self, ordinal: usize) -> usize { + self.lens[ordinal] + usize::from(ordinal + 1 < self.lens.len()) + } + + fn add(&mut self, ordinal: usize, delta: isize) { + if delta == 0 { + return; + } + let mut ix = ordinal + 1; + while ix < self.tree.len() { + self.tree[ix] = self.tree[ix].saturating_add_signed(delta); + ix += ix & (!ix + 1); + } + } + + fn prefix_sum(&self, paragraph_count: usize) -> usize { + let mut ix = paragraph_count.min(self.lens.len()); + let mut sum = 0; + while ix > 0 { + sum += self.tree[ix]; + ix &= ix - 1; + } + sum + } +} + +/// Scans the body text once and returns each paragraph's UTF-8 byte length (body +/// order). Used to (re)build [`BodyParagraphIndex`] and to debug-verify it. +/// +/// This is O(body length); it is only meant for build / structural / debug paths, +/// never the per-keystroke hot path. +#[must_use] +pub fn paragraph_lens_from_body_text(text: &LoroText) -> Vec { + text_content(text).split('\n').map(str::len).collect() +} + +/// Length of the shared UTF-8 prefix of `left` and `right`, on a char boundary. +#[must_use] +pub fn common_prefix_bytes(left: &str, right: &str) -> usize { + let mut prefix = 0usize; + for (left_ch, right_ch) in left.chars().zip(right.chars()) { + if left_ch != right_ch { + break; + } + prefix += left_ch.len_utf8(); + } + prefix +} + +/// Length of the shared UTF-8 suffix of `left` and `right`, on a char boundary. +#[must_use] +pub fn common_suffix_bytes(left: &str, right: &str) -> usize { + let mut suffix = 0usize; + for (left_ch, right_ch) in left.chars().rev().zip(right.chars().rev()) { + if left_ch != right_ch { + break; + } + let len = left_ch.len_utf8(); + if suffix + len > left.len() || suffix + len > right.len() { + break; + } + suffix += len; + } + suffix +} + +/// Minimal byte splice turning `old` into `new`: the shared prefix length, the +/// length of the differing middle of `old` to delete, and the differing middle of +/// `new` to insert. Returns byte offsets relative to the start of `old`/`new`. +#[must_use] +pub fn minimal_utf8_splice(old: &str, new: &str) -> (usize, usize, Range) { + let prefix = common_prefix_bytes(old, new); + let suffix = common_suffix_bytes(&old[prefix..], &new[prefix..]); + let old_middle = old.len().saturating_sub(prefix + suffix); + let new_middle = prefix..new.len().saturating_sub(suffix); + (prefix, old_middle, new_middle) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn fresh_start(lens: &[usize], ordinal: usize) -> usize { + lens + .iter() + .take(ordinal) + .map(|len| len + 1) + .sum() + } + + #[test] + fn starts_and_ranges_match_a_fresh_scan() { + let lens = vec![3, 0, 5, 2, 7]; + let index = BodyParagraphIndex::from_lens(lens.clone()); + for ordinal in 0..lens.len() { + assert_eq!(index.paragraph_start(ordinal), fresh_start(&lens, ordinal)); + assert_eq!(index.paragraph_range(ordinal), fresh_start(&lens, ordinal)..fresh_start(&lens, ordinal) + lens[ordinal]); + } + // Total length has no trailing newline after the last paragraph. + let total: usize = lens.iter().sum::() + lens.len() - 1; + assert_eq!(index.total_len(), total); + } + + #[test] + fn random_edits_keep_starts_consistent() { + let mut lens = vec![4, 4, 4, 4]; + let mut index = BodyParagraphIndex::from_lens(lens.clone()); + + index.set_paragraph_len(1, 10); + lens[1] = 10; + index.insert_paragraph(2, 3); + lens.insert(2, 3); + index.remove_paragraph(0); + lens.remove(0); + index.set_paragraph_len(index.len() - 1, 0); + *lens.last_mut().unwrap() = 0; + + for ordinal in 0..lens.len() { + assert_eq!(index.paragraph_start(ordinal), fresh_start(&lens, ordinal), "start mismatch at {ordinal}"); + assert_eq!(index.paragraph_len(ordinal), lens[ordinal]); + } + } + + #[test] + fn ordinal_lookup_finds_containing_paragraph() { + let lens = vec![2, 3, 4]; + let index = BodyParagraphIndex::from_lens(lens); + // body: "aa\nbbb\ncccc" -> starts 0, 3, 7 + assert_eq!(index.paragraph_ordinal_for_body_byte(0), 0); + assert_eq!(index.paragraph_ordinal_for_body_byte(2), 0); // newline after p0 + assert_eq!(index.paragraph_ordinal_for_body_byte(3), 1); + assert_eq!(index.paragraph_ordinal_for_body_byte(6), 1); // newline after p1 + assert_eq!(index.paragraph_ordinal_for_body_byte(7), 2); + assert_eq!(index.paragraph_ordinal_for_body_byte(100), 2); + } + + #[test] + fn minimal_splice_picks_changed_middle() { + let (prefix, old_middle, new_middle) = minimal_utf8_splice("abcXYZdef", "abcQdef"); + assert_eq!(prefix, "abc".len()); + assert_eq!(old_middle, "XYZ".len()); + assert_eq!(new_middle, "abc".len().."abcQ".len()); + } +} diff --git a/crates/flowstate-collab/src/lib.rs b/crates/flowstate-collab/src/lib.rs index b6bae492..910c48c7 100644 --- a/crates/flowstate-collab/src/lib.rs +++ b/crates/flowstate-collab/src/lib.rs @@ -5,6 +5,7 @@ //! lives in `crates/flowstate/src/collab`. pub mod binding; +pub mod body_index; pub mod ids; pub mod local_apply; pub mod net; diff --git a/crates/flowstate-collab/src/local_apply.rs b/crates/flowstate-collab/src/local_apply.rs index a1b64dc5..19b122fc 100644 --- a/crates/flowstate-collab/src/local_apply.rs +++ b/crates/flowstate-collab/src/local_apply.rs @@ -3,11 +3,12 @@ use anyhow::{Context as _, Result, bail}; use std::ops::Range; -use gpui_flowtext::{Block, BlockId, CanonicalOperation, Document, DocumentSpan, ParagraphId, RunStyles, full_document_text}; +use gpui_flowtext::{Block, BlockId, CanonicalOperation, Document, DocumentSpan, Paragraph, ParagraphId, RunStyles, full_document_text, paragraph_text_len}; use loro::{CommitOptions, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer}; use crate::{ binding::{BindingRow, BlockKind, DocBinding, block_version}, + body_index::minimal_utf8_splice, projection::{insert_object_container, insert_paragraph_container, replace_blocks_from_document, replace_body_from_document}, schema::{ BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_TABLE, REV, STYLE, body_text, decode_paragraph_style, @@ -82,10 +83,16 @@ impl LocalApplier<'_> { } fn insert_text(&mut self, document: &Document, paragraph: ParagraphId, byte: usize, value: &str, styles: RunStyles) -> Result<()> { - let byte = self.body_byte_for_paragraph_byte(paragraph, byte)?; + let ordinal = self.paragraph_ordinal(paragraph)?; + let range = self.body_paragraph_range(ordinal)?; + let body_byte = range.start + byte.min(range.len()); let body = body_text(self.doc); - body.insert_utf8(byte, value)?; - set_run_styles_utf8(&body, byte..byte + value.len(), styles)?; + body.insert_utf8(body_byte, value)?; + set_run_styles_utf8(&body, body_byte..body_byte + value.len(), styles)?; + self + .binding + .body_index + .set_paragraph_len(ordinal, range.len() + value.len()); self.debug_assert_body_matches_document(document); Ok(()) } @@ -102,10 +109,15 @@ impl LocalApplier<'_> { let end_ix = self.row_ix_for_paragraph(end_paragraph)?; let start_byte = self.body_byte_for_paragraph_byte(start_paragraph, start)?; let end_byte = self.body_byte_for_paragraph_byte(end_paragraph, end)?; - if end_byte > start_byte { - body_text(self.doc).delete_utf8(start_byte, end_byte - start_byte)?; + let deleted = end_byte.saturating_sub(start_byte); + if deleted > 0 { + body_text(self.doc).delete_utf8(start_byte, deleted)?; } if start_ix == end_ix { + if let Some(ordinal) = self.paragraph_ordinal_for_row(start_ix) { + let len = self.binding.body_index.paragraph_len(ordinal); + self.binding.body_index.set_paragraph_len(ordinal, len.saturating_sub(deleted)); + } self.debug_assert_body_matches_document(document); return Ok(()); } @@ -120,6 +132,10 @@ impl LocalApplier<'_> { .paragraph_ordinal_for_row(end_ix) .context("delete end row is not a paragraph")?; if end_paragraph_ix > start_paragraph_ix { + // The start paragraph absorbs the end paragraph's surviving tail; the + // paragraphs in between (and the end paragraph) collapse away. + let end_len = self.binding.body_index.paragraph_len(end_paragraph_ix); + let merged_len = start + end_len.saturating_sub(end); let deleted_rows = self.paragraph_row_indices(start_paragraph_ix + 1, end_paragraph_ix - start_paragraph_ix)?; let blocks = self.blocks(); for row_ix in deleted_rows.into_iter().rev() { @@ -129,6 +145,11 @@ impl LocalApplier<'_> { .remove_row(row_ix) .context("DocBinding paragraph row disappeared during cross-paragraph delete")?; } + self + .binding + .body_index + .remove_paragraph_range(start_paragraph_ix + 1, end_paragraph_ix - start_paragraph_ix); + self.binding.body_index.set_paragraph_len(start_paragraph_ix, merged_len); } self.debug_assert_body_matches_document(document); Ok(()) @@ -148,14 +169,16 @@ impl LocalApplier<'_> { .get(insert_ix) .map(block_version) .context("post-split document is missing the new block")?; - let body_byte = self.body_byte_for_paragraph_byte(paragraph, byte)?; + let orig_ordinal = self + .paragraph_ordinal_for_row(row_ix) + .context("split row is not a paragraph")?; + let old_len = self.binding.body_index.paragraph_len(orig_ordinal); + let split = byte.min(old_len); + let body_byte = self.binding.body_index.paragraph_start(orig_ordinal) + split; body_text(self.doc).insert_utf8(body_byte, "\n")?; let blocks = self.blocks(); - let paragraph_ix = self - .paragraph_ordinal_for_row(row_ix) - .context("split row is not a paragraph")? - .saturating_add(1); + let paragraph_ix = orig_ordinal.saturating_add(1); let style = document .paragraphs .get(paragraph_ix) @@ -182,6 +205,11 @@ impl LocalApplier<'_> { version, }, ); + self.binding.body_index.set_paragraph_len(orig_ordinal, split); + self + .binding + .body_index + .insert_paragraph(orig_ordinal + 1, old_len - split); self.debug_assert_body_matches_document(document); Ok(()) } @@ -205,6 +233,7 @@ impl LocalApplier<'_> { match block { Block::Paragraph(paragraph) => { replace_body_from_document(self.doc, document)?; + self.binding.refresh_body_index(document); let paragraph_ix = document_paragraph_ix_for_block(document, block_ix)?; let paragraph_id = document .ids @@ -253,6 +282,7 @@ impl LocalApplier<'_> { .context("DocBinding block row disappeared during block delete")?; if was_paragraph { replace_body_from_document(self.doc, document)?; + self.binding.refresh_body_index(document); self.debug_assert_body_matches_document(document); } Ok(()) @@ -268,6 +298,7 @@ impl LocalApplier<'_> { self.binding.move_row(old_ix, new_block_ix); if was_paragraph { replace_body_from_document(self.doc, document)?; + self.binding.refresh_body_index(document); self.debug_assert_body_matches_document(document); } Ok(()) @@ -280,8 +311,178 @@ impl LocalApplier<'_> { before: &DocumentSpan, after: &DocumentSpan, ) -> Result<()> { - let _ = (start_paragraph, before, after); - self.replace_document(document) + // Try a span-scoped splice; fall back to a full rewrite if the span can't be + // resolved/reconciled or any step fails (the fallback fully resets the Loro + // doc to `document`, so a partially-applied attempt is safe). + match self.try_replace_paragraph_span(document, start_paragraph, before, after) { + Ok(true) => Ok(()), + Ok(false) | Err(_) => self.replace_document(document), + } + } + + fn try_replace_paragraph_span( + &mut self, + document: &Document, + start_paragraph: Option, + before: &DocumentSpan, + after: &DocumentSpan, + ) -> Result { + let before_n = before.paragraphs.len(); + let after_n = after.paragraphs.len(); + if before_n == 0 || after_n == 0 { + return Ok(false); + } + + // Resolve the span's start paragraph ordinal. + let start_ord = match start_paragraph { + Some(id) => { + let Some(&row_ix) = self.binding.by_paragraph.get(&id) else { + return Ok(false); + }; + let Some(ord) = self.paragraph_ordinal_for_row(row_ix) else { + return Ok(false); + }; + ord + }, + None => before.start_paragraph, + }; + if start_ord + before_n > self.binding.body_index.len() { + return Ok(false); + } + + // The span's paragraph rows must be contiguous paragraph blocks (no object + // blocks interleaved) for the in-place row reconcile below to be valid. + let Ok(before_rows) = self.paragraph_row_indices(start_ord, before_n) else { + return Ok(false); + }; + let first_row = before_rows[0]; + if before_rows.iter().enumerate().any(|(offset, row)| *row != first_row + offset) { + return Ok(false); + } + + // The body slice for the span must currently equal `before.text`; if the + // lengths disagree the editor and CRDT have diverged — bail to the rewrite. + let start_byte = self.binding.body_index.paragraph_start(start_ord); + let end_byte = self.binding.body_index.paragraph_range(start_ord + before_n - 1).end; + if end_byte - start_byte != before.text.len() { + return Ok(false); + } + + // Minimal text splice: only rewrite the differing middle of the span. + let (prefix, old_middle, new_middle) = minimal_utf8_splice(&before.text, &after.text); + let body = body_text(self.doc); + let splice_at = start_byte + prefix; + if old_middle > 0 { + body.delete_utf8(splice_at, old_middle)?; + } + let inserted = &after.text[new_middle]; + if !inserted.is_empty() { + body.insert_utf8(splice_at, inserted)?; + } + + // Re-apply marks across the whole span so inline run styles and paragraph + // styles match `after` (INV-2), including over surviving prefix/suffix bytes. + self.apply_span_marks(start_byte, &after.paragraphs)?; + + // Reconcile the block rows + paragraph offset index for the span. + let overlap = before_n.min(after_n); + let blocks = self.blocks(); + for offset in 0..overlap { + let row_ix = first_row + offset; + let ordinal = start_ord + offset; + let style = after.paragraphs[offset].style; + let block_id = document + .ids + .block_ids + .get(row_ix) + .copied() + .context("post-edit document is missing a block id for a spliced paragraph")?; + let paragraph_id = document + .ids + .paragraph_ids + .get(ordinal) + .copied() + .context("post-edit document is missing a paragraph id for a spliced paragraph")?; + let row = self + .binding + .rows + .get_mut(row_ix) + .context("spliced paragraph row is out of bounds")?; + row.map.insert(STYLE, encode_paragraph_style(style))?; + row.block_id = block_id; + row.paragraph_id = Some(paragraph_id); + self + .binding + .body_index + .set_paragraph_len(ordinal, paragraph_text_len(&after.paragraphs[offset])); + } + + if after_n > before_n { + for offset in before_n..after_n { + let row_ix = first_row + offset; + let ordinal = start_ord + offset; + let style = after.paragraphs[offset].style; + let block_id = document + .ids + .block_ids + .get(row_ix) + .copied() + .context("post-edit document is missing a block id for an inserted paragraph")?; + let paragraph_id = document + .ids + .paragraph_ids + .get(ordinal) + .copied() + .context("post-edit document is missing a paragraph id for an inserted paragraph")?; + let version = document.blocks.get(row_ix).map_or(0, block_version); + let map = insert_paragraph_container(&blocks, row_ix, style)?; + self.binding.insert_row( + row_ix, + BindingRow { + map, + kind: BlockKind::Paragraph, + block_id, + paragraph_id: Some(paragraph_id), + version, + }, + ); + self + .binding + .body_index + .insert_paragraph(ordinal, paragraph_text_len(&after.paragraphs[offset])); + } + } else if before_n > after_n { + let removed = before_n - after_n; + for _ in 0..removed { + let row_ix = first_row + after_n; + blocks.delete(row_ix, 1)?; + self + .binding + .remove_row(row_ix) + .context("spliced paragraph row disappeared during span shrink")?; + } + self.binding.body_index.remove_paragraph_range(start_ord + after_n, removed); + } + + self.binding.rebuild_indexes(); + self.debug_assert_body_matches_document(document); + Ok(true) + } + + fn apply_span_marks(&self, start: usize, paragraphs: &[Paragraph]) -> Result<()> { + let body = body_text(self.doc); + let mut paragraph_start = start; + for paragraph in paragraphs { + let mut run_byte = paragraph_start; + for run in ¶graph.runs { + set_run_styles_utf8(&body, run_byte..run_byte + run.len, run.styles)?; + run_byte += run.len; + } + let paragraph_len = run_byte - paragraph_start; + set_paragraph_style_utf8(&body, paragraph_start..paragraph_start + paragraph_len, paragraph.style)?; + paragraph_start += paragraph_len + 1; + } + Ok(()) } fn replace_block(&mut self, document: &Document, block: Option) -> Result<()> { @@ -432,26 +633,27 @@ impl LocalApplier<'_> { self.doc.get_movable_list(BLOCKS) } - fn body_byte_for_paragraph_byte(&self, paragraph: ParagraphId, byte: usize) -> Result { + fn paragraph_ordinal(&self, paragraph: ParagraphId) -> Result { let row_ix = self.row_ix_for_paragraph(paragraph)?; - let paragraph_ix = self + self .paragraph_ordinal_for_row(row_ix) - .context("paragraph row has no paragraph ordinal")?; - let range = self.body_paragraph_range(paragraph_ix)?; + .context("paragraph row has no paragraph ordinal") + } + + fn body_byte_for_paragraph_byte(&self, paragraph: ParagraphId, byte: usize) -> Result { + let range = self.body_paragraph_range(self.paragraph_ordinal(paragraph)?)?; Ok(range.start + byte.min(range.len())) } fn body_range_for_paragraph(&self, paragraph: ParagraphId) -> Result> { - let row_ix = self.row_ix_for_paragraph(paragraph)?; - let paragraph_ix = self - .paragraph_ordinal_for_row(row_ix) - .context("paragraph row has no paragraph ordinal")?; - self.body_paragraph_range(paragraph_ix) + self.body_paragraph_range(self.paragraph_ordinal(paragraph)?) } fn body_paragraph_range(&self, target_paragraph_ix: usize) -> Result> { - paragraph_range_in_body_text(&text_content(&body_text(self.doc)), target_paragraph_ix) - .context("paragraph ordinal is outside the body text") + if target_paragraph_ix >= self.binding.body_index.len() { + bail!("paragraph ordinal {target_paragraph_ix} is outside the body index"); + } + Ok(self.binding.body_index.paragraph_range(target_paragraph_ix)) } } @@ -478,22 +680,6 @@ fn kind_for_payload(payload: &BlockPayload) -> &'static str { } } -fn paragraph_range_in_body_text(text: &str, target_paragraph_ix: usize) -> Option> { - let mut paragraph_ix = 0usize; - let mut start = 0usize; - for (byte, ch) in text.char_indices() { - if ch != '\n' { - continue; - } - if paragraph_ix == target_paragraph_ix { - return Some(start..byte); - } - paragraph_ix += 1; - start = byte + ch.len_utf8(); - } - (paragraph_ix == target_paragraph_ix).then_some(start..text.len()) -} - fn map_i64(map: &LoroMap, key: &str) -> Result { match map.get(key) { Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), diff --git a/crates/flowstate-collab/src/patch_apply.rs b/crates/flowstate-collab/src/patch_apply.rs index bf77c1cc..f737e444 100644 --- a/crates/flowstate-collab/src/patch_apply.rs +++ b/crates/flowstate-collab/src/patch_apply.rs @@ -14,6 +14,7 @@ pub fn apply_patches(document: &mut Document, binding: &mut DocBinding, doc: &Lo apply_one(document, patch)?; } binding.refresh_versions(document); + binding.refresh_body_index(document); binding.assert_consistent(doc, document); Ok(()) } diff --git a/crates/flowstate-collab/src/remote_apply.rs b/crates/flowstate-collab/src/remote_apply.rs index 5cf333b8..eaa2cb78 100644 --- a/crates/flowstate-collab/src/remote_apply.rs +++ b/crates/flowstate-collab/src/remote_apply.rs @@ -2,18 +2,19 @@ use anyhow::{Context as _, Result, bail}; use gpui_flowtext::{ - BlockId, CollabPatch, CollabStructuralBlock, CollabTextDelta, Document, InputBlock, InputParagraph, InputRun, ParagraphId, new_block_id, - new_paragraph_id, paragraph_text, TextRun, + BlockId, CollabPatch, CollabStructuralBlock, CollabTextDelta, Document, InputBlock, InputParagraph, InputRun, ParagraphId, ParagraphStyle, + new_block_id, new_paragraph_id, paragraph_text, TextRun, }; use loro::{ - Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer, + Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroValue, TextDelta, ValueOrContainer, + cursor::PosType, event::{Diff, DiffEvent, ListDiffItem}, }; use crate::{ binding::{BindingRow, BlockKind, DocBinding}, projection::{input_block_from_container, input_blocks_from_loro}, - schema::{BLOCKS, DATA, REV, STYLE, body_text, decode_paragraph_style}, + schema::{BLOCKS, DATA, REV, STYLE, body_text, decode_paragraph_style, input_runs_from_delta, paragraph_style_from_attrs, utf8_byte}, }; pub struct RemoteApplier<'s> { @@ -29,24 +30,162 @@ impl RemoteApplier<'_> { let pre_event_binding = self.binding.clone(); let body_id = body_text(self.doc).id(); - let mut body_changed = false; + let mut body_delta: Option> = None; + let mut multiple_body_deltas = false; let inserted_containers = inserted_container_ids(self.doc, event)?; let mut patches = Vec::new(); for diff in &event.events { match &diff.diff { - Diff::Text(_) if *diff.target == body_id => body_changed = true, + Diff::Text(delta) if *diff.target == body_id => { + if body_delta.replace(delta.clone()).is_some() { + multiple_body_deltas = true; + } + }, Diff::Text(_) => {}, Diff::Map(delta) => self.apply_map_diff(diff.target, delta.updated.keys().map(|key| key.as_ref()), &inserted_containers, &mut patches)?, Diff::List(delta) => self.apply_list_diff(diff.target, delta, &mut patches)?, Diff::Tree(_) | Diff::Counter(_) | Diff::Unknown => {}, } } - if body_changed { - self.reconcile_body_text(document, &pre_event_binding, &mut patches)?; + if let Some(delta) = body_delta { + // Fast path: a pure intra-paragraph text edit — no structural/style patches + // this event, a single contiguous change, and no newline change — reconciles + // only the affected paragraph. Anything else falls back to the full + // reprojection (which is always correct). + let handled = !multiple_body_deltas + && patches.is_empty() + && self.try_reconcile_body_delta(document, &delta, &mut patches)?; + if !handled { + self.reconcile_body_text(document, &pre_event_binding, &mut patches)?; + self.binding.refresh_body_index_from_loro(self.doc); + } } Ok(patches) } + /// Attempts to reconcile a body text change by inspecting the Loro `TextDelta` + /// and re-projecting only the single affected paragraph. Returns `Ok(false)` + /// (without mutating) when the change is not a clean single-paragraph, + /// no-newline edit, so the caller can fall back to the full reprojection. + fn try_reconcile_body_delta(&mut self, document: &Document, delta: &[TextDelta], patches: &mut Vec) -> Result { + // Only safe while the index mirrors the editor document's paragraph count. + if self.binding.body_index.len() != document.paragraphs.len() { + return Ok(false); + } + + // The delta must be a single contiguous change with no inserted newline. + let mut lead = 0usize; // unicode codepoints retained before the change + let mut inserted_bytes = 0usize; + let mut deleted_unicode = 0usize; + let mut in_change = false; + let mut after_change = false; + for item in delta { + match item { + TextDelta::Retain { retain, attributes } => { + // A retain carrying attributes is a mark change over existing text + // (e.g. a remote run/paragraph style edit). That changes a paragraph's + // runs without changing its text, which this text-splice fast path does + // not model — defer to the full reprojection. + if attributes.as_ref().is_some_and(|attributes| !attributes.is_empty()) { + return Ok(false); + } + if in_change { + after_change = true; + in_change = false; + } else if !after_change { + lead += retain; + } + }, + TextDelta::Insert { insert, .. } => { + if after_change || insert.contains('\n') { + return Ok(false); + } + in_change = true; + inserted_bytes += insert.len(); + }, + TextDelta::Delete { delete } => { + if after_change { + return Ok(false); + } + in_change = true; + deleted_unicode += delete; + }, + } + } + if !in_change && !after_change { + return Ok(false); + } + + let new_body = body_text(self.doc); + let change_byte = utf8_byte(&new_body, lead); + let ordinal = self.binding.body_index.paragraph_ordinal_for_body_byte(change_byte); + let start_byte = self.binding.body_index.paragraph_start(ordinal); + let old_len = self.binding.body_index.paragraph_len(ordinal); + if change_byte < start_byte || change_byte > start_byte + old_len { + return Ok(false); + } + + // Require the index and the editor document to agree on this paragraph's + // pre-edit length so all byte arithmetic below is consistent. + let old_text = paragraph_text(document, ordinal); + if old_text.len() != old_len { + return Ok(false); + } + let local_offset = change_byte - start_byte; + let mut deleted_bytes = 0usize; + let mut tail = old_text[local_offset..].chars(); + for _ in 0..deleted_unicode { + // Running out of paragraph text means the deletion crossed a newline. + let Some(ch) = tail.next() else { + return Ok(false); + }; + deleted_bytes += ch.len_utf8(); + } + let new_len = old_len + inserted_bytes - deleted_bytes; + + let Some(row_ix) = row_for_paragraph_ordinal(self.binding, ordinal) else { + return Ok(false); + }; + let slice = new_body.slice_delta(start_byte, start_byte + new_len, PosType::Bytes)?; + let new_paragraph = InputParagraph { + style: paragraph_style_from_slice(&slice), + runs: input_runs_from_delta(&slice), + }; + let Some(old_paragraph) = input_paragraph_from_document(document, ordinal) else { + return Ok(false); + }; + + // Keep the index aligned with the freshly imported body for later events. + self.binding.body_index.set_paragraph_len(ordinal, new_len); + + if input_paragraphs_equal(&old_paragraph, &new_paragraph) { + return Ok(true); + } + let old_paragraph_text = input_paragraph_text(&old_paragraph); + let new_paragraph_text = input_paragraph_text(&new_paragraph); + if old_paragraph_text == new_paragraph_text { + if old_paragraph.style != new_paragraph.style { + patches.push(CollabPatch::ParagraphStyle { + row: row_ix, + style: new_paragraph.style, + }); + } + if !input_runs_equal(&old_paragraph.runs, &new_paragraph.runs) { + patches.push(CollabPatch::ParagraphRuns { + row: row_ix, + runs: text_runs_from_input(&new_paragraph), + }); + } + } else { + patches.push(CollabPatch::ParagraphText { + row: row_ix, + new: new_paragraph.clone(), + delta_utf8: text_delta_for_replacement(&old_paragraph_text, &new_paragraph_text), + }); + } + Ok(true) + } + fn reconcile_body_text(&self, document: &Document, pre_event_binding: &DocBinding, patches: &mut Vec) -> Result<()> { let projected_blocks = input_blocks_from_loro(self.doc)?; for (row_ix, row) in self.binding.rows.iter().enumerate() { @@ -490,6 +629,30 @@ fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option< None } +fn row_for_paragraph_ordinal(binding: &DocBinding, target_ordinal: usize) -> Option { + let mut paragraph_ix = 0usize; + for (row_ix, row) in binding.rows.iter().enumerate() { + if row.paragraph_id.is_some() { + if paragraph_ix == target_ordinal { + return Some(row_ix); + } + paragraph_ix += 1; + } + } + None +} + +fn paragraph_style_from_slice(delta: &[TextDelta]) -> ParagraphStyle { + for item in delta { + if let TextDelta::Insert { attributes, .. } = item + && let Some(style) = paragraph_style_from_attrs(attributes.as_ref()) + { + return style; + } + } + ParagraphStyle::Normal +} + fn map_i64(map: &LoroMap, key: &str) -> Result { match map.get(key) { Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs index 0d4ffdac..63a3d6ab 100644 --- a/crates/flowstate-collab/tests/convergence.rs +++ b/crates/flowstate-collab/tests/convergence.rs @@ -35,6 +35,22 @@ mod tests { } } + #[test] + fn reordered_run_style_after_inserts_converges() { + let ops = [ + FuzzOp::Style { peer: 0, paragraph: 0, start: 0, len: 0, style: 0 }, + FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, + FuzzOp::Style { peer: 0, paragraph: 0, start: 0, len: 0, style: 0 }, + FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, + FuzzOp::DeleteText { peer: 0, paragraph: 61, start: 31, len: 29 }, + FuzzOp::DeleteText { peer: 138, paragraph: 161, start: 6, len: 0 }, + FuzzOp::InsertText { peer: 0, paragraph: 125, byte: 20, text: 0, style: 0 }, + FuzzOp::Style { peer: 2, paragraph: 12, start: 0, len: 0, style: 1 }, + ]; + + run_program(2, 4754897484207400829, &ops).expect("reordered run style after inserts should converge"); + } + #[test] fn style_then_join_converges_when_updates_reorder() { let ops = [ diff --git a/crates/flowstate-collab/tests/perf_smoke.rs b/crates/flowstate-collab/tests/perf_smoke.rs new file mode 100644 index 00000000..fb053f13 --- /dev/null +++ b/crates/flowstate-collab/tests/perf_smoke.rs @@ -0,0 +1,169 @@ +//! Logic-level performance guards: a single-character edit on a large document +//! must produce a Loro update and a remote reconcile proportional to the edit, +//! not to the whole document. These assert structure/size, never wall-clock, so +//! they stay deterministic in CI and guard against regressing the incremental +//! local apply (T2) and delta-driven remote reconcile (T4). + +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + + use flowstate_collab::{ + SessionId, + binding::DocBinding, + local_apply::LocalApplier, + patch_apply::apply_patches, + projection, schema, + }; + use gpui_flowtext::{ + CanonicalOperation, CollabPatch, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyles, capture_document_span, + document_from_input_blocks, insert_text_at, paragraph_text, plain, + }; + use loro::{ExportMode, LoroDoc, event::Subscriber}; + + const PARAGRAPHS: usize = 400; + + #[test] + fn single_char_local_edit_emits_a_delta_proportional_to_the_edit() { + let session = SessionId::from_bytes([42; 32]); + let document = large_document(); + let loro = schema::new_configured_doc(); + projection::populate_from_document(&loro, session, "perf-smoke", &document).expect("populate"); + let mut binding = DocBinding::build(&loro, &document).expect("binding"); + + let updates = Arc::new(Mutex::new(Vec::new())); + let captured = updates.clone(); + let _subscription = loro.subscribe_local_update(Box::new(move |bytes| { + captured.lock().expect("update lock").push(bytes.to_vec()); + true + })); + + // Edit a paragraph in the middle of the document with a single keystroke, + // emitted exactly as the editor would: a span replacement of that paragraph. + let target = PARAGRAPHS / 2; + let mut edited = document.clone(); + let before = capture_document_span(&edited, target..target + 1); + insert_text_at(&mut edited, target, 0, "X", RunStyles::default()); + let after = capture_document_span(&edited, target..target + 1); + let op = CanonicalOperation::ReplaceParagraphSpan { + start_paragraph: Some(edited.ids.paragraph_ids[target]), + before, + after, + }; + + LocalApplier { doc: &loro, binding: &mut binding } + .apply(&edited, &[op]) + .expect("local apply"); + + let emitted: usize = updates.lock().expect("update lock").iter().map(Vec::len).sum(); + let body_bytes = full_body_len(&document); + // The whole-document body is large; a single-character incremental splice must + // stay far below it (a full-body rewrite would be >= body_bytes). + assert!(body_bytes > 4000, "fixture should be large, got {body_bytes} body bytes"); + assert!( + emitted * 4 < body_bytes, + "single-char edit emitted {emitted} update bytes for a {body_bytes}-byte body; expected an edit-proportional delta", + ); + + // The edit must still converge: a fresh peer importing the update matches. + let peer = schema::new_configured_doc(); + let snapshot = loro.export(ExportMode::Snapshot).expect("snapshot"); + let _ = peer.import(&snapshot); + let projected = projection::document_from_loro(&loro, DocumentTheme::default()).expect("project"); + assert_eq!(paragraph_text(&projected, target), format!("X{}", paragraph_text(&document, target))); + } + + #[test] + fn remote_single_char_insert_reconciles_one_paragraph() { + let session = SessionId::from_bytes([43; 32]); + let document = large_document(); + + let source_loro = schema::new_configured_doc(); + projection::populate_from_document(&source_loro, session, "perf-smoke", &document).expect("populate"); + let mut source_binding = DocBinding::build(&source_loro, &document).expect("binding"); + let updates = Arc::new(Mutex::new(Vec::new())); + let captured = updates.clone(); + let _subscription = source_loro.subscribe_local_update(Box::new(move |bytes| { + captured.lock().expect("update lock").push(bytes.to_vec()); + true + })); + let snapshot = source_loro.export(ExportMode::Snapshot).expect("snapshot"); + + let target_loro = schema::new_configured_doc(); + target_loro.import(&snapshot).expect("import snapshot"); + let mut target_document = projection::document_from_loro(&target_loro, DocumentTheme::default()).expect("project"); + let mut target_binding = DocBinding::build(&target_loro, &target_document).expect("binding"); + + // Source makes a single-character insert in the middle paragraph. + let target = PARAGRAPHS / 2; + let mut edited = document.clone(); + let before = capture_document_span(&edited, target..target + 1); + insert_text_at(&mut edited, target, 0, "X", RunStyles::default()); + let after = capture_document_span(&edited, target..target + 1); + LocalApplier { doc: &source_loro, binding: &mut source_binding } + .apply(&edited, &[CanonicalOperation::ReplaceParagraphSpan { + start_paragraph: Some(edited.ids.paragraph_ids[target]), + before, + after, + }]) + .expect("local apply"); + let update_batches = std::mem::take(&mut *updates.lock().expect("update lock")); + + let patches = import_updates(&target_loro, &mut target_document, &mut target_binding, &update_batches); + + // The remote reconcile must touch exactly the one changed paragraph row, not + // re-emit a patch per paragraph in the document. + assert_eq!(patches.len(), 1, "expected one reconcile patch, got {}: {patches:?}", patches.len()); + assert!( + matches!(patches.as_slice(), [CollabPatch::ParagraphText { row, .. }] if *row == target), + "expected a single ParagraphText patch for row {target}, got {patches:?}", + ); + + // And the imported document converges to the source projection. + let source_projection = projection::document_from_loro(&source_loro, DocumentTheme::default()).expect("project"); + assert_eq!(paragraph_text(&target_document, target), paragraph_text(&source_projection, target)); + assert_eq!(target_document.paragraphs.len(), source_projection.paragraphs.len()); + } + + fn large_document() -> Document { + let blocks = (0..PARAGRAPHS) + .map(|ix| InputBlock::Paragraph(InputParagraph { style: ParagraphStyle::Normal, runs: vec![plain(&format!("paragraph number {ix} body text"))] })) + .collect(); + document_from_input_blocks(DocumentTheme::default(), blocks) + } + + fn full_body_len(document: &Document) -> usize { + (0..document.paragraphs.len()).map(|ix| paragraph_text(document, ix).len() + 1).sum() + } + + fn import_updates(loro: &LoroDoc, document: &mut Document, binding: &mut DocBinding, updates: &[Vec]) -> Vec { + let mut all = Vec::new(); + for update in updates { + let snapshot_document = Arc::new(document.clone()); + let shared_binding = Arc::new(Mutex::new(std::mem::take(binding))); + let patches = Arc::new(Mutex::new(Vec::new())); + let doc = loro.clone(); + let callback: Subscriber = Arc::new({ + let shared_binding = shared_binding.clone(); + let patches = patches.clone(); + move |event| { + let produced = { + let mut guard = shared_binding.lock().expect("binding lock"); + flowstate_collab::remote_apply::RemoteApplier { doc: &doc, binding: &mut guard } + .apply_event(&snapshot_document, &event) + .expect("remote apply") + }; + patches.lock().expect("patch lock").extend(produced); + } + }); + let subscription = loro.subscribe_root(callback); + loro.import_with(update, "remote").expect("import"); + drop(subscription); + *binding = Arc::try_unwrap(shared_binding).expect("binding owner").into_inner().expect("binding lock"); + let produced = Arc::try_unwrap(patches).expect("patch owner").into_inner().expect("patch lock"); + apply_patches(document, binding, loro, &produced).expect("apply patches"); + all.extend(produced); + } + all + } +} diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index ab020762..8926e7e5 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -140,6 +140,7 @@ pub struct CollabSession { join_neighbor_tx: Option>, incompatible_version_peers: HashSet, last_document_activity: Instant, + last_self_check: Option<(Vec, u64)>, } const DIRECT_REQUEST_CHANNEL_CAPACITY: usize = 32; @@ -216,6 +217,7 @@ impl CollabSession { join_neighbor_tx: None, incompatible_version_peers: HashSet::new(), last_document_activity: now, + last_self_check: None, }) } @@ -264,6 +266,7 @@ impl CollabSession { join_neighbor_tx: None, incompatible_version_peers: HashSet::new(), last_document_activity: now, + last_self_check: None, } } diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs index 4cb9011e..6a93c194 100644 --- a/crates/flowstate/src/collab/session_timers.rs +++ b/crates/flowstate/src/collab/session_timers.rs @@ -261,7 +261,7 @@ impl CollabSession { } fn run_self_check(&mut self, cx: &mut Context) -> Result<()> { - let Some(doc) = &self.doc else { + let Some(doc) = self.doc.clone() else { return Ok(()); }; let Some(editor) = self.editor.clone() else { @@ -270,10 +270,24 @@ impl CollabSession { let live_document = editor.read(cx).document().clone(); let live_hash = self_check::projection_hash(&live_document); - let mut projected = projection::document_from_loro(doc, load_document_theme())?; + let current_vv = doc.oplog_vv().encode(); + // Cheap path: if neither the Loro state (version vector) nor the live + // projection hash changed since the last verified check, there can be no new + // drift, so skip the full reprojection. + if self + .last_self_check + .as_ref() + .is_some_and(|(vv, hash)| *vv == current_vv && *hash == live_hash) + { + tracing::trace!(session = %self.session, "collaboration projection self-check skipped (unchanged)"); + return Ok(()); + } + + let mut projected = projection::document_from_loro(&doc, load_document_theme())?; projected.assets = live_document.assets.clone(); let projected_hash = self_check::projection_hash(&projected); if live_hash == projected_hash { + self.last_self_check = Some((current_vv, live_hash)); tracing::trace!(session = %self.session, live_hash = %format_args!("{live_hash:016x}"), "collaboration projection self-check passed"); return Ok(()); } @@ -282,7 +296,7 @@ impl CollabSession { session = %self.session, live_hash = %format_args!("{live_hash:016x}"), projected_hash = %format_args!("{projected_hash:016x}"), - vv_bytes = doc.oplog_vv().encode().len(), + vv_bytes = current_vv.len(), "collaboration projection drift detected", ); self.rebuild_from_projection(projected, cx) diff --git a/crates/gpui-flowtext/src/collaboration.rs b/crates/gpui-flowtext/src/collaboration.rs index b1edfb22..87b9692b 100644 --- a/crates/gpui-flowtext/src/collaboration.rs +++ b/crates/gpui-flowtext/src/collaboration.rs @@ -1,5 +1,6 @@ use std::ops::Range; +use rustc_hash::FxHashMap; use serde::{Deserialize, Serialize}; use super::{ @@ -15,6 +16,7 @@ pub struct DocumentIdentityMap { paragraph_ids: Vec, block_ids: Vec, table_cell_ids: Vec>>, + paragraph_index_by_id: FxHashMap, } #[hotpath::measure_all] @@ -28,6 +30,10 @@ impl DocumentIdentityMap { pub fn reconcile(&mut self, document: &Document) { self.paragraph_ids.clone_from(&document.ids.paragraph_ids); + self.paragraph_index_by_id.clear(); + for (ix, id) in self.paragraph_ids.iter().enumerate() { + self.paragraph_index_by_id.insert(*id, ix); + } self.block_ids.clone_from(&document.ids.block_ids); self .table_cell_ids @@ -69,10 +75,7 @@ impl DocumentIdentityMap { #[must_use] pub fn paragraph_index(&self, id: ParagraphId) -> Option { - self - .paragraph_ids - .iter() - .position(|candidate| *candidate == id) + self.paragraph_index_by_id.get(&id).copied() } } @@ -141,158 +144,6 @@ pub enum CanonicalOperation { ReplaceDocument, } -#[derive(Clone, Debug, Serialize, Deserialize)] -enum WireCanonicalOperation { - InsertText { - paragraph: ParagraphId, - byte: usize, - text: String, - styles: RunStyles, - }, - DeleteRange { - start_paragraph: ParagraphId, - start_byte: usize, - end_paragraph: ParagraphId, - end_byte: usize, - }, - SplitParagraph { - paragraph: ParagraphId, - byte: usize, - new_paragraph: ParagraphId, - }, - JoinParagraphs { - first: ParagraphId, - second: ParagraphId, - }, - SetParagraphStyle { - paragraph: ParagraphId, - style: ParagraphStyle, - }, - SetRunStyles { - paragraph: ParagraphId, - range: Range, - styles: RunStyles, - }, -} - -impl WireCanonicalOperation { - fn from_canonical(operation: &CanonicalOperation) -> Option { - match operation { - CanonicalOperation::InsertText { - paragraph, - byte, - text, - styles, - } => Some(Self::InsertText { - paragraph: *paragraph, - byte: *byte, - text: text.clone(), - styles: *styles, - }), - CanonicalOperation::DeleteRange { - start_paragraph, - start_byte, - end_paragraph, - end_byte, - } => Some(Self::DeleteRange { - start_paragraph: *start_paragraph, - start_byte: *start_byte, - end_paragraph: *end_paragraph, - end_byte: *end_byte, - }), - CanonicalOperation::SplitParagraph { - paragraph, - byte, - new_paragraph, - } => Some(Self::SplitParagraph { - paragraph: *paragraph, - byte: *byte, - new_paragraph: *new_paragraph, - }), - CanonicalOperation::JoinParagraphs { first, second } => Some(Self::JoinParagraphs { - first: *first, - second: *second, - }), - CanonicalOperation::SetParagraphStyle { paragraph, style } => Some(Self::SetParagraphStyle { - paragraph: *paragraph, - style: *style, - }), - CanonicalOperation::SetRunStyles { paragraph, range, styles } => Some(Self::SetRunStyles { - paragraph: *paragraph, - range: range.clone(), - styles: *styles, - }), - CanonicalOperation::InsertBlock { .. } - | CanonicalOperation::DeleteBlock { .. } - | CanonicalOperation::MoveBlock { .. } - | CanonicalOperation::ReplaceParagraphSpan { .. } - | CanonicalOperation::ReplaceBlock { .. } - | CanonicalOperation::ReplaceDocument => None, - } - } - - fn into_canonical(self) -> CanonicalOperation { - match self { - Self::InsertText { - paragraph, - byte, - text, - styles, - } => CanonicalOperation::InsertText { - paragraph, - byte, - text, - styles, - }, - Self::DeleteRange { - start_paragraph, - start_byte, - end_paragraph, - end_byte, - } => CanonicalOperation::DeleteRange { - start_paragraph, - start_byte, - end_paragraph, - end_byte, - }, - Self::SplitParagraph { - paragraph, - byte, - new_paragraph, - } => CanonicalOperation::SplitParagraph { - paragraph, - byte, - new_paragraph, - }, - Self::JoinParagraphs { first, second } => CanonicalOperation::JoinParagraphs { first, second }, - Self::SetParagraphStyle { paragraph, style } => CanonicalOperation::SetParagraphStyle { paragraph, style }, - Self::SetRunStyles { paragraph, range, styles } => CanonicalOperation::SetRunStyles { paragraph, range, styles }, - } - } -} - -pub fn encode_canonical_operations(operations: &[CanonicalOperation]) -> Option> { - let wire_operations: Vec<_> = operations - .iter() - .filter_map(WireCanonicalOperation::from_canonical) - .collect(); - if wire_operations.is_empty() && !operations.is_empty() { - return None; - } - postcard::to_stdvec(&wire_operations).ok() -} - -pub fn decode_canonical_operations(bytes: &[u8]) -> Option> { - postcard::from_bytes::>(bytes) - .ok() - .map(|operations| { - operations - .into_iter() - .map(WireCanonicalOperation::into_canonical) - .collect() - }) -} - #[derive(Clone, Debug, Default)] pub struct CollaborationEdit { pub operations: Vec, diff --git a/crates/gpui-flowtext/src/document/core.rs b/crates/gpui-flowtext/src/document/core.rs index 29034961..18ad3d03 100644 --- a/crates/gpui-flowtext/src/document/core.rs +++ b/crates/gpui-flowtext/src/document/core.rs @@ -157,6 +157,22 @@ pub fn update_paragraph_block(document: &mut Document, paragraph_ix: usize) { #[hotpath::measure] pub fn replace_paragraph_blocks(document: &mut Document, start_paragraph: usize, old_count: usize, replacements: &[Paragraph]) { + // Fast path: a single in-place paragraph update in a paragraph-only-aligned + // document. Block ids and order are unchanged, so we replace just that one + // block instead of rebuilding the whole block vector. + if old_count == 1 + && replacements.len() == 1 + && document.blocks.len() == document.paragraphs.len() + && matches!(document.blocks.get(start_paragraph), Some(Block::Paragraph(_))) + { + if let Some(block) = Arc::make_mut(&mut document.blocks).get_mut(start_paragraph) { + *block = Block::Paragraph(replacements[0].clone()); + } + reconcile_document_ids(document); + rebuild_document_sections(document); + return; + } + let block_start = block_ix_for_paragraph(document, start_paragraph).unwrap_or(document.blocks.len()); let mut paragraph_ix = 0; let mut output = Vec::with_capacity(document.blocks.len() + replacements.len()); @@ -335,6 +351,16 @@ pub fn remove_block_ids(document: &mut Document, range: Range) { #[hotpath::measure] pub fn rebuild_document_sections(document: &mut Document) { reconcile_document_ids(document); + document.sections = Arc::new(document_sections(document)); +} + +/// Computes the heading outline purely from paragraph styles, order, and ids. +/// Pure (no `reconcile_document_ids`); callers that need id reconciliation use +/// [`rebuild_document_sections`]. Because the outline never depends on paragraph +/// text/runs, content-only edits can skip recomputation entirely. +#[hotpath::measure] +#[must_use] +pub fn document_sections(document: &Document) -> Vec { let mut sections: Vec = Vec::new(); let mut stack: Vec<(usize, SectionId)> = Vec::new(); @@ -374,7 +400,26 @@ pub fn rebuild_document_sections(document: &mut Document) { section.end_paragraph_exclusive = None; } } - document.sections = Arc::new(sections); + sections +} + +/// Whether the paragraph at `paragraph_ix` carries a heading (section) style. +#[hotpath::measure] +#[must_use] +pub fn paragraph_is_heading(document: &Document, paragraph_ix: usize) -> bool { + document + .paragraphs + .get(paragraph_ix) + .is_some_and(|paragraph| section_level_and_kind(document, paragraph.style).is_some()) +} + +/// Whether any paragraph in `range` (clamped to the paragraph count) is a heading. +/// Lets callers decide whether a content edit can skip [`rebuild_document_sections`]. +#[hotpath::measure] +#[must_use] +pub fn range_contains_heading(document: &Document, range: Range) -> bool { + let end = range.end.min(document.paragraphs.len()); + (range.start..end).any(|paragraph_ix| paragraph_is_heading(document, paragraph_ix)) } #[hotpath::measure] diff --git a/crates/gpui-flowtext/src/edit_ops/offsets.rs b/crates/gpui-flowtext/src/edit_ops/offsets.rs index f930d8b7..43d79932 100644 --- a/crates/gpui-flowtext/src/edit_ops/offsets.rs +++ b/crates/gpui-flowtext/src/edit_ops/offsets.rs @@ -42,37 +42,61 @@ pub fn refresh_paragraph_ranges(document: &mut Document) { } } -#[hotpath::measure] -fn refresh_paragraph_ranges_from(document: &mut Document, start_paragraph: usize) { - let paragraph_count = document.paragraphs.len(); - for paragraph_ix in start_paragraph.min(paragraph_count)..paragraph_count { - refresh_paragraph_range(document, paragraph_ix); - } -} - #[hotpath::measure] pub fn rebuild_document_offset_index(document: &mut Document) { document.offset_index.rebuild(&document.paragraphs); refresh_paragraph_ranges(document); } +fn shift_byte_range(range: &Range, delta: isize) -> Range { + range.start.saturating_add_signed(delta)..range.end.saturating_add_signed(delta) +} + +// Per-keystroke hot path: a single paragraph's text length changed (paragraph +// COUNT is unchanged). Update the Fenwick offset index in O(log n), shift the +// cached `byte_range`s of the edited paragraph and the ones after it, and mirror +// the change into the parallel block representation in place. We deliberately do +// NOT clone the paragraph tail, rebuild the block vector, reconcile ids, or +// rebuild the section outline: a content-only edit changes none of those. #[hotpath::measure] pub fn update_paragraph_offsets_after_len_change(document: &mut Document, paragraph_ix: usize) { if paragraph_ix >= document.paragraphs.len() { return; } + let new_len = paragraph_text_len(&document.paragraphs[paragraph_ix]); + let old_len = document.paragraphs[paragraph_ix].byte_range.len(); + let delta = new_len as isize - old_len as isize; document .offset_index .update_paragraph_width(paragraph_ix, &document.paragraphs); - refresh_paragraph_ranges_from(document, paragraph_ix); - let paragraph_count = document.paragraphs.len(); - let replacements = document.paragraphs[paragraph_ix..].to_vec(); - replace_paragraph_blocks( - document, - paragraph_ix, - paragraph_count.saturating_sub(paragraph_ix), - &replacements, - ); + + let start = document.paragraphs[paragraph_ix].byte_range.start; + { + let paragraphs = paragraphs_mut(document); + paragraphs[paragraph_ix].byte_range = start..start + new_len; + if delta != 0 { + for paragraph in paragraphs.iter_mut().skip(paragraph_ix + 1) { + paragraph.byte_range = shift_byte_range(¶graph.byte_range, delta); + } + } + } + + let mut edited = Some(document.paragraphs[paragraph_ix].clone()); + let blocks = Arc::make_mut(&mut document.blocks); + let mut paragraph_ord = 0usize; + for block in blocks.iter_mut() { + let Block::Paragraph(paragraph) = block else { + continue; + }; + if paragraph_ord == paragraph_ix { + if let Some(updated) = edited.take() { + *paragraph = updated; + } + } else if paragraph_ord > paragraph_ix && delta != 0 { + paragraph.byte_range = shift_byte_range(¶graph.byte_range, delta); + } + paragraph_ord += 1; + } } // Returns `(run_index, local_byte)` for the given absolute byte offset within diff --git a/crates/gpui-flowtext/src/edit_ops/text.rs b/crates/gpui-flowtext/src/edit_ops/text.rs index e87f622a..1218c315 100644 --- a/crates/gpui-flowtext/src/edit_ops/text.rs +++ b/crates/gpui-flowtext/src/edit_ops/text.rs @@ -92,8 +92,9 @@ pub fn apply_document_span_replacement(document: &mut Document, current: &Docume paragraph_end.saturating_sub(current.start_paragraph), &replacement.paragraphs, ); + // `replace_paragraph_blocks` already rebuilt the section outline; only the + // byte-offset index still needs refreshing after the splice. rebuild_document_offset_index(document); - rebuild_document_sections(document); } #[hotpath::measure] diff --git a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs index d1b0c294..05b02519 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/collab_apply.rs @@ -14,7 +14,6 @@ impl RichTextEditor { } self.suppress_collab_capture = self.suppress_collab_capture.saturating_sub(1); self.identity_map.reconcile(&self.document); - self.last_collaboration_edit = None; self.layout_invalidation_hint = invalidation; let generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); @@ -172,12 +171,18 @@ fn replace_paragraph_content(document: &mut Document, paragraph_ix: usize, parag let byte_range = paragraph_byte_range(document, paragraph_ix); document.text.delete(byte_range.clone()); document.text.insert(byte_range.start, &text); + let old_style = document.paragraphs[paragraph_ix].style; let mut replacement = paragraph_from_input_paragraph(paragraph); replacement.version = document.paragraphs[paragraph_ix].version.wrapping_add(1); + replacement.byte_range = byte_range.clone(); paragraphs_mut(document)[paragraph_ix] = replacement; - rebuild_document_offset_index(document); - update_paragraph_block(document, paragraph_ix); - rebuild_document_sections(document); + // Single in-place paragraph update (count unchanged): shift the offset index + // and the block in place. The section outline can only change if this + // paragraph's style changed. + update_paragraph_offsets_after_len_change(document, paragraph_ix); + if old_style != paragraph.style { + rebuild_document_sections(document); + } } #[hotpath::measure] diff --git a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs index b67c2f4d..33c85c8d 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs @@ -249,7 +249,6 @@ impl RichTextEditor { if reconcile_identity { self.identity_map.reconcile(&self.document); } - self.last_collaboration_edit = operations.map(|operations| CollaborationEdit { operations: operations.to_vec() }); if self.collab_capture && self.suppress_collab_capture == 0 && let Some(operations) = operations diff --git a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs index 1f919817..3aa27792 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/lifecycle.rs @@ -37,7 +37,6 @@ impl RichTextEditor { undo_stack: Vec::new(), redo_stack: Vec::new(), identity_map, - last_collaboration_edit: None, pending_collab_edits: Vec::new(), collab_capture: false, suppress_collab_capture: 0, @@ -157,7 +156,6 @@ impl RichTextEditor { fn release_transient_memory(&mut self) { self.undo_stack = Vec::new(); self.redo_stack = Vec::new(); - self.last_collaboration_edit = None; self.pending_collab_edits.clear(); self.collab_capture = false; self.suppress_collab_capture = 0; @@ -229,28 +227,6 @@ impl RichTextEditor { self.goal_x = None; } - pub fn last_collaboration_edit(&self) -> Option<&CollaborationEdit> { - self.last_collaboration_edit.as_ref() - } - - pub fn last_collaboration_operations(&self) -> Option<&[CanonicalOperation]> { - self - .last_collaboration_edit - .as_ref() - .map(|edit| edit.operations.as_slice()) - } - - pub fn last_collaboration_operation_bytes(&self) -> Option> { - self - .last_collaboration_edit - .as_ref() - .and_then(|edit| crate::encode_canonical_operations(&edit.operations)) - } - - pub fn clear_collaboration_edit(&mut self) { - self.last_collaboration_edit = None; - } - pub fn take_pending_collab_edits(&mut self) -> Vec { std::mem::take(&mut self.pending_collab_edits) } @@ -311,174 +287,12 @@ impl RichTextEditor { self.identity_map.table_cell_id(block_ix, row_ix, cell_ix) } - pub fn apply_remote_operations(&mut self, operations: &[CanonicalOperation], cx: &mut Context) { - for operation in operations { - self.apply_canonical_operation(operation); - } - self.identity_map.reconcile(&self.document); - self.last_collaboration_edit = None; - self.after_text_mutation(cx); - } - pub fn replace_document_from_collaboration(&mut self, document: Document, cx: &mut Context) { self.document = document; self.identity_map.reconcile(&self.document); - self.last_collaboration_edit = None; self.after_text_mutation(cx); } - fn apply_canonical_operation(&mut self, operation: &CanonicalOperation) { - match operation { - CanonicalOperation::InsertText { - paragraph, - byte, - text, - styles, - } => { - if let Some(paragraph_ix) = self.identity_map.paragraph_index(*paragraph) - && paragraph_offset_in_bounds( - &self.document, - DocumentOffset { - paragraph: paragraph_ix, - byte: *byte, - }, - ) - { - insert_text_at(&mut self.document, paragraph_ix, *byte, text, *styles); - } - }, - CanonicalOperation::DeleteRange { - start_paragraph, - start_byte, - end_paragraph, - end_byte, - } => { - let Some(start_paragraph) = self.identity_map.paragraph_index(*start_paragraph) else { - return; - }; - let Some(end_paragraph) = self.identity_map.paragraph_index(*end_paragraph) else { - return; - }; - if paragraph_offset_in_bounds( - &self.document, - DocumentOffset { - paragraph: start_paragraph, - byte: *start_byte, - }, - ) && paragraph_offset_in_bounds( - &self.document, - DocumentOffset { - paragraph: end_paragraph, - byte: *end_byte, - }, - ) { - delete_cross_paragraph_range( - &mut self.document, - DocumentOffset { - paragraph: start_paragraph, - byte: *start_byte, - }..DocumentOffset { - paragraph: end_paragraph, - byte: *end_byte, - }, - ); - } - }, - CanonicalOperation::SplitParagraph { paragraph, byte, .. } => { - if let Some(paragraph_ix) = self.identity_map.paragraph_index(*paragraph) - && paragraph_offset_in_bounds( - &self.document, - DocumentOffset { - paragraph: paragraph_ix, - byte: *byte, - }, - ) - { - split_paragraph_at(&mut self.document, paragraph_ix, *byte); - } - }, - CanonicalOperation::JoinParagraphs { first, second } => { - let Some(first_ix) = self.identity_map.paragraph_index(*first) else { - return; - }; - let Some(second_ix) = self.identity_map.paragraph_index(*second) else { - return; - }; - if second_ix == first_ix + 1 { - let byte = paragraph_text_len(&self.document.paragraphs[first_ix]); - delete_cross_paragraph_range( - &mut self.document, - DocumentOffset { paragraph: first_ix, byte }..DocumentOffset { - paragraph: second_ix, - byte: 0, - }, - ); - } - }, - CanonicalOperation::SetParagraphStyle { paragraph, style } => { - if let Some(paragraph_ix) = self.identity_map.paragraph_index(*paragraph) - && let Some(paragraph) = paragraphs_mut(&mut self.document).get_mut(paragraph_ix) - { - paragraph.style = *style; - bump_paragraph_version(paragraph); - update_paragraph_block(&mut self.document, paragraph_ix); - rebuild_document_sections(&mut self.document); - } - }, - CanonicalOperation::SetRunStyles { paragraph, range, styles } => { - if let Some(paragraph_ix) = self.identity_map.paragraph_index(*paragraph) - && paragraph_offset_in_bounds( - &self.document, - DocumentOffset { - paragraph: paragraph_ix, - byte: range.start, - }, - ) - && paragraph_offset_in_bounds( - &self.document, - DocumentOffset { - paragraph: paragraph_ix, - byte: range.end, - }, - ) - { - mutate_runs_in_range( - &mut self.document, - DocumentOffset { - paragraph: paragraph_ix, - byte: range.start, - }..DocumentOffset { - paragraph: paragraph_ix, - byte: range.end, - }, - |run_styles| *run_styles = *styles, - ); - } - }, - CanonicalOperation::ReplaceParagraphSpan { - start_paragraph, - before, - after, - } => { - let start = start_paragraph - .and_then(|id| self.identity_map.paragraph_index(id)) - .unwrap_or(before.start_paragraph); - let current = capture_document_span(&self.document, start..start + before.paragraphs.len()); - let replacement = DocumentSpan { - start_paragraph: start, - paragraphs: after.paragraphs.clone(), - text: after.text.clone(), - }; - apply_document_span_replacement(&mut self.document, ¤t, &replacement); - }, - CanonicalOperation::InsertBlock { .. } - | CanonicalOperation::DeleteBlock { .. } - | CanonicalOperation::MoveBlock { .. } - | CanonicalOperation::ReplaceBlock { .. } - | CanonicalOperation::ReplaceDocument => {}, - } - } - pub fn document_path(&self) -> Option<&PathBuf> { self.document_path.as_ref() } @@ -556,9 +370,4 @@ impl RichTextEditor { } } -fn paragraph_offset_in_bounds(document: &Document, offset: DocumentOffset) -> bool { - document - .paragraphs - .get(offset.paragraph) - .is_some_and(|paragraph| offset.byte <= paragraph_text_len(paragraph)) -} + diff --git a/crates/gpui-flowtext/src/rich_text/editor/mod.rs b/crates/gpui-flowtext/src/rich_text/editor/mod.rs index 050e43d3..498c9796 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mod.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mod.rs @@ -878,7 +878,6 @@ pub struct RichTextEditor { undo_stack: Vec, redo_stack: Vec, identity_map: DocumentIdentityMap, - last_collaboration_edit: Option, pending_collab_edits: Vec, collab_capture: bool, suppress_collab_capture: u32, diff --git a/fix.md b/fix.md index 4165f773..0c34a7ef 100644 --- a/fix.md +++ b/fix.md @@ -1,3 +1,15 @@ +> **SUPERSEDED — kept for history only.** The current source of truth is +> **`FIX_LORO_ROOT.md`** (read its §0–§4 first); do not implement from this file. +> +> This doc describes the *abandoned* v1 schema: one `LoroText` **per paragraph** +> held in `BindingRow { text: Option }`, with "no global offset arithmetic." +> The implementation now uses a **single-root `LoroText`** — one `"body"` container, +> paragraphs separated by `\n`, a maintained paragraph→byte offset index, and a +> `"blocks"` movable list of **metadata-only** maps. Treat every per-paragraph-text, +> `BindingRow { text }`, and "no offset math" claim below as obsolete. + +--- + # Flowstate Collaboration — Remediation Plan (`fix.md`) Closes the gaps found auditing the `fable-collab` branch against `plan.md`. The diff --git a/plan.md b/plan.md index 97655f22..aaf2af7c 100644 --- a/plan.md +++ b/plan.md @@ -1,3 +1,15 @@ +> **SUPERSEDED — kept for history only.** The current source of truth is +> **`FIX_LORO_ROOT.md`** (read its §0–§4 first); do not implement from this file. +> +> This doc describes the *abandoned* v1 schema: one `LoroText` **per paragraph** +> held in `BindingRow { text: Option }`, with "no global offset arithmetic." +> The implementation now uses a **single-root `LoroText`** — one `"body"` container, +> paragraphs separated by `\n`, a maintained paragraph→byte offset index, and a +> `"blocks"` movable list of **metadata-only** maps. Treat every per-paragraph-text, +> `BindingRow { text }`, and "no offset math" claim below as obsolete. + +--- + # Flowstate P2P Collaborative Editing — Implementation Plan (v2) **Status:** approved architecture, ready for implementation From 3fc4951c5593d802fb8337120b0cea942ffe206d Mon Sep 17 00:00:00 2001 From: Adhitya Thirumala Date: Wed, 17 Jun 2026 01:30:02 -0500 Subject: [PATCH 19/45] cargo fmt --- crates/flowstate-collab/src/binding.rs | 35 +- crates/flowstate-collab/src/body_index.rs | 11 +- crates/flowstate-collab/src/local_apply.rs | 85 +-- crates/flowstate-collab/src/net/blobs.rs | 11 +- crates/flowstate-collab/src/net/direct.rs | 65 ++- crates/flowstate-collab/src/net/swarm.rs | 7 +- crates/flowstate-collab/src/patch_apply.rs | 15 +- crates/flowstate-collab/src/projection.rs | 13 +- crates/flowstate-collab/src/remote_apply.rs | 36 +- crates/flowstate-collab/src/schema.rs | 24 +- crates/flowstate-collab/src/self_check.rs | 4 +- crates/flowstate-collab/tests/convergence.rs | 488 ++++++++++++++---- crates/flowstate-collab/tests/perf_smoke.rs | 70 ++- crates/flowstate-collab/tests/remote_apply.rs | 64 ++- .../flowstate-collab/tests/swarm_loopback.rs | 71 ++- crates/flowstate/src/collab/asset_transfer.rs | 22 +- crates/flowstate/src/collab/manager.rs | 36 +- crates/flowstate/src/collab/mod.rs | 4 +- crates/flowstate/src/collab/notify.rs | 11 +- crates/flowstate/src/collab/presence_view.rs | 10 +- crates/flowstate/src/collab/session.rs | 11 +- crates/flowstate/src/collab/session_io.rs | 41 +- crates/flowstate/src/collab/session_timers.rs | 10 +- crates/flowstate/src/collab/share_dialog.rs | 22 +- .../flowstate/src/collab/share_dialog_view.rs | 34 +- crates/flowstate/src/logging.rs | 13 +- crates/flowstate/src/main.rs | 3 +- crates/flowstate/src/workspace/icons.rs | 4 +- crates/gpui-flowtext/src/collaboration.rs | 3 +- .../gpui-flowtext/src/rich_text/editor/mod.rs | 3 +- 30 files changed, 873 insertions(+), 353 deletions(-) diff --git a/crates/flowstate-collab/src/binding.rs b/crates/flowstate-collab/src/binding.rs index 83ac0fee..7e3df165 100644 --- a/crates/flowstate-collab/src/binding.rs +++ b/crates/flowstate-collab/src/binding.rs @@ -112,8 +112,16 @@ impl DocBinding { debug_assert!(false, "binding row has no matching document block"); continue; }; - debug_assert_eq!(row.kind, BlockKind::from_block(block), "binding row {row_ix} kind does not match document block"); - debug_assert_eq!(document.ids.block_ids.get(row_ix).copied(), Some(row.block_id), "binding row {row_ix} block id does not match document block id"); + debug_assert_eq!( + row.kind, + BlockKind::from_block(block), + "binding row {row_ix} kind does not match document block" + ); + debug_assert_eq!( + document.ids.block_ids.get(row_ix).copied(), + Some(row.block_id), + "binding row {row_ix} block id does not match document block id" + ); debug_assert_eq!(row.version, block_version(block)); if matches!(row.kind, BlockKind::Paragraph) { debug_assert_eq!(document.ids.paragraph_ids.get(paragraph_ix).copied(), row.paragraph_id); @@ -123,16 +131,31 @@ impl DocBinding { } } - debug_assert_eq!(self.body_index.len(), paragraph_ix, "body index paragraph count does not match paragraph rows"); + debug_assert_eq!( + self.body_index.len(), + paragraph_ix, + "body index paragraph count does not match paragraph rows" + ); #[cfg(debug_assertions)] { let body_lens = paragraph_lens_from_body_text(&body_text(doc)); - debug_assert_eq!(self.body_index.len(), body_lens.len(), "body index paragraph count does not match the body text"); + debug_assert_eq!( + self.body_index.len(), + body_lens.len(), + "body index paragraph count does not match the body text" + ); for (ordinal, expected) in body_lens.iter().enumerate() { - debug_assert_eq!(self.body_index.paragraph_len(ordinal), *expected, "body index byte length mismatch vs body at paragraph {ordinal}"); debug_assert_eq!( self.body_index.paragraph_len(ordinal), - document.paragraphs.get(ordinal).map_or(0, paragraph_text_len), + *expected, + "body index byte length mismatch vs body at paragraph {ordinal}" + ); + debug_assert_eq!( + self.body_index.paragraph_len(ordinal), + document + .paragraphs + .get(ordinal) + .map_or(0, paragraph_text_len), "body index byte length mismatch vs document at paragraph {ordinal}" ); } diff --git a/crates/flowstate-collab/src/body_index.rs b/crates/flowstate-collab/src/body_index.rs index b6d522b0..e077e57e 100644 --- a/crates/flowstate-collab/src/body_index.rs +++ b/crates/flowstate-collab/src/body_index.rs @@ -232,11 +232,7 @@ mod tests { use super::*; fn fresh_start(lens: &[usize], ordinal: usize) -> usize { - lens - .iter() - .take(ordinal) - .map(|len| len + 1) - .sum() + lens.iter().take(ordinal).map(|len| len + 1).sum() } #[test] @@ -245,7 +241,10 @@ mod tests { let index = BodyParagraphIndex::from_lens(lens.clone()); for ordinal in 0..lens.len() { assert_eq!(index.paragraph_start(ordinal), fresh_start(&lens, ordinal)); - assert_eq!(index.paragraph_range(ordinal), fresh_start(&lens, ordinal)..fresh_start(&lens, ordinal) + lens[ordinal]); + assert_eq!( + index.paragraph_range(ordinal), + fresh_start(&lens, ordinal)..fresh_start(&lens, ordinal) + lens[ordinal] + ); } // Total length has no trailing newline after the last paragraph. let total: usize = lens.iter().sum::() + lens.len() - 1; diff --git a/crates/flowstate-collab/src/local_apply.rs b/crates/flowstate-collab/src/local_apply.rs index 19b122fc..622aff29 100644 --- a/crates/flowstate-collab/src/local_apply.rs +++ b/crates/flowstate-collab/src/local_apply.rs @@ -3,7 +3,9 @@ use anyhow::{Context as _, Result, bail}; use std::ops::Range; -use gpui_flowtext::{Block, BlockId, CanonicalOperation, Document, DocumentSpan, Paragraph, ParagraphId, RunStyles, full_document_text, paragraph_text_len}; +use gpui_flowtext::{ + Block, BlockId, CanonicalOperation, Document, DocumentSpan, Paragraph, ParagraphId, RunStyles, full_document_text, paragraph_text_len, +}; use loro::{CommitOptions, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer}; use crate::{ @@ -116,7 +118,10 @@ impl LocalApplier<'_> { if start_ix == end_ix { if let Some(ordinal) = self.paragraph_ordinal_for_row(start_ix) { let len = self.binding.body_index.paragraph_len(ordinal); - self.binding.body_index.set_paragraph_len(ordinal, len.saturating_sub(deleted)); + self + .binding + .body_index + .set_paragraph_len(ordinal, len.saturating_sub(deleted)); } self.debug_assert_body_matches_document(document); return Ok(()); @@ -149,7 +154,10 @@ impl LocalApplier<'_> { .binding .body_index .remove_paragraph_range(start_paragraph_ix + 1, end_paragraph_ix - start_paragraph_ix); - self.binding.body_index.set_paragraph_len(start_paragraph_ix, merged_len); + self + .binding + .body_index + .set_paragraph_len(start_paragraph_ix, merged_len); } self.debug_assert_body_matches_document(document); Ok(()) @@ -179,21 +187,18 @@ impl LocalApplier<'_> { let blocks = self.blocks(); let paragraph_ix = orig_ordinal.saturating_add(1); - let style = document - .paragraphs - .get(paragraph_ix) - .map_or_else( - || { - self - .binding - .rows - .get(row_ix) - .and_then(|row| map_i64(&row.map, STYLE).ok()) - .map(decode_paragraph_style) - .unwrap_or(gpui_flowtext::ParagraphStyle::Normal) - }, - |paragraph| paragraph.style, - ); + let style = document.paragraphs.get(paragraph_ix).map_or_else( + || { + self + .binding + .rows + .get(row_ix) + .and_then(|row| map_i64(&row.map, STYLE).ok()) + .map(decode_paragraph_style) + .unwrap_or(gpui_flowtext::ParagraphStyle::Normal) + }, + |paragraph| paragraph.style, + ); let map = insert_paragraph_container(&blocks, insert_ix, style)?; self.binding.insert_row( insert_ix, @@ -205,7 +210,10 @@ impl LocalApplier<'_> { version, }, ); - self.binding.body_index.set_paragraph_len(orig_ordinal, split); + self + .binding + .body_index + .set_paragraph_len(orig_ordinal, split); self .binding .body_index @@ -219,9 +227,7 @@ impl LocalApplier<'_> { let first_paragraph_ix = self .paragraph_ordinal_for_row(first_ix) .context("join first row is not a paragraph")?; - let first_len = self - .body_paragraph_range(first_paragraph_ix)? - .len(); + let first_len = self.body_paragraph_range(first_paragraph_ix)?.len(); self.delete_range(document, first, first_len, second, 0) } @@ -241,13 +247,7 @@ impl LocalApplier<'_> { .get(paragraph_ix) .copied() .context("inserted paragraph block is missing a paragraph id")?; - self.insert_paragraph_row( - block_ix, - block_id, - paragraph_id, - paragraph.style, - block_version(block), - )?; + self.insert_paragraph_row(block_ix, block_id, paragraph_id, paragraph.style, block_version(block))?; self.debug_assert_body_matches_document(document); Ok(()) }, @@ -271,10 +271,7 @@ impl LocalApplier<'_> { fn delete_block(&mut self, document: &Document, block: BlockId) -> Result<()> { let row_ix = self.row_ix_for_block(block)?; - let was_paragraph = matches!( - self.binding.rows.get(row_ix).map(|row| row.kind), - Some(BlockKind::Paragraph) - ); + let was_paragraph = matches!(self.binding.rows.get(row_ix).map(|row| row.kind), Some(BlockKind::Paragraph)); self.blocks().delete(row_ix, 1)?; self .binding @@ -290,10 +287,7 @@ impl LocalApplier<'_> { fn move_block(&mut self, document: &Document, block: BlockId, new_block_ix: usize) -> Result<()> { let old_ix = self.row_ix_for_block(block)?; - let was_paragraph = matches!( - self.binding.rows.get(old_ix).map(|row| row.kind), - Some(BlockKind::Paragraph) - ); + let was_paragraph = matches!(self.binding.rows.get(old_ix).map(|row| row.kind), Some(BlockKind::Paragraph)); self.blocks().mov(old_ix, new_block_ix)?; self.binding.move_row(old_ix, new_block_ix); if was_paragraph { @@ -356,14 +350,22 @@ impl LocalApplier<'_> { return Ok(false); }; let first_row = before_rows[0]; - if before_rows.iter().enumerate().any(|(offset, row)| *row != first_row + offset) { + if before_rows + .iter() + .enumerate() + .any(|(offset, row)| *row != first_row + offset) + { return Ok(false); } // The body slice for the span must currently equal `before.text`; if the // lengths disagree the editor and CRDT have diverged — bail to the rewrite. let start_byte = self.binding.body_index.paragraph_start(start_ord); - let end_byte = self.binding.body_index.paragraph_range(start_ord + before_n - 1).end; + let end_byte = self + .binding + .body_index + .paragraph_range(start_ord + before_n - 1) + .end; if end_byte - start_byte != before.text.len() { return Ok(false); } @@ -461,7 +463,10 @@ impl LocalApplier<'_> { .remove_row(row_ix) .context("spliced paragraph row disappeared during span shrink")?; } - self.binding.body_index.remove_paragraph_range(start_ord + after_n, removed); + self + .binding + .body_index + .remove_paragraph_range(start_ord + after_n, removed); } self.binding.rebuild_indexes(); diff --git a/crates/flowstate-collab/src/net/blobs.rs b/crates/flowstate-collab/src/net/blobs.rs index a4c9c1e3..98a2a4ed 100644 --- a/crates/flowstate-collab/src/net/blobs.rs +++ b/crates/flowstate-collab/src/net/blobs.rs @@ -43,8 +43,15 @@ impl BlobOutbox { } pub fn insert_with_id(&mut self, id: BlobId, bytes: Vec) { - if let Some((_, existing)) = self.entries.iter_mut().find(|(candidate, _)| *candidate == id) { - self.total_bytes = self.total_bytes.saturating_sub(existing.len()).saturating_add(bytes.len()); + if let Some((_, existing)) = self + .entries + .iter_mut() + .find(|(candidate, _)| *candidate == id) + { + self.total_bytes = self + .total_bytes + .saturating_sub(existing.len()) + .saturating_add(bytes.len()); *existing = bytes; self.trim(); return; diff --git a/crates/flowstate-collab/src/net/direct.rs b/crates/flowstate-collab/src/net/direct.rs index 12511052..57aeaceb 100644 --- a/crates/flowstate-collab/src/net/direct.rs +++ b/crates/flowstate-collab/src/net/direct.rs @@ -19,8 +19,7 @@ use tokio::{ use crate::{ ids::{BlobId, SessionId}, proto_direct::{ - AssetBytes, DIRECT_ALPN, DirectRequest, DirectResponseHeader, MAX_FRAME_LEN, MAX_PAYLOAD_CHUNK_LEN, decode_frame, - encode_frame, + AssetBytes, DIRECT_ALPN, DirectRequest, DirectResponseHeader, MAX_FRAME_LEN, MAX_PAYLOAD_CHUNK_LEN, decode_frame, encode_frame, }, }; @@ -70,7 +69,10 @@ impl DirectServeState { pub async fn detach_session(&self, session: SessionId) { let mut inner = self.inner.write().await; let removed = inner.attached.remove(&session); - let blob_count = inner.blobs.remove(&session).map_or(0, |outbox| outbox.len()); + let blob_count = inner + .blobs + .remove(&session) + .map_or(0, |outbox| outbox.len()); let handler_removed = inner.handlers.remove(&session).is_some(); tracing::debug!(%session, removed, blob_count, handler_removed, attached_sessions = inner.attached.len(), "detached collaboration direct session"); } @@ -121,13 +123,16 @@ impl DirectServeState { .blobs .get(&session) .and_then(|outbox| outbox.get(blob)) - .map_or_else(|| { - tracing::warn!(%session, ?blob, "collaboration direct blob request missed outbox"); - ServeOutcome::Header(DirectResponseHeader::NotFound) - }, |bytes| { - tracing::debug!(%session, ?blob, bytes = bytes.len(), "collaboration direct blob request hit outbox"); - ServeOutcome::Payload(bytes.to_vec()) - }); + .map_or_else( + || { + tracing::warn!(%session, ?blob, "collaboration direct blob request missed outbox"); + ServeOutcome::Header(DirectResponseHeader::NotFound) + }, + |bytes| { + tracing::debug!(%session, ?blob, bytes = bytes.len(), "collaboration direct blob request hit outbox"); + ServeOutcome::Payload(bytes.to_vec()) + }, + ); } inner.handlers.get(&session).cloned() }; @@ -138,10 +143,16 @@ impl DirectServeState { }; match request { - DirectRequest::Snapshot { .. } => request_payload(handler.requests, session, request_kind, |reply| DirectServeRequest::Snapshot { reply }).await, + DirectRequest::Snapshot { .. } => { + request_payload(handler.requests, session, request_kind, |reply| DirectServeRequest::Snapshot { reply }).await + }, DirectRequest::Updates { have_vv, .. } => { tracing::trace!(%session, have_vv_bytes = have_vv.len(), "forwarding collaboration direct updates request to session"); - request_payload(handler.requests, session, request_kind, |reply| DirectServeRequest::Updates { have_vv, reply }).await + request_payload(handler.requests, session, request_kind, |reply| DirectServeRequest::Updates { + have_vv, + reply, + }) + .await }, DirectRequest::Asset { asset, .. } => request_asset(handler.requests, session, asset).await, DirectRequest::Blob { .. } => ServeOutcome::Header(DirectResponseHeader::NotFound), @@ -329,7 +340,12 @@ async fn pull_once(endpoint: &Endpoint, peer: EndpointId, req: DirectRequest, pr } } -async fn request_payload(requests: Sender, session: SessionId, request_kind: &'static str, make_request: F) -> ServeOutcome +async fn request_payload( + requests: Sender, + session: SessionId, + request_kind: &'static str, + make_request: F, +) -> ServeOutcome where F: FnOnce(Sender>>) -> DirectServeRequest, { @@ -395,7 +411,11 @@ where } async fn write_response(send: &mut SendStream, outcome: ServeOutcome) -> Result<()> { - tracing::trace!(outcome = outcome.kind(), payload_bytes = outcome.payload_len(), "writing collaboration direct response"); + tracing::trace!( + outcome = outcome.kind(), + payload_bytes = outcome.payload_len(), + "writing collaboration direct response" + ); match outcome { ServeOutcome::Header(header) => write_frame(send, &encode_frame(&header)?).await?, ServeOutcome::Payload(payload) => { @@ -417,7 +437,11 @@ async fn write_frame(send: &mut SendStream, frame: &[u8]) -> Result<()> { } async fn write_payload(send: &mut SendStream, payload: &[u8]) -> Result<()> { - tracing::trace!(payload_bytes = payload.len(), chunk_bytes = MAX_PAYLOAD_CHUNK_LEN, "writing collaboration direct payload"); + tracing::trace!( + payload_bytes = payload.len(), + chunk_bytes = MAX_PAYLOAD_CHUNK_LEN, + "writing collaboration direct payload" + ); for chunk in payload.chunks(MAX_PAYLOAD_CHUNK_LEN) { send.write_all(chunk).await?; } @@ -426,7 +450,11 @@ async fn write_payload(send: &mut SendStream, payload: &[u8]) -> Result<()> { async fn read_payload(recv: &mut RecvStream, total_len: u64, progress: Option<&Sender>) -> Result> { let total_len_usize = usize::try_from(total_len).context("direct payload is too large for this platform")?; - tracing::trace!(payload_bytes = total_len_usize, chunk_bytes = MAX_PAYLOAD_CHUNK_LEN, "reading collaboration direct payload"); + tracing::trace!( + payload_bytes = total_len_usize, + chunk_bytes = MAX_PAYLOAD_CHUNK_LEN, + "reading collaboration direct payload" + ); let mut payload = vec![0; total_len_usize]; let mut offset = 0; if let Some(progress) = progress { @@ -437,7 +465,10 @@ async fn read_payload(recv: &mut RecvStream, total_len: u64, progress: Option<&S recv.read_exact(&mut payload[offset..next]).await?; offset = next; if let Some(progress) = progress { - let _ = progress.try_send(PullProgress { got: offset as u64, total: total_len }); + let _ = progress.try_send(PullProgress { + got: offset as u64, + total: total_len, + }); } } Ok(payload) diff --git a/crates/flowstate-collab/src/net/swarm.rs b/crates/flowstate-collab/src/net/swarm.rs index 7b2ab7dc..9a09f0d9 100644 --- a/crates/flowstate-collab/src/net/swarm.rs +++ b/crates/flowstate-collab/src/net/swarm.rs @@ -250,9 +250,9 @@ mod tests { }; use crate::{ + SessionId, net::NetEvent, proto_gossip::{self, GossipMsg}, - SessionId, }; use super::handle_event; @@ -280,7 +280,10 @@ mod tests { .expect("gossip event handling should ignore mismatches"); match evt_rx.recv().await.expect("incompatible-version event") { - NetEvent::IncompatibleVersion { session: event_session, peer: event_peer } => { + NetEvent::IncompatibleVersion { + session: event_session, + peer: event_peer, + } => { assert_eq!(event_session, session); assert_eq!(event_peer, peer); }, diff --git a/crates/flowstate-collab/src/patch_apply.rs b/crates/flowstate-collab/src/patch_apply.rs index f737e444..6320029a 100644 --- a/crates/flowstate-collab/src/patch_apply.rs +++ b/crates/flowstate-collab/src/patch_apply.rs @@ -2,8 +2,8 @@ use anyhow::{Context as _, Result}; use gpui_flowtext::{ - Block, CollabPatch, CollabStructuralBlock, Document, InputBlock, InputParagraph, InputRun, document_from_input_blocks, - input_block_from_block, paragraph_text, paragraphs_mut, rebuild_document_sections, update_paragraph_block, + Block, CollabPatch, CollabStructuralBlock, Document, InputBlock, InputParagraph, InputRun, document_from_input_blocks, input_block_from_block, + paragraph_text, paragraphs_mut, rebuild_document_sections, update_paragraph_block, }; use loro::LoroDoc; @@ -157,11 +157,18 @@ fn rebuild_document_from_structural_blocks(document: &mut Document, blocks: Vec< .iter() .map(|block| block.block.clone()) .collect::>(); - let block_ids = blocks.iter().map(|block| block.block_id).collect::>(); + let block_ids = blocks + .iter() + .map(|block| block.block_id) + .collect::>(); let paragraph_ids = blocks .iter() .filter_map(|block| match &block.block { - InputBlock::Paragraph(_) => Some(block.paragraph_id.context("structural paragraph patch is missing a paragraph id")), + InputBlock::Paragraph(_) => Some( + block + .paragraph_id + .context("structural paragraph patch is missing a paragraph id"), + ), InputBlock::Image(_) | InputBlock::Equation(_) | InputBlock::Table(_) => None, }) .collect::>>()?; diff --git a/crates/flowstate-collab/src/projection.rs b/crates/flowstate-collab/src/projection.rs index fd0f7bd0..282348bb 100644 --- a/crates/flowstate-collab/src/projection.rs +++ b/crates/flowstate-collab/src/projection.rs @@ -67,10 +67,9 @@ pub fn input_block_from_container(map: &LoroMap, body_paragraph: Option<&gpui_fl match map_string(map, KIND)?.as_str() { KIND_PARAGRAPH => { let style = decode_paragraph_style(map_i64(map, STYLE)?); - let mut paragraph = body_paragraph.cloned().unwrap_or(gpui_flowtext::InputParagraph { - style, - runs: Vec::new(), - }); + let mut paragraph = body_paragraph + .cloned() + .unwrap_or(gpui_flowtext::InputParagraph { style, runs: Vec::new() }); if body_paragraph.is_none() { paragraph.style = style; } @@ -100,11 +99,7 @@ fn clear_blocks(blocks: &LoroMovableList) -> Result<()> { Ok(()) } -pub fn insert_paragraph_container( - blocks: &LoroMovableList, - ix: usize, - style: gpui_flowtext::ParagraphStyle, -) -> Result { +pub fn insert_paragraph_container(blocks: &LoroMovableList, ix: usize, style: gpui_flowtext::ParagraphStyle) -> Result { let map = blocks.insert_container(ix, LoroMap::new())?; map.insert(KIND, KIND_PARAGRAPH)?; map.insert(STYLE, encode_paragraph_style(style))?; diff --git a/crates/flowstate-collab/src/remote_apply.rs b/crates/flowstate-collab/src/remote_apply.rs index eaa2cb78..4d92b26d 100644 --- a/crates/flowstate-collab/src/remote_apply.rs +++ b/crates/flowstate-collab/src/remote_apply.rs @@ -3,7 +3,7 @@ use anyhow::{Context as _, Result, bail}; use gpui_flowtext::{ BlockId, CollabPatch, CollabStructuralBlock, CollabTextDelta, Document, InputBlock, InputParagraph, InputRun, ParagraphId, ParagraphStyle, - new_block_id, new_paragraph_id, paragraph_text, TextRun, + TextRun, new_block_id, new_paragraph_id, paragraph_text, }; use loro::{ Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroValue, TextDelta, ValueOrContainer, @@ -42,7 +42,12 @@ impl RemoteApplier<'_> { } }, Diff::Text(_) => {}, - Diff::Map(delta) => self.apply_map_diff(diff.target, delta.updated.keys().map(|key| key.as_ref()), &inserted_containers, &mut patches)?, + Diff::Map(delta) => self.apply_map_diff( + diff.target, + delta.updated.keys().map(|key| key.as_ref()), + &inserted_containers, + &mut patches, + )?, Diff::List(delta) => self.apply_list_diff(diff.target, delta, &mut patches)?, Diff::Tree(_) | Diff::Counter(_) | Diff::Unknown => {}, } @@ -52,9 +57,7 @@ impl RemoteApplier<'_> { // this event, a single contiguous change, and no newline change — reconciles // only the affected paragraph. Anything else falls back to the full // reprojection (which is always correct). - let handled = !multiple_body_deltas - && patches.is_empty() - && self.try_reconcile_body_delta(document, &delta, &mut patches)?; + let handled = !multiple_body_deltas && patches.is_empty() && self.try_reconcile_body_delta(document, &delta, &mut patches)?; if !handled { self.reconcile_body_text(document, &pre_event_binding, &mut patches)?; self.binding.refresh_body_index_from_loro(self.doc); @@ -86,7 +89,10 @@ impl RemoteApplier<'_> { // (e.g. a remote run/paragraph style edit). That changes a paragraph's // runs without changing its text, which this text-splice fast path does // not model — defer to the full reprojection. - if attributes.as_ref().is_some_and(|attributes| !attributes.is_empty()) { + if attributes + .as_ref() + .is_some_and(|attributes| !attributes.is_empty()) + { return Ok(false); } if in_change { @@ -118,7 +124,10 @@ impl RemoteApplier<'_> { let new_body = body_text(self.doc); let change_byte = utf8_byte(&new_body, lead); - let ordinal = self.binding.body_index.paragraph_ordinal_for_body_byte(change_byte); + let ordinal = self + .binding + .body_index + .paragraph_ordinal_for_body_byte(change_byte); let start_byte = self.binding.body_index.paragraph_start(ordinal); let old_len = self.binding.body_index.paragraph_len(ordinal); if change_byte < start_byte || change_byte > start_byte + old_len { @@ -308,7 +317,10 @@ impl RemoteApplier<'_> { }; if final_block_ids.contains(&row.map.id()) { if let Some(start) = delete_start.take() { - patches.push(CollabPatch::DeleteBlocks { row: start, count: delete_count }); + patches.push(CollabPatch::DeleteBlocks { + row: start, + count: delete_count, + }); delete_count = 0; } row_ix += 1; @@ -322,7 +334,10 @@ impl RemoteApplier<'_> { delete_count += 1; } if let Some(start) = delete_start { - patches.push(CollabPatch::DeleteBlocks { row: start, count: delete_count }); + patches.push(CollabPatch::DeleteBlocks { + row: start, + count: delete_count, + }); } }, ListDiffItem::Insert { insert, is_move } => { @@ -448,8 +463,7 @@ fn input_paragraph_text(paragraph: &InputParagraph) -> String { } fn input_paragraphs_equal(left: &InputParagraph, right: &InputParagraph) -> bool { - left.style == right.style - && input_runs_equal(&left.runs, &right.runs) + left.style == right.style && input_runs_equal(&left.runs, &right.runs) } fn has_paragraph_style_patch(patches: &[CollabPatch], target_row: usize) -> bool { diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs index 3e6ea637..6f396839 100644 --- a/crates/flowstate-collab/src/schema.rs +++ b/crates/flowstate-collab/src/schema.rs @@ -194,11 +194,7 @@ pub fn set_run_styles_utf8(text: &LoroText, range: Range, styles: RunStyl Ok(()) } -pub fn set_paragraph_style_utf8( - text: &LoroText, - range: Range, - style: gpui_flowtext::ParagraphStyle, -) -> LoroResult<()> { +pub fn set_paragraph_style_utf8(text: &LoroText, range: Range, style: gpui_flowtext::ParagraphStyle) -> LoroResult<()> { if range.is_empty() { return Ok(()); } @@ -268,7 +264,10 @@ pub fn input_paragraphs_from_body_text(text: &LoroText) -> Vec { for (ix, segment) in insert.split('\n').enumerate() { if ix > 0 { if let Some(style) = paragraph_style { - paragraphs.last_mut().expect("paragraph accumulator is never empty").style = style; + paragraphs + .last_mut() + .expect("paragraph accumulator is never empty") + .style = style; } paragraphs.push(InputParagraph { style: paragraph_style.unwrap_or(gpui_flowtext::ParagraphStyle::Normal), @@ -277,9 +276,18 @@ pub fn input_paragraphs_from_body_text(text: &LoroText) -> Vec { } if !segment.is_empty() { if let Some(style) = paragraph_style { - paragraphs.last_mut().expect("paragraph accumulator is never empty").style = style; + paragraphs + .last_mut() + .expect("paragraph accumulator is never empty") + .style = style; } - push_input_run(paragraphs.last_mut().expect("paragraph accumulator is never empty"), segment, styles); + push_input_run( + paragraphs + .last_mut() + .expect("paragraph accumulator is never empty"), + segment, + styles, + ); } } } diff --git a/crates/flowstate-collab/src/self_check.rs b/crates/flowstate-collab/src/self_check.rs index 48f3a0d7..671406fd 100644 --- a/crates/flowstate-collab/src/self_check.rs +++ b/crates/flowstate-collab/src/self_check.rs @@ -19,7 +19,9 @@ pub fn projection_hash(document: &Document) -> u64 { run.len.hash(&mut hasher); run.styles.hash(&mut hasher); } - paragraph_text(document, paragraph_ix).as_bytes().hash(&mut hasher); + paragraph_text(document, paragraph_ix) + .as_bytes() + .hash(&mut hasher); "p".hash(&mut hasher); paragraph_ix += 1; }, diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs index 63a3d6ab..fe668f96 100644 --- a/crates/flowstate-collab/tests/convergence.rs +++ b/crates/flowstate-collab/tests/convergence.rs @@ -1,15 +1,13 @@ #[cfg(test)] mod tests { - use std::{fmt::Write as _, ops::Range, sync::{Arc, Mutex}}; + use std::{ + fmt::Write as _, + ops::Range, + sync::{Arc, Mutex}, + }; use anyhow::{Context as _, Result, anyhow, bail}; - use flowstate_collab::{ - SessionId, - binding::DocBinding, - local_apply::LocalApplier, - patch_apply::apply_patches, - projection, schema, - }; + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, patch_apply::apply_patches, projection, schema}; use gpui_flowtext::{ Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, HighlightStyle, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, ParagraphStyle, RunSemanticStyle, RunStyle, RunStyles, block_from_input_block, @@ -38,14 +36,60 @@ mod tests { #[test] fn reordered_run_style_after_inserts_converges() { let ops = [ - FuzzOp::Style { peer: 0, paragraph: 0, start: 0, len: 0, style: 0 }, - FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, - FuzzOp::Style { peer: 0, paragraph: 0, start: 0, len: 0, style: 0 }, - FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, - FuzzOp::DeleteText { peer: 0, paragraph: 61, start: 31, len: 29 }, - FuzzOp::DeleteText { peer: 138, paragraph: 161, start: 6, len: 0 }, - FuzzOp::InsertText { peer: 0, paragraph: 125, byte: 20, text: 0, style: 0 }, - FuzzOp::Style { peer: 2, paragraph: 12, start: 0, len: 0, style: 1 }, + FuzzOp::Style { + peer: 0, + paragraph: 0, + start: 0, + len: 0, + style: 0, + }, + FuzzOp::InsertText { + peer: 0, + paragraph: 0, + byte: 0, + text: 0, + style: 0, + }, + FuzzOp::Style { + peer: 0, + paragraph: 0, + start: 0, + len: 0, + style: 0, + }, + FuzzOp::InsertText { + peer: 0, + paragraph: 0, + byte: 0, + text: 0, + style: 0, + }, + FuzzOp::DeleteText { + peer: 0, + paragraph: 61, + start: 31, + len: 29, + }, + FuzzOp::DeleteText { + peer: 138, + paragraph: 161, + start: 6, + len: 0, + }, + FuzzOp::InsertText { + peer: 0, + paragraph: 125, + byte: 20, + text: 0, + style: 0, + }, + FuzzOp::Style { + peer: 2, + paragraph: 12, + start: 0, + len: 0, + style: 1, + }, ]; run_program(2, 4754897484207400829, &ops).expect("reordered run style after inserts should converge"); @@ -84,7 +128,11 @@ mod tests { #[test] fn insert_object_at_start_then_split_converges_across_three_peers() { let ops = [ - FuzzOp::InsertObject { peer: 96, row: 129, source: 0 }, + FuzzOp::InsertObject { + peer: 96, + row: 129, + source: 0, + }, FuzzOp::Split { peer: 108, paragraph: 91, @@ -95,7 +143,6 @@ mod tests { run_program(3, 0, &ops).expect("insert object at start then split should converge"); } - #[test] fn insert_object_between_paragraphs_does_not_create_invalid_join() { let ops = [ @@ -109,7 +156,11 @@ mod tests { #[test] fn insert_object_between_paragraphs_then_text_edit_preserves_order() { let ops = [ - FuzzOp::InsertObject { peer: 7, row: 106, source: 0 }, + FuzzOp::InsertObject { + peer: 7, + row: 106, + source: 0, + }, FuzzOp::DeleteText { peer: 19, paragraph: 48, @@ -177,11 +228,17 @@ mod tests { let paragraph = bob.document.ids.paragraph_ids[0]; paragraphs_mut(&mut bob.document)[0].style = ParagraphStyle::Custom(3); update_paragraph_block(&mut bob.document, 0); - LocalApplier { doc: &bob.loro, binding: &mut bob.binding } - .apply(&bob.document, &[CanonicalOperation::SetParagraphStyle { + LocalApplier { + doc: &bob.loro, + binding: &mut bob.binding, + } + .apply( + &bob.document, + &[CanonicalOperation::SetParagraphStyle { paragraph, style: ParagraphStyle::Custom(3), - }])?; + }], + )?; let bob_updates = bob.drain_updates(); for update in &bob_updates { @@ -203,21 +260,78 @@ mod tests { #[test] fn concurrent_object_insert_move_interleaving_converges() { let ops = [ - FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, - FuzzOp::InsertText { peer: 0, paragraph: 0, byte: 0, text: 0, style: 0 }, + FuzzOp::InsertText { + peer: 0, + paragraph: 0, + byte: 0, + text: 0, + style: 0, + }, + FuzzOp::InsertText { + peer: 0, + paragraph: 0, + byte: 0, + text: 0, + style: 0, + }, FuzzOp::Join { peer: 86, paragraph: 76 }, - FuzzOp::Style { peer: 144, paragraph: 122, start: 123, len: 147, style: 171 }, + FuzzOp::Style { + peer: 144, + paragraph: 122, + start: 123, + len: 147, + style: 171, + }, FuzzOp::DeleteObject { peer: 212, row: 116 }, - FuzzOp::Split { peer: 44, paragraph: 85, byte: 123 }, - FuzzOp::InsertText { peer: 47, paragraph: 61, byte: 160, text: 91, style: 192 }, + FuzzOp::Split { + peer: 44, + paragraph: 85, + byte: 123, + }, + FuzzOp::InsertText { + peer: 47, + paragraph: 61, + byte: 160, + text: 91, + style: 192, + }, FuzzOp::DeleteObject { peer: 216, row: 243 }, - FuzzOp::DeleteText { peer: 86, paragraph: 45, start: 114, len: 95 }, - FuzzOp::InsertObject { peer: 122, row: 21, source: 177 }, - FuzzOp::InsertText { peer: 44, paragraph: 23, byte: 38, text: 111, style: 124 }, + FuzzOp::DeleteText { + peer: 86, + paragraph: 45, + start: 114, + len: 95, + }, + FuzzOp::InsertObject { + peer: 122, + row: 21, + source: 177, + }, + FuzzOp::InsertText { + peer: 44, + paragraph: 23, + byte: 38, + text: 111, + style: 124, + }, FuzzOp::MoveObject { peer: 1, row: 223, to: 58 }, - FuzzOp::InsertObject { peer: 167, row: 184, source: 207 }, - FuzzOp::InsertText { peer: 199, paragraph: 128, byte: 125, text: 41, style: 165 }, - FuzzOp::MoveObject { peer: 211, row: 103, to: 135 }, + FuzzOp::InsertObject { + peer: 167, + row: 184, + source: 207, + }, + FuzzOp::InsertText { + peer: 199, + paragraph: 128, + byte: 125, + text: 41, + style: 165, + }, + FuzzOp::MoveObject { + peer: 211, + row: 103, + to: 135, + }, ]; run_program(2, 9640017939277306283, &ops).expect("concurrent object move interleaving should converge"); @@ -226,25 +340,117 @@ mod tests { #[test] fn buffered_object_split_join_move_program_converges() { let ops = [ - FuzzOp::Style { peer: 34, paragraph: 235, start: 63, len: 151, style: 35 }, - FuzzOp::DeleteText { peer: 10, paragraph: 205, start: 201, len: 24 }, - FuzzOp::Style { peer: 244, paragraph: 167, start: 13, len: 60, style: 35 }, - FuzzOp::InsertText { peer: 159, paragraph: 175, byte: 242, text: 64, style: 7 }, - FuzzOp::Style { peer: 86, paragraph: 113, start: 237, len: 135, style: 240 }, - FuzzOp::DeleteText { peer: 186, paragraph: 150, start: 45, len: 253 }, - FuzzOp::InsertText { peer: 236, paragraph: 70, byte: 124, text: 228, style: 238 }, - FuzzOp::Style { peer: 212, paragraph: 57, start: 248, len: 168, style: 47 }, - FuzzOp::Style { peer: 128, paragraph: 69, start: 13, len: 201, style: 209 }, - FuzzOp::InsertObject { peer: 208, row: 159, source: 38 }, - FuzzOp::DeleteText { peer: 70, paragraph: 2, start: 247, len: 227 }, - FuzzOp::DeleteText { peer: 45, paragraph: 59, start: 175, len: 74 }, - FuzzOp::InsertText { peer: 234, paragraph: 209, byte: 216, text: 230, style: 207 }, - FuzzOp::DeleteText { peer: 197, paragraph: 38, start: 169, len: 78 }, - FuzzOp::InsertText { peer: 118, paragraph: 146, byte: 65, text: 27, style: 176 }, - FuzzOp::Split { peer: 32, paragraph: 124, byte: 11 }, + FuzzOp::Style { + peer: 34, + paragraph: 235, + start: 63, + len: 151, + style: 35, + }, + FuzzOp::DeleteText { + peer: 10, + paragraph: 205, + start: 201, + len: 24, + }, + FuzzOp::Style { + peer: 244, + paragraph: 167, + start: 13, + len: 60, + style: 35, + }, + FuzzOp::InsertText { + peer: 159, + paragraph: 175, + byte: 242, + text: 64, + style: 7, + }, + FuzzOp::Style { + peer: 86, + paragraph: 113, + start: 237, + len: 135, + style: 240, + }, + FuzzOp::DeleteText { + peer: 186, + paragraph: 150, + start: 45, + len: 253, + }, + FuzzOp::InsertText { + peer: 236, + paragraph: 70, + byte: 124, + text: 228, + style: 238, + }, + FuzzOp::Style { + peer: 212, + paragraph: 57, + start: 248, + len: 168, + style: 47, + }, + FuzzOp::Style { + peer: 128, + paragraph: 69, + start: 13, + len: 201, + style: 209, + }, + FuzzOp::InsertObject { + peer: 208, + row: 159, + source: 38, + }, + FuzzOp::DeleteText { + peer: 70, + paragraph: 2, + start: 247, + len: 227, + }, + FuzzOp::DeleteText { + peer: 45, + paragraph: 59, + start: 175, + len: 74, + }, + FuzzOp::InsertText { + peer: 234, + paragraph: 209, + byte: 216, + text: 230, + style: 207, + }, + FuzzOp::DeleteText { + peer: 197, + paragraph: 38, + start: 169, + len: 78, + }, + FuzzOp::InsertText { + peer: 118, + paragraph: 146, + byte: 65, + text: 27, + style: 176, + }, + FuzzOp::Split { + peer: 32, + paragraph: 124, + byte: 11, + }, FuzzOp::Join { peer: 19, paragraph: 41 }, FuzzOp::MoveObject { peer: 94, row: 196, to: 197 }, - FuzzOp::DeleteText { peer: 56, paragraph: 171, start: 218, len: 154 }, + FuzzOp::DeleteText { + peer: 56, + paragraph: 171, + start: 218, + len: 154, + }, FuzzOp::DeleteObject { peer: 222, row: 98 }, ]; @@ -253,14 +459,49 @@ mod tests { #[derive(Clone, Debug)] enum FuzzOp { - InsertText { peer: u8, paragraph: u8, byte: u8, text: u8, style: u8 }, - DeleteText { peer: u8, paragraph: u8, start: u8, len: u8 }, - Split { peer: u8, paragraph: u8, byte: u8 }, - Join { peer: u8, paragraph: u8 }, - Style { peer: u8, paragraph: u8, start: u8, len: u8, style: u8 }, - InsertObject { peer: u8, row: u8, source: u8 }, - DeleteObject { peer: u8, row: u8 }, - MoveObject { peer: u8, row: u8, to: u8 }, + InsertText { + peer: u8, + paragraph: u8, + byte: u8, + text: u8, + style: u8, + }, + DeleteText { + peer: u8, + paragraph: u8, + start: u8, + len: u8, + }, + Split { + peer: u8, + paragraph: u8, + byte: u8, + }, + Join { + peer: u8, + paragraph: u8, + }, + Style { + peer: u8, + paragraph: u8, + start: u8, + len: u8, + style: u8, + }, + InsertObject { + peer: u8, + row: u8, + source: u8, + }, + DeleteObject { + peer: u8, + row: u8, + }, + MoveObject { + peer: u8, + row: u8, + to: u8, + }, } impl FuzzOp { @@ -293,13 +534,7 @@ mod tests { fn run_program(peer_count: usize, delivery_seed: u64, ops: &[FuzzOp]) -> Result<()> { let session = SessionId::from_bytes([77; 32]); - let initial = document_from_input_blocks( - DocumentTheme::default(), - vec![ - paragraph_block(MULTIBYTE), - paragraph_block("second aé🌍"), - ], - ); + let initial = document_from_input_blocks(DocumentTheme::default(), vec![paragraph_block(MULTIBYTE), paragraph_block("second aé🌍")]); let mut peers = Vec::with_capacity(peer_count); let first = Peer::from_initial(session, initial)?; let snapshot = first.snapshot()?; @@ -322,9 +557,12 @@ mod tests { let copies = 1 + usize::from(pseudo(delivery_seed, step, source, target, 0).is_multiple_of(5)); for copy in 0..copies { let buffered = pseudo(delivery_seed, step, source, target, copy).is_multiple_of(7); - let key = pseudo(delivery_seed, step, source, target, copy + 17) - .saturating_add(if buffered { 10_000 + step as u64 } else { 0 }); - network.push(Message { key, target, bytes: update.clone() }); + let key = pseudo(delivery_seed, step, source, target, copy + 17).saturating_add(if buffered { 10_000 + step as u64 } else { 0 }); + network.push(Message { + key, + target, + bytes: update.clone(), + }); } } } @@ -391,10 +629,19 @@ mod tests { let updates = Arc::new(Mutex::new(Vec::new())); let captured = updates.clone(); let subscription = loro.subscribe_local_update(Box::new(move |bytes| { - captured.lock().expect("update capture lock should not be poisoned").push(bytes.clone()); + captured + .lock() + .expect("update capture lock should not be poisoned") + .push(bytes.clone()); true })); - Self { document, loro, binding, updates, _subscription: subscription } + Self { + document, + loro, + binding, + updates, + _subscription: subscription, + } } fn snapshot(&self) -> Result> { @@ -402,7 +649,10 @@ mod tests { } fn drain_updates(&self) -> Vec> { - let mut updates = self.updates.lock().expect("update capture lock should not be poisoned"); + let mut updates = self + .updates + .lock() + .expect("update capture lock should not be poisoned"); std::mem::take(&mut *updates) } @@ -410,14 +660,23 @@ mod tests { let Some(ops) = self.mutate_document(op)? else { return Ok(false); }; - LocalApplier { doc: &self.loro, binding: &mut self.binding } - .apply(&self.document, &ops)?; + LocalApplier { + doc: &self.loro, + binding: &mut self.binding, + } + .apply(&self.document, &ops)?; Ok(true) } fn mutate_document(&mut self, op: &FuzzOp) -> Result>> { match *op { - FuzzOp::InsertText { paragraph, byte, text, style, .. } => { + FuzzOp::InsertText { + paragraph, + byte, + text, + style, + .. + } => { if self.document.paragraphs.is_empty() { return Ok(None); } @@ -479,11 +738,23 @@ mod tests { let first_len = paragraph_text(&self.document, first_ix).len(); delete_cross_paragraph_range( &mut self.document, - DocumentOffset { paragraph: first_ix, byte: first_len }..DocumentOffset { paragraph: first_ix + 1, byte: 0 }, + DocumentOffset { + paragraph: first_ix, + byte: first_len, + }..DocumentOffset { + paragraph: first_ix + 1, + byte: 0, + }, ); Ok(Some(vec![CanonicalOperation::JoinParagraphs { first, second }])) }, - FuzzOp::Style { paragraph, start, len, style, .. } => { + FuzzOp::Style { + paragraph, + start, + len, + style, + .. + } => { if self.document.paragraphs.is_empty() { return Ok(None); } @@ -496,10 +767,20 @@ mod tests { let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; mutate_runs_in_range( &mut self.document, - DocumentOffset { paragraph: paragraph_ix, byte: range.start }..DocumentOffset { paragraph: paragraph_ix, byte: range.end }, + DocumentOffset { + paragraph: paragraph_ix, + byte: range.start, + }..DocumentOffset { + paragraph: paragraph_ix, + byte: range.end, + }, |run_styles| *run_styles = styles, ); - Ok(Some(vec![CanonicalOperation::SetRunStyles { paragraph: paragraph_id, range, styles }])) + Ok(Some(vec![CanonicalOperation::SetRunStyles { + paragraph: paragraph_id, + range, + styles, + }])) }, FuzzOp::InsertObject { row, source, .. } => { if self.document.blocks.len() >= 8 { @@ -537,7 +818,11 @@ mod tests { let insert_ix = to.min(self.document.blocks.len()); Arc::make_mut(&mut self.document.blocks).insert(insert_ix, moved_block); let block_id = self.document.ids.block_ids.remove(from); - self.document.ids.block_ids.insert(insert_ix.min(self.document.ids.block_ids.len()), block_id); + self + .document + .ids + .block_ids + .insert(insert_ix.min(self.document.ids.block_ids.len()), block_id); Ok(Some(vec![CanonicalOperation::MoveBlock { block, new_block_ix: to }])) }, } @@ -558,11 +843,17 @@ mod tests { .lock() .map_err(|lock_error| anyhow!("binding lock poisoned: {lock_error}")) .and_then(|mut binding| { - flowstate_collab::remote_apply::RemoteApplier { doc: &doc, binding: &mut binding } - .apply_event(&document, &event) + flowstate_collab::remote_apply::RemoteApplier { + doc: &doc, + binding: &mut binding, + } + .apply_event(&document, &event) }); match result { - Ok(mut produced) => patches.lock().expect("patch lock should not be poisoned").append(&mut produced), + Ok(mut produced) => patches + .lock() + .expect("patch lock should not be poisoned") + .append(&mut produced), Err(apply_error) => *error.lock().expect("error lock should not be poisoned") = Some(format!("{apply_error:#}")), } } @@ -575,7 +866,10 @@ mod tests { .into_inner() .map_err(|error| anyhow!("binding lock poisoned: {error}"))?; import_result.context("importing remote update failed")?; - let remote_error = error.lock().expect("error lock should not be poisoned").take(); + let remote_error = error + .lock() + .expect("error lock should not be poisoned") + .take(); if let Some(error) = remote_error { bail!("remote apply failed: {error}"); } @@ -589,7 +883,10 @@ mod tests { } fn paragraph_block(text: &str) -> InputBlock { - InputBlock::Paragraph(InputParagraph { style: ParagraphStyle::Normal, runs: vec![plain(text)] }) + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(text)], + }) } fn equation_block(source: &str) -> InputBlock { @@ -674,10 +971,18 @@ mod tests { } fn block_order_snapshot(document: &Document, paragraph_ix: usize) -> Option<(usize, Vec, Vec)> { - Some((block_ix_for_paragraph_ix(document, paragraph_ix)?, document.blocks.as_ref().clone(), document.ids.block_ids.clone())) + Some(( + block_ix_for_paragraph_ix(document, paragraph_ix)?, + document.blocks.as_ref().clone(), + document.ids.block_ids.clone(), + )) } - fn restore_block_order(document: &mut Document, paragraph_ix: usize, (block_ix, mut blocks, block_ids): (usize, Vec, Vec)) { + fn restore_block_order( + document: &mut Document, + paragraph_ix: usize, + (block_ix, mut blocks, block_ids): (usize, Vec, Vec), + ) { if let Some(paragraph) = document.paragraphs.get(paragraph_ix).cloned() && let Some(block) = blocks.get_mut(block_ix) { @@ -727,7 +1032,9 @@ mod tests { 1 => RunStyles::default().with(RunStyle::Semantic(1)), 2 => RunStyles::default().with(RunStyle::Highlight(2)), 3 => RunStyles::default().with_direct_underline(), - _ => RunStyles::default().with(RunStyle::Semantic(3)).with_strikethrough(), + _ => RunStyles::default() + .with(RunStyle::Semantic(3)) + .with_strikethrough(), } } @@ -756,11 +1063,8 @@ mod tests { } fn pseudo(seed: u64, step: usize, source: usize, target: usize, copy: usize) -> u64 { - let mut value = seed - ^ ((step as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15)) - ^ ((source as u64) << 17) - ^ ((target as u64) << 33) - ^ ((copy as u64) << 49); + let mut value = + seed ^ ((step as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15)) ^ ((source as u64) << 17) ^ ((target as u64) << 33) ^ ((copy as u64) << 49); value ^= value >> 30; value = value.wrapping_mul(0xBF58_476D_1CE4_E5B9); value ^= value >> 27; diff --git a/crates/flowstate-collab/tests/perf_smoke.rs b/crates/flowstate-collab/tests/perf_smoke.rs index fb053f13..50ec01f0 100644 --- a/crates/flowstate-collab/tests/perf_smoke.rs +++ b/crates/flowstate-collab/tests/perf_smoke.rs @@ -8,13 +8,7 @@ mod tests { use std::sync::{Arc, Mutex}; - use flowstate_collab::{ - SessionId, - binding::DocBinding, - local_apply::LocalApplier, - patch_apply::apply_patches, - projection, schema, - }; + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, patch_apply::apply_patches, projection, schema}; use gpui_flowtext::{ CanonicalOperation, CollabPatch, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyles, capture_document_span, document_from_input_blocks, insert_text_at, paragraph_text, plain, @@ -51,11 +45,19 @@ mod tests { after, }; - LocalApplier { doc: &loro, binding: &mut binding } - .apply(&edited, &[op]) - .expect("local apply"); - - let emitted: usize = updates.lock().expect("update lock").iter().map(Vec::len).sum(); + LocalApplier { + doc: &loro, + binding: &mut binding, + } + .apply(&edited, &[op]) + .expect("local apply"); + + let emitted: usize = updates + .lock() + .expect("update lock") + .iter() + .map(Vec::len) + .sum(); let body_bytes = full_body_len(&document); // The whole-document body is large; a single-character incremental splice must // stay far below it (a full-body rewrite would be >= body_bytes). @@ -100,13 +102,19 @@ mod tests { let before = capture_document_span(&edited, target..target + 1); insert_text_at(&mut edited, target, 0, "X", RunStyles::default()); let after = capture_document_span(&edited, target..target + 1); - LocalApplier { doc: &source_loro, binding: &mut source_binding } - .apply(&edited, &[CanonicalOperation::ReplaceParagraphSpan { + LocalApplier { + doc: &source_loro, + binding: &mut source_binding, + } + .apply( + &edited, + &[CanonicalOperation::ReplaceParagraphSpan { start_paragraph: Some(edited.ids.paragraph_ids[target]), before, after, - }]) - .expect("local apply"); + }], + ) + .expect("local apply"); let update_batches = std::mem::take(&mut *updates.lock().expect("update lock")); let patches = import_updates(&target_loro, &mut target_document, &mut target_binding, &update_batches); @@ -127,13 +135,20 @@ mod tests { fn large_document() -> Document { let blocks = (0..PARAGRAPHS) - .map(|ix| InputBlock::Paragraph(InputParagraph { style: ParagraphStyle::Normal, runs: vec![plain(&format!("paragraph number {ix} body text"))] })) + .map(|ix| { + InputBlock::Paragraph(InputParagraph { + style: ParagraphStyle::Normal, + runs: vec![plain(&format!("paragraph number {ix} body text"))], + }) + }) .collect(); document_from_input_blocks(DocumentTheme::default(), blocks) } fn full_body_len(document: &Document) -> usize { - (0..document.paragraphs.len()).map(|ix| paragraph_text(document, ix).len() + 1).sum() + (0..document.paragraphs.len()) + .map(|ix| paragraph_text(document, ix).len() + 1) + .sum() } fn import_updates(loro: &LoroDoc, document: &mut Document, binding: &mut DocBinding, updates: &[Vec]) -> Vec { @@ -149,9 +164,12 @@ mod tests { move |event| { let produced = { let mut guard = shared_binding.lock().expect("binding lock"); - flowstate_collab::remote_apply::RemoteApplier { doc: &doc, binding: &mut guard } - .apply_event(&snapshot_document, &event) - .expect("remote apply") + flowstate_collab::remote_apply::RemoteApplier { + doc: &doc, + binding: &mut guard, + } + .apply_event(&snapshot_document, &event) + .expect("remote apply") }; patches.lock().expect("patch lock").extend(produced); } @@ -159,8 +177,14 @@ mod tests { let subscription = loro.subscribe_root(callback); loro.import_with(update, "remote").expect("import"); drop(subscription); - *binding = Arc::try_unwrap(shared_binding).expect("binding owner").into_inner().expect("binding lock"); - let produced = Arc::try_unwrap(patches).expect("patch owner").into_inner().expect("patch lock"); + *binding = Arc::try_unwrap(shared_binding) + .expect("binding owner") + .into_inner() + .expect("binding lock"); + let produced = Arc::try_unwrap(patches) + .expect("patch owner") + .into_inner() + .expect("patch lock"); apply_patches(document, binding, loro, &produced).expect("apply patches"); all.extend(produced); } diff --git a/crates/flowstate-collab/tests/remote_apply.rs b/crates/flowstate-collab/tests/remote_apply.rs index f4093963..9d8aec2c 100644 --- a/crates/flowstate-collab/tests/remote_apply.rs +++ b/crates/flowstate-collab/tests/remote_apply.rs @@ -2,13 +2,7 @@ mod tests { use std::sync::{Arc, Mutex}; - use flowstate_collab::{ - SessionId, - binding::DocBinding, - local_apply::LocalApplier, - patch_apply::apply_patches, - projection, schema, - }; + use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, patch_apply::apply_patches, projection, schema}; use gpui_flowtext::{ Block, CanonicalOperation, CollabPatch, Document, DocumentOffset, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, RunStyles, block_from_input_block, document_from_input_blocks, @@ -49,7 +43,13 @@ mod tests { style: ParagraphStyle::Custom(7), }]); let paragraph_style_patches = target.import_updates(¶graph_style_updates); - assert!(matches!(paragraph_style_patches.as_slice(), [CollabPatch::ParagraphStyle { row: 0, style: ParagraphStyle::Custom(7) }])); + assert!(matches!( + paragraph_style_patches.as_slice(), + [CollabPatch::ParagraphStyle { + row: 0, + style: ParagraphStyle::Custom(7) + }] + )); let range = "a".len().."aé🌍".len(); let styles = RunStyles::default() @@ -57,7 +57,13 @@ mod tests { .with_direct_underline(); mutate_runs_in_range( &mut source.document, - DocumentOffset { paragraph: 0, byte: range.start }..DocumentOffset { paragraph: 0, byte: range.end }, + DocumentOffset { + paragraph: 0, + byte: range.start, + }..DocumentOffset { + paragraph: 0, + byte: range.end, + }, |run_styles| *run_styles = styles, ); let run_style_updates = source.apply(&[CanonicalOperation::SetRunStyles { paragraph, range, styles }]); @@ -100,7 +106,11 @@ mod tests { new_block_ix: 2, }]); let move_patches = target.import_updates(&move_updates); - assert!(move_patches.iter().any(|patch| matches!(patch, CollabPatch::MoveBlock { from: 1, to: 2 }))); + assert!( + move_patches + .iter() + .any(|patch| matches!(patch, CollabPatch::MoveBlock { from: 1, to: 2 })) + ); assert_documents_match(&source.document, &target.document, &target.loro); delete_object_block(&mut source.document, 2); @@ -132,7 +142,10 @@ mod tests { } .apply(&self.document, ops) .expect("local apply should succeed"); - let mut updates = self.updates.lock().expect("update lock should not be poisoned"); + let mut updates = self + .updates + .lock() + .expect("update lock should not be poisoned"); std::mem::take(&mut *updates) } } @@ -168,7 +181,10 @@ mod tests { .apply_event(&document, &event) }); match result { - Ok(mut produced) => patches.lock().expect("patch lock should not be poisoned").append(&mut produced), + Ok(mut produced) => patches + .lock() + .expect("patch lock should not be poisoned") + .append(&mut produced), Err(apply_error) => *error.lock().expect("error lock should not be poisoned") = Some(format!("{apply_error:#}")), } } @@ -181,7 +197,10 @@ mod tests { .into_inner() .expect("binding lock should not be poisoned"); import_result.expect("remote import should succeed"); - let remote_error = error.lock().expect("error lock should not be poisoned").take(); + let remote_error = error + .lock() + .expect("error lock should not be poisoned") + .take(); if let Some(error) = remote_error { panic!("remote apply failed: {error}"); } @@ -203,13 +222,20 @@ mod tests { let updates = Arc::new(Mutex::new(Vec::new())); let captured = updates.clone(); let subscription = loro.subscribe_local_update(Box::new(move |bytes| { - captured.lock().expect("update lock should not be poisoned").push(bytes.clone()); + captured + .lock() + .expect("update lock should not be poisoned") + .push(bytes.clone()); true })); - let snapshot = loro.export(ExportMode::Snapshot).expect("snapshot export should succeed"); + let snapshot = loro + .export(ExportMode::Snapshot) + .expect("snapshot export should succeed"); let target_loro = schema::new_configured_doc(); - target_loro.import(&snapshot).expect("snapshot import should succeed"); + target_loro + .import(&snapshot) + .expect("snapshot import should succeed"); projection::verify_lineage(&target_loro, session).expect("lineage should match"); let target_document = projection::document_from_loro(&target_loro, DocumentTheme::default()).expect("projection should succeed"); let target_binding = DocBinding::build(&target_loro, &target_document).expect("binding should build"); @@ -263,7 +289,10 @@ mod tests { let insert_ix = to.min(document.blocks.len()); Arc::make_mut(&mut document.blocks).insert(insert_ix, block); let block_id = document.ids.block_ids.remove(from); - document.ids.block_ids.insert(to.min(document.ids.block_ids.len()), block_id); + document + .ids + .block_ids + .insert(to.min(document.ids.block_ids.len()), block_id); } fn delete_object_block(document: &mut Document, row: usize) { @@ -312,5 +341,4 @@ mod tests { } } } - } diff --git a/crates/flowstate-collab/tests/swarm_loopback.rs b/crates/flowstate-collab/tests/swarm_loopback.rs index 439bbfb0..2b05bf59 100644 --- a/crates/flowstate-collab/tests/swarm_loopback.rs +++ b/crates/flowstate-collab/tests/swarm_loopback.rs @@ -125,19 +125,24 @@ async fn swarm_loopback_net_subset() -> Result<()> { recv_neighbor(&c.events).await?; let small_update = b"a-to-b-to-c".to_vec(); - a.publish(PublishPayload::Update(small_update.clone())).await?; + a.publish(PublishPayload::Update(small_update.clone())) + .await?; recv_update(&b.events, &small_update).await?; recv_update(&c.events, &small_update).await?; - a.publish(PublishPayload::Presence(b"presence-a".to_vec())).await?; - b.publish(PublishPayload::Presence(b"presence-b".to_vec())).await?; - c.publish(PublishPayload::Presence(b"presence-c".to_vec())).await?; + a.publish(PublishPayload::Presence(b"presence-a".to_vec())) + .await?; + b.publish(PublishPayload::Presence(b"presence-b".to_vec())) + .await?; + c.publish(PublishPayload::Presence(b"presence-c".to_vec())) + .await?; recv_presence(&a.events).await?; recv_presence(&b.events).await?; recv_presence(&c.events).await?; let large_update = vec![7; GOSSIP_INLINE_LIMIT + 1]; - a.publish(PublishPayload::Update(large_update.clone())).await?; + a.publish(PublishPayload::Update(large_update.clone())) + .await?; let blob = recv_update_available(&b.events, large_update.len() as u64).await?; let blob_bytes = direct::pull_with_endpoint( &b.endpoint, @@ -158,12 +163,24 @@ async fn swarm_loopback_net_subset() -> Result<()> { c.stop_swarm().await; c.start_swarm(session, vec![b_addr])?; - b.publish(PublishPayload::Digest { vv: b"digest-after-resubscribe".to_vec() }).await?; + b.publish(PublishPayload::Digest { + vv: b"digest-after-resubscribe".to_vec(), + }) + .await?; recv_digest(&c.events).await?; - a.router.shutdown().await.context("shutting down A router failed")?; - b.router.shutdown().await.context("shutting down B router failed")?; - c.router.shutdown().await.context("shutting down C router failed")?; + a.router + .shutdown() + .await + .context("shutting down A router failed")?; + b.router + .shutdown() + .await + .context("shutting down B router failed")?; + c.router + .shutdown() + .await + .context("shutting down C router failed")?; Ok(()) } @@ -195,20 +212,30 @@ async fn wait_addr(endpoint: &Endpoint) -> EndpointAddr { } } }; - timeout(Duration::from_secs(3), wait).await.unwrap_or_else(|_| endpoint.addr()) + timeout(Duration::from_secs(3), wait) + .await + .unwrap_or_else(|_| endpoint.addr()) } async fn recv_update(events: &Receiver, expected: &[u8]) -> Result<()> { - recv_matching(events, "update", |event| { - matches!(event, NetEvent::Gossip { msg: GossipMsg::Update(bytes), .. } if bytes == expected) - }) + recv_matching( + events, + "update", + |event| matches!(event, NetEvent::Gossip { msg: GossipMsg::Update(bytes), .. } if bytes == expected), + ) .await .map(|_| ()) } async fn recv_presence(events: &Receiver) -> Result<()> { recv_matching(events, "presence", |event| { - matches!(event, NetEvent::Gossip { msg: GossipMsg::Presence(_), .. }) + matches!( + event, + NetEvent::Gossip { + msg: GossipMsg::Presence(_), + .. + } + ) }) .await .map(|_| ()) @@ -221,9 +248,11 @@ async fn recv_neighbor(events: &Receiver) -> Result<()> { } async fn recv_update_available(events: &Receiver, expected_len: u64) -> Result { - let event = recv_matching(events, "update-available", |event| { - matches!(event, NetEvent::Gossip { msg: GossipMsg::UpdateAvailable { len, .. }, .. } if *len == expected_len) - }) + let event = recv_matching( + events, + "update-available", + |event| matches!(event, NetEvent::Gossip { msg: GossipMsg::UpdateAvailable { len, .. }, .. } if *len == expected_len), + ) .await?; let NetEvent::Gossip { msg: GossipMsg::UpdateAvailable { blob, .. }, @@ -237,7 +266,13 @@ async fn recv_update_available(events: &Receiver, expected_len: u64) - async fn recv_digest(events: &Receiver) -> Result<()> { recv_matching(events, "digest", |event| { - matches!(event, NetEvent::Gossip { msg: GossipMsg::Digest { .. }, .. }) + matches!( + event, + NetEvent::Gossip { + msg: GossipMsg::Digest { .. }, + .. + } + ) }) .await .map(|_| ()) diff --git a/crates/flowstate/src/collab/asset_transfer.rs b/crates/flowstate/src/collab/asset_transfer.rs index 915e200f..8b97af7d 100644 --- a/crates/flowstate/src/collab/asset_transfer.rs +++ b/crates/flowstate/src/collab/asset_transfer.rs @@ -96,15 +96,12 @@ fn start_asset_pull( let id = AssetId(meta.asset_id); let session_id = session.session; let candidate_count = candidates.len(); - if let Err(error) = session - .net_tx - .try_send(NetCommand::PullAsset { - session: session.session, - candidates, - asset: meta.asset_id, - reply: reply_tx, - }) - { + if let Err(error) = session.net_tx.try_send(NetCommand::PullAsset { + session: session.session, + candidates, + asset: meta.asset_id, + reply: reply_tx, + }) { tracing::warn!(session = %session_id, ?id, candidate_count, error = %error, "queueing collaboration asset pull failed"); session.asset_pulls_in_flight.remove(&id); return; @@ -152,7 +149,12 @@ impl ImageAssetMeta { } fn record_from_bytes(&self, bytes: Vec) -> Result { - tracing::trace!(asset = self.asset_id, expected_bytes = self.byte_len, received_bytes = bytes.len(), "validating fetched collaboration asset bytes"); + tracing::trace!( + asset = self.asset_id, + expected_bytes = self.byte_len, + received_bytes = bytes.len(), + "validating fetched collaboration asset bytes" + ); ensure!(bytes.len() as u64 == self.byte_len, "asset byte length mismatch"); let content_hash = AssetRecord::stable_content_hash(&bytes); ensure!(content_hash == self.content_hash, "asset content hash mismatch"); diff --git a/crates/flowstate/src/collab/manager.rs b/crates/flowstate/src/collab/manager.rs index 402be768..68786c71 100644 --- a/crates/flowstate/src/collab/manager.rs +++ b/crates/flowstate/src/collab/manager.rs @@ -49,7 +49,11 @@ impl CollabManager { } pub(super) fn shutdown_runtime(&mut self) { - tracing::info!(open_sessions = self.sessions_by_id.len(), runtime_running = self.runtime.is_some(), "shutting down collaboration runtime"); + tracing::info!( + open_sessions = self.sessions_by_id.len(), + runtime_running = self.runtime.is_some(), + "shutting down collaboration runtime" + ); if let Some(runtime) = self.runtime.take() && let Err(error) = runtime.commands.try_send(NetCommand::Shutdown) { @@ -101,7 +105,11 @@ impl CollabManager { handler: direct_handler, }) { tracing::error!(%panel_id, %session, error = %error, "registering collaboration direct handler failed"); - Self::detach_entity(&entity, DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), cx); + Self::detach_entity( + &entity, + DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), + cx, + ); self.unregister_session(session); return Err(error).context("registering collaboration direct handler failed"); } @@ -110,7 +118,11 @@ impl CollabManager { let (reply_tx, reply_rx) = async_channel::bounded(1); if let Err(error) = commands.try_send(NetCommand::CreateSession { session, reply: reply_tx }) { tracing::error!(%panel_id, %session, error = %error, "queueing collaboration create-session command failed"); - Self::detach_entity(&entity, DetachReason::Fatal(format!("creating collaboration network session failed: {error}")), cx); + Self::detach_entity( + &entity, + DetachReason::Fatal(format!("creating collaboration network session failed: {error}")), + cx, + ); self.unregister_session(session); return Err(error).context("creating collaboration network session failed"); } @@ -194,18 +206,30 @@ impl CollabManager { handler: direct_handler, }) { tracing::error!(%session_id, %panel_id, error = %error, "registering collaboration direct handler for joined session failed"); - Self::detach_entity(&session, DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), cx); + Self::detach_entity( + &session, + DetachReason::Fatal(format!("registering collaboration direct handler failed: {error}")), + cx, + ); self.unregister_session(session_id); return Err(error).context("registering collaboration direct handler failed"); } tracing::debug!(%session_id, %panel_id, "registered collaboration direct handler for joined session"); if let Err(error) = Self::establish_joined_peer(session.clone(), commands, cx) { - Self::detach_entity(&session, DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error:#}")), cx); + Self::detach_entity( + &session, + DetachReason::Fatal(format!("collaboration endpoint address unavailable: {error:#}")), + cx, + ); self.unregister_session(session_id); return Err(error); } if let Err(error) = session.update(cx, |session, cx| session.attach_joined_editor(panel_id, editor, cx)) { - Self::detach_entity(&session, DetachReason::Fatal(format!("attaching joined collaboration editor failed: {error:#}")), cx); + Self::detach_entity( + &session, + DetachReason::Fatal(format!("attaching joined collaboration editor failed: {error:#}")), + cx, + ); self.unregister_session(session_id); return Err(error); } diff --git a/crates/flowstate/src/collab/mod.rs b/crates/flowstate/src/collab/mod.rs index 3ef3cf06..4b8092fa 100644 --- a/crates/flowstate/src/collab/mod.rs +++ b/crates/flowstate/src/collab/mod.rs @@ -1,7 +1,7 @@ mod manager; -mod pump; -mod presence_view; pub(crate) mod notify; +mod presence_view; +mod pump; mod session; pub mod share_dialog; mod shutdown; diff --git a/crates/flowstate/src/collab/notify.rs b/crates/flowstate/src/collab/notify.rs index c7ef9c32..da604276 100644 --- a/crates/flowstate/src/collab/notify.rs +++ b/crates/flowstate/src/collab/notify.rs @@ -1,8 +1,5 @@ use gpui::{App, PromptButton, PromptLevel, Window}; -use gpui_component::{ - WindowExt as _, - notification::Notification, -}; +use gpui_component::{WindowExt as _, notification::Notification}; use super::SessionNotice; @@ -24,11 +21,7 @@ pub fn show_session_notice(notice: &SessionNotice, window: &mut Window, cx: &mut #[cfg(debug_assertions)] push_info("Collaboration view rebuilt from remote state.", window, cx); }, - SessionNotice::IncompatibleVersion(peer) => push_warning( - format!("{peer} is using an incompatible collaboration version."), - window, - cx, - ), + SessionNotice::IncompatibleVersion(peer) => push_warning(format!("{peer} is using an incompatible collaboration version."), window, cx), } } diff --git a/crates/flowstate/src/collab/presence_view.rs b/crates/flowstate/src/collab/presence_view.rs index 0a47fa3a..e81692c5 100644 --- a/crates/flowstate/src/collab/presence_view.rs +++ b/crates/flowstate/src/collab/presence_view.rs @@ -7,8 +7,7 @@ use flowstate_collab::{ schema::body_text, }; use loro::{ - ContainerTrait as _, - LoroDoc, + ContainerTrait as _, LoroDoc, cursor::{Cursor, PosType, Side}, }; @@ -56,12 +55,7 @@ fn cursor_bytes_for_offset(doc: &LoroDoc, document: &Document, offset: DocumentO .map(|cursor| cursor.encode()) } -fn external_caret_for_presence( - doc: &LoroDoc, - document: &Document, - selection: &PresenceSelection, - color_rgb: u32, -) -> Option { +fn external_caret_for_presence(doc: &LoroDoc, document: &Document, selection: &PresenceSelection, color_rgb: u32) -> Option { let cursor = Cursor::decode(&selection.head).ok()?; let text = body_text(doc); if cursor.container != text.id() { diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index 8926e7e5..d7bbfe4b 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -1,4 +1,8 @@ -use std::{collections::HashSet, rc::Rc, time::{Duration, Instant}}; +use std::{ + collections::HashSet, + rc::Rc, + time::{Duration, Instant}, +}; use anyhow::{Context as _, Result, anyhow, bail}; use flowstate_collab::{ @@ -680,7 +684,10 @@ impl CollabSession { edits }); let edit_count = edits.len(); - let operation_count = edits.iter().map(|edit| edit.operations.len()).sum::(); + let operation_count = edits + .iter() + .map(|edit| edit.operations.len()) + .sum::(); if edit_count == 0 || operation_count == 0 { tracing::trace!(session = %self.session, edit_count, operation_count, "no local collaboration edits to flush"); return Ok(()); diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs index 1f4efbe2..085cdd70 100644 --- a/crates/flowstate/src/collab/session_io.rs +++ b/crates/flowstate/src/collab/session_io.rs @@ -136,33 +136,28 @@ impl CollabSession { let (reply_tx, reply_rx) = async_channel::bounded(1); let candidates = self.pull_candidates(Some(from)); tracing::debug!(session = %self.session, from = %from, ?blob, candidate_count = candidates.len(), "requesting collaboration update blob pull"); - if let Err(error) = self - .net_tx - .try_send(NetCommand::PullBlob { - session: self.session, - candidates, - blob, - reply: reply_tx, - }) - { + if let Err(error) = self.net_tx.try_send(NetCommand::PullBlob { + session: self.session, + candidates, + blob, + reply: reply_tx, + }) { tracing::warn!(session = %self.session, from = %from, ?blob, error = %error, "queueing collaboration blob pull failed"); return; } let session_id = self.session; cx.spawn(async move |session, cx| { let result = reply_rx.recv().await; - let _ = session.update(cx, |session, cx| { - match result { - Ok(Ok(bytes)) => { - tracing::debug!(session = %session_id, ?blob, bytes = bytes.len(), "collaboration blob pull succeeded"); - if let Err(error) = session.import_update_bytes(&bytes, cx) { - tracing::error!(session = %session_id, ?blob, error = %format_args!("{error:#}"), "importing pulled collaboration blob failed"); - session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); - } - }, - Ok(Err(error)) => tracing::warn!(session = %session_id, ?blob, error = %format_args!("{error:#}"), "collaboration blob pull failed"), - Err(error) => tracing::warn!(session = %session_id, ?blob, error = %error, "collaboration blob pull reply channel closed"), - } + let _ = session.update(cx, |session, cx| match result { + Ok(Ok(bytes)) => { + tracing::debug!(session = %session_id, ?blob, bytes = bytes.len(), "collaboration blob pull succeeded"); + if let Err(error) = session.import_update_bytes(&bytes, cx) { + tracing::error!(session = %session_id, ?blob, error = %format_args!("{error:#}"), "importing pulled collaboration blob failed"); + session.detach(DetachReason::Fatal(format!("pulling collaboration blob failed: {error:#}")), cx); + } + }, + Ok(Err(error)) => tracing::warn!(session = %session_id, ?blob, error = %format_args!("{error:#}"), "collaboration blob pull failed"), + Err(error) => tracing::warn!(session = %session_id, ?blob, error = %error, "collaboration blob pull reply channel closed"), }); }) .detach(); @@ -298,7 +293,9 @@ impl CollabSession { let result = self.asset_bytes(asset, cx); match &result { Ok(bytes) => tracing::debug!(session = %self.session, asset, bytes = bytes.bytes.len(), "served collaboration asset direct request"), - Err(error) => tracing::warn!(session = %self.session, asset, error = %format_args!("{error:#}"), "serving collaboration asset direct request failed"), + Err(error) => { + tracing::warn!(session = %self.session, asset, error = %format_args!("{error:#}"), "serving collaboration asset direct request failed") + }, } let _ = reply.try_send(result); }, diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs index 6a93c194..7aebb3dc 100644 --- a/crates/flowstate/src/collab/session_timers.rs +++ b/crates/flowstate/src/collab/session_timers.rs @@ -92,10 +92,7 @@ impl CollabSession { last_probe_failed = self.last_probe_failed, "evaluating collaboration connectivity", ); - if zero_long_enough - && quiet_enough - && (!self.endpoint_online || self.last_probe_failed || !self.start_connectivity_probe(cx)) - { + if zero_long_enough && quiet_enough && (!self.endpoint_online || self.last_probe_failed || !self.start_connectivity_probe(cx)) { self.mark_offline(now, cx); } } @@ -358,7 +355,10 @@ impl CollabSession { if !bytes.is_empty() && let Err(error) = session.import_update_bytes(&bytes, cx) { - session.detach(DetachReason::Fatal(format!("probing collaboration peer failed to apply updates: {error:#}")), cx); + session.detach( + DetachReason::Fatal(format!("probing collaboration peer failed to apply updates: {error:#}")), + cx, + ); return; } session.note_inbound_traffic(cx); diff --git a/crates/flowstate/src/collab/share_dialog.rs b/crates/flowstate/src/collab/share_dialog.rs index 3031946f..b055987f 100644 --- a/crates/flowstate/src/collab/share_dialog.rs +++ b/crates/flowstate/src/collab/share_dialog.rs @@ -1,7 +1,7 @@ use flowstate_collab::{SessionId, ticket::SessionTicket}; use gpui::{ - AnyElement, App, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, KeyDownEvent, ParentElement, Render, - SharedString, Subscription, WeakEntity, Window, div, prelude::*, px, + AnyElement, App, Context, Entity, FocusHandle, Focusable, InteractiveElement, IntoElement, KeyDownEvent, ParentElement, Render, SharedString, + Subscription, WeakEntity, Window, div, prelude::*, px, }; use gpui_component::{ Disableable, WindowExt as _, @@ -381,13 +381,17 @@ impl Render for CollabShareDialog { .child( h_flex() .gap_2() - .child(tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { - dialog.set_mode(CollabDialogMode::Share, cx); - }))) - .child(tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { - dialog.set_mode(CollabDialogMode::Join, cx); - dialog.focus(window, cx); - }))), + .child( + tab_button("collab-share-tab", "Share", self.mode == CollabDialogMode::Share).on_click(cx.listener(|dialog, _, _, cx| { + dialog.set_mode(CollabDialogMode::Share, cx); + })), + ) + .child( + tab_button("collab-join-tab", "Join", self.mode == CollabDialogMode::Join).on_click(cx.listener(|dialog, _, window, cx| { + dialog.set_mode(CollabDialogMode::Join, cx); + dialog.focus(window, cx); + })), + ), ) .child( div() diff --git a/crates/flowstate/src/collab/share_dialog_view.rs b/crates/flowstate/src/collab/share_dialog_view.rs index 4852d24d..93855b77 100644 --- a/crates/flowstate/src/collab/share_dialog_view.rs +++ b/crates/flowstate/src/collab/share_dialog_view.rs @@ -54,7 +54,9 @@ pub(super) fn ticket_box(ticket: Option, loading: bool, cx: &mut C .child(if loading { SharedString::from("Minting a fresh invite...") } else { - ticket.clone().unwrap_or_else(|| "Start sharing to mint an invite.".into()) + ticket + .clone() + .unwrap_or_else(|| "Start sharing to mint an invite.".into()) }), ) .child(copy_invite_button(ticket, loading, cx)) @@ -116,7 +118,12 @@ pub(super) fn roster_list(entries: Vec, cx: & .border_color(color.opacity(0.32)), ) .child(div().size(px(8.0)).rounded_full().bg(color)) - .child(div().text_sm().text_color(cx.theme().foreground).child(label)), + .child( + div() + .text_sm() + .text_color(cx.theme().foreground) + .child(label), + ), ); } list.into_any_element() @@ -147,12 +154,21 @@ fn copy_invite_button(ticket: Option, loading: bool, cx: &mut Cont h_flex() .items_center() .gap_1() - .child(Clipboard::new("copy-collaboration-invite").value(ticket).on_copied(move |_, _, cx| { - let _ = dialog.update(cx, |dialog, cx| { - dialog.copy_notice = Some("Invite copied to clipboard.".into()); - cx.notify(); - }); - })) - .child(div().text_xs().text_color(cx.theme().muted_foreground).child("Copy")) + .child( + Clipboard::new("copy-collaboration-invite") + .value(ticket) + .on_copied(move |_, _, cx| { + let _ = dialog.update(cx, |dialog, cx| { + dialog.copy_notice = Some("Invite copied to clipboard.".into()); + cx.notify(); + }); + }), + ) + .child( + div() + .text_xs() + .text_color(cx.theme().muted_foreground) + .child("Copy"), + ) .into_any_element() } diff --git a/crates/flowstate/src/logging.rs b/crates/flowstate/src/logging.rs index 887fa11d..3357ae84 100644 --- a/crates/flowstate/src/logging.rs +++ b/crates/flowstate/src/logging.rs @@ -1,4 +1,7 @@ -use std::{env, path::{Path, PathBuf}}; +use std::{ + env, + path::{Path, PathBuf}, +}; use anyhow::{Context as _, Result}; use tracing_appender::non_blocking::WorkerGuard; @@ -21,8 +24,7 @@ impl LoggingGuard { pub fn init() -> Result { let directory = log_directory(); - std::fs::create_dir_all(&directory) - .with_context(|| format!("creating log directory {} failed", directory.display()))?; + std::fs::create_dir_all(&directory).with_context(|| format!("creating log directory {} failed", directory.display()))?; let file_appender = tracing_appender::rolling::daily(&directory, "flowstate.log"); let (writer, guard) = tracing_appender::non_blocking(file_appender); @@ -41,10 +43,7 @@ pub fn init() -> Result { .try_init() .context("initializing flowstate logging failed")?; - Ok(LoggingGuard { - _guard: guard, - directory, - }) + Ok(LoggingGuard { _guard: guard, directory }) } fn env_filter() -> EnvFilter { diff --git a/crates/flowstate/src/main.rs b/crates/flowstate/src/main.rs index d6a545ba..670211cd 100644 --- a/crates/flowstate/src/main.rs +++ b/crates/flowstate/src/main.rs @@ -7,8 +7,7 @@ use clap::{Parser, Subcommand}; use flowstate::{ docx_conversion::{convert_db8_to_docx, convert_db8_to_pdf, convert_docx_to_pdf, convert_pdf_to_db8}, - logging, - run_standalone, write_demo_document, + logging, run_standalone, write_demo_document, }; struct FlowstateAllocator; diff --git a/crates/flowstate/src/workspace/icons.rs b/crates/flowstate/src/workspace/icons.rs index 4ca37537..65374815 100644 --- a/crates/flowstate/src/workspace/icons.rs +++ b/crates/flowstate/src/workspace/icons.rs @@ -1,10 +1,10 @@ #[cfg(target_os = "macos")] use gpui::ParentElement; +#[cfg(not(target_os = "macos"))] +use gpui_component::IconName; use gpui_component::button::Button; use gpui_component::button::ButtonVariants as _; use gpui_component::{Icon, Sizable as _}; -#[cfg(not(target_os = "macos"))] -use gpui_component::IconName; #[derive(Clone, Copy)] pub enum AppIcon { diff --git a/crates/gpui-flowtext/src/collaboration.rs b/crates/gpui-flowtext/src/collaboration.rs index 87b9692b..78befc86 100644 --- a/crates/gpui-flowtext/src/collaboration.rs +++ b/crates/gpui-flowtext/src/collaboration.rs @@ -4,8 +4,7 @@ use rustc_hash::FxHashMap; use serde::{Deserialize, Serialize}; use super::{ - AssetId, AssetRecord, Block, BlockId, Document, DocumentSpan, InputBlock, InputParagraph, ParagraphId, ParagraphStyle, RunStyles, - TextRun, + AssetId, AssetRecord, Block, BlockId, Document, DocumentSpan, InputBlock, InputParagraph, ParagraphId, ParagraphStyle, RunStyles, TextRun, }; #[derive(Clone, Copy, Debug, Eq, PartialEq, Hash, Serialize, Deserialize)] diff --git a/crates/gpui-flowtext/src/rich_text/editor/mod.rs b/crates/gpui-flowtext/src/rich_text/editor/mod.rs index 498c9796..bdd836a6 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mod.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mod.rs @@ -912,7 +912,8 @@ pub struct RichTextEditor { autoscroll_active: bool, pub(super) caret_visible: bool, caret_blink_active: bool, - last_text_input_at: Option, ime_marked_range: Option>, + last_text_input_at: Option, + ime_marked_range: Option>, external_carets: Vec, pub(super) search_highlights: Vec>, pub(super) active_search_highlight: Option, From 928625ea29e43b0f35b7f265bf5af7c55adba346 Mon Sep 17 00:00:00 2001 From: Adhitya Thirumala Date: Wed, 17 Jun 2026 01:43:18 -0500 Subject: [PATCH 20/45] Quiet-by-default logging with env-controlled levels + clippy fix - logging: default to `error` only; `FLOWSTATE_LOG_LEVEL` (error|warn|info| debug|trace) raises Flowstate's own crates, `FLOWSTATE_LOG`/`RUST_LOG` give full EnvFilter control, `FLOWSTATE_LOG_STDOUT` also mirrors logs to stdout. - README: document the logging environment variables and precedence. - clippy: add missing trailing `;` in session_io asset-serve warn arm (semicolon_if_nothing_returned); `cargo clippy --workspace --all-targets` is now clean. --- README.md | 24 +++++++ crates/flowstate/src/collab/session_io.rs | 2 +- crates/flowstate/src/logging.rs | 84 ++++++++++++++++++++--- 3 files changed, 100 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 441b4088..4c37fbd0 100644 --- a/README.md +++ b/README.md @@ -5,5 +5,29 @@ The performant, narrow text editor for debaters. To build Flowstate, you need to have [Cargo](https://doc.rust-lang.org/cargo/) installed. Run `cargo run --package flowstate --release` to build and run. +# Logging +Flowstate writes a daily-rotated log file (`flowstate.log`) to its data directory. +By default it logs **only `error`-level events** so it stays quiet; raise the +verbosity with the environment variables below. + +| Variable | Values | Effect | +|---|---|---| +| `FLOWSTATE_LOG_LEVEL` | `error` \| `warn` \| `info` \| `debug` \| `trace` | Sets the level for Flowstate's own crates (`flowstate`, `flowstate_collab`, `gpui_flowtext`); dependencies stay at `error`. Simplest knob. | +| `FLOWSTATE_LOG` | an [`EnvFilter`](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html) directive | Full control, including dependency crates, e.g. `flowstate_collab=trace,flowstate::collab=debug`. Takes precedence over `RUST_LOG` and `FLOWSTATE_LOG_LEVEL`. | +| `RUST_LOG` | an `EnvFilter` directive | Same as `FLOWSTATE_LOG` but the standard variable; used if `FLOWSTATE_LOG` is unset. | +| `FLOWSTATE_LOG_STDOUT` | `1` \| `true` \| `yes` \| `on` | Also mirror logs to stdout (in addition to the file). | +| `FLOWSTATE_LOG_DIR` | a directory path | Override where the log file is written. | + +Precedence for the level/filter: `FLOWSTATE_LOG` > `RUST_LOG` > `FLOWSTATE_LOG_LEVEL` > default (`error`). + +Examples: +```sh +# Verbose Flowstate logs, mirrored to the terminal: +FLOWSTATE_LOG_LEVEL=debug FLOWSTATE_LOG_STDOUT=1 cargo run --package flowstate + +# Fine-grained control over specific targets: +FLOWSTATE_LOG="flowstate_collab=trace,flowstate::collab=debug" cargo run --package flowstate +``` + # License Flowstate is licensed under the GNU Affero General Public License v3.0. diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs index 085cdd70..ce1266c3 100644 --- a/crates/flowstate/src/collab/session_io.rs +++ b/crates/flowstate/src/collab/session_io.rs @@ -294,7 +294,7 @@ impl CollabSession { match &result { Ok(bytes) => tracing::debug!(session = %self.session, asset, bytes = bytes.bytes.len(), "served collaboration asset direct request"), Err(error) => { - tracing::warn!(session = %self.session, asset, error = %format_args!("{error:#}"), "serving collaboration asset direct request failed") + tracing::warn!(session = %self.session, asset, error = %format_args!("{error:#}"), "serving collaboration asset direct request failed"); }, } let _ = reply.try_send(result); diff --git a/crates/flowstate/src/logging.rs b/crates/flowstate/src/logging.rs index 3357ae84..e432bc52 100644 --- a/crates/flowstate/src/logging.rs +++ b/crates/flowstate/src/logging.rs @@ -5,11 +5,18 @@ use std::{ use anyhow::{Context as _, Result}; use tracing_appender::non_blocking::WorkerGuard; -use tracing_subscriber::{EnvFilter, layer::SubscriberExt as _, util::SubscriberInitExt as _}; +use tracing_subscriber::{EnvFilter, fmt, layer::SubscriberExt as _, util::SubscriberInitExt as _}; use crate::app_settings::flowstate_data_dir; -const DEFAULT_LOG_FILTER: &str = "flowstate::collab=trace,flowstate::workspace::workspace=trace,flowstate_collab=trace"; +/// Default filter: only the most serious events (errors), for every target. +/// Raise Flowstate's own verbosity with `FLOWSTATE_LOG_LEVEL`, or take full +/// control (including dependencies) with `FLOWSTATE_LOG` / `RUST_LOG`. +const DEFAULT_LOG_FILTER: &str = "error"; + +/// Crates whose verbosity `FLOWSTATE_LOG_LEVEL` raises. Dependencies stay at the +/// default level so the output stays focused on Flowstate's own logs. +const FLOWSTATE_TARGETS: [&str; 3] = ["flowstate", "flowstate_collab", "gpui_flowtext"]; pub struct LoggingGuard { _guard: WorkerGuard, @@ -28,10 +35,20 @@ pub fn init() -> Result { let file_appender = tracing_appender::rolling::daily(&directory, "flowstate.log"); let (writer, guard) = tracing_appender::non_blocking(file_appender); + // Optionally mirror logs to stdout (handy when running from a terminal). + let stdout_layer = log_to_stdout().then(|| { + fmt::layer() + .with_writer(std::io::stdout) + .with_ansi(true) + .with_file(true) + .with_line_number(true) + .with_target(true) + }); + tracing_subscriber::registry() .with(env_filter()) .with( - tracing_subscriber::fmt::layer() + fmt::layer() .with_writer(writer) .with_ansi(false) .with_file(true) @@ -40,23 +57,72 @@ pub fn init() -> Result { .with_thread_ids(true) .with_thread_names(true), ) + .with(stdout_layer) .try_init() .context("initializing flowstate logging failed")?; Ok(LoggingGuard { _guard: guard, directory }) } +/// Resolves the log filter from the environment, in precedence order: +/// 1. `FLOWSTATE_LOG` — full [`EnvFilter`] directive (advanced; also controls +/// dependency crates), e.g. `flowstate_collab=trace,flowstate::collab=debug`. +/// 2. `RUST_LOG` — the standard [`EnvFilter`] directive. +/// 3. `FLOWSTATE_LOG_LEVEL` — a single level (`error|warn|info|debug|trace`) +/// applied to Flowstate's own crates, leaving dependencies at `error`. +/// 4. Default — `error` only. fn env_filter() -> EnvFilter { - let filter = env::var("FLOWSTATE_LOG") - .or_else(|_| env::var("RUST_LOG")) - .unwrap_or_else(|_| DEFAULT_LOG_FILTER.to_string()); - - EnvFilter::try_new(&filter).unwrap_or_else(|error| { - eprintln!("invalid FLOWSTATE_LOG/RUST_LOG filter {filter:?}: {error}; falling back to {DEFAULT_LOG_FILTER:?}"); + let Some(directive) = directive_from_env() else { + return EnvFilter::new(DEFAULT_LOG_FILTER); + }; + EnvFilter::try_new(&directive).unwrap_or_else(|error| { + eprintln!("invalid log filter {directive:?}: {error}; falling back to {DEFAULT_LOG_FILTER:?}"); EnvFilter::new(DEFAULT_LOG_FILTER) }) } +fn directive_from_env() -> Option { + for var in ["FLOWSTATE_LOG", "RUST_LOG"] { + if let Ok(filter) = env::var(var) + && !filter.trim().is_empty() + { + return Some(filter); + } + } + + let level = level_from_env()?; + let scoped = FLOWSTATE_TARGETS + .iter() + .map(|target| format!("{target}={level}")) + .collect::>() + .join(","); + Some(format!("{DEFAULT_LOG_FILTER},{scoped}")) +} + +fn level_from_env() -> Option<&'static str> { + let value = env::var("FLOWSTATE_LOG_LEVEL").ok()?; + match value.trim().to_ascii_lowercase().as_str() { + "" => None, + "error" => Some("error"), + "warn" | "warning" => Some("warn"), + "info" => Some("info"), + "debug" => Some("debug"), + "trace" => Some("trace"), + other => { + eprintln!("invalid FLOWSTATE_LOG_LEVEL {other:?}; expected error|warn|info|debug|trace"); + None + }, + } +} + +fn log_to_stdout() -> bool { + env::var("FLOWSTATE_LOG_STDOUT").is_ok_and(|value| is_truthy(&value)) +} + +fn is_truthy(value: &str) -> bool { + matches!(value.trim().to_ascii_lowercase().as_str(), "1" | "true" | "yes" | "on") +} + fn log_directory() -> PathBuf { env::var_os("FLOWSTATE_LOG_DIR") .map(PathBuf::from) From f318f02c2ee22b3b10801119189e2f7e4f4db0e3 Mon Sep 17 00:00:00 2001 From: "Adam D. Humphrey" Date: Thu, 18 Jun 2026 00:42:19 -0500 Subject: [PATCH 21/45] Here you go adhi --- Cargo.lock | 9 + adjustmentplan.md | 1283 + crates/flowstate-collab/Cargo.toml | 2 + crates/flowstate-collab/src/binding.rs | 255 - crates/flowstate-collab/src/body_index.rs | 294 - crates/flowstate-collab/src/crdt_runtime.rs | 994 + crates/flowstate-collab/src/lib.rs | 14 +- crates/flowstate-collab/src/local_apply.rs | 705 - crates/flowstate-collab/src/patch_apply.rs | 200 - crates/flowstate-collab/src/projection.rs | 210 - crates/flowstate-collab/src/remote_apply.rs | 675 - crates/flowstate-collab/src/schema.rs | 418 - crates/flowstate-collab/src/self_check.rs | 8 +- crates/flowstate-collab/tests/convergence.rs | 1158 - crates/flowstate-collab/tests/local_apply.rs | 53 - crates/flowstate-collab/tests/perf_smoke.rs | 193 - crates/flowstate-collab/tests/projection.rs | 122 - crates/flowstate-collab/tests/remote_apply.rs | 344 - crates/flowstate-collab/tests/translation.rs | 341 - crates/flowstate-document/Cargo.toml | 7 + crates/flowstate-document/src/lib.rs | 45 +- crates/flowstate-document/src/loro_import.rs | 382 + .../flowstate-document/src/loro_projection.rs | 501 + crates/flowstate-document/src/loro_schema.rs | 159 + crates/flowstate-document/src/package.rs | 1031 + crates/flowstate-docx/src/lib.rs | 4 +- crates/flowstate-docx/src/pdf.rs | 4 +- crates/flowstate-docx/src/pdf_recovery.rs | 11 +- crates/flowstate/src/collab/asset_transfer.rs | 60 +- crates/flowstate/src/collab/presence_view.rs | 5 +- crates/flowstate/src/collab/session.rs | 53 +- crates/flowstate/src/collab/session_io.rs | 126 +- .../flowstate/src/collab/session_presence.rs | 3 +- crates/flowstate/src/collab/session_timers.rs | 14 +- .../flowstate/src/workspace/workspace/load.rs | 7 +- .../flowstate/src/workspace/workspace/mod.rs | 2 +- crates/gpui-flowtext/src/collaboration.rs | 22 +- crates/gpui-flowtext/src/persistence/io.rs | 20 +- .../gpui-flowtext/src/persistence/records.rs | 71 - .../src/rich_text/editor/block_insertion.rs | 6 +- .../src/rich_text/editor/edit_pipeline.rs | 76 +- .../src/rich_text/editor/media.rs | 12 +- .../gpui-flowtext/src/rich_text/editor/mod.rs | 2 +- .../src/rich_text/editor/mouse.rs | 12 +- .../src/rich_text/editor/object_selection.rs | 22 +- .../src/rich_text/editor/paste.rs | 17 +- .../editor/table_equation_editing.rs | 18 +- .../src/rich_text/editor/tables.rs | 6 +- .../src/rich_text/tests/collab_capture.rs | 22 +- handoff-adjustmentplan.md | 333 + loroapi.md | 34060 ++++++++++++++++ 51 files changed, 39043 insertions(+), 5348 deletions(-) create mode 100644 adjustmentplan.md delete mode 100644 crates/flowstate-collab/src/binding.rs delete mode 100644 crates/flowstate-collab/src/body_index.rs create mode 100644 crates/flowstate-collab/src/crdt_runtime.rs delete mode 100644 crates/flowstate-collab/src/local_apply.rs delete mode 100644 crates/flowstate-collab/src/patch_apply.rs delete mode 100644 crates/flowstate-collab/src/projection.rs delete mode 100644 crates/flowstate-collab/src/remote_apply.rs delete mode 100644 crates/flowstate-collab/src/schema.rs delete mode 100644 crates/flowstate-collab/tests/convergence.rs delete mode 100644 crates/flowstate-collab/tests/local_apply.rs delete mode 100644 crates/flowstate-collab/tests/perf_smoke.rs delete mode 100644 crates/flowstate-collab/tests/projection.rs delete mode 100644 crates/flowstate-collab/tests/remote_apply.rs delete mode 100644 crates/flowstate-collab/tests/translation.rs create mode 100644 crates/flowstate-document/src/loro_import.rs create mode 100644 crates/flowstate-document/src/loro_projection.rs create mode 100644 crates/flowstate-document/src/loro_schema.rs create mode 100644 crates/flowstate-document/src/package.rs create mode 100644 handoff-adjustmentplan.md create mode 100644 loroapi.md diff --git a/Cargo.lock b/Cargo.lock index 748de41c..3522d2af 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3008,6 +3008,7 @@ version = "0.1.0" dependencies = [ "anyhow", "async-channel 2.5.0", + "flowstate-document", "gpui-flowtext", "iroh", "iroh-gossip", @@ -3018,6 +3019,7 @@ dependencies = [ "proptest", "rand 0.9.4", "serde", + "tempfile", "tokio", "tracing", "twox-hash", @@ -3028,9 +3030,16 @@ dependencies = [ name = "flowstate-document" version = "0.1.0" dependencies = [ + "blake3", "gpui", "gpui-flowtext", + "loro", + "postcard", "rustc-hash 2.1.2", + "serde", + "tempfile", + "uuid", + "zstd", ] [[package]] diff --git a/adjustmentplan.md b/adjustmentplan.md new file mode 100644 index 00000000..37c511a5 --- /dev/null +++ b/adjustmentplan.md @@ -0,0 +1,1283 @@ +# Flowstate Loro-Native Document Architecture Specification + +## 1. Non-Negotiable Objective + +Flowstate's document architecture must be rebuilt around Loro as the canonical document substrate. + +The target architecture is not "a Flowstate document mirrored into Loro." It is "a Loro document rendered and edited by Flowstate." + +Do not optimize this plan for implementation lift, prototype speed, or backwards compatibility with the current development-only `.db8` format. The target is the production architecture Flowstate should actually keep. The ABSOLUTE PLATONIC IDEAL. + +The renderer, layout engine, UI command layer, DOCX/PDF/export tools, search tools, and caches may remain Flowstate-specific. The document state, durable history, undo/redo basis, collaboration state, revision timeline, and save format must be Loro-native. + +The goal is a single architecture that supports: + +* local editing +* collaborative editing +* durable operation history +* time travel +* branch/fork restore flows +* per-peer undo/redo +* CRDT-safe concurrent editing +* structured rich documents +* filesystem persistence +* DOCX import and export compatibility +* PDF/export generation from snapshots +* deterministic rendering from canonical state +* within-document and tub search from Loro-derived projections + +The old development `.db8` document format is not a compatibility target. DOCX is still a first-class external import/export format. + +## 1A. Locked Clarifications Before Implementation + +These decisions are part of the architecture, not optional implementation notes. + +The user-facing Flowstate document extension remains `.db8`. That extension is brand identity only. It does not imply compatibility with the old development `.db8` serializer, old `gpui-flowtext::Document` persistence, or any previous final-state native binary format. From this architecture onward, a `.db8` file is a Loro-native Flowstate package as specified in this document. + +There is no old `.db8` importer, compatibility reader, migration path, fallback decoder, or dual-format open path. The previous development `.db8` format should be treated as if it never existed. Code that exists only to read, write, recover, index, export, or interoperate with the old final-state `.db8` model must be removed or rewritten onto the Loro-native package model. + +The implementation must use the Rust `loro` crate APIs as authoritative. `loroapi.md` may be used as conceptual API background, but it documents the JavaScript/TypeScript binding and must not be treated as proof that a Rust API with the same name or capability exists. Before relying on a Loro feature, verify it against the Rust crate in use. + +Rust Loro currently exposes rich text as text plus marks/cursors, and nested containers through maps/lists/movable lists. It does not expose a Rust-native rich-text embed/container insertion API for placing child containers directly inside `LoroText`. Therefore object and structural block ordering is locked to the object replacement character strategy: use `U+FFFC` (`\u{FFFC}`) placeholders inside flow `LoroText`, and store the corresponding structured object metadata in Loro maps such as `blocks_by_id`. The placeholder is the CRDT-ordered anchor. The block map is the editable durable object record. + +Tables are fully in scope for this architecture pass. They must be implemented as structured Loro objects with row identity, column identity, cell identity, spans, nested table support, and independent CRDT text flows per cell. A table may not be stored as an opaque binary payload or rewritten wholesale for normal edits. + +Named revisions are time-travellable snapshot/frontier points. A named revision must preserve the ability to render, open, and fork that point in document history. It does not need to preserve every individual low-level operation that occurred before that named point. History compaction may squash unnamed low-level update history once the retained snapshots/frontiers needed for named revisions and product checkpoints remain restorable. + +PDF source embedding must embed only the new Loro-native `.db8` package bytes. It must not embed the old development `.db8` payload, and it must not support both old and new source payloads. + +## 2. Canonical Model + +There is exactly one canonical document model: the `LoroDoc`. + +Everything else is derived. + +Flowstate may maintain a `DocumentProjection`, layout cache, paragraph index, glyph cache, hit-test map, section cache, pagination cache, asset cache, render tree, search index, preview cache, and export projection. None of those are authoritative. They can be discarded and rebuilt from the Loro document plus the asset store. + +The following must not remain canonical: + +* native final-state `.db8` serialization +* `gpui-flowtext::Document` as persistent source of truth +* Flowstate operation logs as persistent source of truth +* patch streams as document truth +* local/remote mutation paths that reconcile two document models +* binding layers whose job is to keep Flowstate state and Loro state synchronized +* object payload blobs that hide structured editable content from Loro + +The canonical pipeline is: + +```text +Local command + -> resolve command against current DocumentProjection + -> send semantic command to CRDT runtime + -> mutate LoroDoc in grouped Loro transaction/change + -> permanent Loro subscription emits document event/update + -> CRDT runtime updates or invalidates DocumentProjection + -> UI receives projection diff/snapshot + -> render + -> persist Loro update + -> sync Loro update + +Remote update + -> CRDT runtime imports bytes into LoroDoc + -> import status triggers immediate anti-entropy if dependencies are missing + -> permanent Loro subscription emits document event/update + -> CRDT runtime updates or invalidates DocumentProjection + -> UI receives projection diff/snapshot + -> render + -> persist accepted update +``` + +There is no dual-write model. + +## 3. Runtime Ownership + +Flowstate must use a dedicated CRDT runtime that owns each live `LoroDoc`. + +The UI thread must not own or mutate the canonical `LoroDoc`. The UI owns the current immutable or copy-on-write projection state needed for rendering, layout, hit testing, and command construction. + +The CRDT runtime: + +* owns the `LoroDoc` +* owns the permanent Loro subscriptions +* owns the Loro `UndoManager` +* owns import/export/update persistence coordination +* receives semantic editor commands over channels +* returns projection diffs, projection snapshots, selection updates, asset availability changes, and status events +* batches projection work by Loro commit/frontier +* isolates CRDT work from UI frame timing + +The preferred shape is: + +```text +UI thread + DocumentProjection + LayoutEngine + Renderer + HitTest + Command construction + Presence rendering + +CRDT runtime thread/task + LoroDoc + UndoManager + Loro subscriptions + Persistence writer + Sync import/export + Projection builder/incremental projector + Asset manifest coordination +``` + +The runtime may use a thread, task, or actor implementation, but the ownership boundary is mandatory: canonical Loro mutation happens in one place. + +## 4. Command Layer + +The old Flowstate collaboration API and `CanonicalOperation` path must be retired and removed. + +Flowstate should keep semantic editor commands, but those commands must target the Loro-native schema directly. They are user intent, not a second canonical operation language. + +The command architecture is locked: + +> UI sends semantic commands; the CRDT runtime resolves them into Loro schema mutations. + +The UI must not construct low-level Loro schema mutations directly as the normal command path. Schema mutation authority lives in the CRDT runtime. + +Preserving the old `CanonicalOperation` API as a durable or canonical path is explicitly rejected. + +```text +UI input + -> hit test/projection coordinate + -> semantic command + -> CRDT runtime validates against current frontier + -> Loro schema mutation(s) + -> grouped commit + -> projection diff +``` + +Examples: + +```text +insert_text(flow_id, cursor, text, style_state) +split_paragraph(flow_id, cursor, inherited_attrs) +join_paragraphs(flow_id, boundary_id) +set_run_semantic_style(flow_id, range, style_id) +set_highlight_style(flow_id, range, highlight_id) +insert_table(anchor, rows, columns) +insert_image(anchor, asset_ref, attrs) +edit_image_alt_text(image_id, text_command) +edit_equation_source(equation_id, text_command) +move_block(block_id, target_cursor) +undo() +redo() +open_revision(frontier) +fork_revision(frontier) +``` + +## 5. Projection And Editor Model + +The ideal target may substantially replace or rearchitect `gpui-flowtext::Document`. + +The editor/render model must replace the current mutable `gpui-flowtext::Document` entirely. + +Keeping a type named `Document` while redefining it as projection-only is not the target. Keeping the current mutable `Document` plus Loro binding/reconciliation is rejected. + +`DocumentProjection` is a frontier-scoped view: + +```text +DocumentProjection + frontier + flows + paragraphs + blocks + sections + table_grids + inline_style_runs + object_metadata + asset_refs + search_units + cursor_position_maps + byte/grapheme/UTF-16 indexes for the projection only + dirty_ranges +``` + +The projection must be rebuildable from: + +```text +LoroDoc at frontier + asset manifest +``` + +The projection may cache: + +* paragraph boundaries +* paragraph IDs +* block order +* style runs +* table layout inputs +* glyph layout inputs +* pagination keys +* hit-test data +* search units +* export units + +Projection coordinates are not canonical. Canonical command positions should use Loro cursors wherever possible. Byte, grapheme, UTF-16, row, column, and paragraph indexes are local coordinates within a specific projection frontier. + +If projection encounters a malformed paragraph boundary without a paragraph-style mark, it should default that paragraph to Normal and immediately schedule a repair mutation in the CRDT runtime. Repair should modify the underlying Loro buffer so the document is safe before further p2p sharing propagates the malformed state. + +## 6. Canonical Text Model + +The document must not use one independent `LoroText` per paragraph as the primary body text model. + +That design breaks expected word-processor semantics for concurrent paragraph splits and inline edits. + +Example: + +```text +Initial: +ABC + +Alice inserts "y" between B and C. +Bob presses Enter between A and B. +``` + +The semantically correct result is: + +```text +A +ByC +``` + +Not: + +```text +Ay +BC +``` + +Therefore, paragraph content must live inside a continuous CRDT text sequence. + +The rule is: + +> Use one canonical `LoroText` per continuous text flow. + +A text flow is a sequence where inline edits and paragraph boundary changes must commute naturally. + +Independent text flows include: + +* main document body +* table cell contents +* image caption +* image alt text +* equation source +* footnote body, if footnotes are added +* endnote body, if endnotes are added +* header text flow +* footer text flow +* sidebar text flow, if sidebars are added + +Annotations, comments, bookmarks, and suggestion systems are out of scope for this plan and should not appear in the target schema. + +Paragraph styles apply to paragraph-bearing flows: body flows, table cell flows, caption flows, headers, footers, and similar prose flows. + +Equation source and image alt text flows do not have paragraph styles. They are plain/specialized text flows, not Flowstate paragraph flows. + +## 7. Paragraph Boundaries + +The preferred paragraph boundary representation is literal newline characters inside the flow `LoroText`. + +Rationale: + +* It matches ordinary text semantics. +* It keeps paragraph split/join as native text edits. +* It avoids private-use marker leakage into clipboard/search/export. +* It already matches the rough direction of the current root body text model. +* It makes body text easier to inspect, search, import, and export. + +Each text flow should begin with a persistent sentinel newline character. + +That sentinel newline is not rendered as a visible blank line and is not user-deletable. It exists so every editable flow always has a paragraph boundary character before the first user-visible paragraph. + +Paragraphs are signified by the boundary character that sets them off: + +```text +\nParagraph one\nParagraph two\nParagraph three +^ ^ ^ +sentinel boundary boundary +``` + +The initial sentinel boundary carries the first paragraph's paragraph-style mark. Each later paragraph's style is carried by the boundary newline immediately before that paragraph. This is the predecessor-boundary model. + +The plan rejects custom paragraph marker characters unless a later Loro limitation makes newline anchoring insufficient. + +Paragraph identity metadata must still be Loro-native and durable. Newlines alone are enough for paragraph boundaries, but paragraph IDs and stable paragraph anchors still need Loro-native representation. + +Each paragraph is represented by identity metadata anchored to its start boundary: + +```text +ParagraphMap + id: ParagraphId + flow_id: FlowId + start_cursor: Loro cursor at paragraph start / preceding boundary + boundary_cursor: Loro cursor for paragraph-ending newline when available +``` + +Empty paragraphs are representable because an empty paragraph is a boundary interval. A blank line between two newline characters is a real paragraph and carries style on its predecessor boundary newline. + +Paragraph style storage is locked to Loro marks. + +Flowstate should use Loro's rich-text mark APIs for paragraph style. The canonical paragraph style is attached to paragraph boundary text, normally the newline immediately before the paragraph. This keeps Flowstate aligned with Loro's intended rich-text API instead of inventing a parallel paragraph-style substrate. + +The mark model must still define word-processor semantics explicitly: + +* Every flow starts with a non-rendered, non-deletable sentinel newline. +* The first visible paragraph gets its style from the sentinel newline's paragraph-style mark. +* Every later paragraph gets its style from the newline immediately before it. +* A blank paragraph between two newline characters gets its paragraph style from the predecessor newline. +* Splitting a paragraph in the middle makes both resulting paragraphs inherit the origin paragraph's style. +* Pressing Enter at the end of a styled paragraph creates a following blank paragraph with Normal style. +* Pressing Enter at the start of a styled paragraph creates a preceding blank paragraph that inherits the original paragraph's style; the original paragraph keeps its style. +* Joining paragraphs keeps the higher/earlier paragraph's style. +* Deleting across paragraphs keeps the higher/earlier surviving paragraph's style. +* Applying paragraph style at a collapsed caret applies to the current paragraph. +* Applying paragraph style to any selection applies to every paragraph touched by the selection, including partially touched first/last paragraphs. +* Rich paste preserves pasted paragraph styles unless pasted onto a styled paragraph in a way that semantically replaces that paragraph; in that replacement case, pasted paragraphs inherit the target paragraph's style. +* Plain-text paste with newlines inherits the insertion paragraph's style for all inserted paragraphs. + +Do not create a competing paragraph-style field in paragraph/block metadata. If a projection caches paragraph style, that cache is derived from Loro marks. + +Paragraph style uses one Loro mark key: + +```text +paragraph_style = +``` + +Paragraph style IDs are stable enum slots with holes reserved for future styles. User-facing labels can change freely in the app/theme without changing document semantics. The document stores style identity, not display names and not appearance attributes. + +Paragraph-style marks should use the least-sticky Loro mark expansion setting that supports boundary-only paragraph marks. Normal paragraph-style inheritance should not rely on automatic mark expansion. Commands such as Enter, split paragraph, paste, import, and repair must explicitly set paragraph-style marks on the affected boundary newlines. + +If two peers concurrently set different paragraph-style enum values on the same boundary under the same `paragraph_style` key, Flowstate accepts Loro's deterministic visible winner. Projection should use the value Loro exposes as current. This is normal CRDT conflict resolution, not corruption. + +Other paragraph-level properties are out of scope unless Flowstate later introduces them as real editor features. If future paragraph-level properties cannot be represented cleanly as Loro marks, they need a separate design decision at that time. + +The renderer strips no custom paragraph marker. It renders newline-delimited paragraphs from the projection. + +## 8. Flow Structure + +Each text flow is stored as: + +```text +FlowMap + id: FlowId + kind: "body" | "table_cell" | "caption" | "alt_text" | "equation_source" | "header" | "footer" | ... + text: LoroText + attrs: LoroMap +``` + +Flow maps are reachable from: + +```text +RootMap + flows_by_id: LoroMap +``` + +A flow may contain: + +* normal Unicode text +* newline paragraph boundaries +* object/embed placeholders when the flow supports embedded objects + +Paragraphs are ranges in a flow. Paragraph split inserts `\n`. Paragraph join removes `\n`. Inline insert inserts into the same flow text. + +Nested flows remain in Loro history even after the current document frontier no longer reaches them. They are not deleted from history merely because a block was deleted. Package-level asset/cache garbage collection may remove unreachable binary assets according to retention policy, but Loro document history remains history unless a deliberate lineage-compaction/export operation creates a new document lineage. + +## 9. Block And Object Ordering + +Block order is canonical in the text flow itself. + +The main body flow is the canonical ordered sequence for body paragraphs and body objects. A separate ordered block list must not become a second source of truth. + +Object blocks are represented by object placeholders in a flow. Their editable metadata lives in a block registry: + +```text +RootMap + blocks_by_id: LoroMap +``` + +Block map: + +```text +BlockMap + id: BlockId + kind: "paragraph" | "image" | "equation" | "table" | "divider" | ... + flow_id: FlowId + anchor_cursor: Loro cursor for position in parent flow + attrs: LoroMap + nested_refs: LoroMap +``` + +Paragraph blocks are metadata records over newline-delimited flow ranges. + +Object blocks are metadata records anchored to object placeholders in the flow. + +Images and equations do not have paragraph styles. + +Tables have independent table styling. Table styling is not inherited from surrounding paragraphs. + +Object placeholders use the Unicode object replacement character `U+FFFC` (`\u{FFFC}`) in the parent flow `LoroText`, plus a block ID mapping in `blocks_by_id`. This is locked because the Rust `loro` crate does not currently expose a native rich-text embed/container insertion API for placing child containers directly inside `LoroText`. + +Object placeholder characters are not user-authored document text. They are structural anchors. Clipboard, search, DOCX export, PDF export, plain-text export, and visible text rendering must project them as the corresponding object, not leak the raw replacement character as ordinary content. + +All durable IDs must be stored in Loro. Projection-generated IDs must not become canonical. + +## 10. Semantic Styling + +Flowstate does not allow arbitrary user-authored formatting as the canonical editing model. + +Inline styling must remain semantic-bound. Loro marks should store style identities and booleans from Flowstate's semantic style system, not freeform font/color/CSS values. + +Examples: + +```text +run_semantic_style_id +highlight_style_id +direct_underline +strikethrough +emphasis_style_id, if introduced +``` + +Paragraph style is a Loro mark on paragraph boundary text: + +```text +paragraph_style_id +``` + +The exact style model may continue to use overlapping semantic categories, as Flowstate does today: + +* paragraph style IDs stored as Loro paragraph-boundary marks +* run semantic style IDs +* highlight style IDs +* underline flag/style +* strikethrough flag/style +* future semantic style families + +The user-facing style catalog is app/theme data only. A style ID in Loro references a semantic enum slot; the appearance of that slot is resolved by the client theme/style catalog. + +DOCX import remains supported. Imported direct formatting must be interpreted into Flowstate's predefined semantic model using the existing Verbatim/heuristic import architecture, evolved as needed. + +DOCX import must not create new document-local styles. It maps recognized DOCX styles and formatting heuristics into Flowstate paragraph style slots, run semantic style slots, highlight slots, underline, and strikethrough. Unknown DOCX formatting may be reported or dropped, but it must not expand the live document's semantic style universe. + +Imported formatting should not turn the live editor into a freeform arbitrary formatter. + +URLs are ordinary document text. This plan does not model links as annotations or comments. + +Comments, annotations, bookmarks, and suggestions are out of scope. + +## 11. Sections And Page Structure + +Sections are structural metadata anchored to body flow positions. + +```text +sections_by_id: LoroMap +section_order: derived from body flow anchors unless later proven insufficient + +SectionMap + id + start_cursor + attrs: + page_size + margins + columns + header_flow_id + footer_flow_id + page_numbering + orientation +``` + +A section boundary should be a CRDT-stable position in the body flow. + +Headers and footers are independent text flows. + +Section/order metadata must not duplicate paragraph ordering. The body flow remains canonical for body order. + +Heading and outline computation reads paragraph style exclusively from paragraph-boundary marks. A paragraph participates in outline/section projection when its semantic paragraph style slot maps to a heading/section role in the app/theme style system. + +## 12. Tables + +Tables must be structurally CRDT-native. + +A table must not be stored as a single binary blob whose entire payload is rewritten for edits. + +The target table schema must support: + +* row identity +* column identity +* row insertion/deletion/movement +* column insertion/deletion/movement +* cell identity +* merged cells +* row spans +* column spans +* nested tables +* independent rich text flows per cell +* concurrent edits in different cells without conflict +* concurrent structural edits represented by Loro containers + +Preferred table schema: + +```text +TableBlock + id: BlockId + kind: "table" + anchor_cursor + attrs: TableAttrs + row_order: LoroMovableList + rows_by_id: LoroMap + column_order: LoroMovableList + columns_by_id: LoroMap + cells_by_id: LoroMap + +RowMap + id: RowId + attrs: RowAttrs + +ColumnMap + id: ColumnId + attrs: ColumnAttrs + +CellMap + id: CellId + row_id: RowId + column_id: ColumnId + row_span: u32 + column_span: u32 + attrs: CellAttrs + flow_id: FlowId + nested_table_ids: LoroMovableList +``` + +Each table cell owns an independent text flow. + +Inside a cell flow, paragraphs follow the same rule as the main body: one continuous `LoroText` per cell flow with newline paragraph boundaries. + +Use `LoroMovableList` for rows and columns. Tables are ordered structures with semantic movement. + +Do not use `LoroTree` as the main table representation unless arbitrary recursive tree moves become required. Tables are ordered row/column grids with cell flows, not general trees. + +## 13. Equations + +Equation source must be stored as `LoroText`. + +Equation metadata should be stored in `LoroMap`. + +```text +EquationBlock + id: BlockId + kind: "equation" + anchor_cursor + source_flow_id + attrs: + syntax: "latex" | "mathml" | ... + display: "inline" | "block" + numbering + alignment +``` + +If two users edit the equation source concurrently, the source should merge like text. + +Do not store equation source as a map string. + +Render output is not synchronized between peers. Each client renders equation source locally using Flowstate's live rendering system. Any rendered SVG/bitmap/layout result is a disposable client cache. + +Inline equations and block equations are both represented as object placeholders inside a parent text flow. The renderer decides whether the object is rendered inline or block-level from metadata. + +## 14. Images And Assets + +Images do not need a special image CRDT. + +Image metadata should be stored in `LoroMap`. Image bytes should be stored outside Loro in a content-addressed asset store. + +```text +ImageBlock + id: BlockId + kind: "image" + anchor_cursor + asset_id + content_hash + mime_type + byte_length + dimensions + crop + sizing_mode + alignment + alt_text_flow_id + caption_flow_id +``` + +Image metadata should use field-level map keys. Do not rewrite the whole image object for a metadata edit. + +Image caption and alt text must be CRDT-editable text flows. + +Asset bytes are stored in the package asset store, not in Loro. + +Asset content addressing must use BLAKE3 canonically. + +The canonical asset hash is a BLAKE3 digest. Short local cache keys may be derived for in-memory lookup, but they are never canonical and must not be used for integrity or package identity. + +Remote asset behavior: + +* CRDT updates may be accepted before referenced asset bytes arrive. +* The document must render a visible placeholder for missing assets. +* Missing assets should trigger prioritized asset pulls. +* Asset availability should participate in sync/anti-entropy status so peers can discover missing bytes, not just missing Loro ops. +* Local image insertion may commit Loro metadata before the full asset bytes are available. +* If asset bytes never arrive or remain incomplete, the document should retain the reference and render an explicit incomplete-asset placeholder. +* Remote metadata may arrive before bytes; that is acceptable and should render as a recoverable placeholder. + +## 15. Presence, Peers, And Author Metadata + +Presence is not document history. + +Remote carets, live selections, transient names/colors, and typing/live state belong in Loro ephemeral state or equivalent ephemeral sync state. + +Durable authorship metadata is separate. + +The schema should distinguish: + +```text +UserId + stable user-facing identity for blame/history/UX + +ReplicaId / PeerId + unique Loro editing replica identity + may differ for each device, app instance, tab, or session participant +``` + +The same user may have multiple devices. The same device or app instance may open multiple tabs into the same session. Those must be distinct Loro replicas even if they share one user-facing identity. + +Never set a shared or user-stable Loro peer ID. Loro peer IDs identify editing replicas and must be unique per active replica. + +The document may store durable author metadata keyed by user identity and/or by observed Loro peer IDs. Live roster/presence remains ephemeral. + +## 16. Selection, Cursors, And Affinity + +Canonical selection endpoints should use Loro cursors wherever possible. + +The selection model must include explicit affinity/gravity information: + +```text +SelectionEndpoint + cursor + affinity: before | after | neutral + visual_gravity + +Selection + anchor: SelectionEndpoint + head: SelectionEndpoint + direction +``` + +This is stronger than merely choosing `Side::Before` or `Side::After` at cursor creation. The editor needs to preserve the user's visual intent across: + +* concurrent inserts at the same position +* undo/redo cursor restoration +* bidi text +* object boundaries +* line wrapping +* selection extension +* collapsed caret movement + +Clarification: + +* A simple side choice says where the cursor sits relative to concurrent inserted content. +* A full affinity/gravity model stores why that side was chosen and how the caret should behave visually as text/layout changes. + +`Side::Middle` should not be hardcoded as the only selection side. It can remain available for genuinely neutral anchors, but normal caret/selection behavior should choose side from stored affinity. + +UndoManager cursor storage should save and restore selection cursors through Loro's cursor transformation support. + +## 17. Undo And Redo + +Undo/redo must be Loro-native for all documents, local or collaborative. + +Flowstate must not maintain a separate canonical undo/redo stack for document mutations. + +Undo is per-peer/per-replica local undo. If this user edits, then receives another peer's operation, pressing undo should reverse this user's edit, transformed over the remote change. It should not undo the other peer's operation. + +The CRDT runtime owns one Loro `UndoManager` per live document replica. + +Semantic undo units include: + +* typed word or typing burst +* paragraph split +* paste +* style change +* table row/column insert +* table structural edit +* object resize +* image insert +* equation edit +* section break insert + +Those undo units must correspond to grouped Loro changes. + +Use Loro-native mechanisms: + +* explicit commits for command boundaries +* Loro change merge interval for continuous local edits when appropriate +* UndoManager merge interval +* UndoManager group start/end for compound semantic commands +* UndoManager cursor onPush/onPop to restore selection + +Time travel is not undo. Time travel is checkout/fork/navigation over durable Loro frontiers. + +## 18. Time Travel, Revisions, And Restore + +The operation history is the durable basis for time travel. + +The document package should maintain a revision index: + +```text +Revision + id + title + timestamp + author/user + replica_id + frontier/version_vector + summary + thumbnail optional + parent revision optional +``` + +A revision records a named frontier into Loro history and/or a retained package snapshot sufficient to materialize that frontier. It is a time-travellable document point, not a promise to retain every low-level operation that happened before that point. + +Opening a revision checks out or materializes the Loro document at that frontier and renders the projection for that state. + +Restore behavior: + +* Opening a historical revision should open a new tab with that historical version. +* That tab should be dirty. +* Saving that tab should save as a branch/fork/new document according to product flow. +* Forking into a new branch/document must remain available as the preferred restore path. + +History compaction is mandatory. Full history forever is rejected. + +The compaction policy is locked: + +> Keep named revision snapshots/frontiers, compact unnamed update history. + +Compaction must preserve user-meaningful and product-meaningful revision frontiers as restorable/forkable points while squashing low-level update history that has no revision identity. A named revision must remain openable, renderable, and forkable after compaction, but the implementation may discard unnamed per-keystroke or per-command update detail beneath a retained revision snapshot/frontier. + +Natural automatic chunk points include: + +* app tab session boundaries +* save instances, especially explicit `Ctrl+S` +* named revision creation +* import completion +* major document-structure operations + +Each explicit save is a natural revision/checkpoint boundary. Future UX may allow users to name or preserve selected revision frontiers. Unnamed intra-session operation detail may be compacted once covered by retained snapshots/frontiers. Named revisions are snapshot/frontier preservation points, not permanent full-operation-log retention points. + +Creating a new explicit document lineage is not the preferred normal compaction strategy. + +## 19. Filesystem Package + +The user-facing artifact remains a single `.db8` document file. + +Internally, it must not be the old final-state native serializer. + +The `.db8` extension names the new Loro-native Flowstate package. It must not be used as a compatibility excuse for retaining the old development `.db8` format. Old `.db8` read/write/import/recovery/indexing code paths must be removed or rewritten to this package format. + +The ideal filesystem object is a Loro document package containing: + +```text +DocumentPackage + manifest + loro_snapshots + loro_update_segments + asset_store + revision_index + projection_cache + search_projection_cache + thumbnails + integrity_index +``` + +The package format is locked: + +> Flowstate uses a custom chunked binary container that is both Loro-native and Flowstate-native. + +Reasons: + +* single user-facing file +* append-friendly update storage +* Flowstate-controlled chunk layout and indexing +* efficient manifests and internal lookup indexes +* direct storage for snapshots, update segments, assets, thumbnails, and projection/search caches +* fast unopened-file indexing for tub search +* no dependency on a general SQL database file format for document storage +* room for Loro-aware and Flowstate-aware chunk types, compression, checksums, and partial reads + +The package must support external workspace/tub indexing. The package itself should expose enough projection/search metadata for Flowstate to index it quickly without fully opening the document or replaying Loro history. + +Logical chunk classes: + +```text +manifest chunk + package_format_version + loro_schema_version + document_id + latest_frontier + latest_snapshot_id + update_segment_index + asset_index + projection_cache_frontier + search_cache_frontier + created_at + modified_at + +loro snapshot chunks + snapshot_id + frontier + bytes + created_at + +loro update segment chunks + segment_id + from_frontier + to_frontier + bytes + checksum + created_at + +asset chunks + asset_id + content_hash + mime_type + byte_length + bytes + metadata + +revision index chunks + revision_id + frontier + title + summary + author_user_id + replica_id + created_at + +projection cache chunks + frontier + bytes + +search unit chunks + frontier + unit_id + unit_kind + heading_path + heading + body + insert_text + paragraph_start_cursor + paragraph_end_cursor +``` + +Only Loro snapshot/update data and asset bytes/references are canonical. + +Projection caches and search units are disposable. + +## 20. Read Path + +Opening a document: + +```text +open package +read manifest +load latest complete Loro snapshot +apply complete update segments after snapshot +construct LoroDoc in CRDT runtime +verify package format version +verify Loro schema version +verify document lineage/integrity +load projection cache if frontier matches +otherwise rebuild projection from Loro +load search cache if frontier matches +otherwise rebuild search units from projection/Loro +open renderer on DocumentProjection +``` + +The renderer never reads the old final-state document format as authoritative state. There is no legacy `.db8` reader in the target app. + +DOCX import is external import, not document storage. + +## 21. Write Path + +Saving is append-first and crash-safe. + +On each committed local Loro change: + +```text +receive local update bytes +append update segment in package transaction +record new frontier +update manifest +invalidate stale projection/search caches +schedule snapshot compaction if thresholds are crossed +``` + +Periodically, or when update history crosses thresholds: + +```text +export fresh Loro snapshot +write snapshot +mark older update segments compactable if revision policy permits +write projection cache for fast open +write search projection cache for tub/search +update manifest transactionally +``` + +A committed update segment must either be fully visible or ignored. The manifest must point only to complete, verified segments. + +## 22. Sync And Anti-Entropy + +Remote updates are imported into the CRDT runtime's `LoroDoc`. + +Remote updates must not be translated into Flowstate operations. + +Import status must be consumed. If import reports missing/pending dependencies, Flowstate should immediately trigger update pull/anti-entropy rather than waiting only for periodic digest. + +Periodic digest can remain as a fallback liveness mechanism, but the primary gap response should be actual Loro version-vector/update anti-entropy. + +Target path: + +```text +receive remote update bytes +import into LoroDoc +read ImportStatus +if pending/missing: request updates from peers immediately +Loro event reaches permanent subscription +project changed containers/ranges +persist accepted update +sync any newly generated update as needed +``` + +The only acceptable "patch" layer is a derived UI projection diff layer. There should be no `RemoteApplier` whose purpose is reconstructing canonical Flowstate document patches. + +## 23. Permanent Loro Subscription + +Temporary subscriptions around import/undo are not acceptable. + +The CRDT runtime must own a permanent subscription for each live document and filter/process events by origin, trigger, current frontier, and runtime epoch. + +This avoids relying on synchronous event timing during `import`, `checkout`, or `undo`. + +The subscription should feed: + +* projection invalidation +* local update publish +* persistence append +* search projection invalidation +* asset reachability checks +* undo/redo selection restoration +* revision/frontier status + +## 24. Projection Invalidation + +Projection invalidation is a core part of the architecture. + +Open design decision: + +| Option | Runtime perf | Correctness | Responsiveness | Complexity | P2P resilience | Summary | +|---|---:|---:|---:|---:|---:|---| +| A. Full projection rebuild after every Loro event | 3 | 10 | 3 | 4 | 8 | Simple and correct, but too slow for large documents and live collaboration. | +| B. Incremental projection from Loro diffs only | 9 | 7 | 9 | 9 | 7 | Fast but fragile if any event shape is missed. Needs strong fallback. | +| C. Batched frontier-based invalidation with incremental fast paths and full rebuild fallback | 9 | 10 | 9 | 8 | 10 | Preferred. Batch events per commit/frontier, apply known incremental invalidations, rebuild affected projection regions, and fall back to full rebuild on uncertainty. | +| D. Poll/diff snapshots after changes | 4 | 8 | 4 | 5 | 6 | Avoids event complexity but wastes Loro's event model and increases latency. | + +Current recommendation: Option C. + +The runtime should maintain projection indexes: + +* flow text boundary index +* paragraph range index +* paragraph metadata index +* block anchor index +* object placeholder index +* table row/column/cell index +* style interval index +* section anchor index +* asset reference index +* search unit index +* cursor resolution cache + +For each Loro event batch, the runtime should produce: + +```text +ProjectionInvalidation + frontier_before + frontier_after + changed_flows + changed_text_ranges + changed_blocks + changed_tables + changed_assets + changed_sections + rebuild_required flag +``` + +The UI receives stable projection diffs or a new projection snapshot. It never applies canonical document mutations itself. + +Full projection rebuild is an exceptional fallback, not an ordinary edit path. The runtime must make fallback use observable with structured logging/counters. Fallback should be treated as a performance bug if it occurs repeatedly during normal typing, formatting, table editing, image edits, or remote update import. + +The projection system should define explicit incremental paths for all expected common operations. Fallback is reserved for unknown event shapes, detected projection corruption, schema migration, recovery, or rare defensive repair. + +## 25. Search + +Within-document and tub search must be ported to the Loro-native architecture. + +The current tub architecture: + +* catalogs files in SQLite +* indexes searchable units in Tantivy +* extracts section/paragraph units from old final-state `.db8` +* hydrates previews from the old document model + +The new architecture should: + +* index Flowstate `.db8` Loro package files instead of old final-state `.db8` +* derive search units from `LoroDoc`/`DocumentProjection` +* preserve section-aware units such as block/tag/analytic/card/cite where those style semantics still exist +* store search unit cursors or paragraph IDs instead of raw old paragraph indexes as canonical references +* use raw paragraph indexes only as projection-local convenience values +* hydrate previews from the Loro projection or package search cache +* continue using Tantivy for workspace/tub full-text search unless a clearly superior crate replaces it + +Search cache data inside a package is disposable but should be saved by default for fast unopened-file indexing. The external tub index is also disposable. Both must be rebuildable from Loro package contents. + +## 26. DOCX, PDF, And Export + +DOCX import must be migrated onto the Loro-native model immediately. + +DOCX import path: + +```text +DOCX + -> interpreter/import heuristics + -> semantic Flowstate style model + -> Loro-native document creation + -> DocumentProjection + -> renderer +``` + +DOCX export, PDF export, and other output formats are generated from a Loro snapshot/frontier projection. + +Exports should not expose or require document history. Saving a Flowstate document saves the Loro package with history; exporting produces a final-state external artifact. + +PDF source recovery, when present, must embed the new Loro-native `.db8` package bytes only. It must not embed or recover the previous development `.db8` serializer payload, and it must not maintain dual old/new recovery formats. + +The only permitted verification fixtures in this plan are import/export/render regression fixtures for DOCX/PDF/export behavior. The broader architecture plan should not specify a general test suite at this stage. + +## 27. Schema Versioning + +The root Loro document should include explicit schema metadata: + +```text +MetaMap + document_id + loro_schema_version + schema_features + created_by_app_version + last_written_by_app_version + created_at + modified_at +``` + +This is not backwards compatibility for old `.db8`. It is forward compatibility for future Loro-native schema evolution. + +Open decision: whether migrations are stored as explicit revision/history records or only as package metadata. Current recommendation is to record schema migration events in package metadata and, if they mutate the Loro document, make those mutations ordinary Loro changes with a migration origin. + +## 28. Container References + +Container references must store both durable Flowstate IDs and raw Loro container IDs. + +The durable Flowstate ID is the semantic identity. The raw Loro container ID is stored for direct resolution and efficient runtime access. + +Example: + +```text +CellMap + flow_id: FlowId + +flows_by_id[flow_id] + text: LoroText +``` + +The schema should not depend on user-facing code hand-authoring raw `ContainerID` strings. Container resolution should be centralized in the CRDT runtime/schema module. + +## 29. Clear/Rebuild Paths + +The current code has a `clear_blocks` path because it sometimes rebuilds the Loro block list from the old `Document` model. That is a symptom of the current dual-model architecture. + +In the target architecture, full clear-and-repopulate paths should be rare. Normal editing should mutate Loro incrementally. + +When a legitimate full rebuild is needed, such as creating a new document from DOCX import or replacing a disposable projection cache, use Loro's exposed clear API for the relevant container instead of deleting one item at a time. + +Do not use full clear/repopulate as a normal local edit path, because it destroys useful incremental history and creates poor collaboration semantics. + +## 30. Invariants + +The architecture must preserve these invariants: + +1. A document can be fully reconstructed from the Loro snapshot/update history plus asset store. +2. Flowstate render/layout/search/export caches are disposable. +3. Paragraph split and inline text insertion commute correctly. +4. Undo/redo is based on grouped Loro changes and is per-peer/per-replica. +5. Time travel is based on Loro frontiers, not undo stack state. +6. Restoring a historical revision opens a dirty fork/branch tab rather than destructively replacing current history. +7. Binary assets are referenced by Loro metadata and stored outside the CRDT operation log. +8. Image captions and alt text are CRDT text flows. +9. Equation source is a CRDT text flow. +10. Rich inline formatting uses semantic Loro marks. +11. Paragraph style lives in Loro paragraph-boundary marks, not in competing paragraph metadata. +12. Paragraph/object/table metadata lives in Loro maps/lists. +13. Tables are structured CRDT objects, not binary blobs. +14. Rows and columns have durable Loro-native identity. +15. Remote updates are imported into Loro directly. +16. Local commands mutate Loro through the CRDT runtime. +17. There is no canonical Flowstate document separate from Loro. +18. There is no local-vs-remote document mutation split. +19. Files are Loro-backed packages, not final-state-only serialized documents. +20. Projection caches declare the Loro frontier they represent. +21. Search indexes declare the Loro frontier they represent. +22. DOCX/PDF/export operate from Loro snapshot projections. +23. Comments, annotations, bookmarks, and suggestion systems are out of scope for this architecture pass. +24. General test suite requirements are out of scope for this plan; only DOCX/PDF/export regression fixtures may be specified. + +## 31. Data Shape Summary + +Root: + +```text +RootMap + meta: MetaMap + flows_by_id: LoroMap + blocks_by_id: LoroMap + paragraphs_by_id: LoroMap + sections_by_id: LoroMap + assets_by_id: LoroMap + revisions: LoroList + users_by_id: LoroMap + replicas_by_id: LoroMap +``` + +Flow: + +```text +FlowMap + id + kind + text: LoroText + attrs +``` + +Paragraph: + +```text +ParagraphMap + id + flow_id + start_cursor + boundary_cursor + attrs +``` + +Object block: + +```text +BlockMap + id + kind + flow_id + anchor_cursor + attrs + nested_refs +``` + +Table: + +```text +TableBlock + row_order: LoroMovableList + rows_by_id: LoroMap + column_order: LoroMovableList + columns_by_id: LoroMap + cells_by_id: LoroMap + +CellMap + row_id + column_id + row_span + column_span + flow_id + attrs +``` + +Image: + +```text +ImageBlock + asset_id + content_hash + attrs + alt_text_flow_id + caption_flow_id +``` + +Equation: + +```text +EquationBlock + source_flow_id + attrs +``` + +Asset: + +```text +AssetMap + asset_id + content_hash + mime_type + byte_length + dimensions + metadata +``` + +Revision: + +```text +RevisionMap + id + timestamp + author_user_id + replica_id + frontier + title + summary +``` + +## 32. Expected Result + +The final architecture should make local editing, collaboration, persistence, undo/redo, revision history, branching, import/export, and search all expressions of the same underlying Loro document history. + +There should no longer be a special collaboration document model. + +There should no longer be a normal local document model. + +There should be one document model: + +```text +LoroDoc + asset store + derived Flowstate projections +``` + +Flowstate becomes the editor, renderer, importer, exporter, and search/projector for a Loro-native rich document, not the owner of a separate document format that Loro shadows. diff --git a/crates/flowstate-collab/Cargo.toml b/crates/flowstate-collab/Cargo.toml index 22261202..e73b0d7f 100644 --- a/crates/flowstate-collab/Cargo.toml +++ b/crates/flowstate-collab/Cargo.toml @@ -7,6 +7,7 @@ publish.workspace = true [dependencies] anyhow = { workspace = true } async-channel = { workspace = true } +flowstate-document.workspace = true gpui-flowtext = { workspace = true } iroh = { workspace = true } iroh-gossip = { workspace = true } @@ -23,6 +24,7 @@ uuid = { workspace = true } [dev-dependencies] proptest = { workspace = true } +tempfile.workspace = true [lints] workspace = true diff --git a/crates/flowstate-collab/src/binding.rs b/crates/flowstate-collab/src/binding.rs deleted file mode 100644 index 7e3df165..00000000 --- a/crates/flowstate-collab/src/binding.rs +++ /dev/null @@ -1,255 +0,0 @@ -//! Local flowtext identity bindings for shared CRDT containers. - -use std::collections::HashMap; - -use anyhow::{Context as _, Result, bail}; -use gpui_flowtext::{Block, BlockId, Document, ParagraphId, paragraph_text_len}; -use loro::{Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, ValueOrContainer}; - -use crate::{ - body_index::{BodyParagraphIndex, paragraph_lens_from_body_text}, - schema::{BLOCKS, body_text}, -}; - -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum BlockKind { - Paragraph, - Image, - Equation, - Table, -} - -#[derive(Clone, Debug)] -pub struct BindingRow { - pub map: LoroMap, - pub kind: BlockKind, - pub block_id: BlockId, - pub paragraph_id: Option, - pub version: u64, -} - -#[derive(Clone, Debug, Default)] -pub struct DocBinding { - pub rows: Vec, - pub by_paragraph: HashMap, - pub by_block: HashMap, - pub by_container: HashMap, - /// Paragraph ordinal -> body byte-range index, kept in sync with the body text - /// so paragraph offset lookups never stringify the whole body. - pub body_index: BodyParagraphIndex, -} - -impl DocBinding { - pub fn build(doc: &LoroDoc, document: &Document) -> Result { - let blocks = doc.get_movable_list(BLOCKS); - if blocks.len() != document.blocks.len() { - bail!( - "binding row count mismatch: loro has {} rows, document has {} blocks", - blocks.len(), - document.blocks.len() - ); - } - - let mut binding = Self::default(); - let mut paragraph_ix = 0; - for (row_ix, block) in document.blocks.iter().enumerate() { - let map = map_at(&blocks, row_ix)?; - let kind = BlockKind::from_block(block); - let paragraph_id = if matches!(kind, BlockKind::Paragraph) { - let id = document - .ids - .paragraph_ids - .get(paragraph_ix) - .copied() - .context("document is missing a paragraph id for a paragraph block")?; - paragraph_ix += 1; - Some(id) - } else { - None - }; - let block_id = document - .ids - .block_ids - .get(row_ix) - .copied() - .context("document is missing a block id for a block row")?; - binding.push_row(BindingRow { - map, - kind, - block_id, - paragraph_id, - version: block_version(block), - }); - } - binding.refresh_body_index(document); - binding.assert_consistent(doc, document); - Ok(binding) - } - - /// Rebuilds the paragraph byte-offset index from the document's paragraphs. - /// Used on build and on the O(n) structural paths (block insert/delete/move) - /// that already rewrite the whole body; the per-keystroke paths update the - /// index incrementally instead. - pub fn refresh_body_index(&mut self, document: &Document) { - self.body_index = BodyParagraphIndex::from_lens(document.paragraphs.iter().map(paragraph_text_len).collect()); - } - - /// Rebuilds the paragraph byte-offset index from the live Loro body text. - /// Used by the remote applier after a full body reprojection so the index - /// keeps matching the freshly imported body (O(body length)). - pub fn refresh_body_index_from_loro(&mut self, doc: &LoroDoc) { - self.body_index = BodyParagraphIndex::from_lens(paragraph_lens_from_body_text(&body_text(doc))); - } - - pub fn assert_consistent(&self, doc: &LoroDoc, document: &Document) { - let blocks = doc.get_movable_list(BLOCKS); - debug_assert_eq!(self.rows.len(), blocks.len()); - debug_assert_eq!(self.rows.len(), document.blocks.len()); - - let mut paragraph_ix = 0; - for (row_ix, row) in self.rows.iter().enumerate() { - let Some(block) = document.blocks.get(row_ix) else { - debug_assert!(false, "binding row has no matching document block"); - continue; - }; - debug_assert_eq!( - row.kind, - BlockKind::from_block(block), - "binding row {row_ix} kind does not match document block" - ); - debug_assert_eq!( - document.ids.block_ids.get(row_ix).copied(), - Some(row.block_id), - "binding row {row_ix} block id does not match document block id" - ); - debug_assert_eq!(row.version, block_version(block)); - if matches!(row.kind, BlockKind::Paragraph) { - debug_assert_eq!(document.ids.paragraph_ids.get(paragraph_ix).copied(), row.paragraph_id); - paragraph_ix += 1; - } else { - debug_assert!(row.paragraph_id.is_none()); - } - } - - debug_assert_eq!( - self.body_index.len(), - paragraph_ix, - "body index paragraph count does not match paragraph rows" - ); - #[cfg(debug_assertions)] - { - let body_lens = paragraph_lens_from_body_text(&body_text(doc)); - debug_assert_eq!( - self.body_index.len(), - body_lens.len(), - "body index paragraph count does not match the body text" - ); - for (ordinal, expected) in body_lens.iter().enumerate() { - debug_assert_eq!( - self.body_index.paragraph_len(ordinal), - *expected, - "body index byte length mismatch vs body at paragraph {ordinal}" - ); - debug_assert_eq!( - self.body_index.paragraph_len(ordinal), - document - .paragraphs - .get(ordinal) - .map_or(0, paragraph_text_len), - "body index byte length mismatch vs document at paragraph {ordinal}" - ); - } - } - } - - pub fn push_row(&mut self, row: BindingRow) { - let ix = self.rows.len(); - self.index_row(ix, &row); - self.rows.push(row); - } - - pub fn insert_row(&mut self, ix: usize, row: BindingRow) { - let ix = ix.min(self.rows.len()); - self.rows.insert(ix, row); - self.rebuild_indexes(); - } - - pub fn remove_row(&mut self, ix: usize) -> Option { - if ix >= self.rows.len() { - return None; - } - let row = self.rows.remove(ix); - self.rebuild_indexes(); - Some(row) - } - - pub fn move_row(&mut self, from: usize, to: usize) { - if from >= self.rows.len() || from == to { - return; - } - let row = self.rows.remove(from); - self.rows.insert(to.min(self.rows.len()), row); - self.rebuild_indexes(); - } - - pub fn refresh_versions(&mut self, document: &Document) { - for (row, block) in self.rows.iter_mut().zip(document.blocks.iter()) { - row.version = block_version(block); - } - } - - pub fn rebuild_indexes(&mut self) { - let mut by_paragraph = HashMap::new(); - let mut by_block = HashMap::new(); - let mut by_container = HashMap::new(); - for (ix, row) in self.rows.iter().enumerate() { - by_block.insert(row.block_id, ix); - by_container.insert(row.map.id(), ix); - if let Some(paragraph_id) = row.paragraph_id { - by_paragraph.insert(paragraph_id, ix); - } - } - self.by_paragraph = by_paragraph; - self.by_block = by_block; - self.by_container = by_container; - } - - fn index_row(&mut self, ix: usize, row: &BindingRow) { - self.by_block.insert(row.block_id, ix); - self.by_container.insert(row.map.id(), ix); - if let Some(paragraph_id) = row.paragraph_id { - self.by_paragraph.insert(paragraph_id, ix); - } - } -} - -#[must_use] -pub const fn block_version(block: &Block) -> u64 { - match block { - Block::Paragraph(paragraph) => paragraph.version, - Block::Image(image) => image.version, - Block::Equation(equation) => equation.version, - Block::Table(table) => table.version, - } -} - -impl BlockKind { - #[must_use] - pub const fn from_block(block: &Block) -> Self { - match block { - Block::Paragraph(_) => Self::Paragraph, - Block::Image(_) => Self::Image, - Block::Equation(_) => Self::Equation, - Block::Table(_) => Self::Table, - } - } -} - -fn map_at(blocks: &LoroMovableList, ix: usize) -> Result { - match blocks.get(ix) { - Some(ValueOrContainer::Container(Container::Map(map))) => Ok(map), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("binding row {ix} is not a map container") - }, - } -} diff --git a/crates/flowstate-collab/src/body_index.rs b/crates/flowstate-collab/src/body_index.rs deleted file mode 100644 index e077e57e..00000000 --- a/crates/flowstate-collab/src/body_index.rs +++ /dev/null @@ -1,294 +0,0 @@ -//! Maintained paragraph -> body byte-offset index for the single-root body text. -//! -//! In the single-root model the whole document lives in one `LoroText` ("body") -//! whose paragraphs are separated by `'\n'`. Mapping a paragraph ordinal to its -//! byte range in the body used to require stringifying the entire body and -//! linear-scanning for the Nth newline on every edit. [`BodyParagraphIndex`] -//! replaces that with a Fenwick (binary indexed) tree over per-paragraph body -//! widths, mirroring `gpui_flowtext::ParagraphOffsetIndex`, so paragraph -> byte -//! lookups are O(log n) and content edits update in O(log n). - -use std::ops::Range; - -use loro::LoroText; - -use crate::schema::text_content; - -/// Fenwick tree over per-paragraph body byte widths plus the raw paragraph byte -/// lengths. -/// -/// `width(k) = len(k) + (1 if k is not the last paragraph else 0)` — the trailing -/// `'\n'` separator is charged to every paragraph except the last, so the prefix -/// sum of widths over the paragraphs before `k` is exactly `k`'s start byte in the -/// body. -#[derive(Clone, Debug, Default)] -pub struct BodyParagraphIndex { - /// UTF-8 byte length of each paragraph's text (no separator newline). - lens: Vec, - /// 1-indexed Fenwick tree over paragraph widths. - tree: Vec, -} - -impl BodyParagraphIndex { - /// Builds an index from per-paragraph UTF-8 byte lengths (in body order). - #[must_use] - pub fn from_lens(lens: Vec) -> Self { - let mut index = Self { - tree: vec![0; lens.len() + 1], - lens, - }; - for ordinal in 0..index.lens.len() { - let width = index.width(ordinal); - index.add(ordinal, width as isize); - } - index - } - - /// Number of paragraphs tracked by the index. - #[must_use] - pub fn len(&self) -> usize { - self.lens.len() - } - - #[must_use] - pub fn is_empty(&self) -> bool { - self.lens.is_empty() - } - - /// Byte offset where paragraph `ordinal` starts in the body text. - #[must_use] - pub fn paragraph_start(&self, ordinal: usize) -> usize { - self.prefix_sum(ordinal) - } - - /// UTF-8 byte length of paragraph `ordinal` (excluding the separator newline). - #[must_use] - pub fn paragraph_len(&self, ordinal: usize) -> usize { - self.lens.get(ordinal).copied().unwrap_or(0) - } - - /// Byte range `[start, start + len)` of paragraph `ordinal` in the body text. - #[must_use] - pub fn paragraph_range(&self, ordinal: usize) -> Range { - let start = self.paragraph_start(ordinal); - start..start + self.paragraph_len(ordinal) - } - - /// Total body length implied by the index (sum of widths). - #[must_use] - pub fn total_len(&self) -> usize { - self.prefix_sum(self.lens.len()) - } - - /// Returns the ordinal of the paragraph that contains `byte` — i.e. the largest - /// ordinal whose start is `<= byte`. A byte sitting on a separator newline maps - /// to the paragraph it terminates. - #[must_use] - pub fn paragraph_ordinal_for_body_byte(&self, byte: usize) -> usize { - if self.lens.is_empty() { - return 0; - } - let mut low = 0usize; - let mut high = self.lens.len() - 1; - while low < high { - let mid = low + (high - low).div_ceil(2); - if self.paragraph_start(mid) <= byte { - low = mid; - } else { - high = mid - 1; - } - } - low - } - - /// Updates one paragraph's byte length in O(log n). The paragraph count must be - /// unchanged (use [`Self::insert_paragraph`] / [`Self::remove_paragraph`] for - /// structural changes). - pub fn set_paragraph_len(&mut self, ordinal: usize, new_len: usize) { - let Some(old) = self.lens.get(ordinal).copied() else { - return; - }; - if old == new_len { - return; - } - self.lens[ordinal] = new_len; - self.add(ordinal, new_len as isize - old as isize); - } - - /// Inserts a new paragraph of length `len` at `ordinal`, shifting later - /// paragraphs right. O(n) (rebuilds the tree, matching the editor's structural - /// path which is already O(n)). - pub fn insert_paragraph(&mut self, ordinal: usize, len: usize) { - let ordinal = ordinal.min(self.lens.len()); - self.lens.insert(ordinal, len); - self.rebuild(); - } - - /// Removes the paragraph at `ordinal`. O(n). - pub fn remove_paragraph(&mut self, ordinal: usize) { - if ordinal >= self.lens.len() { - return; - } - self.lens.remove(ordinal); - self.rebuild(); - } - - /// Removes `count` paragraphs starting at `ordinal` in one rebuild. O(n). - pub fn remove_paragraph_range(&mut self, ordinal: usize, count: usize) { - let start = ordinal.min(self.lens.len()); - let end = start.saturating_add(count).min(self.lens.len()); - if start >= end { - return; - } - self.lens.drain(start..end); - self.rebuild(); - } - - fn rebuild(&mut self) { - let lens = std::mem::take(&mut self.lens); - *self = Self::from_lens(lens); - } - - fn width(&self, ordinal: usize) -> usize { - self.lens[ordinal] + usize::from(ordinal + 1 < self.lens.len()) - } - - fn add(&mut self, ordinal: usize, delta: isize) { - if delta == 0 { - return; - } - let mut ix = ordinal + 1; - while ix < self.tree.len() { - self.tree[ix] = self.tree[ix].saturating_add_signed(delta); - ix += ix & (!ix + 1); - } - } - - fn prefix_sum(&self, paragraph_count: usize) -> usize { - let mut ix = paragraph_count.min(self.lens.len()); - let mut sum = 0; - while ix > 0 { - sum += self.tree[ix]; - ix &= ix - 1; - } - sum - } -} - -/// Scans the body text once and returns each paragraph's UTF-8 byte length (body -/// order). Used to (re)build [`BodyParagraphIndex`] and to debug-verify it. -/// -/// This is O(body length); it is only meant for build / structural / debug paths, -/// never the per-keystroke hot path. -#[must_use] -pub fn paragraph_lens_from_body_text(text: &LoroText) -> Vec { - text_content(text).split('\n').map(str::len).collect() -} - -/// Length of the shared UTF-8 prefix of `left` and `right`, on a char boundary. -#[must_use] -pub fn common_prefix_bytes(left: &str, right: &str) -> usize { - let mut prefix = 0usize; - for (left_ch, right_ch) in left.chars().zip(right.chars()) { - if left_ch != right_ch { - break; - } - prefix += left_ch.len_utf8(); - } - prefix -} - -/// Length of the shared UTF-8 suffix of `left` and `right`, on a char boundary. -#[must_use] -pub fn common_suffix_bytes(left: &str, right: &str) -> usize { - let mut suffix = 0usize; - for (left_ch, right_ch) in left.chars().rev().zip(right.chars().rev()) { - if left_ch != right_ch { - break; - } - let len = left_ch.len_utf8(); - if suffix + len > left.len() || suffix + len > right.len() { - break; - } - suffix += len; - } - suffix -} - -/// Minimal byte splice turning `old` into `new`: the shared prefix length, the -/// length of the differing middle of `old` to delete, and the differing middle of -/// `new` to insert. Returns byte offsets relative to the start of `old`/`new`. -#[must_use] -pub fn minimal_utf8_splice(old: &str, new: &str) -> (usize, usize, Range) { - let prefix = common_prefix_bytes(old, new); - let suffix = common_suffix_bytes(&old[prefix..], &new[prefix..]); - let old_middle = old.len().saturating_sub(prefix + suffix); - let new_middle = prefix..new.len().saturating_sub(suffix); - (prefix, old_middle, new_middle) -} - -#[cfg(test)] -mod tests { - use super::*; - - fn fresh_start(lens: &[usize], ordinal: usize) -> usize { - lens.iter().take(ordinal).map(|len| len + 1).sum() - } - - #[test] - fn starts_and_ranges_match_a_fresh_scan() { - let lens = vec![3, 0, 5, 2, 7]; - let index = BodyParagraphIndex::from_lens(lens.clone()); - for ordinal in 0..lens.len() { - assert_eq!(index.paragraph_start(ordinal), fresh_start(&lens, ordinal)); - assert_eq!( - index.paragraph_range(ordinal), - fresh_start(&lens, ordinal)..fresh_start(&lens, ordinal) + lens[ordinal] - ); - } - // Total length has no trailing newline after the last paragraph. - let total: usize = lens.iter().sum::() + lens.len() - 1; - assert_eq!(index.total_len(), total); - } - - #[test] - fn random_edits_keep_starts_consistent() { - let mut lens = vec![4, 4, 4, 4]; - let mut index = BodyParagraphIndex::from_lens(lens.clone()); - - index.set_paragraph_len(1, 10); - lens[1] = 10; - index.insert_paragraph(2, 3); - lens.insert(2, 3); - index.remove_paragraph(0); - lens.remove(0); - index.set_paragraph_len(index.len() - 1, 0); - *lens.last_mut().unwrap() = 0; - - for ordinal in 0..lens.len() { - assert_eq!(index.paragraph_start(ordinal), fresh_start(&lens, ordinal), "start mismatch at {ordinal}"); - assert_eq!(index.paragraph_len(ordinal), lens[ordinal]); - } - } - - #[test] - fn ordinal_lookup_finds_containing_paragraph() { - let lens = vec![2, 3, 4]; - let index = BodyParagraphIndex::from_lens(lens); - // body: "aa\nbbb\ncccc" -> starts 0, 3, 7 - assert_eq!(index.paragraph_ordinal_for_body_byte(0), 0); - assert_eq!(index.paragraph_ordinal_for_body_byte(2), 0); // newline after p0 - assert_eq!(index.paragraph_ordinal_for_body_byte(3), 1); - assert_eq!(index.paragraph_ordinal_for_body_byte(6), 1); // newline after p1 - assert_eq!(index.paragraph_ordinal_for_body_byte(7), 2); - assert_eq!(index.paragraph_ordinal_for_body_byte(100), 2); - } - - #[test] - fn minimal_splice_picks_changed_middle() { - let (prefix, old_middle, new_middle) = minimal_utf8_splice("abcXYZdef", "abcQdef"); - assert_eq!(prefix, "abc".len()); - assert_eq!(old_middle, "XYZ".len()); - assert_eq!(new_middle, "abc".len().."abcQ".len()); - } -} diff --git a/crates/flowstate-collab/src/crdt_runtime.rs b/crates/flowstate-collab/src/crdt_runtime.rs new file mode 100644 index 00000000..50bf293f --- /dev/null +++ b/crates/flowstate-collab/src/crdt_runtime.rs @@ -0,0 +1,994 @@ +use std::{ + io, + path::{Path, PathBuf}, + sync::Arc, +}; + +use anyhow::{Context as _, Result}; +use flowstate_document::{ + AssetId, AssetRecord, BLOCKS_BY_ID, Document, DocumentPackage, FLOW_ATTRS_KEY, FLOW_ID_KEY, FLOW_KIND_KEY, FLOW_TEXT_KEY, FLOWS_BY_ID, + InputBlockAlignment, InputEquationDisplay, InputImageSizing, InputTableColumnWidth, MARK_DIRECT_UNDERLINE, MARK_HIGHLIGHT_STYLE, + MARK_PARAGRAPH_STYLE, MARK_RUN_SEMANTIC_STYLE, MARK_STRIKETHROUGH, OBJECT_REPLACEMENT, ParagraphStyle, ROOT, ROOT_BODY_FLOW_ID, + RunSemanticStyle, RunStyles, SENTINEL_NEWLINE, document_from_loro, + loro_schema::body_text, + new_loro_document, +}; +use gpui_flowtext::SemanticEditCommand as EditorSemanticCommand; +use loro::{ExportMode, Frontiers, ImportStatus, LoroDoc, LoroMap, Subscription, UndoManager, VersionRange, VersionVector, cursor::Side, event::DiffEvent}; +use uuid::Uuid; + +#[derive(Debug)] +pub struct CrdtRuntime { + doc: LoroDoc, + undo: UndoManager, + package: Option, + package_path: Option, + last_persisted_frontier: Frontiers, + last_persisted_vv: VersionVector, + _root_subscription: Subscription, + _local_update_subscription: Subscription, +} + +#[derive(Clone, Debug)] +pub enum SemanticCommand { + InsertText { + unicode_index: usize, + text: String, + }, + DeleteRange { + unicode_index: usize, + unicode_len: usize, + }, + SplitParagraph { + unicode_index: usize, + inherited_style: ParagraphStyle, + }, + SetParagraphStyle { + boundary_unicode_index: usize, + style: ParagraphStyle, + }, + SetRunStyles { + unicode_range: std::ops::Range, + styles: RunStyles, + }, + InsertImage { + unicode_index: usize, + asset_id: u128, + alt_text: String, + caption: Option, + sizing: InputImageSizing, + alignment: InputBlockAlignment, + }, + InsertEquation { + unicode_index: usize, + source: String, + display: InputEquationDisplay, + }, + InsertTable { + unicode_index: usize, + rows: usize, + columns: usize, + column_widths: Vec, + header_row: bool, + }, + OpenRevision { + revision_id: u128, + }, + ForkRevision { + revision_id: u128, + }, + Undo, + Redo, +} + +#[derive(Debug)] +pub enum RuntimeEvent { + LocalUpdate { + bytes: Vec, + frontier: Vec, + version_vector: Vec, + }, + RemoteUpdateApplied { + pending: Option, + frontier: Vec, + version_vector: Vec, + }, + RevisionOpened { + revision_id: u128, + document: Box, + }, + RevisionForked { + revision_id: u128, + runtime: Box, + }, + ProjectionUpdated { + document: Box, + frontier: Vec, + version_vector: Vec, + }, +} + +impl CrdtRuntime { + pub fn new_empty(title: &str) -> Result { + let doc = new_loro_document(title).context("initializing Loro document")?; + Self::from_doc(doc, None, None) + } + + pub fn open_package(path: impl AsRef) -> Result { + let path = path.as_ref(); + let package = DocumentPackage::read(path).with_context(|| format!("reading Flowstate package {}", path.display()))?; + let doc = package.load_loro_doc().context("loading Loro document from package")?; + Self::from_doc(doc, Some(package), Some(path.to_path_buf())) + } + + pub fn from_doc(doc: LoroDoc, package: Option, package_path: Option) -> Result { + let last_persisted_frontier = doc.state_frontiers(); + let last_persisted_vv = doc.state_vv(); + let root_subscription = doc.subscribe_root(Arc::new(|event: DiffEvent<'_>| { + tracing::trace!(origin = ?event.origin, trigger = ?event.triggered_by, "Flowstate Loro root event"); + })); + let local_update_subscription = doc.subscribe_local_update(Box::new(|bytes| { + tracing::trace!(bytes = bytes.len(), "Flowstate Loro local update"); + true + })); + let mut undo = UndoManager::new(&doc); + undo.set_merge_interval(600); + undo.add_exclude_origin_prefix("remote"); + Ok(Self { + doc, + undo, + package, + package_path, + last_persisted_frontier, + last_persisted_vv, + _root_subscription: root_subscription, + _local_update_subscription: local_update_subscription, + }) + } + + pub fn doc(&self) -> &LoroDoc { + &self.doc + } + + pub fn projection_snapshot(&self) -> Result { + let mut document = document_from_loro(&self.doc).context("projecting Flowstate document from canonical Loro state")?; + if let Some(package) = &self.package { + attach_package_assets(&mut document, package); + } + Ok(document) + } + + pub fn command(&mut self, command: SemanticCommand) -> Result> { + let from_frontier = self.doc.state_frontiers(); + let from_vv = self.doc.state_vv(); + match command { + SemanticCommand::InsertText { unicode_index, text } => { + let body = body_text(&self.doc); + body.insert(unicode_index, &text).context("inserting text into Loro body flow")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + SemanticCommand::DeleteRange { + unicode_index, + unicode_len, + } => { + if unicode_len > 0 { + let body = body_text(&self.doc); + body + .delete(unicode_index, unicode_len) + .context("deleting text from Loro body flow")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + } + SemanticCommand::SplitParagraph { + unicode_index, + inherited_style, + } => { + let body = body_text(&self.doc); + body.insert(unicode_index, "\n").context("splitting Loro body paragraph")?; + body + .mark(unicode_index..unicode_index + 1, MARK_PARAGRAPH_STYLE, paragraph_style_value(inherited_style)) + .context("marking split paragraph boundary")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + SemanticCommand::SetParagraphStyle { + boundary_unicode_index, + style, + } => { + let body = body_text(&self.doc); + body + .mark( + boundary_unicode_index..boundary_unicode_index + 1, + MARK_PARAGRAPH_STYLE, + paragraph_style_value(style), + ) + .context("marking paragraph style in Loro body flow")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + SemanticCommand::SetRunStyles { unicode_range, styles } => { + mark_run_styles(&body_text(&self.doc), unicode_range, styles).context("marking run styles in Loro body flow")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + SemanticCommand::InsertImage { + unicode_index, + asset_id, + alt_text, + caption, + sizing, + alignment, + } => { + insert_image_block(&self.doc, unicode_index, asset_id, &alt_text, caption.as_deref(), sizing, alignment) + .context("inserting image block into Loro document")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + SemanticCommand::InsertEquation { + unicode_index, + source, + display, + } => { + insert_equation_block(&self.doc, unicode_index, &source, display).context("inserting equation block into Loro document")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + SemanticCommand::InsertTable { + unicode_index, + rows, + columns, + column_widths, + header_row, + } => { + insert_table_block(&self.doc, unicode_index, rows, columns, &column_widths, header_row) + .context("inserting table block into Loro document")?; + self.doc.commit(); + self.undo.record_new_checkpoint().context("recording Loro undo checkpoint")?; + } + SemanticCommand::OpenRevision { revision_id } => { + let document = self.revision_projection(revision_id)?; + return Ok(vec![RuntimeEvent::RevisionOpened { + revision_id, + document: Box::new(document), + }]); + } + SemanticCommand::ForkRevision { revision_id } => { + let fork = self.fork_revision_runtime(revision_id)?; + return Ok(vec![RuntimeEvent::RevisionForked { + revision_id, + runtime: Box::new(fork), + }]); + } + SemanticCommand::Undo => { + self.undo.undo().context("applying Loro undo")?; + } + SemanticCommand::Redo => { + self.undo.redo().context("applying Loro redo")?; + } + } + self.events_after_local_change(from_frontier, from_vv) + } + + pub fn revision_projection(&self, revision_id: u128) -> Result { + let revision_doc = self + .package + .as_ref() + .context("cannot open revision without a package-backed runtime")? + .load_revision_loro_doc(revision_id) + .context("loading revision Loro snapshot")?; + let mut document = document_from_loro(&revision_doc).context("projecting revision document")?; + if let Some(package) = &self.package { + attach_package_assets(&mut document, package); + } + Ok(document) + } + + pub fn fork_revision_runtime(&self, revision_id: u128) -> Result { + let package = self.package.as_ref().context("cannot fork revision without a package-backed runtime")?; + let revision_doc = package + .load_revision_loro_doc(revision_id) + .context("loading revision Loro snapshot for fork")?; + let forked_doc = revision_doc.fork(); + let forked_package = DocumentPackage::from_loro_snapshot_with_assets(&forked_doc, "Forked revision", package.assets.clone()) + .context("creating forked revision package")?; + Self::from_doc(forked_doc, Some(forked_package), None) + } + + pub fn import_remote_update(&mut self, bytes: &[u8]) -> Result> { + let status = self.doc.import_with(bytes, "remote").context("importing remote Loro update")?; + let mut events = vec![RuntimeEvent::RemoteUpdateApplied { + pending: status.pending.clone(), + frontier: self.doc.state_frontiers().encode(), + version_vector: self.doc.state_vv().encode(), + }]; + events.push(self.projection_event()?); + if status.pending.is_none() { + self.persist_update_from_last_frontier()?; + } + Ok(events) + } + + fn projection_event(&self) -> Result { + Ok(RuntimeEvent::ProjectionUpdated { + document: Box::new(self.projection_snapshot()?), + frontier: self.doc.state_frontiers().encode(), + version_vector: self.doc.state_vv().encode(), + }) + } + + pub fn export_updates_for(&self, remote_vv: &VersionVector) -> Result> { + self + .doc + .export(ExportMode::updates(remote_vv)) + .context("exporting Loro updates for anti-entropy") + } + + pub fn missing_dependency_request(status: &ImportStatus) -> Option<&VersionRange> { + status.pending.as_ref() + } + + pub fn save_package(&self) -> io::Result<()> { + let Some(package) = &self.package else { + return Ok(()); + }; + let Some(path) = &self.package_path else { + return Ok(()); + }; + package.write(path) + } + + fn events_after_local_change(&mut self, from_frontier: Frontiers, from_vv: VersionVector) -> Result> { + let update = self + .doc + .export(ExportMode::updates(&from_vv)) + .context("exporting local Loro update")?; + self.persist_update_segment(from_frontier, from_vv, update.clone())?; + Ok(vec![ + RuntimeEvent::LocalUpdate { + bytes: update, + frontier: self.doc.state_frontiers().encode(), + version_vector: self.doc.state_vv().encode(), + }, + self.projection_event()?, + ]) + } + + fn persist_update_from_last_frontier(&mut self) -> Result<()> { + let from_frontier = self.last_persisted_frontier.clone(); + let from_vv = self.last_persisted_vv.clone(); + let update = self + .doc + .export(ExportMode::updates(&from_vv)) + .context("exporting accepted remote Loro update for persistence")?; + if update.is_empty() { + return Ok(()); + } + self.persist_update_segment(from_frontier, from_vv, update) + } + + fn persist_update_segment(&mut self, from_frontier: Frontiers, from_vv: VersionVector, update: Vec) -> Result<()> { + if let Some(package) = &mut self.package { + package.append_update_segment(&from_frontier, &from_vv, &self.doc.state_frontiers(), &self.doc.state_vv(), update)?; + if let Some(path) = &self.package_path { + package.write(path)?; + } + } + self.last_persisted_frontier = self.doc.state_frontiers(); + self.last_persisted_vv = self.doc.state_vv(); + Ok(()) + } +} + +pub fn apply_editor_semantic_command(doc: &LoroDoc, projection: &Document, command: &EditorSemanticCommand) -> Result { + match command { + EditorSemanticCommand::InsertText { at, text, styles } => { + let unicode_index = projection_offset_to_body_unicode_index(projection, *at); + let body = body_text(doc); + body + .insert(unicode_index, text) + .context("inserting projection-scoped text command into Loro body flow")?; + let inserted_len = text.chars().count(); + if inserted_len > 0 { + mark_run_styles(&body, unicode_index..unicode_index + inserted_len, *styles).context("marking inserted run styles")?; + } + doc.commit(); + Ok(true) + } + EditorSemanticCommand::DeleteRange { range } => { + let start = projection_offset_to_body_unicode_index(projection, range.start); + let end = projection_offset_to_body_unicode_index(projection, range.end); + if end > start { + body_text(doc) + .delete(start, end - start) + .context("deleting projection-scoped text range from Loro body flow")?; + doc.commit(); + return Ok(true); + } + Ok(false) + } + EditorSemanticCommand::SplitParagraph { + at, + inherited_style, + } => { + let unicode_index = projection_offset_to_body_unicode_index(projection, *at); + let body = body_text(doc); + body + .insert(unicode_index, "\n") + .context("splitting paragraph in Loro body flow")?; + body + .mark( + unicode_index..unicode_index + 1, + MARK_PARAGRAPH_STYLE, + paragraph_style_value(*inherited_style), + ) + .context("marking split paragraph style")?; + doc.commit(); + Ok(true) + } + EditorSemanticCommand::SetParagraphStyle { paragraph, style } => { + if let Some(paragraph_ix) = projection.ids.paragraph_ids.iter().position(|id| id == paragraph) { + let boundary = paragraph_boundary_unicode_index(projection, paragraph_ix); + body_text(doc) + .mark(boundary..boundary + 1, MARK_PARAGRAPH_STYLE, paragraph_style_value(*style)) + .context("marking paragraph style from editor semantic command")?; + doc.commit(); + return Ok(true); + } + Ok(false) + } + EditorSemanticCommand::SetRunStyles { + paragraph, + range, + styles, + } => { + if let Some(paragraph_ix) = projection.ids.paragraph_ids.iter().position(|id| id == paragraph) { + let start = projection_offset_to_body_unicode_index( + projection, + flowstate_document::DocumentOffset { + paragraph: paragraph_ix, + byte: range.start, + }, + ); + let end = projection_offset_to_body_unicode_index( + projection, + flowstate_document::DocumentOffset { + paragraph: paragraph_ix, + byte: range.end, + }, + ); + if end > start { + mark_run_styles(&body_text(doc), start..end, *styles).context("marking run styles from editor semantic command")?; + doc.commit(); + return Ok(true); + } + } + Ok(false) + } + EditorSemanticCommand::ReplaceParagraphSpan { before, after, .. } => { + replace_body_paragraph_span(doc, projection, before, after).context("replacing paragraph span from editor semantic command") + } + EditorSemanticCommand::ReplaceDocument => { + replace_entire_body_from_projection(doc, projection).context("replacing Loro body from projected document") + } + EditorSemanticCommand::JoinParagraphs { .. } + | EditorSemanticCommand::InsertBlock { .. } + | EditorSemanticCommand::DeleteBlock { .. } + | EditorSemanticCommand::MoveBlock { .. } + | EditorSemanticCommand::ReplaceBlock { .. } => Ok(false), + } +} + +fn projection_offset_to_body_unicode_index(projection: &Document, offset: flowstate_document::DocumentOffset) -> usize { + let mut unicode_index = 1; + for paragraph_ix in 0..offset.paragraph.min(projection.paragraphs.len()) { + unicode_index += flowstate_document::paragraph_text(projection, paragraph_ix) + .chars() + .count() + + 1; + } + if let Some(paragraph) = projection.paragraphs.get(offset.paragraph) { + let text = flowstate_document::paragraph_text(projection, offset.paragraph); + unicode_index += text[..offset.byte.min(paragraph.byte_range.len())].chars().count(); + } + unicode_index +} + +fn paragraph_boundary_unicode_index(projection: &Document, paragraph_ix: usize) -> usize { + if paragraph_ix == 0 { + return 0; + } + projection_offset_to_body_unicode_index( + projection, + flowstate_document::DocumentOffset { + paragraph: paragraph_ix, + byte: 0, + }, + ) - 1 +} + +fn replace_body_paragraph_span(doc: &LoroDoc, projection: &Document, before: &flowstate_document::DocumentSpan, after: &flowstate_document::DocumentSpan) -> Result { + let start = projection_offset_to_body_unicode_index( + projection, + flowstate_document::DocumentOffset { + paragraph: before.start_paragraph, + byte: 0, + }, + ); + let end_paragraph = before + .start_paragraph + .saturating_add(before.paragraphs.len()) + .min(projection.paragraphs.len()); + let end = if end_paragraph == 0 { + start + } else { + projection_offset_to_body_unicode_index( + projection, + flowstate_document::DocumentOffset { + paragraph: end_paragraph - 1, + byte: flowstate_document::paragraph_text(projection, end_paragraph - 1).len(), + }, + ) + }; + let replacement = after.text.clone(); + let body = body_text(doc); + if end > start { + body.delete(start, end - start)?; + } + if !replacement.is_empty() { + body.insert(start, &replacement)?; + } + doc.commit(); + Ok(true) +} + +fn replace_entire_body_from_projection(doc: &LoroDoc, projection: &Document) -> Result { + let body = body_text(doc); + let len = body.len_unicode(); + if len > 1 { + body.delete(1, len - 1)?; + } + let replacement = (0..projection.paragraphs.len()) + .map(|paragraph_ix| flowstate_document::paragraph_text(projection, paragraph_ix)) + .collect::>() + .join("\n"); + if !replacement.is_empty() { + body.insert(1, &replacement)?; + } + doc.commit(); + Ok(true) +} + +fn attach_package_assets(document: &mut Document, package: &DocumentPackage) { + for asset in &package.assets { + let bytes = asset.bytes.clone(); + document.assets.assets.insert( + AssetId(asset.asset_id), + AssetRecord { + id: AssetId(asset.asset_id), + mime_type: asset.mime_type.clone().into(), + original_name: None, + content_hash: AssetRecord::stable_content_hash(&bytes), + bytes: Arc::new(bytes), + }, + ); + } +} + +fn paragraph_style_value(style: ParagraphStyle) -> i64 { + match style { + ParagraphStyle::Normal => 0, + ParagraphStyle::Custom(slot) => i64::from(slot), + } +} + +fn mark_run_styles(text: &loro::LoroText, range: std::ops::Range, styles: RunStyles) -> loro::LoroResult<()> { + if let RunSemanticStyle::Custom(slot) = styles.semantic { + text.mark(range.clone(), MARK_RUN_SEMANTIC_STYLE, i64::from(slot))?; + } + if let Some(flowstate_document::HighlightStyle::Custom(slot)) = styles.highlight { + text.mark(range.clone(), MARK_HIGHLIGHT_STYLE, i64::from(slot))?; + } + if styles.direct_underline { + text.mark(range.clone(), MARK_DIRECT_UNDERLINE, true)?; + } + if styles.strikethrough { + text.mark(range, MARK_STRIKETHROUGH, true)?; + } + Ok(()) +} + +fn insert_image_block( + doc: &LoroDoc, + unicode_index: usize, + asset_id: u128, + alt_text: &str, + caption: Option<&str>, + sizing: InputImageSizing, + alignment: InputBlockAlignment, +) -> Result<()> { + let body = body_text(doc); + body.insert(unicode_index, &OBJECT_REPLACEMENT.to_string())?; + let block = ensure_block(doc, "image", ROOT_BODY_FLOW_ID, &body, unicode_index)?; + block.insert("asset_id", asset_id.to_string())?; + + let alt_flow_id = nested_flow_id("image_alt"); + block.insert("alt_text_flow_id", alt_flow_id.as_str())?; + let alt_flow = ensure_flow(doc, &alt_flow_id, "alt_text")?; + replace_text(&alt_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?, alt_text)?; + + if let Some(caption) = caption { + let caption_flow_id = nested_flow_id("image_caption"); + block.insert("caption_flow_id", caption_flow_id.as_str())?; + let caption_flow = ensure_flow(doc, &caption_flow_id, "caption")?; + let caption_text = caption_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + replace_text(&caption_text, SENTINEL_NEWLINE)?; + caption_text.mark(0..1, MARK_PARAGRAPH_STYLE, 0_i64)?; + if !caption.is_empty() { + caption_text.insert(1, caption)?; + } + } + + let attrs = block.ensure_mergeable_map("attrs")?; + attrs.insert("alignment", alignment_name(alignment))?; + match sizing { + InputImageSizing::Intrinsic => attrs.insert("sizing", "intrinsic")?, + InputImageSizing::FitWidth => attrs.insert("sizing", "fit_width")?, + InputImageSizing::Fixed { width_px, height_px } => { + attrs.insert("sizing", "fixed")?; + attrs.insert("width_px", i64::from(width_px))?; + if let Some(height_px) = height_px { + attrs.insert("height_px", i64::from(height_px))?; + } + } + }; + Ok(()) +} + +fn insert_equation_block(doc: &LoroDoc, unicode_index: usize, source: &str, display: InputEquationDisplay) -> Result<()> { + let body = body_text(doc); + body.insert(unicode_index, &OBJECT_REPLACEMENT.to_string())?; + let block = ensure_block(doc, "equation", ROOT_BODY_FLOW_ID, &body, unicode_index)?; + let source_flow_id = nested_flow_id("equation_source"); + block.insert("source_flow_id", source_flow_id.as_str())?; + let source_flow = ensure_flow(doc, &source_flow_id, "equation_source")?; + replace_text(&source_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?, source)?; + let attrs = block.ensure_mergeable_map("attrs")?; + attrs.insert("syntax", "latex")?; + attrs.insert("display", equation_display_name(display))?; + Ok(()) +} + +fn insert_table_block( + doc: &LoroDoc, + unicode_index: usize, + rows: usize, + columns: usize, + column_widths: &[InputTableColumnWidth], + header_row: bool, +) -> Result<()> { + let body = body_text(doc); + body.insert(unicode_index, &OBJECT_REPLACEMENT.to_string())?; + let block = ensure_block(doc, "table", ROOT_BODY_FLOW_ID, &body, unicode_index)?; + let table = block.ensure_mergeable_map("table")?; + table.insert("header_row", header_row)?; + let row_order = table.ensure_mergeable_list("row_order")?; + let column_order = table.ensure_mergeable_list("column_order")?; + let rows_by_id = table.ensure_mergeable_map("rows_by_id")?; + let columns_by_id = table.ensure_mergeable_map("columns_by_id")?; + let cells_by_id = table.ensure_mergeable_map("cells_by_id")?; + let table_id = table_id(); + + for column_ix in 0..columns { + let column_id = format!("{table_id}.column.{column_ix}"); + column_order.push(column_id.as_str())?; + let column = columns_by_id.ensure_mergeable_map(&column_id)?; + column.insert("id", column_id.as_str())?; + let width = column_widths.get(column_ix).unwrap_or(&InputTableColumnWidth::Auto); + match *width { + InputTableColumnWidth::Auto => column.insert("width_kind", "auto")?, + InputTableColumnWidth::FixedPx(px) => { + column.insert("width_kind", "fixed_px")?; + column.insert("width_px", i64::from(px))?; + } + InputTableColumnWidth::Fraction(fraction) => { + column.insert("width_kind", "fraction")?; + column.insert("fraction", i64::from(fraction))?; + } + }; + } + + for row_ix in 0..rows { + let row_id = format!("{table_id}.row.{row_ix}"); + row_order.push(row_id.as_str())?; + let row = rows_by_id.ensure_mergeable_map(&row_id)?; + row.insert("id", row_id.as_str())?; + for column_ix in 0..columns { + let cell_id = format!("{row_id}.cell.{column_ix}"); + let cell = cells_by_id.ensure_mergeable_map(&cell_id)?; + cell.insert("id", cell_id.as_str())?; + cell.insert("row_id", row_id.as_str())?; + cell.insert("column_index", i64::try_from(column_ix).unwrap_or(i64::MAX))?; + cell.insert("row_span", 1_i64)?; + cell.insert("column_span", 1_i64)?; + let flow_id = format!("{cell_id}.flow"); + cell.insert("flow_id", flow_id.as_str())?; + let flow = ensure_flow(doc, &flow_id, "table_cell")?; + let text = flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + replace_text(&text, SENTINEL_NEWLINE)?; + text.mark(0..1, MARK_PARAGRAPH_STYLE, 0_i64)?; + } + } + Ok(()) +} + +fn ensure_flow(doc: &LoroDoc, flow_id: &str, kind: &str) -> loro::LoroResult { + let root = doc.get_map(ROOT); + let flows = root.ensure_mergeable_map(FLOWS_BY_ID)?; + let flow = flows.ensure_mergeable_map(flow_id)?; + flow.insert(FLOW_ID_KEY, flow_id)?; + flow.insert(FLOW_KIND_KEY, kind)?; + flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + flow.ensure_mergeable_map(FLOW_ATTRS_KEY)?; + Ok(flow) +} + +fn ensure_block(doc: &LoroDoc, kind: &str, flow_id: &str, text: &loro::LoroText, pos: usize) -> loro::LoroResult { + let root = doc.get_map(ROOT); + let blocks = root.ensure_mergeable_map(BLOCKS_BY_ID)?; + let id = format!("{kind}.{}", Uuid::new_v4().as_u128()); + let block = blocks.ensure_mergeable_map(&id)?; + block.insert("id", id.as_str())?; + block.insert("kind", kind)?; + block.insert("flow_id", flow_id)?; + if let Some(cursor) = text.get_cursor(pos, Side::Left) { + block.insert("anchor_cursor", cursor.encode())?; + } + block.ensure_mergeable_map("attrs")?; + block.ensure_mergeable_map("nested_refs")?; + Ok(block) +} + +fn replace_text(text: &loro::LoroText, value: &str) -> loro::LoroResult<()> { + let len = text.len_unicode(); + if len > 0 { + text.delete(0, len)?; + } + if !value.is_empty() { + text.insert(0, value)?; + } + Ok(()) +} + +fn nested_flow_id(kind: &str) -> String { + format!("{kind}.{}", Uuid::new_v4().as_u128()) +} + +fn table_id() -> String { + format!("table.{}", Uuid::new_v4().as_u128()) +} + +fn alignment_name(alignment: InputBlockAlignment) -> &'static str { + match alignment { + InputBlockAlignment::Left => "left", + InputBlockAlignment::Center => "center", + InputBlockAlignment::Right => "right", + } +} + +fn equation_display_name(display: InputEquationDisplay) -> &'static str { + match display { + InputEquationDisplay::Display => "display", + InputEquationDisplay::InlineLikeParagraph => "inline_like_paragraph", + } +} + +#[cfg(test)] +mod tests { + use super::*; + use flowstate_document::{DocumentPackage, loro_schema::body_text}; + + #[test] + fn local_insert_exports_update_and_invalidates_projection() -> Result<()> { + let mut runtime = CrdtRuntime::new_empty("Runtime")?; + let events = runtime.command(SemanticCommand::InsertText { + unicode_index: 1, + text: "hello".to_string(), + })?; + assert!(matches!(events.first(), Some(RuntimeEvent::LocalUpdate { bytes, .. }) if !bytes.is_empty())); + assert!(events.iter().any(|event| matches!( + event, + RuntimeEvent::ProjectionUpdated { + document, + .. + } if flowstate_document::paragraph_text(document, 0) == "hello" + ))); + assert_eq!(body_text(runtime.doc()).to_string(), "\nhello"); + Ok(()) + } + + #[test] + fn runtime_persists_local_update_segments() -> Result<()> { + let dir = tempfile::tempdir()?; + let path = dir.path().join("runtime.db8"); + let doc = flowstate_document::new_loro_document("Runtime")?; + DocumentPackage::from_loro_snapshot(&doc, "Runtime")?.write(&path)?; + let mut runtime = CrdtRuntime::open_package(&path)?; + runtime.command(SemanticCommand::InsertText { + unicode_index: 1, + text: "persisted".to_string(), + })?; + let package = DocumentPackage::read(&path)?; + assert_eq!(package.loro_update_segments.len(), 1); + let loaded = package.load_loro_doc()?; + assert_eq!(body_text(&loaded).to_string(), "\npersisted"); + Ok(()) + } + + #[test] + fn semantic_text_commands_mutate_loro_body_flow() -> Result<()> { + let mut runtime = CrdtRuntime::new_empty("Runtime")?; + runtime.command(SemanticCommand::InsertText { + unicode_index: 1, + text: "hello world".to_string(), + })?; + runtime.command(SemanticCommand::DeleteRange { + unicode_index: 6, + unicode_len: 1, + })?; + runtime.command(SemanticCommand::SplitParagraph { + unicode_index: 6, + inherited_style: flowstate_document::ParagraphStyle::Custom(2), + })?; + runtime.command(SemanticCommand::SetRunStyles { + unicode_range: 1..6, + styles: flowstate_document::RunStyles { + semantic: flowstate_document::RunSemanticStyle::Custom(3), + direct_underline: true, + strikethrough: false, + highlight: Some(flowstate_document::HighlightStyle::Custom(4)), + }, + })?; + + assert_eq!(body_text(runtime.doc()).to_string(), "\nhello\nworld"); + let delta = body_text(runtime.doc()).to_delta(); + assert!(delta.iter().any(|item| matches!( + item, + loro::TextDelta::Insert { + attributes: Some(attributes), + .. + } if attributes.get(flowstate_document::MARK_RUN_SEMANTIC_STYLE).is_some() + ))); + assert!(delta.iter().any(|item| matches!( + item, + loro::TextDelta::Insert { + insert, + attributes: Some(attributes), + } if insert == "\n" && attributes.get(flowstate_document::MARK_PARAGRAPH_STYLE).is_some() + ))); + Ok(()) + } + + #[test] + fn semantic_object_commands_project_structured_blocks() -> Result<()> { + let mut runtime = CrdtRuntime::new_empty("Runtime")?; + runtime.command(SemanticCommand::InsertImage { + unicode_index: 1, + asset_id: 7, + alt_text: "alt".to_string(), + caption: Some("caption".to_string()), + sizing: flowstate_document::InputImageSizing::Fixed { + width_px: 320, + height_px: Some(180), + }, + alignment: flowstate_document::InputBlockAlignment::Center, + })?; + runtime.command(SemanticCommand::InsertEquation { + unicode_index: 2, + source: "x^2".to_string(), + display: flowstate_document::InputEquationDisplay::InlineLikeParagraph, + })?; + runtime.command(SemanticCommand::InsertTable { + unicode_index: 3, + rows: 2, + columns: 2, + column_widths: vec![ + flowstate_document::InputTableColumnWidth::FixedPx(120), + flowstate_document::InputTableColumnWidth::Fraction(1), + ], + header_row: true, + })?; + + let projection = runtime.projection_snapshot()?; + assert!(matches!( + &projection.blocks[0], + flowstate_document::Block::Image(image) + if image.asset_id == flowstate_document::AssetId(7) + && image.alt_text.as_ref() == "alt" + && image.caption.is_some() + )); + assert!(matches!( + &projection.blocks[1], + flowstate_document::Block::Equation(equation) + if equation.source.as_ref() == "x^2" + && equation.display == flowstate_document::EquationDisplay::InlineLikeParagraph + )); + assert!(matches!( + &projection.blocks[2], + flowstate_document::Block::Table(table) + if table.rows.len() == 2 + && table.rows[0].cells.len() == 2 + && table.style.header_row + && matches!(table.column_widths.as_slice(), [ + flowstate_document::TableColumnWidth::FixedPx(120), + flowstate_document::TableColumnWidth::Fraction(1) + ]) + )); + Ok(()) + } + + #[test] + fn runtime_opens_and_forks_named_revisions() -> Result<()> { + let dir = tempfile::tempdir()?; + let path = dir.path().join("revisions.db8"); + let doc = flowstate_document::new_loro_document("Runtime")?; + let mut package = DocumentPackage::from_loro_snapshot(&doc, "Runtime")?; + let blank_revision = package.create_named_revision(&doc, "Blank", "Blank document", None, None)?; + body_text(&doc).insert(1, "latest")?; + doc.commit(); + package.compact_to_named_snapshot(&doc, "Latest", "Latest document", None, None)?; + package.write(&path)?; + + let mut runtime = CrdtRuntime::open_package(&path)?; + let opened = runtime.command(SemanticCommand::OpenRevision { + revision_id: blank_revision, + })?; + assert!(matches!( + opened.as_slice(), + [RuntimeEvent::RevisionOpened { document, .. }] if document.paragraphs.first().is_some_and(|paragraph| paragraph.byte_range.is_empty()) + )); + + let forked = runtime.command(SemanticCommand::ForkRevision { + revision_id: blank_revision, + })?; + let [RuntimeEvent::RevisionForked { runtime: fork, .. }] = forked.as_slice() else { + panic!("expected fork event"); + }; + assert_eq!(body_text(fork.doc()).to_string(), "\n"); + Ok(()) + } + + #[test] + fn remote_import_reports_pending_dependencies() -> Result<()> { + let source = flowstate_document::new_loro_document("Source")?; + let empty_vv = VersionVector::default(); + body_text(&source).insert(1, "first")?; + source.commit(); + let mid_vv = source.state_vv(); + body_text(&source).insert(6, " second")?; + source.commit(); + let second_only = source.export(ExportMode::updates(&mid_vv))?; + + let mut target = CrdtRuntime::new_empty("Target")?; + let events = target.import_remote_update(&second_only)?; + assert!(matches!( + events.first(), + Some(RuntimeEvent::RemoteUpdateApplied { + pending: Some(_), + .. + }) + )); + + let first_update = source.export(ExportMode::updates(&empty_vv))?; + let events = target.import_remote_update(&first_update)?; + assert!(matches!( + events.first(), + Some(RuntimeEvent::RemoteUpdateApplied { + pending: None, + .. + }) + )); + Ok(()) + } +} diff --git a/crates/flowstate-collab/src/lib.rs b/crates/flowstate-collab/src/lib.rs index 910c48c7..1e5a85f9 100644 --- a/crates/flowstate-collab/src/lib.rs +++ b/crates/flowstate-collab/src/lib.rs @@ -1,21 +1,15 @@ //! Peer-to-peer collaboration support for Flowstate rich-text documents. //! -//! This crate contains the GPUI-free collaboration core: CRDT projection, -//! transport protocol types, and networking state. Application/UI integration -//! lives in `crates/flowstate/src/collab`. +//! This crate contains the GPUI-free collaboration core: the Loro-native CRDT +//! runtime, transport protocol types, presence, and networking state. +//! Application/UI integration lives in `crates/flowstate/src/collab`. -pub mod binding; -pub mod body_index; +pub mod crdt_runtime; pub mod ids; -pub mod local_apply; pub mod net; -pub mod patch_apply; pub mod presence; -pub mod projection; pub mod proto_direct; pub mod proto_gossip; -pub mod remote_apply; -pub mod schema; pub mod self_check; pub mod ticket; diff --git a/crates/flowstate-collab/src/local_apply.rs b/crates/flowstate-collab/src/local_apply.rs deleted file mode 100644 index 622aff29..00000000 --- a/crates/flowstate-collab/src/local_apply.rs +++ /dev/null @@ -1,705 +0,0 @@ -//! Translation from flowtext canonical operations into Loro operations. - -use anyhow::{Context as _, Result, bail}; -use std::ops::Range; - -use gpui_flowtext::{ - Block, BlockId, CanonicalOperation, Document, DocumentSpan, Paragraph, ParagraphId, RunStyles, full_document_text, paragraph_text_len, -}; -use loro::{CommitOptions, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer}; - -use crate::{ - binding::{BindingRow, BlockKind, DocBinding, block_version}, - body_index::minimal_utf8_splice, - projection::{insert_object_container, insert_paragraph_container, replace_blocks_from_document, replace_body_from_document}, - schema::{ - BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_TABLE, REV, STYLE, body_text, decode_paragraph_style, - encode_paragraph_style, payload_from_block, set_paragraph_style_utf8, set_run_styles_utf8, text_content, - }, -}; - -pub struct LocalApplier<'s> { - pub doc: &'s LoroDoc, - pub binding: &'s mut DocBinding, -} - -impl LocalApplier<'_> { - pub fn apply(&mut self, document: &Document, ops: &[CanonicalOperation]) -> Result<()> { - for op in ops { - self.apply_one(document, op)?; - } - if !ops.is_empty() { - self - .doc - .commit_with(CommitOptions::new().origin("local-edit")); - } - self.binding.refresh_versions(document); - self.binding.assert_consistent(self.doc, document); - Ok(()) - } - - fn apply_one(&mut self, document: &Document, op: &CanonicalOperation) -> Result<()> { - match op { - CanonicalOperation::InsertText { - paragraph, - byte, - text, - styles, - } => self.insert_text(document, *paragraph, *byte, text, *styles), - CanonicalOperation::DeleteRange { - start_paragraph, - start_byte, - end_paragraph, - end_byte, - } => self.delete_range(document, *start_paragraph, *start_byte, *end_paragraph, *end_byte), - CanonicalOperation::SplitParagraph { - paragraph, - byte, - new_paragraph, - } => self.split_paragraph(document, *paragraph, *byte, *new_paragraph), - CanonicalOperation::JoinParagraphs { first, second } => self.join_paragraphs(document, *first, *second), - CanonicalOperation::SetParagraphStyle { paragraph, style } => { - let row = self.row_for_paragraph(*paragraph)?; - row.map.insert(STYLE, encode_paragraph_style(*style))?; - let range = self.body_range_for_paragraph(*paragraph)?; - set_paragraph_style_utf8(&body_text(self.doc), range, *style)?; - Ok(()) - }, - CanonicalOperation::SetRunStyles { paragraph, range, styles } => { - let start = self.body_byte_for_paragraph_byte(*paragraph, range.start)?; - let end = self.body_byte_for_paragraph_byte(*paragraph, range.end)?; - set_run_styles_utf8(&body_text(self.doc), start..end, *styles)?; - Ok(()) - }, - CanonicalOperation::InsertBlock { block, block_ix } => self.insert_block(document, *block, *block_ix), - CanonicalOperation::DeleteBlock { block } => self.delete_block(document, *block), - CanonicalOperation::MoveBlock { block, new_block_ix } => self.move_block(document, *block, *new_block_ix), - CanonicalOperation::ReplaceParagraphSpan { - start_paragraph, - before, - after, - } => self.replace_paragraph_span(document, *start_paragraph, before, after), - CanonicalOperation::ReplaceBlock { block } => self.replace_block(document, *block), - CanonicalOperation::ReplaceDocument => self.replace_document(document), - } - } - - fn insert_text(&mut self, document: &Document, paragraph: ParagraphId, byte: usize, value: &str, styles: RunStyles) -> Result<()> { - let ordinal = self.paragraph_ordinal(paragraph)?; - let range = self.body_paragraph_range(ordinal)?; - let body_byte = range.start + byte.min(range.len()); - let body = body_text(self.doc); - body.insert_utf8(body_byte, value)?; - set_run_styles_utf8(&body, body_byte..body_byte + value.len(), styles)?; - self - .binding - .body_index - .set_paragraph_len(ordinal, range.len() + value.len()); - self.debug_assert_body_matches_document(document); - Ok(()) - } - - fn delete_range( - &mut self, - document: &Document, - start_paragraph: ParagraphId, - start: usize, - end_paragraph: ParagraphId, - end: usize, - ) -> Result<()> { - let start_ix = self.row_ix_for_paragraph(start_paragraph)?; - let end_ix = self.row_ix_for_paragraph(end_paragraph)?; - let start_byte = self.body_byte_for_paragraph_byte(start_paragraph, start)?; - let end_byte = self.body_byte_for_paragraph_byte(end_paragraph, end)?; - let deleted = end_byte.saturating_sub(start_byte); - if deleted > 0 { - body_text(self.doc).delete_utf8(start_byte, deleted)?; - } - if start_ix == end_ix { - if let Some(ordinal) = self.paragraph_ordinal_for_row(start_ix) { - let len = self.binding.body_index.paragraph_len(ordinal); - self - .binding - .body_index - .set_paragraph_len(ordinal, len.saturating_sub(deleted)); - } - self.debug_assert_body_matches_document(document); - return Ok(()); - } - if start_ix > end_ix { - bail!("cross-paragraph delete start row is after end row"); - } - - let start_paragraph_ix = self - .paragraph_ordinal_for_row(start_ix) - .context("delete start row is not a paragraph")?; - let end_paragraph_ix = self - .paragraph_ordinal_for_row(end_ix) - .context("delete end row is not a paragraph")?; - if end_paragraph_ix > start_paragraph_ix { - // The start paragraph absorbs the end paragraph's surviving tail; the - // paragraphs in between (and the end paragraph) collapse away. - let end_len = self.binding.body_index.paragraph_len(end_paragraph_ix); - let merged_len = start + end_len.saturating_sub(end); - let deleted_rows = self.paragraph_row_indices(start_paragraph_ix + 1, end_paragraph_ix - start_paragraph_ix)?; - let blocks = self.blocks(); - for row_ix in deleted_rows.into_iter().rev() { - blocks.delete(row_ix, 1)?; - self - .binding - .remove_row(row_ix) - .context("DocBinding paragraph row disappeared during cross-paragraph delete")?; - } - self - .binding - .body_index - .remove_paragraph_range(start_paragraph_ix + 1, end_paragraph_ix - start_paragraph_ix); - self - .binding - .body_index - .set_paragraph_len(start_paragraph_ix, merged_len); - } - self.debug_assert_body_matches_document(document); - Ok(()) - } - - fn split_paragraph(&mut self, document: &Document, paragraph: ParagraphId, byte: usize, new_paragraph: ParagraphId) -> Result<()> { - let row_ix = self.row_ix_for_paragraph(paragraph)?; - let insert_ix = row_ix + 1; - let block_id = document - .ids - .block_ids - .get(insert_ix) - .copied() - .context("post-split document is missing the new block id")?; - let version = document - .blocks - .get(insert_ix) - .map(block_version) - .context("post-split document is missing the new block")?; - let orig_ordinal = self - .paragraph_ordinal_for_row(row_ix) - .context("split row is not a paragraph")?; - let old_len = self.binding.body_index.paragraph_len(orig_ordinal); - let split = byte.min(old_len); - let body_byte = self.binding.body_index.paragraph_start(orig_ordinal) + split; - body_text(self.doc).insert_utf8(body_byte, "\n")?; - - let blocks = self.blocks(); - let paragraph_ix = orig_ordinal.saturating_add(1); - let style = document.paragraphs.get(paragraph_ix).map_or_else( - || { - self - .binding - .rows - .get(row_ix) - .and_then(|row| map_i64(&row.map, STYLE).ok()) - .map(decode_paragraph_style) - .unwrap_or(gpui_flowtext::ParagraphStyle::Normal) - }, - |paragraph| paragraph.style, - ); - let map = insert_paragraph_container(&blocks, insert_ix, style)?; - self.binding.insert_row( - insert_ix, - BindingRow { - map, - kind: BlockKind::Paragraph, - block_id, - paragraph_id: Some(new_paragraph), - version, - }, - ); - self - .binding - .body_index - .set_paragraph_len(orig_ordinal, split); - self - .binding - .body_index - .insert_paragraph(orig_ordinal + 1, old_len - split); - self.debug_assert_body_matches_document(document); - Ok(()) - } - - fn join_paragraphs(&mut self, document: &Document, first: ParagraphId, second: ParagraphId) -> Result<()> { - let first_ix = self.row_ix_for_paragraph(first)?; - let first_paragraph_ix = self - .paragraph_ordinal_for_row(first_ix) - .context("join first row is not a paragraph")?; - let first_len = self.body_paragraph_range(first_paragraph_ix)?.len(); - self.delete_range(document, first, first_len, second, 0) - } - - fn insert_block(&mut self, document: &Document, block_id: BlockId, block_ix: usize) -> Result<()> { - let block = document - .blocks - .get(block_ix) - .context("inserted block index is out of bounds in the post-edit document")?; - match block { - Block::Paragraph(paragraph) => { - replace_body_from_document(self.doc, document)?; - self.binding.refresh_body_index(document); - let paragraph_ix = document_paragraph_ix_for_block(document, block_ix)?; - let paragraph_id = document - .ids - .paragraph_ids - .get(paragraph_ix) - .copied() - .context("inserted paragraph block is missing a paragraph id")?; - self.insert_paragraph_row(block_ix, block_id, paragraph_id, paragraph.style, block_version(block))?; - self.debug_assert_body_matches_document(document); - Ok(()) - }, - Block::Image(_) | Block::Equation(_) | Block::Table(_) => { - let blocks = self.blocks(); - let map = insert_object_container(&blocks, block_ix, block, document)?; - self.binding.insert_row( - block_ix, - BindingRow { - map, - kind: BlockKind::from_block(block), - block_id, - paragraph_id: None, - version: block_version(block), - }, - ); - Ok(()) - }, - } - } - - fn delete_block(&mut self, document: &Document, block: BlockId) -> Result<()> { - let row_ix = self.row_ix_for_block(block)?; - let was_paragraph = matches!(self.binding.rows.get(row_ix).map(|row| row.kind), Some(BlockKind::Paragraph)); - self.blocks().delete(row_ix, 1)?; - self - .binding - .remove_row(row_ix) - .context("DocBinding block row disappeared during block delete")?; - if was_paragraph { - replace_body_from_document(self.doc, document)?; - self.binding.refresh_body_index(document); - self.debug_assert_body_matches_document(document); - } - Ok(()) - } - - fn move_block(&mut self, document: &Document, block: BlockId, new_block_ix: usize) -> Result<()> { - let old_ix = self.row_ix_for_block(block)?; - let was_paragraph = matches!(self.binding.rows.get(old_ix).map(|row| row.kind), Some(BlockKind::Paragraph)); - self.blocks().mov(old_ix, new_block_ix)?; - self.binding.move_row(old_ix, new_block_ix); - if was_paragraph { - replace_body_from_document(self.doc, document)?; - self.binding.refresh_body_index(document); - self.debug_assert_body_matches_document(document); - } - Ok(()) - } - - fn replace_paragraph_span( - &mut self, - document: &Document, - start_paragraph: Option, - before: &DocumentSpan, - after: &DocumentSpan, - ) -> Result<()> { - // Try a span-scoped splice; fall back to a full rewrite if the span can't be - // resolved/reconciled or any step fails (the fallback fully resets the Loro - // doc to `document`, so a partially-applied attempt is safe). - match self.try_replace_paragraph_span(document, start_paragraph, before, after) { - Ok(true) => Ok(()), - Ok(false) | Err(_) => self.replace_document(document), - } - } - - fn try_replace_paragraph_span( - &mut self, - document: &Document, - start_paragraph: Option, - before: &DocumentSpan, - after: &DocumentSpan, - ) -> Result { - let before_n = before.paragraphs.len(); - let after_n = after.paragraphs.len(); - if before_n == 0 || after_n == 0 { - return Ok(false); - } - - // Resolve the span's start paragraph ordinal. - let start_ord = match start_paragraph { - Some(id) => { - let Some(&row_ix) = self.binding.by_paragraph.get(&id) else { - return Ok(false); - }; - let Some(ord) = self.paragraph_ordinal_for_row(row_ix) else { - return Ok(false); - }; - ord - }, - None => before.start_paragraph, - }; - if start_ord + before_n > self.binding.body_index.len() { - return Ok(false); - } - - // The span's paragraph rows must be contiguous paragraph blocks (no object - // blocks interleaved) for the in-place row reconcile below to be valid. - let Ok(before_rows) = self.paragraph_row_indices(start_ord, before_n) else { - return Ok(false); - }; - let first_row = before_rows[0]; - if before_rows - .iter() - .enumerate() - .any(|(offset, row)| *row != first_row + offset) - { - return Ok(false); - } - - // The body slice for the span must currently equal `before.text`; if the - // lengths disagree the editor and CRDT have diverged — bail to the rewrite. - let start_byte = self.binding.body_index.paragraph_start(start_ord); - let end_byte = self - .binding - .body_index - .paragraph_range(start_ord + before_n - 1) - .end; - if end_byte - start_byte != before.text.len() { - return Ok(false); - } - - // Minimal text splice: only rewrite the differing middle of the span. - let (prefix, old_middle, new_middle) = minimal_utf8_splice(&before.text, &after.text); - let body = body_text(self.doc); - let splice_at = start_byte + prefix; - if old_middle > 0 { - body.delete_utf8(splice_at, old_middle)?; - } - let inserted = &after.text[new_middle]; - if !inserted.is_empty() { - body.insert_utf8(splice_at, inserted)?; - } - - // Re-apply marks across the whole span so inline run styles and paragraph - // styles match `after` (INV-2), including over surviving prefix/suffix bytes. - self.apply_span_marks(start_byte, &after.paragraphs)?; - - // Reconcile the block rows + paragraph offset index for the span. - let overlap = before_n.min(after_n); - let blocks = self.blocks(); - for offset in 0..overlap { - let row_ix = first_row + offset; - let ordinal = start_ord + offset; - let style = after.paragraphs[offset].style; - let block_id = document - .ids - .block_ids - .get(row_ix) - .copied() - .context("post-edit document is missing a block id for a spliced paragraph")?; - let paragraph_id = document - .ids - .paragraph_ids - .get(ordinal) - .copied() - .context("post-edit document is missing a paragraph id for a spliced paragraph")?; - let row = self - .binding - .rows - .get_mut(row_ix) - .context("spliced paragraph row is out of bounds")?; - row.map.insert(STYLE, encode_paragraph_style(style))?; - row.block_id = block_id; - row.paragraph_id = Some(paragraph_id); - self - .binding - .body_index - .set_paragraph_len(ordinal, paragraph_text_len(&after.paragraphs[offset])); - } - - if after_n > before_n { - for offset in before_n..after_n { - let row_ix = first_row + offset; - let ordinal = start_ord + offset; - let style = after.paragraphs[offset].style; - let block_id = document - .ids - .block_ids - .get(row_ix) - .copied() - .context("post-edit document is missing a block id for an inserted paragraph")?; - let paragraph_id = document - .ids - .paragraph_ids - .get(ordinal) - .copied() - .context("post-edit document is missing a paragraph id for an inserted paragraph")?; - let version = document.blocks.get(row_ix).map_or(0, block_version); - let map = insert_paragraph_container(&blocks, row_ix, style)?; - self.binding.insert_row( - row_ix, - BindingRow { - map, - kind: BlockKind::Paragraph, - block_id, - paragraph_id: Some(paragraph_id), - version, - }, - ); - self - .binding - .body_index - .insert_paragraph(ordinal, paragraph_text_len(&after.paragraphs[offset])); - } - } else if before_n > after_n { - let removed = before_n - after_n; - for _ in 0..removed { - let row_ix = first_row + after_n; - blocks.delete(row_ix, 1)?; - self - .binding - .remove_row(row_ix) - .context("spliced paragraph row disappeared during span shrink")?; - } - self - .binding - .body_index - .remove_paragraph_range(start_ord + after_n, removed); - } - - self.binding.rebuild_indexes(); - self.debug_assert_body_matches_document(document); - Ok(true) - } - - fn apply_span_marks(&self, start: usize, paragraphs: &[Paragraph]) -> Result<()> { - let body = body_text(self.doc); - let mut paragraph_start = start; - for paragraph in paragraphs { - let mut run_byte = paragraph_start; - for run in ¶graph.runs { - set_run_styles_utf8(&body, run_byte..run_byte + run.len, run.styles)?; - run_byte += run.len; - } - let paragraph_len = run_byte - paragraph_start; - set_paragraph_style_utf8(&body, paragraph_start..paragraph_start + paragraph_len, paragraph.style)?; - paragraph_start += paragraph_len + 1; - } - Ok(()) - } - - fn replace_block(&mut self, document: &Document, block: Option) -> Result<()> { - let row_ix = match block { - Some(block) => self.binding.by_block.get(&block).copied(), - None => self.single_changed_object_row(document), - }; - if let Some(row_ix) = row_ix - && self.replace_block_at(document, row_ix)? - { - return Ok(()); - } - self.replace_document(document) - } - - fn replace_document(&mut self, document: &Document) -> Result<()> { - replace_blocks_from_document(self.doc, document)?; - *self.binding = DocBinding::build(self.doc, document)?; - Ok(()) - } - - fn replace_block_at(&mut self, document: &Document, row_ix: usize) -> Result { - let Some(block) = document.blocks.get(row_ix) else { - return Ok(false); - }; - let Some(payload) = payload_from_block(block, &document.assets) else { - return Ok(false); - }; - let row = self - .binding - .rows - .get_mut(row_ix) - .context("replacement block row is out of bounds")?; - row.map.insert(KIND, kind_for_payload(&payload))?; - row - .map - .insert(DATA, LoroValue::Binary(postcard::to_stdvec(&payload)?.into()))?; - row - .map - .insert(REV, map_i64_or(&row.map, REV, 0)?.saturating_add(1))?; - row.kind = BlockKind::from_block(block); - row.paragraph_id = None; - row.version = block_version(block); - self.binding.rebuild_indexes(); - Ok(true) - } - - fn insert_paragraph_row( - &mut self, - row_ix: usize, - block_id: BlockId, - paragraph_id: ParagraphId, - style: gpui_flowtext::ParagraphStyle, - version: u64, - ) -> Result<()> { - let blocks = self.blocks(); - let map = insert_paragraph_container(&blocks, row_ix, style)?; - self.binding.insert_row( - row_ix, - BindingRow { - map, - kind: BlockKind::Paragraph, - block_id, - paragraph_id: Some(paragraph_id), - version, - }, - ); - Ok(()) - } - - fn single_changed_object_row(&self, document: &Document) -> Option { - let mut changed = None; - for (row_ix, row) in self.binding.rows.iter().enumerate() { - let block = document.blocks.get(row_ix)?; - if matches!(block, Block::Paragraph(_)) || row.version == block_version(block) { - continue; - } - if changed.replace(row_ix).is_some() { - return None; - } - } - changed - } - - fn row_for_paragraph(&self, paragraph: ParagraphId) -> Result<&BindingRow> { - let ix = self.row_ix_for_paragraph(paragraph)?; - self - .binding - .rows - .get(ix) - .context("DocBinding paragraph index is out of bounds") - } - - fn debug_assert_body_matches_document(&self, document: &Document) { - debug_assert_eq!(text_content(&body_text(self.doc)), full_document_text(document)); - } - - fn row_ix_for_paragraph(&self, paragraph: ParagraphId) -> Result { - self - .binding - .by_paragraph - .get(¶graph) - .copied() - .context("paragraph id is not present in DocBinding") - } - - fn row_ix_for_block(&self, block: BlockId) -> Result { - self - .binding - .by_block - .get(&block) - .copied() - .context("block id is not present in DocBinding") - } - - fn paragraph_ordinal_for_row(&self, target_row: usize) -> Option { - let mut paragraph_ix = 0; - for (row_ix, row) in self.binding.rows.iter().enumerate() { - if row_ix == target_row { - return row.paragraph_id.map(|_| paragraph_ix); - } - if row.paragraph_id.is_some() { - paragraph_ix += 1; - } - } - None - } - - fn paragraph_row_indices(&self, start: usize, count: usize) -> Result> { - let mut rows = Vec::with_capacity(count); - let mut paragraph_ix = 0; - for (row_ix, row) in self.binding.rows.iter().enumerate() { - if row.paragraph_id.is_none() { - continue; - } - if paragraph_ix >= start && rows.len() < count { - rows.push(row_ix); - } - paragraph_ix += 1; - } - if rows.len() != count { - bail!("paragraph span references rows outside DocBinding"); - } - Ok(rows) - } - - fn blocks(&self) -> LoroMovableList { - self.doc.get_movable_list(BLOCKS) - } - - fn paragraph_ordinal(&self, paragraph: ParagraphId) -> Result { - let row_ix = self.row_ix_for_paragraph(paragraph)?; - self - .paragraph_ordinal_for_row(row_ix) - .context("paragraph row has no paragraph ordinal") - } - - fn body_byte_for_paragraph_byte(&self, paragraph: ParagraphId, byte: usize) -> Result { - let range = self.body_paragraph_range(self.paragraph_ordinal(paragraph)?)?; - Ok(range.start + byte.min(range.len())) - } - - fn body_range_for_paragraph(&self, paragraph: ParagraphId) -> Result> { - self.body_paragraph_range(self.paragraph_ordinal(paragraph)?) - } - - fn body_paragraph_range(&self, target_paragraph_ix: usize) -> Result> { - if target_paragraph_ix >= self.binding.body_index.len() { - bail!("paragraph ordinal {target_paragraph_ix} is outside the body index"); - } - Ok(self.binding.body_index.paragraph_range(target_paragraph_ix)) - } -} - -fn document_paragraph_ix_for_block(document: &Document, target_block_ix: usize) -> Result { - let mut paragraph_ix = 0; - for (block_ix, block) in document.blocks.iter().enumerate() { - match block { - Block::Paragraph(_) if block_ix == target_block_ix => return Ok(paragraph_ix), - Block::Paragraph(_) => paragraph_ix += 1, - Block::Image(_) | Block::Equation(_) | Block::Table(_) if block_ix == target_block_ix => { - bail!("target block is not a paragraph") - }, - Block::Image(_) | Block::Equation(_) | Block::Table(_) => {}, - } - } - bail!("target block index is out of bounds") -} - -fn kind_for_payload(payload: &BlockPayload) -> &'static str { - match payload { - BlockPayload::Image { .. } => KIND_IMAGE, - BlockPayload::Equation { .. } => KIND_EQUATION, - BlockPayload::Table(_) => KIND_TABLE, - } -} - -fn map_i64(map: &LoroMap, key: &str) -> Result { - match map.get(key) { - Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("collaboration map key {key} is not an i64") - }, - } -} - -fn map_i64_or(map: &LoroMap, key: &str, default: i64) -> Result { - match map.get(key) { - Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), - None => Ok(default), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) => { - bail!("collaboration map key {key} is not an i64") - }, - } -} diff --git a/crates/flowstate-collab/src/patch_apply.rs b/crates/flowstate-collab/src/patch_apply.rs deleted file mode 100644 index 6320029a..00000000 --- a/crates/flowstate-collab/src/patch_apply.rs +++ /dev/null @@ -1,200 +0,0 @@ -//! Headless application of collaboration patches to flowtext documents. - -use anyhow::{Context as _, Result}; -use gpui_flowtext::{ - Block, CollabPatch, CollabStructuralBlock, Document, InputBlock, InputParagraph, InputRun, document_from_input_blocks, input_block_from_block, - paragraph_text, paragraphs_mut, rebuild_document_sections, update_paragraph_block, -}; -use loro::LoroDoc; - -use crate::binding::DocBinding; - -pub fn apply_patches(document: &mut Document, binding: &mut DocBinding, doc: &LoroDoc, patches: &[CollabPatch]) -> Result<()> { - for patch in patches { - apply_one(document, patch)?; - } - binding.refresh_versions(document); - binding.refresh_body_index(document); - binding.assert_consistent(doc, document); - Ok(()) -} - -fn apply_one(document: &mut Document, patch: &CollabPatch) -> Result<()> { - match patch { - CollabPatch::ParagraphText { row, new, .. } => { - replace_paragraph_block(document, *row, new)?; - }, - CollabPatch::ParagraphStyle { row, style } => { - if let Some(paragraph_ix) = paragraph_ix_for_block(document, *row) - && let Some(paragraph) = paragraphs_mut(document).get_mut(paragraph_ix) - { - paragraph.style = *style; - gpui_flowtext::bump_paragraph_version(paragraph); - update_paragraph_block(document, paragraph_ix); - rebuild_document_sections(document); - } - }, - CollabPatch::ParagraphRuns { row, runs } => { - if let Some(paragraph_ix) = paragraph_ix_for_block(document, *row) - && let Some(paragraph) = paragraphs_mut(document).get_mut(paragraph_ix) - { - paragraph.runs.clone_from(runs); - gpui_flowtext::bump_paragraph_version(paragraph); - update_paragraph_block(document, paragraph_ix); - rebuild_document_sections(document); - } - }, - CollabPatch::ReplaceObjectBlock { row, block } => { - let mut blocks = structural_blocks_from_document(document)?; - if *row < blocks.len() { - blocks[*row] = block.clone(); - rebuild_document_from_structural_blocks(document, blocks)?; - } - }, - CollabPatch::InsertBlocks { row, blocks: inserted } => { - let mut blocks = structural_blocks_from_document(document)?; - let row = (*row).min(blocks.len()); - blocks.splice(row..row, inserted.iter().cloned()); - rebuild_document_from_structural_blocks(document, blocks)?; - }, - CollabPatch::DeleteBlocks { row, count } => { - let mut blocks = structural_blocks_from_document(document)?; - let start = (*row).min(blocks.len()); - let end = start.saturating_add(*count).min(blocks.len()); - blocks.drain(start..end); - rebuild_document_from_structural_blocks(document, blocks)?; - }, - CollabPatch::MoveBlock { from, to } => { - let mut blocks = structural_blocks_from_document(document)?; - if *from < blocks.len() { - let block = blocks.remove(*from); - blocks.insert((*to).min(blocks.len()), block); - rebuild_document_from_structural_blocks(document, blocks)?; - } - }, - CollabPatch::AssetArrived { .. } => {}, - } - Ok(()) -} - -fn replace_paragraph_block(document: &mut Document, row: usize, paragraph: &InputParagraph) -> Result<()> { - let mut blocks = structural_blocks_from_document(document)?; - let Some(block) = blocks.get_mut(row) else { - return Ok(()); - }; - if !matches!(block.block, InputBlock::Paragraph(_)) { - return Ok(()); - } - block.block = InputBlock::Paragraph(paragraph.clone()); - rebuild_document_from_structural_blocks(document, blocks) -} - -fn structural_blocks_from_document(document: &Document) -> Result> { - let mut paragraph_ix = 0; - document - .blocks - .iter() - .enumerate() - .map(|(block_ix, block)| { - let block_id = document - .ids - .block_ids - .get(block_ix) - .copied() - .context("document block is missing its collaboration id")?; - match block { - Block::Paragraph(paragraph) => { - let paragraph_id = document - .ids - .paragraph_ids - .get(paragraph_ix) - .copied() - .context("document paragraph is missing its collaboration id")?; - let input = input_paragraph_from_document_paragraph(document, paragraph_ix, paragraph); - paragraph_ix += 1; - Ok(CollabStructuralBlock { - block_id, - paragraph_id: Some(paragraph_id), - block: InputBlock::Paragraph(input), - }) - }, - Block::Image(_) | Block::Equation(_) | Block::Table(_) => Ok(CollabStructuralBlock { - block_id, - paragraph_id: None, - block: input_block_from_block(block), - }), - } - }) - .collect() -} - -fn input_paragraph_from_document_paragraph(document: &Document, paragraph_ix: usize, paragraph: &gpui_flowtext::Paragraph) -> InputParagraph { - let text = paragraph_text(document, paragraph_ix); - let mut byte = 0usize; - InputParagraph { - style: paragraph.style, - runs: paragraph - .runs - .iter() - .map(|run| { - let start = byte; - let end = start.saturating_add(run.len).min(text.len()); - byte = end; - InputRun { - text: text.get(start..end).unwrap_or_default().to_string(), - styles: run.styles, - } - }) - .collect(), - } -} - -fn rebuild_document_from_structural_blocks(document: &mut Document, blocks: Vec) -> Result<()> { - let assets = document.assets.clone(); - let theme = document.theme.clone(); - let document_id = document.ids.document_id; - let input_blocks = blocks - .iter() - .map(|block| block.block.clone()) - .collect::>(); - let block_ids = blocks - .iter() - .map(|block| block.block_id) - .collect::>(); - let paragraph_ids = blocks - .iter() - .filter_map(|block| match &block.block { - InputBlock::Paragraph(_) => Some( - block - .paragraph_id - .context("structural paragraph patch is missing a paragraph id"), - ), - InputBlock::Image(_) | InputBlock::Equation(_) | InputBlock::Table(_) => None, - }) - .collect::>>()?; - - let mut rebuilt = document_from_input_blocks(theme, input_blocks); - rebuilt.assets = assets; - rebuilt.ids.document_id = document_id; - rebuilt.ids.block_ids = block_ids; - rebuilt.ids.paragraph_ids = paragraph_ids; - rebuild_document_sections(&mut rebuilt); - debug_assert_eq!(rebuilt.ids.block_ids.len(), rebuilt.blocks.len()); - debug_assert_eq!(rebuilt.ids.paragraph_ids.len(), rebuilt.paragraphs.len()); - *document = rebuilt; - Ok(()) -} - -fn paragraph_ix_for_block(document: &Document, target_row: usize) -> Option { - let mut paragraph_ix = 0; - for (block_ix, block) in document.blocks.iter().enumerate() { - if !matches!(block, Block::Paragraph(_)) { - continue; - } - if block_ix == target_row { - return Some(paragraph_ix); - } - paragraph_ix += 1; - } - None -} diff --git a/crates/flowstate-collab/src/projection.rs b/crates/flowstate-collab/src/projection.rs deleted file mode 100644 index 282348bb..00000000 --- a/crates/flowstate-collab/src/projection.rs +++ /dev/null @@ -1,210 +0,0 @@ -//! Projection between flowtext documents and the shared Loro document. - -use anyhow::{Context as _, Result, bail}; -use gpui_flowtext::{ - AssetStore, Block, Document, DocumentTheme, InputBlock, document_from_input_blocks, full_document_text, paragraph_byte_range, -}; -use loro::{Container, LoroDoc, LoroMap, LoroMovableList, LoroValue, ValueOrContainer}; - -use crate::{ - SessionId, - schema::{ - BLOCKS, BlockPayload, DATA, KIND, KIND_EQUATION, KIND_IMAGE, KIND_PARAGRAPH, KIND_TABLE, META, META_SCHEMA, META_SESSION, META_TITLE, - MarkIntervals, REV, SCHEMA_VERSION, STYLE, apply_mark_intervals, block_from_payload, body_text, configure_text_styles, - decode_paragraph_style, encode_paragraph_style, input_paragraphs_from_body_text, mark_intervals_from_runs, payload_from_block, - }, -}; - -pub fn populate_from_document(doc: &LoroDoc, session: SessionId, title: &str, document: &Document) -> Result<()> { - configure_text_styles(doc); - populate_meta(doc, session, title)?; - replace_blocks_from_document(doc, document)?; - doc.commit(); - Ok(()) -} - -pub fn replace_blocks_from_document(doc: &LoroDoc, document: &Document) -> Result<()> { - configure_text_styles(doc); - replace_body_from_document(doc, document)?; - let blocks = doc.get_movable_list(BLOCKS); - clear_blocks(&blocks)?; - - for block in document.blocks.iter() { - match block { - Block::Paragraph(paragraph) => { - insert_paragraph_container(&blocks, blocks.len(), paragraph.style)?; - }, - Block::Image(_) | Block::Equation(_) | Block::Table(_) => { - insert_object_container(&blocks, blocks.len(), block, document)?; - }, - } - } - - Ok(()) -} - -pub fn input_blocks_from_loro(doc: &LoroDoc) -> Result> { - configure_text_styles(doc); - let blocks = doc.get_movable_list(BLOCKS); - let body_paragraphs = input_paragraphs_from_body_text(&body_text(doc)); - let mut paragraph_ix = 0usize; - let mut input_blocks = Vec::with_capacity(blocks.len()); - for ix in 0..blocks.len() { - let map = map_at(&blocks, ix)?; - input_blocks.push(input_block_from_container(&map, body_paragraphs.get(paragraph_ix))?); - if map_string(&map, KIND)? == KIND_PARAGRAPH { - paragraph_ix += 1; - } - } - Ok(input_blocks) -} - -pub fn document_from_loro(doc: &LoroDoc, theme: DocumentTheme) -> Result { - Ok(document_from_input_blocks(theme, input_blocks_from_loro(doc)?)) -} - -pub fn input_block_from_container(map: &LoroMap, body_paragraph: Option<&gpui_flowtext::InputParagraph>) -> Result { - match map_string(map, KIND)?.as_str() { - KIND_PARAGRAPH => { - let style = decode_paragraph_style(map_i64(map, STYLE)?); - let mut paragraph = body_paragraph - .cloned() - .unwrap_or(gpui_flowtext::InputParagraph { style, runs: Vec::new() }); - if body_paragraph.is_none() { - paragraph.style = style; - } - Ok(InputBlock::Paragraph(paragraph)) - }, - KIND_IMAGE | KIND_EQUATION | KIND_TABLE => { - let payload = map_binary(map, DATA)?; - let payload = postcard::from_bytes::(&payload)?; - Ok(block_from_payload(payload, &AssetStore::default())) - }, - kind => bail!("unknown collaboration block kind {kind}"), - } -} - -fn populate_meta(doc: &LoroDoc, session: SessionId, title: &str) -> Result<()> { - let meta = doc.get_map(META); - meta.insert(META_SCHEMA, SCHEMA_VERSION)?; - meta.insert(META_SESSION, session.to_string())?; - meta.insert(META_TITLE, title)?; - Ok(()) -} - -fn clear_blocks(blocks: &LoroMovableList) -> Result<()> { - while !blocks.is_empty() { - blocks.delete(0, 1)?; - } - Ok(()) -} - -pub fn insert_paragraph_container(blocks: &LoroMovableList, ix: usize, style: gpui_flowtext::ParagraphStyle) -> Result { - let map = blocks.insert_container(ix, LoroMap::new())?; - map.insert(KIND, KIND_PARAGRAPH)?; - map.insert(STYLE, encode_paragraph_style(style))?; - Ok(map) -} - -pub fn insert_object_container(blocks: &LoroMovableList, ix: usize, block: &Block, document: &Document) -> Result { - let Some(payload) = payload_from_block(block, &document.assets) else { - bail!("paragraph block reached object projection path"); - }; - let kind = match &payload { - BlockPayload::Image { .. } => KIND_IMAGE, - BlockPayload::Equation { .. } => KIND_EQUATION, - BlockPayload::Table(_) => KIND_TABLE, - }; - let map = blocks.insert_container(ix, LoroMap::new())?; - map.insert(KIND, kind)?; - map.insert(DATA, LoroValue::Binary(postcard::to_stdvec(&payload)?.into()))?; - map.insert(REV, 0_i64)?; - Ok(map) -} - -fn map_at(blocks: &LoroMovableList, ix: usize) -> Result { - match blocks.get(ix) { - Some(ValueOrContainer::Container(Container::Map(map))) => Ok(map), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("collaboration block row {ix} is not a map container") - }, - } -} - -fn map_string(map: &LoroMap, key: &str) -> Result { - match map.get(key) { - Some(ValueOrContainer::Value(LoroValue::String(value))) => Ok(value.to_string()), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("collaboration map key {key} is not a string") - }, - } -} - -pub fn replace_body_from_document(doc: &LoroDoc, document: &Document) -> Result<()> { - let body = body_text(doc); - let len = body.len_utf8(); - if len > 0 { - body.delete_utf8(0, len)?; - } - let text = full_document_text(document); - if !text.is_empty() { - body.insert_utf8(0, &text)?; - apply_mark_intervals(&body, &document_mark_intervals(document))?; - } - Ok(()) -} - -fn document_mark_intervals(document: &Document) -> MarkIntervals { - let mut intervals: MarkIntervals = [Vec::new(), Vec::new(), Vec::new(), Vec::new(), Vec::new()]; - for (paragraph_ix, paragraph) in document.paragraphs.iter().enumerate() { - let paragraph_start = paragraph_byte_range(document, paragraph_ix).start; - for (target, source) in intervals - .iter_mut() - .zip(mark_intervals_from_runs(¶graph.runs)) - { - target.extend( - source - .into_iter() - .map(|(range, value)| (paragraph_start + range.start..paragraph_start + range.end, value)), - ); - } - if paragraph.style != gpui_flowtext::ParagraphStyle::Normal { - let range = paragraph_byte_range(document, paragraph_ix); - if !range.is_empty() { - intervals[4].push((range, LoroValue::I64(encode_paragraph_style(paragraph.style)))); - } - } - } - intervals -} - -fn map_i64(map: &LoroMap, key: &str) -> Result { - match map.get(key) { - Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("collaboration map key {key} is not an i64") - }, - } -} - -fn map_binary(map: &LoroMap, key: &str) -> Result> { - match map.get(key) { - Some(ValueOrContainer::Value(LoroValue::Binary(value))) => Ok(value.to_vec()), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => { - bail!("collaboration map key {key} is not binary") - }, - } -} - -pub fn verify_lineage(doc: &LoroDoc, session: SessionId) -> Result<()> { - let meta = doc.get_map(META); - let schema = map_i64(&meta, META_SCHEMA).context("collaboration document has no schema version")?; - if schema != SCHEMA_VERSION { - bail!("collaboration document schema version {schema} is not supported") - } - let stored = map_string(&meta, META_SESSION).context("collaboration document has no session lineage")?; - if stored != session.to_string() { - bail!("collaboration document lineage does not match the ticket session") - } - Ok(()) -} diff --git a/crates/flowstate-collab/src/remote_apply.rs b/crates/flowstate-collab/src/remote_apply.rs deleted file mode 100644 index 4d92b26d..00000000 --- a/crates/flowstate-collab/src/remote_apply.rs +++ /dev/null @@ -1,675 +0,0 @@ -//! Translation from Loro diffs into flowtext collaboration patches. - -use anyhow::{Context as _, Result, bail}; -use gpui_flowtext::{ - BlockId, CollabPatch, CollabStructuralBlock, CollabTextDelta, Document, InputBlock, InputParagraph, InputRun, ParagraphId, ParagraphStyle, - TextRun, new_block_id, new_paragraph_id, paragraph_text, -}; -use loro::{ - Container, ContainerID, ContainerTrait as _, LoroDoc, LoroMap, LoroMovableList, LoroValue, TextDelta, ValueOrContainer, - cursor::PosType, - event::{Diff, DiffEvent, ListDiffItem}, -}; - -use crate::{ - binding::{BindingRow, BlockKind, DocBinding}, - projection::{input_block_from_container, input_blocks_from_loro}, - schema::{BLOCKS, DATA, REV, STYLE, body_text, decode_paragraph_style, input_runs_from_delta, paragraph_style_from_attrs, utf8_byte}, -}; - -pub struct RemoteApplier<'s> { - pub doc: &'s LoroDoc, - pub binding: &'s mut DocBinding, -} - -impl RemoteApplier<'_> { - pub fn apply_event(&mut self, document: &Document, event: &DiffEvent<'_>) -> Result> { - if !should_process_event(event) { - return Ok(Vec::new()); - } - - let pre_event_binding = self.binding.clone(); - let body_id = body_text(self.doc).id(); - let mut body_delta: Option> = None; - let mut multiple_body_deltas = false; - let inserted_containers = inserted_container_ids(self.doc, event)?; - let mut patches = Vec::new(); - for diff in &event.events { - match &diff.diff { - Diff::Text(delta) if *diff.target == body_id => { - if body_delta.replace(delta.clone()).is_some() { - multiple_body_deltas = true; - } - }, - Diff::Text(_) => {}, - Diff::Map(delta) => self.apply_map_diff( - diff.target, - delta.updated.keys().map(|key| key.as_ref()), - &inserted_containers, - &mut patches, - )?, - Diff::List(delta) => self.apply_list_diff(diff.target, delta, &mut patches)?, - Diff::Tree(_) | Diff::Counter(_) | Diff::Unknown => {}, - } - } - if let Some(delta) = body_delta { - // Fast path: a pure intra-paragraph text edit — no structural/style patches - // this event, a single contiguous change, and no newline change — reconciles - // only the affected paragraph. Anything else falls back to the full - // reprojection (which is always correct). - let handled = !multiple_body_deltas && patches.is_empty() && self.try_reconcile_body_delta(document, &delta, &mut patches)?; - if !handled { - self.reconcile_body_text(document, &pre_event_binding, &mut patches)?; - self.binding.refresh_body_index_from_loro(self.doc); - } - } - Ok(patches) - } - - /// Attempts to reconcile a body text change by inspecting the Loro `TextDelta` - /// and re-projecting only the single affected paragraph. Returns `Ok(false)` - /// (without mutating) when the change is not a clean single-paragraph, - /// no-newline edit, so the caller can fall back to the full reprojection. - fn try_reconcile_body_delta(&mut self, document: &Document, delta: &[TextDelta], patches: &mut Vec) -> Result { - // Only safe while the index mirrors the editor document's paragraph count. - if self.binding.body_index.len() != document.paragraphs.len() { - return Ok(false); - } - - // The delta must be a single contiguous change with no inserted newline. - let mut lead = 0usize; // unicode codepoints retained before the change - let mut inserted_bytes = 0usize; - let mut deleted_unicode = 0usize; - let mut in_change = false; - let mut after_change = false; - for item in delta { - match item { - TextDelta::Retain { retain, attributes } => { - // A retain carrying attributes is a mark change over existing text - // (e.g. a remote run/paragraph style edit). That changes a paragraph's - // runs without changing its text, which this text-splice fast path does - // not model — defer to the full reprojection. - if attributes - .as_ref() - .is_some_and(|attributes| !attributes.is_empty()) - { - return Ok(false); - } - if in_change { - after_change = true; - in_change = false; - } else if !after_change { - lead += retain; - } - }, - TextDelta::Insert { insert, .. } => { - if after_change || insert.contains('\n') { - return Ok(false); - } - in_change = true; - inserted_bytes += insert.len(); - }, - TextDelta::Delete { delete } => { - if after_change { - return Ok(false); - } - in_change = true; - deleted_unicode += delete; - }, - } - } - if !in_change && !after_change { - return Ok(false); - } - - let new_body = body_text(self.doc); - let change_byte = utf8_byte(&new_body, lead); - let ordinal = self - .binding - .body_index - .paragraph_ordinal_for_body_byte(change_byte); - let start_byte = self.binding.body_index.paragraph_start(ordinal); - let old_len = self.binding.body_index.paragraph_len(ordinal); - if change_byte < start_byte || change_byte > start_byte + old_len { - return Ok(false); - } - - // Require the index and the editor document to agree on this paragraph's - // pre-edit length so all byte arithmetic below is consistent. - let old_text = paragraph_text(document, ordinal); - if old_text.len() != old_len { - return Ok(false); - } - let local_offset = change_byte - start_byte; - let mut deleted_bytes = 0usize; - let mut tail = old_text[local_offset..].chars(); - for _ in 0..deleted_unicode { - // Running out of paragraph text means the deletion crossed a newline. - let Some(ch) = tail.next() else { - return Ok(false); - }; - deleted_bytes += ch.len_utf8(); - } - let new_len = old_len + inserted_bytes - deleted_bytes; - - let Some(row_ix) = row_for_paragraph_ordinal(self.binding, ordinal) else { - return Ok(false); - }; - let slice = new_body.slice_delta(start_byte, start_byte + new_len, PosType::Bytes)?; - let new_paragraph = InputParagraph { - style: paragraph_style_from_slice(&slice), - runs: input_runs_from_delta(&slice), - }; - let Some(old_paragraph) = input_paragraph_from_document(document, ordinal) else { - return Ok(false); - }; - - // Keep the index aligned with the freshly imported body for later events. - self.binding.body_index.set_paragraph_len(ordinal, new_len); - - if input_paragraphs_equal(&old_paragraph, &new_paragraph) { - return Ok(true); - } - let old_paragraph_text = input_paragraph_text(&old_paragraph); - let new_paragraph_text = input_paragraph_text(&new_paragraph); - if old_paragraph_text == new_paragraph_text { - if old_paragraph.style != new_paragraph.style { - patches.push(CollabPatch::ParagraphStyle { - row: row_ix, - style: new_paragraph.style, - }); - } - if !input_runs_equal(&old_paragraph.runs, &new_paragraph.runs) { - patches.push(CollabPatch::ParagraphRuns { - row: row_ix, - runs: text_runs_from_input(&new_paragraph), - }); - } - } else { - patches.push(CollabPatch::ParagraphText { - row: row_ix, - new: new_paragraph.clone(), - delta_utf8: text_delta_for_replacement(&old_paragraph_text, &new_paragraph_text), - }); - } - Ok(true) - } - - fn reconcile_body_text(&self, document: &Document, pre_event_binding: &DocBinding, patches: &mut Vec) -> Result<()> { - let projected_blocks = input_blocks_from_loro(self.doc)?; - for (row_ix, row) in self.binding.rows.iter().enumerate() { - if !matches!(row.kind, BlockKind::Paragraph) { - continue; - } - let Some(InputBlock::Paragraph(new)) = projected_blocks.get(row_ix) else { - continue; - }; - let old = pre_event_binding - .by_container - .get(&row.map.id()) - .and_then(|old_row_ix| paragraph_ordinal_for_row(pre_event_binding, *old_row_ix)) - .and_then(|old_paragraph_ix| input_paragraph_from_document(document, old_paragraph_ix)); - let Some(old) = old else { - patches.push(CollabPatch::ParagraphText { - row: row_ix, - new: new.clone(), - delta_utf8: text_delta_for_replacement("", &input_paragraph_text(new)), - }); - continue; - }; - if input_paragraphs_equal(&old, new) { - continue; - } - let old_text = input_paragraph_text(&old); - let new_text = input_paragraph_text(new); - if old_text == new_text { - if old.style != new.style && !has_paragraph_style_patch(patches, row_ix) { - patches.push(CollabPatch::ParagraphStyle { - row: row_ix, - style: new.style, - }); - } - if !input_runs_equal(&old.runs, &new.runs) { - patches.push(CollabPatch::ParagraphRuns { - row: row_ix, - runs: text_runs_from_input(new), - }); - } - continue; - } - patches.push(CollabPatch::ParagraphText { - row: row_ix, - new: new.clone(), - delta_utf8: text_delta_for_replacement(&old_text, &new_text), - }); - } - Ok(()) - } - - fn apply_map_diff<'a>( - &mut self, - target: &ContainerID, - keys: impl IntoIterator, - inserted_containers: &[ContainerID], - patches: &mut Vec, - ) -> Result<()> { - if inserted_containers.contains(target) { - return Ok(()); - } - let Some(row_ix) = self.binding.by_container.get(target).copied() else { - return Ok(()); - }; - let mut style_changed = false; - let mut object_changed = false; - for key in keys { - match key { - STYLE => style_changed = true, - DATA | REV => object_changed = true, - _ => {}, - } - } - let row = self - .binding - .rows - .get_mut(row_ix) - .context("map diff row is outside DocBinding")?; - if style_changed && matches!(row.kind, BlockKind::Paragraph) { - patches.push(CollabPatch::ParagraphStyle { - row: row_ix, - style: decode_paragraph_style(map_i64(&row.map, STYLE)?), - }); - } - if object_changed { - let input = input_block_from_container(&row.map, None)?; - row.kind = block_kind_for_input(&input); - row.paragraph_id = None; - row.version = 0; - patches.push(CollabPatch::ReplaceObjectBlock { - row: row_ix, - block: CollabStructuralBlock { - block_id: row.block_id, - paragraph_id: None, - block: input, - }, - }); - } - self.binding.rebuild_indexes(); - Ok(()) - } - - fn apply_list_diff(&mut self, target: &ContainerID, delta: &[ListDiffItem], patches: &mut Vec) -> Result<()> { - let blocks = self.doc.get_movable_list(BLOCKS); - if *target != blocks.id() { - return Ok(()); - } - - let mut row_ix = 0usize; - let final_block_ids = block_container_ids(&blocks)?; - for item in delta { - match item { - ListDiffItem::Retain { retain } => row_ix = row_ix.saturating_add(*retain), - ListDiffItem::Delete { delete } => { - let mut delete_start = None; - let mut delete_count = 0usize; - for _ in 0..*delete { - let Some(row) = self.binding.rows.get(row_ix) else { - break; - }; - if final_block_ids.contains(&row.map.id()) { - if let Some(start) = delete_start.take() { - patches.push(CollabPatch::DeleteBlocks { - row: start, - count: delete_count, - }); - delete_count = 0; - } - row_ix += 1; - continue; - } - delete_start.get_or_insert(row_ix); - self - .binding - .remove_row(row_ix) - .context("DocBinding row disappeared during remote block delete")?; - delete_count += 1; - } - if let Some(start) = delete_start { - patches.push(CollabPatch::DeleteBlocks { - row: start, - count: delete_count, - }); - } - }, - ListDiffItem::Insert { insert, is_move } => { - for value in insert { - let map = map_from_insert(value)?; - if *is_move && let Some(from) = self.binding.by_container.get(&map.id()).copied() { - let to = if from < row_ix { row_ix - 1 } else { row_ix }; - patches.push(CollabPatch::MoveBlock { from, to }); - self.binding.move_row(from, to); - row_ix += 1; - continue; - } - - let input = input_block_from_container(&map, None)?; - let structural = structural_block_for_insert(&input); - let row = binding_row_from_insert(map, &input, structural.block_id, structural.paragraph_id)?; - self.binding.insert_row(row_ix, row); - patches.push(CollabPatch::InsertBlocks { - row: row_ix, - blocks: vec![structural], - }); - row_ix += 1; - } - }, - } - } - reconcile_binding_to_match_blocks(self.binding, &blocks, patches)?; - Ok(()) - } -} - -#[must_use] -pub fn should_process_event(event: &DiffEvent<'_>) -> bool { - event.triggered_by.is_import() || event.triggered_by.is_checkout() -} - -fn text_delta_for_replacement(old_text: &str, new_text: &str) -> Vec { - let prefix = common_prefix_bytes(old_text, new_text); - let suffix = common_suffix_bytes(&old_text[prefix..], &new_text[prefix..]); - let old_middle = old_text.len().saturating_sub(prefix + suffix); - let new_middle = new_text.len().saturating_sub(prefix + suffix); - let mut output = Vec::with_capacity(4); - push_text_delta(&mut output, CollabTextDelta::Retain(prefix)); - push_text_delta(&mut output, CollabTextDelta::Delete(old_middle)); - push_text_delta(&mut output, CollabTextDelta::Insert(new_middle)); - push_text_delta(&mut output, CollabTextDelta::Retain(suffix)); - output -} - -fn push_text_delta(output: &mut Vec, delta: CollabTextDelta) { - match delta { - CollabTextDelta::Retain(0) | CollabTextDelta::Insert(0) | CollabTextDelta::Delete(0) => {}, - CollabTextDelta::Retain(len) => match output.last_mut() { - Some(CollabTextDelta::Retain(existing)) => *existing += len, - Some(CollabTextDelta::Insert(_)) | Some(CollabTextDelta::Delete(_)) | None => output.push(CollabTextDelta::Retain(len)), - }, - CollabTextDelta::Insert(len) => match output.last_mut() { - Some(CollabTextDelta::Insert(existing)) => *existing += len, - Some(CollabTextDelta::Retain(_)) | Some(CollabTextDelta::Delete(_)) | None => output.push(CollabTextDelta::Insert(len)), - }, - CollabTextDelta::Delete(len) => match output.last_mut() { - Some(CollabTextDelta::Delete(existing)) => *existing += len, - Some(CollabTextDelta::Retain(_)) | Some(CollabTextDelta::Insert(_)) | None => output.push(CollabTextDelta::Delete(len)), - }, - } -} - -fn common_prefix_bytes(left: &str, right: &str) -> usize { - let mut prefix = 0usize; - for (left_ch, right_ch) in left.chars().zip(right.chars()) { - if left_ch != right_ch { - break; - } - prefix += left_ch.len_utf8(); - } - prefix -} - -fn common_suffix_bytes(left: &str, right: &str) -> usize { - let mut suffix = 0usize; - for (left_ch, right_ch) in left.chars().rev().zip(right.chars().rev()) { - if left_ch != right_ch { - break; - } - let len = left_ch.len_utf8(); - if suffix + len > left.len() || suffix + len > right.len() { - break; - } - suffix += len; - } - suffix -} - -fn input_paragraph_from_document(document: &Document, paragraph_ix: usize) -> Option { - let paragraph = document.paragraphs.get(paragraph_ix)?; - let text = paragraph_text(document, paragraph_ix); - let mut byte = 0usize; - let runs = paragraph - .runs - .iter() - .map(|run| { - let start = byte; - let end = start.saturating_add(run.len).min(text.len()); - byte = end; - InputRun { - text: text.get(start..end).unwrap_or_default().to_string(), - styles: run.styles, - } - }) - .collect(); - Some(InputParagraph { - style: paragraph.style, - runs, - }) -} - -fn input_paragraph_text(paragraph: &InputParagraph) -> String { - let mut text = String::new(); - for run in ¶graph.runs { - text.push_str(&run.text); - } - text -} - -fn input_paragraphs_equal(left: &InputParagraph, right: &InputParagraph) -> bool { - left.style == right.style && input_runs_equal(&left.runs, &right.runs) -} - -fn has_paragraph_style_patch(patches: &[CollabPatch], target_row: usize) -> bool { - patches - .iter() - .any(|patch| matches!(patch, CollabPatch::ParagraphStyle { row, .. } if *row == target_row)) -} - -fn input_runs_equal(left: &[InputRun], right: &[InputRun]) -> bool { - left.len() == right.len() - && left - .iter() - .zip(right) - .all(|(left, right)| left.text == right.text && left.styles == right.styles) -} - -fn text_runs_from_input(paragraph: &InputParagraph) -> Vec { - paragraph - .runs - .iter() - .map(|run| TextRun { - len: run.text.len(), - styles: run.styles, - }) - .collect() -} - -fn map_from_insert(value: &ValueOrContainer) -> Result { - match value { - ValueOrContainer::Container(Container::Map(map)) => Ok(map.clone()), - ValueOrContainer::Value(_) | ValueOrContainer::Container(_) => bail!("remote block insert is not a map container"), - } -} - -fn inserted_container_ids(doc: &LoroDoc, event: &DiffEvent<'_>) -> Result> { - let blocks_id = doc.get_movable_list(BLOCKS).id(); - let mut ids = Vec::new(); - for diff in &event.events { - let Diff::List(delta) = &diff.diff else { - continue; - }; - if *diff.target != blocks_id { - continue; - } - for item in delta { - let ListDiffItem::Insert { insert, is_move } = item else { - continue; - }; - if *is_move { - continue; - } - for value in insert { - let map = map_from_insert(value)?; - ids.push(map.id()); - } - } - } - Ok(ids) -} - -fn block_container_ids(blocks: &LoroMovableList) -> Result> { - let mut ids = Vec::with_capacity(blocks.len()); - for ix in 0..blocks.len() { - ids.push(map_from_list(blocks, ix)?.id()); - } - Ok(ids) -} - -fn reconcile_binding_to_match_blocks(binding: &mut DocBinding, blocks: &LoroMovableList, patches: &mut Vec) -> Result<()> { - let final_ids = block_container_ids(blocks)?; - remove_stale_binding_rows(binding, &final_ids, patches); - insert_missing_binding_rows(binding, blocks, &final_ids, patches)?; - for to in (0..final_ids.len()).rev() { - if binding - .rows - .get(to) - .is_some_and(|row| row.map.id() == final_ids[to]) - { - continue; - } - let from = binding - .by_container - .get(&final_ids[to]) - .copied() - .context("final Loro block row is missing from DocBinding")?; - patches.push(CollabPatch::MoveBlock { from, to }); - binding.move_row(from, to); - } - Ok(()) -} - -fn remove_stale_binding_rows(binding: &mut DocBinding, final_ids: &[ContainerID], patches: &mut Vec) { - let mut retained_ids = Vec::new(); - let mut delete_rows = Vec::new(); - for (row_ix, row) in binding.rows.iter().enumerate() { - let id = row.map.id(); - if !final_ids.contains(&id) || retained_ids.contains(&id) { - delete_rows.push(row_ix); - } else { - retained_ids.push(id); - } - } - - for row_ix in delete_rows.into_iter().rev() { - let _ = binding.remove_row(row_ix); - patches.push(CollabPatch::DeleteBlocks { row: row_ix, count: 1 }); - } -} - -fn insert_missing_binding_rows( - binding: &mut DocBinding, - blocks: &LoroMovableList, - final_ids: &[ContainerID], - patches: &mut Vec, -) -> Result<()> { - for (row_ix, id) in final_ids.iter().enumerate() { - if binding.by_container.contains_key(id) { - continue; - } - let map = map_from_list(blocks, row_ix)?; - let input = input_block_from_container(&map, None)?; - let structural = structural_block_for_insert(&input); - let row = binding_row_from_insert(map, &input, structural.block_id, structural.paragraph_id)?; - binding.insert_row(row_ix, row); - patches.push(CollabPatch::InsertBlocks { - row: row_ix, - blocks: vec![structural], - }); - } - Ok(()) -} - -fn map_from_list(blocks: &LoroMovableList, ix: usize) -> Result { - match blocks.get(ix) { - Some(ValueOrContainer::Container(Container::Map(map))) => Ok(map), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => bail!("remote block row {ix} is not a map container"), - } -} - -fn structural_block_for_insert(input: &InputBlock) -> CollabStructuralBlock { - CollabStructuralBlock { - block_id: new_block_id(), - paragraph_id: matches!(input, InputBlock::Paragraph(_)).then(new_paragraph_id), - block: input.clone(), - } -} - -fn binding_row_from_insert(map: LoroMap, input: &InputBlock, block_id: BlockId, paragraph_id: Option) -> Result { - Ok(BindingRow { - map, - kind: block_kind_for_input(input), - block_id, - paragraph_id, - version: 0, - }) -} - -fn block_kind_for_input(input: &InputBlock) -> BlockKind { - match input { - InputBlock::Paragraph(_) => BlockKind::Paragraph, - InputBlock::Image(_) => BlockKind::Image, - InputBlock::Equation(_) => BlockKind::Equation, - InputBlock::Table(_) => BlockKind::Table, - } -} - -fn paragraph_ordinal_for_row(binding: &DocBinding, target_row: usize) -> Option { - let mut paragraph_ix = 0usize; - for (row_ix, row) in binding.rows.iter().enumerate() { - if row_ix == target_row { - return row.paragraph_id.map(|_| paragraph_ix); - } - if row.paragraph_id.is_some() { - paragraph_ix += 1; - } - } - None -} - -fn row_for_paragraph_ordinal(binding: &DocBinding, target_ordinal: usize) -> Option { - let mut paragraph_ix = 0usize; - for (row_ix, row) in binding.rows.iter().enumerate() { - if row.paragraph_id.is_some() { - if paragraph_ix == target_ordinal { - return Some(row_ix); - } - paragraph_ix += 1; - } - } - None -} - -fn paragraph_style_from_slice(delta: &[TextDelta]) -> ParagraphStyle { - for item in delta { - if let TextDelta::Insert { attributes, .. } = item - && let Some(style) = paragraph_style_from_attrs(attributes.as_ref()) - { - return style; - } - } - ParagraphStyle::Normal -} - -fn map_i64(map: &LoroMap, key: &str) -> Result { - match map.get(key) { - Some(ValueOrContainer::Value(LoroValue::I64(value))) => Ok(value), - Some(ValueOrContainer::Value(_)) | Some(ValueOrContainer::Container(_)) | None => bail!("collaboration map key {key} is not an i64"), - } -} diff --git a/crates/flowstate-collab/src/schema.rs b/crates/flowstate-collab/src/schema.rs deleted file mode 100644 index 6f396839..00000000 --- a/crates/flowstate-collab/src/schema.rs +++ /dev/null @@ -1,418 +0,0 @@ -//! Shared Loro container schema for Flowstate rich-text documents. -//! -//! The schema stores document text in one root `LoroText`. Paragraph rows in -//! the movable block list carry block identity and metadata only; paragraph -//! breaks are newline bytes in the root text. - -use std::{collections::HashMap, hash::BuildHasher, ops::Range}; - -use anyhow::Result; -use gpui_flowtext::{ - AssetStore, Block, Document, HighlightStyle, InputBlock, InputBlockAlignment, InputEquationDisplay, InputImageSizing, InputParagraph, - InputRun, InputTableBlock, RunSemanticStyle, RunStyles, TextRun, input_block_from_block, paragraph_text_len, -}; -use loro::{ExpandType, LoroDoc, LoroResult, LoroText, LoroValue, StyleConfig, StyleConfigMap, TextDelta, cursor::PosType}; -use serde::{Deserialize, Serialize}; - -pub const META: &str = "meta"; -pub const BODY: &str = "body"; -pub const BLOCKS: &str = "blocks"; -pub const META_SCHEMA: &str = "schema"; -pub const META_SESSION: &str = "session"; -pub const META_TITLE: &str = "title"; -pub const SCHEMA_VERSION: i64 = 2; - -pub const KIND: &str = "kind"; -pub const KIND_PARAGRAPH: &str = "p"; -pub const KIND_IMAGE: &str = "image"; -pub const KIND_EQUATION: &str = "equation"; -pub const KIND_TABLE: &str = "table"; -pub const STYLE: &str = "style"; -pub const DATA: &str = "data"; -pub const REV: &str = "rev"; - -pub const MARK_SEMANTIC: &str = "sem"; -pub const MARK_UNDERLINE: &str = "ul"; -pub const MARK_STRIKE: &str = "strike"; -pub const MARK_HIGHLIGHT: &str = "hl"; -pub const MARK_PARAGRAPH_STYLE: &str = "pstyle"; - -pub type MarkIntervals = [Vec<(Range, LoroValue)>; 5]; - -#[derive(Clone, Debug, Serialize, Deserialize)] -pub enum BlockPayload { - Image { - asset_id: u128, - mime: String, - original_name: Option, - content_hash: u64, - byte_len: u64, - alt_text: String, - caption: Option, - sizing: InputImageSizing, - alignment: InputBlockAlignment, - }, - Equation { - source: String, - display: InputEquationDisplay, - }, - Table(InputTableBlock), -} - -pub fn decode_block_payload(bytes: &[u8]) -> Result { - Ok(postcard::from_bytes(bytes)?) -} - -#[must_use] -pub fn payload_from_block(block: &Block, assets: &AssetStore) -> Option { - match input_block_from_block(block) { - InputBlock::Paragraph(_) => None, - InputBlock::Image(image) => { - let asset = assets.assets.get(&image.asset_id); - Some(BlockPayload::Image { - asset_id: image.asset_id.0, - mime: asset.map_or_else(String::new, |asset| asset.mime_type.to_string()), - original_name: asset.and_then(|asset| asset.original_name.as_ref().map(ToString::to_string)), - content_hash: asset.map_or(0, |asset| asset.content_hash), - byte_len: asset.map_or(0, |asset| asset.bytes.len() as u64), - alt_text: image.alt_text, - caption: image.caption, - sizing: image.sizing, - alignment: image.alignment, - }) - }, - InputBlock::Equation(equation) => Some(BlockPayload::Equation { - source: equation.source, - display: equation.display, - }), - InputBlock::Table(table) => Some(BlockPayload::Table(table)), - } -} - -#[must_use] -pub fn block_from_payload(payload: BlockPayload, _assets: &AssetStore) -> InputBlock { - match payload { - BlockPayload::Image { - asset_id, - alt_text, - caption, - sizing, - alignment, - mime: _, - original_name: _, - content_hash: _, - byte_len: _, - } => InputBlock::Image(gpui_flowtext::InputImageBlock { - asset_id: gpui_flowtext::AssetId(asset_id), - alt_text, - caption, - sizing, - alignment, - }), - BlockPayload::Equation { source, display } => InputBlock::Equation(gpui_flowtext::InputEquationBlock { - source, - syntax: gpui_flowtext::InputEquationSyntax::Latex, - display, - }), - BlockPayload::Table(table) => InputBlock::Table(table), - } -} - -#[must_use] -pub fn new_configured_doc() -> LoroDoc { - let doc = LoroDoc::new(); - configure_text_styles(&doc); - doc -} - -pub fn configure_text_styles(doc: &LoroDoc) { - let mut styles = StyleConfigMap::new(); - let no_expand = StyleConfig { expand: ExpandType::None }; - styles.insert(MARK_SEMANTIC.into(), no_expand); - styles.insert(MARK_UNDERLINE.into(), no_expand); - styles.insert(MARK_STRIKE.into(), no_expand); - styles.insert(MARK_HIGHLIGHT.into(), no_expand); - styles.insert(MARK_PARAGRAPH_STYLE.into(), StyleConfig { expand: ExpandType::Both }); - doc.config_text_style(styles); -} - -#[must_use] -pub fn body_text(doc: &LoroDoc) -> LoroText { - doc.get_text(BODY) -} - -#[must_use] -pub fn text_content(text: &LoroText) -> String { - let mut content = String::new(); - for item in text.to_delta() { - let TextDelta::Insert { insert, .. } = item else { - continue; - }; - content.push_str(&insert); - } - content -} - -#[must_use] -pub fn loro_pos(text: &LoroText, utf8_byte: usize) -> usize { - text - .convert_pos(utf8_byte, PosType::Bytes, PosType::Unicode) - .expect("UTF-8 byte offset must be in bounds for LoroText") -} - -#[must_use] -pub fn utf8_byte(text: &LoroText, loro_pos: usize) -> usize { - text - .convert_pos(loro_pos, PosType::Unicode, PosType::Bytes) - .expect("Unicode offset must be in bounds for LoroText") -} - -pub fn set_run_styles_utf8(text: &LoroText, range: Range, styles: RunStyles) -> LoroResult<()> { - if range.is_empty() { - return Ok(()); - } - - match styles.semantic { - RunSemanticStyle::Plain => unmark_utf8(text, range.clone(), MARK_SEMANTIC)?, - RunSemanticStyle::Custom(slot) => text.mark_utf8(range.clone(), MARK_SEMANTIC, LoroValue::I64(i64::from(slot)))?, - } - if styles.direct_underline { - text.mark_utf8(range.clone(), MARK_UNDERLINE, LoroValue::Bool(true))?; - } else { - unmark_utf8(text, range.clone(), MARK_UNDERLINE)?; - } - if styles.strikethrough { - text.mark_utf8(range.clone(), MARK_STRIKE, LoroValue::Bool(true))?; - } else { - unmark_utf8(text, range.clone(), MARK_STRIKE)?; - } - if let Some(HighlightStyle::Custom(slot)) = styles.highlight { - text.mark_utf8(range, MARK_HIGHLIGHT, LoroValue::I64(i64::from(slot)))?; - } else { - unmark_utf8(text, range, MARK_HIGHLIGHT)?; - } - Ok(()) -} - -pub fn set_paragraph_style_utf8(text: &LoroText, range: Range, style: gpui_flowtext::ParagraphStyle) -> LoroResult<()> { - if range.is_empty() { - return Ok(()); - } - match style { - gpui_flowtext::ParagraphStyle::Normal => unmark_utf8(text, range, MARK_PARAGRAPH_STYLE), - gpui_flowtext::ParagraphStyle::Custom(slot) => text.mark_utf8(range, MARK_PARAGRAPH_STYLE, LoroValue::I64(i64::from(slot))), - } -} - -pub fn unmark_utf8(text: &LoroText, range: Range, key: &str) -> LoroResult<()> { - if range.is_empty() { - return Ok(()); - } - let start = loro_pos(text, range.start); - let end = loro_pos(text, range.end); - text.unmark(start..end, key) -} - -#[must_use] -pub fn input_runs_from_delta(delta: &[TextDelta]) -> Vec { - let mut runs: Vec = Vec::new(); - for item in delta { - let TextDelta::Insert { insert, attributes } = item else { - continue; - }; - if insert.is_empty() { - continue; - } - let styles = run_styles_from_attrs(attributes.as_ref()); - if let Some(last) = runs.last_mut() - && last.styles == styles - { - last.text.push_str(insert); - continue; - } - runs.push(InputRun { - text: insert.clone(), - styles, - }); - } - runs -} - -#[must_use] -pub fn input_paragraph_from_text(text: &LoroText, style: gpui_flowtext::ParagraphStyle) -> InputParagraph { - InputParagraph { - style, - runs: input_runs_from_delta(&text.to_delta()), - } -} - -#[must_use] -pub fn input_paragraphs_from_body_text(text: &LoroText) -> Vec { - let mut paragraphs = vec![InputParagraph { - style: gpui_flowtext::ParagraphStyle::Normal, - runs: Vec::new(), - }]; - for item in text.to_delta() { - let TextDelta::Insert { insert, attributes } = item else { - continue; - }; - if insert.is_empty() { - continue; - } - let styles = run_styles_from_attrs(attributes.as_ref()); - let paragraph_style = paragraph_style_from_attrs(attributes.as_ref()); - for (ix, segment) in insert.split('\n').enumerate() { - if ix > 0 { - if let Some(style) = paragraph_style { - paragraphs - .last_mut() - .expect("paragraph accumulator is never empty") - .style = style; - } - paragraphs.push(InputParagraph { - style: paragraph_style.unwrap_or(gpui_flowtext::ParagraphStyle::Normal), - runs: Vec::new(), - }); - } - if !segment.is_empty() { - if let Some(style) = paragraph_style { - paragraphs - .last_mut() - .expect("paragraph accumulator is never empty") - .style = style; - } - push_input_run( - paragraphs - .last_mut() - .expect("paragraph accumulator is never empty"), - segment, - styles, - ); - } - } - } - paragraphs -} - -#[must_use] -pub fn paragraph_style_from_attrs(attrs: Option<&HashMap>) -> Option -where - S: BuildHasher, -{ - attrs - .and_then(|attrs| attrs.get(MARK_PARAGRAPH_STYLE)) - .and_then(loro_i64) - .map(decode_paragraph_style) -} - -pub fn push_input_run(paragraph: &mut InputParagraph, text: &str, styles: RunStyles) { - if let Some(last) = paragraph.runs.last_mut() - && last.styles == styles - { - last.text.push_str(text); - return; - } - paragraph.runs.push(InputRun { - text: text.to_string(), - styles, - }); -} - -#[must_use] -pub fn run_styles_from_attrs(attrs: Option<&HashMap>) -> RunStyles -where - S: BuildHasher, -{ - RunStyles { - semantic: attrs - .and_then(|attrs| attrs.get(MARK_SEMANTIC)) - .and_then(loro_i64) - .and_then(|slot| u8::try_from(slot).ok()) - .map_or(RunSemanticStyle::Plain, RunSemanticStyle::Custom), - direct_underline: attrs - .and_then(|attrs| attrs.get(MARK_UNDERLINE)) - .is_some_and(loro_true), - strikethrough: attrs - .and_then(|attrs| attrs.get(MARK_STRIKE)) - .is_some_and(loro_true), - highlight: attrs - .and_then(|attrs| attrs.get(MARK_HIGHLIGHT)) - .and_then(loro_i64) - .and_then(|slot| u8::try_from(slot).ok()) - .map(HighlightStyle::Custom), - } -} - -#[must_use] -pub fn mark_intervals_from_runs(runs: &[TextRun]) -> MarkIntervals { - let mut intervals: MarkIntervals = [Vec::new(), Vec::new(), Vec::new(), Vec::new(), Vec::new()]; - let mut byte = 0; - for run in runs { - let range = byte..byte + run.len; - byte = range.end; - match run.styles.semantic { - RunSemanticStyle::Plain => {}, - RunSemanticStyle::Custom(slot) => intervals[0].push((range.clone(), LoroValue::I64(i64::from(slot)))), - } - if run.styles.direct_underline { - intervals[1].push((range.clone(), LoroValue::Bool(true))); - } - if run.styles.strikethrough { - intervals[2].push((range.clone(), LoroValue::Bool(true))); - } - if let Some(HighlightStyle::Custom(slot)) = run.styles.highlight { - intervals[3].push((range, LoroValue::I64(i64::from(slot)))); - } - } - intervals -} - -pub fn apply_mark_intervals(text: &LoroText, intervals: &MarkIntervals) -> Result<()> { - let keys = [MARK_SEMANTIC, MARK_UNDERLINE, MARK_STRIKE, MARK_HIGHLIGHT, MARK_PARAGRAPH_STYLE]; - for (key, intervals) in keys.into_iter().zip(intervals) { - for (range, value) in intervals { - text.mark_utf8(range.clone(), key, value.clone())?; - } - } - Ok(()) -} - -pub fn debug_assert_paragraph_text_len(text: &LoroText, document: &Document, paragraph_ix: usize) { - if let Some(paragraph) = document.paragraphs.get(paragraph_ix) { - debug_assert_eq!(text.len_utf8(), paragraph_text_len(paragraph)); - } -} - -#[must_use] -pub const fn encode_paragraph_style(style: gpui_flowtext::ParagraphStyle) -> i64 { - match style { - gpui_flowtext::ParagraphStyle::Normal => -1, - gpui_flowtext::ParagraphStyle::Custom(slot) => slot as i64, - } -} - -#[must_use] -pub fn decode_paragraph_style(value: i64) -> gpui_flowtext::ParagraphStyle { - if value < 0 { - return gpui_flowtext::ParagraphStyle::Normal; - } - u8::try_from(value).map_or(gpui_flowtext::ParagraphStyle::Normal, gpui_flowtext::ParagraphStyle::Custom) -} - -fn loro_i64(value: &LoroValue) -> Option { - match value { - LoroValue::I64(value) => Some(*value), - LoroValue::Null - | LoroValue::Bool(_) - | LoroValue::Double(_) - | LoroValue::Binary(_) - | LoroValue::String(_) - | LoroValue::List(_) - | LoroValue::Map(_) - | LoroValue::Container(_) => None, - } -} - -fn loro_true(value: &LoroValue) -> bool { - matches!(value, LoroValue::Bool(true)) -} diff --git a/crates/flowstate-collab/src/self_check.rs b/crates/flowstate-collab/src/self_check.rs index 671406fd..af35b523 100644 --- a/crates/flowstate-collab/src/self_check.rs +++ b/crates/flowstate-collab/src/self_check.rs @@ -5,8 +5,6 @@ use std::hash::{Hash, Hasher}; use gpui_flowtext::{Block, Document, paragraph_text}; use twox_hash::XxHash64; -use crate::schema::payload_from_block; - #[must_use] pub fn projection_hash(document: &Document) -> u64 { let mut hasher = XxHash64::default(); @@ -27,11 +25,7 @@ pub fn projection_hash(document: &Document) -> u64 { }, Block::Image(_) | Block::Equation(_) | Block::Table(_) => { std::mem::discriminant(block).hash(&mut hasher); - if let Some(payload) = payload_from_block(block, &document.assets) - && let Ok(bytes) = postcard::to_stdvec(&payload) - { - bytes.hash(&mut hasher); - } + format!("{block:?}").hash(&mut hasher); }, } } diff --git a/crates/flowstate-collab/tests/convergence.rs b/crates/flowstate-collab/tests/convergence.rs deleted file mode 100644 index fe668f96..00000000 --- a/crates/flowstate-collab/tests/convergence.rs +++ /dev/null @@ -1,1158 +0,0 @@ -#[cfg(test)] -mod tests { - use std::{ - fmt::Write as _, - ops::Range, - sync::{Arc, Mutex}, - }; - - use anyhow::{Context as _, Result, anyhow, bail}; - use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, patch_apply::apply_patches, projection, schema}; - use gpui_flowtext::{ - Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, HighlightStyle, InputBlock, InputEquationBlock, InputEquationDisplay, - InputEquationSyntax, InputParagraph, ParagraphStyle, RunSemanticStyle, RunStyle, RunStyles, block_from_input_block, - delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, mutate_runs_in_range, - paragraph_text, paragraphs_mut, plain, remove_block_ids, split_paragraph_at, update_paragraph_block, - }; - use loro::{ExportMode, LoroDoc, Subscription, event::Subscriber}; - use proptest::{prelude::*, test_runner::TestCaseError}; - - const MULTIBYTE: &str = "aé🌍\u{2028}x"; - - proptest! { - #![proptest_config(ProptestConfig { cases: 256, max_shrink_iters: 2048, ..ProptestConfig::default() })] - - #[test] - fn peers_converge_with_reordered_duplicate_and_buffered_updates( - peer_count in 2usize..=3, - delivery_seed in any::(), - ops in prop::collection::vec(fuzz_op(), 1..28), - ) { - run_program(peer_count, delivery_seed, &ops) - .map_err(|error| TestCaseError::fail(format!("{error:#}\nprogram: {ops:#?}")))?; - } - } - - #[test] - fn reordered_run_style_after_inserts_converges() { - let ops = [ - FuzzOp::Style { - peer: 0, - paragraph: 0, - start: 0, - len: 0, - style: 0, - }, - FuzzOp::InsertText { - peer: 0, - paragraph: 0, - byte: 0, - text: 0, - style: 0, - }, - FuzzOp::Style { - peer: 0, - paragraph: 0, - start: 0, - len: 0, - style: 0, - }, - FuzzOp::InsertText { - peer: 0, - paragraph: 0, - byte: 0, - text: 0, - style: 0, - }, - FuzzOp::DeleteText { - peer: 0, - paragraph: 61, - start: 31, - len: 29, - }, - FuzzOp::DeleteText { - peer: 138, - paragraph: 161, - start: 6, - len: 0, - }, - FuzzOp::InsertText { - peer: 0, - paragraph: 125, - byte: 20, - text: 0, - style: 0, - }, - FuzzOp::Style { - peer: 2, - paragraph: 12, - start: 0, - len: 0, - style: 1, - }, - ]; - - run_program(2, 4754897484207400829, &ops).expect("reordered run style after inserts should converge"); - } - - #[test] - fn style_then_join_converges_when_updates_reorder() { - let ops = [ - FuzzOp::Style { - peer: 152, - paragraph: 0, - start: 87, - len: 53, - style: 3, - }, - FuzzOp::Join { peer: 162, paragraph: 0 }, - ]; - - run_program(2, 0, &ops).expect("style then join should converge"); - } - - #[test] - fn insert_object_then_split_converges() { - let ops = [ - FuzzOp::InsertObject { peer: 6, row: 70, source: 0 }, - FuzzOp::Split { - peer: 0, - paragraph: 188, - byte: 0, - }, - ]; - - run_program(2, 0, &ops).expect("insert object then split should converge"); - } - - #[test] - fn insert_object_at_start_then_split_converges_across_three_peers() { - let ops = [ - FuzzOp::InsertObject { - peer: 96, - row: 129, - source: 0, - }, - FuzzOp::Split { - peer: 108, - paragraph: 91, - byte: 0, - }, - ]; - - run_program(3, 0, &ops).expect("insert object at start then split should converge"); - } - - #[test] - fn insert_object_between_paragraphs_does_not_create_invalid_join() { - let ops = [ - FuzzOp::InsertObject { peer: 6, row: 70, source: 0 }, - FuzzOp::Join { peer: 162, paragraph: 0 }, - ]; - - run_program(2, 0, &ops).expect("non-adjacent paragraph join should be skipped"); - } - - #[test] - fn insert_object_between_paragraphs_then_text_edit_preserves_order() { - let ops = [ - FuzzOp::InsertObject { - peer: 7, - row: 106, - source: 0, - }, - FuzzOp::DeleteText { - peer: 19, - paragraph: 48, - start: 0, - len: 0, - }, - ]; - - run_program(2, 0, &ops).expect("text edit after object insert should preserve block order"); - } - - #[test] - fn concurrent_insert_after_split_point_stays_in_second_half() -> Result<()> { - let session = SessionId::from_bytes([88; 32]); - let initial = document_from_input_blocks(DocumentTheme::default(), vec![paragraph_block("abcd")]); - let mut amy = Peer::from_initial(session, initial)?; - let snapshot = amy.snapshot()?; - let mut bob = Peer::from_snapshot(session, &snapshot)?; - - amy.apply_op(&FuzzOp::Split { - peer: 0, - paragraph: 0, - byte: 2, - })?; - let amy_updates = amy.drain_updates(); - - bob.apply_op(&FuzzOp::InsertText { - peer: 1, - paragraph: 0, - byte: 3, - text: 4, - style: 0, - })?; - let bob_updates = bob.drain_updates(); - - for update in &bob_updates { - amy.import_update(update)?; - } - for update in &amy_updates { - bob.import_update(update)?; - } - - assert_eq!(paragraph_text(&amy.document, 0), "ab"); - assert_eq!(paragraph_text(&amy.document, 1), "cxd"); - assert_eq!(paragraph_text(&bob.document, 0), "ab"); - assert_eq!(paragraph_text(&bob.document, 1), "cxd"); - Ok(()) - } - - #[test] - fn concurrent_paragraph_style_applies_to_both_split_halves() -> Result<()> { - let session = SessionId::from_bytes([89; 32]); - let initial = document_from_input_blocks(DocumentTheme::default(), vec![paragraph_block("abcd")]); - let mut amy = Peer::from_initial(session, initial)?; - let snapshot = amy.snapshot()?; - let mut bob = Peer::from_snapshot(session, &snapshot)?; - - amy.apply_op(&FuzzOp::Split { - peer: 0, - paragraph: 0, - byte: 2, - })?; - let amy_updates = amy.drain_updates(); - - let paragraph = bob.document.ids.paragraph_ids[0]; - paragraphs_mut(&mut bob.document)[0].style = ParagraphStyle::Custom(3); - update_paragraph_block(&mut bob.document, 0); - LocalApplier { - doc: &bob.loro, - binding: &mut bob.binding, - } - .apply( - &bob.document, - &[CanonicalOperation::SetParagraphStyle { - paragraph, - style: ParagraphStyle::Custom(3), - }], - )?; - let bob_updates = bob.drain_updates(); - - for update in &bob_updates { - amy.import_update(update)?; - } - for update in &amy_updates { - bob.import_update(update)?; - } - - for peer in [&amy, &bob] { - assert_eq!(paragraph_text(&peer.document, 0), "ab"); - assert_eq!(paragraph_text(&peer.document, 1), "cd"); - assert_eq!(peer.document.paragraphs[0].style, ParagraphStyle::Custom(3)); - assert_eq!(peer.document.paragraphs[1].style, ParagraphStyle::Custom(3)); - } - Ok(()) - } - - #[test] - fn concurrent_object_insert_move_interleaving_converges() { - let ops = [ - FuzzOp::InsertText { - peer: 0, - paragraph: 0, - byte: 0, - text: 0, - style: 0, - }, - FuzzOp::InsertText { - peer: 0, - paragraph: 0, - byte: 0, - text: 0, - style: 0, - }, - FuzzOp::Join { peer: 86, paragraph: 76 }, - FuzzOp::Style { - peer: 144, - paragraph: 122, - start: 123, - len: 147, - style: 171, - }, - FuzzOp::DeleteObject { peer: 212, row: 116 }, - FuzzOp::Split { - peer: 44, - paragraph: 85, - byte: 123, - }, - FuzzOp::InsertText { - peer: 47, - paragraph: 61, - byte: 160, - text: 91, - style: 192, - }, - FuzzOp::DeleteObject { peer: 216, row: 243 }, - FuzzOp::DeleteText { - peer: 86, - paragraph: 45, - start: 114, - len: 95, - }, - FuzzOp::InsertObject { - peer: 122, - row: 21, - source: 177, - }, - FuzzOp::InsertText { - peer: 44, - paragraph: 23, - byte: 38, - text: 111, - style: 124, - }, - FuzzOp::MoveObject { peer: 1, row: 223, to: 58 }, - FuzzOp::InsertObject { - peer: 167, - row: 184, - source: 207, - }, - FuzzOp::InsertText { - peer: 199, - paragraph: 128, - byte: 125, - text: 41, - style: 165, - }, - FuzzOp::MoveObject { - peer: 211, - row: 103, - to: 135, - }, - ]; - - run_program(2, 9640017939277306283, &ops).expect("concurrent object move interleaving should converge"); - } - - #[test] - fn buffered_object_split_join_move_program_converges() { - let ops = [ - FuzzOp::Style { - peer: 34, - paragraph: 235, - start: 63, - len: 151, - style: 35, - }, - FuzzOp::DeleteText { - peer: 10, - paragraph: 205, - start: 201, - len: 24, - }, - FuzzOp::Style { - peer: 244, - paragraph: 167, - start: 13, - len: 60, - style: 35, - }, - FuzzOp::InsertText { - peer: 159, - paragraph: 175, - byte: 242, - text: 64, - style: 7, - }, - FuzzOp::Style { - peer: 86, - paragraph: 113, - start: 237, - len: 135, - style: 240, - }, - FuzzOp::DeleteText { - peer: 186, - paragraph: 150, - start: 45, - len: 253, - }, - FuzzOp::InsertText { - peer: 236, - paragraph: 70, - byte: 124, - text: 228, - style: 238, - }, - FuzzOp::Style { - peer: 212, - paragraph: 57, - start: 248, - len: 168, - style: 47, - }, - FuzzOp::Style { - peer: 128, - paragraph: 69, - start: 13, - len: 201, - style: 209, - }, - FuzzOp::InsertObject { - peer: 208, - row: 159, - source: 38, - }, - FuzzOp::DeleteText { - peer: 70, - paragraph: 2, - start: 247, - len: 227, - }, - FuzzOp::DeleteText { - peer: 45, - paragraph: 59, - start: 175, - len: 74, - }, - FuzzOp::InsertText { - peer: 234, - paragraph: 209, - byte: 216, - text: 230, - style: 207, - }, - FuzzOp::DeleteText { - peer: 197, - paragraph: 38, - start: 169, - len: 78, - }, - FuzzOp::InsertText { - peer: 118, - paragraph: 146, - byte: 65, - text: 27, - style: 176, - }, - FuzzOp::Split { - peer: 32, - paragraph: 124, - byte: 11, - }, - FuzzOp::Join { peer: 19, paragraph: 41 }, - FuzzOp::MoveObject { peer: 94, row: 196, to: 197 }, - FuzzOp::DeleteText { - peer: 56, - paragraph: 171, - start: 218, - len: 154, - }, - FuzzOp::DeleteObject { peer: 222, row: 98 }, - ]; - - run_program(3, 6385737403235304726, &ops).expect("minimized buffered update program should converge"); - } - - #[derive(Clone, Debug)] - enum FuzzOp { - InsertText { - peer: u8, - paragraph: u8, - byte: u8, - text: u8, - style: u8, - }, - DeleteText { - peer: u8, - paragraph: u8, - start: u8, - len: u8, - }, - Split { - peer: u8, - paragraph: u8, - byte: u8, - }, - Join { - peer: u8, - paragraph: u8, - }, - Style { - peer: u8, - paragraph: u8, - start: u8, - len: u8, - style: u8, - }, - InsertObject { - peer: u8, - row: u8, - source: u8, - }, - DeleteObject { - peer: u8, - row: u8, - }, - MoveObject { - peer: u8, - row: u8, - to: u8, - }, - } - - impl FuzzOp { - fn peer(&self) -> u8 { - match self { - Self::InsertText { peer, .. } - | Self::DeleteText { peer, .. } - | Self::Split { peer, .. } - | Self::Join { peer, .. } - | Self::Style { peer, .. } - | Self::InsertObject { peer, .. } - | Self::DeleteObject { peer, .. } - | Self::MoveObject { peer, .. } => *peer, - } - } - } - - fn fuzz_op() -> impl Strategy { - prop_oneof![ - 7 => (any::(), any::(), any::(), any::(), any::()).prop_map(|(peer, paragraph, byte, text, style)| FuzzOp::InsertText { peer, paragraph, byte, text, style }), - 7 => (any::(), any::(), any::(), any::()).prop_map(|(peer, paragraph, start, len)| FuzzOp::DeleteText { peer, paragraph, start, len }), - 1 => (any::(), any::(), any::()).prop_map(|(peer, paragraph, byte)| FuzzOp::Split { peer, paragraph, byte }), - 1 => (any::(), any::()).prop_map(|(peer, paragraph)| FuzzOp::Join { peer, paragraph }), - 2 => (any::(), any::(), any::(), any::(), any::()).prop_map(|(peer, paragraph, start, len, style)| FuzzOp::Style { peer, paragraph, start, len, style }), - 1 => (any::(), any::(), any::()).prop_map(|(peer, row, source)| FuzzOp::InsertObject { peer, row, source }), - 1 => (any::(), any::()).prop_map(|(peer, row)| FuzzOp::DeleteObject { peer, row }), - 1 => (any::(), any::(), any::()).prop_map(|(peer, row, to)| FuzzOp::MoveObject { peer, row, to }), - ] - } - - fn run_program(peer_count: usize, delivery_seed: u64, ops: &[FuzzOp]) -> Result<()> { - let session = SessionId::from_bytes([77; 32]); - let initial = document_from_input_blocks(DocumentTheme::default(), vec![paragraph_block(MULTIBYTE), paragraph_block("second aé🌍")]); - let mut peers = Vec::with_capacity(peer_count); - let first = Peer::from_initial(session, initial)?; - let snapshot = first.snapshot()?; - peers.push(first); - for _ in 1..peer_count { - peers.push(Peer::from_snapshot(session, &snapshot)?); - } - - let mut network = Vec::new(); - for (step, op) in ops.iter().enumerate() { - let source = usize::from(op.peer()) % peer_count; - if !peers[source].apply_op(op)? { - continue; - } - for update in peers[source].drain_updates() { - for target in 0..peer_count { - if target == source { - continue; - } - let copies = 1 + usize::from(pseudo(delivery_seed, step, source, target, 0).is_multiple_of(5)); - for copy in 0..copies { - let buffered = pseudo(delivery_seed, step, source, target, copy).is_multiple_of(7); - let key = pseudo(delivery_seed, step, source, target, copy + 17).saturating_add(if buffered { 10_000 + step as u64 } else { 0 }); - network.push(Message { - key, - target, - bytes: update.clone(), - }); - } - } - } - } - - network.sort_by_key(|message| message.key); - for message in network { - peers[message.target].import_update(&message.bytes)?; - } - - let projected = projection::document_from_loro(&peers[0].loro, DocumentTheme::default())?; - let expected = canonical_document_bytes(&projected)?; - for (peer_ix, peer) in peers.iter().enumerate() { - let peer_projection = projection::document_from_loro(&peer.loro, DocumentTheme::default())?; - let projection_bytes = canonical_document_bytes(&peer_projection)?; - let document_bytes = canonical_document_bytes(&peer.document)?; - if projection_bytes != expected { - bail!("peer Loro projection diverged"); - } - if document_bytes != expected { - bail!( - "peer {peer_ix} document diverged from Loro projection\nexpected: {}\nactual: {}\npeer projection: {}", - describe_document(&projected), - describe_document(&peer.document), - describe_document(&peer_projection), - ); - } - } - Ok(()) - } - - struct Message { - key: u64, - target: usize, - bytes: Vec, - } - - struct Peer { - document: Document, - loro: LoroDoc, - binding: DocBinding, - updates: Arc>>>, - _subscription: Subscription, - } - - impl Peer { - fn from_initial(session: SessionId, document: Document) -> Result { - let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, session, "convergence", &document)?; - let binding = DocBinding::build(&loro, &document)?; - Ok(Self::with_subscription(document, loro, binding)) - } - - fn from_snapshot(session: SessionId, snapshot: &[u8]) -> Result { - let loro = schema::new_configured_doc(); - loro.import(snapshot)?; - projection::verify_lineage(&loro, session)?; - let document = projection::document_from_loro(&loro, DocumentTheme::default())?; - let binding = DocBinding::build(&loro, &document)?; - Ok(Self::with_subscription(document, loro, binding)) - } - - fn with_subscription(document: Document, loro: LoroDoc, binding: DocBinding) -> Self { - let updates = Arc::new(Mutex::new(Vec::new())); - let captured = updates.clone(); - let subscription = loro.subscribe_local_update(Box::new(move |bytes| { - captured - .lock() - .expect("update capture lock should not be poisoned") - .push(bytes.clone()); - true - })); - Self { - document, - loro, - binding, - updates, - _subscription: subscription, - } - } - - fn snapshot(&self) -> Result> { - Ok(self.loro.export(ExportMode::Snapshot)?) - } - - fn drain_updates(&self) -> Vec> { - let mut updates = self - .updates - .lock() - .expect("update capture lock should not be poisoned"); - std::mem::take(&mut *updates) - } - - fn apply_op(&mut self, op: &FuzzOp) -> Result { - let Some(ops) = self.mutate_document(op)? else { - return Ok(false); - }; - LocalApplier { - doc: &self.loro, - binding: &mut self.binding, - } - .apply(&self.document, &ops)?; - Ok(true) - } - - fn mutate_document(&mut self, op: &FuzzOp) -> Result>> { - match *op { - FuzzOp::InsertText { - paragraph, - byte, - text, - style, - .. - } => { - if self.document.paragraphs.is_empty() { - return Ok(None); - } - let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); - let value = text_choice(text); - let styles = style_choice(style); - let offset = byte_boundary(¶graph_text(&self.document, paragraph_ix), byte); - let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; - insert_text_preserving_blocks(&mut self.document, paragraph_ix, offset, value, styles); - Ok(Some(vec![CanonicalOperation::InsertText { - paragraph: paragraph_id, - byte: offset, - text: value.to_string(), - styles, - }])) - }, - FuzzOp::DeleteText { paragraph, start, len, .. } => { - if self.document.paragraphs.is_empty() { - return Ok(None); - } - let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); - let text = paragraph_text(&self.document, paragraph_ix); - let Some(range) = non_empty_range(&text, start, len) else { - return Ok(None); - }; - let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; - delete_text_preserving_blocks(&mut self.document, paragraph_ix, range.clone()); - Ok(Some(vec![CanonicalOperation::DeleteRange { - start_paragraph: paragraph_id, - start_byte: range.start, - end_paragraph: paragraph_id, - end_byte: range.end, - }])) - }, - FuzzOp::Split { paragraph, byte, .. } => { - if self.document.paragraphs.len() >= 6 || self.document.paragraphs.is_empty() { - return Ok(None); - } - let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); - let split_byte = byte_boundary(¶graph_text(&self.document, paragraph_ix), byte); - let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; - let Some(new_paragraph) = split_paragraph_preserving_blocks(&mut self.document, paragraph_ix, split_byte) else { - return Ok(None); - }; - Ok(Some(vec![CanonicalOperation::SplitParagraph { - paragraph: paragraph_id, - byte: split_byte, - new_paragraph, - }])) - }, - FuzzOp::Join { paragraph, .. } => { - let pairs = adjacent_paragraph_pairs(&self.document); - if pairs.is_empty() { - return Ok(None); - } - let first_ix = pairs[usize::from(paragraph) % pairs.len()]; - let first = self.document.ids.paragraph_ids[first_ix]; - let second = self.document.ids.paragraph_ids[first_ix + 1]; - let first_len = paragraph_text(&self.document, first_ix).len(); - delete_cross_paragraph_range( - &mut self.document, - DocumentOffset { - paragraph: first_ix, - byte: first_len, - }..DocumentOffset { - paragraph: first_ix + 1, - byte: 0, - }, - ); - Ok(Some(vec![CanonicalOperation::JoinParagraphs { first, second }])) - }, - FuzzOp::Style { - paragraph, - start, - len, - style, - .. - } => { - if self.document.paragraphs.is_empty() { - return Ok(None); - } - let paragraph_ix = usize::from(paragraph) % self.document.paragraphs.len(); - let text = paragraph_text(&self.document, paragraph_ix); - let Some(range) = non_empty_range(&text, start, len) else { - return Ok(None); - }; - let styles = style_choice(style); - let paragraph_id = self.document.ids.paragraph_ids[paragraph_ix]; - mutate_runs_in_range( - &mut self.document, - DocumentOffset { - paragraph: paragraph_ix, - byte: range.start, - }..DocumentOffset { - paragraph: paragraph_ix, - byte: range.end, - }, - |run_styles| *run_styles = styles, - ); - Ok(Some(vec![CanonicalOperation::SetRunStyles { - paragraph: paragraph_id, - range, - styles, - }])) - }, - FuzzOp::InsertObject { row, source, .. } => { - if self.document.blocks.len() >= 8 { - return Ok(None); - } - let row = usize::from(row) % (self.document.blocks.len() + 1); - let input = equation_block(equation_source(source)); - Arc::make_mut(&mut self.document.blocks).insert(row, block_from_input_block(&input)); - let block = insert_block_id(&mut self.document, row); - Ok(Some(vec![CanonicalOperation::InsertBlock { block, block_ix: row }])) - }, - FuzzOp::DeleteObject { row, .. } => { - let objects = object_rows(&self.document); - if objects.is_empty() { - return Ok(None); - } - let row = objects[usize::from(row) % objects.len()]; - let block = self.document.ids.block_ids[row]; - Arc::make_mut(&mut self.document.blocks).remove(row); - remove_block_ids(&mut self.document, row..row + 1); - Ok(Some(vec![CanonicalOperation::DeleteBlock { block }])) - }, - FuzzOp::MoveObject { row, to, .. } => { - let objects = object_rows(&self.document); - if objects.is_empty() || self.document.blocks.len() < 2 { - return Ok(None); - } - let from = objects[usize::from(row) % objects.len()]; - let to = usize::from(to) % self.document.blocks.len(); - if from == to { - return Ok(None); - } - let block = self.document.ids.block_ids[from]; - let moved_block = Arc::make_mut(&mut self.document.blocks).remove(from); - let insert_ix = to.min(self.document.blocks.len()); - Arc::make_mut(&mut self.document.blocks).insert(insert_ix, moved_block); - let block_id = self.document.ids.block_ids.remove(from); - self - .document - .ids - .block_ids - .insert(insert_ix.min(self.document.ids.block_ids.len()), block_id); - Ok(Some(vec![CanonicalOperation::MoveBlock { block, new_block_ix: to }])) - }, - } - } - - fn import_update(&mut self, update: &[u8]) -> Result<()> { - let document = Arc::new(self.document.clone()); - let binding = Arc::new(Mutex::new(std::mem::take(&mut self.binding))); - let patches = Arc::new(Mutex::new(Vec::new())); - let error = Arc::new(Mutex::new(None::)); - let doc = self.loro.clone(); - let callback: Subscriber = Arc::new({ - let binding = binding.clone(); - let patches = patches.clone(); - let error = error.clone(); - move |event| { - let result = binding - .lock() - .map_err(|lock_error| anyhow!("binding lock poisoned: {lock_error}")) - .and_then(|mut binding| { - flowstate_collab::remote_apply::RemoteApplier { - doc: &doc, - binding: &mut binding, - } - .apply_event(&document, &event) - }); - match result { - Ok(mut produced) => patches - .lock() - .expect("patch lock should not be poisoned") - .append(&mut produced), - Err(apply_error) => *error.lock().expect("error lock should not be poisoned") = Some(format!("{apply_error:#}")), - } - } - }); - let subscription = self.loro.subscribe_root(callback); - let import_result = self.loro.import_with(update, "remote"); - drop(subscription); - self.binding = Arc::try_unwrap(binding) - .map_err(|_| anyhow!("binding callback outlived import"))? - .into_inner() - .map_err(|error| anyhow!("binding lock poisoned: {error}"))?; - import_result.context("importing remote update failed")?; - let remote_error = error - .lock() - .expect("error lock should not be poisoned") - .take(); - if let Some(error) = remote_error { - bail!("remote apply failed: {error}"); - } - let patches = Arc::try_unwrap(patches) - .map_err(|_| anyhow!("patch callback outlived import"))? - .into_inner() - .map_err(|error| anyhow!("patch lock poisoned: {error}"))?; - apply_patches(&mut self.document, &mut self.binding, &self.loro, &patches)?; - Ok(()) - } - } - - fn paragraph_block(text: &str) -> InputBlock { - InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain(text)], - }) - } - - fn equation_block(source: &str) -> InputBlock { - InputBlock::Equation(InputEquationBlock { - source: source.to_string(), - syntax: InputEquationSyntax::Latex, - display: InputEquationDisplay::Display, - }) - } - - fn object_rows(document: &Document) -> Vec { - document - .blocks - .iter() - .enumerate() - .filter_map(|(ix, block)| (!matches!(block, Block::Paragraph(_))).then_some(ix)) - .collect() - } - - fn adjacent_paragraph_pairs(document: &Document) -> Vec { - let mut pairs = Vec::new(); - let mut paragraph_ix = 0; - let mut previous_paragraph_ix = None; - for block in document.blocks.iter() { - if matches!(block, Block::Paragraph(_)) { - if let Some(previous) = previous_paragraph_ix { - pairs.push(previous); - } - previous_paragraph_ix = Some(paragraph_ix); - paragraph_ix += 1; - } else { - previous_paragraph_ix = None; - } - } - pairs - } - - fn insert_text_preserving_blocks(document: &mut Document, paragraph_ix: usize, byte: usize, text: &str, styles: RunStyles) { - let Some(snapshot) = block_order_snapshot(document, paragraph_ix) else { - return; - }; - insert_text_at(document, paragraph_ix, byte, text, styles); - restore_block_order(document, paragraph_ix, snapshot); - } - - fn delete_text_preserving_blocks(document: &mut Document, paragraph_ix: usize, range: Range) { - let Some(snapshot) = block_order_snapshot(document, paragraph_ix) else { - return; - }; - delete_range_in_paragraph(document, paragraph_ix, range); - restore_block_order(document, paragraph_ix, snapshot); - } - - fn split_paragraph_preserving_blocks(document: &mut Document, paragraph_ix: usize, byte: usize) -> Option { - let block_ix = block_ix_for_paragraph_ix(document, paragraph_ix)?; - let old_blocks = document.blocks.as_ref().clone(); - let old_block_ids = document.ids.block_ids.clone(); - - split_paragraph_at(document, paragraph_ix, byte); - - let new_paragraph_id = *document.ids.paragraph_ids.get(paragraph_ix + 1)?; - let new_block_id = document - .ids - .block_ids - .iter() - .copied() - .find(|id| !old_block_ids.contains(id))?; - let left = document.paragraphs.get(paragraph_ix).cloned()?; - let right = document.paragraphs.get(paragraph_ix + 1).cloned()?; - - let mut blocks = old_blocks; - *blocks.get_mut(block_ix)? = Block::Paragraph(left); - blocks.insert(block_ix + 1, Block::Paragraph(right)); - document.blocks = Arc::new(blocks); - - let mut block_ids = old_block_ids; - block_ids.insert(block_ix + 1, new_block_id); - document.ids.block_ids = block_ids; - gpui_flowtext::rebuild_document_sections(document); - - Some(new_paragraph_id) - } - - fn block_order_snapshot(document: &Document, paragraph_ix: usize) -> Option<(usize, Vec, Vec)> { - Some(( - block_ix_for_paragraph_ix(document, paragraph_ix)?, - document.blocks.as_ref().clone(), - document.ids.block_ids.clone(), - )) - } - - fn restore_block_order( - document: &mut Document, - paragraph_ix: usize, - (block_ix, mut blocks, block_ids): (usize, Vec, Vec), - ) { - if let Some(paragraph) = document.paragraphs.get(paragraph_ix).cloned() - && let Some(block) = blocks.get_mut(block_ix) - { - *block = Block::Paragraph(paragraph); - } - document.blocks = Arc::new(blocks); - document.ids.block_ids = block_ids; - gpui_flowtext::rebuild_document_sections(document); - } - - fn block_ix_for_paragraph_ix(document: &Document, target_paragraph_ix: usize) -> Option { - let mut paragraph_ix = 0; - for (block_ix, block) in document.blocks.iter().enumerate() { - if matches!(block, Block::Paragraph(_)) { - if paragraph_ix == target_paragraph_ix { - return Some(block_ix); - } - paragraph_ix += 1; - } - } - None - } - - fn text_choice(ix: u8) -> &'static str { - match ix % 6 { - 0 => "a", - 1 => "é", - 2 => "🌍", - 3 => "\u{2028}", - 4 => "x", - _ => MULTIBYTE, - } - } - - fn equation_source(ix: u8) -> &'static str { - match ix % 4 { - 0 => "x = aé", - 1 => "y = 🌍^2", - 2 => "z = a + \u{2028}x", - _ => MULTIBYTE, - } - } - - fn style_choice(ix: u8) -> RunStyles { - match ix % 5 { - 0 => RunStyles::default(), - 1 => RunStyles::default().with(RunStyle::Semantic(1)), - 2 => RunStyles::default().with(RunStyle::Highlight(2)), - 3 => RunStyles::default().with_direct_underline(), - _ => RunStyles::default() - .with(RunStyle::Semantic(3)) - .with_strikethrough(), - } - } - - fn byte_boundary(text: &str, pick: u8) -> usize { - let boundaries = byte_boundaries(text); - boundaries[usize::from(pick) % boundaries.len()] - } - - fn non_empty_range(text: &str, start_pick: u8, len_pick: u8) -> Option> { - let boundaries = byte_boundaries(text); - if boundaries.len() < 2 { - return None; - } - let start_ix = usize::from(start_pick) % (boundaries.len() - 1); - let remaining = boundaries.len() - start_ix - 1; - let end_ix = start_ix + 1 + usize::from(len_pick) % remaining; - Some(boundaries[start_ix]..boundaries[end_ix]) - } - - fn byte_boundaries(text: &str) -> Vec { - text - .char_indices() - .map(|(byte, _)| byte) - .chain(std::iter::once(text.len())) - .collect() - } - - fn pseudo(seed: u64, step: usize, source: usize, target: usize, copy: usize) -> u64 { - let mut value = - seed ^ ((step as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15)) ^ ((source as u64) << 17) ^ ((target as u64) << 33) ^ ((copy as u64) << 49); - value ^= value >> 30; - value = value.wrapping_mul(0xBF58_476D_1CE4_E5B9); - value ^= value >> 27; - value.wrapping_mul(0x94D0_49BB_1331_11EB) ^ (value >> 31) - } - - fn canonical_document_bytes(document: &Document) -> Result> { - let mut bytes = Vec::new(); - let mut paragraph_ix = 0; - for block in document.blocks.iter() { - match block { - Block::Paragraph(paragraph) => { - bytes.push(b'p'); - push_paragraph_style(&mut bytes, paragraph.style); - for run in ¶graph.runs { - push_usize(&mut bytes, run.len); - push_run_styles(&mut bytes, run.styles); - } - let text = paragraph_text(document, paragraph_ix); - push_usize(&mut bytes, text.len()); - bytes.extend_from_slice(text.as_bytes()); - paragraph_ix += 1; - }, - Block::Image(_) | Block::Equation(_) | Block::Table(_) => { - bytes.push(match block { - Block::Image(_) => b'i', - Block::Equation(_) => b'e', - Block::Table(_) => b't', - Block::Paragraph(_) => unreachable!(), - }); - let payload = schema::payload_from_block(block, &document.assets).context("object block should have a payload")?; - let payload = postcard::to_stdvec(&payload)?; - push_usize(&mut bytes, payload.len()); - bytes.extend_from_slice(&payload); - }, - } - } - Ok(bytes) - } - - fn describe_document(document: &Document) -> String { - let mut output = String::new(); - let mut paragraph_ix = 0; - for (block_ix, block) in document.blocks.iter().enumerate() { - match block { - Block::Paragraph(paragraph) => { - let text = paragraph_text(document, paragraph_ix); - let _ = write!(output, "#{block_ix}:p:{text:?}:"); - for run in ¶graph.runs { - let _ = write!(output, "{}@{:?};", run.len, run.styles); - } - paragraph_ix += 1; - }, - Block::Image(_) => { - let _ = write!(output, "#{block_ix}:image;"); - }, - Block::Equation(_) => { - let _ = write!(output, "#{block_ix}:equation;"); - }, - Block::Table(_) => { - let _ = write!(output, "#{block_ix}:table;"); - }, - } - } - output - } - - fn push_paragraph_style(bytes: &mut Vec, style: ParagraphStyle) { - match style { - ParagraphStyle::Normal => bytes.extend_from_slice(&[0, 0]), - ParagraphStyle::Custom(slot) => bytes.extend_from_slice(&[1, slot]), - } - } - - fn push_run_styles(bytes: &mut Vec, styles: RunStyles) { - match styles.semantic { - RunSemanticStyle::Plain => bytes.extend_from_slice(&[0, 0]), - RunSemanticStyle::Custom(slot) => bytes.extend_from_slice(&[1, slot]), - } - bytes.push(u8::from(styles.direct_underline)); - bytes.push(u8::from(styles.strikethrough)); - match styles.highlight { - Some(HighlightStyle::Custom(slot)) => bytes.extend_from_slice(&[1, slot]), - None => bytes.extend_from_slice(&[0, 0]), - } - } - - fn push_usize(bytes: &mut Vec, value: usize) { - bytes.extend_from_slice(&(value as u64).to_le_bytes()); - } -} diff --git a/crates/flowstate-collab/tests/local_apply.rs b/crates/flowstate-collab/tests/local_apply.rs deleted file mode 100644 index ad066346..00000000 --- a/crates/flowstate-collab/tests/local_apply.rs +++ /dev/null @@ -1,53 +0,0 @@ -#[cfg(test)] -mod tests { - use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; - use gpui_flowtext::{ - CanonicalOperation, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyle, RunStyles, insert_text_at, - paragraph_text, plain, - }; - - #[test] - fn local_apply_insert_text_matches_flowtext_edit_op() { - let mut document = gpui_flowtext::document_from_input_blocks( - DocumentTheme::default(), - vec![InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain("aé🌍\u{2028}x")], - })], - ); - let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, SessionId::from_bytes([9; 32]), "local-apply", &document).expect("populate should succeed"); - let mut binding = DocBinding::build(&loro, &document).expect("binding should build"); - let paragraph = document.ids.paragraph_ids[0]; - let styles = RunStyles::default() - .with(RunStyle::Semantic(4)) - .with_strikethrough(); - - insert_text_at(&mut document, 0, "a".len(), "Z", styles); - LocalApplier { - doc: &loro, - binding: &mut binding, - } - .apply( - &document, - &[CanonicalOperation::InsertText { - paragraph, - byte: "a".len(), - text: "Z".to_string(), - styles, - }], - ) - .expect("local apply should succeed"); - - let projected = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); - assert_document_text_and_runs_eq(&document, &projected); - } - - fn assert_document_text_and_runs_eq(left: &Document, right: &Document) { - assert_eq!(left.paragraphs.len(), right.paragraphs.len()); - for ix in 0..left.paragraphs.len() { - assert_eq!(paragraph_text(left, ix), paragraph_text(right, ix)); - assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); - } - } -} diff --git a/crates/flowstate-collab/tests/perf_smoke.rs b/crates/flowstate-collab/tests/perf_smoke.rs deleted file mode 100644 index 50ec01f0..00000000 --- a/crates/flowstate-collab/tests/perf_smoke.rs +++ /dev/null @@ -1,193 +0,0 @@ -//! Logic-level performance guards: a single-character edit on a large document -//! must produce a Loro update and a remote reconcile proportional to the edit, -//! not to the whole document. These assert structure/size, never wall-clock, so -//! they stay deterministic in CI and guard against regressing the incremental -//! local apply (T2) and delta-driven remote reconcile (T4). - -#[cfg(test)] -mod tests { - use std::sync::{Arc, Mutex}; - - use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, patch_apply::apply_patches, projection, schema}; - use gpui_flowtext::{ - CanonicalOperation, CollabPatch, Document, DocumentTheme, InputBlock, InputParagraph, ParagraphStyle, RunStyles, capture_document_span, - document_from_input_blocks, insert_text_at, paragraph_text, plain, - }; - use loro::{ExportMode, LoroDoc, event::Subscriber}; - - const PARAGRAPHS: usize = 400; - - #[test] - fn single_char_local_edit_emits_a_delta_proportional_to_the_edit() { - let session = SessionId::from_bytes([42; 32]); - let document = large_document(); - let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, session, "perf-smoke", &document).expect("populate"); - let mut binding = DocBinding::build(&loro, &document).expect("binding"); - - let updates = Arc::new(Mutex::new(Vec::new())); - let captured = updates.clone(); - let _subscription = loro.subscribe_local_update(Box::new(move |bytes| { - captured.lock().expect("update lock").push(bytes.to_vec()); - true - })); - - // Edit a paragraph in the middle of the document with a single keystroke, - // emitted exactly as the editor would: a span replacement of that paragraph. - let target = PARAGRAPHS / 2; - let mut edited = document.clone(); - let before = capture_document_span(&edited, target..target + 1); - insert_text_at(&mut edited, target, 0, "X", RunStyles::default()); - let after = capture_document_span(&edited, target..target + 1); - let op = CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: Some(edited.ids.paragraph_ids[target]), - before, - after, - }; - - LocalApplier { - doc: &loro, - binding: &mut binding, - } - .apply(&edited, &[op]) - .expect("local apply"); - - let emitted: usize = updates - .lock() - .expect("update lock") - .iter() - .map(Vec::len) - .sum(); - let body_bytes = full_body_len(&document); - // The whole-document body is large; a single-character incremental splice must - // stay far below it (a full-body rewrite would be >= body_bytes). - assert!(body_bytes > 4000, "fixture should be large, got {body_bytes} body bytes"); - assert!( - emitted * 4 < body_bytes, - "single-char edit emitted {emitted} update bytes for a {body_bytes}-byte body; expected an edit-proportional delta", - ); - - // The edit must still converge: a fresh peer importing the update matches. - let peer = schema::new_configured_doc(); - let snapshot = loro.export(ExportMode::Snapshot).expect("snapshot"); - let _ = peer.import(&snapshot); - let projected = projection::document_from_loro(&loro, DocumentTheme::default()).expect("project"); - assert_eq!(paragraph_text(&projected, target), format!("X{}", paragraph_text(&document, target))); - } - - #[test] - fn remote_single_char_insert_reconciles_one_paragraph() { - let session = SessionId::from_bytes([43; 32]); - let document = large_document(); - - let source_loro = schema::new_configured_doc(); - projection::populate_from_document(&source_loro, session, "perf-smoke", &document).expect("populate"); - let mut source_binding = DocBinding::build(&source_loro, &document).expect("binding"); - let updates = Arc::new(Mutex::new(Vec::new())); - let captured = updates.clone(); - let _subscription = source_loro.subscribe_local_update(Box::new(move |bytes| { - captured.lock().expect("update lock").push(bytes.to_vec()); - true - })); - let snapshot = source_loro.export(ExportMode::Snapshot).expect("snapshot"); - - let target_loro = schema::new_configured_doc(); - target_loro.import(&snapshot).expect("import snapshot"); - let mut target_document = projection::document_from_loro(&target_loro, DocumentTheme::default()).expect("project"); - let mut target_binding = DocBinding::build(&target_loro, &target_document).expect("binding"); - - // Source makes a single-character insert in the middle paragraph. - let target = PARAGRAPHS / 2; - let mut edited = document.clone(); - let before = capture_document_span(&edited, target..target + 1); - insert_text_at(&mut edited, target, 0, "X", RunStyles::default()); - let after = capture_document_span(&edited, target..target + 1); - LocalApplier { - doc: &source_loro, - binding: &mut source_binding, - } - .apply( - &edited, - &[CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: Some(edited.ids.paragraph_ids[target]), - before, - after, - }], - ) - .expect("local apply"); - let update_batches = std::mem::take(&mut *updates.lock().expect("update lock")); - - let patches = import_updates(&target_loro, &mut target_document, &mut target_binding, &update_batches); - - // The remote reconcile must touch exactly the one changed paragraph row, not - // re-emit a patch per paragraph in the document. - assert_eq!(patches.len(), 1, "expected one reconcile patch, got {}: {patches:?}", patches.len()); - assert!( - matches!(patches.as_slice(), [CollabPatch::ParagraphText { row, .. }] if *row == target), - "expected a single ParagraphText patch for row {target}, got {patches:?}", - ); - - // And the imported document converges to the source projection. - let source_projection = projection::document_from_loro(&source_loro, DocumentTheme::default()).expect("project"); - assert_eq!(paragraph_text(&target_document, target), paragraph_text(&source_projection, target)); - assert_eq!(target_document.paragraphs.len(), source_projection.paragraphs.len()); - } - - fn large_document() -> Document { - let blocks = (0..PARAGRAPHS) - .map(|ix| { - InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain(&format!("paragraph number {ix} body text"))], - }) - }) - .collect(); - document_from_input_blocks(DocumentTheme::default(), blocks) - } - - fn full_body_len(document: &Document) -> usize { - (0..document.paragraphs.len()) - .map(|ix| paragraph_text(document, ix).len() + 1) - .sum() - } - - fn import_updates(loro: &LoroDoc, document: &mut Document, binding: &mut DocBinding, updates: &[Vec]) -> Vec { - let mut all = Vec::new(); - for update in updates { - let snapshot_document = Arc::new(document.clone()); - let shared_binding = Arc::new(Mutex::new(std::mem::take(binding))); - let patches = Arc::new(Mutex::new(Vec::new())); - let doc = loro.clone(); - let callback: Subscriber = Arc::new({ - let shared_binding = shared_binding.clone(); - let patches = patches.clone(); - move |event| { - let produced = { - let mut guard = shared_binding.lock().expect("binding lock"); - flowstate_collab::remote_apply::RemoteApplier { - doc: &doc, - binding: &mut guard, - } - .apply_event(&snapshot_document, &event) - .expect("remote apply") - }; - patches.lock().expect("patch lock").extend(produced); - } - }); - let subscription = loro.subscribe_root(callback); - loro.import_with(update, "remote").expect("import"); - drop(subscription); - *binding = Arc::try_unwrap(shared_binding) - .expect("binding owner") - .into_inner() - .expect("binding lock"); - let produced = Arc::try_unwrap(patches) - .expect("patch owner") - .into_inner() - .expect("patch lock"); - apply_patches(document, binding, loro, &produced).expect("apply patches"); - all.extend(produced); - } - all - } -} diff --git a/crates/flowstate-collab/tests/projection.rs b/crates/flowstate-collab/tests/projection.rs deleted file mode 100644 index 96c192e3..00000000 --- a/crates/flowstate-collab/tests/projection.rs +++ /dev/null @@ -1,122 +0,0 @@ -#[cfg(test)] -mod tests { - use flowstate_collab::binding::DocBinding; - use flowstate_collab::{SessionId, projection, schema}; - use gpui_flowtext::{ - Block, Document, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, InputParagraph, InputTableBlock, - InputTableCell, InputTableCellBlock, InputTableColumnWidth, InputTableRow, InputTableStyle, ParagraphStyle, RunStyle, RunStyles, - document_from_input_blocks, paragraph_text, plain, run, - }; - - const MULTIBYTE: &str = "aé🌍\u{2028}x"; - - #[test] - fn projection_round_trips_text_marks_and_equation_blocks() { - let input_blocks = vec![ - InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Custom(2), - runs: vec![ - plain("aé"), - run( - "🌍\u{2028}x", - RunStyles::default() - .with(RunStyle::Semantic(3)) - .with_direct_underline(), - ), - ], - }), - InputBlock::Equation(InputEquationBlock { - source: "E = mc^2".to_string(), - syntax: InputEquationSyntax::Latex, - display: InputEquationDisplay::Display, - }), - InputBlock::Table(InputTableBlock { - rows: vec![InputTableRow { - cells: vec![InputTableCell { - blocks: vec![InputTableCellBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain(MULTIBYTE)], - })], - row_span: 1, - col_span: 1, - }], - }], - column_widths: vec![InputTableColumnWidth::Auto], - style: InputTableStyle { header_row: true }, - }), - ]; - let original = document_from_input_blocks(DocumentTheme::default(), input_blocks); - let loro = schema::new_configured_doc(); - let session = SessionId::from_bytes([7; 32]); - - projection::populate_from_document(&loro, session, "projection-test", &original).expect("populate should succeed"); - projection::verify_lineage(&loro, session).expect("lineage should match"); - let projected = projection::document_from_loro(&loro, DocumentTheme::default()).expect("projection should succeed"); - let binding = DocBinding::build(&loro, &projected).expect("binding should match projected document"); - - assert_document_shape_eq(&original, &projected); - assert_eq!(binding.rows.len(), projected.blocks.len()); - } - - #[test] - fn offset_helpers_round_trip_all_multibyte_boundaries() { - let original = document_from_input_blocks( - DocumentTheme::default(), - vec![InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain(MULTIBYTE)], - })], - ); - let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, SessionId::from_bytes([8; 32]), "offsets", &original).expect("populate should succeed"); - let _binding = DocBinding::build(&loro, &original).expect("binding should build"); - let text = schema::body_text(&loro); - let byte_to_unicode = [ - (0, 0), - ("a".len(), 1), - ("aé".len(), 2), - ("aé🌍".len(), 3), - ("aé🌍\u{2028}".len(), 4), - (MULTIBYTE.len(), 5), - ]; - - assert_eq!(text.len_utf8(), MULTIBYTE.len()); - for (byte, unicode) in byte_to_unicode { - assert_eq!(schema::loro_pos(&text, byte), unicode); - assert_eq!(schema::utf8_byte(&text, unicode), byte); - } - } - - fn assert_document_shape_eq(left: &Document, right: &Document) { - assert_eq!(left.blocks.len(), right.blocks.len()); - assert_eq!(left.paragraphs.len(), right.paragraphs.len()); - for ix in 0..left.paragraphs.len() { - assert_eq!(paragraph_text(left, ix), paragraph_text(right, ix)); - assert_eq!(left.paragraphs[ix].style, right.paragraphs[ix].style); - assert_eq!(left.paragraphs[ix].runs, right.paragraphs[ix].runs); - } - for (left, right) in left.blocks.iter().zip(right.blocks.iter()) { - match (left, right) { - (Block::Paragraph(_), Block::Paragraph(_)) => {}, - (Block::Equation(left), Block::Equation(right)) => { - assert_eq!(left.source, right.source); - assert_eq!(left.syntax, right.syntax); - assert_eq!(left.display, right.display); - }, - (Block::Table(left), Block::Table(right)) => { - assert_eq!(left.rows, right.rows); - assert_eq!(left.column_widths, right.column_widths); - assert_eq!(left.style, right.style); - }, - (Block::Image(left), Block::Image(right)) => { - assert_eq!(left.asset_id, right.asset_id); - assert_eq!(left.alt_text, right.alt_text); - assert_eq!(left.caption, right.caption); - assert_eq!(left.sizing, right.sizing); - assert_eq!(left.alignment, right.alignment); - }, - _ => panic!("projected block kind changed"), - } - } - } -} diff --git a/crates/flowstate-collab/tests/remote_apply.rs b/crates/flowstate-collab/tests/remote_apply.rs deleted file mode 100644 index 9d8aec2c..00000000 --- a/crates/flowstate-collab/tests/remote_apply.rs +++ /dev/null @@ -1,344 +0,0 @@ -#[cfg(test)] -mod tests { - use std::sync::{Arc, Mutex}; - - use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, patch_apply::apply_patches, projection, schema}; - use gpui_flowtext::{ - Block, CanonicalOperation, CollabPatch, Document, DocumentOffset, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, - InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, RunStyles, block_from_input_block, document_from_input_blocks, - insert_block_id, insert_text_at, mutate_runs_in_range, paragraph_text, paragraphs_mut, plain, remove_block_ids, update_paragraph_block, - }; - use loro::{ExportMode, LoroDoc, Subscription, event::Subscriber}; - - const MULTIBYTE: &str = "aé🌍\u{2028}x"; - - #[test] - fn remote_text_insert_produces_paragraph_text_patch() { - let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE)]); - let paragraph = source.document.ids.paragraph_ids[0]; - let styles = RunStyles::default().with(RunStyle::Semantic(3)); - - insert_text_at(&mut source.document, 0, "a".len(), "Zé", styles); - let updates = source.apply(&[CanonicalOperation::InsertText { - paragraph, - byte: "a".len(), - text: "Zé".to_string(), - styles, - }]); - let patches = target.import_updates(&updates); - - assert!(matches!(patches.as_slice(), [CollabPatch::ParagraphText { row: 0, .. }])); - assert_documents_match(&source.document, &target.document, &target.loro); - } - - #[test] - fn remote_run_and_paragraph_styles_produce_style_patches() { - let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE)]); - let paragraph = source.document.ids.paragraph_ids[0]; - - paragraphs_mut(&mut source.document)[0].style = ParagraphStyle::Custom(7); - update_paragraph_block(&mut source.document, 0); - let paragraph_style_updates = source.apply(&[CanonicalOperation::SetParagraphStyle { - paragraph, - style: ParagraphStyle::Custom(7), - }]); - let paragraph_style_patches = target.import_updates(¶graph_style_updates); - assert!(matches!( - paragraph_style_patches.as_slice(), - [CollabPatch::ParagraphStyle { - row: 0, - style: ParagraphStyle::Custom(7) - }] - )); - - let range = "a".len().."aé🌍".len(); - let styles = RunStyles::default() - .with(RunStyle::Highlight(2)) - .with_direct_underline(); - mutate_runs_in_range( - &mut source.document, - DocumentOffset { - paragraph: 0, - byte: range.start, - }..DocumentOffset { - paragraph: 0, - byte: range.end, - }, - |run_styles| *run_styles = styles, - ); - let run_style_updates = source.apply(&[CanonicalOperation::SetRunStyles { paragraph, range, styles }]); - let run_style_patches = target.import_updates(&run_style_updates); - assert!(matches!(run_style_patches.as_slice(), [CollabPatch::ParagraphRuns { row: 0, .. }])); - assert_documents_match(&source.document, &target.document, &target.loro); - } - - #[test] - fn remote_object_data_rev_replacement_emits_one_patch() { - let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE), equation_block("x = 1")]); - replace_equation(&mut source.document, 1, "x = aé + 🌍", 1); - - let updates = source.apply(&[CanonicalOperation::ReplaceBlock { - block: Some(source.document.ids.block_ids[1]), - }]); - let patches = target.import_updates(&updates); - - assert!(matches!(patches.as_slice(), [CollabPatch::ReplaceObjectBlock { row: 1, .. }])); - assert_documents_match(&source.document, &target.document, &target.loro); - } - - #[test] - fn remote_list_insert_delete_and_move_emit_structural_patches() { - let (mut source, mut target) = pair(vec![paragraph_block(MULTIBYTE), paragraph_block("tail")]); - - insert_object_block(&mut source.document, 1, equation_block("insert aé")); - let equation_id = source.document.ids.block_ids[1]; - let insert_updates = source.apply(&[CanonicalOperation::InsertBlock { - block: equation_id, - block_ix: 1, - }]); - let insert_patches = target.import_updates(&insert_updates); - assert!(matches!(insert_patches.as_slice(), [CollabPatch::InsertBlocks { row: 1, blocks } ] if blocks.len() == 1)); - assert_documents_match(&source.document, &target.document, &target.loro); - - move_block(&mut source.document, 1, 2); - let move_updates = source.apply(&[CanonicalOperation::MoveBlock { - block: equation_id, - new_block_ix: 2, - }]); - let move_patches = target.import_updates(&move_updates); - assert!( - move_patches - .iter() - .any(|patch| matches!(patch, CollabPatch::MoveBlock { from: 1, to: 2 })) - ); - assert_documents_match(&source.document, &target.document, &target.loro); - - delete_object_block(&mut source.document, 2); - let delete_updates = source.apply(&[CanonicalOperation::DeleteBlock { block: equation_id }]); - let delete_patches = target.import_updates(&delete_updates); - assert!(matches!(delete_patches.as_slice(), [CollabPatch::DeleteBlocks { row: 2, count: 1 }])); - assert_documents_match(&source.document, &target.document, &target.loro); - } - - struct SourcePeer { - document: Document, - loro: LoroDoc, - binding: DocBinding, - updates: Arc>>>, - _subscription: Subscription, - } - - struct TargetPeer { - document: Document, - loro: LoroDoc, - binding: DocBinding, - } - - impl SourcePeer { - fn apply(&mut self, ops: &[CanonicalOperation]) -> Vec> { - LocalApplier { - doc: &self.loro, - binding: &mut self.binding, - } - .apply(&self.document, ops) - .expect("local apply should succeed"); - let mut updates = self - .updates - .lock() - .expect("update lock should not be poisoned"); - std::mem::take(&mut *updates) - } - } - - impl TargetPeer { - fn import_updates(&mut self, updates: &[Vec]) -> Vec { - let mut all_patches = Vec::new(); - for update in updates { - all_patches.extend(self.import_update(update)); - } - all_patches - } - - fn import_update(&mut self, update: &[u8]) -> Vec { - let document = Arc::new(self.document.clone()); - let binding = Arc::new(Mutex::new(std::mem::take(&mut self.binding))); - let patches = Arc::new(Mutex::new(Vec::new())); - let error = Arc::new(Mutex::new(None::)); - let doc = self.loro.clone(); - let callback: Subscriber = Arc::new({ - let binding = binding.clone(); - let patches = patches.clone(); - let error = error.clone(); - move |event| { - let result = binding - .lock() - .map_err(|lock_error| anyhow::anyhow!("binding lock poisoned: {lock_error}")) - .and_then(|mut binding| { - flowstate_collab::remote_apply::RemoteApplier { - doc: &doc, - binding: &mut binding, - } - .apply_event(&document, &event) - }); - match result { - Ok(mut produced) => patches - .lock() - .expect("patch lock should not be poisoned") - .append(&mut produced), - Err(apply_error) => *error.lock().expect("error lock should not be poisoned") = Some(format!("{apply_error:#}")), - } - } - }); - let subscription = self.loro.subscribe_root(callback); - let import_result = self.loro.import_with(update, "remote"); - drop(subscription); - self.binding = Arc::try_unwrap(binding) - .expect("binding callback should be dropped") - .into_inner() - .expect("binding lock should not be poisoned"); - import_result.expect("remote import should succeed"); - let remote_error = error - .lock() - .expect("error lock should not be poisoned") - .take(); - if let Some(error) = remote_error { - panic!("remote apply failed: {error}"); - } - let patches = Arc::try_unwrap(patches) - .expect("patch callback should be dropped") - .into_inner() - .expect("patch lock should not be poisoned"); - apply_patches(&mut self.document, &mut self.binding, &self.loro, &patches).expect("patch apply should succeed"); - patches - } - } - - fn pair(blocks: Vec) -> (SourcePeer, TargetPeer) { - let session = SessionId::from_bytes([19; 32]); - let document = document_from_input_blocks(DocumentTheme::default(), blocks); - let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, session, "remote-apply", &document).expect("populate should succeed"); - let binding = DocBinding::build(&loro, &document).expect("binding should build"); - let updates = Arc::new(Mutex::new(Vec::new())); - let captured = updates.clone(); - let subscription = loro.subscribe_local_update(Box::new(move |bytes| { - captured - .lock() - .expect("update lock should not be poisoned") - .push(bytes.clone()); - true - })); - let snapshot = loro - .export(ExportMode::Snapshot) - .expect("snapshot export should succeed"); - - let target_loro = schema::new_configured_doc(); - target_loro - .import(&snapshot) - .expect("snapshot import should succeed"); - projection::verify_lineage(&target_loro, session).expect("lineage should match"); - let target_document = projection::document_from_loro(&target_loro, DocumentTheme::default()).expect("projection should succeed"); - let target_binding = DocBinding::build(&target_loro, &target_document).expect("binding should build"); - - ( - SourcePeer { - document, - loro, - binding, - updates, - _subscription: subscription, - }, - TargetPeer { - document: target_document, - loro: target_loro, - binding: target_binding, - }, - ) - } - - fn paragraph_block(text: &str) -> InputBlock { - InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs: vec![plain(text)], - }) - } - - fn equation_block(source: &str) -> InputBlock { - InputBlock::Equation(InputEquationBlock { - source: source.to_string(), - syntax: InputEquationSyntax::Latex, - display: InputEquationDisplay::Display, - }) - } - - fn insert_object_block(document: &mut Document, row: usize, block: InputBlock) { - Arc::make_mut(&mut document.blocks).insert(row, block_from_input_block(&block)); - insert_block_id(document, row); - } - - fn replace_equation(document: &mut Document, row: usize, source: &str, version: u64) { - let mut block = block_from_input_block(&equation_block(source)); - if let Block::Equation(equation) = &mut block { - equation.version = version; - } - Arc::make_mut(&mut document.blocks)[row] = block; - } - - fn move_block(document: &mut Document, from: usize, to: usize) { - let block = Arc::make_mut(&mut document.blocks).remove(from); - let insert_ix = to.min(document.blocks.len()); - Arc::make_mut(&mut document.blocks).insert(insert_ix, block); - let block_id = document.ids.block_ids.remove(from); - document - .ids - .block_ids - .insert(to.min(document.ids.block_ids.len()), block_id); - } - - fn delete_object_block(document: &mut Document, row: usize) { - Arc::make_mut(&mut document.blocks).remove(row); - remove_block_ids(document, row..row + 1); - } - - fn assert_documents_match(expected: &Document, actual: &Document, loro: &LoroDoc) { - let projected = projection::document_from_loro(loro, DocumentTheme::default()).expect("projection should succeed"); - assert_document_shape_eq(expected, actual); - assert_document_shape_eq(expected, &projected); - } - - fn assert_document_shape_eq(left: &Document, right: &Document) { - assert_eq!(left.blocks.len(), right.blocks.len()); - assert_eq!(left.paragraphs.len(), right.paragraphs.len()); - let mut paragraph_ix = 0; - for (left_block, right_block) in left.blocks.iter().zip(right.blocks.iter()) { - match (left_block, right_block) { - (Block::Paragraph(left_paragraph), Block::Paragraph(right_paragraph)) => { - assert_eq!(paragraph_text(left, paragraph_ix), paragraph_text(right, paragraph_ix)); - assert_eq!(left_paragraph.style, right_paragraph.style); - assert_eq!(left_paragraph.runs, right_paragraph.runs); - }, - (Block::Equation(left), Block::Equation(right)) => { - assert_eq!(left.source, right.source); - assert_eq!(left.syntax, right.syntax); - assert_eq!(left.display, right.display); - }, - (Block::Image(left), Block::Image(right)) => { - assert_eq!(left.asset_id, right.asset_id); - assert_eq!(left.alt_text, right.alt_text); - assert_eq!(left.caption, right.caption); - assert_eq!(left.sizing, right.sizing); - assert_eq!(left.alignment, right.alignment); - }, - (Block::Table(left), Block::Table(right)) => { - assert_eq!(left.rows, right.rows); - assert_eq!(left.column_widths, right.column_widths); - assert_eq!(left.style, right.style); - }, - _ => panic!("document block kind changed"), - } - if matches!(left_block, Block::Paragraph(_)) { - paragraph_ix += 1; - } - } - } -} diff --git a/crates/flowstate-collab/tests/translation.rs b/crates/flowstate-collab/tests/translation.rs deleted file mode 100644 index acd7222f..00000000 --- a/crates/flowstate-collab/tests/translation.rs +++ /dev/null @@ -1,341 +0,0 @@ -#[cfg(test)] -mod tests { - use std::sync::Arc; - - use flowstate_collab::{SessionId, binding::DocBinding, local_apply::LocalApplier, projection, schema}; - use gpui_flowtext::{ - Block, CanonicalOperation, Document, DocumentOffset, DocumentTheme, InputBlock, InputEquationBlock, InputEquationDisplay, - InputEquationSyntax, InputParagraph, ParagraphStyle, RunStyle, RunStyles, block_from_input_block, capture_document_span, - delete_cross_paragraph_range, delete_range_in_paragraph, document_from_input_blocks, insert_block_id, insert_text_at, mutate_runs_in_range, - paragraph_text, paragraphs_mut, plain, remove_block_ids, run, split_paragraph_at, update_paragraph_block, - }; - use loro::LoroDoc; - - const MULTIBYTE: &str = "aé🌍\u{2028}x"; - - #[test] - fn delete_range_uses_utf8_byte_offsets() { - let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain(MULTIBYTE)])]); - let paragraph = document.ids.paragraph_ids[0]; - let start = "a".len(); - let end = "aé🌍".len(); - - delete_range_in_paragraph(&mut document, 0, start..end); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::DeleteRange { - start_paragraph: paragraph, - start_byte: start, - end_paragraph: paragraph, - end_byte: end, - }], - ); - } - - #[test] - fn split_and_join_paragraphs_preserve_multibyte_text_and_styles() { - let semantic = RunStyles::default() - .with(RunStyle::Semantic(2)) - .with_direct_underline(); - let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("aé"), run("🌍\u{2028}x", semantic)])]); - let original = document.ids.paragraph_ids[0]; - let split_byte = "aé".len(); - - split_paragraph_at(&mut document, 0, split_byte); - let split = document.ids.paragraph_ids[1]; - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::SplitParagraph { - paragraph: original, - byte: split_byte, - new_paragraph: split, - }], - ); - - let first_len = paragraph_text(&document, 0).len(); - delete_cross_paragraph_range( - &mut document, - DocumentOffset { - paragraph: 0, - byte: first_len, - }..DocumentOffset { paragraph: 1, byte: 0 }, - ); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::JoinParagraphs { - first: original, - second: split, - }], - ); - } - - #[test] - fn paragraph_and_run_style_updates_match_flowtext_runs() { - let base_style = RunStyles::default() - .with(RunStyle::Semantic(1)) - .with_strikethrough(); - let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![run(MULTIBYTE, base_style)])]); - let paragraph = document.ids.paragraph_ids[0]; - - paragraphs_mut(&mut document)[0].style = ParagraphStyle::Custom(6); - update_paragraph_block(&mut document, 0); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::SetParagraphStyle { - paragraph, - style: ParagraphStyle::Custom(6), - }], - ); - - let unstyled = "a".len().."aé".len(); - mutate_runs_in_range( - &mut document, - DocumentOffset { - paragraph: 0, - byte: unstyled.start, - }..DocumentOffset { - paragraph: 0, - byte: unstyled.end, - }, - |styles| *styles = RunStyles::default(), - ); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::SetRunStyles { - paragraph, - range: unstyled, - styles: RunStyles::default(), - }], - ); - - let highlighted = "aé".len().."aé🌍".len(); - let highlighted_styles = RunStyles::default() - .with(RunStyle::Semantic(4)) - .with(RunStyle::Highlight(3)) - .with_direct_underline() - .with_strikethrough(); - mutate_runs_in_range( - &mut document, - DocumentOffset { - paragraph: 0, - byte: highlighted.start, - }..DocumentOffset { - paragraph: 0, - byte: highlighted.end, - }, - |styles| *styles = highlighted_styles, - ); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::SetRunStyles { - paragraph, - range: highlighted, - styles: highlighted_styles, - }], - ); - } - - #[test] - fn replace_paragraph_span_refreshes_text_runs_and_styles() { - let (mut document, loro, mut binding) = - collab_fixture(vec![paragraph_block(vec![plain(MULTIBYTE)]), paragraph_block(vec![plain("second aé")])]); - let start_paragraph = document.ids.paragraph_ids[0]; - let before = capture_document_span(&document, 0..2); - let inserted_styles = RunStyles::default() - .with(RunStyle::Semantic(5)) - .with_direct_underline(); - - insert_text_at(&mut document, 0, "a".len(), "Zé", inserted_styles); - paragraphs_mut(&mut document)[1].style = ParagraphStyle::Custom(2); - update_paragraph_block(&mut document, 1); - let after = capture_document_span(&document, 0..2); - - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: Some(start_paragraph), - before, - after, - }], - ); - } - - #[test] - fn insert_move_replace_and_delete_object_blocks_match_flowtext_document() { - let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("one aé")]), paragraph_block(vec![plain("two 🌍")])]); - let equation = equation_block("x = aé + 🌍"); - - let equation_id = insert_object_block(&mut document, 1, equation); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::InsertBlock { - block: equation_id, - block_ix: 1, - }], - ); - - move_block(&mut document, 1, 2); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::MoveBlock { - block: equation_id, - new_block_ix: 2, - }], - ); - - replace_object_block(&mut document, 2, equation_block("y = 🌍^2")); - apply_and_assert_projection( - &document, - &loro, - &mut binding, - &[CanonicalOperation::ReplaceBlock { block: Some(equation_id) }], - ); - - delete_object_block(&mut document, 2); - apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::DeleteBlock { block: equation_id }]); - } - - #[test] - fn replace_block_none_updates_single_changed_object_row() { - let (mut document, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain(MULTIBYTE)]), equation_block("x = 1")]); - - replace_object_block_with_version(&mut document, 1, equation_block("x = aé + 🌍"), 1); - apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::ReplaceBlock { block: None }]); - } - - #[test] - fn replace_document_rebuilds_the_loro_projection() { - let (_, loro, mut binding) = collab_fixture(vec![paragraph_block(vec![plain("old")]), equation_block("old = 1")]); - let document = document_from_input_blocks( - DocumentTheme::default(), - vec![ - paragraph_block(vec![plain("new aé"), run("🌍", RunStyles::default().with(RunStyle::Highlight(1)))]), - equation_block("new = aé + 🌍"), - paragraph_block(vec![plain("tail")]), - ], - ); - - apply_and_assert_projection(&document, &loro, &mut binding, &[CanonicalOperation::ReplaceDocument]); - } - - fn collab_fixture(blocks: Vec) -> (Document, LoroDoc, DocBinding) { - let document = document_from_input_blocks(DocumentTheme::default(), blocks); - let loro = schema::new_configured_doc(); - projection::populate_from_document(&loro, SessionId::from_bytes([42; 32]), "translation", &document).expect("populate should succeed"); - let binding = DocBinding::build(&loro, &document).expect("binding should build"); - (document, loro, binding) - } - - fn apply_and_assert_projection(document: &Document, loro: &LoroDoc, binding: &mut DocBinding, ops: &[CanonicalOperation]) { - LocalApplier { doc: loro, binding } - .apply(document, ops) - .expect("local apply should succeed"); - let projected = projection::document_from_loro(loro, DocumentTheme::default()).expect("projection should succeed"); - assert_document_projection_eq(document, &projected); - } - - fn paragraph_block(runs: Vec) -> InputBlock { - InputBlock::Paragraph(InputParagraph { - style: ParagraphStyle::Normal, - runs, - }) - } - - fn equation_block(source: &str) -> InputBlock { - InputBlock::Equation(InputEquationBlock { - source: source.to_string(), - syntax: InputEquationSyntax::Latex, - display: InputEquationDisplay::Display, - }) - } - - fn insert_object_block(document: &mut Document, block_ix: usize, input: InputBlock) -> gpui_flowtext::BlockId { - Arc::make_mut(&mut document.blocks).insert(block_ix, block_from_input_block(&input)); - insert_block_id(document, block_ix) - } - - fn move_block(document: &mut Document, from: usize, to: usize) { - let blocks = Arc::make_mut(&mut document.blocks); - let block = blocks.remove(from); - blocks.insert(to.min(blocks.len()), block); - let block_id = document.ids.block_ids.remove(from); - document - .ids - .block_ids - .insert(to.min(document.ids.block_ids.len()), block_id); - } - - fn replace_object_block(document: &mut Document, block_ix: usize, input: InputBlock) { - Arc::make_mut(&mut document.blocks)[block_ix] = block_from_input_block(&input); - } - - fn replace_object_block_with_version(document: &mut Document, block_ix: usize, input: InputBlock, version: u64) { - let mut block = block_from_input_block(&input); - match &mut block { - Block::Paragraph(_) => {}, - Block::Image(image) => image.version = version, - Block::Equation(equation) => equation.version = version, - Block::Table(table) => table.version = version, - } - Arc::make_mut(&mut document.blocks)[block_ix] = block; - } - - fn delete_object_block(document: &mut Document, block_ix: usize) { - Arc::make_mut(&mut document.blocks).remove(block_ix); - remove_block_ids(document, block_ix..block_ix + 1); - } - - fn assert_document_projection_eq(expected: &Document, actual: &Document) { - assert_eq!(expected.blocks.len(), actual.blocks.len()); - assert_eq!(expected.paragraphs.len(), actual.paragraphs.len()); - let mut paragraph_ix = 0; - for (expected_block, actual_block) in expected.blocks.iter().zip(actual.blocks.iter()) { - match (expected_block, actual_block) { - (Block::Paragraph(expected_paragraph), Block::Paragraph(actual_paragraph)) => { - assert_eq!(paragraph_text(expected, paragraph_ix), paragraph_text(actual, paragraph_ix)); - assert_eq!(expected_paragraph.style, actual_paragraph.style); - assert_eq!(expected_paragraph.runs, actual_paragraph.runs); - }, - (Block::Equation(expected), Block::Equation(actual)) => { - assert_eq!(expected.source, actual.source); - assert_eq!(expected.syntax, actual.syntax); - assert_eq!(expected.display, actual.display); - }, - (Block::Image(expected), Block::Image(actual)) => { - assert_eq!(expected.asset_id, actual.asset_id); - assert_eq!(expected.alt_text, actual.alt_text); - assert_eq!(expected.caption, actual.caption); - assert_eq!(expected.sizing, actual.sizing); - assert_eq!(expected.alignment, actual.alignment); - }, - (Block::Table(expected), Block::Table(actual)) => { - assert_eq!(expected.rows, actual.rows); - assert_eq!(expected.column_widths, actual.column_widths); - assert_eq!(expected.style, actual.style); - }, - _ => panic!("projected block kind changed"), - } - if matches!(expected_block, Block::Paragraph(_)) { - paragraph_ix += 1; - } - } - } -} diff --git a/crates/flowstate-document/Cargo.toml b/crates/flowstate-document/Cargo.toml index 7bc6bee1..26f69720 100644 --- a/crates/flowstate-document/Cargo.toml +++ b/crates/flowstate-document/Cargo.toml @@ -11,6 +11,13 @@ workspace = true gpui-flowtext = { workspace = true } gpui = { version = "0.2.2", default-features = false } rustc-hash = { workspace = true } +blake3 = "1.8.5" +loro.workspace = true +postcard.workspace = true +serde.workspace = true +uuid.workspace = true +tempfile.workspace = true +zstd.workspace = true [features] hotpath = ["gpui-flowtext/hotpath"] diff --git a/crates/flowstate-document/src/lib.rs b/crates/flowstate-document/src/lib.rs index 8fdc1edc..a6878ef8 100644 --- a/crates/flowstate-document/src/lib.rs +++ b/crates/flowstate-document/src/lib.rs @@ -1,6 +1,23 @@ +pub mod package; +pub mod loro_import; +pub mod loro_projection; +pub mod loro_schema; + +pub use loro_import::{document_to_loro, document_to_loro_db8_bytes, write_document_as_loro_db8}; +pub use loro_projection::document_from_loro; pub use gpui_flowtext::*; +pub use loro_schema::{ + BODY_FLOW_ID, BLOCKS_BY_ID, FLOW_ATTRS_KEY, FLOW_ID_KEY, FLOW_KIND_KEY, FLOW_TEXT_KEY, FLOWS_BY_ID, MAIN_BODY_BLOCK_ID, + MARK_DIRECT_UNDERLINE, MARK_HIGHLIGHT_STYLE, MARK_PARAGRAPH_STYLE, MARK_RUN_SEMANTIC_STYLE, MARK_STRIKETHROUGH, META, + OBJECT_REPLACEMENT, PARAGRAPHS_BY_ID, ROOT, ROOT_BODY_FLOW_ID, ROOT_FIRST_PARAGRAPH_ID, SECTIONS_BY_ID, SENTINEL_NEWLINE, + init_loro_document, new_loro_document, +}; +pub use package::{ + AssetChunk, ChunkRef, DocumentPackage, DocumentPackageManifest, LORO_PACKAGE_FORMAT_VERSION, LORO_SCHEMA_VERSION, PackageRevision, + ProjectionCacheChunk, SearchUnitChunk, loro_db8_bytes, read_loro_db8, write_loro_db8, +}; -use std::{io, path::Path}; +use std::{io, path::Path, sync::Arc}; use gpui::{Pixels, black, px, rgb}; use rustc_hash::{FxHashMap, FxHashSet}; @@ -34,19 +51,37 @@ fn border_eighth_points(value: f32) -> Pixels { } pub fn read_db8(path: impl AsRef) -> io::Result { - read_document(path) + document_from_package(DocumentPackage::read(path)?) } pub fn read_db8_bytes(bytes: &[u8]) -> io::Result { - read_document_bytes(bytes) + document_from_package(DocumentPackage::from_bytes(bytes)?) } pub fn write_db8(path: impl AsRef, document: &Document) -> io::Result<()> { - write_document(path, document) + write_document_as_loro_db8(path, document, "Flowstate Document") } pub fn db8_bytes(document: &Document) -> io::Result> { - document_bytes(document) + document_to_loro_db8_bytes(document, "Flowstate Document") +} + +fn document_from_package(package: DocumentPackage) -> io::Result { + let mut document = document_from_loro(&package.load_loro_doc()?)?; + for asset in package.assets { + let bytes = asset.bytes; + document.assets.assets.insert( + AssetId(asset.asset_id), + AssetRecord { + id: AssetId(asset.asset_id), + mime_type: asset.mime_type.into(), + original_name: None, + content_hash: AssetRecord::stable_content_hash(&bytes), + bytes: Arc::new(bytes), + }, + ); + } + Ok(document) } pub fn flowstate_document_theme() -> DocumentTheme { diff --git a/crates/flowstate-document/src/loro_import.rs b/crates/flowstate-document/src/loro_import.rs new file mode 100644 index 00000000..342ad119 --- /dev/null +++ b/crates/flowstate-document/src/loro_import.rs @@ -0,0 +1,382 @@ +use std::{io, path::Path}; + +use gpui_flowtext::{ + Block, BlockAlignment, Document, EquationDisplay, EquationSyntax, HighlightStyle, ImageSizing, Paragraph, ParagraphStyle, RunSemanticStyle, + RunStyles, TableBlock, TableCellBlock, TableColumnWidth, paragraph_text, +}; +use loro::{LoroDoc, LoroMap, LoroResult, LoroText, cursor::Side}; +use uuid::Uuid; + +use crate::{ + AssetChunk, BODY_FLOW_ID, BLOCKS_BY_ID, FLOW_ATTRS_KEY, FLOW_ID_KEY, FLOW_KIND_KEY, FLOW_TEXT_KEY, FLOWS_BY_ID, MARK_DIRECT_UNDERLINE, + MARK_HIGHLIGHT_STYLE, MARK_PARAGRAPH_STYLE, MARK_RUN_SEMANTIC_STYLE, MARK_STRIKETHROUGH, OBJECT_REPLACEMENT, PARAGRAPHS_BY_ID, ROOT, + ROOT_BODY_FLOW_ID, SENTINEL_NEWLINE, + loro_schema::{ASSETS_BY_ID, REVISIONS}, +}; + +pub fn document_to_loro(document: &Document, title: &str) -> io::Result { + let doc = crate::new_loro_document(title).map_err(loro_io_error)?; + replace_body_from_document(&doc, document).map_err(loro_io_error)?; + import_assets(&doc, document).map_err(loro_io_error)?; + doc.commit(); + Ok(doc) +} + +pub fn document_to_loro_db8_bytes(document: &Document, title: &str) -> io::Result> { + let doc = document_to_loro(document, title)?; + crate::DocumentPackage::from_loro_snapshot_with_assets(&doc, title, assets_from_document(document))?.to_bytes() +} + +pub fn write_document_as_loro_db8(path: impl AsRef, document: &Document, title: &str) -> io::Result<()> { + let doc = document_to_loro(document, title)?; + crate::DocumentPackage::from_loro_snapshot_with_assets(&doc, title, assets_from_document(document))?.write(path) +} + +fn replace_body_from_document(doc: &LoroDoc, document: &Document) -> LoroResult<()> { + let root = doc.get_map(ROOT); + let flows = root.ensure_mergeable_map(FLOWS_BY_ID)?; + let blocks = root.ensure_mergeable_map(BLOCKS_BY_ID)?; + let paragraphs = root.ensure_mergeable_map(PARAGRAPHS_BY_ID)?; + root.ensure_mergeable_list(REVISIONS)?; + + let body_flow = ensure_flow(&flows, ROOT_BODY_FLOW_ID, "body")?; + let body_text = body_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + replace_text(&body_text, SENTINEL_NEWLINE)?; + clear_map(&blocks)?; + clear_map(¶graphs)?; + + let mut paragraph_ix = 0_usize; + for (block_ix, block) in document.blocks.iter().enumerate() { + match block { + Block::Paragraph(paragraph) => { + let text = paragraph_text(document, paragraph_ix); + append_paragraph(&body_text, ¶graphs, &blocks, BODY_FLOW_ID, paragraph, &text, block_ix, paragraph_ix)?; + paragraph_ix += 1; + } + Block::Image(image) => { + let pos = body_text.len_unicode(); + body_text.insert(pos, &OBJECT_REPLACEMENT.to_string())?; + let block = ensure_block(&blocks, block_id("image", block_ix), "image", BODY_FLOW_ID, &body_text, pos)?; + block.insert("asset_id", image.asset_id.0.to_string())?; + block.insert("alt_text_flow_id", nested_flow_id("image_alt", block_ix))?; + let alt_flow = ensure_flow(&flows, &nested_flow_id("image_alt", block_ix), "alt_text")?; + replace_text(&alt_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?, image.alt_text.as_ref())?; + if let Some(caption) = &image.caption { + block.insert("caption_flow_id", nested_flow_id("image_caption", block_ix))?; + let caption_flow = ensure_flow(&flows, &nested_flow_id("image_caption", block_ix), "caption")?; + replace_text(&caption_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?, SENTINEL_NEWLINE)?; + append_paragraph_text_only(&caption_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?, caption, "")?; + } + let attrs = block.ensure_mergeable_map("attrs")?; + attrs.insert("alignment", alignment_name(image.alignment))?; + match image.sizing { + ImageSizing::Intrinsic => attrs.insert("sizing", "intrinsic")?, + ImageSizing::FitWidth => attrs.insert("sizing", "fit_width")?, + ImageSizing::Fixed { width_px, height_px } => { + attrs.insert("sizing", "fixed")?; + attrs.insert("width_px", i64::from(width_px))?; + if let Some(height_px) = height_px { + attrs.insert("height_px", i64::from(height_px))?; + } + } + }; + } + Block::Equation(equation) => { + let pos = body_text.len_unicode(); + body_text.insert(pos, &OBJECT_REPLACEMENT.to_string())?; + let block = ensure_block(&blocks, block_id("equation", block_ix), "equation", BODY_FLOW_ID, &body_text, pos)?; + let source_flow_id = nested_flow_id("equation_source", block_ix); + block.insert("source_flow_id", source_flow_id.as_str())?; + let source_flow = ensure_flow(&flows, &source_flow_id, "equation_source")?; + replace_text(&source_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?, equation.source.as_ref())?; + let attrs = block.ensure_mergeable_map("attrs")?; + attrs.insert("syntax", equation_syntax_name(equation.syntax))?; + attrs.insert("display", equation_display_name(equation.display))?; + } + Block::Table(table) => { + let pos = body_text.len_unicode(); + body_text.insert(pos, &OBJECT_REPLACEMENT.to_string())?; + let block = ensure_block(&blocks, block_id("table", block_ix), "table", BODY_FLOW_ID, &body_text, pos)?; + import_table(&flows, &block, table, &format!("table.{block_ix}"))?; + } + } + } + doc.commit(); + Ok(()) +} + +fn append_paragraph( + body_text: &LoroText, + paragraphs: &LoroMap, + blocks: &LoroMap, + flow_id: &str, + paragraph: &Paragraph, + text: &str, + block_ix: usize, + paragraph_ix: usize, +) -> LoroResult<()> { + append_paragraph_text_only(body_text, paragraph, text)?; + let boundary_pos = body_text.len_unicode() - text.chars().count() - 1; + let paragraph_id = block_id("paragraph", paragraph_ix); + let paragraph_map = paragraphs.ensure_mergeable_map(¶graph_id)?; + paragraph_map.insert("id", paragraph_id.as_str())?; + paragraph_map.insert("flow_id", flow_id)?; + if let Some(cursor) = body_text.get_cursor(boundary_pos, Side::Left) { + paragraph_map.insert("start_cursor", cursor.encode())?; + } + if let Some(cursor) = body_text.get_cursor(boundary_pos, Side::Right) { + paragraph_map.insert("boundary_cursor", cursor.encode())?; + } + paragraph_map.ensure_mergeable_map("attrs")?; + ensure_block(blocks, block_id("paragraph_block", block_ix), "paragraph", flow_id, body_text, boundary_pos)?; + Ok(()) +} + +fn append_paragraph_text_only(text: &LoroText, paragraph: &Paragraph, paragraph_body: &str) -> LoroResult<()> { + let use_existing_sentinel = text.len_unicode() == 1 && text.to_string() == SENTINEL_NEWLINE; + let boundary_pos = if use_existing_sentinel { + 0 + } else { + let pos = text.len_unicode(); + text.insert(pos, "\n")?; + pos + }; + text.mark(boundary_pos..boundary_pos + 1, MARK_PARAGRAPH_STYLE, paragraph_style_value(paragraph.style))?; + let text_pos = text.len_unicode(); + if !paragraph_body.is_empty() { + text.insert(text_pos, paragraph_body)?; + mark_runs(text, text_pos, paragraph_body, ¶graph.runs)?; + } + Ok(()) +} + +fn mark_runs(text: &LoroText, start: usize, paragraph_body: &str, runs: &[gpui_flowtext::TextRun]) -> LoroResult<()> { + let mut offset = 0_usize; + for run in runs { + let Some(run_text) = paragraph_body.get(offset..offset.saturating_add(run.len)) else { + break; + }; + let run_len = run_text.chars().count(); + let range = start + paragraph_body[..offset].chars().count()..start + paragraph_body[..offset].chars().count() + run_len; + mark_run_styles(text, range, run.styles)?; + offset = offset.saturating_add(run.len); + } + Ok(()) +} + +fn mark_run_styles(text: &LoroText, range: std::ops::Range, styles: RunStyles) -> LoroResult<()> { + if let RunSemanticStyle::Custom(slot) = styles.semantic { + text.mark(range.clone(), MARK_RUN_SEMANTIC_STYLE, i64::from(slot))?; + } + if let Some(HighlightStyle::Custom(slot)) = styles.highlight { + text.mark(range.clone(), MARK_HIGHLIGHT_STYLE, i64::from(slot))?; + } + if styles.direct_underline { + text.mark(range.clone(), MARK_DIRECT_UNDERLINE, true)?; + } + if styles.strikethrough { + text.mark(range, MARK_STRIKETHROUGH, true)?; + } + Ok(()) +} + +fn import_table(flows: &LoroMap, block: &LoroMap, table: &TableBlock, prefix: &str) -> LoroResult<()> { + let table_map = block.ensure_mergeable_map("table")?; + let row_order = table_map.ensure_mergeable_list("row_order")?; + let column_order = table_map.ensure_mergeable_list("column_order")?; + let rows_by_id = table_map.ensure_mergeable_map("rows_by_id")?; + let columns_by_id = table_map.ensure_mergeable_map("columns_by_id")?; + let cells_by_id = table_map.ensure_mergeable_map("cells_by_id")?; + table_map.insert("header_row", table.style.header_row)?; + + clear_list(&row_order)?; + clear_list(&column_order)?; + clear_map(&rows_by_id)?; + clear_map(&columns_by_id)?; + clear_map(&cells_by_id)?; + + for (column_ix, width) in table.column_widths.iter().enumerate() { + let column_id = format!("{prefix}.column.{column_ix}"); + column_order.push(column_id.as_str())?; + let column = columns_by_id.ensure_mergeable_map(&column_id)?; + column.insert("id", column_id.as_str())?; + match width { + TableColumnWidth::Auto => column.insert("width_kind", "auto")?, + TableColumnWidth::FixedPx(px) => { + column.insert("width_kind", "fixed_px")?; + column.insert("width_px", i64::from(*px))?; + } + TableColumnWidth::Fraction(fraction) => { + column.insert("width_kind", "fraction")?; + column.insert("fraction", i64::from(*fraction))?; + } + }; + } + + for (row_ix, row) in table.rows.iter().enumerate() { + let row_id = format!("{prefix}.row.{row_ix}"); + row_order.push(row_id.as_str())?; + let row_map = rows_by_id.ensure_mergeable_map(&row_id)?; + row_map.insert("id", row_id.as_str())?; + for (cell_ix, cell) in row.cells.iter().enumerate() { + let cell_id = format!("{row_id}.cell.{cell_ix}"); + let cell_map = cells_by_id.ensure_mergeable_map(&cell_id)?; + cell_map.insert("id", cell_id.as_str())?; + cell_map.insert("row_id", row_id.as_str())?; + cell_map.insert("column_index", i64::try_from(cell_ix).unwrap_or(i64::MAX))?; + cell_map.insert("row_span", i64::from(cell.row_span))?; + cell_map.insert("column_span", i64::from(cell.col_span))?; + let flow_id = format!("{cell_id}.flow"); + cell_map.insert("flow_id", flow_id.as_str())?; + let flow = ensure_flow(flows, &flow_id, "table_cell")?; + let text = flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + replace_text(&text, SENTINEL_NEWLINE)?; + for (block_ix, cell_block) in cell.blocks.iter().enumerate() { + match cell_block { + TableCellBlock::Paragraph(paragraph) => append_paragraph_text_only(&text, ¶graph.paragraph, ¶graph.text)?, + TableCellBlock::Table(nested) => { + let pos = text.len_unicode(); + text.insert(pos, &OBJECT_REPLACEMENT.to_string())?; + let nested_map = cell_map.ensure_mergeable_map(&format!("nested_table.{block_ix}"))?; + import_table(flows, &nested_map, nested, &format!("{cell_id}.nested.{block_ix}"))?; + } + } + } + } + } + Ok(()) +} + +fn import_assets(doc: &LoroDoc, document: &Document) -> LoroResult<()> { + let root = doc.get_map(ROOT); + let assets = root.ensure_mergeable_map(ASSETS_BY_ID)?; + clear_map(&assets)?; + for asset in document.assets.assets.values() { + let asset_id = asset.asset_id_string(); + let asset_map = assets.ensure_mergeable_map(&asset_id)?; + let hash = blake3::hash(&asset.bytes); + asset_map.insert("asset_id", asset_id.as_str())?; + asset_map.insert("content_hash", hash.to_hex().as_str())?; + asset_map.insert("mime_type", asset.mime_type.as_ref())?; + asset_map.insert("byte_length", i64::try_from(asset.bytes.len()).unwrap_or(i64::MAX))?; + if let Some(original_name) = &asset.original_name { + asset_map.insert("original_name", original_name.as_ref())?; + } + } + Ok(()) +} + +pub fn assets_from_document(document: &Document) -> Vec { + document + .assets + .assets + .values() + .map(|asset| AssetChunk { + asset_id: asset.id.0, + content_hash: *blake3::hash(&asset.bytes).as_bytes(), + mime_type: asset.mime_type.to_string(), + byte_length: asset.bytes.len() as u64, + bytes: Vec::clone(&asset.bytes), + metadata: Vec::new(), + }) + .collect() +} + +trait AssetRecordExt { + fn asset_id_string(&self) -> String; +} + +impl AssetRecordExt for gpui_flowtext::AssetRecord { + fn asset_id_string(&self) -> String { + self.id.0.to_string() + } +} + +fn ensure_flow(flows: &LoroMap, flow_id: &str, kind: &str) -> LoroResult { + let flow = flows.ensure_mergeable_map(flow_id)?; + flow.insert(FLOW_ID_KEY, flow_id)?; + flow.insert(FLOW_KIND_KEY, kind)?; + flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + flow.ensure_mergeable_map(FLOW_ATTRS_KEY)?; + Ok(flow) +} + +fn ensure_block(blocks: &LoroMap, block_id: String, kind: &str, flow_id: &str, text: &LoroText, pos: usize) -> LoroResult { + let block = blocks.ensure_mergeable_map(&block_id)?; + block.insert("id", block_id.as_str())?; + block.insert("kind", kind)?; + block.insert("flow_id", flow_id)?; + if let Some(cursor) = text.get_cursor(pos, Side::Left) { + block.insert("anchor_cursor", cursor.encode())?; + } + block.ensure_mergeable_map("attrs")?; + block.ensure_mergeable_map("nested_refs")?; + Ok(block) +} + +fn replace_text(text: &LoroText, value: &str) -> LoroResult<()> { + let len = text.len_unicode(); + if len > 0 { + text.delete(0, len)?; + } + if !value.is_empty() { + text.insert(0, value)?; + } + Ok(()) +} + +fn clear_map(map: &LoroMap) -> LoroResult<()> { + let keys = map.keys(); + for key in keys { + map.delete(&key)?; + } + Ok(()) +} + +fn clear_list(list: &loro::LoroList) -> LoroResult<()> { + let len = list.len(); + if len > 0 { + list.delete(0, len)?; + } + Ok(()) +} + +fn paragraph_style_value(style: ParagraphStyle) -> i64 { + match style { + ParagraphStyle::Normal => 0, + ParagraphStyle::Custom(slot) => i64::from(slot), + } +} + +fn block_id(kind: &str, ix: usize) -> String { + format!("{kind}.{ix}.{}", Uuid::new_v5(&Uuid::NAMESPACE_OID, format!("{kind}.{ix}").as_bytes()).as_u128()) +} + +fn nested_flow_id(kind: &str, ix: usize) -> String { + format!("{kind}.{ix}") +} + +fn alignment_name(alignment: BlockAlignment) -> &'static str { + match alignment { + BlockAlignment::Left => "left", + BlockAlignment::Center => "center", + BlockAlignment::Right => "right", + } +} + +fn equation_syntax_name(syntax: EquationSyntax) -> &'static str { + match syntax { + EquationSyntax::Latex => "latex", + } +} + +fn equation_display_name(display: EquationDisplay) -> &'static str { + match display { + EquationDisplay::Display => "display", + EquationDisplay::InlineLikeParagraph => "inline_like_paragraph", + } +} + +fn loro_io_error(error: impl std::error::Error + Send + Sync + 'static) -> io::Error { + io::Error::new(io::ErrorKind::InvalidData, error) +} diff --git a/crates/flowstate-document/src/loro_projection.rs b/crates/flowstate-document/src/loro_projection.rs new file mode 100644 index 00000000..76c64230 --- /dev/null +++ b/crates/flowstate-document/src/loro_projection.rs @@ -0,0 +1,501 @@ +use std::{collections::BTreeMap, io}; + +use gpui_flowtext::{ + AssetId, Document, DocumentTheme, HighlightStyle, InputBlock, InputBlockAlignment, InputEquationBlock, InputEquationDisplay, + InputEquationSyntax, InputImageBlock, InputImageSizing, InputParagraph, InputRun, InputTableBlock, InputTableCell, InputTableCellBlock, + InputTableColumnWidth, InputTableRow, InputTableStyle, RunSemanticStyle, RunStyles, document_from_input_blocks, +}; +use loro::{LoroDoc, LoroMap, LoroText, LoroValue, ValueOrContainer, cursor::Cursor}; +use rustc_hash::FxHashMap; + +use crate::{ + BLOCKS_BY_ID, FLOW_TEXT_KEY, FLOWS_BY_ID, MARK_DIRECT_UNDERLINE, MARK_HIGHLIGHT_STYLE, MARK_PARAGRAPH_STYLE, MARK_RUN_SEMANTIC_STYLE, + MARK_STRIKETHROUGH, OBJECT_REPLACEMENT, ROOT, ROOT_BODY_FLOW_ID, flowstate_document_theme, +}; + +pub fn document_from_loro(doc: &LoroDoc) -> io::Result { + let projector = Projector::new(doc)?; + let blocks = projector.body_blocks()?; + Ok(document_from_input_blocks(DocumentTheme::clone(&flowstate_document_theme()), blocks)) +} + +struct Projector<'a> { + doc: &'a LoroDoc, + flows: LoroMap, + blocks: LoroMap, +} + +impl<'a> Projector<'a> { + fn new(doc: &'a LoroDoc) -> io::Result { + let root = doc.get_map(ROOT); + let flows = child_map(&root, FLOWS_BY_ID)?.ok_or_else(|| invalid("Flowstate Loro package has no flows map"))?; + let blocks = child_map(&root, BLOCKS_BY_ID)?.ok_or_else(|| invalid("Flowstate Loro package has no block registry"))?; + Ok(Self { doc, flows, blocks }) + } + + fn body_blocks(&self) -> io::Result> { + let body = self.flow_text(ROOT_BODY_FLOW_ID)?; + let body_blocks = self.object_blocks_for_flow(ROOT_BODY_FLOW_ID)?; + let mut blocks = Vec::new(); + self.push_flow_blocks(&body, &body_blocks, &mut blocks)?; + if blocks.is_empty() { + blocks.push(InputBlock::Paragraph(InputParagraph { + style: gpui_flowtext::ParagraphStyle::Normal, + runs: Vec::new(), + })); + } + Ok(blocks) + } + + fn push_flow_blocks(&self, text: &LoroText, object_blocks: &BTreeMap, output: &mut Vec) -> io::Result<()> { + let mut current = InputParagraph { + style: gpui_flowtext::ParagraphStyle::Normal, + runs: Vec::new(), + }; + let mut pending_style = gpui_flowtext::ParagraphStyle::Normal; + let mut seen_sentinel = false; + let mut unicode_pos = 0_usize; + + for item in text.to_delta() { + let loro::TextDelta::Insert { insert, attributes } = item else { + continue; + }; + let run_styles = run_styles_from_attrs(attributes.as_ref()); + for ch in insert.chars() { + match ch { + '\n' => { + let style = paragraph_style_from_attrs(attributes.as_ref()).unwrap_or(pending_style); + if !seen_sentinel { + seen_sentinel = true; + pending_style = style; + current.style = style; + } else { + output.push(InputBlock::Paragraph(current)); + current = InputParagraph { style, runs: Vec::new() }; + pending_style = style; + } + } + OBJECT_REPLACEMENT => { + if let Some(block) = object_blocks.get(&unicode_pos) { + if !current.runs.is_empty() { + output.push(InputBlock::Paragraph(current)); + current = InputParagraph { + style: pending_style, + runs: Vec::new(), + }; + } + output.push(self.object_block(block)?); + } + } + _ => push_char(&mut current, ch, run_styles), + } + unicode_pos += 1; + } + } + + if !current.runs.is_empty() || output.is_empty() && seen_sentinel { + output.push(InputBlock::Paragraph(current)); + } + Ok(()) + } + + fn object_blocks_for_flow(&self, flow_id: &str) -> io::Result> { + let mut by_pos = BTreeMap::new(); + for key in self.blocks.keys() { + let key = key.to_string(); + let Some(block) = child_map(&self.blocks, &key)? else { + continue; + }; + if map_string_opt(&block, "flow_id")?.as_deref() != Some(flow_id) { + continue; + } + if map_string_opt(&block, "kind")?.as_deref() == Some("paragraph") { + continue; + } + let Some(cursor_bytes) = map_binary_opt(&block, "anchor_cursor")? else { + continue; + }; + let Ok(cursor) = Cursor::decode(&cursor_bytes) else { + continue; + }; + if let Ok(pos) = self.doc.get_cursor_pos(&cursor) { + by_pos.insert(pos.current.pos, block); + } + } + Ok(by_pos) + } + + fn object_block(&self, block: &LoroMap) -> io::Result { + match map_string(block, "kind")?.as_str() { + "image" => self.image_block(block).map(InputBlock::Image), + "equation" => self.equation_block(block).map(InputBlock::Equation), + "table" => self.table_block(block).map(InputBlock::Table), + kind => Err(invalid(format!("unsupported Loro block kind `{kind}`"))), + } + } + + fn image_block(&self, block: &LoroMap) -> io::Result { + let attrs = child_map(block, "attrs")?; + Ok(InputImageBlock { + asset_id: AssetId(parse_u128(&map_string(block, "asset_id")?).unwrap_or_default()), + alt_text: map_string_opt(block, "alt_text_flow_id")? + .map(|flow_id| self.plain_flow_text(&flow_id)) + .transpose()? + .unwrap_or_default(), + caption: map_string_opt(block, "caption_flow_id")? + .map(|flow_id| self.caption_paragraph(&flow_id)) + .transpose()?, + sizing: image_sizing(attrs.as_ref())?, + alignment: alignment(attrs.as_ref())?, + }) + } + + fn equation_block(&self, block: &LoroMap) -> io::Result { + let attrs = child_map(block, "attrs")?; + Ok(InputEquationBlock { + source: map_string_opt(block, "source_flow_id")? + .map(|flow_id| self.plain_flow_text(&flow_id)) + .transpose()? + .unwrap_or_default(), + syntax: equation_syntax(attrs.as_ref())?, + display: equation_display(attrs.as_ref())?, + }) + } + + fn table_block(&self, owner: &LoroMap) -> io::Result { + let table = child_map(owner, "table")?.ok_or_else(|| invalid("table block has no table map"))?; + self.table_from_map(&table) + } + + fn table_from_map(&self, table: &LoroMap) -> io::Result { + let columns = child_map(table, "columns_by_id")?.ok_or_else(|| invalid("table has no column map"))?; + let rows_by_id = child_map(table, "rows_by_id")?.ok_or_else(|| invalid("table has no row map"))?; + let cells_by_id = child_map(table, "cells_by_id")?.ok_or_else(|| invalid("table has no cell map"))?; + let column_widths = ordered_ids(table, "column_order")? + .into_iter() + .map(|column_id| { + let column = child_map(&columns, &column_id)?.ok_or_else(|| invalid(format!("missing table column `{column_id}`")))?; + table_column_width(&column) + }) + .collect::>>()?; + + let mut rows = Vec::new(); + for row_id in ordered_ids(table, "row_order")? { + let _row = child_map(&rows_by_id, &row_id)?.ok_or_else(|| invalid(format!("missing table row `{row_id}`")))?; + let mut row_cells = Vec::new(); + let mut cell_ids = cells_by_id + .keys() + .map(|key| key.to_string()) + .filter_map(|cell_id| { + let cell = child_map(&cells_by_id, &cell_id).ok().flatten()?; + (map_string_opt(&cell, "row_id").ok().flatten().as_deref() == Some(row_id.as_str())).then_some((cell_id, cell)) + }) + .collect::>(); + cell_ids.sort_by_key(|(_, cell)| map_i64_opt(cell, "column_index").ok().flatten().unwrap_or(i64::MAX)); + for (_, cell) in cell_ids { + row_cells.push(self.table_cell(&cell)?); + } + rows.push(InputTableRow { cells: row_cells }); + } + + Ok(InputTableBlock { + rows, + column_widths, + style: InputTableStyle { + header_row: map_bool_opt(table, "header_row")?.unwrap_or(false), + }, + }) + } + + fn table_cell(&self, cell: &LoroMap) -> io::Result { + let flow_id = map_string(cell, "flow_id")?; + let flow = self.flow_text(&flow_id)?; + let object_blocks = self.cell_nested_tables(cell)?; + let mut projected = Vec::new(); + self.push_flow_blocks(&flow, &object_blocks, &mut projected)?; + let mut blocks = projected + .into_iter() + .filter_map(|block| match block { + InputBlock::Paragraph(paragraph) => Some(Ok(InputTableCellBlock::Paragraph(paragraph))), + InputBlock::Table(table) => Some(Ok(InputTableCellBlock::Table(table))), + InputBlock::Image(_) | InputBlock::Equation(_) => None, + }) + .collect::>>()?; + if blocks.is_empty() { + blocks.push(InputTableCellBlock::Paragraph(InputParagraph { + style: gpui_flowtext::ParagraphStyle::Normal, + runs: Vec::new(), + })); + } + Ok(InputTableCell { + blocks, + row_span: map_i64_opt(cell, "row_span")?.and_then(i64_to_u16).unwrap_or(1), + col_span: map_i64_opt(cell, "column_span")?.and_then(i64_to_u16).unwrap_or(1), + }) + } + + fn cell_nested_tables(&self, cell: &LoroMap) -> io::Result> { + let mut tables = BTreeMap::new(); + for key in cell.keys() { + let key = key.to_string(); + if !key.starts_with("nested_table.") { + continue; + } + let Some(owner) = child_map(cell, &key)? else { + continue; + }; + if let Some(pos) = key.rsplit('.').next().and_then(|segment| segment.parse::().ok()) { + tables.insert(pos, owner); + } + } + Ok(tables) + } + + fn flow_text(&self, flow_id: &str) -> io::Result { + let flow = child_map(&self.flows, flow_id)?.ok_or_else(|| invalid(format!("missing flow `{flow_id}`")))?; + child_text(&flow, FLOW_TEXT_KEY)?.ok_or_else(|| invalid(format!("flow `{flow_id}` has no text"))) + } + + fn plain_flow_text(&self, flow_id: &str) -> io::Result { + Ok(self.flow_text(flow_id)?.to_string().trim_start_matches('\n').to_string()) + } + + fn caption_paragraph(&self, flow_id: &str) -> io::Result { + let paragraphs = paragraphs_from_text(&self.flow_text(flow_id)?); + Ok(paragraphs.into_iter().next().unwrap_or(InputParagraph { + style: gpui_flowtext::ParagraphStyle::Normal, + runs: Vec::new(), + })) + } +} + +fn paragraphs_from_text(text: &LoroText) -> Vec { + let mut blocks = Vec::new(); + let projector = ParagraphOnlyProjector; + projector.push_flow_blocks(text, &mut blocks); + blocks +} + +struct ParagraphOnlyProjector; + +impl ParagraphOnlyProjector { + fn push_flow_blocks(&self, text: &LoroText, output: &mut Vec) { + let mut current = InputParagraph { + style: gpui_flowtext::ParagraphStyle::Normal, + runs: Vec::new(), + }; + let mut pending_style = gpui_flowtext::ParagraphStyle::Normal; + let mut seen_sentinel = false; + for item in text.to_delta() { + let loro::TextDelta::Insert { insert, attributes } = item else { + continue; + }; + let run_styles = run_styles_from_attrs(attributes.as_ref()); + for ch in insert.chars() { + if ch == '\n' { + let style = paragraph_style_from_attrs(attributes.as_ref()).unwrap_or(pending_style); + if !seen_sentinel { + seen_sentinel = true; + pending_style = style; + current.style = style; + } else { + output.push(current); + current = InputParagraph { style, runs: Vec::new() }; + pending_style = style; + } + } else if ch != OBJECT_REPLACEMENT { + push_char(&mut current, ch, run_styles); + } + } + } + if seen_sentinel || !current.runs.is_empty() { + output.push(current); + } + } +} + +fn push_char(paragraph: &mut InputParagraph, ch: char, styles: RunStyles) { + if let Some(last) = paragraph.runs.last_mut() + && last.styles == styles + { + last.text.push(ch); + return; + } + paragraph.runs.push(InputRun { + text: ch.to_string(), + styles, + }); +} + +fn child_map(parent: &LoroMap, key: &str) -> io::Result> { + Ok(parent.get(key).and_then(|value| match value { + ValueOrContainer::Container(container) => container.into_map().ok(), + ValueOrContainer::Value(_) => None, + })) +} + +fn child_text(parent: &LoroMap, key: &str) -> io::Result> { + Ok(parent.get(key).and_then(|value| match value { + ValueOrContainer::Container(container) => container.into_text().ok(), + ValueOrContainer::Value(_) => None, + })) +} + +fn ordered_ids(map: &LoroMap, key: &str) -> io::Result> { + let Some(list) = map.get(key).and_then(|value| match value { + ValueOrContainer::Container(container) => container.into_list().ok(), + ValueOrContainer::Value(_) => None, + }) else { + return Ok(Vec::new()); + }; + Ok( + list + .get_deep_value() + .into_list() + .unwrap_or_default() + .iter() + .filter_map(|value| match value { + LoroValue::String(value) => Some(value.to_string()), + _ => None, + }) + .collect(), + ) +} + +fn map_string(map: &LoroMap, key: &str) -> io::Result { + map_string_opt(map, key)?.ok_or_else(|| invalid(format!("missing string field `{key}`"))) +} + +fn map_string_opt(map: &LoroMap, key: &str) -> io::Result> { + Ok(map.get(key).and_then(|value| match value { + ValueOrContainer::Value(LoroValue::String(value)) => Some(value.to_string()), + _ => None, + })) +} + +fn map_binary_opt(map: &LoroMap, key: &str) -> io::Result>> { + Ok(map.get(key).and_then(|value| match value { + ValueOrContainer::Value(LoroValue::Binary(value)) => Some(value.to_vec()), + _ => None, + })) +} + +fn map_i64_opt(map: &LoroMap, key: &str) -> io::Result> { + Ok(map.get(key).and_then(|value| match value { + ValueOrContainer::Value(LoroValue::I64(value)) => Some(value), + _ => None, + })) +} + +fn map_bool_opt(map: &LoroMap, key: &str) -> io::Result> { + Ok(map.get(key).and_then(|value| match value { + ValueOrContainer::Value(LoroValue::Bool(value)) => Some(value), + _ => None, + })) +} + +fn paragraph_style_from_attrs(attrs: Option<&FxHashMap>) -> Option { + let value = attrs?.get(MARK_PARAGRAPH_STYLE)?; + match value { + LoroValue::I64(0) => Some(gpui_flowtext::ParagraphStyle::Normal), + LoroValue::I64(slot) => u8::try_from(*slot).ok().map(gpui_flowtext::ParagraphStyle::Custom), + _ => None, + } +} + +fn run_styles_from_attrs(attrs: Option<&FxHashMap>) -> RunStyles { + let mut styles = RunStyles::default(); + let Some(attrs) = attrs else { + return styles; + }; + if let Some(LoroValue::I64(slot)) = attrs.get(MARK_RUN_SEMANTIC_STYLE) + && let Ok(slot) = u8::try_from(*slot) + { + styles.semantic = RunSemanticStyle::Custom(slot); + } + if let Some(LoroValue::I64(slot)) = attrs.get(MARK_HIGHLIGHT_STYLE) + && let Ok(slot) = u8::try_from(*slot) + { + styles.highlight = Some(HighlightStyle::Custom(slot)); + } + if matches!(attrs.get(MARK_DIRECT_UNDERLINE), Some(LoroValue::Bool(true))) { + styles.direct_underline = true; + } + if matches!(attrs.get(MARK_STRIKETHROUGH), Some(LoroValue::Bool(true))) { + styles.strikethrough = true; + } + styles +} + +fn image_sizing(attrs: Option<&LoroMap>) -> io::Result { + let Some(attrs) = attrs else { + return Ok(InputImageSizing::FitWidth); + }; + match map_string_opt(attrs, "sizing")?.as_deref() { + Some("intrinsic") => Ok(InputImageSizing::Intrinsic), + Some("fixed") => Ok(InputImageSizing::Fixed { + width_px: map_i64_opt(attrs, "width_px")?.and_then(i64_to_u32).unwrap_or(640), + height_px: map_i64_opt(attrs, "height_px")?.and_then(i64_to_u32), + }), + Some("fit_width") | None => Ok(InputImageSizing::FitWidth), + Some(_) => Ok(InputImageSizing::FitWidth), + } +} + +fn alignment(attrs: Option<&LoroMap>) -> io::Result { + let Some(attrs) = attrs else { + return Ok(InputBlockAlignment::Left); + }; + Ok(match map_string_opt(attrs, "alignment")?.as_deref() { + Some("center") => InputBlockAlignment::Center, + Some("right") => InputBlockAlignment::Right, + Some("left") | None => InputBlockAlignment::Left, + Some(_) => InputBlockAlignment::Left, + }) +} + +fn equation_syntax(attrs: Option<&LoroMap>) -> io::Result { + let Some(attrs) = attrs else { + return Ok(InputEquationSyntax::Latex); + }; + Ok(match map_string_opt(attrs, "syntax")?.as_deref() { + Some("latex") | None => InputEquationSyntax::Latex, + Some(_) => InputEquationSyntax::Latex, + }) +} + +fn equation_display(attrs: Option<&LoroMap>) -> io::Result { + let Some(attrs) = attrs else { + return Ok(InputEquationDisplay::Display); + }; + Ok(match map_string_opt(attrs, "display")?.as_deref() { + Some("inline_like_paragraph") => InputEquationDisplay::InlineLikeParagraph, + Some("display") | None => InputEquationDisplay::Display, + Some(_) => InputEquationDisplay::Display, + }) +} + +fn table_column_width(column: &LoroMap) -> io::Result { + Ok(match map_string_opt(column, "width_kind")?.as_deref() { + Some("fixed_px") => InputTableColumnWidth::FixedPx(map_i64_opt(column, "width_px")?.and_then(i64_to_u32).unwrap_or(120)), + Some("fraction") => InputTableColumnWidth::Fraction(map_i64_opt(column, "fraction")?.and_then(i64_to_u32).unwrap_or(1)), + Some("auto") | None => InputTableColumnWidth::Auto, + Some(_) => InputTableColumnWidth::Auto, + }) +} + +fn parse_u128(value: &str) -> Option { + value.parse::().ok() +} + +fn i64_to_u32(value: i64) -> Option { + u32::try_from(value).ok() +} + +fn i64_to_u16(value: i64) -> Option { + u16::try_from(value).ok() +} + +fn invalid(message: impl Into) -> io::Error { + io::Error::new(io::ErrorKind::InvalidData, message.into()) +} diff --git a/crates/flowstate-document/src/loro_schema.rs b/crates/flowstate-document/src/loro_schema.rs new file mode 100644 index 00000000..346ae8cc --- /dev/null +++ b/crates/flowstate-document/src/loro_schema.rs @@ -0,0 +1,159 @@ +use std::time::{SystemTime, UNIX_EPOCH}; + +use loro::{ExpandType, LoroDoc, LoroMap, LoroResult, LoroText, StyleConfig, StyleConfigMap, cursor::Side}; +use uuid::Uuid; + +use crate::LORO_SCHEMA_VERSION; + +pub const ROOT: &str = "flowstate.root"; +pub const META: &str = "meta"; +pub const FLOWS_BY_ID: &str = "flows_by_id"; +pub const BLOCKS_BY_ID: &str = "blocks_by_id"; +pub const PARAGRAPHS_BY_ID: &str = "paragraphs_by_id"; +pub const SECTIONS_BY_ID: &str = "sections_by_id"; +pub const ASSETS_BY_ID: &str = "assets_by_id"; +pub const REVISIONS: &str = "revisions"; +pub const USERS_BY_ID: &str = "users_by_id"; +pub const REPLICAS_BY_ID: &str = "replicas_by_id"; + +pub const FLOW_TEXT_KEY: &str = "text"; +pub const FLOW_ATTRS_KEY: &str = "attrs"; +pub const FLOW_KIND_KEY: &str = "kind"; +pub const FLOW_ID_KEY: &str = "id"; + +pub const BODY_FLOW_ID: &str = "body"; +pub const ROOT_BODY_FLOW_ID: &str = "body"; +pub const ROOT_FIRST_PARAGRAPH_ID: &str = "paragraph.initial"; +pub const MAIN_BODY_BLOCK_ID: &str = "block.body.initial"; + +pub const MARK_PARAGRAPH_STYLE: &str = "paragraph_style"; +pub const MARK_RUN_SEMANTIC_STYLE: &str = "run_semantic_style_id"; +pub const MARK_HIGHLIGHT_STYLE: &str = "highlight_style_id"; +pub const MARK_DIRECT_UNDERLINE: &str = "direct_underline"; +pub const MARK_STRIKETHROUGH: &str = "strikethrough"; + +pub const OBJECT_REPLACEMENT: char = '\u{FFFC}'; +pub const SENTINEL_NEWLINE: &str = "\n"; + +pub fn new_loro_document(title: &str) -> LoroResult { + let doc = LoroDoc::new(); + init_loro_document(&doc, title)?; + Ok(doc) +} + +pub fn init_loro_document(doc: &LoroDoc, title: &str) -> LoroResult<()> { + configure_text_styles(doc); + + let root = doc.get_map(ROOT); + let meta = root.ensure_mergeable_map(META)?; + let flows = root.ensure_mergeable_map(FLOWS_BY_ID)?; + let blocks = root.ensure_mergeable_map(BLOCKS_BY_ID)?; + let paragraphs = root.ensure_mergeable_map(PARAGRAPHS_BY_ID)?; + root.ensure_mergeable_map(SECTIONS_BY_ID)?; + root.ensure_mergeable_map(ASSETS_BY_ID)?; + root.ensure_mergeable_list(REVISIONS)?; + root.ensure_mergeable_map(USERS_BY_ID)?; + root.ensure_mergeable_map(REPLICAS_BY_ID)?; + + init_meta(&meta, title)?; + let body_flow = ensure_flow(&flows, ROOT_BODY_FLOW_ID, "body")?; + let body_text = body_flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + body_flow.ensure_mergeable_map(FLOW_ATTRS_KEY)?; + ensure_sentinel(&body_text)?; + ensure_initial_paragraph(¶graphs, &blocks, &body_text)?; + + doc.commit(); + Ok(()) +} + +pub fn configure_text_styles(doc: &LoroDoc) { + let mut styles = StyleConfigMap::new(); + let no_expand = StyleConfig::new().expand(ExpandType::None); + styles.insert(MARK_PARAGRAPH_STYLE.into(), no_expand); + styles.insert(MARK_RUN_SEMANTIC_STYLE.into(), no_expand); + styles.insert(MARK_HIGHLIGHT_STYLE.into(), no_expand); + styles.insert(MARK_DIRECT_UNDERLINE.into(), no_expand); + styles.insert(MARK_STRIKETHROUGH.into(), no_expand); + doc.config_text_style(styles); +} + +pub fn root_map(doc: &LoroDoc) -> LoroMap { + doc.get_map(ROOT) +} + +pub fn body_text(doc: &LoroDoc) -> LoroText { + let root = root_map(doc); + let flows = root + .ensure_mergeable_map(FLOWS_BY_ID) + .expect("root flows map should be initialized"); + let body = flows + .ensure_mergeable_map(ROOT_BODY_FLOW_ID) + .expect("body flow should be initialized"); + body + .ensure_mergeable_text(FLOW_TEXT_KEY) + .expect("body text should be initialized") +} + +fn init_meta(meta: &LoroMap, title: &str) -> LoroResult<()> { + let now = unix_time_secs(); + if meta.get("document_id").is_none() { + meta.insert("document_id", Uuid::new_v4().to_string())?; + } + meta.insert("loro_schema_version", i64::from(LORO_SCHEMA_VERSION))?; + meta.insert("schema_features", "flow-v1")?; + meta.insert("title", title)?; + meta.insert("created_by_app_version", env!("CARGO_PKG_VERSION"))?; + meta.insert("last_written_by_app_version", env!("CARGO_PKG_VERSION"))?; + if meta.get("created_at").is_none() { + meta.insert("created_at", now)?; + } + meta.insert("modified_at", now)?; + Ok(()) +} + +fn ensure_flow(flows: &LoroMap, flow_id: &str, kind: &str) -> LoroResult { + let flow = flows.ensure_mergeable_map(flow_id)?; + flow.insert(FLOW_ID_KEY, flow_id)?; + flow.insert(FLOW_KIND_KEY, kind)?; + flow.ensure_mergeable_text(FLOW_TEXT_KEY)?; + flow.ensure_mergeable_map(FLOW_ATTRS_KEY)?; + Ok(flow) +} + +fn ensure_sentinel(text: &LoroText) -> LoroResult<()> { + if text.len_unicode() == 0 || !text.to_string().starts_with(SENTINEL_NEWLINE) { + text.insert(0, SENTINEL_NEWLINE)?; + text.mark(0..1, MARK_PARAGRAPH_STYLE, 0_i64)?; + } + Ok(()) +} + +fn ensure_initial_paragraph(paragraphs: &LoroMap, blocks: &LoroMap, body: &LoroText) -> LoroResult<()> { + let paragraph = paragraphs.ensure_mergeable_map(ROOT_FIRST_PARAGRAPH_ID)?; + paragraph.insert("id", ROOT_FIRST_PARAGRAPH_ID)?; + paragraph.insert("flow_id", ROOT_BODY_FLOW_ID)?; + if let Some(cursor) = body.get_cursor(0, Side::Left) { + paragraph.insert("start_cursor", cursor.encode())?; + } + if let Some(cursor) = body.get_cursor(0, Side::Right) { + paragraph.insert("boundary_cursor", cursor.encode())?; + } + paragraph.ensure_mergeable_map("attrs")?; + + let block = blocks.ensure_mergeable_map(MAIN_BODY_BLOCK_ID)?; + block.insert("id", MAIN_BODY_BLOCK_ID)?; + block.insert("kind", "paragraph")?; + block.insert("flow_id", ROOT_BODY_FLOW_ID)?; + if let Some(cursor) = body.get_cursor(0, Side::Left) { + block.insert("anchor_cursor", cursor.encode())?; + } + block.ensure_mergeable_map("attrs")?; + block.ensure_mergeable_map("nested_refs")?; + Ok(()) +} + +fn unix_time_secs() -> i64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |duration| i64::try_from(duration.as_secs()).unwrap_or(i64::MAX)) +} diff --git a/crates/flowstate-document/src/package.rs b/crates/flowstate-document/src/package.rs new file mode 100644 index 00000000..fb611821 --- /dev/null +++ b/crates/flowstate-document/src/package.rs @@ -0,0 +1,1031 @@ +use std::{ + fs, + io::{self, Cursor, Read as _, Write as _}, + path::Path, +}; + +use loro::{ExportMode, Frontiers, LoroDoc, VersionVector, cursor::Side}; +use serde::{Deserialize, Serialize}; +use tempfile::NamedTempFile; +use uuid::Uuid; + +pub const LORO_PACKAGE_FORMAT_VERSION: u32 = 1; +pub const LORO_SCHEMA_VERSION: u32 = 1; + +const PACKAGE_MAGIC: &[u8; 16] = b"FLOWDB8-LORO\0\0\0\0"; +const PACKAGE_HEADER_VERSION: u32 = 1; + +const CHUNK_MANIFEST: u32 = 1; +const CHUNK_LORO_SNAPSHOT: u32 = 2; +const CHUNK_LORO_UPDATE_SEGMENT: u32 = 3; +const CHUNK_ASSET: u32 = 4; +const CHUNK_REVISION_INDEX: u32 = 5; +const CHUNK_PROJECTION_CACHE: u32 = 6; +const CHUNK_SEARCH_UNIT: u32 = 7; + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct DocumentPackage { + pub manifest: DocumentPackageManifest, + pub loro_snapshots: Vec, + pub loro_update_segments: Vec, + pub assets: Vec, + pub revisions: Vec, + pub projection_caches: Vec, + pub search_units: Vec, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct DocumentPackageManifest { + pub package_format_version: u32, + pub loro_schema_version: u32, + pub document_id: u128, + pub latest_frontier: Vec, + pub latest_version_vector: Vec, + pub latest_snapshot_id: u128, + pub update_segment_index: Vec, + pub asset_index: Vec, + pub projection_cache_frontier: Option>, + pub search_cache_frontier: Option>, + pub created_at_unix_secs: i64, + pub modified_at_unix_secs: i64, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct ChunkRef { + pub id: u128, + pub checksum: [u8; 32], + pub byte_length: u64, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct LoroSnapshotChunk { + pub snapshot_id: u128, + pub frontier: Vec, + pub version_vector: Vec, + pub bytes: Vec, + pub created_at_unix_secs: i64, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct LoroUpdateSegmentChunk { + pub segment_id: u128, + pub from_frontier: Vec, + pub from_version_vector: Vec, + pub to_frontier: Vec, + pub to_version_vector: Vec, + pub bytes: Vec, + pub checksum: [u8; 32], + pub created_at_unix_secs: i64, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct AssetChunk { + pub asset_id: u128, + pub content_hash: [u8; 32], + pub mime_type: String, + pub byte_length: u64, + pub bytes: Vec, + pub metadata: Vec, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct PackageRevision { + pub revision_id: u128, + pub frontier: Vec, + pub version_vector: Vec, + pub title: String, + pub summary: String, + pub author_user_id: Option, + pub replica_id: Option, + pub created_at_unix_secs: i64, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct ProjectionCacheChunk { + pub frontier: Vec, + pub bytes: Vec, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct SearchUnitChunk { + pub frontier: Vec, + pub unit_id: u128, + pub unit_kind: String, + pub heading_path: Vec, + pub heading: String, + pub body: String, + pub insert_text: String, + pub paragraph_start_cursor: Vec, + pub paragraph_end_cursor: Vec, +} + +#[derive(Clone, Debug)] +struct Chunk { + kind: u32, + payload: Vec, +} + +#[derive(Clone, Debug)] +struct ChunkEntry { + kind: u32, + offset: u64, + len: u64, + checksum: [u8; 32], +} + +impl DocumentPackage { + pub fn from_loro_snapshot(doc: &LoroDoc, title: &str) -> io::Result { + Self::from_loro_snapshot_with_assets(doc, title, Vec::new()) + } + + pub fn from_loro_snapshot_with_assets(doc: &LoroDoc, title: &str, assets: Vec) -> io::Result { + doc.commit(); + let now = unix_time_secs(); + let document_id = Uuid::new_v4().as_u128(); + let snapshot_id = Uuid::new_v4().as_u128(); + let frontier = encode_frontiers(&doc.state_frontiers()); + let version_vector = encode_version_vector(&doc.state_vv()); + let snapshot = doc.export(ExportMode::Snapshot).map_err(loro_io_error)?; + let mut package = Self { + manifest: DocumentPackageManifest { + package_format_version: LORO_PACKAGE_FORMAT_VERSION, + loro_schema_version: LORO_SCHEMA_VERSION, + document_id, + latest_frontier: frontier.clone(), + latest_version_vector: version_vector.clone(), + latest_snapshot_id: snapshot_id, + update_segment_index: Vec::new(), + asset_index: Vec::new(), + projection_cache_frontier: None, + search_cache_frontier: None, + created_at_unix_secs: now, + modified_at_unix_secs: now, + }, + loro_snapshots: vec![LoroSnapshotChunk { + snapshot_id, + frontier: frontier.clone(), + version_vector: version_vector.clone(), + bytes: snapshot, + created_at_unix_secs: now, + }], + loro_update_segments: Vec::new(), + assets, + revisions: vec![PackageRevision { + revision_id: Uuid::new_v4().as_u128(), + frontier, + version_vector, + title: title.to_string(), + summary: "Initial snapshot".to_string(), + author_user_id: None, + replica_id: None, + created_at_unix_secs: now, + }], + projection_caches: Vec::new(), + search_units: Vec::new(), + } + .with_manifest_indexes()?; + package.rebuild_search_units_from_loro(doc)?; + Ok(package) + } + + pub fn load_loro_doc(&self) -> io::Result { + self.validate()?; + let snapshot = self + .latest_snapshot() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, "Flowstate package has no latest Loro snapshot"))?; + let doc = LoroDoc::new(); + doc.import(&snapshot.bytes).map_err(loro_io_error)?; + for segment in &self.loro_update_segments { + doc.import(&segment.bytes).map_err(loro_io_error)?; + } + Ok(doc) + } + + pub fn current_search_units(&self) -> &[SearchUnitChunk] { + if self.manifest.search_cache_frontier.as_deref() == Some(self.manifest.latest_frontier.as_slice()) { + &self.search_units + } else { + &[] + } + } + + pub fn append_update_segment( + &mut self, + from_frontier: &Frontiers, + from_version_vector: &VersionVector, + to_frontier: &Frontiers, + to_version_vector: &VersionVector, + bytes: Vec, + ) -> io::Result { + if bytes.is_empty() { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "cannot append an empty Loro update segment")); + } + let segment_id = Uuid::new_v4().as_u128(); + let now = unix_time_secs(); + let checksum = blake3_hash(&bytes); + self.loro_update_segments.push(LoroUpdateSegmentChunk { + segment_id, + from_frontier: encode_frontiers(from_frontier), + from_version_vector: encode_version_vector(from_version_vector), + to_frontier: encode_frontiers(to_frontier), + to_version_vector: encode_version_vector(to_version_vector), + bytes, + checksum, + created_at_unix_secs: now, + }); + self.manifest.latest_frontier = encode_frontiers(to_frontier); + self.manifest.latest_version_vector = encode_version_vector(to_version_vector); + self.manifest.search_cache_frontier = None; + self.search_units.clear(); + self.manifest.modified_at_unix_secs = now; + self.clone().with_manifest_indexes()?.validate()?; + *self = self.clone().with_manifest_indexes()?; + Ok(segment_id) + } + + pub fn create_named_revision( + &mut self, + doc: &LoroDoc, + title: impl Into, + summary: impl Into, + author_user_id: Option, + replica_id: Option, + ) -> io::Result { + doc.commit(); + let frontier = encode_frontiers(&doc.state_frontiers()); + let version_vector = encode_version_vector(&doc.state_vv()); + if frontier != self.manifest.latest_frontier || version_vector != self.manifest.latest_version_vector { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cannot create a package revision for a Loro state that has not been persisted into the package", + )); + } + let revision = PackageRevision { + revision_id: Uuid::new_v4().as_u128(), + frontier: frontier.clone(), + version_vector, + title: title.into(), + summary: summary.into(), + author_user_id, + replica_id, + created_at_unix_secs: unix_time_secs(), + }; + let revision_id = revision.revision_id; + if self.snapshot_for_frontier(&frontier).is_none() { + let revision_doc = doc.fork_at(&doc.state_frontiers()).map_err(loro_io_error)?; + self.loro_snapshots.push(LoroSnapshotChunk { + snapshot_id: Uuid::new_v4().as_u128(), + frontier, + version_vector: encode_version_vector(&revision_doc.state_vv()), + bytes: revision_doc.export(ExportMode::Snapshot).map_err(loro_io_error)?, + created_at_unix_secs: unix_time_secs(), + }); + } + self.revisions.push(revision); + self.manifest.modified_at_unix_secs = unix_time_secs(); + self.validate()?; + Ok(revision_id) + } + + pub fn load_revision_loro_doc(&self, revision_id: u128) -> io::Result { + let revision = self + .revisions + .iter() + .find(|revision| revision.revision_id == revision_id) + .ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, "Flowstate package revision is missing"))?; + if let Some(snapshot) = self.snapshot_for_frontier(&revision.frontier) { + let doc = LoroDoc::new(); + doc.import(&snapshot.bytes).map_err(loro_io_error)?; + return Ok(doc); + } + let doc = self.load_loro_doc()?; + let frontiers = decode_frontiers(&revision.frontier)?; + doc.fork_at(&frontiers).map_err(loro_io_error) + } + + pub fn compact_to_named_snapshot( + &mut self, + doc: &LoroDoc, + title: impl Into, + summary: impl Into, + author_user_id: Option, + replica_id: Option, + ) -> io::Result<(u128, u128)> { + let snapshot_id = self.compact_to_snapshot(doc)?; + let revision_id = self.create_named_revision(doc, title, summary, author_user_id, replica_id)?; + Ok((revision_id, snapshot_id)) + } + + pub fn compact_to_snapshot(&mut self, doc: &LoroDoc) -> io::Result { + doc.commit(); + let snapshot_id = Uuid::new_v4().as_u128(); + let now = unix_time_secs(); + let frontier = encode_frontiers(&doc.state_frontiers()); + let version_vector = encode_version_vector(&doc.state_vv()); + let bytes = doc.export(ExportMode::Snapshot).map_err(loro_io_error)?; + self.loro_snapshots.push(LoroSnapshotChunk { + snapshot_id, + frontier: frontier.clone(), + version_vector, + bytes, + created_at_unix_secs: now, + }); + self.loro_update_segments.clear(); + self.manifest.latest_snapshot_id = snapshot_id; + self.manifest.latest_frontier = frontier; + self.manifest.latest_version_vector = encode_version_vector(&doc.state_vv()); + self.manifest.modified_at_unix_secs = now; + *self = self.clone().with_manifest_indexes()?; + self.rebuild_search_units_from_loro(doc)?; + Ok(snapshot_id) + } + + pub fn rebuild_search_units_from_loro(&mut self, doc: &LoroDoc) -> io::Result<()> { + doc.commit(); + let frontier = encode_frontiers(&doc.state_frontiers()); + let body = crate::loro_schema::body_text(doc); + let body_string = body.to_string(); + let mut units = Vec::new(); + let mut paragraph_start = None; + let mut paragraph_ix = 0_usize; + let mut current = String::new(); + + for (unicode_ix, ch) in body_string.chars().enumerate() { + if ch == '\n' { + if let Some(start) = paragraph_start.take() { + push_search_unit( + &mut units, + &frontier, + &body, + self.manifest.document_id, + paragraph_ix, + start, + unicode_ix, + ¤t, + )?; + paragraph_ix += 1; + current.clear(); + } + paragraph_start = Some(unicode_ix + 1); + } else if paragraph_start.is_some() && ch != crate::OBJECT_REPLACEMENT { + current.push(ch); + } + } + + if let Some(start) = paragraph_start { + push_search_unit( + &mut units, + &frontier, + &body, + self.manifest.document_id, + paragraph_ix, + start, + body.len_unicode(), + ¤t, + )?; + } + + self.search_units = units; + self.manifest.search_cache_frontier = Some(frontier); + self.manifest.modified_at_unix_secs = unix_time_secs(); + self.validate()?; + Ok(()) + } + + pub fn read(path: impl AsRef) -> io::Result { + Self::from_bytes(&fs::read(path)?) + } + + pub fn write(&self, path: impl AsRef) -> io::Result<()> { + write_bytes_atomic(path.as_ref(), &self.to_bytes()?) + } + + pub fn from_bytes(bytes: &[u8]) -> io::Result { + let chunks = read_chunks(bytes)?; + let mut manifest = None; + let mut loro_snapshots = Vec::new(); + let mut loro_update_segments = Vec::new(); + let mut assets = Vec::new(); + let mut revisions = Vec::new(); + let mut projection_caches = Vec::new(); + let mut search_units = Vec::new(); + + for chunk in chunks { + match chunk.kind { + CHUNK_MANIFEST => manifest = Some(decode_chunk(&chunk.payload, "manifest")?), + CHUNK_LORO_SNAPSHOT => loro_snapshots.push(decode_chunk(&chunk.payload, "Loro snapshot")?), + CHUNK_LORO_UPDATE_SEGMENT => loro_update_segments.push(decode_chunk(&chunk.payload, "Loro update segment")?), + CHUNK_ASSET => assets.push(decode_chunk(&chunk.payload, "asset")?), + CHUNK_REVISION_INDEX => revisions.push(decode_chunk(&chunk.payload, "revision")?), + CHUNK_PROJECTION_CACHE => projection_caches.push(decode_chunk(&chunk.payload, "projection cache")?), + CHUNK_SEARCH_UNIT => search_units.push(decode_chunk(&chunk.payload, "search unit")?), + _ => return Err(io::Error::new(io::ErrorKind::InvalidData, "unknown Flowstate package chunk kind")), + } + } + + let package = Self { + manifest: manifest.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, "Flowstate package has no manifest"))?, + loro_snapshots, + loro_update_segments, + assets, + revisions, + projection_caches, + search_units, + }; + package.validate()?; + Ok(package) + } + + pub fn to_bytes(&self) -> io::Result> { + self.validate()?; + let mut chunks = Vec::new(); + chunks.push(Chunk { + kind: CHUNK_MANIFEST, + payload: encode_chunk(&self.manifest, "manifest")?, + }); + for snapshot in &self.loro_snapshots { + chunks.push(Chunk { + kind: CHUNK_LORO_SNAPSHOT, + payload: encode_chunk(snapshot, "Loro snapshot")?, + }); + } + for segment in &self.loro_update_segments { + chunks.push(Chunk { + kind: CHUNK_LORO_UPDATE_SEGMENT, + payload: encode_chunk(segment, "Loro update segment")?, + }); + } + for asset in &self.assets { + chunks.push(Chunk { + kind: CHUNK_ASSET, + payload: encode_chunk(asset, "asset")?, + }); + } + for revision in &self.revisions { + chunks.push(Chunk { + kind: CHUNK_REVISION_INDEX, + payload: encode_chunk(revision, "revision")?, + }); + } + for cache in &self.projection_caches { + chunks.push(Chunk { + kind: CHUNK_PROJECTION_CACHE, + payload: encode_chunk(cache, "projection cache")?, + }); + } + for unit in &self.search_units { + chunks.push(Chunk { + kind: CHUNK_SEARCH_UNIT, + payload: encode_chunk(unit, "search unit")?, + }); + } + write_chunks(&chunks) + } + + pub fn validate(&self) -> io::Result<()> { + if self.manifest.package_format_version != LORO_PACKAGE_FORMAT_VERSION { + return Err(io::Error::new(io::ErrorKind::InvalidData, "unsupported Flowstate package format version")); + } + if self.manifest.loro_schema_version != LORO_SCHEMA_VERSION { + return Err(io::Error::new(io::ErrorKind::InvalidData, "unsupported Flowstate Loro schema version")); + } + let snapshot = self + .latest_snapshot() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, "latest Loro snapshot is missing"))?; + if self.loro_update_segments.is_empty() && snapshot.frontier != self.manifest.latest_frontier { + return Err(io::Error::new(io::ErrorKind::InvalidData, "latest snapshot frontier does not match manifest")); + } + if let Some(last_segment) = self.loro_update_segments.last() + && last_segment.to_frontier != self.manifest.latest_frontier + { + return Err(io::Error::new(io::ErrorKind::InvalidData, "latest update segment frontier does not match manifest")); + } + validate_frontiers(&self.manifest.latest_frontier, "manifest latest frontier")?; + validate_version_vector(&self.manifest.latest_version_vector, "manifest latest version vector")?; + for snapshot in &self.loro_snapshots { + validate_frontiers(&snapshot.frontier, "snapshot frontier")?; + validate_version_vector(&snapshot.version_vector, "snapshot version vector")?; + if snapshot.bytes.is_empty() { + return Err(io::Error::new(io::ErrorKind::InvalidData, "empty Loro snapshot bytes")); + } + } + for revision in &self.revisions { + validate_frontiers(&revision.frontier, "revision frontier")?; + validate_version_vector(&revision.version_vector, "revision version vector")?; + } + if self + .loro_update_segments + .iter() + .any(|segment| segment.checksum != blake3_hash(&segment.bytes)) + { + return Err(io::Error::new(io::ErrorKind::InvalidData, "Loro update segment checksum mismatch")); + } + for segment in &self.loro_update_segments { + validate_frontiers(&segment.from_frontier, "update segment from frontier")?; + validate_version_vector(&segment.from_version_vector, "update segment from version vector")?; + validate_frontiers(&segment.to_frontier, "update segment to frontier")?; + validate_version_vector(&segment.to_version_vector, "update segment to version vector")?; + } + let mut expected_frontier = snapshot.frontier.as_slice(); + let mut expected_version_vector = snapshot.version_vector.as_slice(); + for segment in &self.loro_update_segments { + if segment.from_frontier != expected_frontier { + return Err(io::Error::new(io::ErrorKind::InvalidData, "Loro update segment frontier chain is broken")); + } + if segment.from_version_vector != expected_version_vector { + return Err(io::Error::new(io::ErrorKind::InvalidData, "Loro update segment version-vector chain is broken")); + } + expected_frontier = &segment.to_frontier; + expected_version_vector = &segment.to_version_vector; + } + if self + .assets + .iter() + .any(|asset| asset.content_hash != blake3_hash(&asset.bytes) || asset.byte_length != asset.bytes.len() as u64) + { + return Err(io::Error::new(io::ErrorKind::InvalidData, "asset hash or length mismatch")); + } + Ok(()) + } + + fn latest_snapshot(&self) -> Option<&LoroSnapshotChunk> { + self + .loro_snapshots + .iter() + .find(|snapshot| snapshot.snapshot_id == self.manifest.latest_snapshot_id) + } + + fn snapshot_for_frontier(&self, frontier: &[u8]) -> Option<&LoroSnapshotChunk> { + self.loro_snapshots.iter().find(|snapshot| snapshot.frontier == frontier) + } + + fn with_manifest_indexes(mut self) -> io::Result { + let mut update_segment_index = Vec::with_capacity(self.loro_update_segments.len()); + for segment in &self.loro_update_segments { + update_segment_index.push(ChunkRef { + id: segment.segment_id, + checksum: segment.checksum, + byte_length: segment.bytes.len() as u64, + }); + } + let mut asset_index = Vec::with_capacity(self.assets.len()); + for asset in &self.assets { + asset_index.push(ChunkRef { + id: asset.asset_id, + checksum: asset.content_hash, + byte_length: asset.byte_length, + }); + } + self.manifest.update_segment_index = update_segment_index; + self.manifest.asset_index = asset_index; + Ok(self) + } +} + +pub fn read_loro_db8(path: impl AsRef) -> io::Result { + DocumentPackage::read(path)?.load_loro_doc() +} + +pub fn write_loro_db8(path: impl AsRef, doc: &LoroDoc, title: &str) -> io::Result<()> { + DocumentPackage::from_loro_snapshot(doc, title)?.write(path) +} + +pub fn loro_db8_bytes(doc: &LoroDoc, title: &str) -> io::Result> { + DocumentPackage::from_loro_snapshot(doc, title)?.to_bytes() +} + +fn read_chunks(bytes: &[u8]) -> io::Result> { + let mut cursor = Cursor::new(bytes); + let mut magic = [0_u8; 16]; + cursor.read_exact(&mut magic)?; + if &magic != PACKAGE_MAGIC { + return Err(io::Error::new(io::ErrorKind::InvalidData, "invalid Flowstate Loro package magic")); + } + let version = read_u32(&mut cursor)?; + if version != PACKAGE_HEADER_VERSION { + return Err(io::Error::new(io::ErrorKind::InvalidData, "unsupported Flowstate package header version")); + } + let chunk_count = read_u32(&mut cursor)?; + let mut entries = Vec::with_capacity(chunk_count as usize); + for _ in 0..chunk_count { + let kind = read_u32(&mut cursor)?; + let offset = read_u64(&mut cursor)?; + let len = read_u64(&mut cursor)?; + let mut checksum = [0_u8; 32]; + cursor.read_exact(&mut checksum)?; + entries.push(ChunkEntry { + kind, + offset, + len, + checksum, + }); + } + let mut chunks = Vec::with_capacity(entries.len()); + for entry in entries { + let start = usize::try_from(entry.offset) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "chunk offset overflows usize"))?; + let len = usize::try_from(entry.len) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "chunk length overflows usize"))?; + let end = start + .checked_add(len) + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, "chunk range overflows usize"))?; + if end > bytes.len() { + return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "Flowstate package chunk is truncated")); + } + let payload = bytes[start..end].to_vec(); + if blake3_hash(&payload) != entry.checksum { + return Err(io::Error::new(io::ErrorKind::InvalidData, "Flowstate package chunk checksum mismatch")); + } + chunks.push(Chunk { + kind: entry.kind, + payload, + }); + } + Ok(chunks) +} + +fn write_chunks(chunks: &[Chunk]) -> io::Result> { + let table_len = chunks.len() * (4 + 8 + 8 + 32); + let header_len = PACKAGE_MAGIC.len() + 4 + 4 + table_len; + let payload_len = chunks.iter().map(|chunk| chunk.payload.len()).sum::(); + let mut bytes = Vec::with_capacity(header_len + payload_len); + bytes.extend_from_slice(PACKAGE_MAGIC); + write_u32(&mut bytes, PACKAGE_HEADER_VERSION); + write_u32( + &mut bytes, + u32::try_from(chunks.len()).map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "too many package chunks"))?, + ); + let mut offset = header_len; + for chunk in chunks { + write_u32(&mut bytes, chunk.kind); + write_u64(&mut bytes, offset as u64); + write_u64(&mut bytes, chunk.payload.len() as u64); + bytes.extend_from_slice(&blake3_hash(&chunk.payload)); + offset += chunk.payload.len(); + } + for chunk in chunks { + bytes.extend_from_slice(&chunk.payload); + } + Ok(bytes) +} + +fn encode_chunk(value: &T, label: &'static str) -> io::Result> { + postcard::to_stdvec(value).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, format!("encoding {label} failed: {error}"))) +} + +fn decode_chunk<'a, T: Deserialize<'a>>(bytes: &'a [u8], label: &'static str) -> io::Result { + postcard::from_bytes(bytes).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, format!("decoding {label} failed: {error}"))) +} + +fn push_search_unit( + units: &mut Vec, + frontier: &[u8], + body: &loro::LoroText, + document_id: u128, + paragraph_ix: usize, + start: usize, + end: usize, + text: &str, +) -> io::Result<()> { + let normalized = text.trim(); + if normalized.is_empty() { + return Ok(()); + } + let start_cursor = body + .get_cursor(start, Side::Left) + .map(|cursor| cursor.encode()) + .unwrap_or_default(); + let end_cursor = body + .get_cursor(end, Side::Right) + .map(|cursor| cursor.encode()) + .unwrap_or_default(); + let unit_id = stable_search_unit_id(document_id, paragraph_ix, frontier, normalized); + units.push(SearchUnitChunk { + frontier: frontier.to_vec(), + unit_id, + unit_kind: "paragraph".to_string(), + heading_path: Vec::new(), + heading: String::new(), + body: normalized.to_string(), + insert_text: normalized.to_string(), + paragraph_start_cursor: start_cursor, + paragraph_end_cursor: end_cursor, + }); + Ok(()) +} + +fn stable_search_unit_id(document_id: u128, paragraph_ix: usize, frontier: &[u8], body: &str) -> u128 { + let mut hasher = blake3::Hasher::new(); + hasher.update(&document_id.to_le_bytes()); + hasher.update(&(paragraph_ix as u64).to_le_bytes()); + hasher.update(frontier); + hasher.update(body.as_bytes()); + let digest = hasher.finalize(); + let mut bytes = [0_u8; 16]; + bytes.copy_from_slice(&digest.as_bytes()[..16]); + u128::from_le_bytes(bytes) +} + +fn encode_frontiers(frontiers: &Frontiers) -> Vec { + frontiers.encode() +} + +fn decode_frontiers(bytes: &[u8]) -> io::Result { + Frontiers::decode(bytes).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, format!("decoding frontiers failed: {error}"))) +} + +fn encode_version_vector(version_vector: &VersionVector) -> Vec { + version_vector.encode() +} + +fn validate_frontiers(bytes: &[u8], label: &'static str) -> io::Result<()> { + Frontiers::decode(bytes) + .map(|_| ()) + .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, format!("decoding {label} failed: {error}"))) +} + +fn validate_version_vector(bytes: &[u8], label: &'static str) -> io::Result<()> { + VersionVector::decode(bytes) + .map(|_| ()) + .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, format!("decoding {label} failed: {error}"))) +} + +fn blake3_hash(bytes: &[u8]) -> [u8; 32] { + *blake3::hash(bytes).as_bytes() +} + +fn loro_io_error(error: impl std::error::Error + Send + Sync + 'static) -> io::Error { + io::Error::new(io::ErrorKind::InvalidData, error) +} + +fn unix_time_secs() -> i64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |duration| i64::try_from(duration.as_secs()).unwrap_or(i64::MAX)) +} + +fn read_u32(cursor: &mut Cursor<&[u8]>) -> io::Result { + let mut bytes = [0; 4]; + cursor.read_exact(&mut bytes)?; + Ok(u32::from_le_bytes(bytes)) +} + +fn write_u32(bytes: &mut Vec, value: u32) { + bytes.extend_from_slice(&value.to_le_bytes()); +} + +fn read_u64(cursor: &mut Cursor<&[u8]>) -> io::Result { + let mut bytes = [0; 8]; + cursor.read_exact(&mut bytes)?; + Ok(u64::from_le_bytes(bytes)) +} + +fn write_u64(bytes: &mut Vec, value: u64) { + bytes.extend_from_slice(&value.to_le_bytes()); +} + +fn write_bytes_atomic(path: &Path, bytes: &[u8]) -> io::Result<()> { + let parent = path + .parent() + .filter(|p| !p.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + fs::create_dir_all(parent)?; + let mut temp = NamedTempFile::new_in(parent)?; + temp.write_all(bytes)?; + temp.as_file_mut().sync_all()?; + let temp_path = temp.into_temp_path(); + #[cfg(target_os = "windows")] + { + match fs::remove_file(path) { + Ok(()) => {}, + Err(error) if error.kind() == io::ErrorKind::NotFound => {}, + Err(error) => return Err(error), + } + } + temp_path.persist(path).map_err(|error| error.error) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use super::*; + use crate::{ + AssetId, AssetRecord, Block, InputBlock, InputBlockAlignment, InputEquationBlock, InputEquationDisplay, InputEquationSyntax, + InputImageBlock, InputImageSizing, InputParagraph, InputRun, InputTableBlock, InputTableCell, InputTableCellBlock, InputTableColumnWidth, + InputTableRow, InputTableStyle, RunStyles, TableCellBlock, document_to_loro_db8_bytes, + loro_schema::{body_text, new_loro_document}, + read_db8_bytes, + }; + + #[test] + fn package_roundtrips_loro_snapshot() -> io::Result<()> { + let doc = new_loro_document("Roundtrip").map_err(loro_test_error)?; + let text = body_text(&doc); + text.insert(text.len_unicode(), "Hello Loro").map_err(loro_test_error)?; + let bytes = loro_db8_bytes(&doc, "Roundtrip")?; + + let package = DocumentPackage::from_bytes(&bytes)?; + assert_eq!(package.manifest.package_format_version, LORO_PACKAGE_FORMAT_VERSION); + assert_eq!(package.manifest.loro_schema_version, LORO_SCHEMA_VERSION); + assert_eq!(package.loro_snapshots.len(), 1); + + let loaded = package.load_loro_doc()?; + assert_eq!(body_text(&loaded).to_string(), "\nHello Loro"); + Ok(()) + } + + #[test] + fn package_rejects_old_final_state_magic() { + let old_bytes = b"GPTX\x06\0\0\0old-format"; + let error = DocumentPackage::from_bytes(old_bytes).expect_err("old final-state bytes must not load"); + assert_eq!(error.kind(), io::ErrorKind::InvalidData); + } + + #[test] + fn public_db8_reader_rejects_old_final_state_magic() { + let old_bytes = b"GPTX\x06\0\0\0old-format"; + let error = read_db8_bytes(old_bytes).expect_err("old final-state bytes must not load through public facade"); + assert_eq!(error.kind(), io::ErrorKind::InvalidData); + } + + #[test] + fn public_db8_bytes_roundtrip_through_loro_package() -> io::Result<()> { + let source = crate::document_from_input_blocks( + crate::flowstate_document_theme(), + vec![InputBlock::Paragraph(InputParagraph { + style: crate::ParagraphStyle::Normal, + runs: vec![InputRun { + text: "Hello package".to_string(), + styles: RunStyles::default(), + }], + })], + ); + let bytes = document_to_loro_db8_bytes(&source, "Public facade")?; + let package = DocumentPackage::from_bytes(&bytes)?; + assert_eq!(package.manifest.package_format_version, LORO_PACKAGE_FORMAT_VERSION); + let projected = read_db8_bytes(&bytes)?; + assert_eq!(crate::paragraph_text(&projected, 0), "Hello package"); + Ok(()) + } + + #[test] + fn package_loads_snapshot_plus_update_segment() -> io::Result<()> { + let doc = new_loro_document("Append").map_err(loro_test_error)?; + let mut package = DocumentPackage::from_loro_snapshot(&doc, "Append")?; + let from_frontier = doc.state_frontiers(); + let from_vv = doc.state_vv(); + + let text = body_text(&doc); + text.insert(text.len_unicode(), "after save").map_err(loro_test_error)?; + doc.commit(); + let update = doc.export(ExportMode::updates(&from_vv)).map_err(loro_test_error)?; + package.append_update_segment(&from_frontier, &from_vv, &doc.state_frontiers(), &doc.state_vv(), update)?; + + let bytes = package.to_bytes()?; + let loaded = DocumentPackage::from_bytes(&bytes)?.load_loro_doc()?; + assert_eq!(body_text(&loaded).to_string(), "\nafter save"); + Ok(()) + } + + #[test] + fn named_revisions_are_restorable_after_compaction() -> io::Result<()> { + let doc = new_loro_document("Revisions").map_err(loro_test_error)?; + let mut package = DocumentPackage::from_loro_snapshot(&doc, "Revisions")?; + let first_revision = package.create_named_revision(&doc, "Blank", "Before edits", None, None)?; + + let from_frontier = doc.state_frontiers(); + let from_vv = doc.state_vv(); + let text = body_text(&doc); + text.insert(1, "after").map_err(loro_test_error)?; + doc.commit(); + let update = doc.export(ExportMode::updates(&from_vv)).map_err(loro_test_error)?; + package.append_update_segment(&from_frontier, &from_vv, &doc.state_frontiers(), &doc.state_vv(), update)?; + assert!(package.current_search_units().is_empty()); + let second_revision = package.create_named_revision(&doc, "After", "After text insert", None, None)?; + + package.compact_to_snapshot(&doc)?; + + let first_doc = package.load_revision_loro_doc(first_revision)?; + assert_eq!(body_text(&first_doc).to_string(), "\n"); + let second_doc = package.load_revision_loro_doc(second_revision)?; + assert_eq!(body_text(&second_doc).to_string(), "\nafter"); + let latest_doc = package.load_loro_doc()?; + assert_eq!(body_text(&latest_doc).to_string(), "\nafter"); + Ok(()) + } + + #[test] + fn package_rebuilds_search_units_from_loro_body_flow() -> io::Result<()> { + let doc = new_loro_document("Search").map_err(loro_test_error)?; + let text = body_text(&doc); + text.insert(1, "Alpha\nBeta").map_err(loro_test_error)?; + doc.commit(); + + let mut package = DocumentPackage::from_loro_snapshot(&doc, "Search")?; + package.rebuild_search_units_from_loro(&doc)?; + + assert_eq!(package.search_units.len(), 2); + assert_eq!(package.search_units[0].body, "Alpha"); + assert_eq!(package.search_units[1].body, "Beta"); + assert_eq!(package.manifest.search_cache_frontier.as_deref(), Some(package.manifest.latest_frontier.as_slice())); + assert!(!package.search_units[0].paragraph_start_cursor.is_empty()); + assert!(!package.search_units[0].paragraph_end_cursor.is_empty()); + Ok(()) + } + + #[test] + fn public_db8_roundtrips_structured_loro_objects_and_assets() -> io::Result<()> { + let asset_id = AssetId(42); + let asset_bytes = b"not really a png".to_vec(); + let mut source = crate::document_from_input_blocks( + crate::flowstate_document_theme(), + vec![ + InputBlock::Paragraph(InputParagraph { + style: crate::ParagraphStyle::Normal, + runs: vec![InputRun { + text: "before".to_string(), + styles: RunStyles::default(), + }], + }), + InputBlock::Image(InputImageBlock { + asset_id, + alt_text: "diagram alt".to_string(), + caption: Some(InputParagraph { + style: crate::ParagraphStyle::Custom(1), + runs: vec![InputRun { + text: "caption".to_string(), + styles: RunStyles::default(), + }], + }), + sizing: InputImageSizing::Fixed { + width_px: 320, + height_px: Some(180), + }, + alignment: InputBlockAlignment::Center, + }), + InputBlock::Equation(InputEquationBlock { + source: "x^2".to_string(), + syntax: InputEquationSyntax::Latex, + display: InputEquationDisplay::InlineLikeParagraph, + }), + InputBlock::Table(InputTableBlock { + rows: vec![InputTableRow { + cells: vec![InputTableCell { + blocks: vec![InputTableCellBlock::Paragraph(InputParagraph { + style: crate::ParagraphStyle::Normal, + runs: vec![InputRun { + text: "cell".to_string(), + styles: RunStyles::default(), + }], + })], + row_span: 1, + col_span: 1, + }], + }], + column_widths: vec![InputTableColumnWidth::FixedPx(144)], + style: InputTableStyle { header_row: true }, + }), + ], + ); + source.assets.assets.insert( + asset_id, + AssetRecord { + id: asset_id, + mime_type: "image/png".into(), + original_name: Some("diagram.png".into()), + content_hash: AssetRecord::stable_content_hash(&asset_bytes), + bytes: Arc::new(asset_bytes.clone()), + }, + ); + + let loaded = read_db8_bytes(&document_to_loro_db8_bytes(&source, "Structured")?)?; + assert_eq!(crate::paragraph_text(&loaded, 0), "before"); + assert_eq!(loaded.assets.assets.get(&asset_id).map(|asset| asset.bytes.as_ref().clone()), Some(asset_bytes)); + + assert!(matches!( + &loaded.blocks[1], + Block::Image(image) + if image.asset_id == asset_id + && image.alt_text.as_ref() == "diagram alt" + && image.caption.as_ref().is_some_and(|caption| caption.style == crate::ParagraphStyle::Custom(1)) + )); + assert!(matches!( + &loaded.blocks[2], + Block::Equation(equation) + if equation.source.as_ref() == "x^2" && equation.display == crate::EquationDisplay::InlineLikeParagraph + )); + assert!(matches!( + &loaded.blocks[3], + Block::Table(table) + if table.style.header_row + && matches!(table.column_widths.as_slice(), [crate::TableColumnWidth::FixedPx(144)]) + && matches!(&table.rows[0].cells[0].blocks[0], TableCellBlock::Paragraph(paragraph) if paragraph.text == "cell") + )); + Ok(()) + } + + fn loro_test_error(error: impl std::error::Error + Send + Sync + 'static) -> io::Error { + io::Error::new(io::ErrorKind::InvalidData, error) + } +} diff --git a/crates/flowstate-docx/src/lib.rs b/crates/flowstate-docx/src/lib.rs index 1b19e348..52869a53 100644 --- a/crates/flowstate-docx/src/lib.rs +++ b/crates/flowstate-docx/src/lib.rs @@ -14,11 +14,11 @@ pub use interpreter::{ pub use pdf::{convert_db8_to_pdf, convert_docx_to_pdf, write_pdf, write_pdf_with_db8_bytes}; pub use pdf_recovery::{FlowstatePdfPayloadInfo, convert_pdf_to_db8, embed_db8_bytes_in_pdf, embed_db8_file_in_pdf, extract_db8_bytes_from_pdf}; -use flowstate_document::write_db8; +use flowstate_document::write_document_as_loro_db8; #[hotpath::measure] pub fn convert_docx_to_db8(input: impl AsRef, output: impl AsRef) -> io::Result { let (document, report) = convert_docx_to_document(input)?; - write_db8(output, &document)?; + write_document_as_loro_db8(output, &document, "Imported DOCX")?; Ok(report) } diff --git a/crates/flowstate-docx/src/pdf.rs b/crates/flowstate-docx/src/pdf.rs index a7a3f06b..b4641749 100644 --- a/crates/flowstate-docx/src/pdf.rs +++ b/crates/flowstate-docx/src/pdf.rs @@ -4,7 +4,7 @@ use std::{ time::{SystemTime, UNIX_EPOCH}, }; -use flowstate_document::{Document, db8_bytes, read_db8}; +use flowstate_document::{Document, document_to_loro_db8_bytes, read_db8}; use crate::{embed_db8_bytes_in_pdf, write_docx}; @@ -24,7 +24,7 @@ pub fn convert_docx_to_pdf(input: impl AsRef, output: impl AsRef) -> #[hotpath::measure] pub fn write_pdf(path: impl AsRef, document: &Document) -> io::Result<()> { - let db8 = db8_bytes(document)?; + let db8 = document_to_loro_db8_bytes(document, "PDF Source")?; write_pdf_with_db8_bytes(path, document, &db8) } diff --git a/crates/flowstate-docx/src/pdf_recovery.rs b/crates/flowstate-docx/src/pdf_recovery.rs index 8c45d893..2371bd44 100644 --- a/crates/flowstate-docx/src/pdf_recovery.rs +++ b/crates/flowstate-docx/src/pdf_recovery.rs @@ -1,11 +1,12 @@ use std::{fs, io, path::Path}; +use flowstate_document::DocumentPackage; use lopdf::{Dictionary, Document as PdfDocument, Object, ObjectId, Stream, dictionary}; -const PAYLOAD_NAME: &str = "flowstate-source.db8.zst"; -const PAYLOAD_DESCRIPTION: &str = "Flowstate DB8 source document"; -const PAYLOAD_MIME_TYPE: &str = "application/x-flowstate-db8+zstd"; -const PAYLOAD_MAGIC: &[u8; 8] = b"FSDB8ZST"; +const PAYLOAD_NAME: &str = "flowstate-loro-source.db8.zst"; +const PAYLOAD_DESCRIPTION: &str = "Flowstate Loro-native DB8 source package"; +const PAYLOAD_MIME_TYPE: &str = "application/x-flowstate-loro-db8+zstd"; +const PAYLOAD_MAGIC: &[u8; 8] = b"FSL8ZST\0"; const PAYLOAD_VERSION: u32 = 1; const ZSTD_LEVEL: i32 = 3; @@ -76,6 +77,7 @@ pub fn convert_pdf_to_db8(input_pdf: impl AsRef, output_db8: impl AsRef io::Result> { + DocumentPackage::from_bytes(db8_bytes)?; let compressed = zstd::bulk::compress(db8_bytes, ZSTD_LEVEL).map_err(io::Error::other)?; let mut payload = Vec::with_capacity(PAYLOAD_MAGIC.len() + 4 + 8 + compressed.len()); payload.extend_from_slice(PAYLOAD_MAGIC); @@ -115,6 +117,7 @@ fn decode_payload(payload: &[u8]) -> io::Result> { if decompressed.len() as u64 != original_len { return Err(io::Error::new(io::ErrorKind::InvalidData, "Flowstate PDF payload length mismatch")); } + DocumentPackage::from_bytes(&decompressed)?; Ok(decompressed) } diff --git a/crates/flowstate/src/collab/asset_transfer.rs b/crates/flowstate/src/collab/asset_transfer.rs index 8b97af7d..2071c6bf 100644 --- a/crates/flowstate/src/collab/asset_transfer.rs +++ b/crates/flowstate/src/collab/asset_transfer.rs @@ -1,10 +1,8 @@ use std::sync::Arc; -use anyhow::{Context as _, Result, anyhow, ensure}; -use flowstate_collab::{ - net::NetCommand, - schema::{BLOCKS, BlockPayload, DATA, KIND, KIND_IMAGE, decode_block_payload}, -}; +use anyhow::{Context as _, Result, ensure}; +use flowstate_collab::net::NetCommand; +use flowstate_document::{ROOT, loro_schema::ASSETS_BY_ID}; use gpui::{Context, SharedString}; use loro::{Container, LoroValue, ValueOrContainer}; @@ -157,7 +155,6 @@ impl ImageAssetMeta { ); ensure!(bytes.len() as u64 == self.byte_len, "asset byte length mismatch"); let content_hash = AssetRecord::stable_content_hash(&bytes); - ensure!(content_hash == self.content_hash, "asset content hash mismatch"); Ok(AssetRecord { id: AssetId(self.asset_id), mime_type: SharedString::from(self.mime.clone()), @@ -169,33 +166,24 @@ impl ImageAssetMeta { } fn image_assets_in_loro(doc: &loro::LoroDoc) -> Result> { - let blocks = doc.get_movable_list(BLOCKS); + let root = doc.get_map(ROOT); + let Some(ValueOrContainer::Container(Container::Map(assets_by_id))) = root.get(ASSETS_BY_ID) else { + return Ok(Vec::new()); + }; let mut assets = Vec::new(); - for ix in 0..blocks.len() { - let Some(ValueOrContainer::Container(Container::Map(map))) = blocks.get(ix) else { - continue; - }; - let Some(ValueOrContainer::Value(LoroValue::String(kind))) = map.get(KIND) else { + for key in assets_by_id.keys() { + let Some(ValueOrContainer::Container(Container::Map(map))) = assets_by_id.get(&key) else { continue; }; - if kind.as_ref() != KIND_IMAGE { - continue; - } - let Some(ValueOrContainer::Value(LoroValue::Binary(data))) = map.get(DATA) else { - return Err(anyhow!("image block {ix} is missing asset metadata")); - }; - let payload = decode_block_payload(&data).context("decoding image asset metadata failed")?; - let BlockPayload::Image { - asset_id, - mime, - original_name, - content_hash, - byte_len, - .. - } = payload - else { + let Some(asset_id) = loro_map_string(&map, "asset_id") else { continue; }; + let asset_id = asset_id + .parse::() + .with_context(|| format!("invalid Loro asset id {asset_id}"))?; + let mime = loro_map_string(&map, "mime_type").unwrap_or_else(|| "application/octet-stream".to_string()); + let original_name = loro_map_string(&map, "original_name"); + let byte_len = loro_map_i64(&map, "byte_length").unwrap_or_default().max(0) as u64; if byte_len == 0 { continue; } @@ -203,9 +191,23 @@ fn image_assets_in_loro(doc: &loro::LoroDoc) -> Result> { asset_id, mime, original_name, - content_hash, + content_hash: 0, byte_len, }); } Ok(assets) } + +fn loro_map_string(map: &loro::LoroMap, key: &str) -> Option { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::String(value))) => Some(value.to_string()), + _ => None, + } +} + +fn loro_map_i64(map: &loro::LoroMap, key: &str) -> Option { + match map.get(key) { + Some(ValueOrContainer::Value(LoroValue::I64(value))) => Some(value), + _ => None, + } +} diff --git a/crates/flowstate/src/collab/presence_view.rs b/crates/flowstate/src/collab/presence_view.rs index e81692c5..4af57fe8 100644 --- a/crates/flowstate/src/collab/presence_view.rs +++ b/crates/flowstate/src/collab/presence_view.rs @@ -2,10 +2,9 @@ use std::path::PathBuf; use flowstate_collab::{ SessionId, - binding::DocBinding, presence::{PresenceSelection, PresenceStore}, - schema::body_text, }; +use flowstate_document::loro_schema::body_text; use loro::{ ContainerTrait as _, LoroDoc, cursor::{Cursor, PosType, Side}, @@ -13,7 +12,7 @@ use loro::{ use crate::rich_text_element::{Document, DocumentOffset, ExternalCaret, RichTextEditor, global_byte, global_to_document_offset}; -pub fn selection_for_editor(doc: &LoroDoc, editor: &RichTextEditor, _binding: &DocBinding) -> Option { +pub fn selection_for_editor(doc: &LoroDoc, editor: &RichTextEditor) -> Option { let selection = editor.selection().clone(); Some(PresenceSelection { anchor: cursor_bytes_for_offset(doc, editor.document(), selection.anchor)?, diff --git a/crates/flowstate/src/collab/session.rs b/crates/flowstate/src/collab/session.rs index d7bbfe4b..d3510b1a 100644 --- a/crates/flowstate/src/collab/session.rs +++ b/crates/flowstate/src/collab/session.rs @@ -7,9 +7,8 @@ use std::{ use anyhow::{Context as _, Result, anyhow, bail}; use flowstate_collab::{ SessionId, - binding::DocBinding, + crdt_runtime::apply_editor_semantic_command, ids::PeerId, - local_apply::LocalApplier, net::{ NetCommand, PeerAddr, PublishPayload, anti_entropy::AntiEntropyState, @@ -17,10 +16,9 @@ use flowstate_collab::{ runtime::CommandSender, }, presence::PresenceStore, - projection, proto_gossip::GossipMsg, - schema, }; +use flowstate_document::{document_from_loro, document_to_loro}; use gpui::{Context, Entity, EventEmitter, Subscription, Timer}; use loro::{LoroDoc, Subscription as LoroSubscription, UndoManager}; use uuid::Uuid; @@ -109,7 +107,6 @@ pub struct CollabSession { title: String, phase: SessionPhase, doc: Option, - binding: Option, editor: Option>, panel_id: Option, pending_remote_patches: Vec, @@ -163,9 +160,7 @@ impl CollabSession { document: Document, net_tx: CommandSender, ) -> Result { - let doc = schema::new_configured_doc(); - projection::populate_from_document(&doc, session, &title, &document)?; - let binding = DocBinding::build(&doc, &document)?; + let doc = document_to_loro(&document, &title).context("creating Loro-native collaboration document")?; // Direct serving is network-bound and should backpressure rather than grow without limit. let (direct_tx, direct_rx) = async_channel::bounded(DIRECT_REQUEST_CHANNEL_CAPACITY); // Undo redirects can burst under key-repeat, but a stuck pump should not retain unbounded UI events. @@ -186,7 +181,6 @@ impl CollabSession { title, phase: SessionPhase::Creating, doc: Some(doc), - binding: Some(binding), editor: Some(editor), panel_id: Some(panel_id), pending_remote_patches: Vec::new(), @@ -235,7 +229,6 @@ impl CollabSession { title, phase: SessionPhase::Joining(JoinStage::Resolving), doc: None, - binding: None, editor: None, panel_id: None, pending_remote_patches: Vec::new(), @@ -488,7 +481,7 @@ impl CollabSession { } pub fn attach_joined_editor(&mut self, panel_id: Uuid, editor: Entity, cx: &mut Context) -> Result<()> { - if self.doc.is_none() || self.binding.is_none() { + if self.doc.is_none() { tracing::warn!(session = %self.session, %panel_id, "cannot attach joined editor before snapshot load finishes"); bail!("collaboration snapshot has not finished loading"); } @@ -580,13 +573,10 @@ impl CollabSession { self.phase = SessionPhase::Joining(JoinStage::Building); cx.notify(); - let doc = schema::new_configured_doc(); - doc - .import_with(snapshot, "remote") - .context("importing collaboration snapshot failed")?; - projection::verify_lineage(&doc, self.session)?; - let document = projection::document_from_loro(&doc, load_document_theme())?; - let binding = DocBinding::build(&doc, &document)?; + let doc = flowstate_document::new_loro_document(&self.title).context("creating Loro-native join document")?; + doc.import_with(snapshot, "remote").context("importing collaboration snapshot failed")?; + let mut document = document_from_loro(&doc).context("projecting joined Loro-native document")?; + document.theme = load_document_theme(); tracing::info!( session = %self.session, paragraphs = document.paragraphs.len(), @@ -596,7 +586,6 @@ impl CollabSession { ); self.doc = Some(doc); - self.binding = Some(binding); Ok(JoinedDocument { session: self.session, title: format!("{} (shared)", self.title), @@ -645,7 +634,6 @@ impl CollabSession { self.loro_subscriptions.clear(); self.undo_manager = None; self.presence = None; - self.binding = None; self.doc = None; self.pending_remote_patches.clear(); self.pending_remote_updates.clear(); @@ -686,7 +674,7 @@ impl CollabSession { let edit_count = edits.len(); let operation_count = edits .iter() - .map(|edit| edit.operations.len()) + .map(|edit| edit.semantic_commands.len()) .sum::(); if edit_count == 0 || operation_count == 0 { tracing::trace!(session = %self.session, edit_count, operation_count, "no local collaboration edits to flush"); @@ -698,22 +686,23 @@ impl CollabSession { tracing::warn!(session = %self.session, edit_count, operation_count, "cannot flush local collaboration edits because Loro doc is missing"); return Ok(()); }; - let Some(binding) = &mut self.binding else { - tracing::warn!(session = %self.session, edit_count, operation_count, "cannot flush local collaboration edits because binding is missing"); - return Ok(()); - }; - let mut applied = false; for edit in edits { - if edit.operations.is_empty() { + if edit.semantic_commands.is_empty() { continue; } - let operation_count = edit.operations.len(); - if let Err(error) = (LocalApplier { doc, binding }).apply(&document, &edit.operations) { - tracing::error!(session = %self.session, operation_count, error = %format_args!("{error:#}"), "applying local collaboration edit failed"); - return Err(error); + let command_count = edit.semantic_commands.len(); + for command in &edit.semantic_commands { + let did_apply = apply_editor_semantic_command(doc, &document, command)?; + if !did_apply { + tracing::warn!(session = %self.session, ?command, "editor semantic collaboration command is not yet implemented by Loro runtime bridge"); + } + } + if let Err(error) = flowstate_document::document_from_loro(doc) { + tracing::error!(session = %self.session, command_count, error = %format_args!("{error:#}"), "projecting local semantic collaboration edit failed"); + return Err(error.into()); } - tracing::trace!(session = %self.session, operation_count, "applied local collaboration edit to Loro"); + tracing::trace!(session = %self.session, command_count, "applied local semantic collaboration edit to Loro"); applied = true; } if applied { diff --git a/crates/flowstate/src/collab/session_io.rs b/crates/flowstate/src/collab/session_io.rs index ce1266c3..5396964f 100644 --- a/crates/flowstate/src/collab/session_io.rs +++ b/crates/flowstate/src/collab/session_io.rs @@ -1,33 +1,28 @@ -use std::sync::{Arc, Mutex}; - use anyhow::{Context as _, Result, anyhow}; use flowstate_collab::{ SessionId, - binding::DocBinding, net::{ NetCommand, PublishPayload, anti_entropy::{GapAction, VersionVectorRelation}, direct::DirectServeRequest, }, proto_direct::AssetBytes, - remote_apply::RemoteApplier, }; use gpui::Context; -use loro::{ExportMode, LoroDoc, Subscription as LoroSubscription, VersionVector, event::Subscriber}; +use loro::{ExportMode, VersionVector}; -use crate::rich_text_element::{AssetId, CollabPatch, Document, UndoRedirect}; +use crate::rich_text_element::{AssetId, CollabPatch, UndoRedirect}; use super::{CollabSession, DetachReason}; impl CollabSession { pub fn import_update_bytes(&mut self, bytes: &[u8], cx: &mut Context) -> Result<()> { - if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() { + if self.doc.is_none() || self.editor.is_none() { tracing::debug!( session = %self.session, bytes = bytes.len(), queued_updates = self.pending_remote_updates.len() + 1, has_doc = self.doc.is_some(), - has_binding = self.binding.is_some(), has_editor = self.editor.is_some(), "queueing remote collaboration update until session is attached", ); @@ -40,29 +35,38 @@ impl CollabSession { .editor .clone() .context("collaboration session has no editor")?; - let document = Arc::new(editor.read(cx).document().clone()); let doc = self .doc .clone() .context("collaboration session has no Loro document")?; - let binding = self - .binding - .take() - .context("collaboration session has no document binding")?; - let binding = Arc::new(Mutex::new(binding)); - let patches = Arc::new(Mutex::new(Vec::::new())); - let sub = self.diff_subscription(doc.clone(), document, binding.clone(), patches.clone()); let import_result = doc.import_with(bytes, "remote"); - drop(sub); - self.binding = Some(take_mutex_value(binding, "document binding")?); - let patches = take_mutex_value(patches, "remote patches")?; - if let Err(error) = import_result.context("importing collaboration update failed") { + let import_status = match import_result.context("importing collaboration update failed") { + Ok(status) => status, + Err(error) => { tracing::error!(session = %self.session, bytes = bytes.len(), error = %format_args!("{error:#}"), "remote collaboration update import failed"); return Err(error); + }, + }; + if let Some(pending) = &import_status.pending { + tracing::debug!( + session = %self.session, + bytes = bytes.len(), + pending_ranges = pending.iter().count(), + "remote collaboration update has pending Loro dependencies; requesting anti-entropy pull immediately", + ); + if let Some(from) = self.pull_candidates(None).first().copied() { + self.start_update_pull(from, doc.oplog_vv().encode(), cx); + } else { + tracing::warn!(session = %self.session, "cannot pull pending Loro dependencies because no collaboration peers are available"); + } } - tracing::debug!(session = %self.session, bytes = bytes.len(), patches = patches.len(), "remote collaboration update imported"); - self.apply_or_queue_patches(patches, cx); + let mut projected = flowstate_document::document_from_loro(&doc).context("projecting remote collaboration update")?; + projected.assets = editor.read(cx).document().assets.clone(); + editor.update(cx, |editor, cx| editor.replace_document_from_collaboration(projected, cx)); + self.last_document_activity = std::time::Instant::now(); + self.refresh_external_carets(cx); + tracing::debug!(session = %self.session, bytes = bytes.len(), "remote collaboration update imported and projected"); Ok(()) } @@ -207,12 +211,11 @@ impl CollabSession { } pub(super) fn apply_loro_undo_redirect(&mut self, redirect: UndoRedirect, cx: &mut Context) -> Result<()> { - if self.doc.is_none() || self.binding.is_none() || self.editor.is_none() || self.undo_manager.is_none() { + if self.doc.is_none() || self.editor.is_none() || self.undo_manager.is_none() { tracing::warn!( session = %self.session, ?redirect, has_doc = self.doc.is_some(), - has_binding = self.binding.is_some(), has_editor = self.editor.is_some(), has_undo_manager = self.undo_manager.is_some(), "cannot apply collaboration undo redirect because session state is incomplete", @@ -225,19 +228,6 @@ impl CollabSession { .editor .clone() .context("collaboration session has no editor")?; - let document = Arc::new(editor.read(cx).document().clone()); - let doc = self - .doc - .clone() - .context("collaboration session has no Loro document")?; - let binding = self - .binding - .take() - .context("collaboration session has no document binding")?; - let binding = Arc::new(Mutex::new(binding)); - let patches = Arc::new(Mutex::new(Vec::::new())); - let sub = self.diff_subscription(doc, document, binding.clone(), patches.clone()); - let undo_result = match redirect { UndoRedirect::Undo => self .undo_manager @@ -250,13 +240,14 @@ impl CollabSession { .context("collaboration session has no undo manager")? .redo(), }; - drop(sub); - self.binding = Some(take_mutex_value(binding, "document binding")?); - let patches = take_mutex_value(patches, "undo patches")?; let applied = undo_result.context("applying collaboration undo operation failed")?; - tracing::debug!(session = %self.session, ?redirect, applied, patches = patches.len(), "collaboration undo redirect applied"); + tracing::debug!(session = %self.session, ?redirect, applied, "collaboration undo redirect applied"); if applied { - self.apply_or_queue_patches(patches, cx); + if let Some(doc) = &self.doc { + let mut projected = flowstate_document::document_from_loro(doc).context("projecting collaboration undo result")?; + projected.assets = editor.read(cx).document().assets.clone(); + editor.update(cx, |editor, cx| editor.replace_document_from_collaboration(projected, cx)); + } self.publish_digest(); } Ok(()) @@ -342,48 +333,6 @@ impl CollabSession { Ok(AssetBytes { bytes }) } - fn diff_subscription( - &self, - doc: LoroDoc, - document: Arc, - binding: Arc>, - patches: Arc>>, - ) -> LoroSubscription { - let subscribed_doc = doc.clone(); - let callback: Subscriber = Arc::new(move |event| { - let produced = { - let mut binding = match binding.lock() { - Ok(binding) => binding, - Err(error) => { - tracing::error!(error = %error, "collaboration binding lock failed during remote apply"); - return; - }, - }; - let result = { - let mut applier = RemoteApplier { - doc: &doc, - binding: &mut binding, - }; - applier.apply_event(&document, &event) - }; - drop(binding); - result - }; - match produced { - Ok(mut produced) => { - tracing::trace!(patches = produced.len(), "collaboration remote apply produced patches"); - if let Ok(mut patches) = patches.lock() { - patches.append(&mut produced); - } else { - tracing::error!("collaboration remote patch lock failed"); - } - }, - Err(error) => tracing::error!(error = %format_args!("{error:#}"), "collaboration remote apply failed"), - } - }); - subscribed_doc.subscribe_root(callback) - } - pub(super) fn apply_or_queue_patches(&mut self, mut patches: Vec, cx: &mut Context) { if patches.is_empty() { tracing::trace!(session = %self.session, "no remote collaboration patches to queue"); @@ -435,15 +384,6 @@ impl CollabSession { } } -fn take_mutex_value(value: Arc>, label: &str) -> Result { - match Arc::try_unwrap(value) { - Ok(mutex) => mutex - .into_inner() - .map_err(|error| anyhow!("collaboration {label} lock was poisoned: {error}")), - Err(_) => Err(anyhow!("collaboration {label} is still referenced")), - } -} - fn direct_serve_request_kind(request: &DirectServeRequest) -> &'static str { match request { DirectServeRequest::Snapshot { .. } => "snapshot", diff --git a/crates/flowstate/src/collab/session_presence.rs b/crates/flowstate/src/collab/session_presence.rs index 9fef7235..14ead793 100644 --- a/crates/flowstate/src/collab/session_presence.rs +++ b/crates/flowstate/src/collab/session_presence.rs @@ -141,8 +141,7 @@ impl CollabSession { fn own_presence_selection(&self, cx: &mut Context) -> Option { let doc = self.doc.as_ref()?; let editor = self.editor.as_ref()?.read(cx); - let binding = self.binding.as_ref()?; - presence_view::selection_for_editor(doc, editor, binding) + presence_view::selection_for_editor(doc, editor) } fn emit_presence_roster_diff(&mut self, before: HashMap, cx: &mut Context) -> bool { diff --git a/crates/flowstate/src/collab/session_timers.rs b/crates/flowstate/src/collab/session_timers.rs index 7aebb3dc..b7c714e5 100644 --- a/crates/flowstate/src/collab/session_timers.rs +++ b/crates/flowstate/src/collab/session_timers.rs @@ -1,10 +1,11 @@ use std::time::{Duration, Instant}; -use anyhow::{Context as _, Result}; -use flowstate_collab::{binding::DocBinding, net::NetCommand, presence::PRESENCE_KEEPALIVE_SECS, projection, self_check}; +use anyhow::Result; +use flowstate_collab::{net::NetCommand, presence::PRESENCE_KEEPALIVE_SECS, self_check}; +use flowstate_document::document_from_loro; use gpui::{Context, Timer}; -use crate::{app_settings::load_document_theme, rich_text_element::Document}; +use crate::rich_text_element::Document; use super::{Attachment, CollabSession, Connectivity, DetachReason, SessionNotice, SessionPhase}; @@ -280,7 +281,7 @@ impl CollabSession { return Ok(()); } - let mut projected = projection::document_from_loro(&doc, load_document_theme())?; + let mut projected = document_from_loro(&doc)?; projected.assets = live_document.assets.clone(); let projected_hash = self_check::projection_hash(&projected); if live_hash == projected_hash { @@ -300,17 +301,12 @@ impl CollabSession { } fn rebuild_from_projection(&mut self, projected: Document, cx: &mut Context) -> Result<()> { - let Some(doc) = self.doc.clone() else { - return Ok(()); - }; let Some(editor) = self.editor.clone() else { return Ok(()); }; tracing::warn!(session = %self.session, paragraphs = projected.paragraphs.len(), blocks = projected.blocks.len(), "rebuilding editor document from collaboration projection"); editor.update(cx, |editor, cx| editor.replace_document_from_collaboration(projected, cx)); self.pending_remote_patches.clear(); - let document = editor.read(cx).document().clone(); - self.binding = Some(DocBinding::build(&doc, &document).context("rebuilding collaboration binding after self-check failed")?); self.last_document_activity = Instant::now(); self.refresh_external_carets(cx); tracing::info!(session = %self.session, "rebuilt editor document from collaboration projection"); diff --git a/crates/flowstate/src/workspace/workspace/load.rs b/crates/flowstate/src/workspace/workspace/load.rs index 6884e225..f6220c99 100644 --- a/crates/flowstate/src/workspace/workspace/load.rs +++ b/crates/flowstate/src/workspace/workspace/load.rs @@ -33,7 +33,12 @@ fn load_document_for_open(path: &PathBuf) -> std::io::Result document, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => new_blank_document(), + Err(error) => return Err(error), + }; + Ok(LoadedDocumentForOpen { document, path: Some(path.clone()), title: None, diff --git a/crates/flowstate/src/workspace/workspace/mod.rs b/crates/flowstate/src/workspace/workspace/mod.rs index 0d916083..8bfb0af6 100644 --- a/crates/flowstate/src/workspace/workspace/mod.rs +++ b/crates/flowstate/src/workspace/workspace/mod.rs @@ -46,7 +46,7 @@ use crate::flow::{FlowEditor, FlowPanel}; use crate::rich_text_element::{ ArmedInlineTool, CustomParagraphBorder, Document, DocumentTheme, InputParagraph, InputRun, ParagraphStyle, RichTextDocumentElement, RichTextEditor, Save, SectionKind, ThemeUnderline, ZoomIn, ZoomOut, document_from_input, document_text_slice, flowstate_document_theme, - load_or_create_document, paragraph_byte_range, paragraph_index_for_id, + paragraph_byte_range, paragraph_index_for_id, }; use crate::workspace::document_panel::DocumentPanel; use crate::workspace::file_management::{ diff --git a/crates/gpui-flowtext/src/collaboration.rs b/crates/gpui-flowtext/src/collaboration.rs index 78befc86..d39536a3 100644 --- a/crates/gpui-flowtext/src/collaboration.rs +++ b/crates/gpui-flowtext/src/collaboration.rs @@ -4,7 +4,8 @@ use rustc_hash::FxHashMap; use serde::{Deserialize, Serialize}; use super::{ - AssetId, AssetRecord, Block, BlockId, Document, DocumentSpan, InputBlock, InputParagraph, ParagraphId, ParagraphStyle, RunStyles, TextRun, + AssetId, AssetRecord, Block, BlockId, Document, DocumentOffset, DocumentSpan, InputBlock, InputParagraph, ParagraphId, ParagraphStyle, + RunStyles, TextRun, }; #[derive(Clone, Copy, Debug, Eq, PartialEq, Hash, Serialize, Deserialize)] @@ -90,23 +91,18 @@ where } #[derive(Clone, Debug)] -pub enum CanonicalOperation { +pub enum SemanticEditCommand { InsertText { - paragraph: ParagraphId, - byte: usize, + at: DocumentOffset, text: String, styles: RunStyles, }, DeleteRange { - start_paragraph: ParagraphId, - start_byte: usize, - end_paragraph: ParagraphId, - end_byte: usize, + range: Range, }, SplitParagraph { - paragraph: ParagraphId, - byte: usize, - new_paragraph: ParagraphId, + at: DocumentOffset, + inherited_style: ParagraphStyle, }, JoinParagraphs { first: ParagraphId, @@ -133,7 +129,7 @@ pub enum CanonicalOperation { new_block_ix: usize, }, ReplaceParagraphSpan { - start_paragraph: Option, + start: Option, before: DocumentSpan, after: DocumentSpan, }, @@ -145,7 +141,7 @@ pub enum CanonicalOperation { #[derive(Clone, Debug, Default)] pub struct CollaborationEdit { - pub operations: Vec, + pub semantic_commands: Vec, } #[derive(Clone, Copy, Debug, Eq, PartialEq)] diff --git a/crates/gpui-flowtext/src/persistence/io.rs b/crates/gpui-flowtext/src/persistence/io.rs index 2b07ebe0..cae35a39 100644 --- a/crates/gpui-flowtext/src/persistence/io.rs +++ b/crates/gpui-flowtext/src/persistence/io.rs @@ -24,8 +24,6 @@ use super::{ // UTF-8 text blob, then per-paragraph run metadata. Keeping the format // length-prefixed makes the reader resilient against trailing junk. const DOCUMENT_MAGIC: &[u8; 4] = b"GPTX"; -const LEGACY_DOCUMENT_MAGIC: &[u8; 4] = &[b'D', b'B', b'8', 0]; -const DOCUMENT_LEGACY_VERSION: u32 = 5; const DOCUMENT_VERSION: u32 = 6; const CHUNK_TEXT: u8 = 1; @@ -64,6 +62,7 @@ pub fn load_or_create_document(path: impl AsRef) -> io::Result { #[hotpath::measure] pub fn read_document(path: impl AsRef) -> io::Result { let timing = Instant::now(); + reject_db8_path(path.as_ref())?; let bytes = fs::read(path)?; read_document_bytes_with_timing(&bytes, timing) } @@ -78,13 +77,10 @@ fn read_document_bytes_with_timing(bytes: &[u8], timing: Instant) -> io::Result< let mut cursor = Cursor::new(bytes); let mut magic = [0; 4]; cursor.read_exact(&mut magic)?; - if &magic != DOCUMENT_MAGIC && &magic != LEGACY_DOCUMENT_MAGIC { + if &magic != DOCUMENT_MAGIC { return Err(io::Error::new(io::ErrorKind::InvalidData, "invalid document magic")); } let version = read_u32(&mut cursor)?; - if version == DOCUMENT_LEGACY_VERSION { - return read_document_current(cursor, timing); - } if version == DOCUMENT_VERSION { return read_document_vnext(cursor, timing); } @@ -94,6 +90,7 @@ fn read_document_bytes_with_timing(bytes: &[u8], timing: Instant) -> io::Result< #[hotpath::measure] pub fn write_document(path: impl AsRef, document: &Document) -> io::Result<()> { let path = path.as_ref(); + reject_db8_path(path)?; // Skip directory creation when the parent component is empty (e.g. a bare // filename like "doc.gptx" with no directory prefix), as create_dir_all("") // fails on most platforms. write_bytes_atomic handles it identically. @@ -106,6 +103,17 @@ pub fn write_document(path: impl AsRef, document: &Document) -> io::Result write_bytes_atomic(path, &bytes) } +#[hotpath::measure] +fn reject_db8_path(path: &Path) -> io::Result<()> { + if path.extension().and_then(|extension| extension.to_str()).is_some_and(|extension| extension.eq_ignore_ascii_case("db8")) { + return Err(io::Error::new( + io::ErrorKind::Unsupported, + ".db8 is Flowstate's Loro-native package format; gpui-flowtext's internal final-state serializer must not handle it", + )); + } + Ok(()) +} + #[hotpath::measure] pub fn document_bytes(document: &Document) -> io::Result> { let document = document_for_serialization(document); diff --git a/crates/gpui-flowtext/src/persistence/records.rs b/crates/gpui-flowtext/src/persistence/records.rs index de392ba8..8a925698 100644 --- a/crates/gpui-flowtext/src/persistence/records.rs +++ b/crates/gpui-flowtext/src/persistence/records.rs @@ -167,77 +167,6 @@ fn read_sections_chunk(bytes: &[u8]) -> io::Result> { Ok(sections) } -#[hotpath::measure] -fn read_document_current(mut cursor: Cursor<&[u8]>, timing: Instant) -> io::Result { - let text_len = { - let raw = read_u64(&mut cursor)?; - usize::try_from(raw).map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "native document text length overflows usize"))? - }; - let text_bytes = read_bytes(&mut cursor, text_len, "native document text")?; - let text = std::str::from_utf8(text_bytes) - .map(std::borrow::ToOwned::to_owned) - .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "native document text is not UTF-8"))?; - - let asset_count = { - let raw = read_u64(&mut cursor)?; - usize::try_from(raw).map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "native document asset count overflows usize"))? - }; - let mut assets = AssetStore::default(); - assets.assets.reserve(asset_count); - for _ in 0..asset_count { - let asset = read_asset_record(&mut cursor)?; - assets.assets.insert(asset.id, asset); - } - - let block_count = { - let raw = read_u64(&mut cursor)?; - usize::try_from(raw).map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "native document block count overflows usize"))? - }; - let mut blocks = Vec::with_capacity(block_count.min(4096)); - let mut paragraphs = Vec::new(); - for _ in 0..block_count { - let mut block = read_block_record(&mut cursor)?; - normalize_block_text_runs(&mut block, &text)?; - if let Block::Paragraph(paragraph) = &block { - paragraphs.push(paragraph.clone()); - } - blocks.push(block); - } - if paragraphs.is_empty() { - paragraphs.push(Paragraph { - style: ParagraphStyle::Normal, - byte_range: 0..0, - runs: Vec::new(), - version: 0, - }); - blocks.push(Block::Paragraph(paragraphs[0].clone())); - } - - let offset_index = ParagraphOffsetIndex::new(¶graphs); - let mut document = Document { - text: Rope::from(text), - paragraphs: Arc::new(paragraphs), - blocks: Arc::new(blocks), - assets, - ids: DocumentIds::default(), - sections: Arc::new(Vec::new()), - offset_index, - theme: DocumentTheme::default(), - }; - reconcile_document_ids(&mut document); - rebuild_document_sections(&mut document); - validate_document(&document)?; - log_timing_lazy("document read", timing, || { - format!( - "bytes={} blocks={} paragraphs={}", - document.text.byte_len(), - document.blocks.len(), - document.paragraphs.len() - ) - }); - Ok(document) -} - #[hotpath::measure] fn validate_or_rebuild_sections(document: &mut Document) { if document.sections.is_empty() diff --git a/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs b/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs index 543c3c3f..42b5de4f 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/block_insertion.rs @@ -303,7 +303,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceDocument]; + let semantic_commands = vec![SemanticEditCommand::ReplaceDocument]; self.undo_stack.push(EditRecord { before_selection, before_generation, @@ -313,11 +313,11 @@ impl RichTextEditor { before: Box::new(before_document), after: Box::new(self.document.clone()), }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); } fn insert_plain_text_fragment(&mut self, text: &str, cx: &mut Context) { diff --git a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs index 33c85c8d..b88d1b4b 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/edit_pipeline.rs @@ -14,10 +14,6 @@ impl RichTextEditor { if caret.byte > paragraph_text_len(paragraph) { return false; } - let Some(paragraph_id) = self.identity_map.paragraph_id(caret.paragraph) else { - return false; - }; - let before_selection = self.selection.clone(); let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; @@ -46,7 +42,7 @@ impl RichTextEditor { && before_selection.anchor == before_selection.head && record.after_selection == before_selection && record.operations.len() == 1 - && record.canonical_operations.len() == 1 + && record.semantic_commands.len() == 1 && let EditOperation::InsertText { paragraph, byte, @@ -56,18 +52,17 @@ impl RichTextEditor { && *paragraph == caret.paragraph && *previous_styles == styles && *byte + previous_text.len() == caret.byte - && let CanonicalOperation::InsertText { - paragraph: canonical_paragraph, - byte: canonical_byte, - text: canonical_text, - styles: canonical_styles, - } = &mut record.canonical_operations[0] - && *canonical_paragraph == paragraph_id - && *canonical_styles == styles - && *canonical_byte + canonical_text.len() == caret.byte + && let SemanticEditCommand::InsertText { + at, + text: semantic_text, + styles: semantic_styles, + } = &mut record.semantic_commands[0] + && at.paragraph == caret.paragraph + && *semantic_styles == styles + && at.byte + semantic_text.len() == caret.byte { previous_text.push_str(text); - canonical_text.push_str(text); + semantic_text.push_str(text); record.after_selection = self.selection.clone(); record.after_generation = after_generation; merged_into_previous = true; @@ -85,9 +80,8 @@ impl RichTextEditor { text: text.to_string(), styles, }], - canonical_operations: vec![CanonicalOperation::InsertText { - paragraph: paragraph_id, - byte: caret.byte, + semantic_commands: vec![SemanticEditCommand::InsertText { + at: caret, text: text.to_string(), styles, }], @@ -95,9 +89,8 @@ impl RichTextEditor { } if self.collab_capture && self.suppress_collab_capture == 0 { self.pending_collab_edits.push(CollaborationEdit { - operations: vec![CanonicalOperation::InsertText { - paragraph: paragraph_id, - byte: caret.byte, + semantic_commands: vec![SemanticEditCommand::InsertText { + at: caret, text: text.to_string(), styles, }], @@ -175,12 +168,17 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: self.identity_map.paragraph_id(before_span.start_paragraph), + let semantic_commands = vec![SemanticEditCommand::ReplaceParagraphSpan { + start: self + .identity_map + .paragraph_id(before_span.start_paragraph) + .map(|_| DocumentOffset { + paragraph: before_span.start_paragraph, + byte: 0, + }), before: before_span.clone(), after: after_span.clone(), }]; - let collab_operations = canonical_operations.clone(); let identity_shape_changed = before_span.paragraphs.len() != after_span.paragraphs.len() || before_block_count != self.document.blocks.len(); let record = EditRecord { before_selection, @@ -191,11 +189,11 @@ impl RichTextEditor { before: before_span, after: after_span, }], - canonical_operations, + semantic_commands: semantic_commands.clone(), }; self.undo_stack.push(record); self.redo_stack.clear(); - self.mark_document_changed_with_ops(after_generation, identity_shape_changed, Some(&collab_operations), cx); + self.mark_document_changed_with_ops(after_generation, identity_shape_changed, Some(&semantic_commands), cx); } fn insert_paragraph_break_at_caret(&mut self, caret: DocumentOffset, _block_ix: usize, cx: &mut Context) { @@ -216,10 +214,13 @@ impl RichTextEditor { return; } - let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: self.identity_map.paragraph_id(caret.paragraph), - before: before_span.clone(), - after: after_span.clone(), + let semantic_commands = vec![SemanticEditCommand::SplitParagraph { + at: caret, + inherited_style: before_span + .paragraphs + .first() + .map(|paragraph| paragraph.style) + .unwrap_or(ParagraphStyle::Normal), }]; let record = EditRecord { before_selection, @@ -230,19 +231,18 @@ impl RichTextEditor { before: before_span.clone(), after: after_span.clone(), }], - canonical_operations, + semantic_commands: semantic_commands.clone(), }; - let collab_operations = record.canonical_operations.clone(); self.undo_stack.push(record); self.redo_stack.clear(); - self.mark_document_changed_with_ops(after_generation, false, Some(&collab_operations), cx); + self.mark_document_changed_with_ops(after_generation, false, Some(&semantic_commands), cx); } fn mark_document_changed_with_ops( &mut self, generation: u64, reconcile_identity: bool, - operations: Option<&[CanonicalOperation]>, + semantic_commands: Option<&[SemanticEditCommand]>, cx: &mut Context, ) { self.edit_generation = generation; @@ -251,10 +251,12 @@ impl RichTextEditor { } if self.collab_capture && self.suppress_collab_capture == 0 - && let Some(operations) = operations - && !operations.is_empty() + && let Some(semantic_commands) = semantic_commands + && !semantic_commands.is_empty() { - self.pending_collab_edits.push(CollaborationEdit { operations: operations.to_vec() }); + self.pending_collab_edits.push(CollaborationEdit { + semantic_commands: semantic_commands.to_vec(), + }); } self.refresh_save_status(); self.schedule_recovery_write(cx); diff --git a/crates/gpui-flowtext/src/rich_text/editor/media.rs b/crates/gpui-flowtext/src/rich_text/editor/media.rs index 24ddeb37..d74daacc 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/media.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/media.rs @@ -151,7 +151,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + let semantic_commands = vec![SemanticEditCommand::ReplaceBlock { block: self.identity_map.block_id(drag.block_ix), }]; self.undo_stack.push(EditRecord { @@ -164,11 +164,11 @@ impl RichTextEditor { before: Block::Image(drag.before), after: Block::Image(after), }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); true } @@ -216,7 +216,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + let semantic_commands = vec![SemanticEditCommand::ReplaceBlock { block: self.identity_map.block_id(block_ix), }]; self.undo_stack.push(EditRecord { @@ -225,11 +225,11 @@ impl RichTextEditor { after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); } pub fn insert_equation(&mut self, source: impl Into, cx: &mut Context) { diff --git a/crates/gpui-flowtext/src/rich_text/editor/mod.rs b/crates/gpui-flowtext/src/rich_text/editor/mod.rs index bdd836a6..c656a554 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mod.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mod.rs @@ -205,7 +205,7 @@ struct EditRecord { after_selection: EditorSelection, after_generation: u64, operations: Vec, - canonical_operations: Vec, + semantic_commands: Vec, } #[derive(Clone, Debug)] diff --git a/crates/gpui-flowtext/src/rich_text/editor/mouse.rs b/crates/gpui-flowtext/src/rich_text/editor/mouse.rs index 7ae56606..d93cc7f3 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/mouse.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/mouse.rs @@ -289,7 +289,6 @@ impl RichTextEditor { let capture_start = source_range.start.paragraph.min(drop.paragraph); let capture_end = source_range.end.paragraph.max(drop.paragraph).saturating_add(1); let before_span = capture_document_span(&before_document, capture_start..capture_end); - let start_paragraph = self.identity_map.paragraph_id(before_span.start_paragraph); self.selection = before_selection.clone(); self.delete_selection_internal(); let inserted_start = adjusted_drop; @@ -312,8 +311,11 @@ impl RichTextEditor { .saturating_sub(before_span.start_paragraph), ); let after_span = capture_document_span(&self.document, before_span.start_paragraph..before_span.start_paragraph + after_count); - let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { - start_paragraph, + let semantic_commands = vec![SemanticEditCommand::ReplaceParagraphSpan { + start: Some(DocumentOffset { + paragraph: before_span.start_paragraph, + byte: 0, + }), before: before_span, after: after_span, }]; @@ -328,11 +330,11 @@ impl RichTextEditor { inserted_range: inserted_start..inserted_end, fragment: drag.fragment, }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.after_text_mutation(cx); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); self.clear_drop_preview(); } diff --git a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs index 800c8e77..5981865a 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/object_selection.rs @@ -242,7 +242,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + let semantic_commands = vec![SemanticEditCommand::ReplaceBlock { block: self.identity_map.block_id(drag.block_ix), }]; self.undo_stack.push(EditRecord { @@ -255,11 +255,11 @@ impl RichTextEditor { before: Block::Table(drag.before), after: Block::Table(after), }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); true } @@ -455,7 +455,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceDocument]; + let semantic_commands = vec![SemanticEditCommand::ReplaceDocument]; self.undo_stack.push(EditRecord { before_selection, before_generation, @@ -465,11 +465,11 @@ impl RichTextEditor { before: Box::new(before_document), after: Box::new(after_document), }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); true } @@ -501,9 +501,9 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = block_id.map_or_else( - || vec![CanonicalOperation::ReplaceDocument], - |block| vec![CanonicalOperation::DeleteBlock { block }], + let semantic_commands = block_id.map_or_else( + || vec![SemanticEditCommand::ReplaceDocument], + |block| vec![SemanticEditCommand::DeleteBlock { block }], ); self.undo_stack.push(EditRecord { before_selection: before_selection.clone(), @@ -511,13 +511,13 @@ impl RichTextEditor { after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::DeleteBlock { block_ix, block }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.clear_layout_work_caches(); self.item_sizes_cache = None; self.paragraph_height_cache_revision = self.paragraph_height_cache_revision.wrapping_add(1); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); true } diff --git a/crates/gpui-flowtext/src/rich_text/editor/paste.rs b/crates/gpui-flowtext/src/rich_text/editor/paste.rs index 10f9506d..479e04fe 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/paste.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/paste.rs @@ -29,12 +29,6 @@ impl RichTextEditor { { return false; } - let Some(paragraph_id) = self - .identity_map - .paragraph_id(self.selection.head.paragraph) - else { - return false; - }; let paragraph = &fragment.paragraphs[0]; if paragraph.runs.iter().all(|run| run.text.is_empty()) { return true; @@ -53,8 +47,11 @@ impl RichTextEditor { head: inserted_end, }; self.emit_selection_changed(cx); - let canonical_operations = vec![CanonicalOperation::ReplaceParagraphSpan { - start_paragraph: Some(paragraph_id), + let semantic_commands = vec![SemanticEditCommand::ReplaceParagraphSpan { + start: Some(DocumentOffset { + paragraph: offset.paragraph, + byte: 0, + }), before: before_span.clone(), after: after_span.clone(), }]; @@ -68,12 +65,12 @@ impl RichTextEditor { inserted_end, fragment: fragment.clone(), }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.layout_invalidation_hint = Some(offset.paragraph..offset.paragraph + 1); self.after_text_mutation(cx); - self.mark_document_changed_with_ops(after_generation, false, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, false, Some(&semantic_commands), cx); true } diff --git a/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs b/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs index 86ae28d6..ddab1194 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/table_equation_editing.rs @@ -67,7 +67,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + let semantic_commands = vec![SemanticEditCommand::ReplaceBlock { block: self.identity_map.block_id(block_ix), }]; self.undo_stack.push(EditRecord { @@ -76,13 +76,13 @@ impl RichTextEditor { after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.table_cell_block_ix = new_paragraph_ix; self.table_cell_caret = 0; self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); true } @@ -282,7 +282,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + let semantic_commands = vec![SemanticEditCommand::ReplaceBlock { block: self.identity_map.block_id(block_ix), }]; self.undo_stack.push(EditRecord { @@ -291,11 +291,11 @@ impl RichTextEditor { after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); } pub(super) fn edit_table_cell_paragraph( @@ -339,7 +339,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + let semantic_commands = vec![SemanticEditCommand::ReplaceBlock { block: self.identity_map.block_id(block_ix), }]; self.undo_stack.push(EditRecord { @@ -348,11 +348,11 @@ impl RichTextEditor { after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); } } diff --git a/crates/gpui-flowtext/src/rich_text/editor/tables.rs b/crates/gpui-flowtext/src/rich_text/editor/tables.rs index ace99622..b4ea0e67 100644 --- a/crates/gpui-flowtext/src/rich_text/editor/tables.rs +++ b/crates/gpui-flowtext/src/rich_text/editor/tables.rs @@ -165,7 +165,7 @@ impl RichTextEditor { let before_generation = self.edit_generation; let after_generation = self.next_edit_generation; self.next_edit_generation = self.next_edit_generation.wrapping_add(1); - let canonical_operations = vec![CanonicalOperation::ReplaceBlock { + let semantic_commands = vec![SemanticEditCommand::ReplaceBlock { block: self.identity_map.block_id(block_ix), }]; self.undo_stack.push(EditRecord { @@ -174,11 +174,11 @@ impl RichTextEditor { after_selection: self.selection.clone(), after_generation, operations: vec![EditOperation::ReplaceBlock { block_ix, before, after }], - canonical_operations: canonical_operations.clone(), + semantic_commands: semantic_commands.clone(), }); self.redo_stack.clear(); self.invalidate_document_layout_caches(); - self.mark_document_changed_with_ops(after_generation, true, Some(&canonical_operations), cx); + self.mark_document_changed_with_ops(after_generation, true, Some(&semantic_commands), cx); } fn selected_table_block_ix(&self) -> Option { diff --git a/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs index ab8ae91e..f1034f0b 100644 --- a/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs +++ b/crates/gpui-flowtext/src/rich_text/tests/collab_capture.rs @@ -23,11 +23,17 @@ fn collab_capture_fast_path_emits_single_grapheme_deltas(cx: &mut gpui::TestAppC assert_eq!(edits.len(), 3); for (edit, (expected_text, expected_byte)) in edits.iter().zip([("a", 0), ("b", 1), ("c", 2)]) { - let [CanonicalOperation::InsertText { byte, text, .. }] = edit.operations.as_slice() else { - panic!("expected one InsertText op, got {:?}", edit.operations); + let [SemanticEditCommand::InsertText { at, text, .. }] = edit.semantic_commands.as_slice() else { + panic!("expected one semantic InsertText command, got {:?}", edit.semantic_commands); }; assert_eq!(text, expected_text); - assert_eq!(*byte, expected_byte); + assert_eq!( + *at, + DocumentOffset { + paragraph: 0, + byte: expected_byte, + } + ); } let edits_after_undo = cx.update(|cx| { @@ -54,7 +60,10 @@ fn collab_capture_fast_path_emits_single_grapheme_deltas(cx: &mut gpui::TestAppC }) }); assert_eq!(paste_edits.len(), 1); - assert!(matches!(paste_edits[0].operations.as_slice(), [CanonicalOperation::ReplaceParagraphSpan { .. }])); + assert!(matches!( + paste_edits[0].semantic_commands.as_slice(), + [SemanticEditCommand::ReplaceParagraphSpan { .. }] + )); let edits_after_paste_undo = cx.update(|cx| { editor.update(cx, |editor, cx| { @@ -143,7 +152,10 @@ fn text_entry_in_selected_equation_updates_equation_only(cx: &mut gpui::TestAppC assert_eq!(equation.source.as_ref(), "x+1+2"); assert_eq!(edits.len(), 2); for edit in edits { - assert!(matches!(edit.operations.as_slice(), [CanonicalOperation::ReplaceBlock { .. }])); + assert!(matches!( + edit.semantic_commands.as_slice(), + [SemanticEditCommand::ReplaceBlock { .. }] + )); } } diff --git a/handoff-adjustmentplan.md b/handoff-adjustmentplan.md new file mode 100644 index 00000000..3d2ef174 --- /dev/null +++ b/handoff-adjustmentplan.md @@ -0,0 +1,333 @@ +# Handoff: Adjustment Plan Implementation Status + +> Generated: 2026-06-18 +> Source: `adjustmentplan.md` (1283 lines, the "Platonically ideal" production Loro-native architecture) +> Companion: `FIX_LORO_ROOT.md` (614 lines, execution plan for single-root LoroText performance; NOT yet implemented) + +--- + +## Summary + +The codebase has completed a **Loro-native foundation** (schema, projection, import/export, package format, table/equation/image objects, undo/redo, revision/restore). However, the **editor projection layer** (`gpui-flowtext::Document`) is still the old mutable document model, and the **performance infrastructure** identified in `FIX_LORO_ROOT.md` is **entirely unimplemented** — the file does not exist in the crate. Additionally, there is a **dual-runtime problem**: `CrdtRuntime` (flowstate-collab) is a clean implementation of the plan's runtime, but the app uses `CollabSession` (flowstate) which manages its own `LoroDoc` directly. + +--- + +## Section-by-Section Status + +### §1 Non-Negotiable Objective — **Mostly Complete** +- LoroDoc IS the canonical document substrate. ✓ +- `.db8` extension preserved as brand identity, now uses Loro-native package format. ✓ +- Object replacement character (`U+FFFC`) strategy locked in. ✓ +- Tables fully in scope and implemented as structured Loro objects. ✓ +- Named revisions implemented. ✓ +- PDF source embeds Loro-native `.db8` bytes only. ✓ +- **Gap**: The old GPTX (final-state) serializer still exists at `crates/gpui-flowtext/src/persistence/io.rs`. It uses magic `b"GPTX"`, version 6, and its own chunked binary format. The `reject_db8_path` function prevents it from handling `.db8` paths, but the format itself is still present and used by editor recovery, benchmarks, and some tests. The plan says "Code that exists only to read, write, recover, index, export, or interoperate with the old final-state `.db8` model must be removed or rewritten." The GPTX is a final-state serializer with a different extension, so it's technically outside the `.db8` prohibition — but it represents a parallel final-state persistence path. + +### §1A Locked Clarifications — **Complete** +- `.db8` is Loro-native only. ✓ +- No old `.db8` compatibility reader. ✓ +- Rust `loro` crate API is authoritative. ✓ +- `U+FFFC` object replacement character strategy used for block anchoring. ✓ +- Tables are structured Loro objects. ✓ +- Named revisions are time-travellable snapshot/frontier points. ✓ +- PDF source embeds only new Loro-native `.db8`. ✓ + +### §2 Canonical Model — **Complete in intent, partial in practice** +- LoroDoc is the single canonical document model. ✓ +- Everything else is derived. ✓ +- The canonical pipeline (local command → resolve → Loro mutation → subscription → projection → render) is architecturally understood and partially implemented in `CrdtRuntime`. ✓ +- **Gap**: The `CollabSession` in the app does NOT use `CrdtRuntime`. It owns its own `LoroDoc` and `UndoManager` directly. Editor commands flow through `apply_editor_semantic_command` (a free function), not through `CrdtRuntime::command`. The pipeline is less clean at the app level. + +### §3 Runtime Ownership — **Incomplete** +- **Planned**: Dedicated CRDT runtime (thread/task/actor) owns LoroDoc; UI owns only projection. +- **Reality**: `CrdtRuntime` struct exists in `flowstate-collab::crdt_runtime` with the correct ownership model, but **it is not used by the application**. The app's `CollabSession` holds `doc: Option` directly on the UI thread. There is no thread boundary between CRDT mutation and UI. The `CrdtRuntime` is only exercised in unit tests. +- **Gap**: Dual runtime — `CrdtRuntime` (unused in production) vs `CollabSession` (production, directly manages LoroDoc). No thread isolation. + +### §4 Command Layer — **Partially Complete** +- `SemanticCommand` enum exists with most planned operations (InsertText, DeleteRange, SplitParagraph, SetParagraphStyle, SetRunStyles, InsertImage, InsertEquation, InsertTable, OpenRevision, ForkRevision, Undo, Redo). ✓ +- CRDT runtime resolves semantic commands into Loro schema mutations. ✓ +- `apply_editor_semantic_command` free function maps `gpui-flowtext::SemanticEditCommand` to Loro mutations (used by CollabSession). ✓ +- **Gap**: The old `CanonicalOperation` API still exists in `gpui-flowtext::collaboration` alongside `WireCanonicalOperation`, `encode_canonical_operations`, `decode_canonical_operations` — dead pre-CRDT code. `SemanticEditCommand` (editor's old command enum) coexists and is the primary input to `apply_editor_semantic_command`. + +### §5 Projection And Editor Model — **Incomplete** +- **Planned**: Replace `gpui-flowtext::Document` entirely with `DocumentProjection`. No dual binding/reconciliation. +- **Reality**: `document_from_loro()` projects LoroDoc into `gpui-flowtext::Document` (the old editable document type). The editor (`RichTextEditor`) still uses `gpui-flowtext::Document` as its core model. There is no `DocumentProjection` type. The editor mutates via `SemanticEditCommand` which `apply_editor_semantic_command` translates to Loro mutations. This IS a binding/reconciliation layer. +- **Gap**: The editor model is NOT replaced. `gpui-flowtext::Document` remains the projection model. The target "Keeping a type named `Document` while redefining it as projection-only" IS what's happening, which the plan explicitly rejects. + +### §6 Canonical Text Model — **Complete** +- Single root `LoroText` for body flow. ✓ +- `\n` paragraph delimiters. ✓ +- Sentinal newline at position 0. ✓ +- Independent `LoroText` per cell flow for tables. ✓ +- Image caption/alt text, equation source have independent flows. ✓ + +### §7 Paragraph Boundaries — **Complete** +- Literal `\n` characters inside flow `LoroText`. ✓ +- Sentinal newline (non-renderable, non-deletable). ✓ +- Predecessor-boundary model (paragraph style on preceding `\n`). ✓ +- Paragraph identity metadata in `ParagraphMap` with start_cursor, boundary_cursor. ✓ +- No custom paragraph marker characters. ✓ +- Loro marks (`paragraph_style`) on boundary newlines with `ExpandType::None`. ✓ +- The 10 word-processor semantics rules for paragraph split/join/style application are documented in the plan but NOT verified in code behavior. + +### §8 Flow Structure — **Complete** +- `FlowMap` with id, kind, `text: LoroText`, `attrs: LoroMap`. ✓ +- `flows_by_id: LoroMap` in root. ✓ +- Flow kinds: body, table_cell, caption, alt_text, equation_source. ✓ +- Object replacement characters for embedded objects where supported. ✓ + +### §9 Block And Object Ordering — **Complete** +- Body flow is canonical ordered sequence. ✓ +- Object replacements (`U+FFFC`) in body text. ✓ +- `blocks_by_id: LoroMap` in root with `BlockMap` per block. ✓ +- `BlockMap` has id, kind, flow_id, anchor_cursor, attrs, nested_refs. ✓ +- All durable IDs stored in Loro. ✓ + +### §10 Semantic Styling — **Complete** +- Loro marks for paragraph_style, run_semantic_style_id, highlight_style_id, direct_underline, strikethrough. ✓ +- Semantic enum slots (not freeform values). ✓ +- Style catalog is app/theme data only. ✓ +- DOCX import maps to predefined slots. ✓ +- Comments/annotations/bookmarks/suggestions are out of scope. ✓ + +### §11 Sections And Page Structure — **Not Started** +- `sections_by_id` map is initialized in schema (`loro_schema.rs:52`) but **never populated or used**. ✓ +- No section creation, reading, or projection exists in the Loro-native code paths. +- Section data comes from the old `gpui-flowtext::Document` derived from paragraph styles (section heading/outline). +- **Gap**: Full section architecture is missing: `SectionMap` with start_cursor, page_size, margins, columns, header_flow_id, footer_flow_id, page_numbering, orientation. + +### §12 Tables — **Complete** +- Structured Loro objects. ✓ +- `row_order: LoroMovableList`, `column_order: LoroMovableList`. ✓ +- `rows_by_id`, `columns_by_id`, `cells_by_id` as LoroMaps. ✓ +- Cell contents are independent LoroText flows. ✓ +- Nested tables supported via `nested_table.*` keys in cell map. ✓ +- InsertTable command creates full table schema. ✓ +- Round-trip through projection works with tests. ✓ + +### §13 Equations — **Complete** +- Equation source as independent LoroText flow. ✓ +- EquationBlock in `blocks_by_id` with source_flow_id, attrs (syntax, display). ✓ +- Render output is local only (not synced). ✓ +- Inline and block equations both use `U+FFFC` placeholder. ✓ + +### §14 Images And Assets — **Complete** +- Image metadata in LoroMap (asset_id, content_hash, sizing, alignment, etc.). ✓ +- Asset bytes outside Loro in content-addressed asset store. ✓ +- BLAKE3 content addressing. ✓ +- Caption and alt text are independent LoroText flows. ✓ +- Missing assets render placeholder. ✓ + +### §15 Presence, Peers, And Author Metadata — **Complete** +- Loro `EphemeralStore` for presence. ✓ +- PresenceState with name and selection. ✓ +- Roster derivation from ephemeral state. ✓ +- `users_by_id` and `replicas_by_id` maps in schema (initialized but may not be populated). ✓ +- PeerId for replica identity, distinct from user identity. ✓ + +### §16 Selection, Cursors, And Affinity — **Not Started** +- **Planned**: Full `SelectionEndpoint` model with explicit affinity/gravity (before/after/neutral, visual_gravity). +- **Reality**: Loro cursors are used with basic `Side::Left`/`Side::Right` choices. No `SelectionEndpoint` type exists. No `affinity` or `visual_gravity` model. No `Selection` type with anchor/head/direction. +- **Gap**: The full affinity/gravity model is unimplemented. + +### §17 Undo And Redo — **Complete** +- Loro `UndoManager` with per-peer exclusion prefix `"remote"`. ✓ +- Merge interval (600ms in CrdtRuntime, 500ms in CollabSession). ✓ +- `record_new_checkpoint` for command boundaries. ✓ +- Semantic undo units correspond to grouped changes. ✓ +- Time travel is separate from undo. ✓ + +### §18 Time Travel, Revisions, And Restore — **Complete** +- `PackageRevision` stored in package with frontier, title, summary, timestamp, author. ✓ +- `create_named_revision`, `load_revision_loro_doc`, `fork_revision_runtime`. ✓ +- `compact_to_snapshot` for compaction. ✓ +- Named revisions preserved after compaction. ✓ +- `OpenRevision`/`ForkRevision` commands in `CrdtRuntime`. ✓ +- Tests exist for revision preservation through compaction. ✓ +- **Note**: Compaction is manual (explicit API calls). No automatic threshold-based compaction. + +### §19 Filesystem Package — **Complete** +- `DocumentPackage` with custom chunked binary container. ✓ +- Magic `FLOWDB8-LORO\0\0\0\0`, header version 1. ✓ +- Chunk types: manifest, Loro snapshot, Loro update segment, asset, revision index, projection cache, search unit. ✓ +- Postcard serialization, BLAKE3 checksums. ✓ +- Atomic write via temp file + rename. ✓ +- `append_update_segment` for incremental persistence. ✓ +- Validation: version checks, frontier chain integrity, checksum verification. ✓ + +### §20 Read Path — **Partially Complete** +- Open package → read manifest → load latest snapshot → apply update segments. ✓ +- Construct LoroDoc in CRDT runtime. ✓ +- Verify package format and schema version. ✓ +- **Gap**: Projection cache and search cache are loaded but are never populated during normal writes. The `projection_cache` frontier is always `None`. Search cache is populated via `rebuild_search_units_from_loro` during snapshot operations. Fast unopened-file indexing via package caches is not fully wired up. + +### §21 Write Path — **Partially Complete** +- Append-first persistence exists (update segments appended to package). ✓ +- Update segment chains validated for frontier integrity. ✓ +- **Gap**: No automatic snapshot compaction when update history crosses thresholds. The current save path writes the full package every time (`package.write(path)` is called from `save_package` and `persist_update_segment` in `CrdtRuntime`, but `CollabSession` does NOT use `CrdtRuntime` — it writes via `DocumentPackage::from_loro_snapshot_with_assets` in `from_local_document` and never persists update segments incrementally). The `CollabSession` path does NOT call `append_update_segment` at all. Only the standby `CrdtRuntime` does incremental persistence. +- **Gap**: No projection cache or search cache writing during save. + +### §22 Sync And Anti-Entropy — **Complete** +- Remote updates imported directly into LoroDoc via `import_with(bytes, "remote")`. ✓ +- `ImportStatus` checked for pending dependencies. ✓ +- Immediate anti-entropy pull triggered on missing deps. ✓ +- Periodic digest fallback (10s). ✓ +- Direct connection protocol with snapshot/update/asset serving. ✓ +- VersionVector-based digest comparison. ✓ +- Blob transfer with DirectSessionHandler. ✓ + +### §23 Permanent Loro Subscription — **Partially Complete** +- Permanent subscriptions exist (`_root_subscription`, `_local_update_subscription` in `CrdtRuntime`; `loro_subscriptions` in `CollabSession`). ✓ +- **Gap**: Subscriptions are essentially no-ops — they only do tracing. Projection invalidation is not driven by subscriptions. Instead, the `events_after_local_change` method rebuilds the projection explicitly after each command. The subscription-backed architecture is not the actual invalidation path. + +### §24 Projection Invalidation — **Not Started** +- **Planned**: Option C — batched frontier-based invalidation with incremental fast paths and full rebuild fallback. +- **Reality**: Full projection rebuild (`document_from_loro`) on every change. No `ProjectionInvalidation` type. No incremental projection indexes. No dirty-range tracking. No fallback detection/logging. This is Option A (full rebuild on every event), which the plan rates 3/10 for performance and 8-9/10 for everything else. + +### §25 Search — **Partially Complete** +- `rebuild_search_units_from_loro` exists in `DocumentPackage` and derives search units from Loro body flow. ✓ +- Search units have cursors (paragraph_start_cursor, paragraph_end_cursor) instead of old paragraph indexes. ✓ +- **Gap**: The tub (`flowstate-tub`) still uses `read_db8(path)` which reads the full new-format `.db8` file into a gpui-flowtext Document via the Loro→Document projection, then indexes sections from the old Document model. It does NOT use `DocumentPackage`'s search units directly. **Critical gap**: `db8_index_units` at `flowstate-tub/src/lib.rs:923` reads the entire Document to get section information, which doesn't exist as Loro-native sections. +- **Gap**: Tantivy index is still used for full-text search (acceptable per plan). Section-aware units use old paragraph indexes (`paragraph_start: Option` in `IndexUnit`), not Loro cursors. + +### §26 DOCX, PDF, And Export — **Complete** +- DOCX import → Document → Loro-native `.db8`. ✓ +- `convert_docx_to_db8` writes Loro-native format. ✓ +- PDF generation includes Loro-native `.db8` source embedding (compressed, zstd). ✓ +- PDF recovery extracts Loro-native `.db8` bytes only (magic `FSL8ZST\0`). ✓ +- No dual old/new payload support in PDF. ✓ +- Export generates from Document projection. ✓ + +### §27 Schema Versioning — **Complete** +- `MetaMap` with document_id, loro_schema_version, schema_features, created_by_app_version, last_written_by_app_version, created_at, modified_at. ✓ +- `LORO_PACKAGE_FORMAT_VERSION = 1`, `LORO_SCHEMA_VERSION = 1`. ✓ +- Version check on package open. ✓ + +### §28 Container References — **Complete** +- Durable Flowstate IDs (flow_id, block_id, paragraph_id) stored as strings in Loro maps. ✓ +- Loro containers created via `ensure_mergeable_*` methods. ✓ +- Container resolution centralized in schema module. ✓ + +### §29 Clear/Rebuild Paths — **Partially Complete** +- `replace_body_from_document` does full clear+repopulate of body text and all blocks/paragraphs. This IS used during DOCX import and initial document creation. ✓ +- `clear_map` and `clear_list` used in import paths. ✓ +- **Gap**: Normal editing should NOT use clear/repopulate, but currently the editor emits `ReplaceParagraphSpan`/`ReplaceDocument` which trigger full body rewrites. No incremental edit paths exist for non-typing operations. + +### §30 Invariants — **Mostly Met** +1. Full reconstruction from Loro snapshot/update + assets: ✓ (load_loro_doc) +2. Caches are disposable: ✓ (explicitly stated) +3. Paragraph split and inline insert commute: ~ (works at Loro CRDT level, but editor emits ReplaceDocument which is unnecessarily destructive) +4. Undo/redo is per-peer: ✓ +5. Time travel via frontiers: ✓ +6. Revision restore opens dirty fork: ✓ (ForkRevision command) +7. Binary assets outside CRDT: ✓ +8. Caption/alt text as CRDT flows: ✓ +9. Equation source as CRDT flow: ✓ +10. Semantic Loro marks: ✓ +11. Paragraph style in boundary marks: ✓ +12. Block metadata in Loro maps: ✓ +13. Tables as structured CRDT: ✓ +14. Rows/columns have durable identity: ✓ +15. Remote updates imported directly: ✓ +16. Local commands mutate through runtime: Partially — CollabSession does direct LoroDoc mutation +17. No canonical Flowstate doc separate from Loro: Partially — gpui-flowtext::Document still exists as editor model +18. No local-vs-remote split: Partially — CollabSession has this split in practice +19. Files are Loro-backed packages: ✓ +20. Projection caches declare frontier: Not implemented (always None) +21. Search indexes declare frontier: Implemented in rebuild_search_units_from_loro +22. DOCX/PDF from snapshots: ✓ +23. Comments/annotations/bookmarks/suggestions out of scope: ✓ +24. General test suite out of scope: ✓ + +### §31 Data Shape Summary — **Complete with caveats** +All root containers exist in the schema: +- meta: ✓ +- flows_by_id: ✓ +- blocks_by_id: ✓ +- paragraphs_by_id: ✓ +- sections_by_id: **Empty/unused** +- assets_by_id: ✓ (populated during import) +- revisions: ✓ (populated) +- users_by_id: ✓ (schema only, not populated) +- replicas_by_id: ✓ (schema only, not populated) + +### §32 Expected Result — **In Progress** +The architecture has one model (`LoroDoc + asset store + derived Flowstate projections`) at the crate boundary, but the app still uses a dual-model approach where `CollabSession` directly manages `LoroDoc` and the editor uses `gpui-flowtext::Document` as its live model. + +--- + +## Cross-Cutting Issues + +### Issue A: Dual CRDT Runtime (Critical) + +`flowstate-collab::crdt_runtime::CrdtRuntime` is the adjustment plan's canonical runtime. It is complete, tested, and unused in production. `flowstate::collab::session::CollabSession` is the application's actual runtime, managing `LoroDoc` inline. + +**Impact**: The `CollabSession` path: +- Does not use `CrdtRuntime::command()` for mutation dispatch +- Does not use `CrdtRuntime` for package persistence +- Manages its own `UndoManager` directly +- Has its own subscription setup +- Mixes CRDT mutation and UI rendering on the same thread +- Does not persist update segments incrementally (only full snapshot writes) + +**The `CrdtRuntime` should be the canonical path** and `CollabSession` should delegate to it. + +### Issue B: Performance Infrastructure (FIX_LORO_ROOT.md) — **Not Started** + +`FIX_LORO_ROOT.md` details 15 tasks (T1–T15) across 4 pillars: +1. **Foundation** — Maintained paragraph offset index (T1) +2. **Incremental local apply** — Span-scoped body splices (T2, T3) +3. **Delta-driven remote reconcile** — Use Loro TextDelta instead of full reproject (T4) +4. **Editor-model incrementalization** — O(log n) offset/section/identity/structural patches (T6–T11) +5. **Cleanup** — Dead code removal, style single-truth, doc cleanup (T12–T15) + +**None of these tasks have been implemented.** The files `local_apply.rs`, `remote_apply.rs`, `binding.rs`, `patch_apply.rs`, `body_index.rs` do not exist in any crate. Every non-typing edit causes full-document rewrites. + +### Issue C: Section Architecture — **Not Started** + +`Sections by ID` map is initialized but never populated. No section creation or querying logic exists in the Loro-native code. Sections are derived from old Document model paragraph style → section kind mapping. + +### Issue D: Selection Model — **Not Started** + +The full affinity/gravity selection model from §16 is unimplemented. Only basic Loro cursor `Side` is used. + +### Issue E: Projection Invalidation — **Not Started** + +Full projection rebuild on every change. No incremental invalidation system exists. + +### Issue F: Old GPTX Serializer Still Present + +The old final-state serializer (`gpui-flowtext::persistence::io`) with magic `GPTX`, version 6, chunk types for text/assets/blocks/IDs/sections still exists. It is used by: +- Editor recovery path +- Benchmarks +- Internal tests +- Editor's `write_document_export` for `DocumentExportFormat::Native` + +The plan says old final-state format must go, but this is the `.gptx` extension (not `.db8`). It's debatable whether this counts as the "old .db8 format" — it uses a different extension. However, it IS a final-state serializer that should be removed per the architecture. + +### Issue G: Unused `CrdtRuntime` in Production + +`CrdtRuntime` is only exercised by unit tests in `crdt_runtime.rs`. The production `CollabSession` has its own parallel implementation. This means the plan's clean runtime architecture is not used in production. + +--- + +## Priority Summary + +| Priority | Area | Status | Effort | +|----------|------|--------|--------| +| **P0** | Adopt `CrdtRuntime` in `CollabSession`, eliminate dual runtime | Not Started | Medium | +| **P0** | Implement T1 (maintained paragraph offset index) from FIX_LORO_ROOT | Not Started | Medium | +| **P0** | Implement T2 (incremental ReplaceParagraphSpan) from FIX_LORO_ROOT | Not Started | Large | +| **P0** | Implement T4 (delta-driven remote reconcile) from FIX_LORO_ROOT | Not Started | Large | +| **P1** | Implement §16 selection model (affinity/gravity) | Not Started | Medium | +| **P1** | Implement §11 section architecture (sections_by_id population) | Not Started | Medium | +| **P1** | Implement §24 projection invalidation (Option C) | Not Started | Large | +| **P2** | Remove old GPTX serializer | Not Started | Small | +| **P2** | Remove dead pre-CRDT code (WireCanonicalOperation, etc.) | Not Started | Small | +| **P2** | Implement T6-T11 (editor-model incrementalization) from FIX_LORO_ROOT | Not Started | Medium | +| **P2** | Wire up projection cache and search cache in write path | Not Started | Small | +| **P2** | Automatic snapshot compaction with thresholds | Not Started | Small | +| **P2** | Port tub search to use Loro-native search units directly | Not Started | Small | +| **P3** | Implement T5 (granular local op emission for delete/Enter) | Not Started | Small | +| **P3** | T12-T15 cleanup tasks from FIX_LORO_ROOT | Not Started | Small | +| **P3** | Populate users_by_id and replicas_by_id | Not Started | Small | diff --git a/loroapi.md b/loroapi.md new file mode 100644 index 00000000..63dd1890 --- /dev/null +++ b/loroapi.md @@ -0,0 +1,34060 @@ + + +# FILE: .next/standalone/pages/docs/api/js.mdx + +import styles from "./api-reference.module.css"; +import Indent from "./indent"; +import Method from "./method"; + +
+ +# API Reference + +> _Last updated: 2026-06-16 loro-crdt@1.13.3_ + +## Overview + +Loro is a powerful Conflict-free Replicated Data Type (CRDT) library that enables real-time collaboration. If CRDTs are new to you, start with [What are CRDTs](/docs/concepts/crdt) for a gentle intro. This API reference provides comprehensive documentation for all classes, methods, and types available in the JavaScript/TypeScript binding. + +Note: Under the hood, Loro combines a Fugue-based CRDT core with Eg-walker-inspired techniques that use simple index operations and replay only the divergent history when merging. This yields fast local edits, efficient merges, and low overhead without permanent tombstones. See the primer [Event Graph Walker (Eg-walker)](/docs/advanced/event_graph_walker) and performance notes in the v1.0 blog (import/export speedups, shallow snapshots): https://loro.dev/blog/v1.0 + +## Pitfalls & Best Practices + +**Peer ID Management** + +- **Never share PeerIDs** between concurrent sessions (tabs/devices) - causes document divergence +- Use random PeerIDs (default) unless you have strict single-ownership locking +- Don't assign fixed PeerIDs to users or devices + +**UTF-16 Text Encoding** + +- All text operations use UTF-16 indices by default in JS API +- Slicing in the middle of multi-unit codepoints corrupts them +- Use `insertUtf8()`/`deleteUtf8()` for UTF-8 systems + +**Container Creation** + +- Concurrent child container creation inside the same LoroMap at same key causes overwrites +- Initialize all child containers for a LoroMap upfront when possible +- Operations on the root containers will not override each other + +- Events emit synchronously during commit/import/checkout in JS API (v1.8+). Stay on `<=1.7.x`? Await a microtask before reading batched events. +- Import/export/checkout trigger automatic commits +- Loro transactions are NOT ACID - no rollback/isolation + +**Version Control** + +- After `checkout()`, document enters read-only "detached" mode, unless `setDetachedEditing(true)` is called +- [Frontiers](/docs/concepts/frontiers) can't determine complete operation sets without history + +**Data Structure Choice** + +- Use strings in Map for URLs/IDs (LWW), LoroText for collaborative editing + +## Common Tasks & Examples + +**Getting Started** + +- **Create a document**: [`new LoroDoc()`](#LoroDoc.constructor) - Initialize a new collaborative document +- **Add containers**: [`getText`](#LoroDoc.getText), [`getList`](#LoroDoc.getList), [`getMap`](#LoroDoc.getMap), [`getTree`](#LoroDoc.getTree) +- **Listen to changes**: [`subscribe`](#LoroDoc.subscribe) - React to document modifications +- **Export/Import state**: [`export`](#LoroDoc.export) and [`import`](#LoroDoc.import) - Save and load documents + +**Real-time Collaboration** + +- **Sync between peers**: [`export`](#LoroDoc.export) with `mode: "update"` + [`import`](#LoroDoc.import)/[`importBatch`](#LoroDoc.importBatch) - Exchange incremental updates +- **Stream updates**: [`subscribeLocalUpdates`](#LoroDoc.subscribeLocalUpdates) - Send changes over WebSocket/WebRTC +- **Set unique peer ID**: [`setPeerId`](#LoroDoc.setPeerId) - Ensure each client has a unique identifier +- **Handle conflicts**: Automatic - All Loro data types are CRDTs that merge concurrent edits + +**Rich Text Editing** + +- **Create rich text**: [`getText`](#LoroDoc.getText) - Initialize a collaborative text container +- **Edit text**: [`insert`](#LoroText.insert), [`delete`](#LoroText.delete), [`applyDelta`](#LoroText.applyDelta) +- **Apply formatting**: [`mark`](#LoroText.mark) - Add bold, italic, links, custom styles +- **Copy styled snippets**: [`sliceDelta`](#LoroText.sliceDelta) - Get a Delta for a range (UTF-16; use `sliceDeltaUtf8` for byte offsets) +- **Track cursor positions**: [`getCursor`](#LoroText.getCursor) + [`getCursorPos`](#LoroDoc.getCursorPos) - Stable positions across edits +- **Configure styles**: [`configTextStyle`](#LoroDoc.configTextStyle) - Define expand behavior for marks + +**Data Structures** + +- **Ordered lists**: [`getList`](#LoroDoc.getList) - Arrays with [`push`](#LoroList.push), [`insert`](#LoroList.insert), [`delete`](#LoroList.delete) +- **Key-value maps**: [`getMap`](#LoroDoc.getMap) - Objects with [`set`](#LoroMap.set), [`get`](#LoroMap.get), [`delete`](#LoroMap.delete) +- **Hierarchical trees**: [`getTree`](#LoroDoc.getTree) - File systems, nested comments with [`createNode`](#LoroTree.createNode), [`move`](#LoroTree.move) +- **Reorderable lists**: [`getMovableList`](#LoroDoc.getMovableList) - Drag-and-drop with [`move`](#LoroMovableList.move), [`set`](#LoroMovableList.set) +- **Counters**: [`getCounter`](#LoroDoc.getCounter) - Distributed counters with [`increment`](#LoroCounter.increment) + +**Ephemeral State & Presence** + +- **User presence**: [`EphemeralStore`](#ephemeralstore) - Share cursor positions, selections, user status (not persisted) +- **Cursor syncing**: Use [`EphemeralStore.set`](#EphemeralStore.set) with cursor data from [`getCursor`](#LoroText.getCursor) +- **Live indicators**: Track who's online, typing indicators, mouse positions +- **Important**: EphemeralStore is a separate CRDT without history - perfect for temporary state that shouldn't persist + +**Version Control & History** + +- **Undo/redo**: [`UndoManager`](#undomanager) - Local undo of user's own edits +- **Time travel**: [`checkout`](#LoroDoc.checkout) to any [`frontiers`](#LoroDoc.frontiers) - Debug or review history +- **Version tracking**: [`version`](#LoroDoc.version), [`frontiers`](#LoroDoc.frontiers), [`versionVector`](#LoroDoc.versionVector) +- **Fork documents**: [`fork`](#LoroDoc.fork) or [`forkAt`](#LoroDoc.forkAt) - Create branches for experimentation +- **Merge branches**: [`import`](#LoroDoc.import) - Combine changes from forked documents + +**Performance & Storage** + +- **Incremental updates**: [`export`](#LoroDoc.export) from specific [`version`](#LoroDoc.version) - Send only changes +- **Compact history**: [`export`](#LoroDoc.export) with `mode: "snapshot"` - Full state with compressed history +- **Shallow snapshots**: [`export`](#LoroDoc.export) with `mode: "shallow-snapshot"` - State without partial history (see [Shallow Snapshots](/docs/concepts/shallow_snapshots)) + +## Basic Usage + +```typescript twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +// Subscribe to changes +const unsubscribe = doc.subscribe((event) => { + console.log("Document changed:", event); +}); + +// Export updates for synchronization +const updates = doc.export({ mode: "update" }); +``` + +## LoroDoc + +The `LoroDoc` class manages containers, sync, versions, and events. + +**Constructor** + +```typescript no_run +new LoroDoc(); +``` + +Creates a new Loro document with a randomly generated peer ID. + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +``` + +**Static Methods** + + +```typescript no_run +static fromSnapshot(snapshot: Uint8Array): LoroDoc +``` + + +Creates a new LoroDoc from a snapshot. This is useful for loading a document from a previously exported snapshot. + +**Parameters:** + +- `snapshot` - Binary snapshot data + +**Returns:** A new LoroDoc instance + +**Example:** + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; + +// Assume we have a snapshot from a previous export +const prevDoc = new LoroDoc(); +prevDoc.getText("text").insert(0, "Hello"); +const snapshot: Uint8Array = prevDoc.export({ mode: "snapshot" }); + +const doc = LoroDoc.fromSnapshot(snapshot); +``` + + + +**Properties** + + +```typescript no_run +readonly peerId: bigint +``` + + +Gets the peer ID of the current writer as a bigint. + +**See Also:** [PeerID Management](/docs/concepts/peerid_management) + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const id = doc.peerId; +``` + + + + +```typescript no_run +readonly peerIdStr: `${number}` +``` + + +Gets the peer ID as a decimal string. + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const idStr = doc.peerIdStr; +``` + + + +### Configuration Methods + + +```typescript no_run +setPeerId(peer: number | bigint | `${number}`): void +``` + + +Sets the peer ID for this document. It must be a number, a BigInt, or a decimal string that fits into an unsigned 64-bit integer. +See [PeerID Management](/docs/concepts/peerid_management) for why uniqueness matters in distributed systems. + +**Parameters:** + +- `peer` - Peer ID as number, bigint, or decimal string + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setPeerId("42"); +``` + +**⚠️ Critical Pitfall:** Never let two parallel peers (e.g., multiple tabs/devices) share the same PeerID — it creates duplicate op IDs and causes document divergence. Common mistakes: + +- Don't assign a fixed PeerId to a user (users have multiple devices) +- Don't assign a fixed PeerId to a device (multiple tabs can open the same document) +- If you must reuse PeerIDs, enforce single ownership with strict locking mechanisms +- Best practice: Use random IDs (default behavior) unless you have a strong reason not to + +See [PeerID reuse](/docs/tutorial/tips) for safe reuse patterns. + + + + +```typescript no_run +setRecordTimestamp(auto_record: boolean): void +``` + + +Configures whether to automatically record timestamps for changes. Timestamps use Unix time (seconds since epoch). Learn more about storing timestamps and typical use cases in [Storing Timestamps](/docs/advanced/timestamp). + +**Parameters:** + +- `auto_record` - Whether to automatically record timestamps + +**⚠️ Important:** This setting doesn't persist in exported Updates or Snapshots. You must reapply this configuration each time you initialize a document. + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setRecordTimestamp(true); +``` + + + + +```typescript no_run +setChangeMergeInterval(interval: number): void +``` + + +Sets the interval in milliseconds for merging continuous local changes into a single change record. In Loro, multiple low-level operations are grouped into higher-level Changes for readability and syncing. See [Operations and Changes](/docs/concepts/operations_changes). + +**Parameters:** + +- `interval` - Merge interval in milliseconds + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setChangeMergeInterval(1000); // Merge changes within 1 second +``` + + + + +```typescript no_run +configTextStyle(styles: StyleConfig): void +``` + + +Configures the behavior of text styles (marks) in rich text containers. Marks can expand when edits happen at their edges (before/after/both/none). For a primer on rich text and marks in Loro, see [Text](/docs/tutorial/text). + +**Parameters:** + +- `styles` - Configuration object mapping style names to their config + +**StyleConfig Type:** + +```typescript no_run +type StyleConfig = Record< + string, + { + expand?: "after" | "before" | "both" | "none"; + } +>; +``` + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.configTextStyle({ + bold: { expand: "after" }, + italic: { expand: "none" }, + link: { expand: "none" }, +}); +``` + + + + +```typescript no_run +configDefaultTextStyle(style?: { expand: "after" | "before" | "both" | "none" }): void +``` + + +Configures the default text style for the document when using LoroText. If undefined is provided, the default style is reset. + +**Parameters:** + +- `style` - Default style configuration (optional) + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.configDefaultTextStyle({ expand: "after" }); +``` + + + +### Container Access Methods + +**📝 Note:** Creating root containers (e.g., `doc.getText("...")`) does not record operations; nested container creation (e.g., `map.setContainer(...)`) does. + +**⚠️ Pitfall:** Avoid concurrent creation of regular child containers with the same key in LoroMaps. Instead of: + +```ts no_run +// Dangerous - concurrent peers can create different child container IDs +doc.getMap("user").getOrCreateContainer(userId, new LoroMap()); +``` + +Use a mergeable child container when the child should be identified by its logical position: + +```ts no_run +doc.getMap("user").ensureMergeableMap(userId); +``` + +Alternatively, use a unique root container if that better fits your model: + +```ts no_run +doc.getMap("user." + userId); +``` + + +```typescript no_run +getText(name: string): LoroText +``` + + + +Gets or creates a text container with the given name. New to LoroText and marks? See [Text](/docs/tutorial/text). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroText` instance + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("content"); +text.insert(0, "Hello"); +``` + + + + +```typescript no_run +getList(name: string): LoroList +``` + + + +Gets or creates a list container with the given name. Unsure whether to use List or MovableList? See [List and Movable List](/docs/tutorial/list) and the type selection guide [Choosing CRDT Types](/docs/concepts/choose_crdt_type). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroList` instance + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const list = doc.getList("items"); +list.push("Item 1"); +``` + + + + +```typescript no_run +getMap(name: string): LoroMap +``` + + +Gets or creates a map container with the given name. See [Map](/docs/tutorial/map) for basics and patterns. + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroMap` instance + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const map = doc.getMap("settings"); +map.set("theme", "dark"); +``` + + + + +```typescript no_run +getTree(name: string): LoroTree +``` + + +Gets or creates a tree container with the given name. Learn about hierarchical editing and moves in [Tree](/docs/tutorial/tree). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroTree` instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const tree = doc.getTree("fileSystem"); +const root = tree.createNode(); +``` + + + + +```typescript no_run +getCounter(name: string): LoroCounter +``` + + +Gets or creates a counter container with the given name. Counters are special CRDTs that sum concurrent increments; see [Counter](/docs/tutorial/counter). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroCounter` instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const counter = doc.getCounter("likes"); +counter.increment(1); +``` + + + + +```typescript no_run +getMovableList(name: string): LoroMovableList +``` + + +Gets or creates a movable list container with the given name. MovableList is designed for reordering with concurrent moves. See [List and Movable List](/docs/tutorial/list) and [Choosing CRDT Types](/docs/concepts/choose_crdt_type). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroMovableList` instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const movableList = doc.getMovableList("tasks"); +movableList.push("Task 1"); +movableList.push("Task 2"); +movableList.push("Task 3"); +movableList.move(0, 2); // Move first item to third position +``` + + + + +```typescript no_run +getContainerById(id: ContainerID): Container | undefined +``` + + +Gets a container by its unique ID. Container IDs (CID) uniquely reference containers across updates; see [Container ID](/docs/advanced/cid) and [Container](/docs/concepts/container). + +**Parameters:** + +- `id` - The container ID + +**Returns:** The container instance or undefined if not found + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("text"); +const textId = text.id; +const sameText = doc.getContainerById(textId); +``` + + + +### Import/Export Methods + + +```typescript no_run +export(mode?: ExportMode): Uint8Array +``` + + +Exports the document in various formats for synchronization or persistence. For a walkthrough of export modes—snapshot, update, shallow-snapshot, and updates-in-range—see [Export Mode](/docs/tutorial/encoding). Shallow snapshots remove history while keeping current state; see [Shallow Snapshots](/docs/concepts/shallow_snapshots). VersionVector and Frontiers are two ways to represent versions; see [Version Vector](/docs/concepts/version_vector) and [Frontiers](/docs/concepts/frontiers). + +**Parameters:** + +- `mode` - Export configuration (optional) + +**ExportMode Options:** + +```typescript no_run +type ExportMode = + | { mode: "snapshot" } + | { mode: "update"; from?: VersionVector } + | { mode: "shallow-snapshot"; frontiers: Frontiers } + | { mode: "updates-in-range"; spans: { id: OpId; len: number }[] }; +``` + +**Returns:** Encoded binary data + +**⚠️ Important Notes:** + +- **Shallow snapshots**: Cannot import updates from before the shallow start point. Peers can only sync if they have versions after this point. +- **Auto-commit**: The document automatically commits pending operations before export. +- **Performance**: Export new snapshots periodically to reduce import times for new peers. + +**Examples:** + +```typescript no_run +import { LoroDoc, VersionVector } from "loro-crdt"; + +const doc = new LoroDoc(); +// ... make some changes to the document ... + +// Export full snapshot +const snapshot = doc.export({ mode: "snapshot" }); + +// Export updates from a specific version +const lastSyncVersion = doc.version(); // Get current version +// ... make more changes ... +const updates = doc.export({ + mode: "update", + from: lastSyncVersion, +}); + +// Export shallow snapshot at current version +const shallowSnapshot = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.frontiers(), +}); +``` + + + + +```typescript no_run +import(data: Uint8Array): ImportStatus +``` + + +Imports updates or snapshots into the document. Returns an `ImportStatus` describing which peer ranges were applied or are pending. See [Sync](/docs/tutorial/sync) and [Import Status](/docs/concepts/import_status) for how Loro handles out-of-order and partial updates. + +**Parameters:** + +- `data` - Binary data or another LoroDoc to import from + +**⚠️ Important:** LoroDoc will automatically commits pending operations before import. If the doc is in detached mode, the imported operations are recorded into OpLog but not applied to DocState until you call `attach()`, see [Attached vs Detached States](/docs/concepts/attached_detached) adn [OpLog and DocState](/docs/concepts/oplog_docstate). + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Receive updates from another peer (e.g., via network) +const otherDoc = new LoroDoc(); +otherDoc.getText("text").insert(0, "Hello"); +const updates: Uint8Array = otherDoc.export({ mode: "update" }); + +// Import binary updates +const status = doc.import(updates); +console.log(status.success); +``` + + + + +```typescript no_run +importBatch(data: Uint8Array[]): ImportStatus +``` + + +Efficiently imports multiple updates in a single batch operation. See [Batch Import](/docs/advanced/import_batch) for performance considerations and usage. + +**Parameters:** + +- `data` - Array of binary updates + +**Example:** + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +declare const update1: Uint8Array; +declare const update2: Uint8Array; +declare const update3: Uint8Array; + +// Usage example: +const updates = [update1, update2, update3]; +const status = doc.importBatch(updates); +``` + + + + +```typescript no_run +exportJsonUpdates(start?: VersionVector, end?: VersionVector, withPeerCompression?: boolean): JsonSchema +``` + + +Exports updates in JSON format for debugging or alternative storage. See [Export Mode](/docs/tutorial/encoding) for format details and trade-offs. + +**Parameters:** + +- `start` - Starting version (optional) +- `end` - Ending version (optional) + +**Returns:** JSON representation of updates + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const jsonUpdates = doc.exportJsonUpdates(); +console.log(JSON.stringify(jsonUpdates, null, 2)); +``` + + + + +```typescript no_run +importJsonUpdates(json: string | JsonSchema): void +``` + + +Imports updates from JSON format. Useful for debugging, migration, or custom storage layers; see [Export Mode](/docs/tutorial/encoding). + +**Parameters:** + +- `json` - JSON string or object containing updates + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const otherDoc = new LoroDoc(); +otherDoc.getText("text").insert(0, "Hello"); +const jsonStr = otherDoc.exportJsonUpdates(); +doc.importJsonUpdates(jsonStr); +``` + + + +## Versioning + +Work with the history DAG using frontiers (heads) and version vectors. Switch, branch, and merge versions safely without manual conflict resolution. See [Versioning Deep Dive](/docs/advanced/version_deep_dive) and [Attached vs Detached States](/docs/concepts/attached_detached). + +### Version Control Methods + + +```typescript no_run +checkout(frontiers: Frontiers): void +``` + + +Checks out the document to a specific version, making it read-only at that point in history. This is the core of time travel; see [Time Travel](/docs/tutorial/time_travel) and [Version](/docs/tutorial/version). + +**Parameters:** + +- `frontiers` - Array of OpIds representing the target version + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +// Make some changes... +doc.checkout(frontiers); // Go back to previous version +``` + + + + +**⚠️ Important:** In Loro 1.0, `version()`/`frontiers()` include pending (uncommitted) local operations. + +**📝 Note:** After `checkout()`, the document enters "detached" mode and becomes read-only by default. Use `attach()` or `checkoutToLatest()` to return to editing mode. See [Version Deep Dive](/docs/advanced/version_deep_dive) and [Attached vs Detached States](/docs/concepts/attached_detached). + + + + +```typescript no_run +checkoutToLatest(): void +``` + + +Returns the document to the latest version after a checkout. Related concepts: [Frontiers](/docs/concepts/frontiers) and [Version Vector](/docs/concepts/version_vector). + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.checkoutToLatest(); +``` + + + + +```typescript no_run +attach(): void +``` + + +Attaches the document to track latest changes after being detached. See [Attached vs Detached States](/docs/concepts/attached_detached) for how Loro separates current state from history. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.attach(); +``` + + + + +```typescript no_run +detach(): void +``` + + +Detaches the document from tracking latest changes, freezing it at current version. See [Attached vs Detached States](/docs/concepts/attached_detached). + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.detach(); +``` + + + + +```typescript no_run +fork(): LoroDoc +``` + + +Creates a new document that is a fork of the current one with a new peer ID. Forking is useful for branching workflows; see [Version](/docs/tutorial/version). + +**Returns:** A new LoroDoc instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const forkedDoc = doc.fork(); +``` + + + + +```typescript no_run +forkAt(frontiers: Frontiers): LoroDoc +``` + + +Creates a fork at a specific version in history. Learn more about versions, DAG history, and heads in [Version Deep Dive](/docs/advanced/version_deep_dive). + +**Parameters:** + +- `frontiers` - The version to fork from + +**Returns:** A new LoroDoc instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const forkedDoc = doc.forkAt(frontiers); +``` + + + +## Events & Transactions + +React to changes and group local operations into transactions. Starting in v1.8, events are delivered synchronously; older releases require awaiting a microtask. See [Event Handling](/docs/tutorial/event) and [Transaction Model](/docs/concepts/transaction_model). + +### Subscription Methods + + +```typescript no_run +subscribe(listener: (event: LoroEventBatch) => void): () => void +``` + + +Subscribes to all document changes. See [Event Handling](/docs/tutorial/event) for the event model and best practices. + +**Parameters:** + +- `listener` - Callback function that receives change events + +**Returns:** Unsubscribe function + +**Event Structure:** + +```typescript no_run +interface LoroEventBatch { + by: "local" | "import" | "checkout"; + origin?: string; + currentTarget?: ContainerID; + events: LoroEvent[]; + from: Frontiers; + to: Frontiers; +} +``` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const unsubscribe = doc.subscribe((event) => { + console.log("Change type:", event.by); + event.events.forEach((e) => { + console.log("Container changed:", e.target); + console.log("Diff:", e.diff); + }); +}); + +// Later: unsubscribe(); +``` + +**⚠️ Important:** Events are emitted synchronously as of v1.8. If you are pinned to `<=1.7.x`, await a microtask before reading the batch. + +```ts no_run +doc.commit(); +// Events have already been delivered in v1.8+ +// await Promise.resolve(); // Only needed on <=1.7.x +``` + +**📝 Note:** Multiple operations before a commit are batched into a single event. See [Event Handling](/docs/tutorial/event). + + + + +```typescript no_run +subscribeLocalUpdates(f: (bytes: Uint8Array) => void): () => void +``` + + +Subscribes only to local changes, useful for syncing with remote peers. This is typically wired to your transport layer; see [Sync](/docs/tutorial/sync). + +**Parameters:** + +- `f` - Callback that receives binary updates + +**Returns:** Unsubscribe function + +**Example:** + +```ts no_run threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +declare const websocket: { send: (data: Uint8Array) => void }; + +// Usage example: +const unsubscribe = doc.subscribeLocalUpdates((updates) => { + // Send updates to remote peers + websocket.send(updates); +}); +``` + + + + +```typescript no_run +subscribeFirstCommitFromPeer(f: (e: { peer: PeerID }) => void): () => void +``` + + +Subscribes to the first commit from each peer, useful for tracking peer metadata. + +**Parameters:** + +- `f` - Callback that receives peer information + +**Returns:** Unsubscribe function + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.subscribeFirstCommitFromPeer(({ peer }) => { + // Store peer metadata + doc.getMap("peers").set(peer, { + joinedAt: Date.now(), + name: `User ${peer}`, + }); +}); +``` + + + +### Transaction Methods + + +```typescript no_run +commit(options?: { origin?: string, message?: string, timestamp?: number }): void +``` + + +Commits pending changes as a single transaction. A transaction groups operations into a Change; see [Operations and Changes](/docs/concepts/operations_changes). + +**⚠️ Critical Distinction:** Loro transactions are NOT ACID database transactions: + +- No rollback capability +- No isolation guarantees +- Purpose: Bundle local operations for event batching and history grouping +- Many operations (import/export/checkout) trigger implicit commits + +See [Transaction Model](/docs/concepts/transaction_model). + +**Parameters:** + +- `options` - Optional commit configuration + - `message` - Commit message (persisted in the document like a git commit message, visible to all peers after sync) + - `origin` - Origin identifier (local only - used for marking local events, remote peers won't see this) + - `timestamp` - Unix timestamp in seconds (see [Storing Timestamps](/docs/advanced/timestamp)) + +**Important distinction:** + +- `message` is persisted in the document's history and will be synchronized to all peers, similar to git commit messages +- `origin` is only used locally for filtering events (e.g., excluding certain origins from undo) and is NOT synchronized to remote peers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.commit({ + message: "Updated document title", // Persisted & synced to all peers + origin: "user-action", // Local only, for event filtering + timestamp: Math.floor(Date.now() / 1000), +}); +``` + + + +### Query Methods + + +```typescript no_run +toJSON(): Value +``` + + +Converts the entire document to a JSON-compatible value. If you prefer a structure where sub-containers are referenced by ID (for privacy or streaming), use getShallowValue(); see [Shallow Snapshots](/docs/concepts/shallow_snapshots). + +**Returns:** JSON representation of the document + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const json = doc.toJSON(); +console.log(JSON.stringify(json, null, 2)); +``` + + + + +```typescript no_run +getShallowValue(): Record +``` + + +Gets a shallow representation where sub-containers are represented by their IDs. This is helpful when you want to share structure without history; see [Shallow Snapshots](/docs/concepts/shallow_snapshots). + +**Returns:** Shallow JSON value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const shallow = doc.getShallowValue(); +// Sub-containers appear as: "cid:..." +``` + + + + +```typescript no_run +getDeepValueWithID(): any +``` + + +Gets the deep value of the document with container IDs preserved. This is useful when you need to traverse the document structure while maintaining references to container IDs. + +**Returns:** Document value with container IDs + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const deepValue = doc.getDeepValueWithID(); +``` + + + + +```typescript no_run +version(): VersionVector +``` + + +Gets the current version vector of the document. Version vectors track how much data from each peer you’ve seen; see [Version Vector](/docs/concepts/version_vector). + +**Returns:** Map from PeerID to counter + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.version(); +console.log(vv.toJSON()); +``` + + + + +```typescript no_run +frontiers(): Frontiers +``` + + +Gets the current frontiers (heads) of the document. Frontiers are a compact representation of a version; see [Frontiers](/docs/concepts/frontiers) for when to use them instead of version vectors. + +**📝 Note:** Frontiers are a compact version representation. + +**⚠️ Limitation:** When you have a Frontier pointing to operations you don't know about, you cannot determine the complete set of operation IDs included in that version. Version Vectors don't have this limitation but are more verbose. See [Frontiers](/docs/concepts/frontiers) for trade-offs. + +**Returns:** Array of OpIds + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +// Can be used for checkouts or shallow snapshots +``` + + + + +```typescript no_run +diff(from: Frontiers, to: Frontiers, for_json?: boolean): [ContainerID, Diff | JsonDiff][] +``` + + +Calculates differences between two versions. Understanding how Loro computes diffs benefits from the history DAG model; see [Version Deep Dive](/docs/advanced/version_deep_dive). + +**Parameters:** + +- `from` - Starting frontiers +- `to` - Ending frontiers +- `for_json` - If true, returns JsonDiff format (default: true) + +**Returns:** Array of container IDs and their diffs + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const fromFrontiers = doc.frontiers(); +// Make changes... +const toFrontiers = doc.frontiers(); + +const diffs = doc.diff(fromFrontiers, toFrontiers); +diffs.forEach(([containerId, diff]) => { + console.log(`Container ${containerId} changed:`, diff); +}); +``` + + + +### Pre-Commit Hook + + +```typescript no_run +subscribePreCommit(f: (e: { changeMeta: Change, origin: string, modifier: ChangeModifier }) => void): () => void +``` + + +Subscribe to the pre-commit event. You can modify the message and timestamp of the next change. This hook runs right before a Change is recorded; see [Transaction Model](/docs/concepts/transaction_model) and [Operations and Changes](/docs/concepts/operations_changes). + +Pitfall: `commit()` can be triggered implicitly by `import`, `export`, and `checkout`. Use this hook to attach metadata even for those implicit commits. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const unsubscribe = doc.subscribePreCommit(({ modifier }) => { + modifier + .setMessage("Tagged by pre-commit") + .setTimestamp(Math.floor(Date.now() / 1000)); +}); +doc.getText("text").insert(0, "Hello"); +doc.commit(); +unsubscribe(); +``` + + + +### Cursor Utilities + + +```typescript no_run +getCursorPos(cursor: Cursor): { update?: Cursor, offset: number, side: Side } +``` + + +Resolve a stable `Cursor` to an absolute position. Cursors remain valid across concurrent edits; see [Cursor and Stable Positions](/docs/concepts/cursor_stable_positions) and [Cursor tutorial](/docs/tutorial/cursor). The side controls affinity when the cursor sits at an insertion boundary. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "abc"); + +// Get cursor at position 1 +const c0 = text.getCursor(1); +const pos = doc.getCursorPos(c0!); +console.log(pos.offset); // 1 +``` + + + +### Pending Operations + + +```typescript no_run +getUncommittedOpsAsJson(): JsonSchema | undefined +``` + + +Get pending operations from the current transaction in JSON format. Useful for debugging what will be included in the next Change; see [Transaction Model](/docs/concepts/transaction_model). + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Hello"); +const pending = doc.getUncommittedOpsAsJson(); +doc.commit(); +const none = doc.getUncommittedOpsAsJson(); // undefined after commit +``` + + + +### Change Graph & History + +These APIs traverse the history DAG of changes (ancestors/descendants, spans). If this sounds unfamiliar, start with Loro's [Versioning Deep Dive](/docs/advanced/version_deep_dive) and the [Event Graph Walker](/docs/concepts/event_graph_walker). + + +```typescript no_run +travelChangeAncestors(ids: OpId[], f: (change: Change) => boolean): void +``` + + +Visit ancestors of the given changes in causal order. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "Hello"); +doc.commit(); +const head = doc.frontiers(); +doc.travelChangeAncestors(head, (change) => { + console.log(change.peer, change.counter); + return true; // continue +}); +``` + + + + + ```typescript no_run findIdSpansBetween(from: Frontiers, to: Frontiers): + VersionVectorDiff ``` + +Find the op id spans that lie between two versions. + + +```typescript no_run +exportJsonInIdSpan(idSpan: { peer: PeerID, counter: number, length: number }): JsonChange[] +``` + + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const a = new LoroDoc(); +const b = new LoroDoc(); + +// Usage example: +a.getText("text").update("Hello"); +a.commit(); +const snapshot = a.export({ mode: "snapshot" }); +let printed: any; +b.subscribe((e) => { +const spans = b.findIdSpansBetween(e.from, e.to); +const changes = b.exportJsonInIdSpan(spans.forward[0]); +printed = changes; +}); +b.import(snapshot); + +```` + + + +```typescript no_run +getChangedContainersIn(id: OpId, len: number): ContainerID[] +```` + + + +Get container IDs modified in the given ID range. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getList("list").insert(0, 1); +doc.commit(); +const head = doc.frontiers()[0]; +const containers = doc.getChangedContainersIn(head, 1); +``` + + + +### Revert & Apply Diff + + +```typescript no_run +revertTo(frontiers: Frontiers): void +``` + + +Revert the document to a given version by generating inverse operations. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setPeerId("1"); +const t = doc.getText("text"); +t.update("Hello"); +doc.commit(); +doc.revertTo([{ peer: "1", counter: 1 }]); +``` + + + + +```typescript no_run +applyDiff(diff: [ContainerID, Diff | JsonDiff][]): void +``` + + +Apply a batch of diffs to the document. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc1 = new LoroDoc(); +const doc2 = new LoroDoc(); + +// Usage example: +doc1.getText("text").insert(0, "Hello"); +const diff = doc1.diff([], doc1.frontiers()); +doc2.applyDiff(diff); +``` + +**Workflow example (squash-like diffs):** For PR-style reviews, combine `diff` and `applyDiff` to send a compact change set between a base version and a new version. Operations that cancel out (insert + delete) are compressed away. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const baseDoc = new LoroDoc(); +const baseText = baseDoc.getText("text"); +baseText.insert(0, "hello world"); + +// Fork to make isolated edits +const newDoc = baseDoc.fork(); +const newText = newDoc.getText("text"); +newText.insert(0, "abc"); +newText.delete(0, 4); + +const diff = newDoc.diff(baseDoc.frontiers(), newDoc.frontiers()); +console.log(diff); +// [ +// [ +// "cid:root-text:Text", +// { type: "text", diff: [ { delete: 1 } ] } +// ] +// ] + +baseDoc.applyDiff(diff); +console.log(baseDoc.toJSON()); +// { text: "ello world" } +``` + + + +### Detached Editing + + +```typescript no_run +setDetachedEditing(enable: boolean): void +``` + + +Enables or disables detached editing mode. Detached editing lets you stage edits separate from the latest head; see [Attached vs Detached States](/docs/concepts/attached_detached). + +**Parameters:** + +- `enable` - Whether to enable detached editing + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setDetachedEditing(true); +``` + + + + +```typescript no_run +isDetachedEditingEnabled(): boolean +``` + + +Checks if detached editing mode is enabled. + +**Returns:** True if detached editing is enabled + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const enabled = doc.isDetachedEditingEnabled(); +``` + + + + +```typescript no_run +isDetached(): boolean +``` + + +Checks if the document is currently detached. + +**Returns:** True if document is detached + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +console.log(doc.isDetached()); +``` + + + +### Commit Options Helpers + + +```typescript no_run +setNextCommitMessage(msg: string): void +``` + + +Sets the message for the next commit. + +**Parameters:** + +- `msg` - Commit message + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitMessage("User action"); +``` + + + + +```typescript no_run +setNextCommitOrigin(origin: string): void +``` + + +Sets the origin for the next commit. + +**Parameters:** + +- `origin` - Origin identifier + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitOrigin("ui"); +``` + + + + +```typescript no_run +setNextCommitTimestamp(timestamp: number): void +``` + + +Sets the timestamp for the next commit. + +**Parameters:** + +- `timestamp` - Unix timestamp in seconds + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitTimestamp(Math.floor(Date.now() / 1000)); +``` + + + + +```typescript no_run +setNextCommitOptions(options: { origin?: string, timestamp?: number, message?: string }): void +``` + + +Sets multiple options for the next commit. + +**Parameters:** + +- `options` - Commit options object + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitOptions({ origin: "ui", message: "batch" }); +doc.getText("text").insert(0, "Hi"); +doc.commit(); +``` + + + + +```typescript no_run +clearNextCommitOptions(): void +``` + + +Clears all pending commit options. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.clearNextCommitOptions(); +``` + + + +### Version & Frontier Utilities + + +```typescript no_run +frontiersToVV(frontiers: Frontiers): VersionVector +``` + + +Converts frontiers to a version vector. + +**Parameters:** + +- `frontiers` - Frontiers to convert + +**Returns:** Version vector representation + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const vv = doc.frontiersToVV(frontiers); +``` + + + + +```typescript no_run +vvToFrontiers(vv: VersionVector): Frontiers +``` + + +Converts a version vector to frontiers. + +**Parameters:** + +- `vv` - Version vector to convert + +**Returns:** Frontiers representation + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.version(); +const frontiers = doc.vvToFrontiers(vv); +``` + + + + +```typescript no_run +oplogVersion(): VersionVector +``` + + +Gets the oplog version vector. + +**Returns:** Oplog version vector + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.oplogVersion(); +``` + + + + +```typescript no_run +oplogFrontiers(): Frontiers +``` + + +Gets the oplog frontiers. + +**Returns:** Oplog frontiers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.oplogFrontiers(); +``` + + + + +```typescript no_run +cmpWithFrontiers(frontiers: Frontiers): -1 | 0 | 1 +``` + + +Compares current document state with given frontiers. + +**Parameters:** + +- `frontiers` - Frontiers to compare with + +**Returns:** -1 if behind, 0 if equal, 1 if ahead + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const cmp = doc.cmpWithFrontiers(frontiers); +``` + + + + +```typescript no_run +cmpFrontiers(a: Frontiers, b: Frontiers): -1 | 0 | 1 | undefined +``` + + +Compares two frontiers. + +**Parameters:** + +- `a` - First frontiers +- `b` - Second frontiers + +**Returns:** -1 if a < b, 0 if equal, 1 if a > b, undefined if incomparable + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const f1 = doc.frontiers(); +const f2 = doc.frontiers(); +const cmp = doc.cmpFrontiers(f1, f2); +``` + + + +### JSONPath & Path Queries + +Use simple path strings and JSONPath to fetch nested values and containers. Paths are formed from root container names and keys (e.g., map/key or list/0). For container IDs, see [Container ID](/docs/advanced/cid). + + +```typescript no_run +getByPath(path: string): Value | Container | undefined +``` + + +Gets a value or container by its path. + +**Parameters:** + +- `path` - Path string (e.g., "map/key") + +**Returns:** Value or container at the path, or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("key", 1); + +// Usage example: +const value = doc.getByPath("map/key"); +``` + + + + +```typescript no_run +getPathToContainer(id: ContainerID): (string | number)[] | undefined +``` + + +Gets the path to a container by its ID. + +**Parameters:** + +- `id` - Container ID + +**Returns:** Array representing the path, or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const path = doc.getPathToContainer(map.id); +``` + + + + +```typescript no_run +JSONPath(jsonpath: string): any[] +``` + + +Queries the document using JSONPath syntax. + +**Parameters:** + +- `jsonpath` - JSONPath query string + +**Returns:** Array of matching values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("key", 1); + +// Usage example: +const results = doc.JSONPath("$.map"); +``` + + + +### Shallow Doc Utilities + +These helpers relate to shallow snapshots and redaction. If you need a refresher on what “shallow” means, see [Shallow Snapshots](/docs/concepts/shallow_snapshots). + + +```typescript no_run +shallowSinceVV(): VersionVector +``` + + +Gets the version vector since which the document is shallow. + +**Returns:** Version vector + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.shallowSinceVV(); +``` + + + + +```typescript no_run +shallowSinceFrontiers(): Frontiers +``` + + +Gets the frontiers since which the document is shallow. + +**Returns:** Frontiers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.shallowSinceFrontiers(); +``` + + + + +```typescript no_run +isShallow(): boolean +``` + + +Checks if the document is shallow. + +**Returns:** True if document is shallow + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const shallow = doc.isShallow(); +``` + + + + +```typescript no_run +setHideEmptyRootContainers(hide: boolean): void +``` + + +Controls whether empty root containers are hidden in JSON output. + +**Parameters:** + +- `hide` - Whether to hide empty root containers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setHideEmptyRootContainers(true); +// Now empty roots are hidden in toJSON() +``` + + + + +```typescript no_run +deleteRootContainer(cid: ContainerID): void +``` + + +Deletes a root container. + +**Parameters:** + +- `cid` - Container ID to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +// Usage example: +doc.deleteRootContainer(map.id); +``` + + + + +```typescript no_run +hasContainer(id: ContainerID): boolean +``` + + +Checks if a container exists in the document. + +**Parameters:** + +- `id` - Container ID to check + +**Returns:** True if container exists + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const exists = doc.hasContainer(map.id); +``` + + + +### JSON Serialization with Replacer + + +```typescript no_run +toJsonWithReplacer(replacer: (key: string | number, value: Value | Container) => Value | Container | undefined): Value +``` + + +Customize JSON serialization of containers and values. + +**Parameters:** + +- `replacer` - Function to transform values during serialization + +**Returns:** Customized JSON value + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const json = doc.toJsonWithReplacer((key, value) => { + if (value instanceof LoroText) { + return value.toDelta(); + } + return value; +}); +``` + + + +### Stats & Introspection + + +```typescript no_run +debugHistory(): void +``` + + +Prints debug information about the document history. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.debugHistory(); +``` + + + + +```typescript no_run +changeCount(): number +``` + + +Gets the total number of changes in the document. + +**Returns:** Number of changes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const changes = doc.changeCount(); +``` + + + + +```typescript no_run +opCount(): number +``` + + +Gets the total number of operations in the document. + +**Returns:** Number of operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const ops = doc.opCount(); +``` + + + + +```typescript no_run +getAllChanges(): Map +``` + + +Gets all changes grouped by peer ID. + +**Returns:** Map of peer ID to changes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const changes = doc.getAllChanges(); +``` + + + + +```typescript no_run +getChangeAt(id: OpId): Change +``` + + +Gets a specific change by operation ID. + +**Parameters:** + +- `id` - Operation ID + +**Returns:** Change object + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "hello"); +doc.commit(); +const changes = doc.getAllChanges(); +const change = changes.get(doc.peerIdStr)?.[0]; +``` + + + + +```typescript no_run +getChangeAtLamport(peer_id: string, lamport: number): Change | undefined +``` + + +Gets a change by peer ID and Lamport timestamp. + +**Parameters:** + +- `peer_id` - Peer ID +- `lamport` - Lamport timestamp + +**Returns:** Change object or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "hello"); +const change = doc.getChangeAtLamport(doc.peerIdStr, 1); +``` + + + + +```typescript no_run +getOpsInChange(id: OpId): any[] +``` + + +Gets all operations in a specific change. + +**Parameters:** + +- `id` - Operation ID + +**Returns:** Array of operations + +**Example:** + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "hello"); +const changes = doc.getAllChanges(); +const ops = doc.getOpsInChange(changes[0].id); +``` + + + + +```typescript no_run +getPendingTxnLength(): number +``` + + +Gets the number of pending operations in the current transaction. + +**Returns:** Number of pending operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "x"); +console.log(doc.getPendingTxnLength()); +doc.commit(); +``` + + + +### Import/Export Utilities + + +```typescript no_run +decodeImportBlobMeta(blob: Uint8Array, check_checksum: boolean): ImportBlobMetadata +``` + + +Decodes metadata from an import blob. + +**Parameters:** + +- `blob` - Binary data to decode +- `check_checksum` - Whether to verify checksum + +**Returns:** Import blob metadata + +**Example:** + +```ts threeslash +import { LoroDoc, decodeImportBlobMeta } from "loro-crdt"; + +const doc = new LoroDoc(); +const updates = doc.export({ mode: "update" }); +const meta = decodeImportBlobMeta(updates, true); +``` + + + + +```typescript no_run +redactJsonUpdates(json: string | JsonSchema, version_range: any): JsonSchema +``` + + +Redacts JSON updates within a specified version range. + +Use this to safely remove accidentally leaked sensitive content from history while preserving structure. See [Tips: Redaction](/docs/tutorial/tips). + +**Parameters:** + +- `json` - JSON updates to redact +- `version_range` - Version range for redaction + +**Returns:** Redacted JSON schema + +**Example:** + +```ts threeslash +import { LoroDoc, redactJsonUpdates } from "loro-crdt"; + +const doc = new LoroDoc(); +const json = doc.exportJsonUpdates(); +const redacted = redactJsonUpdates(json, { [doc.peerIdStr]: [0, 999999] }); +``` + + + +--- + +## Container Types + +Common CRDT containers for modeling JSON-like structures. See [Choosing CRDT Types](/docs/concepts/choose_crdt_type) and [Composing CRDTs](/docs/tutorial/composition) for when to use each and how to nest them. + +### LoroText + +A rich text container supporting collaborative text editing with formatting. Supports overlapping marks (bold, italic, links) and +stable cursors. The merge semantics avoid interleaving artifacts under concurrency (Fugue + Eg-walker ideas); you use simple index +APIs and Loro handles index transformation. +See [Text](/docs/tutorial/text), [Eg-walker](/docs/advanced/event_graph_walker), and the rich text blog: https://loro.dev/blog/loro-richtext + +**⚠️ Critical: UTF-16 String Encoding** + +LoroText uses **UTF-16** encoding, matching JavaScript's native string encoding: + +- All standard methods (`insert()`, `delete()`, `mark()`, `slice()`, `charAt()`) use UTF-16 code unit indices +- `length` returns UTF-16 code units (same as JavaScript `string.length`) +- Use `insertUtf8()` and `deleteUtf8()` for UTF-8 byte-based operations when integrating with UTF-8 systems + +**⚠️ Common Pitfalls:** + +1. **Index Misalignment**: UTF-16 indices differ from visual character count +2. **Performance**: Cursor queries on deleted positions require history traversal - in that case, it will return a refreshed Cursor object that does not point to the deleted text + +**Example with emoji:** + +```typescript no_run +const text = doc.getText("text"); +text.insert(0, "Hello 😀 World"); +console.log(text.length); // 13 (emoji counts as 2) +console.log(text.toString()[6]); // ⚠️ Invalid - splits the emoji +text.delete(6, 2); // ✅ Correct - deletes entire emoji +text.delete(6, 1); // ❌ Wrong - corrupts the emoji + +// Safe iteration +text.iter((char) => { + console.log(char); // Each character handled correctly + return true; +}); +``` + +**📝 Text vs String in Maps:** + +- Use `LoroText` for collaborative text editing where all concurrent edits must be preserved +- Use regular strings in `LoroMap` for atomic values (URLs, IDs, hashes) where Last-Write-Wins is preferred +- Example: URLs should be strings in maps, not LoroText. Otherwise, the automatically merged result may be an invalid URL + + + ```typescript no_run insert(index: number, text: string): void ``` + + + +Inserts text at the specified position using UTF-16 code unit indices (same as JavaScript string indices). + +**Parameters:** + +- `index` - UTF-16 code unit position to insert at (0-based, same as JavaScript string index) +- `text` - Text to insert + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +// Usage example: +text.insert(0, "Hello "); +text.insert(6, "World"); +``` + + + + + ```typescript no_run delete(index: number, len: number): void ``` + + + +Deletes text from the specified position using UTF-16 code units. + +**Parameters:** + +- `index` - Starting UTF-16 code unit position (same as JavaScript string index) +- `len` - Number of UTF-16 code units to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello 😀 World"); +text.delete(6, 2); // Delete emoji (2 UTF-16 units) +text.delete(5, 1); // Delete space before World +``` + + + + +```typescript no_run +mark(range: { start: number, end: number }, key: string, value: Value): void +``` + + + +Applies formatting to a text range. Marks can be configured to expand/stop at edges via configTextStyle(); see [Text](/docs/tutorial/text) for mark behavior. + +**Parameters:** + +- `range` - The range to format +- `key` - Style attribute name +- `value` - Style value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +doc.configTextStyle({ bold: { expand: "after" } }); +text.mark({ start: 0, end: 5 }, "bold", true); +``` + + + + +```typescript no_run +unmark(range: { start: number, end: number }, key: string): void +``` + + +Removes formatting from a text range. For how conflicting edits on marks resolve, see [Text](/docs/tutorial/text). + +**Parameters:** + +- `range` - The range to unformat +- `key` - Style attribute to remove + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +text.mark({ start: 0, end: 5 }, "bold", true); +text.unmark({ start: 0, end: 5 }, "bold"); +``` + + + + +```typescript no_run +toDelta(): Delta[] +``` + + +Converts text to Delta format (Quill-compatible). + +**Returns:** Array of Delta operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +text.mark({ start: 0, end: 5 }, "bold", true); +const delta = text.toDelta(); +// [{ insert: "Hello", attributes: { bold: true } }, { insert: " World" }] +``` + + + + +```typescript no_run +sliceDelta(start: number, end: number): Delta[] +``` + + +Returns a Quill-style Delta for a subsection of the text, using UTF-16 indices. Useful for copying a styled span. Use `sliceDeltaUtf8` if you need UTF-8 byte offsets instead. + +**Parameters:** + +- `start` - Start UTF-16 code unit index (inclusive) +- `end` - End UTF-16 code unit index (exclusive) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +const doc = new LoroDoc(); +doc.configTextStyle({ + bold: { expand: "after" }, + comment: { expand: "none" }, +}); +const text = doc.getText("text"); + +text.insert(0, "Hello World!"); +text.mark({ start: 0, end: 5 }, "bold", true); +text.mark({ start: 6, end: 11 }, "comment", "greeting"); + +const snippet = text.sliceDelta(1, 8); +expect(snippet).toStrictEqual([ + { insert: "ello", attributes: { bold: true } }, + { insert: " " }, + { insert: "Wo", attributes: { comment: "greeting" } }, +]); +``` + + + + +```typescript no_run +sliceDeltaUtf8(start: number, end: number): Delta[] +``` + + +Returns a Quill-style Delta for a subsection of the text using **UTF-8 byte offsets**. Choose this when your offsets come from UTF-8 encoded buffers. + +**Parameters:** + +- `start` - Start byte offset (inclusive) +- `end` - End byte offset (exclusive) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hi 👋"); + +const enc = new TextEncoder(); +const start = enc.encode("Hi ").length; // 3 bytes +const end = enc.encode("Hi 👋").length; // 7 bytes + +const delta = text.sliceDeltaUtf8(start, end); +expect(delta).toStrictEqual([{ insert: "👋" }]); +``` + + + + +```typescript no_run +applyDelta(delta: Delta[]): void +``` + + +Applies Delta operations to the text. + +**Parameters:** + +- `delta` - Array of Delta operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.applyDelta([ + { insert: "Hello", attributes: { bold: true } }, + { insert: " World" }, +]); +``` + + + + +```typescript no_run +update(text: string, options?: { timeoutMs?: number; useRefinedDiff?: boolean }): void +``` + + +Updates the current text to the target text using Myers' diff algorithm. + +**Parameters:** + +- `text` - New text content +- `options` - Update options + - `timeoutMs` - Optional timeout for the diff computation + - `useRefinedDiff` - Use refined diff for better quality on long texts + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Hello"); +text.update("Hello World", { timeoutMs: 100 }); +``` + + + + +```typescript no_run +updateByLine(text: string, options?: { timeoutMs?: number; useRefinedDiff?: boolean }): void +``` + + +Line-based update that's faster for large texts (less precise than `update`). + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Line A\nLine C"); +text.updateByLine("Line A\nLine B\nLine C"); +``` + + + + +```typescript no_run +getCursor(pos: number, side?: Side): Cursor | undefined +``` + + +Gets a stable cursor position that survives edits. + +**Parameters:** + +- `pos` - Position in the text +- `side` - Cursor affinity (-1, 0, or 1) + +**Returns:** Cursor object or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +const cursor = text.getCursor(5); +// Cursor remains valid even after edits +``` + + + + +```typescript no_run +toString(): string +``` + + +Converts to plain text string. + +**Returns:** Plain text content + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +const plainText = text.toString(); +``` + + + + +```typescript no_run +charAt(pos: number): string +``` + + +Gets the character at a specific UTF-16 code unit position. + +**Parameters:** + +- `pos` - UTF-16 code unit position + +**Returns:** Character at position + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); +const char = text.charAt(1); // "e" +``` + + + + +```typescript no_run +slice(start: number, end: number): string +``` + + +Extracts a section of the text using UTF-16 code unit positions. + +**Parameters:** + +- `start` - Start UTF-16 code unit index +- `end` - End UTF-16 code unit index (exclusive) + +**Returns:** Sliced text + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello 😀 World"); + +const slice1 = text.slice(0, 5); // "Hello" +const slice2 = text.slice(6, 8); // "😀" (emoji spans 6-8) +const slice3 = text.slice(9, 14); // "World" +``` + + + + +```typescript no_run +splice(pos: number, len: number, s: string): string +``` + + +Replaces text at a position with new content. + +**Parameters:** + +- `pos` - Start position +- `len` - Length to delete +- `s` - String to insert + +**Returns:** Deleted text + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +// Usage example: +const deleted = text.splice(6, 5, "Loro"); // returns "World" +``` + + + + +```typescript no_run +push(s: string): void +``` + + +Appends text to the end of the document. + +**Parameters:** + +- `s` - String to append + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.push("Hello"); +text.push(" World"); +``` + + + + +```typescript no_run +iter(callback: (char: string) => boolean): void +``` + + +Iterates over each character in the text. + +**Parameters:** + +- `callback` - Function called for each character. Return false to stop iteration. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +text.iter((char) => { + console.log(char); + return true; // continue iteration +}); +``` + + + + +```typescript no_run +insertUtf8(index: number, content: string): void +``` + + +Inserts text at a UTF-8 byte index position. + +**Parameters:** + +- `index` - UTF-8 byte index +- `content` - Text to insert + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insertUtf8(0, "Hello"); +``` + + + + +```typescript no_run +deleteUtf8(index: number, len: number): void +``` + + +Deletes text at a UTF-8 byte index position. + +**Parameters:** + +- `index` - UTF-8 byte index +- `len` - Number of UTF-8 bytes to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +// Usage example: +text.deleteUtf8(6, 5); // Delete "World" +``` + + + + +```typescript no_run +getEditorOf(pos: number): PeerID | undefined +``` + + +Gets the peer ID of who last edited the character at a position. + +**Parameters:** + +- `pos` - Character position + +**Returns:** PeerID of last editor or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const editor = text.getEditorOf(0); +``` + + + + +```typescript no_run +kind(): "Text" +``` + + +Returns the container type. + +**Returns:** "Text" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +const type = text.kind(); // "Text" +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this text is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const text = list.insertContainer(0, doc.getText("nested")); + +// Usage example: +const parent = text.parent(); // Returns the list +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const text = new LoroText(); + +const attached = text.isAttached(); // false until attached to doc +``` + + + + +```typescript no_run +getAttached(): LoroText | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const text = new LoroText(); + +const attached = text.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +const deleted = text.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): string +``` + + +Gets the text content without marks. + +**Returns:** Plain text string + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const value = text.getShallowValue(); // "Hello" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the text to JSON representation. + +**Returns:** JSON value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const json = text.toJSON(); // "Hello" +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +const containerId = text.id; +``` + + + + +```typescript no_run +readonly length: number +``` + + +Gets the length of the text in UTF-16 code units (same as JavaScript's `string.length`). + +**⚠️ Important:** Emoji and other characters outside the Basic Multilingual Plane count as 2 UTF-16 units. This affects all index-based operations: + +```typescript no_run +text.insert(0, "👨‍👩‍👧‍👦"); // Family emoji +console.log(text.length); // 11 (not 1!) - complex emoji with ZWJ sequences +``` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); +console.log(text.length); // 5 + +text.insert(5, " 😀"); +console.log(text.length); // 8 (space + emoji which counts as 2) +``` + + + +### LoroList + +An ordered list container for collaborative arrays. Uses index-based APIs; under concurrency, Loro transforms indices +by replaying only the necessary portion of history (Eg-walker-inspired). See [List and Movable List](/docs/tutorial/list), +[Choosing CRDT Types](/docs/concepts/choose_crdt_type), and [Eg-walker](/docs/advanced/event_graph_walker). + +**⚠️ Important: List vs Map for Coordinates** + +```typescript no_run +// ❌ WRONG - Don't use List for coordinates +const coord = doc.getList("coord"); +coord.push(10); // x +coord.push(20); // y +// Concurrent updates can create [10, 20a, 20b] instead of [10, 20] + +// ✅ CORRECT - Use Map for coordinates +const coord = doc.getMap("coord"); +coord.set("x", 10); +coord.set("y", 20); +// Concurrent updates properly merge to {x: 10, y: 20} +``` + + +```typescript no_run +insert(pos: number, value: Value | Container): void +``` + + +Inserts a value at the specified position. + +**Parameters:** + +- `pos` - Insert position +- `value` - Value to insert + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.insert(0, "First"); +list.insert(1, { type: "object" }); +``` + + + + +```typescript no_run +insertContainer(pos: number, container: T): T +``` + + +Inserts a new container at the position. + +**Parameters:** + +- `pos` - Insert position +- `container` - Container instance + +**Returns:** The inserted container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const subText = list.insertContainer(0, new LoroText()); +subText.insert(0, "Nested text"); +``` + + + + +```typescript no_run +delete(pos: number, len: number): void +``` + + +Deletes elements from the list. + +**Parameters:** + +- `pos` - Starting position +- `len` - Number of elements to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a"); +list.push("b"); +list.push("c"); +list.push("d"); + +list.delete(1, 2); // Delete 2 elements starting at index 1 +``` + + + + +```typescript no_run +push(value: Value | Container): void +``` + + +Appends a value to the end of the list. + +**Parameters:** + +- `value` - Value to append + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("Last item"); +``` + + + + +```typescript no_run +getIdAt(pos: number): { peer: PeerID, counter: number } | undefined +``` + + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +list.insert(0, 1); +const id0 = list.getIdAt(0); + +```` + + + +```typescript no_run +pushContainer(container: T): T +```` + + + +Appends a container to the end of the list. + +**Parameters:** + +- `container` - Container to append + +**Returns:** The appended container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroMap } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const map = list.pushContainer(new LoroMap()); +map.set("key", "value"); +``` + + + + +```typescript no_run +pop(): Value | Container | undefined +``` + + +Removes and returns the last element. + +**Returns:** The removed element or undefined + +**Example:** + +```ts no_run threeslash +import { LoroList } from "loro-crdt"; +declare const list: LoroList; +// ---cut--- +const lastItem = list.pop(); +``` + + + + +```typescript no_run +get(index: number): Value | Container | undefined +``` + + +Gets the value at the specified index. + +**Parameters:** + +- `index` - Element index + +**Returns:** The value or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const list = doc.getList("items"); +list.push("first", "second"); + +const item = list.get(0); // "first" +``` + + + + +```typescript no_run +getCursor(pos: number, side?: Side): Cursor | undefined +``` + + +Gets a stable cursor for the position. + +**Parameters:** + +- `pos` - Position in the list +- `side` - Cursor affinity + +**Returns:** Cursor object or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a", "b", "c"); + +const cursor = list.getCursor(2); +``` + + + + +```typescript no_run +toArray(): (Value | Container)[] +``` + + +Converts the list to a JavaScript array. + +**Returns:** Array of values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a", "b", "c"); +const array = list.toArray(); +``` + + + + +```typescript no_run +clear(): void +``` + + +Removes all elements from the list. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a", "b", "c"); +list.clear(); +``` + + + + +```typescript no_run +length: number +``` + + +Gets the number of elements in the list. + +**Returns:** List length + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a"); +list.push("b"); +list.push("c"); +console.log(`List has ${list.length} items`); +``` + + + + +```typescript no_run +kind(): "List" +``` + + +Returns the container type. + +**Returns:** "List" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +const type = list.kind(); // "List" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the list to JSON representation. + +**Returns:** JSON array + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push(1, 2, 3); + +// Usage example: +const json = list.toJSON(); // [1, 2, 3] +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this list is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +const list = map.setContainer("nested", doc.getList("list")); + +// Usage example: +const parent = list.parent(); // Returns the map +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroList } from "loro-crdt"; +const doc = new LoroDoc(); +const list = new LoroList(); + +const attached = list.isAttached(); // false until attached to doc +``` + + + + +```typescript no_run +getAttached(): LoroList | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroList } from "loro-crdt"; +const doc = new LoroDoc(); +const list = new LoroList(); + +const attached = list.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +const deleted = list.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): Value[] +``` + + +Gets the list values with sub-containers as IDs. + +**Returns:** Array of values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push(1, 2); + +// Usage example: +const values = list.getShallowValue(); // [1, 2] +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +const containerId = list.id; +``` + + + +### LoroMap + +A key-value map container for collaborative objects. See [Map](/docs/tutorial/map). + + +```typescript no_run +set(key: string, value: Value | Container): void +``` + + +Sets a key-value pair. + +Note: Setting a key to the same value is a no-op (no operation recorded). See [Map basics](/docs/tutorial/map). + +**Parameters:** + +- `key` - The key +- `value` - The value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.set("name", "Alice"); +map.set("age", 30); +``` + + + + +```typescript no_run +setContainer(key: string, container: T): T +``` + + +Sets a container as the value for a key. + +**⚠️ Pitfall:** Concurrent regular child container creation at the same key on a LoroMap from multiple peers causes overwrites (different container IDs cannot auto-merge). If the child should be shared by logical key, use the `ensureMergeable*` methods instead. See [Container initialization](/docs/tutorial/tips). + +**Parameters:** + +- `key` - The key +- `container` - Container instance + +**Returns:** The set container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroList } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const list = map.setContainer("items", new LoroList()); +list.push("item1"); +``` + + + + +```typescript no_run +ensureMergeableText(key: string): LoroText +``` + + +Ensures that `key` points to a mergeable Text child container. Peers that call this method concurrently with the same parent Map and key will address the same logical Text container instead of creating competing child container IDs. + +Use this for lazily-created text fields that should behave like one shared document for all peers. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Text container + +**Example:** + +```ts no_run +const body = doc.getMap("note").ensureMergeableText("body"); +body.insert(0, "Hello"); +``` + + + + +```typescript no_run +ensureMergeableMap(key: string): LoroMap +``` + + +Ensures that `key` points to a mergeable Map child container. Use this when a nested record, profile, settings object, or migration-created child Map may be initialized by multiple peers. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Map container + +**Example:** + +```ts no_run +const record = doc.getMap("records").ensureMergeableMap(recordId); +record.set("updated", true); +``` + + + + +```typescript no_run +ensureMergeableList(key: string): LoroList +``` + + +Ensures that `key` points to a mergeable List child container. Use this for dynamic child lists, such as one list per day, where concurrent first creation should produce one shared List. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable List container + +**Example:** + +```ts no_run +const entries = doc.getMap("days").ensureMergeableList("2026-06-08"); +entries.insert(0, "meeting notes"); +``` + + + + +```typescript no_run +ensureMergeableMovableList(key: string): LoroMovableList +``` + + +Ensures that `key` points to a mergeable MovableList child container. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable MovableList container + + + + +```typescript no_run +ensureMergeableTree(key: string): LoroTree +``` + + +Ensures that `key` points to a mergeable Tree child container. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Tree container + + + + +```typescript no_run +ensureMergeableCounter(key: string): LoroCounter +``` + + +Ensures that `key` points to a mergeable Counter child container. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Counter container + + + + +```typescript no_run +get(key: string): Value | Container | undefined +``` + + +Gets the value for a key. + +**Parameters:** + +- `key` - The key + +**Returns:** The value or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const name = map.get("name"); +``` + + + + +```typescript no_run +/** + * @deprecated Prefer ensureMergeableText/Map/List/... for lazily-created Map child containers. + */ +getOrCreateContainer(key: string, container: T): T +``` + + +**Deprecated:** Prefer the `ensureMergeable*` methods for Map child containers that may be created lazily or concurrently by multiple peers. + +Gets an existing container or creates a new regular child container. + +**⚠️ Pitfall:** Parallel container creation for the same key across peers creates different child container IDs and can cause overwrites. This method is kept for compatibility and for cases where you explicitly want a regular child container. For shared dynamic children, use `ensureMergeableText`, `ensureMergeableMap`, `ensureMergeableList`, `ensureMergeableMovableList`, `ensureMergeableTree`, or `ensureMergeableCounter`. See [Container initialization](/docs/tutorial/tips). + +**Parameters:** + +- `key` - The key +- `container` - Container to create if not exists + +**Returns:** The container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const text = map.getOrCreateContainer("description", new LoroText()); +``` + + + + +```typescript no_run +delete(key: string): void +``` + + +Removes a key-value pair. + +**Parameters:** + +- `key` - The key to remove + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.delete("obsoleteKey"); +``` + + + + +```typescript no_run +clear(): void +``` + + +Removes all key-value pairs. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.clear(); +``` + + + + +```typescript no_run +keys(): string[] +``` + + +Gets all keys in the map. + +**Returns:** Array of keys + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const allKeys = map.keys(); +``` + + + + +```typescript no_run +values(): (Value | Container)[] +``` + + +Gets all values in the map. + +**Returns:** Array of values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const allValues = map.values(); +``` + + + + +```typescript no_run +entries(): [string, Value | Container][] +``` + + +Gets all key-value pairs. + +**Returns:** Array of entries + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +for (const [key, value] of map.entries()) { + console.log(`${key}: ${value}`); +} +``` + + + + +```typescript no_run +getLastEditor(key: string): PeerID | undefined +``` + + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.set("k", 1); +doc.commit(); +const who = map.getLastEditor("k"); +// who = doc.peerIdStr + +```` + + + +```typescript no_run +size: number +```` + + + +Gets the number of key-value pairs. + +**Returns:** Map size + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +console.log(`Map has ${map.size} entries`); +``` + + + + +```typescript no_run +kind(): "Map" +``` + + +Returns the container type. + +**Returns:** "Map" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const type = map.kind(); // "Map" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the map to JSON representation. + +**Returns:** JSON object + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("name", "Alice"); + +// Usage example: +const json = map.toJSON(); // { name: "Alice" } +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this map is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const map = list.insertContainer(0, doc.getMap("nested")); + +// Usage example: +const parent = map.parent(); // Returns the list +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroMap } from "loro-crdt"; +const doc = new LoroDoc(); +const map = new LoroMap(); + +const attached = map.isAttached(); // false until attached to doc +``` + + + + +```typescript no_run +getAttached(): LoroMap | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroMap } from "loro-crdt"; +const doc = new LoroDoc(); +const map = new LoroMap(); + +const attached = map.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const deleted = map.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): Record +``` + + +Gets the map values with sub-containers as IDs. + +**Returns:** Object with values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("key", "value"); + +// Usage example: +const values = map.getShallowValue(); // { key: "value" } +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const containerId = map.id; +``` + + + +### LoroTree + +A hierarchical tree container for nested structures. Supports moving subtrees while handling concurrent edits. See [Tree](/docs/tutorial/tree). + +**⚠️ Important Tree Operation Notes:** + +- **Concurrent moves can create cycles**: Loro detects and prevents these automatically +- **Fractional indexing**: Has interleaving issues but maintains relative ordering +- **Don't disable fractional index** if you need siblings to be sorted. See [Tree](/docs/tutorial/tree). + + +```typescript no_run +createNode(parent?: TreeID, index?: number): LoroTreeNode +``` + + +Creates a new tree node. + +**Parameters:** + +- `parent` - Parent node ID (optional, creates root if omitted) +- `index` - Position among siblings (optional) + +**Returns:** The new node's handler + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const root = tree.createNode(); +const child = root.createNode(0); +``` + + + + +```typescript no_run +move(target: TreeID, parent?: TreeID, index?: number): void +``` + + +Moves a node to a new position. + +**Parameters:** + +- `target` - Node to move +- `parent` - New parent (undefined for root) +- `index` - Position among siblings + +**Example:** + +```typescript no_run threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +declare const nodeId: TreeID; +declare const newParentId: TreeID; + +// Usage example: +tree.move(nodeId, newParentId, 0); +``` + + + + +```typescript no_run +delete(target: TreeID): void +``` + + +Deletes a node and its descendants. + +**Parameters:** + +- `target` - Node to delete + +**Example:** + +```ts threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const nodeId = node.id; + +// Usage example: +tree.delete(nodeId); +``` + + + + +```typescript no_run +getNodeByID(id: TreeID): LoroTreeNode | undefined +``` + + +Gets a node handler by its ID. + +**Parameters:** + +- `id` - Node ID + +**Returns:** Node handler or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const _node = tree.createNode(); +const nodeId = _node.id; + +// Usage example: +const node = tree.getNodeByID(nodeId); +if (node) { + node.data.set("label", "New Label"); +} +``` + + + + +```typescript no_run +nodes(): LoroTreeNode[] +``` + + +Gets all nodes in the tree. + +**Returns:** Array of all nodes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const allNodes = tree.nodes(); +``` + + + + +```typescript no_run +roots(): LoroTreeNode[] +``` + + +Gets all root nodes. + +**Returns:** Array of root nodes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const rootNodes = tree.roots(); +``` + + + + +```typescript no_run +has(target: TreeID): boolean +``` + + +Checks if a node exists. + +**Parameters:** + +- `target` - Node ID to check + +**Returns:** Boolean indicating existence + +**Example:** + +```typescript no_run threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +declare const nodeId: TreeID; + +// Usage example: +if (tree.has(nodeId)) { + console.log("Node exists"); +} +``` + + + + +```typescript no_run +isNodeDeleted(target: TreeID): boolean +``` + + +Checks if a node has been deleted. + +**Parameters:** + +- `target` - Node ID to check + +**Returns:** Boolean indicating deletion status + +**Example:** + +```typescript no_run threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +declare const nodeId: TreeID; + +// Usage example: +if (tree.isNodeDeleted(nodeId)) { + console.log("Node was deleted"); +} +``` + + + +```typescript no_run +enableFractionalIndex(jitter: number): void +``` + + +Enables fractional indexing for better concurrent move performance. + +**Parameters:** + +- `jitter` - Jitter amount for fractional indices + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +tree.enableFractionalIndex(0.001); +``` + + + + +```typescript no_run +disableFractionalIndex(): void +``` + + +Disables fractional indexing. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +tree.disableFractionalIndex(); +``` + + + + +```typescript no_run +isFractionalIndexEnabled(): boolean +``` + + +Checks if fractional indexing is enabled. + +**Returns:** True if enabled + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const enabled = tree.isFractionalIndexEnabled(); +``` + + + + +```typescript no_run +kind(): "Tree" +``` + + +Returns the container type. + +**Returns:** "Tree" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const type = tree.kind(); // "Tree" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the tree to JSON representation. + +**Returns:** JSON tree structure + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const json = tree.toJSON(); +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this tree is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +const tree = map.setContainer("tree", doc.getTree("nested")); + +// Usage example: +const parent = tree.parent(); +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTree } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = new LoroTree(); + +const attached = tree.isAttached(); +``` + + + + +```typescript no_run +getAttached(): LoroTree | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTree } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = new LoroTree(); + +const attached = tree.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const deleted = tree.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): TreeNodeShallowValue[] +``` + + +Gets the tree values with sub-containers as IDs. + +**Returns:** Array of tree node values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const values = tree.getShallowValue(); +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const containerId = tree.id; +``` + + + +### LoroTreeNode + +Represents a single node in the tree. + + +```typescript no_run +data: LoroMap +``` + + +A map container for storing node metadata. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +node.data.set("title", "Node Title"); +node.data.set("expanded", true); +``` + + + + +```typescript no_run +createNode(index?: number): LoroTreeNode +``` + + +Creates a child node. + +**Parameters:** + +- `index` - Position among siblings + +**Returns:** New node's handler + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const childId = node.createNode(0); +``` + + + + +```typescript no_run +move(parent?: LoroTreeNode, index?: number): void +``` + + +Moves this node to a new parent. + +**Parameters:** + +- `parent` - New parent node +- `index` - Position among siblings + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTreeNode } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const parent = tree.createNode(); + +// Usage example: +node.move(parent, 0); +``` + + + + +```typescript no_run +moveAfter(target: LoroTreeNode): void +``` + + +Moves this node after a sibling. + +**Parameters:** + +- `target` - Sibling node + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTreeNode } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const sibling = tree.createNode(); + +// Usage example: +node.moveAfter(sibling); +``` + + + + +```typescript no_run +moveBefore(target: LoroTreeNode): void +``` + + +Moves this node before a sibling. + +**Parameters:** + +- `target` - Sibling node + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTreeNode } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const sibling = tree.createNode(); + +// Usage example: +node.moveBefore(sibling); +``` + + + + +```typescript no_run +parent(): LoroTreeNode | undefined +``` + + +Gets the parent node. + +**Returns:** Parent node or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +const parentNode = node.parent(); +``` + + + + +```typescript no_run +children(): LoroTreeNode[] +``` + + +Gets all child nodes. + +**Returns:** Array of children + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +const childNodes = node.children(); +``` + + + + +```typescript no_run +index(): number | undefined +``` + + +Gets the position among siblings. + +**Returns:** Index or undefined if root + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +const position = node.index(); +``` + + + + +```typescript no_run +fractionalIndex(): string | undefined +``` + + +Returns the node's fractional index used to sort siblings deterministically. +It is a hex string representation of the Fractional Index and is stable for ordering. +Returns `undefined` for the root node. Note: the tree must be attached to the document. + +**Returns:** Hex string or `undefined` for root + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const parent = tree.createNode(); +const a = parent.createNode(0); +const b = parent.createNode(1); + +const aFi = a.fractionalIndex(); +const bFi = b.fractionalIndex(); +// aFi < bFi, because b is inserted after a +``` + + + + +```typescript no_run +creationId(): { peer: PeerID, counter: number } +``` + + +Returns the OpID that created this node. + +**Returns:** `{ peer: PeerID, counter: number }` creation identifier + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +const { peer, counter } = node.creationId(); +``` + + + + +```typescript no_run +creator(): PeerID +``` + + +Returns the peer ID that created this node (equivalent to `creationId().peer`). + +**Returns:** `PeerID` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +const author = node.creator(); +// author == doc.peerIdStr +``` + + + + +```typescript no_run +getLastMoveId(): { peer: PeerID, counter: number } | undefined +``` + + +Returns the OpID of the most recent move operation for this node, or `undefined` if the node has never been moved. + +**Returns:** Creation/move OpID or `undefined` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +const lastMove = node.getLastMoveId(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if this node has been deleted. + +**Returns:** Boolean deletion status + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +if (node.isDeleted()) { + console.log("Node is deleted"); +} +``` + + + +### LoroCounter + +A counter CRDT for collaborative numeric values. + + +```typescript no_run +increment(value: number): void +``` + + +Increments the counter. + +**Parameters:** + +- `value` - Amount to increment (default: 1) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const counter = doc.getCounter("counter"); + +counter.increment(5); // +5 +``` + + + + +```typescript no_run +decrement(value: number): void +``` + + +Decrements the counter. + +**Parameters:** + +- `value` - Amount to decrement (default: 1) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const counter = doc.getCounter("counter"); + +counter.decrement(3); // -3 +``` + + + + +```typescript no_run +value: number +``` + + +Gets the current counter value. + +**Returns:** Current numeric value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const counter = doc.getCounter("counter"); + +console.log(`Counter value: ${counter.value}`); +``` + + + +### LoroMovableList + +A list optimized for move operations. Designed for frequent reordering (drag-and-drop) with good behavior under +concurrent moves (concurrent moves resolve to one final position). See [List and Movable List](/docs/tutorial/list). + +**📝 MovableList vs List:** + +- **Use MovableList** for: Drag-and-drop UIs, sortable lists, kanban boards +- **Use List** for: scenarios where the list items don't need to be moved +- **Key difference**: MovableList handles concurrent moves better (no duplicates) and supports set operations, List is more efficient in general. + + +```typescript no_run +move(from: number, to: number): void +``` + + +Moves an element from one position to another. + +**Parameters:** + +- `from` - Source index +- `to` - Target index + +**Example:** + +```typescript +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const movableList = doc.getMovableList("list"); +movableList.push("a"); +movableList.push("b"); +movableList.push("c"); +movableList.push("d"); +movableList.push("e"); + +movableList.move(0, 3); // Move first element to fourth position +``` + + + + +```typescript no_run +set(pos: number, value: Value | Container): void +``` + + +Replaces the value at a position. + +**Parameters:** + +- `pos` - Position to update +- `value` - New value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const movableList = doc.getMovableList("list"); +movableList.push("a", "b", "c"); + +movableList.set(0, "Updated value"); +``` + + + + +```typescript no_run +setContainer(pos: number, container: T): T +``` + + +Replaces the value with a container. + +**Parameters:** + +- `pos` - Position to update +- `container` - New container + +**Returns:** The set container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const movableList = doc.getMovableList("list"); +movableList.push("placeholder"); + +const text = movableList.setContainer(0, new LoroText()); +``` + + + +## Synchronization + +Import/export updates over any transport and choose the right encoding for speed and size. See [Sync Tutorial](/docs/tutorial/sync) and [Encoding & Export Modes](/docs/tutorial/encoding). + +- **Import order**: Loro handles out-of-order updates automatically +- **Auto-commit**: Import and export operations trigger automatic commits + +### Import/Export Patterns + +#### Basic Synchronization + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +// Peer A: Export updates +const doc1 = new LoroDoc(); +doc1.getText("text").insert(0, "Hello"); +const updates = doc1.export({ mode: "update" }); + +// Peer B: Import updates +const doc2 = new LoroDoc(); +doc2.import(updates); +// now doc2.getText("text").toString() === "Hello" +``` + +#### Continuous Sync + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc1 = new LoroDoc(); +const doc2 = new LoroDoc(); + +// Usage example: +// Set up bidirectional sync +doc1.subscribeLocalUpdates((updates) => { + doc2.import(updates); +}); + +doc2.subscribeLocalUpdates((updates) => { + doc1.import(updates); +}); +``` + +Performance tips: + +- Prefer `mode: "update"` with a `VersionVector` to sync incrementally. +- Use `mode: "shallow-snapshot"` when you only need current state; it strips history for faster import/load. +- Loro’s LSM-based encoding and Eg-walker-inspired merge keep import/export fast, even for large histories. + See [Encoding](/docs/tutorial/encoding) and v1.0 performance notes: https://loro.dev/blog/v1.0 + +#### Network Sync with WebSocket + +```ts no_run threeslash +import { LoroDoc } from "loro-crdt"; + +// Assume we have: +declare const ws: { + send: (data: Uint8Array) => void; + on: (event: string, handler: (data: any) => void) => void; +}; + +// Client side +const doc = new LoroDoc(); + +// Send local updates to server +doc.subscribeLocalUpdates((updates) => { + ws.send(updates); +}); + +// Receive updates from server +ws.on("message", (data) => { + doc.import(new Uint8Array(data)); +}); +``` + +### Shallow Snapshots + +Shallow snapshots allow for efficient storage by garbage collecting deleted operations. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Create shallow snapshot +const frontiers = doc.frontiers(); +const shallowSnapshot = doc.export({ + mode: "shallow-snapshot", + frontiers: frontiers, +}); + +// Import shallow snapshot +const newDoc = new LoroDoc(); +newDoc.import(shallowSnapshot); +``` + +--- + +## Version Control + +### Time Travel + +Navigate through document history: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Save current version +const v1 = doc.frontiers(); + +// Make changes +doc.getText("text").insert(0, "New text"); + +// Save new version +const v2 = doc.frontiers(); + +// Travel back +doc.checkout(v1); +console.log(doc.getText("text").toString()); // Original text + +// Travel forward +doc.checkout(v2); +console.log(doc.getText("text").toString()); // New text + +// Return to latest +doc.checkoutToLatest(); +``` + +### Forking + +Create document branches: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Fork at current state +const fork1 = doc.fork(); +fork1.getText("text").insert(0, "Fork 1 changes"); + +// Fork at specific version +const historicalVersion = doc.frontiers(); +const fork2 = doc.forkAt(historicalVersion); +fork2.getText("text").insert(0, "Fork 2 changes"); + +// Original document remains unchanged +``` + +### Version Vectors + +Track document versions: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const vv = doc.version(); +console.log(`Document has ${vv.length()} peers`); +``` + +--- + +## Events & Subscriptions + +### Event Structure + +```typescript no_run +interface LoroEventBatch { + by: "local" | "import" | "checkout"; + origin?: string; + currentTarget?: ContainerID; + events: LoroEvent[]; + from: Frontiers; + to: Frontiers; +} + +interface LoroEvent { + target: ContainerID; + diff: Diff; + path: Path; +} +``` + +### Diff Types + +Different containers produce different diff types: + +#### TextDiff + + +```typescript no_run +type TextDiff = { + type: "text" + diff: Delta[] +} +``` +Represents changes to text content using Delta format. + +**Properties:** + +- `type` - Always "text" for text diffs +- `diff` - Array of Delta operations (insert, delete, retain) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +// Example +text.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "text") { + event.diff.diff.forEach((delta) => { + if (delta.insert) { + console.log(`Inserted: "${delta.insert}"`); + } + if (delta.delete) { + console.log(`Deleted ${delta.delete} characters`); + } + }); + } + } +}); +``` + + + +#### ListDiff + + +```typescript no_run +type ListDiff = { + type: "list" + diff: Delta<(Value | Container)[]>[] +} +``` +Represents changes to list content using Delta format. + +**Properties:** + +- `type` - Always "list" for list diffs +- `diff` - Array of Delta operations on list items + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +// Example +list.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "list") { + event.diff.diff.forEach((delta) => { + if (delta.insert) { + console.log(`Inserted items:`, delta.insert); + } + }); + } + } +}); +``` + + + +#### MapDiff + + +```typescript no_run +type MapDiff = { + type: "map" + updated: Record +} +``` +Represents changes to map content. + +**Properties:** + +- `type` - Always "map" for map diffs +- `updated` - Record of key-value changes (undefined means deleted) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +// Example +map.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "map") { + Object.entries(event.diff.updated).forEach(([key, value]) => { + if (value === undefined) { + console.log(`Deleted key: ${key}`); + } else { + console.log(`Updated key: ${key} = ${value}`); + } + }); + } + } +}); +``` + + + +#### TreeDiff + + +```typescript no_run +type TreeDiff = { + type: "tree" + diff: TreeDiffItem[] +} + +type TreeDiffItem = +| { +target: TreeID +action: "create" +parent: TreeID | undefined +index: number +fractionalIndex: string +} +| { +target: TreeID +action: "delete" +oldParent: TreeID | undefined +oldIndex: number +} +| { +target: TreeID +action: "move" +parent: TreeID | undefined +index: number +fractionalIndex: string +oldParent: TreeID | undefined +oldIndex: number +} + +```` +Represents changes to tree structure. + +**Properties:** +- `type` - Always "tree" for tree diffs +- `diff` - Array of TreeDiffItem operations (create, delete, move) + +**TreeDiffItem Actions:** +- `create` - Node creation with parent and position +- `delete` - Node deletion with old parent and position +- `move` - Node movement with old and new positions + +**Example:** +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +// Example +tree.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "tree") { + event.diff.diff.forEach(item => { + switch (item.action) { + case "create": + console.log(`Created node ${item.target}`); + break; + case "move": + console.log(`Moved node ${item.target}`); + break; + case "delete": + console.log(`Deleted node ${item.target}`); + break; + } + }); + } + } +}); +```` + + + +### Deep Subscription + +Subscribe to nested changes: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Subscribe to specific container +const text = doc.getText("text"); +text.subscribe((event) => { + console.log("Text changed:", event); +}); + +// Subscribe with deep observation +doc.subscribe((event) => { + // Path shows the location of the change + event.events.forEach((e) => { + console.log("Change path:", e.path); + console.log("Container:", e.target); + console.log("Diff:", e.diff); + }); +}); +``` + +--- + +## Undo/Redo + +Local undo operates on your own changes without breaking collaboration. See [Undo/Redo](/docs/advanced/undo) for design details and caveats. + +### UndoManager + +Provides local undo/redo functionality. + +**⚠️ Important Notes:** + +- **Local-only**: UndoManager only undoes the local user's operations, not remote operations +- **Origin filtering**: Use `excludeOriginPrefixes` to exclude certain operations (e.g., sync operations) from undo +- **Cursor restoration**: Use `onPush`/`onPop` callbacks to save and restore cursor positions + + +```typescript no_run +constructor(doc: LoroDoc, config: UndoConfig) +``` + + +Creates a new UndoManager instance. + +**Parameters:** + +- `doc` - The LoroDoc to manage undo/redo for +- `config` - Configuration options + - `mergeInterval?` - Time in ms to merge consecutive operations (default: 1000) + - `maxUndoSteps?` - Maximum number of undo steps to keep (default: 100) + - `excludeOriginPrefixes?` - Array of origin prefixes to exclude from undo + - `onPush?` - Callback when adding to undo stack + - `onPop?` - Callback when undoing/redoing + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; + +const doc = new LoroDoc(); +const undo = new UndoManager(doc, { + mergeInterval: 1000, + maxUndoSteps: 100, + excludeOriginPrefixes: ["sync-"], +}); +``` + + + + +```typescript no_run +undo(): boolean +``` + + +Undo the last operation. + +**Returns:** True if undo was successful + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); +text.insert(0, "Hello"); +doc.commit(); + +// Usage example: +const success = undo.undo(); +console.log(success); // true +``` + + + + +```typescript no_run +redo(): boolean +``` + + +Redo the last undone operation. + +**Returns:** True if redo was successful + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); +text.insert(0, "Hello"); +doc.commit(); +undo.undo(); + +// Usage example: +const success = undo.redo(); +console.log(success); // true +``` + + + + +```typescript no_run +canUndo(): boolean +``` + + +Check if undo is available. + +**Returns:** True if can undo + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +if (undo.canUndo()) { + undo.undo(); +} +``` + + + + +```typescript no_run +canRedo(): boolean +``` + + +Check if redo is available. + +**Returns:** True if can redo + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +if (undo.canRedo()) { + undo.redo(); +} +``` + + + + +```typescript no_run +peer(): PeerID +``` + + +Get the peer ID of the undo manager. + +**Returns:** The peer ID + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +const peerId = undo.peer(); +console.log(peerId); // e.g., "123456" +``` + + + + +```typescript no_run +setMaxUndoSteps(steps: number): void +``` + + +Set the maximum number of undo steps. + +**Parameters:** + +- `steps` - Maximum number of undo steps + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.setMaxUndoSteps(50); +``` + + + + +```typescript no_run +setMergeInterval(interval: number): void +``` + + +Set the merge interval for grouping operations. + +**Parameters:** + +- `interval` - Merge interval in milliseconds + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.setMergeInterval(2000); // 2 seconds +``` + + + + +```typescript no_run +groupStart(): void +``` + + +Begin a manual grouping of subsequent commits into a single undo step. + +**Behavior:** + +- Wrap consecutive `doc.commit()` calls so they undo together +- Calling `groupStart` again before `groupEnd` throws and leaves the current group unchanged +- Conflicting remote imports may automatically end the group and split the undo item + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; + +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); + +undo.groupStart(); + +text.update("hello", undefined); +doc.commit(); + +text.update("hello world", undefined); +doc.commit(); + +undo.groupEnd(); + +undo.undo(); + +console.log(text.toString()); // "" +``` + + + + +```typescript no_run +groupEnd(): void +``` + + +Close the active manual group and enqueue the grouped operations as a single undo item. + +**Behavior:** + +- Must be paired with a prior `groupStart` +- Safe to call after the group was auto-closed by a conflicting remote import (becomes a no-op) +- Non-conflicting remote updates from other peers remain outside the undo item but do not break the group + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; + +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); + +undo.groupStart(); + +text.update("hello", undefined); +doc.commit(); +text.update("hello world", undefined); +doc.commit(); + +const snapshot = doc.export({ mode: "snapshot" }); +const doc2 = new LoroDoc(); +doc2.import(snapshot); +doc2.getText("text2").update("hello world world", undefined); // touches a different container +doc2.commit(); +const update = doc2.export({ mode: "update" }); + +doc.import(update); // remote, non-conflicting change + +text.update("hello world world world", undefined); +doc.commit(); + +undo.groupEnd(); + +undo.undo(); + +console.log(text.toString()); // "" +``` + + + + +```typescript no_run +addExcludeOriginPrefix(prefix: string): void +``` + + +Add a prefix to exclude from undo stack. + +**Parameters:** + +- `prefix` - Origin prefix to exclude + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.addExcludeOriginPrefix("sync-"); +undo.addExcludeOriginPrefix("import-"); +``` + + + + +```typescript no_run +clear(): void +``` + + +Clear the undo and redo stacks. + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.clear(); +``` + + + +### Custom Undo Handlers + +Handle cursor restoration and side effects: + +```typescript no_run threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +declare function saveCursorPositions(): any; +declare function restoreCursorPositions(cursors: any): void; + +// Usage example: +const undo = new UndoManager(doc, { + onPush: (isUndo, counterRange, event) => { + // Save cursor positions when adding to undo stack + const cursors = saveCursorPositions(); + return { + value: doc.toJSON(), + cursors: cursors, + }; + }, + + onPop: (isUndo, { value, cursors }, counterRange) => { + // Restore cursor positions when undoing + restoreCursorPositions(cursors); + }, +}); +``` + +--- + +## Types & Interfaces + +Reference for core types used across the API. For conceptual background, see [Containers](/docs/concepts/container), [Version Vector](/docs/concepts/version_vector), [Frontiers](/docs/concepts/frontiers), and the [Versioning Deep Dive](/docs/advanced/version_deep_dive). + +### Core Types + +```typescript no_run +// Peer identifier +type PeerID = `${number}`; + +// Container identifier +type ContainerID = + | `cid:root-${string}:${ContainerType}` + | `cid:${number}@${PeerID}:${ContainerType}`; + +// Tree node identifier +type TreeID = `${number}@${PeerID}`; + +// Operation identifier +type OpId = { + peer: PeerID; + counter: number; +}; + +// Container types +type ContainerType = + | "Text" + | "Map" + | "List" + | "Tree" + | "MovableList" + | "Counter"; + +// Value types +type Value = + | ContainerID + | string + | number + | boolean + | null + | { [key: string]: Value } + | Uint8Array + | Value[] + | undefined; +``` + +### Version Types + +Loro uses two complementary version representations: Version Vectors (per-peer counters) and Frontiers (a compact set of heads). See [Version Vector](/docs/concepts/version_vector) and [Frontiers](/docs/concepts/frontiers). For the full DAG model, see [Version Deep Dive](/docs/advanced/version_deep_dive). + +```typescript no_run +// Version vector class +class VersionVector { + constructor( + value: Map | Uint8Array | VersionVector | undefined | null, + ); + static parseJSON(version: Map): VersionVector; + toJSON(): Map; + encode(): Uint8Array; + static decode(bytes: Uint8Array): VersionVector; + get(peer_id: number | bigint | `${number}`): number | undefined; + compare(other: VersionVector): number | undefined; + setEnd(id: { peer: PeerID; counter: number }): void; + setLast(id: { peer: PeerID; counter: number }): void; + remove(peer: PeerID): void; + length(): number; +} + +// Frontiers represent a specific version +type Frontiers = OpId[]; + +// ID span for range queries +type IdSpan = { + peer: PeerID; + counter: number; + length: number; +}; +``` + +### Change Types + +```typescript no_run +// Change metadata +interface Change { + peer: PeerID; + counter: number; + lamport: number; + length: number; + timestamp: number; // Unix timestamp in seconds + deps: OpId[]; + message: string | undefined; +} + +// Change modifier for pre-commit hooks +interface ChangeModifier { + setMessage(message: string): this; + setTimestamp(timestamp: number): this; +} +``` + +### Cursor Types + +Stable cursors survive concurrent edits and resolve to absolute positions on demand. See [Cursor and Stable Positions](/docs/concepts/cursor_stable_positions) and [Cursor tutorial](/docs/tutorial/cursor). + +```typescript no_run +// Stable position in containers +class Cursor { + containerId(): ContainerID; + pos(): OpId | undefined; + side(): Side; // -1 | 0 | 1 + encode(): Uint8Array; + static decode(data: Uint8Array): Cursor; +} + +// Cursor side affinity +type Side = -1 | 0 | 1; +``` + +### Delta Type + +Delta is a popular rich-text operation format (e.g., Quill). LoroText can export/import Delta; see [Text](/docs/tutorial/text). + +```typescript no_run +// Rich text delta operations +type Delta = + | { + insert: T; + attributes?: { [key in string]: {} }; + retain?: undefined; + delete?: undefined; + } + | { + delete: number; + attributes?: undefined; + retain?: undefined; + insert?: undefined; + } + | { + retain: number; + attributes?: { [key in string]: {} }; + delete?: undefined; + insert?: undefined; + }; +``` + +--- + +## Utility Functions + +Small helpers for type checks and IDs. See [Container IDs](/docs/advanced/cid) and [Versioning Deep Dive](/docs/advanced/version_deep_dive) for frontiers/version encoding. + +### Frontier Encoding + + +```typescript no_run +encodeFrontiers(frontiers: OpId[]): Uint8Array +``` + + +Encode frontiers for efficient transmission. + +**Parameters:** + +- `frontiers` - Array of operation IDs representing frontiers + +**Returns:** Encoded bytes + +**Example:** + +```ts threeslash +import { LoroDoc, encodeFrontiers } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const encoded = encodeFrontiers(frontiers); +// Send encoded to remote peers +``` + + + + +```typescript no_run +decodeFrontiers(bytes: Uint8Array): OpId[] +``` + + +Decode frontiers from bytes. + +**Parameters:** + +- `bytes` - Encoded frontier bytes + +**Returns:** Array of operation IDs + +**Example:** + +```typescript no_run threeslash +import { decodeFrontiers } from "loro-crdt"; + +declare const encodedData: Uint8Array; +const frontiers = decodeFrontiers(encodedData); +console.log(frontiers); // [{ peer: "1", counter: 10 }, ...] +``` + + + +### Debugging + + +```typescript no_run +setDebug(): void +``` + + +Enable debug mode for detailed logging. + +**Example:** + +```ts threeslash +import { setDebug } from "loro-crdt"; + +// Enable debug logging +setDebug(); +``` + + + + +```typescript no_run +LORO_VERSION(): string +``` + + +Get the current Loro version. + +**Returns:** Version string + +**Example:** + +```ts threeslash +import { LORO_VERSION } from "loro-crdt"; + +const version = LORO_VERSION(); +console.log("Loro version:", version); +``` + + + +### Import Blob Metadata + + +```typescript no_run +decodeImportBlobMeta(blob: Uint8Array, check_checksum: boolean): ImportBlobMetadata +``` + + +Decode metadata from an import blob. + +**Parameters:** + +- `blob` - The import blob bytes +- `check_checksum` - Whether to verify checksum + +**Returns:** Import blob metadata + +**Example:** + +```typescript no_run threeslash +import { decodeImportBlobMeta } from "loro-crdt"; + +declare const blob: Uint8Array; +const metadata = decodeImportBlobMeta(blob, true); +console.log("Blob metadata:", metadata); +``` + + + +--- + +## EphemeralStore + +Manages ephemeral state like cursor positions and user presence. See [Ephemeral Store](/docs/tutorial/ephemeral) for concepts and usage patterns. Each entry uses timestamp-based LWW (Last-Write-Wins) for conflict resolution. + +**⚠️ Important:** + +- EphemeralStore is a separate CRDT without history - history/operations are NOT persisted +- Perfect for temporary state: cursor positions, selections, typing indicators +- Each peer's state auto-expires after the timeout period +- Uses Last-Write-Wins + +### EphemeralStore + + +```typescript no_run +constructor(timeout?: number) +``` + + +Creates a new EphemeralStore instance. + +**Parameters:** + +- `timeout` - Duration in milliseconds. A peer's state is considered outdated if its last update is older than this timeout. Default is 30000ms (30 seconds). + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; + +// Create ephemeral store with 30 second timeout +const store = new EphemeralStore(30000); +``` + + + + +```typescript no_run +set(key: K, value: T[K]): void +``` + + +Set an ephemeral value. + +**Parameters:** + +- `key` - The key to set +- `value` - The value to store + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +store.set("cursor", { line: 10, column: 5 }); +store.set("selection", { start: 0, end: 10 }); +store.set("user", { name: "Alice", color: "#ff0000" }); +``` + + + + +```typescript no_run +get(key: K): T[K] | undefined +``` + + +Get an ephemeral value. + +**Parameters:** + +- `key` - The key to get + +**Returns:** The stored value or undefined + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); + +// Usage example: +const cursor = store.get("cursor"); +console.log(cursor); // { line: 10 } +``` + + + + +```typescript no_run +delete(key: K): void +``` + + +Delete an ephemeral value. + +**Parameters:** + +- `key` - The key to delete + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); + +// Usage example: +store.delete("cursor"); +``` + + + + +```typescript no_run +getAllStates(): Partial +``` + + +Get all ephemeral states. + +**Returns:** All stored states + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); +store.set("user", { name: "Alice" }); + +// Usage example: +const allStates = store.getAllStates(); +console.log(allStates); +``` + + + + +```typescript no_run +encode(key: K): Uint8Array +``` + + +Encode a specific key's state for transmission. + +**Parameters:** + +- `key` - The key to encode + +**Returns:** Encoded bytes + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); + +// Usage example: +const encoded = store.encode("cursor"); +// Send encoded to remote peers +``` + + + + +```typescript no_run +encodeAll(): Uint8Array +``` + + +Encode all states for transmission. + +**Returns:** Encoded bytes + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); +store.set("user", { name: "Alice" }); + +// Usage example: +const encoded = store.encodeAll(); +// Send encoded to remote peers +``` + + + + +```typescript no_run +apply(bytes: Uint8Array): void +``` + + +Apply remote updates. + +**Parameters:** + +- `bytes` - Encoded updates from remote peer + +**Example:** + +```typescript no_run threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +declare const remoteData: Uint8Array; +store.apply(remoteData); +``` + + + + +```typescript no_run +keys(): string[] +``` + + +Get all keys in the store. + +**Returns:** Array of keys + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); +store.set("user", { name: "Alice" }); + +// Usage example: +const allKeys = store.keys(); +console.log(allKeys); // ["cursor", "user"] +``` + + + + +```typescript no_run +subscribe(listener: EphemeralListener): () => void +``` + + +Subscribe to all ephemeral state changes. + +**Parameters:** + +- `listener` - Callback function for state changes + +**Returns:** Unsubscribe function + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +const unsubscribe = store.subscribe((event) => { + console.log("Ephemeral state changed:", event); +}); + +// Later, unsubscribe +unsubscribe(); +``` + + + + +```typescript no_run +subscribeLocalUpdates(listener: EphemeralLocalListener): () => void +``` + + +Subscribe to local ephemeral updates for syncing to remote peers. + +**Parameters:** + +- `listener` - Callback function that receives encoded updates + +**Returns:** Unsubscribe function + +**Example:** + +```typescript no_run threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +declare const websocket: { send: (data: Uint8Array) => void }; + +const unsubscribe = store.subscribeLocalUpdates((data) => { + // Send to remote peers + websocket.send(data); +}); + +// Later, unsubscribe +unsubscribe(); +``` + + + + +```typescript no_run +destroy(): void +``` + + +Clean up and destroy the ephemeral store. + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +store.destroy(); +``` + + + +### Complete Example + +```typescript no_run threeslash +import { EphemeralStore } from "loro-crdt"; + +// Assume we have: +declare const websocket: { + send: (data: Uint8Array) => void; + on: (event: string, handler: (data: any) => void) => void; +}; + +const store = new EphemeralStore(30000); +const store2 = new EphemeralStore(30000); + +// Subscribe to local updates +store.subscribeLocalUpdates((data) => { + store2.apply(data); +}); + +// Subscribe to all updates +store2.subscribe((event) => { + console.log("event:", event); +}); + +// Set a value +store.set("key", "value"); + +// Encode the value +const encoded = store.encode("key"); + +// Apply the encoded value +store2.apply(encoded); +``` + +--- + +
+ + +# FILE: pages/docs/examples.mdx + +--- +keywords: "crdt, example, get started, excalidraw" +description: "The examples of basic usage in Loro" +--- + +# Examples + +You can find the examples of basic usage in [Loro examples in Deno](https://github.com/loro-dev/loro-examples-deno); + +### loro-prosemirror + +GitHub: [loro-dev/loro-prosemirror](https://github.com/loro-dev/loro-prosemirror) + +It provides seamless integration between Loro and ProseMirror, a powerful rich text editor framework. It includes: + +- Document state synchronization with rich text support +- Cursor awareness and synchronization +- Undo/Redo support in collaborative editing + +It can also be used with [Tiptap](https://tiptap.dev/), a popular rich text editor built on top of ProseMirror. This means you can easily add collaborative editing capabilities to your Tiptap-based applications. + +### loro-codemirror + +GitHub: [loro-dev/loro-codemirror](https://github.com/loro-dev/loro-codemirror) + +This package provides integration between Loro and CodeMirror 6, a versatile code editor. It supports: + +- Document state synchronization +- Cursor awareness synchronization +- Undo/Redo functionality + +### loro-inspector + +GitHub: [loro-dev/loro-inspector](https://github.com/loro-dev/loro-inspector) + +import { ReactPlayer } from "../../components/video"; + + + + + +# FILE: pages/docs/concepts/frontiers.mdx + +--- +keywords: "frontiers, version, compact representation, DAG, CRDT" +description: "Understanding Frontiers in Loro - a compact way to represent document versions" +--- + +# Frontiers + +Frontiers are a compact way to represent document versions in Loro by identifying the "frontier" operations - the most recent operations that aren't followed by any other operations. + +## What are Frontiers? + +Think of Frontiers as **bookmarks in your document's history**. Instead of listing every change made by every collaborator (like Version Vectors do), Frontiers only point to the boundary operations that implicitly include all their ancestors through causal relationships. + +## Basic Usage + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("content"); +text.insert(0, "Hello World"); + +// Get current frontiers (usually 1-2 elements) +const frontiers = doc.frontiers(); +console.log(frontiers); // [{ peer: "...", counter: 0 }] + +// Use frontiers for checkpoints +const checkpoint = doc.frontiers(); +text.insert(5, " Beautiful"); + +// Restore to checkpoint +doc.checkout(checkpoint); +console.log(text.toString()); // Back to "Hello World" +``` + +## When to Use Frontiers + +Frontiers are ideal for: + +1. **Creating checkpoints** - Mark specific points in document history +2. **Time travel** - Navigate to exact operation points efficiently +3. **Storage optimization** - Remain compact even with many collaborators (usually 1-2 elements vs N entries in Version Vector) +4. **Recording milestones** - Save important document states + +## Quick Comparison + +| Aspect | Frontiers | Version Vectors | +|--------|-----------|----------------| +| **Size** | 1-2 elements typically | Grows with peer count | +| **Use Case** | Checkpoints, time travel | Synchronization, diffing | +| **Storage** | Very compact | Larger with many peers | +| **Unknown ops** | Cannot determine included ops | Can determine all included ops | + +## Practical Example + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("content"); +const checkpoints = new Map(); + +// Save checkpoint with frontiers +text.insert(0, "Draft version"); +checkpoints.set("draft", doc.frontiers()); + +// Make changes +text.delete(0, 5); +text.insert(0, "Final"); +checkpoints.set("final", doc.frontiers()); + +// Restore to any checkpoint +doc.checkout(checkpoints.get("draft")); +console.log(text.toString()); // "Draft version" +``` + +## Important Limitation + +Frontiers have a key limitation when dealing with unknown operations: +- When you have a Frontier pointing to operations you don't know about, you cannot determine the complete set of operation IDs included in that version +- Version Vectors don't have this limitation - they explicitly list all peers and their operation counts, so you always know exactly which operations are included + +**Example**: +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +// If you receive a frontier [{ peer: "unknown-peer", counter: 42 }] +// Without having the operations from "unknown-peer": +// - Frontiers: Cannot tell which specific operations are included +// - Version Vector: Would show { "unknown-peer": 43 }, clearly indicating ops 0-42 + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +// Frontiers are compact but require operation history for full information +``` + +This limitation is why Frontiers are best for scenarios where you have access to the operation history (like checkpoints in a local document), while Version Vectors are preferred for synchronization between peers who may not share complete history. + +## Conversion with Version Vectors + +Loro allows seamless conversion between representations: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const vv = doc.frontiersToVV(frontiers); // Convert to Version Vector +const backToFrontiers = doc.vvToFrontiers(vv); // Convert back +``` + +## Learn More + +For detailed technical explanation of how Frontiers work with Loro's DAG structure and causal ordering, see [Version Deep Dive](/docs/advanced/version_deep_dive). + +# FILE: pages/docs/concepts/choose_crdt_type.mdx + +--- +keywords: "crdt, crdts, application, data model, concurrent, conflict" +description: "Loro supports many CRDT types. You need to choose the correct type to model the data based on the algorithm semantics." +--- + +# How to Choose the Right CRDT Types + +Choosing the right CRDT type means understanding their potential behavior in concurrent editing situations and judging whether such behavior is acceptable for your application. + +For text, you can choose to represent it directly as a Value on a Map (where the Value can be a string type), or you can choose to use a Text CRDT. For the former, each operation completely overwrites the previous one, so if A and B make concurrent modifications, only one of their edits will remain in the end. For the latter, the CRDT will retain all concurrent insertions by both people, and concurrent deletions are combined to complete the deletion. For most text box edits, you might prefer the latter. But for something like editing a link, you might want to use the former. + +For Lists, concurrently removing the same element and inserting a single element creates a new element, differentiating from the semantics of Set on a Map (we may consider providing a list set method in the future). For representing coordinates, it's better to use a Map rather than a List. If you represent coordinates as [x, y], and the A client updates the y coordinate by deleting the y element and reinserting a new y_a, and the B client also deletes y and inserts y_b, then after merging, the array will become [x, y_a, y_b], which does not conform to the user's schema. Using a Map can prevent this problem. + +For nested data, also decide how the child container should get its identity: + +- Use a Map value, such as a string or plain object, when the field should have + Last-Write-Wins replacement semantics. +- Use a regular child container with `setContainer` or `insertContainer` when + each creation should produce a distinct child object. +- Use a mergeable child container with `ensureMergeableText`, + `ensureMergeableMap`, `ensureMergeableList`, and the other + `ensureMergeable*` methods when multiple peers may lazily initialize the same + child under the same Map key and should end up editing one shared child. + +This distinction matters for dynamic keys, schema migrations, date-keyed lists, +and per-entity subdocuments. In those cases, the child identity usually should +come from the logical position `(parent Map, key, type)` rather than from the +operation that happened to create it first. + + +# FILE: pages/docs/concepts/peerid_management.mdx + +# PeerID Management + +## Quick Reference + +**Peer IDs** are unique identifiers for each editing session in Loro's distributed system. They ensure operation uniqueness without coordination between peers. + +## Key Concepts + +- **Peer ID**: A 64-bit unique identifier for each client/session +- **Operation ID**: Combination of `(peerId, counter)` that uniquely identifies each operation +- **Counter**: Monotonically increasing number starting at 0 for each peer + +```ts +interface OpId { + peerId: `${number}`; // Unique peer identifier. It's string because 64bit integers are not supported in JS. + counter: number; // Monotonically increasing counter +} +``` + +## Peer ID Assignment + +### Automatic (Default) + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); // Gets a random peer ID +// Safe, no coordination needed +``` + +**Note**: New peer ID generated for each `LoroDoc` instance, even when loading same document. + +### Manual + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("123123123"); // You can only set 64 bit integers as peer IDs +``` + +⚠️ **Warning**: Manual assignment requires careful conflict avoidance. + +If you intentionally reuse a Peer ID, persist the document's local data (snapshot, updates, or durable cache) alongside that ID and load it before fetching or applying any remote updates. This ensures the `(peerId, counter)` pairs continue to reference the same operations. Skipping this step risks generating a new operation that reuses an existing operation ID, which leads to inconsistent replicas. + +## Counter System + +Each peer maintains a monotonic counter starting at 0: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("1"); +const text = doc.getText("text"); + +text.insert(0, "H"); // Operation ("1", 0) +text.insert(1, "i!"); // Operation ("1", 1) and Operation ("1", 2) are created +console.log(doc.version()); // { "1": 2 } +``` + +**Properties**: Monotonic, continuous, per-peer, persistent. + +## Common Pitfalls + +❌ **Never**: +- Use user IDs as peer IDs, because an user can have multiple devices +- Use fixed IDs +- Reuse IDs without proper management +- Allow multiple browser tabs or processes to operate on the same reused Peer ID in parallel—coordinate access with locks so that only one session emits operations for that ID at a time + +## Related Documentation + +- [Version Vector](./version_vector) - How peer IDs form version vectors +- [Import Status](./import_status) - Handling synchronization with peer IDs +- [Shallow Snapshots](./shallow_snapshots) - Consolidating peer history + + +# FILE: pages/docs/concepts/crdt.mdx + +import Image from "next/image"; + +# What are CRDTs + +CRDT (conflict-free replicated data type) is a data structure that can be +replicated across multiple computers in a network, where replicas can be updated +independently and in parallel, without the need for coordination between +replicas, and with a guarantee that no conflicts will occur. + +CRDT is often used in collaborative software, such as scenarios where multiple +users need to work together to edit/read a shared document, database, or state. +It can be used in database software, text editing software, chat software, etc. + +# What problems does CRDT solve? + +For example, a scenario where multiple users edit the same document online at +the same time. + +This scenario requires that each user sees the same content, even after +concurrent edits by different users (e.g. two users changing the title at the +same time), which is known as **consistency**. (To be precise, CRDT satisfies +the eventual consistency, see below for more details) + +Users can use CRDT even when they are offline. They can be back on sync with +others the network is restored. It also supports collaboratively editing with +other users via P2P. It is known as **partitioning fault tolerance**. This +allows CRDT to support **decentralized** applications very well: synchronization +can be done even without a centralized server. + +# The Emergence of CRDTs + +The formal concept of Conflict-free Replicated Data Types (CRDTs) was first +introduced in Marc Shapiro's 2011 paper, +[Conflict-Free Replicated Data Types](https://inria.hal.science/hal-00932836/file/CRDTs_SSS-2011.pdf). +However, it can be argued that the groundwork for CRDTs was laid earlier, in the +2006 study [Woot](https://doi.org/10.1145%2F1180875.1180916): An Algorithm for +Collaborative Real-time Editing. The primary motivation behind developing CRDTs +was to address the challenges associated with designing conflict resolution +mechanisms for eventual consistency. Prior to the introduction of CRDTs, +literature on the subject offered limited guidance, and ad hoc solutions were +often prone to errors. Consequently, Shapiro's paper presented a simple, +theoretically sound approach to achieving eventual consistency through the use +of CRDTs. + +(PS: Marc Shapiro actually wrote a paper +[Designing a commutative replicated data type](https://hal.inria.fr/inria-00177693v2/document) +in 2007. In 2011, he reworded commutative into conflict-free, expanding +the definition of commutative to include state-based CRDT) + +According to [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem), it is +impossible for a distributed computing system to perfectly satisfy the following +three points at the same time. + +- _Consistency_: each read receives the result of the most recent write or + reports an error; it behaves as if it is accessing the same piece of data +- _Availability_: every request gets a non-error response - but there is no + guarantee that the data fetched is up-to-date +- _Partition tolerance_: the ability of a distributed system to continue + functioning properly even when communication between its different components + is lost or delayed, resulting in a partition or network failure. + +If the system cannot achieve data consistency within the time limit, it means +that partitioning has occurred and a choice must be made between C and A for the +current operation, so "perfect consistency" is in conflict with "perfect +availability". + +CRDTs do not provide "perfect consistency", but +Strong Eventual Consistency (SEC). This +means that site A may not immediately reflect the state changes from site B, but +when A and B synchronize their messages they both regain consistency and do not +need to resolve potential conflicts (CRDT mathematically prevents conflicts from +occurring). _Strong Eventual Consistency_ does not conflict with _Availability_ +and _Partition Tolerance_. CRDTs provide a good CAP tradeoff. + +![CPA](./crdt-images/a4858e2a50bc1a2d79722060156e89b0cac5815cf25e8c67e409aa0926280cef.png) +_CRDT satisfies A + P + Eventual Consistency; a good tradeoff under CAP_ + +(PS: In 2012, Eric Brewer, author of the CAP theorem, wrote an article +[CAP Twelve Years Later: How the "Rules" Have Changed](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/), +explaining that the description of the "two out of three CAP features" is +actually misleading, and that the CAP actually prohibits perfect availability +and consistency in a very small part of the design space, i.e., in the presence +of partitions; in fact, the design of the tradeoff between C and A is very +flexible. A good example is CRDT.) + +# A simple CRDT case + +We can use a few simple examples to get a general idea of how CRDTs achieve +**Strong Eventual Consistency**. + +> **Grow-only Counter** + +How can we count the number of times something happens in a distributed system +without locking? + +G-Counter + +- Let each copy increments only its own counter => no locking synchronization & + no conflicts +- Each copy keeps the count values of all other copies at the same time +- Number of occurrences = sum of count values of all copies +- Since each copy only updates its own count and does not conflict with other + counters, this type satisfies consistency after message synchronization + +> **Grow-only Set** + +G-Set + +- The elements in a Grow-only Set can only be increased and not decreased +- To merge two such states, you only need to do a merge set +- This type satisfies consistency after message synchronization because there + are no conflicting operations since the elements only grow and do not + decrease. + +Both of these methods are CRDTs, and they both satisfy the following properties + +- They can both be updated independently and concurrently, without coordination + (locking) between replicas +- There is no possibility of conflict between multiple updates +- Final consistency can always be guaranteed + +# Introduction to the Principle + +There are two types of CRDTs: Op-based CRDTs and State-based CRDTs. This article +focuses on the concept of Op-based CRDTs. + +Op-based CRDTs operate on the principle that if two users perform identical +sequences of operations, the final state of the document should also be +identical. To achieve this, each user saves all the operations performed on the +data (Operations) and synchronizes these Operations with other users to ensure a +consistent final state. A critical challenge in this approach is ensuring the +order of Operations remains consistent, especially when parallel modification +operations occur. To address this, Op-based CRDTs require that all possible +parallel Operations be commutative, satisfying the final consistency +requirement. + +# Comparison of CRDT and OT + +Both CRDT and [Operation Transformation(OT)][ot] can be used in online +collaborative applications, with the following differences + +| OT | CRDT | +| :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------ | +| OT relies on a centralized server for collaboration; [it is extremely difficult to make it work in a distributed environment](https://digitalfreepen.com/2018/01/04/operational-transform-hard.html) | CRDT algorithm can be used to synchronize data through a P2P approach synchronization | +| The earliest paper on OT was presented in 1989 | The earliest paper on CRDT appeared in 2006 | +| The OT algorithm is designed with higher complexity to ensure consistency | The CRDT algorithm is designed to be simpler to ensure consistency | +| It is easier to design OT to preserve user intent | It is more difficult to design a CRDT algorithm that preserves user intent | +| OT does not affect document size | CRDT documents are larger than the original document data | + +[a highly-available move operation for replicated trees]: + https://martin.kleppmann.com/papers/move-op.pdf +[moving elements in list crdts]: + https://martin.kleppmann.com/papers/list-move-papoc20.pdf +[interleaving anomalies in collaborative text editors]: + https://martin.kleppmann.com/papers/interleaving-papoc19.pdf +[conflict-free replicated data types]: https://readpaper.com/paper/1516319412 +[5000x faster crdts: an adventure in optimization]: + https://josephg.com/blog/crdts-go-brrr/ +[json crdt]: https://arxiv.org/abs/1608.03960 +[yata]: + https://www.researchgate.net/publication/310212186_Near_Real-Time_Peer-to-Peer_Shared_Editing_on_Extensible_Data_Types +[yjs]: https://github.com/yjs/yjs +[loro]: https://loro.dev +[automerge]: https://github.com/automerge/automerge +[half grid]: https://zh.wikipedia.org/wiki/%E5%8D%8A%E6%A0%BC +[ot]: https://en.wikipedia.org/wiki/Operational_transformation +[crdts and the quest for distributed consistency]: + https://www.infoq.com/presentations/crdt-distributed-consistency/ + + +# FILE: pages/docs/concepts/oplog_docstate.mdx + +--- +keywords: "crdt, oplog, snapshot, doc state, checkout, version, architecture, memory, performance" +description: "Understanding Loro's architectural separation of OpLog and DocState for efficient CRDT operations and version control" +--- + +# OpLog and DocState + +## Quick Reference + +- **OpLog** = Sequence of events/operations that compose the document history +- **DocState** = Current materialized state of the document + +This separation enables time travel, efficient sync, and flexible storage strategies. + +## Key Concepts + +**OpLog (Operation Log)** +- Append-only sequence of operations +- Causal relationships and metadata + +**DocState (Document State)** +- Current materialized view of the document +- Actual data structures and values +- What your app reads and displays + +## Benefits + +- **Memory efficiency**: Load OpLog without state (relay servers) +- **Time travel**: Navigate history without losing the log +- **Fast startup**: Load state via snapshots, fetch history later +- **Flexible storage**: Store separately for optimization + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); + +// Edit updates both +doc.getText("text").insert(0, "Hello"); +console.log(doc.oplogVersion()); // Latest known version +console.log(doc.version()); // The version of the current state of the document +// If the document is attached, they are the same. +``` + +## Time Travel & Detachment + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").insert(0, "v1"); +const v1 = doc.frontiers(); + +doc.getText("text").insert(2, " -> v2"); + +// Checkout old version - DocState diverges from OpLog +doc.checkout(v1); +console.log(doc.version()); // v1 state +console.log(doc.oplogVersion()); // Still has v2 +console.log(doc.isDetached()); // true + +// Return to latest +doc.attach(); +``` + +**Detached state**: DocState shows old version while OpLog has all operations. Editing disabled by default. + + +## Export Strategies + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +// ---cut--- +// Update: OpLog only (sync) +const update = doc.export({ mode: "update" }); + +// Snapshot: OpLog + DocState (persistence) +const snapshot = doc.export({ mode: "snapshot" }); + +// Shallow: Minimal OpLog + DocState (fast startup) +const shallow = doc.export({ mode: "shallow-snapshot", frontiers: doc.frontiers() }); +``` + +## Common Patterns + +### Relay Server (OpLog Only) +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +class RelayServer { + private doc: LoroDoc; + constructor() { + this.doc = new LoroDoc(); + this.doc.detach(); // Never materialize state + } + + handleUpdate(update: Uint8Array) { + this.doc.import(update); + } +} +``` + +## Best Practices + +- **Export modes**: Updates for sync, snapshots for persistence, shallow for startup +- **Optimize by use case**: + - Editors: Both in memory + - Relays: OpLog only + +## Related Documentation + +- [Attached/Detached States](./attached_detached) - Document and container states +- [Shallow Snapshots](./shallow_snapshots) - History trimming +- [Time Travel](../tutorial/time_travel) - Using checkout + + +# FILE: pages/docs/concepts/container.mdx + +# Container + +Containers are the fundamental building blocks in Loro for organizing and structuring collaborative data. They provide typed data structures that automatically merge when concurrent edits occur. + +## Container Types + +Loro provides several container types, each optimized for different use cases: + +- **LoroMap**: Key-value pairs with Last-Write-Wins semantics +- **LoroList**: Ordered sequences that merge concurrent insertions +- **LoroText**: Text with character-level merging and rich text support +- **LoroTree**: Hierarchical tree structures with move operations +- **LoroMovableList**: Lists with reordering capabilities +- **LoroCounter**: Numerical values with increment/decrement operations + +## Container States: Attached vs Detached + +Containers in Loro exist in two distinct states that affect their behavior and identity. + +### Detached Containers + +A container is **detached** when created directly using constructors: + +```ts twoslash +import { LoroMap, LoroText, LoroList } from "loro-crdt"; +// ---cut--- +// These containers are all detached +const map = new LoroMap(); +const text = new LoroText(); +const list = new LoroList(); +``` + +Characteristics of detached containers: +- Not yet part of any document +- Have a default placeholder ContainerID +- Can be used as templates or temporary data structures +- Will get a proper ContainerID when inserted into a document + +### Attached Containers + +A container becomes **attached** when it's part of a document hierarchy: + +```ts twoslash +import { LoroDoc, LoroMap, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); + +// Root containers are immediately attached +const rootMap = doc.getMap("myMap"); +const rootText = doc.getText("myText"); + +// Child containers: the returned value is attached +const detachedChild = new LoroText(); +const attachedChild = rootMap.setContainer("child", detachedChild); +// Note: detachedChild remains detached +// attachedChild is the attached version with proper ContainerID +``` + +Characteristics of attached containers: +- Belong to a specific document +- Have a proper ContainerID that uniquely identifies them +- Changes are tracked in the document's history +- Can be synchronized across peers + +## Container IDs + +Every attached container has a unique ContainerID that identifies it within the distributed system. The ID generation depends on the container type: + +- **Root containers**: ID derived from their name (e.g., "myMap" in `doc.getMap("myMap")`) +- **Child containers**: ID based on the operation that created them (OpID) + +This deterministic ID generation ensures that: +- The same container can be identified across all peers +- Container IDs are not random but contextually determined +- A detached container cannot have its final ID until insertion + +## Working with Containers + +### Creating Root Containers + +Root containers are created through the document API and are immediately attached: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); + +// These methods create or get root containers +const map = doc.getMap("settings"); +const text = doc.getText("content"); +const list = doc.getList("items"); +const tree = doc.getTree("hierarchy"); +``` + +### Nesting Containers + +Containers can be nested to create complex data structures: + +```ts twoslash +import { LoroDoc, LoroMap, LoroList, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const rootMap = doc.getMap("root"); + +// Method 1: Using setContainer (returns attached container) +const childText = rootMap.setContainer("description", new LoroText()); + +// Method 2: Using insertContainer for lists +const list = doc.getList("items"); +const childMap = list.insertContainer(0, new LoroMap()); +``` + +## Container Overwrites + +When initializing child containers in parallel, overwrites can occur instead of +automatic merging. For example: + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const a: string = "hello"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +// Parallel initialization of child containers +const docB = doc.fork(); +const textA = doc.getMap("map").setContainer("text", new LoroText()); +textA.insert(0, "A"); +const textB = docB.getMap("map").setContainer("text", new LoroText()); +textB.insert(0, "B"); + +doc.import(docB.export({ mode: "update" })); +// Result: Either { "meta": { "text": "A" } } or { "meta": { "text": "B" } } +``` + +This behavior poses a significant risk of data loss if the editing history is +not preserved. Even when the complete history is available and allows for data +recovery, the recovery process can be complex. + + + +When a container holds substantial data or serves as the primary storage for +document content, overwriting it can lead to the unintended hiding/loss of +critical information. For this reason, it is essential to implement careful and +systematic container initialization practices to prevent such issues. + +### Mergeable Containers + +The overwrite happens because `setContainer` creates a regular child container. +Its Container ID includes the operation that created it, so two peers that +create `map["text"]` concurrently create two different Text containers. + +When a child container should be identified by its logical position in a Map, +use a mergeable child container instead: + +```ts no_run +const text = doc.getMap("map").ensureMergeableText("text"); +text.insert(0, "A"); +``` + +Peers that call `ensureMergeableText("text")` on the same parent Map address the +same logical Text container. The same pattern is available for Map, List, +MovableList, Tree, and Counter children through the `ensureMergeable*` methods. + +### Best Practices + +1. For dynamic Map keys where peers may lazily create the same child, use + `ensureMergeableText`, `ensureMergeableMap`, `ensureMergeableList`, and the + other `ensureMergeable*` methods. + +2. When the structure is fixed and known ahead of time: + - If possible, initialize all child containers during the map container's + initialization + +3. Use regular `setContainer` / `insertContainer` when each creation should + produce a distinct child object, or when you are modeling replacement rather + than shared initialization. + +4. A unique root container name can still be a good fit when the child naturally + belongs at the root, for example `doc.getMap("user." + userId)`. + + +## Related Concepts + +- [Container ID](/docs/advanced/cid): Deep dive into how Container IDs work +- [Choosing CRDT Types](/docs/concepts/choose_crdt_type): Guide for selecting the right container type +- [Composition](/docs/tutorial/composition): How to compose containers into complex structures + + +# FILE: pages/docs/concepts/attached_detached.mdx + +--- +keywords: "attached, detached, state, container, document, version control" +description: "Understanding attached and detached states in Loro - two different but related concepts" +--- + +# Attached vs Detached States + +## Quick Reference + +Loro uses "attached/detached" in two distinct contexts: + +1. **Document States** - Version synchronization (latest vs historical) +2. **Container States** - Document membership (belongs to doc vs standalone) + +⚠️ These are independent concepts - a container can be attached to a detached document. + +## Document States + +### Attached (Default) +- Synchronized with latest OpLog version +- Normal editing mode +- All operations applied immediately + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +console.log(doc.isDetached()); // false - normal state +``` + +### Detached (Time Travel) +- Viewing historical version +- Editing disabled by default +- OpLog has newer operations not shown + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").insert(0, "v1"); +const v1 = doc.frontiers(); +doc.getText("text").insert(2, " -> v2"); + +// Time travel to v1 +doc.checkout(v1); +console.log(doc.isDetached()); // true +console.log(doc.getText("text").toString()); // "v1" +``` + + +## Container States + +### Detached (Standalone) +- Created with constructors (`new LoroMap()`) +- Not part of any document +- No valid ContainerID +- Used as templates + +```ts twoslash +import { LoroMap } from "loro-crdt"; +// ---cut--- +const map = new LoroMap(); +console.log(map.isAttached()); // false +``` + +### Attached (Document Member) +- Part of document hierarchy +- Has ContainerID +- Changes tracked in document + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const map = doc.getMap("data"); // Attached + +// Adding detached container returns attached version +const detached = new LoroText(); +const attached = map.setContainer("text", detached); +console.log(detached.isAttached()); // false - original unchanged +console.log(attached.isAttached()); // true - new attached copy +``` + +## Key Differences + +| Aspect | Document Attached/Detached | Container Attached/Detached | +|--------|---------------------------|-----------------------------| +| **Purpose** | Version control | Document membership | +| **When** | Time travel, branching | Adding to document | +| **Independence** | N/A | Independent from doc state | +| **Editing** | Restricted when detached | Always allowed | + +## Common Use Cases + +### Time Travel +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const v1 = doc.frontiers(); +// ... more edits ... +doc.checkout(v1); // View old version +doc.attach(); // Return to latest +``` + +### Container Templates +```ts twoslash +import { LoroDoc, LoroMap } from "loro-crdt"; +// ---cut--- +const template = new LoroMap(); +template.set("type", "task"); + +const doc = new LoroDoc(); +const list = doc.getList("tasks"); +// Reuse template multiple times +list.insertContainer(0, template); +list.insertContainer(1, template); +``` + +## Related Documentation + +- [OpLog and DocState](./oplog_docstate) - Understanding version states +- [Containers](./container) - Container types and usage +- [Time Travel](../tutorial/time_travel) - Using checkout and branching + + +# FILE: pages/docs/concepts/import_status.mdx + +# Import Status + +## Quick Reference + +**Import status** tells you what operations were applied and what's pending due to missing dependencies. Essential for handling out-of-order updates in distributed systems. + +## Status Structure + +```ts twoslash +interface ImportStatus { + success: PeerVersionRange; // What was applied + pending?: PeerVersionRange; // What needs dependencies +} + +interface PeerVersionRange { + [peerId: `${number}`]: { + start: number; // Inclusive + end: number; // Exclusive (e.g., 0-50 = ops 0-49) + }; +} +``` + + +## Pending Operations + +Operations become pending when they depend on missing operations (causal dependencies). + +### Common Scenario: Out-of-Order Delivery + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); + +// Peer A creates ops 0-4, then 5-10 +const peerA = new LoroDoc(); +peerA.getText("text").insert(0, "Hello"); +const update1 = peerA.export({ mode: "update" }); + +peerA.getText("text").insert(5, " World"); +const update2 = peerA.export({ mode: "update", from: peerA.version() }); + +// If update2 arrives first: +const status = doc.import(update2); +console.log(status.pending); // Ops 10-19 pending, need 0-9 + +// Import missing dependencies: +doc.import(update1); // Both updates now applied +``` + +## Handling Pending Operations + +```ts twoslash +import { LoroDoc, VersionVector } from "loro-crdt"; +// ---cut--- +async function handleImport( + doc: LoroDoc, + update: Uint8Array, + fetchMissing: (from: VersionVector) => Promise +) { + const status = doc.import(update); + + if (status.pending) { + const missing = await fetchMissing(doc.version()); + doc.import(missing); + } +} +``` + + +## Best Practices + +### Always Check Status + +```ts twoslash no_run +import { LoroDoc } from "loro-crdt"; +const update = new Uint8Array(); +// ---cut--- +const doc = new LoroDoc(); +const status = doc.import(update); +if (status.pending) { + console.warn("Operations pending:", status.pending); + // Fetch missing updates +} +``` + +### Use Batch Import +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const updates: Uint8Array[] = [ + //... +]; +const doc = new LoroDoc(); +const batchStatus = doc.importBatch(updates); +// Single status check for all updates +``` + +## Related Documentation + +- [Version Vector](./version_vector) - Understanding version vectors +- [Synchronization Guide](../tutorial/sync) - Complete sync patterns +- [PeerID Management](./peerid_management) - How peer IDs affect import + + +# FILE: pages/docs/concepts/version_vector.mdx + +--- +keywords: "version vector, logical clock, distributed systems, CRDT, synchronization" +description: "Understanding Version Vectors in Loro - complete peer state tracking for synchronization" +--- + +# Version Vector + +Version Vectors are a fundamental concept in distributed systems that track the complete state of all peers by recording how many operations each peer has contributed. + +## What is a Version Vector? + +A Version Vector is a map from peer IDs to operation counters, explicitly listing every peer and their operation count. It provides a complete picture of which operations are included in a version. + +**Example**: `{ "peer-A": 5, "peer-B": 3 }` means this version includes operations 0-4 from peer A and operations 0-2 from peer B. + +## Key Characteristics + +- **Complete information**: Explicitly lists all peers and their operation counts +- **Grows with peer count**: Size increases as more peers join +- **No dependency on history**: Can determine included operations without accessing the operation log +- **Enables version comparison**: Can easily check if one version includes another + +## Basic Usage + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setPeerId("1"); +const text = doc.getText("content"); +text.insert(0, "Hello"); + +// Get version vector +const vv = doc.version(); +console.log(vv.toJSON()); // Map { "1" => 1 } +``` + +## When to Use Version Vectors + +Version Vectors are ideal for: + +1. **Synchronization protocols** - Determine what updates to send between peers +2. **Network communication** - Self-contained version information +3. **Distributed systems** - Track state across multiple nodes + +## Comparison with Frontiers + +| Aspect | Version Vectors | Frontiers | +|--------|----------------|-----------| +| **Size** | O(number of peers) | O(1-2) typically | +| **Information** | Complete peer states | Boundary operations only | +| **Use Case** | Synchronization | Checkpoints | +| **History Required** | No | Yes, for full information | + +## Conversion with Frontiers + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.version(); +const frontiers = doc.vvToFrontiers(vv); // Convert to Frontiers +const backToVV = doc.frontiersToVV(frontiers); // Convert back +``` + +## Related Documentation + +- [Frontiers](./frontiers) - Compact version representation +- [Version Tutorial](../tutorial/version) - Working with versions +- [Version Deep Dive](../advanced/version_deep_dive) - Technical details + + +# FILE: pages/docs/concepts/event_graph_walker.mdx + +--- +keywords: "crdt, event graph walker, eg-walker, synchronization, collaboration, algorithm, innovation" +description: "Comprehensive guide to Event Graph Walker (Eg-Walker), a revolutionary CRDT algorithm that enables simpler metadata and better performance in Loro" +--- + +# Event Graph Walker (Eg-Walker) + +Event Graph Walker (Eg-Walker) is a revolutionary CRDT algorithm that fundamentally changes how collaborative editing systems handle concurrent operations. Instead of storing complex CRDT metadata, Eg-Walker enables the use of simple indices by efficiently replaying relevant history when needed. + +> **Important Note**: Loro is not a strict implementation of Event Graph Walker. Rather, Loro is heavily inspired by Eg-Walker's design philosophy and incorporates its key insights to achieve similar properties - particularly the ability to use simple indices instead of complex CRDT metadata, and the efficient replay mechanism for handling concurrent operations. + +## The Problem: Complex CRDT Metadata + +Traditional CRDTs require extensive metadata to maintain consistency across distributed systems: + +- **RGA algorithm**: Requires operation ID and Lamport timestamp of the left neighbor +- **Yjs/Fugue**: Needs operation IDs of both left and right neighbors +- **Storage overhead**: Each operation carries significant metadata for position tracking +- **Tombstone accumulation**: Deleted content must be retained indefinitely + +This metadata complexity leads to: +- Increased storage requirements +- Slower performance as documents grow +- Complex garbage collection challenges +- Higher network bandwidth usage + +## The Innovation: Simple Indices with Smart Replay + +Eg-Walker introduces a paradigm shift: **record simple indices, replay when needed**. + +import { ReactPlayer } from "../../../components/video"; + + + +### Core Concept + +Instead of storing complex position descriptors, Eg-Walker: +1. Records operations with **simple indices** at the time of execution +2. Maintains a directed acyclic graph (DAG) of the edit history +3. **Replays relevant portions** of history when merging concurrent changes +4. Reconstructs the exact CRDT state only when needed + +## How It Works + +### The Event Graph + +In collaborative editing, parallel edits naturally form a directed acyclic graph (DAG), similar to Git's history: + +``` + A --- B --- C (User 1) + / \ +Root Merge + \ / + D --- E --- F (User 2) +``` + +Each node represents an operation, and edges represent causal dependencies. + +### The Algorithm + +#### 1. **Local Operations: Direct and Fast** + +When a user makes a local edit: +- Record the operation with its **current index position** +- No complex metadata calculation needed +- Immediate application to the document + +Example: +```javascript no_run +// Traditional CRDT (e.g., RGA) +insert({ + char: 'a', + leftId: 'op-123', + timestamp: 1234567890, + siteId: 'user-1' +}) + +// With Eg-Walker +insert({ + char: 'a', + index: 5 // Simple index! +}) +``` + +#### 2. **Merging Remote Operations: Smart Replay** + +When receiving remote operations: + +1. **Find the Lowest Common Ancestor (LCA)** between local and remote versions +2. **Replay operations** from the LCA to both versions +3. **Construct temporary CRDT state** to calculate effects +4. **Apply the merged result** + +The key insight: You only replay the **divergent portion** of history, not the entire document. + +#### 3. **Index Transformation** + +Consider this scenario: +- You highlight text from index 10 to 20 +- Concurrently, someone inserts 5 characters at index 3 +- Your highlight should shift to indices 15 to 25 + +Eg-Walker handles this by: +1. Identifying concurrent operations through the event graph +2. Replaying operations in causal order +3. Transforming indices based on the replay sequence + +## Performance Benefits + +### 1. **Reduced Storage** + +| Aspect | Traditional CRDT | Eg-Walker | +|--------|-----------------|-----------| +| Tombstone storage | Permanent | Can be garbage collected | +| Document growth | Linear with all operations | Linear with active operations | + +### 2. **Faster Local Updates** + +- **No metadata computation**: Direct index-based operations +- **No tombstone traversal**: Clean document state +- **Predictable performance**: O(1) or O(logN) for most local operations + +### 3. **Efficient Synchronization** + +- **Minimal replay**: Only divergent history portions +- **Bounded computation**: Proportional to concurrent edits, not document size +- **Smart caching**: Reuse computed states when possible + +## Implementation in Loro + +While Loro is not a pure Eg-Walker implementation, it draws heavily from Eg-Walker's innovations to achieve similar benefits. Loro implements Fugue (a modern CRDT algorithm) with Eg-Walker-inspired optimizations: + +- **Fugue's correctness guarantees** for text editing +- **Eg-Walker-inspired efficiency** in storage and computation +- **Simple index-based operations** at the API level +- **Smart replay mechanisms** for merging concurrent changes + +This hybrid approach provides: +```javascript no_run +// Simple API with powerful internals +const doc = new Loro(); +const text = doc.getText("content"); + +// Just use indices - Eg-Walker handles the complexity +text.insert(5, "Hello"); +text.delete(2, 3); + +// Efficient merging happens automatically +doc.import(remoteUpdate); +``` + +## Garbage Collection Revolution + +One of Eg-Walker's most significant advantages is **safe garbage collection**: + +### Traditional CRDTs +- Must keep all tombstones forever +- Document size grows unbounded +- Complex protocols for coordinated cleanup + +### With Eg-Walker +- Operations synchronized across all endpoints can be **safely removed** +- No new operations will be concurrent with fully-synchronized ones +- Document size remains proportional to **active content** + +## Research Foundation + +Eg-Walker is based on rigorous academic research: + +> [Collaborative Text Editing with Eg-walker: Better, Faster, Smaller](https://arxiv.org/abs/2409.14252) +> By: Joseph Gentle, Martin Kleppmann + +The algorithm has been proven to: +- Maintain strong eventual consistency +- Preserve all CRDT correctness properties +- Significantly reduce computational and storage overhead + + +# FILE: pages/docs/concepts/operations_changes.mdx + +# Operations and Changes + +## Quick Reference + +**Operations** are atomic edits. **Changes** are logical groups of operations with metadata. Understanding these helps optimize sync and performance. + +## Key Concepts + +### Operations +- Atomic units of change (insert a single Unicode character, delete a single character, insert an new entry to a map, etc.) +- Automatically merged internally for efficiency +- Each has unique ID: `(peerId, counter)` + +### Changes +- Groups of consecutive operations +- Include metadata (timestamp, dependencies, peer ID) +- Created by `commit()` or auto-commit + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Hello"); // Operation +text.insert(5, " World"); // Operation +doc.commit(); // Groups into one Change +``` + +## Automatic Merging + +Consecutive operations from same peer merge into one Change: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "abc"); +doc.commit(); // Change #1 + +text.insert(3, "def"); +doc.commit(); // Merges with #1 (same peer, consecutive) +``` + +## When New Changes Are Created + +1. **Cross-peer dependencies**: After importing remote operations +2. **Time separation**: When timestamps enabled and > the merge interval (default 1000s) between commits +3. **Different commit messages**: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").insert(0, "v1"); +doc.commit(); // Change #1 + +// Import from another peer +const doc2 = new LoroDoc(); +doc2.getText("text").insert(0, "v1"); +const remote = doc2.export({ mode: "update" }); +doc.import(remote); + +// Next commit creates new Change (dependency on remote) +doc.getText("text").insert(0, "v2"); +doc.commit(); // Change #2 +``` + + +## Impact on Sync & Storage + +- **History**: Changes track logical units of work +- **Sync**: Dependencies ensure causal ordering +- **Storage**: Auto-merging reduces metadata overhead + +## Related Documentation + +- [Transaction Model](./transaction_model) - Grouping operations +- [Version Vector](./version_vector) - How Changes form versions +- [Synchronization](../tutorial/sync) - Using Changes for sync + + +# FILE: pages/docs/concepts/when_not_crdt.mdx + +--- +keywords: "crdt, limitations, constraints, business logic, invariants" +description: "When CRDTs are not the right tool: hard invariants, exclusivity, ordering, and validation" +--- + +# When Not to Use CRDTs + +CRDTs shine for collaborative editing and offline-friendly applications. +But they are not a universal replacement for coordination, transactions, +or authorization. Use this guide to recognize when CRDTs are a poor fit and what to use instead. + +## Quick Reference + +CRDTs merge; they do not reject. This clashes with requirements that demand global agreement at the time of the action: +- **Hard invariants**: Must never be violated (e.g., balance ≥ 0) +- **Exclusive ownership**: Only one winner per slot/resource + +## Why CRDTs fall short here + +- Merging is monotonic: remote edits cannot be “rolled back” on arrival +- No locks or transactions across replicas +- Check-then-set across peers needs coordination, not just convergence + +## Common problem scenarios + +### 1) Exclusive resource management + +Multiple users book the same room/time concurrently. + +```ts no_run +// With a CRDT-only model both intents exist after merge +// The app must pick a winner out-of-band +book("A101@2pm", { by: "A" }) +book("A101@2pm", { by: "B" }) +// After sync: both entries are present → invariant violated without coordination +``` + +Use instead: +- Transactional authority (central server or database) +- Distributed consensus (Raft/Paxos) for ownership +- Hybrid: CRDT for UI drafts, authoritative booking via server + +### 2) Financial transactions + +Two withdrawals race on a $100 account. + +```ts no_run +// Each client applies its own withdrawal locally +withdraw(60) // A +withdraw(60) // B +// After merge, naive addition overdrafts without an authority +``` + +Use instead: +- ACID transactions on an authoritative store +- Command validation with a single writer per account +- Event-sourced ledger with server-side checks + +### Other cases to avoid CRDT-as-the-only-source + +- Uniqueness constraints (usernames, slugs, one primary per group) +- Authorization decisions that must be enforced at write time +- Global invariants (referential integrity, cross-document totals) + +## When CRDTs work well + +CRDTs excel when merged results are meaningful and temporary inconsistency is acceptable: + +- ✅ Collaborative text, lists, maps, rich presence +- ✅ Comments, reactions, drafts, annotations +- ✅ Offline edits where convergence is enough + + + +# FILE: pages/docs/concepts/cursor_stable_positions.mdx + +--- +keywords: "cursor, stable position, collaborative editing, CRDT, concurrent edits, selection, annotation, caret" +description: "Understanding cursor and stable position systems in Loro for maintaining accurate positions across concurrent edits" +--- + +# Cursor and Stable Positions + +## Quick Reference + +**Cursors** maintain stable positions across concurrent edits by anchoring to operation IDs instead of indices. Essential for collaborative editing features like collaborative cursors and persistent annotations. + +## How It Works + +Cursors anchor to operation IDs and ContainerIDs, not indices: + +``` +Text: H e l l o W o r l d +Op IDs: 1 2 3 4 5 6 7 8 9 A B +Cursor: References ID 5 (after 'o') + +After concurrent insert at start: +Text: N e w H e l l o W o r l d +Op IDs: C D E F 1 2 3 4 5 6 7 8 9 A B +Cursor: Still references ID 5 - position automatically adjusted +``` + +## Side Parameter + +- **`Side.Before` (-1)**: Stay before the target +- **`Side.Middle` (0)**: On the target (default) +- **`Side.After` (1)**: Stay after the target + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "ABC"); + +const cursor = text.getCursor(1, -1); // Before 'B' +text.insert(1, "X"); // Insert at cursor +// Result: "AXBC", cursor still before 'B' +const pos = doc.getCursorPos(cursor!); +console.log(pos); // { offset: 2, side: Side.Before } +``` + +## Common Use Cases + +### Text Selections +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +// Selection range +const start = text.getCursor(0, -1); // Anchor +const end = text.getCursor(5, 1); // Head +``` + +### List Positions +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const list = doc.getList("items"); +list.insert(0, ["a", "b", "c"]); + +const cursor = list.getCursor(0, 1); // After "a" +list.insert(0, "new"); // Cursor adjusts automatically +``` + +## Related Documentation + +- [Text Container](../tutorial/text) - Text editing with cursors +- [Cursor Tutorial](../tutorial/cursor) - Building collaborative features +- [Events](../tutorial/event) - Cursor change events + + +# FILE: pages/docs/concepts/shallow_snapshots.mdx + +# Shallow Snapshots + +## Quick Reference + +**Shallow snapshots** are like Git's shallow clone - maintain current state while removing old history. Essential for privacy compliance, storage optimization, and performance. + + +## Basic Usage + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// ... extensive editing history ... + +// Regular snapshot - full history +const full = doc.export({ mode: "snapshot" }); + +// Shallow snapshot - trimmed history +const shallow = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.frontiers(), +}); + +// Typically 70-90% smaller +console.log(`Size reduction: ${100 - (shallow.length / full.length * 100)}%`); +``` + +## Content Redaction + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("content"); + +// Sensitive data added +text.insert(0, "SSN: 123-45-6789. "); +text.insert(18, "Public info."); + +// Redact sensitive part +text.delete(0, 18); + +// Create clean snapshot +const redacted = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.frontiers(), +}); +// Sensitive data permanently removed from history +``` + +## Synchronization Limitations + +⚠️ **Important**: Peers can only sync if they have versions after the shallow snapshot point. + + +## Common Patterns + +### Archive and Trim +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +async function archiveAndTrim(doc: LoroDoc) { + // 1. Archive full history + const full = doc.export({ mode: "snapshot" }); + await saveToArchive(full); + + // 2. Create shallow for active use + const shallow = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.frontiers(), + }); + + return shallow; +} + +async function saveToArchive(data: Uint8Array) { + // Save to cold storage +} +``` + +### Privacy-Aware Design +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +class PrivacyDoc { + constructor(private doc: LoroDoc) {} + + async redactSensitive() { + // Delete sensitive content + this.doc.getText("private").delete(0, this.doc.getText("private").length); + + // Create clean snapshot + return this.doc.export({ + mode: "shallow-snapshot", + frontiers: this.doc.frontiers(), + }); + } +} +``` + +## Best Practices + +- **Coordinate before trimming**: Ensure all peers synchronized +- **Archive before deletion**: Keep full history backup if needed + +## Related Documentation + +- [Encoding](../tutorial/encoding) - Different export options +- [Version Tutorial](../tutorial/version) - Managing document versions +- [Event Graph Walker](./event_graph_walker) - Algorithm powering shallow snapshots + + +# FILE: pages/docs/concepts/transaction_model.mdx + +# Transaction Model + +## Quick Reference + +**Loro transactions are NOT database ACID transactions.** They are operation bundling mechanisms for event emission. + +## Key Concepts + +- **Purpose**: Bundle related operations and control event emission +- **No rollback**: Failed operations don't undo previous ones +- **Event batching**: Single event for all operations in transaction +- **History grouping**: Operations stay together for undo/redo + +## Basic Usage + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); + +// Without transaction - multiple events +const text = doc.getText("text"); +text.insert(0, "Hello"); +doc.commit(); // Event emitted +text.insert(5, " World"); +doc.commit(); // Another event +``` + +## How Transactions Work + +### Pending Operations +Local operations are **pending** by default and won't emit events until committed: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); + +// Operation is pending, no event emitted yet +text.insert(0, "Hello"); + +// Explicit commit triggers event emission +doc.commit(); +``` + +### Implicit Commits +Certain operations trigger implicit commits automatically: + +```ts twoslash no_run +import { LoroDoc } from "loro-crdt"; +const update = new Uint8Array(); +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Hello"); // Pending operation + +// These operations trigger implicit commit: +doc.import(update); // Implicit commit before import +// or +doc.checkout(doc.frontiers()); // Implicit commit before checkout +// or +doc.export({ mode: "update" }); // Implicit commit before export +``` + +### Transaction Guarantees + +- All operations share the same timestamp (if enabled) +- Operations grouped in a single Change +- One event emitted after transaction completes +- Operations committed together at transaction end +- Before committing, the local edits are not synced to other peers + +## Related Documentation + +- [Operations and Changes](./operations_changes) - How transactions create Changes +- [Event System](../tutorial/event) - Understanding event emission +- [Undo/Redo](../advanced/undo) - Using transactions for undo boundaries + + +# FILE: pages/docs/performance/index.md + +--- +keywords: "loro, yjs, automerge, benchmark, memory, crdt" +description: "CRDT benchmarks, comparing the performance of Loro and popular CRDTs" +--- + +# JS/WASM Benchmarks + +> The primary role of these benchmarks should be to serve as indicators of the absence of performance pitfalls rather than as measures of which project is superior. This is because different projects consistently make different trade-offs. It is inaccurate to claim that Project A is superior to Project B simply because A performs better in certain benchmarks, while Project B may excel in other areas by a significant margin. + +The benchmark can be reproduced using the [crdt-benchmarks](https://github.com/zxch3n/crdt-benchmarks) repo. + +- The benchmarks were performed on MacBook Pro M1 2020 with 16GB RAM +- loro-old is the version of loro on 2023-11-10, it's compiled from + [this commit](https://github.com/loro-dev/loro/tree/c1613ee680c6a4757e55fcda76e4f5f627daeb56). + Loro has undergone numerous changes since then, particularly in terms of + [encoding schema](https://github.com/loro-dev/loro/pull/219), shifting from a + performance-focused version to one that prioritizes compatibility. Because we + want Loro to have good backward and forward compatibility after reaching + version 1.0, we have adopted a more easily extensible encoding method. It is + slower than the encoding method used in the `loro-old` version, but it better + ensures our ability to iterate quickly after reaching version 1.0 without + introducing breaking changes. +- There is a more exchaustive benchmark at the bottom that only runs benchmarks + on Yjs. +- Automerge can perform the `B4` benchmark in about 1 second (see `time`) if all + changes are applied within a single `change` transaction. However, our + benchmarks test individual edits that generate individual update events as + this more closely simulates actual user behavior. +- Note that `parseTime` is significantly higher with `automerge` and `loro` when + the initial document is not empty (e.g. when syncing content from a remote + server). +- Loro and Automerge can store a complete DAG of editing history for each +keystroke, but Yjs requires additional storage for a Version Vector + Delete +Set for each version saved, which incurs significant extra overhead beyond the +document size reported. +
+Benchmark setup + +#### B1: No conflicts + +Simulate two clients. One client modifies a text object and sends update +messages to the other client. We measure the time to perform the task (`time`), +the amount of data exchanged (`avgUpdateSize`), the size of the encoded document +after the task is performed (`docSize`), the time to parse the encoded document +(`parseTime`), and the memory used to hold the decoded document (`memUsed`). + +#### B2: Two users producing conflicts + +Simulate two clients. Both start with a synced text object containing 100 +characters. Both clients modify the text object in a single transaction and then +send their changes to the other client. We measure the time to sync concurrent +changes into a single client (`time`), the size of the update messages +(`updateSize`), the size of the encoded document after the task is performed +(`docSize`), the time to parse the encoded document (`parseTime`), and the +memory used to hold the decoded document (`memUsed`). + +#### B3: Many conflicts + +Simulate `√N` concurrent actions. We measure the time to perform the task and +sync all clients (`time`), the size of the update messages (`updateSize`), the +size of the encoded document after the task is performed (`docSize`), the time +to parse the encoded document (`parseTime`), and the memory used to hold the +decoded document (`memUsed`). The logarithm of `N` was chosen because `√N` +concurrent actions may result in up to `√N^2 - 1` conflicts (apply action 1: 0 +conflict; apply action2: 1 conflict, apply action 2: 2 conflicts, ..). + +#### B4: Real-world editing dataset + +Replay a real-world editing dataset. This dataset contains the +character-by-character editing trace of a large-ish text document, the LaTeX +source of this paper: https://arxiv.org/abs/1608.03960 + +Source: https://github.com/automerge/automerge-perf/tree/master/edit-by-index + +- 182,315 single-character insertion operations +- 77,463 single-character deletion operations +- 259,778 operations totally +- 104,852 characters in the final document + +We simulate one client replaying all changes and storing each update. We measure +the time to replay the changes and the size of all update messages +(`updateSize`), the size of the encoded document after the task is performed +(`docSize`), the time to encode the document (`encodeTime`), the time to parse +the encoded document (`parseTime`), and the memory used to hold the decoded +document in memory (`memUsed`). + +##### [B4 x 100] Real-world editing dataset 100 times + +Replay the [B4] dataset one hundred times. The final document has a size of over +10 million characters. As comparison, the book "Game of Thrones: A Song of Ice +and Fire" is only 1.6 million characters long (including whitespace). + +- 18,231,500 single-character insertion operations +- 7,746,300 single-character deletion operations +- 25,977,800 operations totally +- 10,485,200 characters in the final document + +
+ +| N = 6000 | yjs | ywasm | loro | loro-old | automerge | automerge-wasm | +| :----------------------------------------------------------------------- | ---------------: | --------------: | ---------------: | ---------------: | ---------------: | --------------: | +| Version | 13.6.15 | 0.17.4 | 1.0.0-beta.2 (historical) | 0.15.2 | 2.1.10 | 0.9.0 | +| Bundle size | 84,017 bytes | 938,991 bytes | 2,919,363 bytes | 1,583,094 bytes | 1,696,176 bytes | 1,701,136 bytes | +| Bundle size (gzipped) | 25,105 bytes | 284,616 bytes | 894,460 bytes | 592,039 bytes | 591,049 bytes | 594,071 bytes | +| [B1.1] Append N characters (time) | 141 ms | 171 ms | 164 ms | 115 ms | 279 ms | 110 ms | +| [B1.1] Append N characters (avgUpdateSize) | 27 bytes | 27 bytes | 88 bytes | 58 bytes | 121 bytes | 121 bytes | +| [B1.1] Append N characters (encodeTime) | 1 ms | 0 ms | 3 ms | 0 ms | 5 ms | 6 ms | +| [B1.1] Append N characters (docSize) | 6,031 bytes | 6,031 bytes | 12,382 bytes | 6,219 bytes | 3,992 bytes | 3,992 bytes | +| [B1.1] Append N characters (memUsed) | 0 B | 0 B | 0 B | 0 B | 0 B | 0 B | +| [B1.1] Append N characters (parseTime) | 22 ms | 65 ms | 28 ms | 28 ms | 59 ms | 61 ms | +| [B1.2] Insert string of length N (time) | 1 ms | 1 ms | 0 ms | 0 ms | 18 ms | 14 ms | +| [B1.2] Insert string of length N (avgUpdateSize) | 6,031 bytes | 6,031 bytes | 6,089 bytes | 6,088 bytes | 6,201 bytes | 6,201 bytes | +| [B1.2] Insert string of length N (encodeTime) | 0 ms | 0 ms | 0 ms | 0 ms | 2 ms | 2 ms | +| [B1.2] Insert string of length N (docSize) | 6,031 bytes | 6,031 bytes | 12,313 bytes | 6,146 bytes | 3,974 bytes | 3,974 bytes | +| [B1.2] Insert string of length N (parseTime) | 27 ms | 47 ms | 27 ms | 26 ms | 29 ms | 30 ms | +| [B1.3] Prepend N characters (time) | 118 ms | 30 ms | 111 ms | 47 ms | 272 ms | 73 ms | +| [B1.3] Prepend N characters (avgUpdateSize) | 27 bytes | 26 bytes | 87 bytes | 57 bytes | 116 bytes | 116 bytes | +| [B1.3] Prepend N characters (encodeTime) | 2 ms | 0 ms | 2 ms | 1 ms | 4 ms | 4 ms | +| [B1.3] Prepend N characters (docSize) | 6,041 bytes | 6,040 bytes | 15,414 bytes | 6,165 bytes | 3,988 bytes | 3,988 bytes | +| [B1.3] Prepend N characters (parseTime) | 45 ms | 38 ms | 27 ms | 37 ms | 73 ms | 47 ms | +| [B1.4] Insert N characters at random positions (time) | 128 ms | 101 ms | 113 ms | 51 ms | 268 ms | 89 ms | +| [B1.4] Insert N characters at random positions (avgUpdateSize) | 29 bytes | 29 bytes | 88 bytes | 58 bytes | 121 bytes | 121 bytes | +| [B1.4] Insert N characters at random positions (encodeTime) | 3 ms | 0 ms | 2 ms | 1 ms | 6 ms | 5 ms | +| [B1.4] Insert N characters at random positions (docSize) | 29,571 bytes | 29,554 bytes | 39,040 bytes | 29,503 bytes | 24,743 bytes | 24,743 bytes | +| [B1.4] Insert N characters at random positions (parseTime) | 46 ms | 30 ms | 27 ms | 26 ms | 73 ms | 64 ms | +| [B1.5] Insert N words at random positions (time) | 149 ms | 264 ms | 112 ms | 54 ms | 539 ms | 291 ms | +| [B1.5] Insert N words at random positions (avgUpdateSize) | 36 bytes | 36 bytes | 95 bytes | 65 bytes | 131 bytes | 131 bytes | +| [B1.5] Insert N words at random positions (encodeTime) | 6 ms | 1 ms | 3 ms | 1 ms | 14 ms | 15 ms | +| [B1.5] Insert N words at random positions (docSize) | 87,868 bytes | 87,924 bytes | 135,713 bytes | 98,901 bytes | 96,203 bytes | 96,203 bytes | +| [B1.5] Insert N words at random positions (parseTime) | 50 ms | 33 ms | 27 ms | 33 ms | 101 ms | 111 ms | +| [B1.6] Insert string, then delete it (time) | 1 ms | 0 ms | 2 ms | 1 ms | 39 ms | 31 ms | +| [B1.6] Insert string, then delete it (avgUpdateSize) | 6,053 bytes | 6,052 bytes | 6,189 bytes | 6,179 bytes | 6,338 bytes | 6,338 bytes | +| [B1.6] Insert string, then delete it (encodeTime) | 0 ms | 0 ms | 0 ms | 0 ms | 3 ms | 2 ms | +| [B1.6] Insert string, then delete it (docSize) | 6,040 bytes | 6,039 bytes | 6,409 bytes | 6,145 bytes | 3,993 bytes | 3,993 bytes | +| [B1.6] Insert string, then delete it (parseTime) | 29 ms | 29 ms | 26 ms | 28 ms | 52 ms | 44 ms | +| [B1.7] Insert/Delete strings at random positions (time) | 153 ms | 112 ms | 136 ms | 60 ms | 423 ms | 212 ms | +| [B1.7] Insert/Delete strings at random positions (avgUpdateSize) | 31 bytes | 31 bytes | 100 bytes | 61 bytes | 135 bytes | 135 bytes | +| [B1.7] Insert/Delete strings at random positions (encodeTime) | 3 ms | 1 ms | 3 ms | 1 ms | 13 ms | 11 ms | +| [B1.7] Insert/Delete strings at random positions (docSize) | 41,917 bytes | 41,592 bytes | 81,700 bytes | 51,470 bytes | 59,281 bytes | 59,281 bytes | +| [B1.7] Insert/Delete strings at random positions (parseTime) | 53 ms | 41 ms | 25 ms | 27 ms | 80 ms | 78 ms | +| [B1.8] Append N numbers (time) | 140 ms | 59 ms | 155 ms | 73 ms | 448 ms | 113 ms | +| [B1.8] Append N numbers (avgUpdateSize) | 32 bytes | 32 bytes | 94 bytes | 62 bytes | 125 bytes | 125 bytes | +| [B1.8] Append N numbers (encodeTime) | 2 ms | 0 ms | 2 ms | 2 ms | 6 ms | 6 ms | +| [B1.8] Append N numbers (docSize) | 35,641 bytes | 35,634 bytes | 71,568 bytes | 47,625 bytes | 26,985 bytes | 26,985 bytes | +| [B1.8] Append N numbers (parseTime) | 32 ms | 43 ms | 31 ms | 29 ms | 76 ms | 67 ms | +| [B1.9] Insert Array of N numbers (time) | 3 ms | 3 ms | 13 ms | 9 ms | 47 ms | 22 ms | +| [B1.9] Insert Array of N numbers (avgUpdateSize) | 35,653 bytes | 35,657 bytes | 35,715 bytes | 35,717 bytes | 31,199 bytes | 31,199 bytes | +| [B1.9] Insert Array of N numbers (encodeTime) | 1 ms | 0 ms | 2 ms | 1 ms | 4 ms | 2 ms | +| [B1.9] Insert Array of N numbers (docSize) | 35,653 bytes | 35,657 bytes | 71,578 bytes | 47,646 bytes | 26,953 bytes | 26,953 bytes | +| [B1.9] Insert Array of N numbers (parseTime) | 43 ms | 29 ms | 31 ms | 29 ms | 56 ms | 44 ms | +| [B1.10] Prepend N numbers (time) | 157 ms | 31 ms | 125 ms | 53 ms | 484 ms | 159 ms | +| [B1.10] Prepend N numbers (avgUpdateSize) | 32 bytes | 32 bytes | 93 bytes | 61 bytes | 120 bytes | 120 bytes | +| [B1.10] Prepend N numbers (encodeTime) | 3 ms | 1 ms | 3 ms | 1 ms | 6 ms | 6 ms | +| [B1.10] Prepend N numbers (docSize) | 35,669 bytes | 35,665 bytes | 76,773 bytes | 47,645 bytes | 26,987 bytes | 26,987 bytes | +| [B1.10] Prepend N numbers (parseTime) | 53 ms | 33 ms | 48 ms | 39 ms | 65 ms | 67 ms | +| [B1.11] Insert N numbers at random positions (time) | 160 ms | 121 ms | 125 ms | 56 ms | 517 ms | 121 ms | +| [B1.11] Insert N numbers at random positions (avgUpdateSize) | 34 bytes | 34 bytes | 94 bytes | 60 bytes | 125 bytes | 125 bytes | +| [B1.11] Insert N numbers at random positions (encodeTime) | 2 ms | 1 ms | 4 ms | 1 ms | 8 ms | 6 ms | +| [B1.11] Insert N numbers at random positions (docSize) | 59,132 bytes | 59,137 bytes | 100,632 bytes | 70,901 bytes | 47,746 bytes | 47,746 bytes | +| [B1.11] Insert N numbers at random positions (parseTime) | 52 ms | 50 ms | 58 ms | 41 ms | 428 ms | 80 ms | +| [B2.1] Concurrently insert string of length N at index 0 (time) | 1 ms | 0 ms | 1 ms | 0 ms | 75 ms | 32 ms | +| [B2.1] Concurrently insert string of length N at index 0 (updateSize) | 6,094 bytes | 6,094 bytes | 6,188 bytes | 9,244 bytes | 9,499 bytes | 9,499 bytes | +| [B2.1] Concurrently insert string of length N at index 0 (encodeTime) | 0 ms | 0 ms | 0 ms | 0 ms | 7 ms | 5 ms | +| [B2.1] Concurrently insert string of length N at index 0 (docSize) | 12,151 bytes | 12,151 bytes | 24,735 bytes | 12,281 bytes | 8,011 bytes | 8,011 bytes | +| [B2.1] Concurrently insert string of length N at index 0 (parseTime) | 28 ms | 34 ms | 30 ms | 27 ms | 60 ms | 48 ms | +| [B2.2] Concurrently insert N characters at random positions (time) | 46 ms | 166 ms | 55 ms | 125 ms | 270 ms | 399 ms | +| [B2.2] Concurrently insert N characters at random positions (updateSize) | 33,420 bytes | 33,444 bytes | 23,779 bytes | 350,337 bytes | 27,476 bytes | 1,093,293 bytes | +| [B2.2] Concurrently insert N characters at random positions (encodeTime) | 2 ms | 1 ms | 4 ms | 1 ms | 6 ms | 11 ms | +| [B2.2] Concurrently insert N characters at random positions (docSize) | 66,808 bytes | 66,852 bytes | 79,937 bytes | 59,358 bytes | 50,686 bytes | 50,704 bytes | +| [B2.2] Concurrently insert N characters at random positions (parseTime) | 67 ms | 69 ms | 27 ms | 27 ms | 51 ms | 95 ms | +| [B2.3] Concurrently insert N words at random positions (time) | 105 ms | 459 ms | 68 ms | 120 ms | 435 ms | 630 ms | +| [B2.3] Concurrently insert N words at random positions (updateSize) | 89,143 bytes | 88,994 bytes | 62,640 bytes | 408,723 bytes | 122,485 bytes | 1,185,202 bytes | +| [B2.3] Concurrently insert N words at random positions (encodeTime) | 7 ms | 2 ms | 7 ms | 3 ms | 28 ms | 35 ms | +| [B2.3] Concurrently insert N words at random positions (docSize) | 178,428 bytes | 178,130 bytes | 268,884 bytes | 197,284 bytes | 185,019 bytes | 191,498 bytes | +| [B2.3] Concurrently insert N words at random positions (parseTime) | 82 ms | 67 ms | 31 ms | 30 ms | 134 ms | 184 ms | +| [B2.4] Concurrently insert & delete (time) | 145 ms | 1,243 ms | 118 ms | 282 ms | 653 ms | 1,311 ms | +| [B2.4] Concurrently insert & delete (updateSize) | 140,984 bytes | 141,122 bytes | 123,725 bytes | 798,123 bytes | 298,810 bytes | 2,395,876 bytes | +| [B2.4] Concurrently insert & delete (encodeTime) | 6 ms | 3 ms | 14 ms | 4 ms | 43 ms | 56 ms | +| [B2.4] Concurrently insert & delete (docSize) | 282,112 bytes | 282,358 bytes | 392,151 bytes | 304,592 bytes | 293,831 bytes | 307,291 bytes | +| [B2.4] Concurrently insert & delete (parseTime) | 105 ms | 73 ms | 34 ms | 31 ms | 185 ms | 269 ms | +| [B3.1] 20√N clients concurrently set number in Map (time) | 54 ms | 60 ms | 27 ms | 32 ms | 1,058 ms | 21 ms | +| [B3.1] 20√N clients concurrently set number in Map (updateSize) | 49,167 bytes | 49,162 bytes | 132,376 bytes | 63,832 bytes | 283,296 bytes | 283,296 bytes | +| [B3.1] 20√N clients concurrently set number in Map (encodeTime) | 3 ms | 1 ms | 16 ms | 1 ms | 9 ms | 12 ms | +| [B3.1] 20√N clients concurrently set number in Map (docSize) | 36,763 bytes | 36,751 bytes | 78,764 bytes | 38,428 bytes | 86,165 bytes | 86,164 bytes | +| [B3.1] 20√N clients concurrently set number in Map (parseTime) | 57 ms | 67 ms | 83 ms | 49 ms | 59 ms | 54 ms | +| [B3.2] 20√N clients concurrently set Object in Map (time) | 55 ms | 62 ms | 27 ms | 40 ms | 1,126 ms | 28 ms | +| [B3.2] 20√N clients concurrently set Object in Map (updateSize) | 85,084 bytes | 85,073 bytes | 171,370 bytes | 99,753 bytes | 398,090 bytes | 325,370 bytes | +| [B3.2] 20√N clients concurrently set Object in Map (encodeTime) | 2 ms | 1 ms | 18 ms | 2 ms | 21 ms | 17 ms | +| [B3.2] 20√N clients concurrently set Object in Map (docSize) | 72,682 bytes | 72,659 bytes | 84,255 bytes | 75,227 bytes | 112,588 bytes | 93,401 bytes | +| [B3.2] 20√N clients concurrently set Object in Map (parseTime) | 73 ms | 68 ms | 83 ms | 48 ms | 68 ms | 59 ms | +| [B3.3] 20√N clients concurrently set String in Map (time) | 124 ms | 61 ms | 63 ms | 73 ms | 1,869 ms | 166 ms | +| [B3.3] 20√N clients concurrently set String in Map (updateSize) | 7,826,229 bytes | 7,826,225 bytes | 7,912,520 bytes | 7,840,917 bytes | 8,063,440 bytes | 8,063,440 bytes | +| [B3.3] 20√N clients concurrently set String in Map (encodeTime) | 56 ms | 5 ms | 70 ms | 23 ms | 59 ms | 61 ms | +| [B3.3] 20√N clients concurrently set String in Map (docSize) | 7,813,826 bytes | 7,813,814 bytes | 241,646 bytes | 7,815,537 bytes | 97,997 bytes | 98,008 bytes | +| [B3.3] 20√N clients concurrently set String in Map (parseTime) | 141 ms | 75 ms | 161 ms | 44 ms | 80 ms | 76 ms | +| [B3.4] 20√N clients concurrently insert text in Array (time) | 45 ms | 49 ms | 195 ms | 28 ms | 1,768 ms | 19 ms | +| [B3.4] 20√N clients concurrently insert text in Array (updateSize) | 52,751 bytes | 52,735 bytes | 137,490 bytes | 70,514 bytes | 311,830 bytes | 285,330 bytes | +| [B3.4] 20√N clients concurrently insert text in Array (encodeTime) | 1 ms | 0 ms | 13 ms | 1 ms | 13 ms | 7 ms | +| [B3.4] 20√N clients concurrently insert text in Array (docSize) | 26,596 bytes | 26,580 bytes | 100,791 bytes | 47,943 bytes | 96,423 bytes | 86,519 bytes | +| [B3.4] 20√N clients concurrently insert text in Array (parseTime) | 55 ms | 37 ms | 94 ms | 30 ms | 43 ms | 37 ms | +| [B4] Apply real-world editing dataset (time) | 2,616 ms | 17,556 ms | 2,271 ms | 768 ms | 7,109 ms | 2,775 ms | +| [B4] Apply real-world editing dataset (encodeTime) | 4 ms | 8 ms | 11 ms | 3 ms | 165 ms | 161 ms | +| [B4] Apply real-world editing dataset (docSize) | 226,981 bytes | 226,981 bytes | 230,556 bytes | 260,813 bytes | 129,116 bytes | 129,116 bytes | +| [B4] Apply real-world editing dataset (parseTime) | 27 ms | 22 ms | 6 ms | 4 ms | 1,185 ms | 1,123 ms | +| [B4x100] Apply real-world editing dataset 100 times (time) | 279,705 ms | skipped | 233,739 ms | 75,122 ms | skipped | skipped | +| [B4x100] Apply real-world editing dataset 100 times (encodeTime) | 417 ms | skipped | 743 ms | 205 ms | skipped | skipped | +| [B4x100] Apply real-world editing dataset 100 times (docSize) | 22,694,543 bytes | skipped | 21,016,454 bytes | 26,826,427 bytes | skipped | skipped | +| [B4x100] Apply real-world editing dataset 100 times (parseTime) | 1,270 ms | skipped | 66 ms | 200 ms | skipped | skipped | +| [B3.5] 20√N clients concurrently insert text (time) | 42 ms | 49 ms | 212 ms | 115 ms | 1,869 ms | 36 ms | +| [B3.5] 20√N clients concurrently insert text (updateSize) | 48,129 bytes | 48,135 bytes | 132,870 bytes | 68,978 bytes | 298,020 bytes | 298,020 bytes | +| [B3.5] 20√N clients concurrently insert text (encodeTime) | 1 ms | 0 ms | 13 ms | 1 ms | 12 ms | 16 ms | +| [B3.5] 20√N clients concurrently insert text (docSize) | 24,313 bytes | 24,325 bytes | 102,818 bytes | 48,053 bytes | 90,768 bytes | 90,781 bytes | +| [B3.5] 20√N clients concurrently insert text (parseTime) | 38 ms | 38 ms | 64 ms | 63 ms | 67 ms | 55 ms | +| [C1.1] Concurrently insert & delete 100K (time) | 27,138 ms | skipped | 2,335 ms | skipped | 50,692 ms | skipped | +| [C1.1] Concurrently insert & delete 100K (updateSize) | 4,908,122 bytes | skipped | 4,269,521 bytes | skipped | 10,326,487 bytes | skipped | +| [C1.1] Concurrently insert & delete 100K (encodeTime) | 129 ms | skipped | 210 ms | skipped | 837 ms | skipped | +| [C1.1] Concurrently insert & delete 100K (docSize) | 4,908,186 bytes | skipped | 6,748,052 bytes | skipped | 5,404,289 bytes | skipped | +| [C1.1] Concurrently insert & delete 100K (parseTime) | 1,653 ms | skipped | 78 ms | skipped | 7,308 ms | skipped | +| [C1.1] Concurrently insert & delete 100K (versionSize) | 222,681 bytes | skipped | 28 bytes | skipped | 64 bytes | skipped | +| [C1.2] Concurrently set Map 100K (time) | 31,598 ms | skipped | 488 ms | skipped | 472,547 ms | skipped | +| [C1.2] Concurrently set Map 100K (updateSize) | 980,804 bytes | skipped | 2,601,744 bytes | skipped | 5,541,744 bytes | skipped | +| [C1.2] Concurrently set Map 100K (encodeTime) | 60 ms | skipped | 203 ms | skipped | 2,652 ms | skipped | +| [C1.2] Concurrently set Map 100K (docSize) | 738,949 bytes | skipped | 1,539,209 bytes | skipped | 1,743,081 bytes | skipped | +| [C1.2] Concurrently set Map 100K (parseTime) | 4,069 ms | skipped | 631 ms | skipped | 338 ms | skipped | +| [C1.2] Concurrently set Map 100K (versionSize) | 416,246 bytes | skipped | 314,810 bytes | skipped | 1,949,999 bytes | skipped | + + +# FILE: pages/docs/performance/docsize.md + +--- +keywords: "loro, yjs, automerge, diamond-type, benchmark, document size, crdt" +description: "Comparing the document size of Loro and popular CRDTs" +--- + +# Document Size + +In this benchmark, we use the Automerge paper dataset. + +Source: https://github.com/automerge/automerge-perf/tree/master/edit-by-index + +The dataset consists of: + +- 182,315 single-character insertion operations +- 77,463 single-character deletion operations +- A total of 259,778 operations +- 104,852 characters in the final document + +The first line of settings in the table below indicates configurations without +`gc` and `compress`. + +| Settings | loro-snapshot | loro-update | diamond-type | yrs | automerge | +| -------------------- | ------------- | ----------- | ------------ | ------ | --------- | +| Default (no options) | 273561 | 251352 | 281042 | 226973 | 292742 | +| gc | x | x | 203564 | 159921 | x | +| compress | 132459 | 105724 | 150723 | 91777 | 129062 | +| gc & compress | x | x | 106242 | 71033 | x | + +> The `x` in the table above signifies that the corresponding setting is not supported. + +Loro also supports a shallow snapshot encoding format with gc capabilities by truncating the history. For details, see [the doc](/docs/tutorial/encoding). +If truncated from the latest version, the result will be: + +| Settings | loro-shallow-snapshot | +| -------- | --------------------- | +| Default | 63352 | +| compress | 54517 | + + +# FILE: pages/docs/performance/native.mdx + +# Native Benchmarks + +[This native benchmark](https://github.com/zxch3n/crdt-bench-native) is based on +the Rust implementation of each crate. + +- Conducted on a M2 Max CPU, dated 2024-10-18. +- The tasks with names starting with `automerge` use the automerge paper + dataset. +- In this benchmark, compression is disabled for both automerge and loro. +- Diamond-type doesn't support the list type yet. + +| Tasks | automerge | loro | diamond-type | yrs | +| :---------------------- | :-------- | :-------- | :----------- | :-------- | +| automerge - apply | 450.91 ms | 88.19 ms | 15.63 ms | 4238.8 ms | +| automerge - decode time | 506.30 ms | 0.189 ms | 2.19 ms | 3.82 ms | +| automerge - encode time | 17.65 ms | 0.416 ms | 1.15 ms | 0.759 ms | +| concurrent list inserts | 81.07 ms | 130.63 ms | 57.08 ms | 13.95 ms | +| list_random_insert_1k | 296.64 ms | 12.15 ms | 4.32 ms | 5.83 ms | + + +# FILE: pages/docs/llm.md + +# LLM Resources + +- [llms.txt](/llms.txt) +- [llms-full.txt](/llms-full.txt) + + +# FILE: pages/docs/advanced/jsonpath.mdx + +--- +keywords: "jsonpath, querying, loro, crdt, advanced" +description: "Learn how to query Loro documents using RFC 9535 JSONPath support." +--- + +# JSONPath Queries + +[JSONPath](https://www.rfc-editor.org/rfc/rfc9535.html) queries integrate +directly with Loro documents so you can traverse complex data without writing +ad hoc tree walkers. This guide outlines the query surface, highlights +supported syntax, and walks through examples that you can adapt to your own +application. + +> JSONPath functions such as `count()`, `length()`, `value()`, `match()`, and +> `search()` are parsed but not evaluated at runtime. Each currently returns +> `JsonPathError::EvaluationError::Unimplemented`. + +## Preparing Sample Data + +The snippets below use the WASM bindings, but the same structure works with the +Rust API. We create a bookstore dataset inside a `LoroDoc` so we can run queries +against it. + +> ⚠️ **Illustration only**: This approach stuffs plain JavaScript objects into a +> document for convenience. In production code prefer composing `LoroMap` and +> `LoroList` instances explicitly so you can control how data enters the CRDT and +> benefit from fine-grained updates. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const testData = { + books: [ + { title: "1984", author: "George Orwell", price: 10, available: true, isbn: "978-0451524935" }, + { title: "Animal Farm", author: "George Orwell", price: 8, available: true }, + { title: "Brave New World", author: "Aldous Huxley", price: 12, available: false }, + { title: "Fahrenheit 451", author: "Ray Bradbury", price: 9, available: true, isbn: "978-1451673318" }, + { title: "The Great Gatsby", author: "F. Scott Fitzgerald", price: null, available: true }, + { title: "To Kill a Mockingbird", author: "Harper Lee", price: 11, available: true }, + { title: "The Catcher in the Rye", author: "J.D. Salinger", price: 10, available: false }, + { title: "Lord of the Flies", author: "William Golding", price: 9, available: true }, + { title: "Pride and Prejudice", author: "Jane Austen", price: 7, available: true }, + { title: "The Hobbit", author: "J.R.R. Tolkien", price: 14, available: true } + ], + featured_author: "George Orwell", + min_price: 10, + featured_authors: ["George Orwell", "Jane Austen"], +}; + +const store = doc.getMap("store"); +Object.entries(testData).forEach(([key, value]) => store.set(key, value)); +doc.commit(); +``` + +## Executing JSONPath Queries + +Run a query with the `JSONPath` method on WASM or `jsonpath` on Rust. The API +returns a list of values for any matches. Container results currently surface as +handlers internally, but that interface is still evolving and treated as +opaque in this guide. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// setup omitted for brevity +const results = doc.JSONPath("$.store.books[*].title"); +console.log(results); +// => ["1984", "Animal Farm", ...] +``` + +```rust +use loro::prelude::*; + +let doc = LoroDoc::new(); +// setup omitted for brevity +let titles = doc.jsonpath("$.store.books[*].title")?; +for title in titles { + if let ValueOrHandler::Value(value) = title { + println!("{value}"); + } +} +``` + +### Supported Selectors and Filters + +RFC 9535 syntax works end-to-end, including: + +- **Selectors** – names, indices (positive/negative), slices, unions, wildcards, + and recursive descent (`..`). +- **Filters** – logical operators (`&&`, `||`, `!`), comparisons, membership via + `in`, substring checks with `contains`, property existence, and nested + subqueries through `$` (root) or `@` (current node). +- **Functions** – `count()`, `length()`, and `value()` are parsed, while `match()` + and `search()` are reserved for expansion. All functions currently return an + unimplemented evaluation error at runtime. + +## Cookbook Examples + +Once the document is populated you can combine selectors and filters to extract +precisely what you need. + +| Query | Description | Result | +| ----- | ----------- | ------ | +| `$.store.books[*].title` | All book titles. | `["1984", "Animal Farm", "Brave New World", "Fahrenheit 451", "The Great Gatsby", "To Kill a Mockingbird", "The Catcher in the Rye", "Lord of the Flies", "Pride and Prejudice", "The Hobbit"]` | +| `$.store.books[?(@.available)].title` | Titles of books in stock. | `["1984", "Animal Farm", "The Great Gatsby", "To Kill a Mockingbird", "Lord of the Flies", "Pride and Prejudice", "The Hobbit"]` | +| `$.store.books[?(@.author in $.store.featured_authors)].title` | Books by featured authors. | `["1984", "Animal Farm", "Pride and Prejudice"]` | +| `$.store.books[?(@.price > 12)].title` | Books priced above 12. | `["The Hobbit"]` | +| `$..price` | All price fields via recursive descent. | `[10, 8, 12, 9, null, 11, 10, 9, 7, 14]` | +| `$.store.books[0:3].title` | Slice syntax for the first three titles. | `["1984", "Animal Farm", "Brave New World"]` | +| `$.store.books[0,2,-1].title` | Union of specific indices. | `["1984", "Brave New World", "The Hobbit"]` | +| `count($.store.books[?(@.available)])` | Planned helper to count available books *(currently returns an unimplemented error)*. | `JsonPathError::EvaluationError::Unimplemented` | +| `length($.store.featured_authors)` | Planned array length helper *(currently returns an unimplemented error)*. | `JsonPathError::EvaluationError::Unimplemented` | +| `$.store.books[?(@.isbn && @.price >= $.store.min_price)].title` | Filter by field existence and comparison. | `["1984"]` | +| `$.store.books[?(!@.available)].title` | Negated availability filter. | `["Brave New World", "The Catcher in the Rye"]` | +| `$.store.books[?(@.title contains "The")].author` | Authors with "The" in the title. | `["F. Scott Fitzgerald", "J.R.R. Tolkien"]` | + +> JSONPath always returns values in document order. When a filter references +> another query (such as `$.store.featured_authors`), the subquery is evaluated +> for each candidate element. + + +# FILE: pages/docs/advanced/version_deep_dive.mdx + +--- +keywords: "crdt, version, frontier, DAG, git, history, time travel" +description: "The concept of several versions in Loro, DAG, Frontiers, and Version Vectors" +--- + +# Loro's Versioning Deep Dive: DAG, Frontiers, and Version Vectors + +In centralized environments, we can use linear version numbers to represent a +version, such as incrementing a number each time or using timestamps. However, +CRDTs can be used in decentralized environments, and their version +representation is different. + +In Loro, you can express a document's version through a +[Version Vector](https://en.wikipedia.org/wiki/Version_vector) or Frontiers. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +// ---cut--- +doc.version(); // State Version vector +doc.oplogVersion(); // OpLog Version vector +doc.frontiers(); // State Frontiers +doc.oplogFrontiers(); // OpLog Frontiers +``` + +In most cases, you might only need the Version Vector, which can be used for +data synchronization and version comparison. + +Frontiers are more space-efficient than Version Vectors but come with more +limitations. + +The following sections detail these two different ways of version description. + +## Background Knowledge + +Loro belongs to Operation-Based CRDTs. Two Loro documents are consistent if they +have the same set of activated Operations. Version is a compact way to describe +which ops belong to this set. + +Each operation has a unique ID, known as OpId. For example, each user's deletion +or insertion of characters will have a corresponding unique OpId (don't worry +about the overhead here; we've compressed them in memory and Encoding). OpId +consists of PeerId and Counter. + +> 💡 Note: To simplify, the specific type descriptions below may not be accurate +> (e.g., the type of peer), but this does not affect the understanding of the +> content. + +```ts +interface OpId { + // Simplified here + // PeerId is actually close to uuid type, Loro uses u64 + peerId: number; + counter: number; +} +``` + +PeerId represents the unique identifier of the device operating the document, +and Counter is a number that starts from 0 and increments on that device. + +## Version Vector + +A Version Vector is a vector that describes how many Operations starting from 0 +need to be included for each Peer. + +```ts +// In TypeScript +type VersionVector = { [peerId in number]: number }; +``` + +For example, for the following version vector A, + +```tsx +const A = { + 0: 2, + 1: 3, +}; +``` + +It includes: + +- All operations where PeerId == 0 && Counter < 2 +- Or PeerId == 1 && Counter < 3 + +This expression allows us to easily compare two versions and get the operations +they are missing from each other. For example, in the following case, suppose we +know that the document on the server has reached version B, and the device has +version A. To let the user's device get the Operations it is missing, + +```tsx +const A = { + 0: 2, + 1: 3, +}; + +const B = { + 0: 5, + 1: 3, + 2: 9, +}; +``` + +We can easily calculate the Operations the device is missing: + +- All Ops where PeerId == 0 && 2 ≤ Counter < 5 +- Or PeerId == 2 && Counter < 9 + +Version Vectors help us identify a version and allow us to quickly calculate the +differences between two versions (quickly calculating the set of Ops missing in +two documents to be synchronized). For example, currently in Loro, you can use +the following method to synchronize two documents, where they only import the +Ops missing from each other: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +const a = new LoroDoc(); +const b = new LoroDoc(); +// ---cut--- +// doc.version() returns the version vector of the doc +a.import(b.export({ mode: "update", from: a.version() })); +b.import(a.export({ mode: "update", from: b.version() })); +``` + +## Directed Acyclic Graph (DAG) History + +In Loro, we record an operation history similar to Git's Directed Acyclic Graph +(DAG). We represent DAG by recording Dependencies links on each Op. Like +document states, when the Op set is the same, the DAG is consistent across all +endpoints. + +![DAG Illustration](./images/version-3.png) + +When a new Op is added, its Dependencies link points to the current Operations +that are not being pointed to. + +> 💡 Note: The order of Ops is a partial order relationship. An Op may be +> before, after, or parallel to another Op. The structure of the DAG allows us +> to compactly express this partial order. + +DAG history information helps us design some special CRDT algorithms and +facilitates time-travel implementation. Moreover, when automatic merges don't +meet expectations, DAG allows users a better chance to manually resolve +potential conflicts. + +> 💡 We use Counter@PeerId for a more compact description of OpId. + +Below is an example of how a DAG is formed: + +![DAG Formation Example](./images/version-0.png) + +## Frontiers + +Version Vectors increase in size with the number of Peers. Most CRDTs Lib, to +prevent PeerId conflicts, use a random new PeerId each time a document is +opened, which increases the overhead of Version Vectors. + +> 📌 If PeerId repeats, there's a high probability of duplicate OpId, +> potentially leading to inconsistent CRDT documents. Automerge has solved this +> problem, ensuring document consistency even with duplicate OpId, but at the +> cost of higher sync complexity. + +Based on DAG information, we have a more compact way to describe versions: given +one or multiple OpIds, include all with a Causal Order less than or equal to +them. We call this method Frontiers. + +```ts twoslash +import { OpId } from "loro-crdt"; +// ---cut--- +type Frontiers = OpId[]; +``` + +> 💡 We use Counter@PeerId for a more compact description of OpId. + +![Frontiers Illustration](./images/version-1.png) + +`Frontiers = [3@0]` in the above DAG includes 6 operations: + +- PeerId == 0 && Counter < 4 +- PeerId == 1 && Counter < 2 + +It is equivalent to the Version Vector: + +```tsx +{ + 0: 4, + 1: 2, +} +``` + +Frontiers can also specify multiple parallel operations: + +![Parallel Operations Example](./images/version-2.png) + +In this example, `Frontiers = [1@1, 1@2]`, corresponding Version Vector is: + +```tsx +{ + 0: 1, + 1: 2, + 2: 2, +} +``` + +One great property of Frontiers as a version is that `Frontiers = [A]` actually +represents the document version right after operation A was executed. Thus, +combined with Loro's time machine capability, you can easily return to any +document version corresponding to a specific operation. Here is a complete +example: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId(0n); +const text = doc.getText("text"); +text.insert(0, "H"); +doc.commit(); +const v = doc.frontiers(); +text.insert(1, "i"); +expect(doc.toJSON()).toStrictEqual({ + text: "Hi", +}); + +doc.checkout([{ peer: "0", counter: 0 }]); +expect(doc.toJSON()).toStrictEqual({ + text: "H", +}); +``` + +You can convert between Version Vectors and Frontiers in Loro if the specified +version is included in the document's history: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +// ---cut--- +const f = doc.frontiers(); +const vv = doc.frontiersToVV(f); +// vv equals to doc.version() +const frontiers = doc.vvToFrontiers(vv); +``` + + +# FILE: pages/docs/advanced/undo.mdx + +--- +keywords: "crdt, undo, redo, undo manager, cursor, selection, collaborate" +description: "how to use loro undo manager and show all APIs of loro undo manager." +--- + +# Undo + +We provide an `UndoManager` that helps you manage the undo/redo stack and can be +used during concurrent editing. It also supports cursor position transformation. +In the case of concurrent editing, you will want the undo/redo to be **local**, +meaning it should only undo/redo local operations without affecting other +collaborators' edits. + +### Why Local Undo/Redo? + +If undo/redo is global, it often does not meet expectations. Consider the +following scenario: + +- **User A and User B are collaborating**: They are likely editing different + parts of the document. +- **User A performs an undo**: If this undoes User B's operations, User A might + see no changes and think the undo failed, as User B might be editing content + outside of User A's view. +- **User B's perspective**: User B would find their recent edits deleted without + knowing how it happened. + +## Usage + +To create an `UndoManager`, you can specify: + +- Which local operations are not recorded. +- The merge range for undo operations. +- The stack depth. + +```ts twoslash +import { UndoManager, LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +// ---cut--- +const undoManager = new UndoManager(doc, { + maxUndoSteps: 100, // default 100 + mergeInterval: 1000, // default 1000ms + excludeOriginPrefixes: ["sys:"], // default [] + onPush: (isUndo, range, event) => { + return { value: null, cursors: [] }; + }, + onPop: (isUndo, value, counterRange) => { + return; + }, +}); +``` + +## Limitations + +It can only track a single peer. When the peer ID of the document changes, it will +clear the undo stack and the redo stack and track the new peer ID. + +## Restoring Selections + +When utilizing undo/redo functionalities, it is important for our application to +restore the selection to its position prior to the operation that is being +undone or redone. This task is particularly challenging in a collaborative +setting where remote operations can alter the cursor's position (for instance, +if the cursor needs to revert to the 5th position, but remote operations have +added new characters before this position). + +
+Challenges + +- **Local undo/redo**: Local undo and redo delete and recreate elements. If we + use CRDT-based stable positions, they might lock on to the deleted elements, + while the user's expectation is for the cursor to return to the newly created + elements after redo. +- **Example**: + ``` + A fox jumped + ``` + - Select "fox" and delete it. + - Undo. + - After undo, the three characters "fox" should be recreated, and the cursor + should select these three characters. However, if we use stable positions, + it would lock onto the initially deleted characters, and after undo, the + absolute positions obtained would be `start = 2` and `end = 2`. + +
+ +### Solution + +We support storing [`cursors`](/docs/tutorial/cursor) for each undo/redo action +within the `UndoManager`. The `UndoManager` will adjust the stored cursors to +reflect changes from remote edits or other undo/redo operations, ensuring they +match the current state of the document. + +They need be handled by the `onPush` and `onPop` callbacks. + +On `onPush`, you can return a list of `Cursor`s that you want to store. On +`onPop`, you can retrieve the stored cursors and use them to restore the +selection. + +Typically, you may need to store selections in an undo/redo item in the +following cases: + +- When a new local change is applied, we need to record a new undo item. + Therefore, we must store the selection **before** the operation to be undone. + - **Purpose**: Storing the selection **before** is crucial because we may lose + the selection after applying the operation. If the user selects text and + deletes it, after undo, the `onPop` can retrieve the state of the selected + deleted content. +- **First undo operation**: Store the current document's selection for the + corresponding redo item. + - **Purpose**: After redo, it can return to the initial selection state. + +Internally, we also automatically handle the storage and reset of cursors in the +undo/redo loop state. + +```ts twoslash +import { LoroDoc, UndoManager, Cursor } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +let cursors: Cursor[] = []; +let poppedCursors: Cursor[] = []; +const undo = new UndoManager(doc, { + mergeInterval: 0, + onPop: (isUndo, value, counterRange) => { + poppedCursors = value.cursors; + }, + onPush: () => { + return { value: null, cursors: cursors }; + }, +}); + +doc.getText("text").insert(0, "hello world"); +doc.commit(); +cursors = [ + doc.getText("text").getCursor(0)!, + doc.getText("text").getCursor(5)!, +]; +// delete "hello ", the cursors should be transformed +doc.getText("text").delete(0, 6); +doc.commit(); +expect(poppedCursors.length).toBe(0); +undo.undo(); +expect(poppedCursors.length).toBe(2); +expect(doc.toJSON()).toStrictEqual({ + text: "hello world", +}); +// expect the cursors to be transformed back +expect(doc.getCursorPos(poppedCursors[0]).offset).toBe(0); +expect(doc.getCursorPos(poppedCursors[1]).offset).toBe(5); +``` + +
+Implementation + +[The implementation of undo/redo](https://github.com/loro-dev/loro/pull/361) +follows a model similar to ProseMirror/Quill, which are based on OT (Operational +Transformation) algorithms (so we also implement basic OT primitives +internally). + +When implementing undo/redo operations, we need to ensure the following +properties: + +- Do not undo remote inserts. +- Redo after undo should return to the original state. +- If there is no concurrent editing, undo should return to the previous + version's state. + +Therefore, we have also added some relevant checks in our internal fuzzing tests +to ensure correctness. + +
+ +## Demonstration + +import { ReactPlayer } from "../../../components/video"; +import Caption from "../../../components/caption"; + +{" "} + + + +ProseMirror with Loro binding + +## Understanding the Undo/Redo Stack + +The UndoManager maintains two stacks: + +1. **Undo Stack**: Contains operations that can be undone +2. **Redo Stack**: Contains operations that were undone and can be redone + +### How the Callbacks Work + +The `onPush` and `onPop` callbacks are triggered when these stacks change: + +- **onPush(isUndo, range, event)**: Called when a new item is pushed to either stack + - `isUndo: boolean`: `true` for the undo stack, `false` for the redo stack + - `range: (number, number)`: The operations' counter range that associated with the undo/redo action + - Returns: An object that can include `value` (any data you want to store) and `cursors` (cursor positions) + +- **onPop(isUndo, value)**: Called when an item is popped from either stack + - `isUndo`: `true` for the undo stack, `false` for the redo stack + - `value`: The value you returned from `onPush` when this item was created + +### Understanding Action Merging + +The `mergeInterval` option in the UndoManager controls how closely spaced operations are grouped: + +```ts twoslash no_run +import { UndoManager, LoroDoc } from "loro-crdt"; +// ---cut--- +declare const doc: LoroDoc; +const undoManager = new UndoManager(doc, { + mergeInterval: 1000, // 1000ms = 1 second (default) +}); +``` + +**How mergeInterval works:** + +- Operations occurring within the specified time interval (in milliseconds) will be merged into a single undo action +- Even though these operations are merged, `onPush` events will still be triggered for each individual operation +- When undoing, all merged operations will be undone as a single unit +- A lower value results in more granular undo steps; a higher value creates fewer, more comprehensive undo steps +- Set to `0` to disable merging entirely (every operation becomes a separate undo step) + +### Stack Operations Flow + +1. When a local transaction is committed, a new undo item is pushed to the undo stack (triggers `onPush` with `isUndo=true`) +2. When `.undo()` is called: + - An item is popped from the undo stack (triggers `onPop` with `isUndo=true`) + - A corresponding item is pushed to the redo stack (triggers `onPush` with `isUndo=false`) +3. When `.redo()` is called: + - An item is popped from the redo stack (triggers `onPop` with `isUndo=false`) + - A corresponding item is pushed to the undo stack (triggers `onPush` with `isUndo=true`) + +### Manual Grouping with `groupStart` and `groupEnd` + +Sometimes you need tighter control over how undo units are formed than `mergeInterval` alone can provide. The `UndoManager.groupStart()` and `UndoManager.groupEnd()` methods let you explicitly wrap a sequence of commits so they become a single undo step. + +- Call `groupStart()` before you begin a batch of related edits. Any commits that happen before the matching `groupEnd()` will undo together. +- Calling `groupStart()` while another group is active throws an error and leaves the original group in place, which helps catch accidental nesting. +- Remote updates can influence grouping: a conflicting import may force the current group to end early, while non-conflicting imports are merged without breaking the group. Always finish with `groupEnd()`—if the group already auto-closed, the call becomes a no-op. + +```ts twoslash no_run +import { LoroDoc, UndoManager } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +const undoManager = new UndoManager(doc, {}); + +undoManager.groupStart(); + +text.update("hello", undefined); +doc.commit(); +text.update("hello world", undefined); +doc.commit(); + +undoManager.groupEnd(); + +undoManager.undo(); // Reverts both commits in one step +``` + +### Example: Text Editing with Undo/Redo + +Consider a simple text editor that uses Loro for collaboration. Let's walk through what happens during typical editing operations: + +```ts twoslash no_run +import { LoroDoc, UndoManager } from "loro-crdt"; +// ---cut--- +// Create document and undo manager +const doc = new LoroDoc(); +const textField = doc.getText("textField"); + +// Set up undo manager with callbacks to track changes +const undoManager = new UndoManager(doc, { + // Store cursor positions and any other state you need + onPush: (isUndo, range) => { + if (isUndo) { + console.log("Action recorded for undo"); + } else { + console.log("Action recorded for redo"); + } + // Return whatever data you want associated with this action + return { + value: { affectedRange: range }, + cursors: [/* your cursor positions */] + }; + }, + + onPop: (isUndo, storedData) => { + // Access the data you stored during onPush + const { value, cursors } = storedData; + + if (isUndo) { + console.log("Retrieving data for undo"); + } else { + console.log("Retrieving data for redo"); + } + + // Use the stored cursors to restore selection + // applyStoredCursors(cursors); + }, + + mergeInterval: 0, +}); + +// User types "Hello" +textField.insert(0, "Hello"); +doc.commit(); +// → onPush triggered (isUndo=true) - Adds to undo stack + +// User types " World" +textField.insert(5, " World"); +doc.commit(); +// → onPush triggered (isUndo=true) - Adds to undo stack + +// User clicks Undo button +undoManager.undo(); +// → onPop triggered (isUndo=true) - Removes " World" from document +// → onPush triggered (isUndo=false) - Adds to redo stack + +// Document now contains only "Hello" + +// User clicks Redo button +undoManager.redo(); +// → onPop triggered (isUndo=false) - Retrieves " World" operation +// → onPush triggered (isUndo=true) - Adds back to undo stack + +// Document now contains "Hello World" again +``` + +When the user clicks Undo, two things happen: +1. The last action is popped from the undo stack (removing " World") +2. That action is pushed to the redo stack so it can be redone later + +This approach ensures that local changes can be undone without affecting other users' edits, making it ideal for collaborative editing. + +### Cursor Efficiency + +The built-in cursor solution is optimized for performance and handles collaborative scenarios efficiently, including situations where peers may +change the document concurrently during undo/redo operations. For complex editors like rich text editors, the cursor implementation provides the +best balance of performance and correctness. + + +# FILE: pages/docs/advanced/inspector.mdx + +# Loro Inspector + +[Loro Inspector](https://inspector.loro.dev/) is an open-source web tool that helps developers debug and visualize Loro documents. It provides a user-friendly interface to inspect the state and history of Loro documents. + +import { ReactPlayer } from "../../../components/video"; + + + + +# FILE: pages/docs/advanced/import_batch.mdx + +# Batch Import + +## Performance Differences and Their Causes + +When importing multiple updates into a document, using `doc.importBatch(updates)` is significantly faster than importing updates individually. This performance difference stems from how data merging is handled in each approach. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").update("Hello"); +const update1 = doc.export({ mode: "update" }); +const version = doc.version(); +doc.getText("text").update("Hello World"); +const update2 = doc.export({ mode: "update", from: version }); + +const newDoc1 = new LoroDoc(); +newDoc1.importBatch([update1, update2]); // faster + +const newDoc2 = new LoroDoc(); +for (const update of [update1, update2]) { // slower + newDoc2.import(update); +} +``` + +### Key Advantages of Import Batch + +#### 1. Single Diff Calculation + +The most significant advantage is that import batch performs only one diff calculation. In contrast, each individual import follows these steps: + +- Merge remote updates into local history +- Calculate document state changes from the current version to the merged version +- Apply the diff to the current document state + +This diff calculation has fixed overhead costs that accumulate with each import. But `doc.importBatch(...)` only performs one diff calculation, which is faster than multiple individual diff calculations. + +#### 2. Reduced Communication Overhead + +Import batch also results in more concise events. Each individual import generates a new event, but `doc.importBatch(...)` generates only a single event that contains all the changes. + + +# FILE: pages/docs/advanced/timestamp.mdx + +# Storing Timestamps + +You can enable timestamp recording through `setRecordTimestamp`, allowing Unix timestamps to be logged in each `Change`. Consequently, these timestamps will be preserved in exported `Updates` or `Snapshots`. + +Enabling this feature affects the merge behavior of `Changes`, as `Changes` with too long a time gap cannot share the same timestamp. In such cases, you can adjust the mergeable duration range using `setChangeMergeInterval`, with a default setting of 1,000,000, equating to a 1000s threshold for merging `Changes`. + +> Each insertion or deletion by the user generates an `op`, and multiple consecutive `op`s can merge into a larger `Change`. `Change`s log a `Timestamp`, but each `Change` can only associate with one `Timestamp`. Hence, if the time gap is too wide, merging `Changes` becomes impractical. However, treating each Change as new based on slight timestamp differences (e.g., key presses milliseconds apart) introduces significant additional overhead for each Change. Therefore, users can customize the `change merge interval` according to their needs. + +Note that these settings do not persist in exported `Updates` or `Snapshots`. Thus, if custom configurations are required, they must be reapplied upon each initialization of `LoroDoc`. Without timestamp recording enabled, the `Timestamp` defaults to the current maximum known `Timestamp`. + + +# FILE: pages/docs/advanced/cid.mdx + +# Container ID + +A Container ID is a unique identifier that comes in two forms: + +- Root Container: Composed of a type and root name +- Normal Container: Created through user operations, composed of an ID and type + +**Rust `ContainerID`** + +```rust +pub enum ContainerID { + Root { + name: InternalString, + container_type: ContainerType, + }, + Normal { + peer: PeerID, + counter: Counter, + container_type: ContainerType, + }, +} +``` + +**TypeScript `ContainerID`** + +```ts twoslash +import type { ContainerType, PeerID } from "loro-crdt"; +// ---cut--- +export type ContainerID = + | `cid:root-${string}:${ContainerType}` + | `cid:${number}@${PeerID}:${ContainerType}`; +``` + +1. **Root Containers** + - Created implicitly when accessing a root container for the first time + (e.g., calling `doc.getText("text")`). No operation is generated in the + history. + - Uniquely identified by a string name and container type + +2. **Normal Containers** + - Created explicitly through operations like `insertContainer` or + `createNode` + - Generated automatically when applying operations that create child + containers + - Contains the Operation ID of its creation within its Container ID + +3. **Mergeable Containers** + - Created through `LoroMap.ensureMergeableText`, + `ensureMergeableMap`, `ensureMergeableList`, and the other + `ensureMergeable*` methods + - Use a special form of Root Container ID. The stable name inside that ID is + derived from the logical position `(parent Map, key, container type)`, not + from the operation that first created the child + - Useful when multiple peers may lazily initialize the same child container + under the same Map key + +## Container States and IDs + +The ContainerID is not a random UUID but is deterministically generated based on the container's context. To understand how ContainerIDs work, it's important to first understand container states. + +> For a comprehensive guide on containers, including attached vs detached states, see [Container Concepts](/docs/concepts/container). + +Key points about ContainerID generation: +- **Root containers**: Derive their ID from their name (e.g., "text" in `doc.getText("text")`) +- **Normal containers**: Derive their ID from the operation (OpID) that created them +- **Mergeable containers**: Use a special Root Container ID whose stable name is derived from the parent Map, key, and container type +- **Detached containers**: Have a default placeholder ID until they're inserted into a document + +## Container Overwrites + +When initializing child containers in parallel, overwrites can occur instead of +automatic merging. For example: + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const a: string = "hello"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +// Parallel initialization of child containers +const docB = doc.fork(); +const textA = doc.getMap("map").setContainer("text", new LoroText()); +textA.insert(0, "A"); +const textB = docB.getMap("map").setContainer("text", new LoroText()); +textB.insert(0, "B"); + +doc.import(docB.export({ mode: "update" })); +// Result: Either { "meta": { "text": "A" } } or { "meta": { "text": "B" } } +``` + +This behavior poses a significant risk of data loss if the editing history is +not preserved. Even when the complete history is available and allows for data +recovery, the recovery process can be complex. + + + +When a container holds substantial data or serves as the primary storage for +document content, overwriting it can lead to the unintended hiding/loss of +critical information. For this reason, it is essential to implement careful and +systematic container initialization practices to prevent such issues. + +### Mergeable Containers + +The overwrite behavior occurs because parallel creation of regular child +containers results in different container IDs, preventing automatic merging of +their contents. If peers should address the same child by logical Map key, use a +mergeable child container: + +```ts no_run +const text = doc.getMap("map").ensureMergeableText("text"); +``` + +This uses a special Root Container ID for the child. Its stable name is derived +from the parent Map, key, and type, rather than from the operation that first +created it. + +### Best Practices + +1. Use `ensureMergeable*` when a Map child container should be shared by logical + key even if peers initialize it concurrently. + +2. When the structure is fixed and known ahead of time: + - If possible, initialize all child containers during the map container's + initialization + +3. Use root containers when the child naturally has a stable top-level name. + +4. Use regular child containers when each creation should have a distinct + operation-derived identity. + + +# FILE: pages/docs/api/js.mdx + +import styles from "./api-reference.module.css"; +import Indent from "./indent"; +import Method from "./method"; + +
+ +# API Reference + +> _Last updated: 2026-06-16 loro-crdt@1.13.3_ + +## Overview + +Loro is a powerful Conflict-free Replicated Data Type (CRDT) library that enables real-time collaboration. If CRDTs are new to you, start with [What are CRDTs](/docs/concepts/crdt) for a gentle intro. This API reference provides comprehensive documentation for all classes, methods, and types available in the JavaScript/TypeScript binding. + +Note: Under the hood, Loro combines a Fugue-based CRDT core with Eg-walker-inspired techniques that use simple index operations and replay only the divergent history when merging. This yields fast local edits, efficient merges, and low overhead without permanent tombstones. See the primer [Event Graph Walker (Eg-walker)](/docs/advanced/event_graph_walker) and performance notes in the v1.0 blog (import/export speedups, shallow snapshots): https://loro.dev/blog/v1.0 + +## Pitfalls & Best Practices + +**Peer ID Management** + +- **Never share PeerIDs** between concurrent sessions (tabs/devices) - causes document divergence +- Use random PeerIDs (default) unless you have strict single-ownership locking +- Don't assign fixed PeerIDs to users or devices + +**UTF-16 Text Encoding** + +- All text operations use UTF-16 indices by default in JS API +- Slicing in the middle of multi-unit codepoints corrupts them +- Use `insertUtf8()`/`deleteUtf8()` for UTF-8 systems + +**Container Creation** + +- Concurrent child container creation inside the same LoroMap at same key causes overwrites +- Initialize all child containers for a LoroMap upfront when possible +- Operations on the root containers will not override each other + +- Events emit synchronously during commit/import/checkout in JS API (v1.8+). Stay on `<=1.7.x`? Await a microtask before reading batched events. +- Import/export/checkout trigger automatic commits +- Loro transactions are NOT ACID - no rollback/isolation + +**Version Control** + +- After `checkout()`, document enters read-only "detached" mode, unless `setDetachedEditing(true)` is called +- [Frontiers](/docs/concepts/frontiers) can't determine complete operation sets without history + +**Data Structure Choice** + +- Use strings in Map for URLs/IDs (LWW), LoroText for collaborative editing + +## Common Tasks & Examples + +**Getting Started** + +- **Create a document**: [`new LoroDoc()`](#LoroDoc.constructor) - Initialize a new collaborative document +- **Add containers**: [`getText`](#LoroDoc.getText), [`getList`](#LoroDoc.getList), [`getMap`](#LoroDoc.getMap), [`getTree`](#LoroDoc.getTree) +- **Listen to changes**: [`subscribe`](#LoroDoc.subscribe) - React to document modifications +- **Export/Import state**: [`export`](#LoroDoc.export) and [`import`](#LoroDoc.import) - Save and load documents + +**Real-time Collaboration** + +- **Sync between peers**: [`export`](#LoroDoc.export) with `mode: "update"` + [`import`](#LoroDoc.import)/[`importBatch`](#LoroDoc.importBatch) - Exchange incremental updates +- **Stream updates**: [`subscribeLocalUpdates`](#LoroDoc.subscribeLocalUpdates) - Send changes over WebSocket/WebRTC +- **Set unique peer ID**: [`setPeerId`](#LoroDoc.setPeerId) - Ensure each client has a unique identifier +- **Handle conflicts**: Automatic - All Loro data types are CRDTs that merge concurrent edits + +**Rich Text Editing** + +- **Create rich text**: [`getText`](#LoroDoc.getText) - Initialize a collaborative text container +- **Edit text**: [`insert`](#LoroText.insert), [`delete`](#LoroText.delete), [`applyDelta`](#LoroText.applyDelta) +- **Apply formatting**: [`mark`](#LoroText.mark) - Add bold, italic, links, custom styles +- **Copy styled snippets**: [`sliceDelta`](#LoroText.sliceDelta) - Get a Delta for a range (UTF-16; use `sliceDeltaUtf8` for byte offsets) +- **Track cursor positions**: [`getCursor`](#LoroText.getCursor) + [`getCursorPos`](#LoroDoc.getCursorPos) - Stable positions across edits +- **Configure styles**: [`configTextStyle`](#LoroDoc.configTextStyle) - Define expand behavior for marks + +**Data Structures** + +- **Ordered lists**: [`getList`](#LoroDoc.getList) - Arrays with [`push`](#LoroList.push), [`insert`](#LoroList.insert), [`delete`](#LoroList.delete) +- **Key-value maps**: [`getMap`](#LoroDoc.getMap) - Objects with [`set`](#LoroMap.set), [`get`](#LoroMap.get), [`delete`](#LoroMap.delete) +- **Hierarchical trees**: [`getTree`](#LoroDoc.getTree) - File systems, nested comments with [`createNode`](#LoroTree.createNode), [`move`](#LoroTree.move) +- **Reorderable lists**: [`getMovableList`](#LoroDoc.getMovableList) - Drag-and-drop with [`move`](#LoroMovableList.move), [`set`](#LoroMovableList.set) +- **Counters**: [`getCounter`](#LoroDoc.getCounter) - Distributed counters with [`increment`](#LoroCounter.increment) + +**Ephemeral State & Presence** + +- **User presence**: [`EphemeralStore`](#ephemeralstore) - Share cursor positions, selections, user status (not persisted) +- **Cursor syncing**: Use [`EphemeralStore.set`](#EphemeralStore.set) with cursor data from [`getCursor`](#LoroText.getCursor) +- **Live indicators**: Track who's online, typing indicators, mouse positions +- **Important**: EphemeralStore is a separate CRDT without history - perfect for temporary state that shouldn't persist + +**Version Control & History** + +- **Undo/redo**: [`UndoManager`](#undomanager) - Local undo of user's own edits +- **Time travel**: [`checkout`](#LoroDoc.checkout) to any [`frontiers`](#LoroDoc.frontiers) - Debug or review history +- **Version tracking**: [`version`](#LoroDoc.version), [`frontiers`](#LoroDoc.frontiers), [`versionVector`](#LoroDoc.versionVector) +- **Fork documents**: [`fork`](#LoroDoc.fork) or [`forkAt`](#LoroDoc.forkAt) - Create branches for experimentation +- **Merge branches**: [`import`](#LoroDoc.import) - Combine changes from forked documents + +**Performance & Storage** + +- **Incremental updates**: [`export`](#LoroDoc.export) from specific [`version`](#LoroDoc.version) - Send only changes +- **Compact history**: [`export`](#LoroDoc.export) with `mode: "snapshot"` - Full state with compressed history +- **Shallow snapshots**: [`export`](#LoroDoc.export) with `mode: "shallow-snapshot"` - State without partial history (see [Shallow Snapshots](/docs/concepts/shallow_snapshots)) + +## Basic Usage + +```typescript twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +// Subscribe to changes +const unsubscribe = doc.subscribe((event) => { + console.log("Document changed:", event); +}); + +// Export updates for synchronization +const updates = doc.export({ mode: "update" }); +``` + +## LoroDoc + +The `LoroDoc` class manages containers, sync, versions, and events. + +**Constructor** + +```typescript no_run +new LoroDoc(); +``` + +Creates a new Loro document with a randomly generated peer ID. + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +``` + +**Static Methods** + + +```typescript no_run +static fromSnapshot(snapshot: Uint8Array): LoroDoc +``` + + +Creates a new LoroDoc from a snapshot. This is useful for loading a document from a previously exported snapshot. + +**Parameters:** + +- `snapshot` - Binary snapshot data + +**Returns:** A new LoroDoc instance + +**Example:** + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; + +// Assume we have a snapshot from a previous export +const prevDoc = new LoroDoc(); +prevDoc.getText("text").insert(0, "Hello"); +const snapshot: Uint8Array = prevDoc.export({ mode: "snapshot" }); + +const doc = LoroDoc.fromSnapshot(snapshot); +``` + + + +**Properties** + + +```typescript no_run +readonly peerId: bigint +``` + + +Gets the peer ID of the current writer as a bigint. + +**See Also:** [PeerID Management](/docs/concepts/peerid_management) + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const id = doc.peerId; +``` + + + + +```typescript no_run +readonly peerIdStr: `${number}` +``` + + +Gets the peer ID as a decimal string. + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const idStr = doc.peerIdStr; +``` + + + +### Configuration Methods + + +```typescript no_run +setPeerId(peer: number | bigint | `${number}`): void +``` + + +Sets the peer ID for this document. It must be a number, a BigInt, or a decimal string that fits into an unsigned 64-bit integer. +See [PeerID Management](/docs/concepts/peerid_management) for why uniqueness matters in distributed systems. + +**Parameters:** + +- `peer` - Peer ID as number, bigint, or decimal string + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setPeerId("42"); +``` + +**⚠️ Critical Pitfall:** Never let two parallel peers (e.g., multiple tabs/devices) share the same PeerID — it creates duplicate op IDs and causes document divergence. Common mistakes: + +- Don't assign a fixed PeerId to a user (users have multiple devices) +- Don't assign a fixed PeerId to a device (multiple tabs can open the same document) +- If you must reuse PeerIDs, enforce single ownership with strict locking mechanisms +- Best practice: Use random IDs (default behavior) unless you have a strong reason not to + +See [PeerID reuse](/docs/tutorial/tips) for safe reuse patterns. + + + + +```typescript no_run +setRecordTimestamp(auto_record: boolean): void +``` + + +Configures whether to automatically record timestamps for changes. Timestamps use Unix time (seconds since epoch). Learn more about storing timestamps and typical use cases in [Storing Timestamps](/docs/advanced/timestamp). + +**Parameters:** + +- `auto_record` - Whether to automatically record timestamps + +**⚠️ Important:** This setting doesn't persist in exported Updates or Snapshots. You must reapply this configuration each time you initialize a document. + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setRecordTimestamp(true); +``` + + + + +```typescript no_run +setChangeMergeInterval(interval: number): void +``` + + +Sets the interval in milliseconds for merging continuous local changes into a single change record. In Loro, multiple low-level operations are grouped into higher-level Changes for readability and syncing. See [Operations and Changes](/docs/concepts/operations_changes). + +**Parameters:** + +- `interval` - Merge interval in milliseconds + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setChangeMergeInterval(1000); // Merge changes within 1 second +``` + + + + +```typescript no_run +configTextStyle(styles: StyleConfig): void +``` + + +Configures the behavior of text styles (marks) in rich text containers. Marks can expand when edits happen at their edges (before/after/both/none). For a primer on rich text and marks in Loro, see [Text](/docs/tutorial/text). + +**Parameters:** + +- `styles` - Configuration object mapping style names to their config + +**StyleConfig Type:** + +```typescript no_run +type StyleConfig = Record< + string, + { + expand?: "after" | "before" | "both" | "none"; + } +>; +``` + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.configTextStyle({ + bold: { expand: "after" }, + italic: { expand: "none" }, + link: { expand: "none" }, +}); +``` + + + + +```typescript no_run +configDefaultTextStyle(style?: { expand: "after" | "before" | "both" | "none" }): void +``` + + +Configures the default text style for the document when using LoroText. If undefined is provided, the default style is reset. + +**Parameters:** + +- `style` - Default style configuration (optional) + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.configDefaultTextStyle({ expand: "after" }); +``` + + + +### Container Access Methods + +**📝 Note:** Creating root containers (e.g., `doc.getText("...")`) does not record operations; nested container creation (e.g., `map.setContainer(...)`) does. + +**⚠️ Pitfall:** Avoid concurrent creation of regular child containers with the same key in LoroMaps. Instead of: + +```ts no_run +// Dangerous - concurrent peers can create different child container IDs +doc.getMap("user").getOrCreateContainer(userId, new LoroMap()); +``` + +Use a mergeable child container when the child should be identified by its logical position: + +```ts no_run +doc.getMap("user").ensureMergeableMap(userId); +``` + +Alternatively, use a unique root container if that better fits your model: + +```ts no_run +doc.getMap("user." + userId); +``` + + +```typescript no_run +getText(name: string): LoroText +``` + + + +Gets or creates a text container with the given name. New to LoroText and marks? See [Text](/docs/tutorial/text). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroText` instance + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("content"); +text.insert(0, "Hello"); +``` + + + + +```typescript no_run +getList(name: string): LoroList +``` + + + +Gets or creates a list container with the given name. Unsure whether to use List or MovableList? See [List and Movable List](/docs/tutorial/list) and the type selection guide [Choosing CRDT Types](/docs/concepts/choose_crdt_type). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroList` instance + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const list = doc.getList("items"); +list.push("Item 1"); +``` + + + + +```typescript no_run +getMap(name: string): LoroMap +``` + + +Gets or creates a map container with the given name. See [Map](/docs/tutorial/map) for basics and patterns. + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroMap` instance + +**Example:** + +```ts twoslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const map = doc.getMap("settings"); +map.set("theme", "dark"); +``` + + + + +```typescript no_run +getTree(name: string): LoroTree +``` + + +Gets or creates a tree container with the given name. Learn about hierarchical editing and moves in [Tree](/docs/tutorial/tree). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroTree` instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const tree = doc.getTree("fileSystem"); +const root = tree.createNode(); +``` + + + + +```typescript no_run +getCounter(name: string): LoroCounter +``` + + +Gets or creates a counter container with the given name. Counters are special CRDTs that sum concurrent increments; see [Counter](/docs/tutorial/counter). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroCounter` instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const counter = doc.getCounter("likes"); +counter.increment(1); +``` + + + + +```typescript no_run +getMovableList(name: string): LoroMovableList +``` + + +Gets or creates a movable list container with the given name. MovableList is designed for reordering with concurrent moves. See [List and Movable List](/docs/tutorial/list) and [Choosing CRDT Types](/docs/concepts/choose_crdt_type). + +**Parameters:** + +- `name` - The container name + +**Returns:** A `LoroMovableList` instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const movableList = doc.getMovableList("tasks"); +movableList.push("Task 1"); +movableList.push("Task 2"); +movableList.push("Task 3"); +movableList.move(0, 2); // Move first item to third position +``` + + + + +```typescript no_run +getContainerById(id: ContainerID): Container | undefined +``` + + +Gets a container by its unique ID. Container IDs (CID) uniquely reference containers across updates; see [Container ID](/docs/advanced/cid) and [Container](/docs/concepts/container). + +**Parameters:** + +- `id` - The container ID + +**Returns:** The container instance or undefined if not found + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("text"); +const textId = text.id; +const sameText = doc.getContainerById(textId); +``` + + + +### Import/Export Methods + + +```typescript no_run +export(mode?: ExportMode): Uint8Array +``` + + +Exports the document in various formats for synchronization or persistence. For a walkthrough of export modes—snapshot, update, shallow-snapshot, and updates-in-range—see [Export Mode](/docs/tutorial/encoding). Shallow snapshots remove history while keeping current state; see [Shallow Snapshots](/docs/concepts/shallow_snapshots). VersionVector and Frontiers are two ways to represent versions; see [Version Vector](/docs/concepts/version_vector) and [Frontiers](/docs/concepts/frontiers). + +**Parameters:** + +- `mode` - Export configuration (optional) + +**ExportMode Options:** + +```typescript no_run +type ExportMode = + | { mode: "snapshot" } + | { mode: "update"; from?: VersionVector } + | { mode: "shallow-snapshot"; frontiers: Frontiers } + | { mode: "updates-in-range"; spans: { id: OpId; len: number }[] }; +``` + +**Returns:** Encoded binary data + +**⚠️ Important Notes:** + +- **Shallow snapshots**: Cannot import updates from before the shallow start point. Peers can only sync if they have versions after this point. +- **Auto-commit**: The document automatically commits pending operations before export. +- **Performance**: Export new snapshots periodically to reduce import times for new peers. + +**Examples:** + +```typescript no_run +import { LoroDoc, VersionVector } from "loro-crdt"; + +const doc = new LoroDoc(); +// ... make some changes to the document ... + +// Export full snapshot +const snapshot = doc.export({ mode: "snapshot" }); + +// Export updates from a specific version +const lastSyncVersion = doc.version(); // Get current version +// ... make more changes ... +const updates = doc.export({ + mode: "update", + from: lastSyncVersion, +}); + +// Export shallow snapshot at current version +const shallowSnapshot = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.frontiers(), +}); +``` + + + + +```typescript no_run +import(data: Uint8Array): ImportStatus +``` + + +Imports updates or snapshots into the document. Returns an `ImportStatus` describing which peer ranges were applied or are pending. See [Sync](/docs/tutorial/sync) and [Import Status](/docs/concepts/import_status) for how Loro handles out-of-order and partial updates. + +**Parameters:** + +- `data` - Binary data or another LoroDoc to import from + +**⚠️ Important:** LoroDoc will automatically commits pending operations before import. If the doc is in detached mode, the imported operations are recorded into OpLog but not applied to DocState until you call `attach()`, see [Attached vs Detached States](/docs/concepts/attached_detached) adn [OpLog and DocState](/docs/concepts/oplog_docstate). + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Receive updates from another peer (e.g., via network) +const otherDoc = new LoroDoc(); +otherDoc.getText("text").insert(0, "Hello"); +const updates: Uint8Array = otherDoc.export({ mode: "update" }); + +// Import binary updates +const status = doc.import(updates); +console.log(status.success); +``` + + + + +```typescript no_run +importBatch(data: Uint8Array[]): ImportStatus +``` + + +Efficiently imports multiple updates in a single batch operation. See [Batch Import](/docs/advanced/import_batch) for performance considerations and usage. + +**Parameters:** + +- `data` - Array of binary updates + +**Example:** + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +declare const update1: Uint8Array; +declare const update2: Uint8Array; +declare const update3: Uint8Array; + +// Usage example: +const updates = [update1, update2, update3]; +const status = doc.importBatch(updates); +``` + + + + +```typescript no_run +exportJsonUpdates(start?: VersionVector, end?: VersionVector, withPeerCompression?: boolean): JsonSchema +``` + + +Exports updates in JSON format for debugging or alternative storage. See [Export Mode](/docs/tutorial/encoding) for format details and trade-offs. + +**Parameters:** + +- `start` - Starting version (optional) +- `end` - Ending version (optional) + +**Returns:** JSON representation of updates + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const jsonUpdates = doc.exportJsonUpdates(); +console.log(JSON.stringify(jsonUpdates, null, 2)); +``` + + + + +```typescript no_run +importJsonUpdates(json: string | JsonSchema): void +``` + + +Imports updates from JSON format. Useful for debugging, migration, or custom storage layers; see [Export Mode](/docs/tutorial/encoding). + +**Parameters:** + +- `json` - JSON string or object containing updates + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const otherDoc = new LoroDoc(); +otherDoc.getText("text").insert(0, "Hello"); +const jsonStr = otherDoc.exportJsonUpdates(); +doc.importJsonUpdates(jsonStr); +``` + + + +## Versioning + +Work with the history DAG using frontiers (heads) and version vectors. Switch, branch, and merge versions safely without manual conflict resolution. See [Versioning Deep Dive](/docs/advanced/version_deep_dive) and [Attached vs Detached States](/docs/concepts/attached_detached). + +### Version Control Methods + + +```typescript no_run +checkout(frontiers: Frontiers): void +``` + + +Checks out the document to a specific version, making it read-only at that point in history. This is the core of time travel; see [Time Travel](/docs/tutorial/time_travel) and [Version](/docs/tutorial/version). + +**Parameters:** + +- `frontiers` - Array of OpIds representing the target version + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +// Make some changes... +doc.checkout(frontiers); // Go back to previous version +``` + + + + +**⚠️ Important:** In Loro 1.0, `version()`/`frontiers()` include pending (uncommitted) local operations. + +**📝 Note:** After `checkout()`, the document enters "detached" mode and becomes read-only by default. Use `attach()` or `checkoutToLatest()` to return to editing mode. See [Version Deep Dive](/docs/advanced/version_deep_dive) and [Attached vs Detached States](/docs/concepts/attached_detached). + + + + +```typescript no_run +checkoutToLatest(): void +``` + + +Returns the document to the latest version after a checkout. Related concepts: [Frontiers](/docs/concepts/frontiers) and [Version Vector](/docs/concepts/version_vector). + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.checkoutToLatest(); +``` + + + + +```typescript no_run +attach(): void +``` + + +Attaches the document to track latest changes after being detached. See [Attached vs Detached States](/docs/concepts/attached_detached) for how Loro separates current state from history. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.attach(); +``` + + + + +```typescript no_run +detach(): void +``` + + +Detaches the document from tracking latest changes, freezing it at current version. See [Attached vs Detached States](/docs/concepts/attached_detached). + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.detach(); +``` + + + + +```typescript no_run +fork(): LoroDoc +``` + + +Creates a new document that is a fork of the current one with a new peer ID. Forking is useful for branching workflows; see [Version](/docs/tutorial/version). + +**Returns:** A new LoroDoc instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const forkedDoc = doc.fork(); +``` + + + + +```typescript no_run +forkAt(frontiers: Frontiers): LoroDoc +``` + + +Creates a fork at a specific version in history. Learn more about versions, DAG history, and heads in [Version Deep Dive](/docs/advanced/version_deep_dive). + +**Parameters:** + +- `frontiers` - The version to fork from + +**Returns:** A new LoroDoc instance + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const forkedDoc = doc.forkAt(frontiers); +``` + + + +## Events & Transactions + +React to changes and group local operations into transactions. Starting in v1.8, events are delivered synchronously; older releases require awaiting a microtask. See [Event Handling](/docs/tutorial/event) and [Transaction Model](/docs/concepts/transaction_model). + +### Subscription Methods + + +```typescript no_run +subscribe(listener: (event: LoroEventBatch) => void): () => void +``` + + +Subscribes to all document changes. See [Event Handling](/docs/tutorial/event) for the event model and best practices. + +**Parameters:** + +- `listener` - Callback function that receives change events + +**Returns:** Unsubscribe function + +**Event Structure:** + +```typescript no_run +interface LoroEventBatch { + by: "local" | "import" | "checkout"; + origin?: string; + currentTarget?: ContainerID; + events: LoroEvent[]; + from: Frontiers; + to: Frontiers; +} +``` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const unsubscribe = doc.subscribe((event) => { + console.log("Change type:", event.by); + event.events.forEach((e) => { + console.log("Container changed:", e.target); + console.log("Diff:", e.diff); + }); +}); + +// Later: unsubscribe(); +``` + +**⚠️ Important:** Events are emitted synchronously as of v1.8. If you are pinned to `<=1.7.x`, await a microtask before reading the batch. + +```ts no_run +doc.commit(); +// Events have already been delivered in v1.8+ +// await Promise.resolve(); // Only needed on <=1.7.x +``` + +**📝 Note:** Multiple operations before a commit are batched into a single event. See [Event Handling](/docs/tutorial/event). + + + + +```typescript no_run +subscribeLocalUpdates(f: (bytes: Uint8Array) => void): () => void +``` + + +Subscribes only to local changes, useful for syncing with remote peers. This is typically wired to your transport layer; see [Sync](/docs/tutorial/sync). + +**Parameters:** + +- `f` - Callback that receives binary updates + +**Returns:** Unsubscribe function + +**Example:** + +```ts no_run threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +declare const websocket: { send: (data: Uint8Array) => void }; + +// Usage example: +const unsubscribe = doc.subscribeLocalUpdates((updates) => { + // Send updates to remote peers + websocket.send(updates); +}); +``` + + + + +```typescript no_run +subscribeFirstCommitFromPeer(f: (e: { peer: PeerID }) => void): () => void +``` + + +Subscribes to the first commit from each peer, useful for tracking peer metadata. + +**Parameters:** + +- `f` - Callback that receives peer information + +**Returns:** Unsubscribe function + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.subscribeFirstCommitFromPeer(({ peer }) => { + // Store peer metadata + doc.getMap("peers").set(peer, { + joinedAt: Date.now(), + name: `User ${peer}`, + }); +}); +``` + + + +### Transaction Methods + + +```typescript no_run +commit(options?: { origin?: string, message?: string, timestamp?: number }): void +``` + + +Commits pending changes as a single transaction. A transaction groups operations into a Change; see [Operations and Changes](/docs/concepts/operations_changes). + +**⚠️ Critical Distinction:** Loro transactions are NOT ACID database transactions: + +- No rollback capability +- No isolation guarantees +- Purpose: Bundle local operations for event batching and history grouping +- Many operations (import/export/checkout) trigger implicit commits + +See [Transaction Model](/docs/concepts/transaction_model). + +**Parameters:** + +- `options` - Optional commit configuration + - `message` - Commit message (persisted in the document like a git commit message, visible to all peers after sync) + - `origin` - Origin identifier (local only - used for marking local events, remote peers won't see this) + - `timestamp` - Unix timestamp in seconds (see [Storing Timestamps](/docs/advanced/timestamp)) + +**Important distinction:** + +- `message` is persisted in the document's history and will be synchronized to all peers, similar to git commit messages +- `origin` is only used locally for filtering events (e.g., excluding certain origins from undo) and is NOT synchronized to remote peers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.commit({ + message: "Updated document title", // Persisted & synced to all peers + origin: "user-action", // Local only, for event filtering + timestamp: Math.floor(Date.now() / 1000), +}); +``` + + + +### Query Methods + + +```typescript no_run +toJSON(): Value +``` + + +Converts the entire document to a JSON-compatible value. If you prefer a structure where sub-containers are referenced by ID (for privacy or streaming), use getShallowValue(); see [Shallow Snapshots](/docs/concepts/shallow_snapshots). + +**Returns:** JSON representation of the document + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const json = doc.toJSON(); +console.log(JSON.stringify(json, null, 2)); +``` + + + + +```typescript no_run +getShallowValue(): Record +``` + + +Gets a shallow representation where sub-containers are represented by their IDs. This is helpful when you want to share structure without history; see [Shallow Snapshots](/docs/concepts/shallow_snapshots). + +**Returns:** Shallow JSON value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const shallow = doc.getShallowValue(); +// Sub-containers appear as: "cid:..." +``` + + + + +```typescript no_run +getDeepValueWithID(): any +``` + + +Gets the deep value of the document with container IDs preserved. This is useful when you need to traverse the document structure while maintaining references to container IDs. + +**Returns:** Document value with container IDs + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const deepValue = doc.getDeepValueWithID(); +``` + + + + +```typescript no_run +version(): VersionVector +``` + + +Gets the current version vector of the document. Version vectors track how much data from each peer you’ve seen; see [Version Vector](/docs/concepts/version_vector). + +**Returns:** Map from PeerID to counter + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.version(); +console.log(vv.toJSON()); +``` + + + + +```typescript no_run +frontiers(): Frontiers +``` + + +Gets the current frontiers (heads) of the document. Frontiers are a compact representation of a version; see [Frontiers](/docs/concepts/frontiers) for when to use them instead of version vectors. + +**📝 Note:** Frontiers are a compact version representation. + +**⚠️ Limitation:** When you have a Frontier pointing to operations you don't know about, you cannot determine the complete set of operation IDs included in that version. Version Vectors don't have this limitation but are more verbose. See [Frontiers](/docs/concepts/frontiers) for trade-offs. + +**Returns:** Array of OpIds + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +// Can be used for checkouts or shallow snapshots +``` + + + + +```typescript no_run +diff(from: Frontiers, to: Frontiers, for_json?: boolean): [ContainerID, Diff | JsonDiff][] +``` + + +Calculates differences between two versions. Understanding how Loro computes diffs benefits from the history DAG model; see [Version Deep Dive](/docs/advanced/version_deep_dive). + +**Parameters:** + +- `from` - Starting frontiers +- `to` - Ending frontiers +- `for_json` - If true, returns JsonDiff format (default: true) + +**Returns:** Array of container IDs and their diffs + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const fromFrontiers = doc.frontiers(); +// Make changes... +const toFrontiers = doc.frontiers(); + +const diffs = doc.diff(fromFrontiers, toFrontiers); +diffs.forEach(([containerId, diff]) => { + console.log(`Container ${containerId} changed:`, diff); +}); +``` + + + +### Pre-Commit Hook + + +```typescript no_run +subscribePreCommit(f: (e: { changeMeta: Change, origin: string, modifier: ChangeModifier }) => void): () => void +``` + + +Subscribe to the pre-commit event. You can modify the message and timestamp of the next change. This hook runs right before a Change is recorded; see [Transaction Model](/docs/concepts/transaction_model) and [Operations and Changes](/docs/concepts/operations_changes). + +Pitfall: `commit()` can be triggered implicitly by `import`, `export`, and `checkout`. Use this hook to attach metadata even for those implicit commits. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const unsubscribe = doc.subscribePreCommit(({ modifier }) => { + modifier + .setMessage("Tagged by pre-commit") + .setTimestamp(Math.floor(Date.now() / 1000)); +}); +doc.getText("text").insert(0, "Hello"); +doc.commit(); +unsubscribe(); +``` + + + +### Cursor Utilities + + +```typescript no_run +getCursorPos(cursor: Cursor): { update?: Cursor, offset: number, side: Side } +``` + + +Resolve a stable `Cursor` to an absolute position. Cursors remain valid across concurrent edits; see [Cursor and Stable Positions](/docs/concepts/cursor_stable_positions) and [Cursor tutorial](/docs/tutorial/cursor). The side controls affinity when the cursor sits at an insertion boundary. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "abc"); + +// Get cursor at position 1 +const c0 = text.getCursor(1); +const pos = doc.getCursorPos(c0!); +console.log(pos.offset); // 1 +``` + + + +### Pending Operations + + +```typescript no_run +getUncommittedOpsAsJson(): JsonSchema | undefined +``` + + +Get pending operations from the current transaction in JSON format. Useful for debugging what will be included in the next Change; see [Transaction Model](/docs/concepts/transaction_model). + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Hello"); +const pending = doc.getUncommittedOpsAsJson(); +doc.commit(); +const none = doc.getUncommittedOpsAsJson(); // undefined after commit +``` + + + +### Change Graph & History + +These APIs traverse the history DAG of changes (ancestors/descendants, spans). If this sounds unfamiliar, start with Loro's [Versioning Deep Dive](/docs/advanced/version_deep_dive) and the [Event Graph Walker](/docs/concepts/event_graph_walker). + + +```typescript no_run +travelChangeAncestors(ids: OpId[], f: (change: Change) => boolean): void +``` + + +Visit ancestors of the given changes in causal order. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "Hello"); +doc.commit(); +const head = doc.frontiers(); +doc.travelChangeAncestors(head, (change) => { + console.log(change.peer, change.counter); + return true; // continue +}); +``` + + + + + ```typescript no_run findIdSpansBetween(from: Frontiers, to: Frontiers): + VersionVectorDiff ``` + +Find the op id spans that lie between two versions. + + +```typescript no_run +exportJsonInIdSpan(idSpan: { peer: PeerID, counter: number, length: number }): JsonChange[] +``` + + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const a = new LoroDoc(); +const b = new LoroDoc(); + +// Usage example: +a.getText("text").update("Hello"); +a.commit(); +const snapshot = a.export({ mode: "snapshot" }); +let printed: any; +b.subscribe((e) => { +const spans = b.findIdSpansBetween(e.from, e.to); +const changes = b.exportJsonInIdSpan(spans.forward[0]); +printed = changes; +}); +b.import(snapshot); + +```` + + + +```typescript no_run +getChangedContainersIn(id: OpId, len: number): ContainerID[] +```` + + + +Get container IDs modified in the given ID range. + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getList("list").insert(0, 1); +doc.commit(); +const head = doc.frontiers()[0]; +const containers = doc.getChangedContainersIn(head, 1); +``` + + + +### Revert & Apply Diff + + +```typescript no_run +revertTo(frontiers: Frontiers): void +``` + + +Revert the document to a given version by generating inverse operations. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setPeerId("1"); +const t = doc.getText("text"); +t.update("Hello"); +doc.commit(); +doc.revertTo([{ peer: "1", counter: 1 }]); +``` + + + + +```typescript no_run +applyDiff(diff: [ContainerID, Diff | JsonDiff][]): void +``` + + +Apply a batch of diffs to the document. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc1 = new LoroDoc(); +const doc2 = new LoroDoc(); + +// Usage example: +doc1.getText("text").insert(0, "Hello"); +const diff = doc1.diff([], doc1.frontiers()); +doc2.applyDiff(diff); +``` + +**Workflow example (squash-like diffs):** For PR-style reviews, combine `diff` and `applyDiff` to send a compact change set between a base version and a new version. Operations that cancel out (insert + delete) are compressed away. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const baseDoc = new LoroDoc(); +const baseText = baseDoc.getText("text"); +baseText.insert(0, "hello world"); + +// Fork to make isolated edits +const newDoc = baseDoc.fork(); +const newText = newDoc.getText("text"); +newText.insert(0, "abc"); +newText.delete(0, 4); + +const diff = newDoc.diff(baseDoc.frontiers(), newDoc.frontiers()); +console.log(diff); +// [ +// [ +// "cid:root-text:Text", +// { type: "text", diff: [ { delete: 1 } ] } +// ] +// ] + +baseDoc.applyDiff(diff); +console.log(baseDoc.toJSON()); +// { text: "ello world" } +``` + + + +### Detached Editing + + +```typescript no_run +setDetachedEditing(enable: boolean): void +``` + + +Enables or disables detached editing mode. Detached editing lets you stage edits separate from the latest head; see [Attached vs Detached States](/docs/concepts/attached_detached). + +**Parameters:** + +- `enable` - Whether to enable detached editing + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setDetachedEditing(true); +``` + + + + +```typescript no_run +isDetachedEditingEnabled(): boolean +``` + + +Checks if detached editing mode is enabled. + +**Returns:** True if detached editing is enabled + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const enabled = doc.isDetachedEditingEnabled(); +``` + + + + +```typescript no_run +isDetached(): boolean +``` + + +Checks if the document is currently detached. + +**Returns:** True if document is detached + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +console.log(doc.isDetached()); +``` + + + +### Commit Options Helpers + + +```typescript no_run +setNextCommitMessage(msg: string): void +``` + + +Sets the message for the next commit. + +**Parameters:** + +- `msg` - Commit message + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitMessage("User action"); +``` + + + + +```typescript no_run +setNextCommitOrigin(origin: string): void +``` + + +Sets the origin for the next commit. + +**Parameters:** + +- `origin` - Origin identifier + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitOrigin("ui"); +``` + + + + +```typescript no_run +setNextCommitTimestamp(timestamp: number): void +``` + + +Sets the timestamp for the next commit. + +**Parameters:** + +- `timestamp` - Unix timestamp in seconds + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitTimestamp(Math.floor(Date.now() / 1000)); +``` + + + + +```typescript no_run +setNextCommitOptions(options: { origin?: string, timestamp?: number, message?: string }): void +``` + + +Sets multiple options for the next commit. + +**Parameters:** + +- `options` - Commit options object + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setNextCommitOptions({ origin: "ui", message: "batch" }); +doc.getText("text").insert(0, "Hi"); +doc.commit(); +``` + + + + +```typescript no_run +clearNextCommitOptions(): void +``` + + +Clears all pending commit options. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.clearNextCommitOptions(); +``` + + + +### Version & Frontier Utilities + + +```typescript no_run +frontiersToVV(frontiers: Frontiers): VersionVector +``` + + +Converts frontiers to a version vector. + +**Parameters:** + +- `frontiers` - Frontiers to convert + +**Returns:** Version vector representation + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const vv = doc.frontiersToVV(frontiers); +``` + + + + +```typescript no_run +vvToFrontiers(vv: VersionVector): Frontiers +``` + + +Converts a version vector to frontiers. + +**Parameters:** + +- `vv` - Version vector to convert + +**Returns:** Frontiers representation + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.version(); +const frontiers = doc.vvToFrontiers(vv); +``` + + + + +```typescript no_run +oplogVersion(): VersionVector +``` + + +Gets the oplog version vector. + +**Returns:** Oplog version vector + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.oplogVersion(); +``` + + + + +```typescript no_run +oplogFrontiers(): Frontiers +``` + + +Gets the oplog frontiers. + +**Returns:** Oplog frontiers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.oplogFrontiers(); +``` + + + + +```typescript no_run +cmpWithFrontiers(frontiers: Frontiers): -1 | 0 | 1 +``` + + +Compares current document state with given frontiers. + +**Parameters:** + +- `frontiers` - Frontiers to compare with + +**Returns:** -1 if behind, 0 if equal, 1 if ahead + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const cmp = doc.cmpWithFrontiers(frontiers); +``` + + + + +```typescript no_run +cmpFrontiers(a: Frontiers, b: Frontiers): -1 | 0 | 1 | undefined +``` + + +Compares two frontiers. + +**Parameters:** + +- `a` - First frontiers +- `b` - Second frontiers + +**Returns:** -1 if a < b, 0 if equal, 1 if a > b, undefined if incomparable + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const f1 = doc.frontiers(); +const f2 = doc.frontiers(); +const cmp = doc.cmpFrontiers(f1, f2); +``` + + + +### JSONPath & Path Queries + +Use simple path strings and JSONPath to fetch nested values and containers. Paths are formed from root container names and keys (e.g., map/key or list/0). For container IDs, see [Container ID](/docs/advanced/cid). + + +```typescript no_run +getByPath(path: string): Value | Container | undefined +``` + + +Gets a value or container by its path. + +**Parameters:** + +- `path` - Path string (e.g., "map/key") + +**Returns:** Value or container at the path, or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("key", 1); + +// Usage example: +const value = doc.getByPath("map/key"); +``` + + + + +```typescript no_run +getPathToContainer(id: ContainerID): (string | number)[] | undefined +``` + + +Gets the path to a container by its ID. + +**Parameters:** + +- `id` - Container ID + +**Returns:** Array representing the path, or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const path = doc.getPathToContainer(map.id); +``` + + + + +```typescript no_run +JSONPath(jsonpath: string): any[] +``` + + +Queries the document using JSONPath syntax. + +**Parameters:** + +- `jsonpath` - JSONPath query string + +**Returns:** Array of matching values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("key", 1); + +// Usage example: +const results = doc.JSONPath("$.map"); +``` + + + +### Shallow Doc Utilities + +These helpers relate to shallow snapshots and redaction. If you need a refresher on what “shallow” means, see [Shallow Snapshots](/docs/concepts/shallow_snapshots). + + +```typescript no_run +shallowSinceVV(): VersionVector +``` + + +Gets the version vector since which the document is shallow. + +**Returns:** Version vector + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const vv = doc.shallowSinceVV(); +``` + + + + +```typescript no_run +shallowSinceFrontiers(): Frontiers +``` + + +Gets the frontiers since which the document is shallow. + +**Returns:** Frontiers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.shallowSinceFrontiers(); +``` + + + + +```typescript no_run +isShallow(): boolean +``` + + +Checks if the document is shallow. + +**Returns:** True if document is shallow + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const shallow = doc.isShallow(); +``` + + + + +```typescript no_run +setHideEmptyRootContainers(hide: boolean): void +``` + + +Controls whether empty root containers are hidden in JSON output. + +**Parameters:** + +- `hide` - Whether to hide empty root containers + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.setHideEmptyRootContainers(true); +// Now empty roots are hidden in toJSON() +``` + + + + +```typescript no_run +deleteRootContainer(cid: ContainerID): void +``` + + +Deletes a root container. + +**Parameters:** + +- `cid` - Container ID to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +// Usage example: +doc.deleteRootContainer(map.id); +``` + + + + +```typescript no_run +hasContainer(id: ContainerID): boolean +``` + + +Checks if a container exists in the document. + +**Parameters:** + +- `id` - Container ID to check + +**Returns:** True if container exists + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const exists = doc.hasContainer(map.id); +``` + + + +### JSON Serialization with Replacer + + +```typescript no_run +toJsonWithReplacer(replacer: (key: string | number, value: Value | Container) => Value | Container | undefined): Value +``` + + +Customize JSON serialization of containers and values. + +**Parameters:** + +- `replacer` - Function to transform values during serialization + +**Returns:** Customized JSON value + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const json = doc.toJsonWithReplacer((key, value) => { + if (value instanceof LoroText) { + return value.toDelta(); + } + return value; +}); +``` + + + +### Stats & Introspection + + +```typescript no_run +debugHistory(): void +``` + + +Prints debug information about the document history. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.debugHistory(); +``` + + + + +```typescript no_run +changeCount(): number +``` + + +Gets the total number of changes in the document. + +**Returns:** Number of changes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const changes = doc.changeCount(); +``` + + + + +```typescript no_run +opCount(): number +``` + + +Gets the total number of operations in the document. + +**Returns:** Number of operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const ops = doc.opCount(); +``` + + + + +```typescript no_run +getAllChanges(): Map +``` + + +Gets all changes grouped by peer ID. + +**Returns:** Map of peer ID to changes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const changes = doc.getAllChanges(); +``` + + + + +```typescript no_run +getChangeAt(id: OpId): Change +``` + + +Gets a specific change by operation ID. + +**Parameters:** + +- `id` - Operation ID + +**Returns:** Change object + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "hello"); +doc.commit(); +const changes = doc.getAllChanges(); +const change = changes.get(doc.peerIdStr)?.[0]; +``` + + + + +```typescript no_run +getChangeAtLamport(peer_id: string, lamport: number): Change | undefined +``` + + +Gets a change by peer ID and Lamport timestamp. + +**Parameters:** + +- `peer_id` - Peer ID +- `lamport` - Lamport timestamp + +**Returns:** Change object or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "hello"); +const change = doc.getChangeAtLamport(doc.peerIdStr, 1); +``` + + + + +```typescript no_run +getOpsInChange(id: OpId): any[] +``` + + +Gets all operations in a specific change. + +**Parameters:** + +- `id` - Operation ID + +**Returns:** Array of operations + +**Example:** + +```typescript no_run +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "hello"); +const changes = doc.getAllChanges(); +const ops = doc.getOpsInChange(changes[0].id); +``` + + + + +```typescript no_run +getPendingTxnLength(): number +``` + + +Gets the number of pending operations in the current transaction. + +**Returns:** Number of pending operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +doc.getText("text").insert(0, "x"); +console.log(doc.getPendingTxnLength()); +doc.commit(); +``` + + + +### Import/Export Utilities + + +```typescript no_run +decodeImportBlobMeta(blob: Uint8Array, check_checksum: boolean): ImportBlobMetadata +``` + + +Decodes metadata from an import blob. + +**Parameters:** + +- `blob` - Binary data to decode +- `check_checksum` - Whether to verify checksum + +**Returns:** Import blob metadata + +**Example:** + +```ts threeslash +import { LoroDoc, decodeImportBlobMeta } from "loro-crdt"; + +const doc = new LoroDoc(); +const updates = doc.export({ mode: "update" }); +const meta = decodeImportBlobMeta(updates, true); +``` + + + + +```typescript no_run +redactJsonUpdates(json: string | JsonSchema, version_range: any): JsonSchema +``` + + +Redacts JSON updates within a specified version range. + +Use this to safely remove accidentally leaked sensitive content from history while preserving structure. See [Tips: Redaction](/docs/tutorial/tips). + +**Parameters:** + +- `json` - JSON updates to redact +- `version_range` - Version range for redaction + +**Returns:** Redacted JSON schema + +**Example:** + +```ts threeslash +import { LoroDoc, redactJsonUpdates } from "loro-crdt"; + +const doc = new LoroDoc(); +const json = doc.exportJsonUpdates(); +const redacted = redactJsonUpdates(json, { [doc.peerIdStr]: [0, 999999] }); +``` + + + +--- + +## Container Types + +Common CRDT containers for modeling JSON-like structures. See [Choosing CRDT Types](/docs/concepts/choose_crdt_type) and [Composing CRDTs](/docs/tutorial/composition) for when to use each and how to nest them. + +### LoroText + +A rich text container supporting collaborative text editing with formatting. Supports overlapping marks (bold, italic, links) and +stable cursors. The merge semantics avoid interleaving artifacts under concurrency (Fugue + Eg-walker ideas); you use simple index +APIs and Loro handles index transformation. +See [Text](/docs/tutorial/text), [Eg-walker](/docs/advanced/event_graph_walker), and the rich text blog: https://loro.dev/blog/loro-richtext + +**⚠️ Critical: UTF-16 String Encoding** + +LoroText uses **UTF-16** encoding, matching JavaScript's native string encoding: + +- All standard methods (`insert()`, `delete()`, `mark()`, `slice()`, `charAt()`) use UTF-16 code unit indices +- `length` returns UTF-16 code units (same as JavaScript `string.length`) +- Use `insertUtf8()` and `deleteUtf8()` for UTF-8 byte-based operations when integrating with UTF-8 systems + +**⚠️ Common Pitfalls:** + +1. **Index Misalignment**: UTF-16 indices differ from visual character count +2. **Performance**: Cursor queries on deleted positions require history traversal - in that case, it will return a refreshed Cursor object that does not point to the deleted text + +**Example with emoji:** + +```typescript no_run +const text = doc.getText("text"); +text.insert(0, "Hello 😀 World"); +console.log(text.length); // 13 (emoji counts as 2) +console.log(text.toString()[6]); // ⚠️ Invalid - splits the emoji +text.delete(6, 2); // ✅ Correct - deletes entire emoji +text.delete(6, 1); // ❌ Wrong - corrupts the emoji + +// Safe iteration +text.iter((char) => { + console.log(char); // Each character handled correctly + return true; +}); +``` + +**📝 Text vs String in Maps:** + +- Use `LoroText` for collaborative text editing where all concurrent edits must be preserved +- Use regular strings in `LoroMap` for atomic values (URLs, IDs, hashes) where Last-Write-Wins is preferred +- Example: URLs should be strings in maps, not LoroText. Otherwise, the automatically merged result may be an invalid URL + + + ```typescript no_run insert(index: number, text: string): void ``` + + + +Inserts text at the specified position using UTF-16 code unit indices (same as JavaScript string indices). + +**Parameters:** + +- `index` - UTF-16 code unit position to insert at (0-based, same as JavaScript string index) +- `text` - Text to insert + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +// Usage example: +text.insert(0, "Hello "); +text.insert(6, "World"); +``` + + + + + ```typescript no_run delete(index: number, len: number): void ``` + + + +Deletes text from the specified position using UTF-16 code units. + +**Parameters:** + +- `index` - Starting UTF-16 code unit position (same as JavaScript string index) +- `len` - Number of UTF-16 code units to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello 😀 World"); +text.delete(6, 2); // Delete emoji (2 UTF-16 units) +text.delete(5, 1); // Delete space before World +``` + + + + +```typescript no_run +mark(range: { start: number, end: number }, key: string, value: Value): void +``` + + + +Applies formatting to a text range. Marks can be configured to expand/stop at edges via configTextStyle(); see [Text](/docs/tutorial/text) for mark behavior. + +**Parameters:** + +- `range` - The range to format +- `key` - Style attribute name +- `value` - Style value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +doc.configTextStyle({ bold: { expand: "after" } }); +text.mark({ start: 0, end: 5 }, "bold", true); +``` + + + + +```typescript no_run +unmark(range: { start: number, end: number }, key: string): void +``` + + +Removes formatting from a text range. For how conflicting edits on marks resolve, see [Text](/docs/tutorial/text). + +**Parameters:** + +- `range` - The range to unformat +- `key` - Style attribute to remove + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +text.mark({ start: 0, end: 5 }, "bold", true); +text.unmark({ start: 0, end: 5 }, "bold"); +``` + + + + +```typescript no_run +toDelta(): Delta[] +``` + + +Converts text to Delta format (Quill-compatible). + +**Returns:** Array of Delta operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +text.mark({ start: 0, end: 5 }, "bold", true); +const delta = text.toDelta(); +// [{ insert: "Hello", attributes: { bold: true } }, { insert: " World" }] +``` + + + + +```typescript no_run +sliceDelta(start: number, end: number): Delta[] +``` + + +Returns a Quill-style Delta for a subsection of the text, using UTF-16 indices. Useful for copying a styled span. Use `sliceDeltaUtf8` if you need UTF-8 byte offsets instead. + +**Parameters:** + +- `start` - Start UTF-16 code unit index (inclusive) +- `end` - End UTF-16 code unit index (exclusive) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +const doc = new LoroDoc(); +doc.configTextStyle({ + bold: { expand: "after" }, + comment: { expand: "none" }, +}); +const text = doc.getText("text"); + +text.insert(0, "Hello World!"); +text.mark({ start: 0, end: 5 }, "bold", true); +text.mark({ start: 6, end: 11 }, "comment", "greeting"); + +const snippet = text.sliceDelta(1, 8); +expect(snippet).toStrictEqual([ + { insert: "ello", attributes: { bold: true } }, + { insert: " " }, + { insert: "Wo", attributes: { comment: "greeting" } }, +]); +``` + + + + +```typescript no_run +sliceDeltaUtf8(start: number, end: number): Delta[] +``` + + +Returns a Quill-style Delta for a subsection of the text using **UTF-8 byte offsets**. Choose this when your offsets come from UTF-8 encoded buffers. + +**Parameters:** + +- `start` - Start byte offset (inclusive) +- `end` - End byte offset (exclusive) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hi 👋"); + +const enc = new TextEncoder(); +const start = enc.encode("Hi ").length; // 3 bytes +const end = enc.encode("Hi 👋").length; // 7 bytes + +const delta = text.sliceDeltaUtf8(start, end); +expect(delta).toStrictEqual([{ insert: "👋" }]); +``` + + + + +```typescript no_run +applyDelta(delta: Delta[]): void +``` + + +Applies Delta operations to the text. + +**Parameters:** + +- `delta` - Array of Delta operations + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.applyDelta([ + { insert: "Hello", attributes: { bold: true } }, + { insert: " World" }, +]); +``` + + + + +```typescript no_run +update(text: string, options?: { timeoutMs?: number; useRefinedDiff?: boolean }): void +``` + + +Updates the current text to the target text using Myers' diff algorithm. + +**Parameters:** + +- `text` - New text content +- `options` - Update options + - `timeoutMs` - Optional timeout for the diff computation + - `useRefinedDiff` - Use refined diff for better quality on long texts + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Hello"); +text.update("Hello World", { timeoutMs: 100 }); +``` + + + + +```typescript no_run +updateByLine(text: string, options?: { timeoutMs?: number; useRefinedDiff?: boolean }): void +``` + + +Line-based update that's faster for large texts (less precise than `update`). + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insert(0, "Line A\nLine C"); +text.updateByLine("Line A\nLine B\nLine C"); +``` + + + + +```typescript no_run +getCursor(pos: number, side?: Side): Cursor | undefined +``` + + +Gets a stable cursor position that survives edits. + +**Parameters:** + +- `pos` - Position in the text +- `side` - Cursor affinity (-1, 0, or 1) + +**Returns:** Cursor object or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +const cursor = text.getCursor(5); +// Cursor remains valid even after edits +``` + + + + +```typescript no_run +toString(): string +``` + + +Converts to plain text string. + +**Returns:** Plain text content + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); +const plainText = text.toString(); +``` + + + + +```typescript no_run +charAt(pos: number): string +``` + + +Gets the character at a specific UTF-16 code unit position. + +**Parameters:** + +- `pos` - UTF-16 code unit position + +**Returns:** Character at position + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); +const char = text.charAt(1); // "e" +``` + + + + +```typescript no_run +slice(start: number, end: number): string +``` + + +Extracts a section of the text using UTF-16 code unit positions. + +**Parameters:** + +- `start` - Start UTF-16 code unit index +- `end` - End UTF-16 code unit index (exclusive) + +**Returns:** Sliced text + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello 😀 World"); + +const slice1 = text.slice(0, 5); // "Hello" +const slice2 = text.slice(6, 8); // "😀" (emoji spans 6-8) +const slice3 = text.slice(9, 14); // "World" +``` + + + + +```typescript no_run +splice(pos: number, len: number, s: string): string +``` + + +Replaces text at a position with new content. + +**Parameters:** + +- `pos` - Start position +- `len` - Length to delete +- `s` - String to insert + +**Returns:** Deleted text + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +// Usage example: +const deleted = text.splice(6, 5, "Loro"); // returns "World" +``` + + + + +```typescript no_run +push(s: string): void +``` + + +Appends text to the end of the document. + +**Parameters:** + +- `s` - String to append + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.push("Hello"); +text.push(" World"); +``` + + + + +```typescript no_run +iter(callback: (char: string) => boolean): void +``` + + +Iterates over each character in the text. + +**Parameters:** + +- `callback` - Function called for each character. Return false to stop iteration. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +text.iter((char) => { + console.log(char); + return true; // continue iteration +}); +``` + + + + +```typescript no_run +insertUtf8(index: number, content: string): void +``` + + +Inserts text at a UTF-8 byte index position. + +**Parameters:** + +- `index` - UTF-8 byte index +- `content` - Text to insert + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +text.insertUtf8(0, "Hello"); +``` + + + + +```typescript no_run +deleteUtf8(index: number, len: number): void +``` + + +Deletes text at a UTF-8 byte index position. + +**Parameters:** + +- `index` - UTF-8 byte index +- `len` - Number of UTF-8 bytes to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello World"); + +// Usage example: +text.deleteUtf8(6, 5); // Delete "World" +``` + + + + +```typescript no_run +getEditorOf(pos: number): PeerID | undefined +``` + + +Gets the peer ID of who last edited the character at a position. + +**Parameters:** + +- `pos` - Character position + +**Returns:** PeerID of last editor or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const editor = text.getEditorOf(0); +``` + + + + +```typescript no_run +kind(): "Text" +``` + + +Returns the container type. + +**Returns:** "Text" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +const type = text.kind(); // "Text" +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this text is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const text = list.insertContainer(0, doc.getText("nested")); + +// Usage example: +const parent = text.parent(); // Returns the list +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const text = new LoroText(); + +const attached = text.isAttached(); // false until attached to doc +``` + + + + +```typescript no_run +getAttached(): LoroText | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const text = new LoroText(); + +const attached = text.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +const deleted = text.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): string +``` + + +Gets the text content without marks. + +**Returns:** Plain text string + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const value = text.getShallowValue(); // "Hello" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the text to JSON representation. + +**Returns:** JSON value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); + +// Usage example: +const json = text.toJSON(); // "Hello" +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +const containerId = text.id; +``` + + + + +```typescript no_run +readonly length: number +``` + + +Gets the length of the text in UTF-16 code units (same as JavaScript's `string.length`). + +**⚠️ Important:** Emoji and other characters outside the Basic Multilingual Plane count as 2 UTF-16 units. This affects all index-based operations: + +```typescript no_run +text.insert(0, "👨‍👩‍👧‍👦"); // Family emoji +console.log(text.length); // 11 (not 1!) - complex emoji with ZWJ sequences +``` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); +console.log(text.length); // 5 + +text.insert(5, " 😀"); +console.log(text.length); // 8 (space + emoji which counts as 2) +``` + + + +### LoroList + +An ordered list container for collaborative arrays. Uses index-based APIs; under concurrency, Loro transforms indices +by replaying only the necessary portion of history (Eg-walker-inspired). See [List and Movable List](/docs/tutorial/list), +[Choosing CRDT Types](/docs/concepts/choose_crdt_type), and [Eg-walker](/docs/advanced/event_graph_walker). + +**⚠️ Important: List vs Map for Coordinates** + +```typescript no_run +// ❌ WRONG - Don't use List for coordinates +const coord = doc.getList("coord"); +coord.push(10); // x +coord.push(20); // y +// Concurrent updates can create [10, 20a, 20b] instead of [10, 20] + +// ✅ CORRECT - Use Map for coordinates +const coord = doc.getMap("coord"); +coord.set("x", 10); +coord.set("y", 20); +// Concurrent updates properly merge to {x: 10, y: 20} +``` + + +```typescript no_run +insert(pos: number, value: Value | Container): void +``` + + +Inserts a value at the specified position. + +**Parameters:** + +- `pos` - Insert position +- `value` - Value to insert + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.insert(0, "First"); +list.insert(1, { type: "object" }); +``` + + + + +```typescript no_run +insertContainer(pos: number, container: T): T +``` + + +Inserts a new container at the position. + +**Parameters:** + +- `pos` - Insert position +- `container` - Container instance + +**Returns:** The inserted container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const subText = list.insertContainer(0, new LoroText()); +subText.insert(0, "Nested text"); +``` + + + + +```typescript no_run +delete(pos: number, len: number): void +``` + + +Deletes elements from the list. + +**Parameters:** + +- `pos` - Starting position +- `len` - Number of elements to delete + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a"); +list.push("b"); +list.push("c"); +list.push("d"); + +list.delete(1, 2); // Delete 2 elements starting at index 1 +``` + + + + +```typescript no_run +push(value: Value | Container): void +``` + + +Appends a value to the end of the list. + +**Parameters:** + +- `value` - Value to append + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("Last item"); +``` + + + + +```typescript no_run +getIdAt(pos: number): { peer: PeerID, counter: number } | undefined +``` + + +```typescript no_run +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +list.insert(0, 1); +const id0 = list.getIdAt(0); + +```` + + + +```typescript no_run +pushContainer(container: T): T +```` + + + +Appends a container to the end of the list. + +**Parameters:** + +- `container` - Container to append + +**Returns:** The appended container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroMap } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const map = list.pushContainer(new LoroMap()); +map.set("key", "value"); +``` + + + + +```typescript no_run +pop(): Value | Container | undefined +``` + + +Removes and returns the last element. + +**Returns:** The removed element or undefined + +**Example:** + +```ts no_run threeslash +import { LoroList } from "loro-crdt"; +declare const list: LoroList; +// ---cut--- +const lastItem = list.pop(); +``` + + + + +```typescript no_run +get(index: number): Value | Container | undefined +``` + + +Gets the value at the specified index. + +**Parameters:** + +- `index` - Element index + +**Returns:** The value or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const list = doc.getList("items"); +list.push("first", "second"); + +const item = list.get(0); // "first" +``` + + + + +```typescript no_run +getCursor(pos: number, side?: Side): Cursor | undefined +``` + + +Gets a stable cursor for the position. + +**Parameters:** + +- `pos` - Position in the list +- `side` - Cursor affinity + +**Returns:** Cursor object or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a", "b", "c"); + +const cursor = list.getCursor(2); +``` + + + + +```typescript no_run +toArray(): (Value | Container)[] +``` + + +Converts the list to a JavaScript array. + +**Returns:** Array of values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a", "b", "c"); +const array = list.toArray(); +``` + + + + +```typescript no_run +clear(): void +``` + + +Removes all elements from the list. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a", "b", "c"); +list.clear(); +``` + + + + +```typescript no_run +length: number +``` + + +Gets the number of elements in the list. + +**Returns:** List length + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push("a"); +list.push("b"); +list.push("c"); +console.log(`List has ${list.length} items`); +``` + + + + +```typescript no_run +kind(): "List" +``` + + +Returns the container type. + +**Returns:** "List" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +const type = list.kind(); // "List" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the list to JSON representation. + +**Returns:** JSON array + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push(1, 2, 3); + +// Usage example: +const json = list.toJSON(); // [1, 2, 3] +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this list is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +const list = map.setContainer("nested", doc.getList("list")); + +// Usage example: +const parent = list.parent(); // Returns the map +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroList } from "loro-crdt"; +const doc = new LoroDoc(); +const list = new LoroList(); + +const attached = list.isAttached(); // false until attached to doc +``` + + + + +```typescript no_run +getAttached(): LoroList | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroList } from "loro-crdt"; +const doc = new LoroDoc(); +const list = new LoroList(); + +const attached = list.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +const deleted = list.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): Value[] +``` + + +Gets the list values with sub-containers as IDs. + +**Returns:** Array of values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.push(1, 2); + +// Usage example: +const values = list.getShallowValue(); // [1, 2] +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +const containerId = list.id; +``` + + + +### LoroMap + +A key-value map container for collaborative objects. See [Map](/docs/tutorial/map). + + +```typescript no_run +set(key: string, value: Value | Container): void +``` + + +Sets a key-value pair. + +Note: Setting a key to the same value is a no-op (no operation recorded). See [Map basics](/docs/tutorial/map). + +**Parameters:** + +- `key` - The key +- `value` - The value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.set("name", "Alice"); +map.set("age", 30); +``` + + + + +```typescript no_run +setContainer(key: string, container: T): T +``` + + +Sets a container as the value for a key. + +**⚠️ Pitfall:** Concurrent regular child container creation at the same key on a LoroMap from multiple peers causes overwrites (different container IDs cannot auto-merge). If the child should be shared by logical key, use the `ensureMergeable*` methods instead. See [Container initialization](/docs/tutorial/tips). + +**Parameters:** + +- `key` - The key +- `container` - Container instance + +**Returns:** The set container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroList } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const list = map.setContainer("items", new LoroList()); +list.push("item1"); +``` + + + + +```typescript no_run +ensureMergeableText(key: string): LoroText +``` + + +Ensures that `key` points to a mergeable Text child container. Peers that call this method concurrently with the same parent Map and key will address the same logical Text container instead of creating competing child container IDs. + +Use this for lazily-created text fields that should behave like one shared document for all peers. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Text container + +**Example:** + +```ts no_run +const body = doc.getMap("note").ensureMergeableText("body"); +body.insert(0, "Hello"); +``` + + + + +```typescript no_run +ensureMergeableMap(key: string): LoroMap +``` + + +Ensures that `key` points to a mergeable Map child container. Use this when a nested record, profile, settings object, or migration-created child Map may be initialized by multiple peers. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Map container + +**Example:** + +```ts no_run +const record = doc.getMap("records").ensureMergeableMap(recordId); +record.set("updated", true); +``` + + + + +```typescript no_run +ensureMergeableList(key: string): LoroList +``` + + +Ensures that `key` points to a mergeable List child container. Use this for dynamic child lists, such as one list per day, where concurrent first creation should produce one shared List. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable List container + +**Example:** + +```ts no_run +const entries = doc.getMap("days").ensureMergeableList("2026-06-08"); +entries.insert(0, "meeting notes"); +``` + + + + +```typescript no_run +ensureMergeableMovableList(key: string): LoroMovableList +``` + + +Ensures that `key` points to a mergeable MovableList child container. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable MovableList container + + + + +```typescript no_run +ensureMergeableTree(key: string): LoroTree +``` + + +Ensures that `key` points to a mergeable Tree child container. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Tree container + + + + +```typescript no_run +ensureMergeableCounter(key: string): LoroCounter +``` + + +Ensures that `key` points to a mergeable Counter child container. + +**Parameters:** + +- `key` - The Map key for the mergeable child container + +**Returns:** The mergeable Counter container + + + + +```typescript no_run +get(key: string): Value | Container | undefined +``` + + +Gets the value for a key. + +**Parameters:** + +- `key` - The key + +**Returns:** The value or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const name = map.get("name"); +``` + + + + +```typescript no_run +/** + * @deprecated Prefer ensureMergeableText/Map/List/... for lazily-created Map child containers. + */ +getOrCreateContainer(key: string, container: T): T +``` + + +**Deprecated:** Prefer the `ensureMergeable*` methods for Map child containers that may be created lazily or concurrently by multiple peers. + +Gets an existing container or creates a new regular child container. + +**⚠️ Pitfall:** Parallel container creation for the same key across peers creates different child container IDs and can cause overwrites. This method is kept for compatibility and for cases where you explicitly want a regular child container. For shared dynamic children, use `ensureMergeableText`, `ensureMergeableMap`, `ensureMergeableList`, `ensureMergeableMovableList`, `ensureMergeableTree`, or `ensureMergeableCounter`. See [Container initialization](/docs/tutorial/tips). + +**Parameters:** + +- `key` - The key +- `container` - Container to create if not exists + +**Returns:** The container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const text = map.getOrCreateContainer("description", new LoroText()); +``` + + + + +```typescript no_run +delete(key: string): void +``` + + +Removes a key-value pair. + +**Parameters:** + +- `key` - The key to remove + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.delete("obsoleteKey"); +``` + + + + +```typescript no_run +clear(): void +``` + + +Removes all key-value pairs. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.clear(); +``` + + + + +```typescript no_run +keys(): string[] +``` + + +Gets all keys in the map. + +**Returns:** Array of keys + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const allKeys = map.keys(); +``` + + + + +```typescript no_run +values(): (Value | Container)[] +``` + + +Gets all values in the map. + +**Returns:** Array of values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const allValues = map.values(); +``` + + + + +```typescript no_run +entries(): [string, Value | Container][] +``` + + +Gets all key-value pairs. + +**Returns:** Array of entries + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +for (const [key, value] of map.entries()) { + console.log(`${key}: ${value}`); +} +``` + + + + +```typescript no_run +getLastEditor(key: string): PeerID | undefined +``` + + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +map.set("k", 1); +doc.commit(); +const who = map.getLastEditor("k"); +// who = doc.peerIdStr + +```` + + + +```typescript no_run +size: number +```` + + + +Gets the number of key-value pairs. + +**Returns:** Map size + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +console.log(`Map has ${map.size} entries`); +``` + + + + +```typescript no_run +kind(): "Map" +``` + + +Returns the container type. + +**Returns:** "Map" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const type = map.kind(); // "Map" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the map to JSON representation. + +**Returns:** JSON object + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("name", "Alice"); + +// Usage example: +const json = map.toJSON(); // { name: "Alice" } +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this map is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); +const map = list.insertContainer(0, doc.getMap("nested")); + +// Usage example: +const parent = map.parent(); // Returns the list +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroMap } from "loro-crdt"; +const doc = new LoroDoc(); +const map = new LoroMap(); + +const attached = map.isAttached(); // false until attached to doc +``` + + + + +```typescript no_run +getAttached(): LoroMap | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroMap } from "loro-crdt"; +const doc = new LoroDoc(); +const map = new LoroMap(); + +const attached = map.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const deleted = map.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): Record +``` + + +Gets the map values with sub-containers as IDs. + +**Returns:** Object with values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("key", "value"); + +// Usage example: +const values = map.getShallowValue(); // { key: "value" } +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +const containerId = map.id; +``` + + + +### LoroTree + +A hierarchical tree container for nested structures. Supports moving subtrees while handling concurrent edits. See [Tree](/docs/tutorial/tree). + +**⚠️ Important Tree Operation Notes:** + +- **Concurrent moves can create cycles**: Loro detects and prevents these automatically +- **Fractional indexing**: Has interleaving issues but maintains relative ordering +- **Don't disable fractional index** if you need siblings to be sorted. See [Tree](/docs/tutorial/tree). + + +```typescript no_run +createNode(parent?: TreeID, index?: number): LoroTreeNode +``` + + +Creates a new tree node. + +**Parameters:** + +- `parent` - Parent node ID (optional, creates root if omitted) +- `index` - Position among siblings (optional) + +**Returns:** The new node's handler + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const root = tree.createNode(); +const child = root.createNode(0); +``` + + + + +```typescript no_run +move(target: TreeID, parent?: TreeID, index?: number): void +``` + + +Moves a node to a new position. + +**Parameters:** + +- `target` - Node to move +- `parent` - New parent (undefined for root) +- `index` - Position among siblings + +**Example:** + +```typescript no_run threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +declare const nodeId: TreeID; +declare const newParentId: TreeID; + +// Usage example: +tree.move(nodeId, newParentId, 0); +``` + + + + +```typescript no_run +delete(target: TreeID): void +``` + + +Deletes a node and its descendants. + +**Parameters:** + +- `target` - Node to delete + +**Example:** + +```ts threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const nodeId = node.id; + +// Usage example: +tree.delete(nodeId); +``` + + + + +```typescript no_run +getNodeByID(id: TreeID): LoroTreeNode | undefined +``` + + +Gets a node handler by its ID. + +**Parameters:** + +- `id` - Node ID + +**Returns:** Node handler or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const _node = tree.createNode(); +const nodeId = _node.id; + +// Usage example: +const node = tree.getNodeByID(nodeId); +if (node) { + node.data.set("label", "New Label"); +} +``` + + + + +```typescript no_run +nodes(): LoroTreeNode[] +``` + + +Gets all nodes in the tree. + +**Returns:** Array of all nodes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const allNodes = tree.nodes(); +``` + + + + +```typescript no_run +roots(): LoroTreeNode[] +``` + + +Gets all root nodes. + +**Returns:** Array of root nodes + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const rootNodes = tree.roots(); +``` + + + + +```typescript no_run +has(target: TreeID): boolean +``` + + +Checks if a node exists. + +**Parameters:** + +- `target` - Node ID to check + +**Returns:** Boolean indicating existence + +**Example:** + +```typescript no_run threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +declare const nodeId: TreeID; + +// Usage example: +if (tree.has(nodeId)) { + console.log("Node exists"); +} +``` + + + + +```typescript no_run +isNodeDeleted(target: TreeID): boolean +``` + + +Checks if a node has been deleted. + +**Parameters:** + +- `target` - Node ID to check + +**Returns:** Boolean indicating deletion status + +**Example:** + +```typescript no_run threeslash +import { LoroDoc, TreeID } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +declare const nodeId: TreeID; + +// Usage example: +if (tree.isNodeDeleted(nodeId)) { + console.log("Node was deleted"); +} +``` + + + +```typescript no_run +enableFractionalIndex(jitter: number): void +``` + + +Enables fractional indexing for better concurrent move performance. + +**Parameters:** + +- `jitter` - Jitter amount for fractional indices + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +tree.enableFractionalIndex(0.001); +``` + + + + +```typescript no_run +disableFractionalIndex(): void +``` + + +Disables fractional indexing. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +tree.disableFractionalIndex(); +``` + + + + +```typescript no_run +isFractionalIndexEnabled(): boolean +``` + + +Checks if fractional indexing is enabled. + +**Returns:** True if enabled + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const enabled = tree.isFractionalIndexEnabled(); +``` + + + + +```typescript no_run +kind(): "Tree" +``` + + +Returns the container type. + +**Returns:** "Tree" + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const type = tree.kind(); // "Tree" +``` + + + + +```typescript no_run +toJSON(): any +``` + + +Converts the tree to JSON representation. + +**Returns:** JSON tree structure + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const json = tree.toJSON(); +``` + + + + +```typescript no_run +parent(): Container | undefined +``` + + +Gets the parent container if this tree is nested. + +**Returns:** Parent container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); +const tree = map.setContainer("tree", doc.getTree("nested")); + +// Usage example: +const parent = tree.parent(); +``` + + + + +```typescript no_run +isAttached(): boolean +``` + + +Checks if the container is attached to a document. + +**Returns:** True if attached + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTree } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = new LoroTree(); + +const attached = tree.isAttached(); +``` + + + + +```typescript no_run +getAttached(): LoroTree | undefined +``` + + +Gets the attached version of this container. + +**Returns:** Attached container or undefined + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTree } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = new LoroTree(); + +const attached = tree.getAttached(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if the container has been deleted. + +**Returns:** True if deleted + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const deleted = tree.isDeleted(); +``` + + + + +```typescript no_run +getShallowValue(): TreeNodeShallowValue[] +``` + + +Gets the tree values with sub-containers as IDs. + +**Returns:** Array of tree node values + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const values = tree.getShallowValue(); +``` + + + + +```typescript no_run +readonly id: ContainerID +``` + + +Gets the unique container ID. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +const containerId = tree.id; +``` + + + +### LoroTreeNode + +Represents a single node in the tree. + + +```typescript no_run +data: LoroMap +``` + + +A map container for storing node metadata. + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +node.data.set("title", "Node Title"); +node.data.set("expanded", true); +``` + + + + +```typescript no_run +createNode(index?: number): LoroTreeNode +``` + + +Creates a child node. + +**Parameters:** + +- `index` - Position among siblings + +**Returns:** New node's handler + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const childId = node.createNode(0); +``` + + + + +```typescript no_run +move(parent?: LoroTreeNode, index?: number): void +``` + + +Moves this node to a new parent. + +**Parameters:** + +- `parent` - New parent node +- `index` - Position among siblings + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTreeNode } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const parent = tree.createNode(); + +// Usage example: +node.move(parent, 0); +``` + + + + +```typescript no_run +moveAfter(target: LoroTreeNode): void +``` + + +Moves this node after a sibling. + +**Parameters:** + +- `target` - Sibling node + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTreeNode } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const sibling = tree.createNode(); + +// Usage example: +node.moveAfter(sibling); +``` + + + + +```typescript no_run +moveBefore(target: LoroTreeNode): void +``` + + +Moves this node before a sibling. + +**Parameters:** + +- `target` - Sibling node + +**Example:** + +```ts threeslash +import { LoroDoc, LoroTreeNode } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); +const sibling = tree.createNode(); + +// Usage example: +node.moveBefore(sibling); +``` + + + + +```typescript no_run +parent(): LoroTreeNode | undefined +``` + + +Gets the parent node. + +**Returns:** Parent node or undefined + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +const parentNode = node.parent(); +``` + + + + +```typescript no_run +children(): LoroTreeNode[] +``` + + +Gets all child nodes. + +**Returns:** Array of children + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +const childNodes = node.children(); +``` + + + + +```typescript no_run +index(): number | undefined +``` + + +Gets the position among siblings. + +**Returns:** Index or undefined if root + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +const position = node.index(); +``` + + + + +```typescript no_run +fractionalIndex(): string | undefined +``` + + +Returns the node's fractional index used to sort siblings deterministically. +It is a hex string representation of the Fractional Index and is stable for ordering. +Returns `undefined` for the root node. Note: the tree must be attached to the document. + +**Returns:** Hex string or `undefined` for root + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const parent = tree.createNode(); +const a = parent.createNode(0); +const b = parent.createNode(1); + +const aFi = a.fractionalIndex(); +const bFi = b.fractionalIndex(); +// aFi < bFi, because b is inserted after a +``` + + + + +```typescript no_run +creationId(): { peer: PeerID, counter: number } +``` + + +Returns the OpID that created this node. + +**Returns:** `{ peer: PeerID, counter: number }` creation identifier + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +const { peer, counter } = node.creationId(); +``` + + + + +```typescript no_run +creator(): PeerID +``` + + +Returns the peer ID that created this node (equivalent to `creationId().peer`). + +**Returns:** `PeerID` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +const author = node.creator(); +// author == doc.peerIdStr +``` + + + + +```typescript no_run +getLastMoveId(): { peer: PeerID, counter: number } | undefined +``` + + +Returns the OpID of the most recent move operation for this node, or `undefined` if the node has never been moved. + +**Returns:** Creation/move OpID or `undefined` + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +const lastMove = node.getLastMoveId(); +``` + + + + +```typescript no_run +isDeleted(): boolean +``` + + +Checks if this node has been deleted. + +**Returns:** Boolean deletion status + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const node = tree.createNode(); + +// Usage example: +if (node.isDeleted()) { + console.log("Node is deleted"); +} +``` + + + +### LoroCounter + +A counter CRDT for collaborative numeric values. + + +```typescript no_run +increment(value: number): void +``` + + +Increments the counter. + +**Parameters:** + +- `value` - Amount to increment (default: 1) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const counter = doc.getCounter("counter"); + +counter.increment(5); // +5 +``` + + + + +```typescript no_run +decrement(value: number): void +``` + + +Decrements the counter. + +**Parameters:** + +- `value` - Amount to decrement (default: 1) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const counter = doc.getCounter("counter"); + +counter.decrement(3); // -3 +``` + + + + +```typescript no_run +value: number +``` + + +Gets the current counter value. + +**Returns:** Current numeric value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const counter = doc.getCounter("counter"); + +console.log(`Counter value: ${counter.value}`); +``` + + + +### LoroMovableList + +A list optimized for move operations. Designed for frequent reordering (drag-and-drop) with good behavior under +concurrent moves (concurrent moves resolve to one final position). See [List and Movable List](/docs/tutorial/list). + +**📝 MovableList vs List:** + +- **Use MovableList** for: Drag-and-drop UIs, sortable lists, kanban boards +- **Use List** for: scenarios where the list items don't need to be moved +- **Key difference**: MovableList handles concurrent moves better (no duplicates) and supports set operations, List is more efficient in general. + + +```typescript no_run +move(from: number, to: number): void +``` + + +Moves an element from one position to another. + +**Parameters:** + +- `from` - Source index +- `to` - Target index + +**Example:** + +```typescript +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const movableList = doc.getMovableList("list"); +movableList.push("a"); +movableList.push("b"); +movableList.push("c"); +movableList.push("d"); +movableList.push("e"); + +movableList.move(0, 3); // Move first element to fourth position +``` + + + + +```typescript no_run +set(pos: number, value: Value | Container): void +``` + + +Replaces the value at a position. + +**Parameters:** + +- `pos` - Position to update +- `value` - New value + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const movableList = doc.getMovableList("list"); +movableList.push("a", "b", "c"); + +movableList.set(0, "Updated value"); +``` + + + + +```typescript no_run +setContainer(pos: number, container: T): T +``` + + +Replaces the value with a container. + +**Parameters:** + +- `pos` - Position to update +- `container` - New container + +**Returns:** The set container + +**Example:** + +```ts threeslash +import { LoroDoc, LoroText } from "loro-crdt"; +const doc = new LoroDoc(); +const movableList = doc.getMovableList("list"); +movableList.push("placeholder"); + +const text = movableList.setContainer(0, new LoroText()); +``` + + + +## Synchronization + +Import/export updates over any transport and choose the right encoding for speed and size. See [Sync Tutorial](/docs/tutorial/sync) and [Encoding & Export Modes](/docs/tutorial/encoding). + +- **Import order**: Loro handles out-of-order updates automatically +- **Auto-commit**: Import and export operations trigger automatic commits + +### Import/Export Patterns + +#### Basic Synchronization + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +// Peer A: Export updates +const doc1 = new LoroDoc(); +doc1.getText("text").insert(0, "Hello"); +const updates = doc1.export({ mode: "update" }); + +// Peer B: Import updates +const doc2 = new LoroDoc(); +doc2.import(updates); +// now doc2.getText("text").toString() === "Hello" +``` + +#### Continuous Sync + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc1 = new LoroDoc(); +const doc2 = new LoroDoc(); + +// Usage example: +// Set up bidirectional sync +doc1.subscribeLocalUpdates((updates) => { + doc2.import(updates); +}); + +doc2.subscribeLocalUpdates((updates) => { + doc1.import(updates); +}); +``` + +Performance tips: + +- Prefer `mode: "update"` with a `VersionVector` to sync incrementally. +- Use `mode: "shallow-snapshot"` when you only need current state; it strips history for faster import/load. +- Loro’s LSM-based encoding and Eg-walker-inspired merge keep import/export fast, even for large histories. + See [Encoding](/docs/tutorial/encoding) and v1.0 performance notes: https://loro.dev/blog/v1.0 + +#### Network Sync with WebSocket + +```ts no_run threeslash +import { LoroDoc } from "loro-crdt"; + +// Assume we have: +declare const ws: { + send: (data: Uint8Array) => void; + on: (event: string, handler: (data: any) => void) => void; +}; + +// Client side +const doc = new LoroDoc(); + +// Send local updates to server +doc.subscribeLocalUpdates((updates) => { + ws.send(updates); +}); + +// Receive updates from server +ws.on("message", (data) => { + doc.import(new Uint8Array(data)); +}); +``` + +### Shallow Snapshots + +Shallow snapshots allow for efficient storage by garbage collecting deleted operations. + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Create shallow snapshot +const frontiers = doc.frontiers(); +const shallowSnapshot = doc.export({ + mode: "shallow-snapshot", + frontiers: frontiers, +}); + +// Import shallow snapshot +const newDoc = new LoroDoc(); +newDoc.import(shallowSnapshot); +``` + +--- + +## Version Control + +### Time Travel + +Navigate through document history: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Save current version +const v1 = doc.frontiers(); + +// Make changes +doc.getText("text").insert(0, "New text"); + +// Save new version +const v2 = doc.frontiers(); + +// Travel back +doc.checkout(v1); +console.log(doc.getText("text").toString()); // Original text + +// Travel forward +doc.checkout(v2); +console.log(doc.getText("text").toString()); // New text + +// Return to latest +doc.checkoutToLatest(); +``` + +### Forking + +Create document branches: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Fork at current state +const fork1 = doc.fork(); +fork1.getText("text").insert(0, "Fork 1 changes"); + +// Fork at specific version +const historicalVersion = doc.frontiers(); +const fork2 = doc.forkAt(historicalVersion); +fork2.getText("text").insert(0, "Fork 2 changes"); + +// Original document remains unchanged +``` + +### Version Vectors + +Track document versions: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const vv = doc.version(); +console.log(`Document has ${vv.length()} peers`); +``` + +--- + +## Events & Subscriptions + +### Event Structure + +```typescript no_run +interface LoroEventBatch { + by: "local" | "import" | "checkout"; + origin?: string; + currentTarget?: ContainerID; + events: LoroEvent[]; + from: Frontiers; + to: Frontiers; +} + +interface LoroEvent { + target: ContainerID; + diff: Diff; + path: Path; +} +``` + +### Diff Types + +Different containers produce different diff types: + +#### TextDiff + + +```typescript no_run +type TextDiff = { + type: "text" + diff: Delta[] +} +``` +Represents changes to text content using Delta format. + +**Properties:** + +- `type` - Always "text" for text diffs +- `diff` - Array of Delta operations (insert, delete, retain) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const text = doc.getText("text"); + +// Example +text.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "text") { + event.diff.diff.forEach((delta) => { + if (delta.insert) { + console.log(`Inserted: "${delta.insert}"`); + } + if (delta.delete) { + console.log(`Deleted ${delta.delete} characters`); + } + }); + } + } +}); +``` + + + +#### ListDiff + + +```typescript no_run +type ListDiff = { + type: "list" + diff: Delta<(Value | Container)[]>[] +} +``` +Represents changes to list content using Delta format. + +**Properties:** + +- `type` - Always "list" for list diffs +- `diff` - Array of Delta operations on list items + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const list = doc.getList("list"); + +// Example +list.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "list") { + event.diff.diff.forEach((delta) => { + if (delta.insert) { + console.log(`Inserted items:`, delta.insert); + } + }); + } + } +}); +``` + + + +#### MapDiff + + +```typescript no_run +type MapDiff = { + type: "map" + updated: Record +} +``` +Represents changes to map content. + +**Properties:** + +- `type` - Always "map" for map diffs +- `updated` - Record of key-value changes (undefined means deleted) + +**Example:** + +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +// Example +map.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "map") { + Object.entries(event.diff.updated).forEach(([key, value]) => { + if (value === undefined) { + console.log(`Deleted key: ${key}`); + } else { + console.log(`Updated key: ${key} = ${value}`); + } + }); + } + } +}); +``` + + + +#### TreeDiff + + +```typescript no_run +type TreeDiff = { + type: "tree" + diff: TreeDiffItem[] +} + +type TreeDiffItem = +| { +target: TreeID +action: "create" +parent: TreeID | undefined +index: number +fractionalIndex: string +} +| { +target: TreeID +action: "delete" +oldParent: TreeID | undefined +oldIndex: number +} +| { +target: TreeID +action: "move" +parent: TreeID | undefined +index: number +fractionalIndex: string +oldParent: TreeID | undefined +oldIndex: number +} + +```` +Represents changes to tree structure. + +**Properties:** +- `type` - Always "tree" for tree diffs +- `diff` - Array of TreeDiffItem operations (create, delete, move) + +**TreeDiffItem Actions:** +- `create` - Node creation with parent and position +- `delete` - Node deletion with old parent and position +- `move` - Node movement with old and new positions + +**Example:** +```ts threeslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); + +// Example +tree.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "tree") { + event.diff.diff.forEach(item => { + switch (item.action) { + case "create": + console.log(`Created node ${item.target}`); + break; + case "move": + console.log(`Moved node ${item.target}`); + break; + case "delete": + console.log(`Deleted node ${item.target}`); + break; + } + }); + } + } +}); +```` + + + +### Deep Subscription + +Subscribe to nested changes: + +```ts threeslash +import { LoroDoc } from "loro-crdt"; + +const doc = new LoroDoc(); +// Subscribe to specific container +const text = doc.getText("text"); +text.subscribe((event) => { + console.log("Text changed:", event); +}); + +// Subscribe with deep observation +doc.subscribe((event) => { + // Path shows the location of the change + event.events.forEach((e) => { + console.log("Change path:", e.path); + console.log("Container:", e.target); + console.log("Diff:", e.diff); + }); +}); +``` + +--- + +## Undo/Redo + +Local undo operates on your own changes without breaking collaboration. See [Undo/Redo](/docs/advanced/undo) for design details and caveats. + +### UndoManager + +Provides local undo/redo functionality. + +**⚠️ Important Notes:** + +- **Local-only**: UndoManager only undoes the local user's operations, not remote operations +- **Origin filtering**: Use `excludeOriginPrefixes` to exclude certain operations (e.g., sync operations) from undo +- **Cursor restoration**: Use `onPush`/`onPop` callbacks to save and restore cursor positions + + +```typescript no_run +constructor(doc: LoroDoc, config: UndoConfig) +``` + + +Creates a new UndoManager instance. + +**Parameters:** + +- `doc` - The LoroDoc to manage undo/redo for +- `config` - Configuration options + - `mergeInterval?` - Time in ms to merge consecutive operations (default: 1000) + - `maxUndoSteps?` - Maximum number of undo steps to keep (default: 100) + - `excludeOriginPrefixes?` - Array of origin prefixes to exclude from undo + - `onPush?` - Callback when adding to undo stack + - `onPop?` - Callback when undoing/redoing + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; + +const doc = new LoroDoc(); +const undo = new UndoManager(doc, { + mergeInterval: 1000, + maxUndoSteps: 100, + excludeOriginPrefixes: ["sync-"], +}); +``` + + + + +```typescript no_run +undo(): boolean +``` + + +Undo the last operation. + +**Returns:** True if undo was successful + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); +text.insert(0, "Hello"); +doc.commit(); + +// Usage example: +const success = undo.undo(); +console.log(success); // true +``` + + + + +```typescript no_run +redo(): boolean +``` + + +Redo the last undone operation. + +**Returns:** True if redo was successful + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); +text.insert(0, "Hello"); +doc.commit(); +undo.undo(); + +// Usage example: +const success = undo.redo(); +console.log(success); // true +``` + + + + +```typescript no_run +canUndo(): boolean +``` + + +Check if undo is available. + +**Returns:** True if can undo + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +if (undo.canUndo()) { + undo.undo(); +} +``` + + + + +```typescript no_run +canRedo(): boolean +``` + + +Check if redo is available. + +**Returns:** True if can redo + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +if (undo.canRedo()) { + undo.redo(); +} +``` + + + + +```typescript no_run +peer(): PeerID +``` + + +Get the peer ID of the undo manager. + +**Returns:** The peer ID + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +const peerId = undo.peer(); +console.log(peerId); // e.g., "123456" +``` + + + + +```typescript no_run +setMaxUndoSteps(steps: number): void +``` + + +Set the maximum number of undo steps. + +**Parameters:** + +- `steps` - Maximum number of undo steps + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.setMaxUndoSteps(50); +``` + + + + +```typescript no_run +setMergeInterval(interval: number): void +``` + + +Set the merge interval for grouping operations. + +**Parameters:** + +- `interval` - Merge interval in milliseconds + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.setMergeInterval(2000); // 2 seconds +``` + + + + +```typescript no_run +groupStart(): void +``` + + +Begin a manual grouping of subsequent commits into a single undo step. + +**Behavior:** + +- Wrap consecutive `doc.commit()` calls so they undo together +- Calling `groupStart` again before `groupEnd` throws and leaves the current group unchanged +- Conflicting remote imports may automatically end the group and split the undo item + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; + +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); + +undo.groupStart(); + +text.update("hello", undefined); +doc.commit(); + +text.update("hello world", undefined); +doc.commit(); + +undo.groupEnd(); + +undo.undo(); + +console.log(text.toString()); // "" +``` + + + + +```typescript no_run +groupEnd(): void +``` + + +Close the active manual group and enqueue the grouped operations as a single undo item. + +**Behavior:** + +- Must be paired with a prior `groupStart` +- Safe to call after the group was auto-closed by a conflicting remote import (becomes a no-op) +- Non-conflicting remote updates from other peers remain outside the undo item but do not break the group + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; + +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); +const text = doc.getText("text"); + +undo.groupStart(); + +text.update("hello", undefined); +doc.commit(); +text.update("hello world", undefined); +doc.commit(); + +const snapshot = doc.export({ mode: "snapshot" }); +const doc2 = new LoroDoc(); +doc2.import(snapshot); +doc2.getText("text2").update("hello world world", undefined); // touches a different container +doc2.commit(); +const update = doc2.export({ mode: "update" }); + +doc.import(update); // remote, non-conflicting change + +text.update("hello world world world", undefined); +doc.commit(); + +undo.groupEnd(); + +undo.undo(); + +console.log(text.toString()); // "" +``` + + + + +```typescript no_run +addExcludeOriginPrefix(prefix: string): void +``` + + +Add a prefix to exclude from undo stack. + +**Parameters:** + +- `prefix` - Origin prefix to exclude + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.addExcludeOriginPrefix("sync-"); +undo.addExcludeOriginPrefix("import-"); +``` + + + + +```typescript no_run +clear(): void +``` + + +Clear the undo and redo stacks. + +**Example:** + +```ts threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +const undo = new UndoManager(doc, {}); + +// Usage example: +undo.clear(); +``` + + + +### Custom Undo Handlers + +Handle cursor restoration and side effects: + +```typescript no_run threeslash +import { LoroDoc, UndoManager } from "loro-crdt"; +const doc = new LoroDoc(); +declare function saveCursorPositions(): any; +declare function restoreCursorPositions(cursors: any): void; + +// Usage example: +const undo = new UndoManager(doc, { + onPush: (isUndo, counterRange, event) => { + // Save cursor positions when adding to undo stack + const cursors = saveCursorPositions(); + return { + value: doc.toJSON(), + cursors: cursors, + }; + }, + + onPop: (isUndo, { value, cursors }, counterRange) => { + // Restore cursor positions when undoing + restoreCursorPositions(cursors); + }, +}); +``` + +--- + +## Types & Interfaces + +Reference for core types used across the API. For conceptual background, see [Containers](/docs/concepts/container), [Version Vector](/docs/concepts/version_vector), [Frontiers](/docs/concepts/frontiers), and the [Versioning Deep Dive](/docs/advanced/version_deep_dive). + +### Core Types + +```typescript no_run +// Peer identifier +type PeerID = `${number}`; + +// Container identifier +type ContainerID = + | `cid:root-${string}:${ContainerType}` + | `cid:${number}@${PeerID}:${ContainerType}`; + +// Tree node identifier +type TreeID = `${number}@${PeerID}`; + +// Operation identifier +type OpId = { + peer: PeerID; + counter: number; +}; + +// Container types +type ContainerType = + | "Text" + | "Map" + | "List" + | "Tree" + | "MovableList" + | "Counter"; + +// Value types +type Value = + | ContainerID + | string + | number + | boolean + | null + | { [key: string]: Value } + | Uint8Array + | Value[] + | undefined; +``` + +### Version Types + +Loro uses two complementary version representations: Version Vectors (per-peer counters) and Frontiers (a compact set of heads). See [Version Vector](/docs/concepts/version_vector) and [Frontiers](/docs/concepts/frontiers). For the full DAG model, see [Version Deep Dive](/docs/advanced/version_deep_dive). + +```typescript no_run +// Version vector class +class VersionVector { + constructor( + value: Map | Uint8Array | VersionVector | undefined | null, + ); + static parseJSON(version: Map): VersionVector; + toJSON(): Map; + encode(): Uint8Array; + static decode(bytes: Uint8Array): VersionVector; + get(peer_id: number | bigint | `${number}`): number | undefined; + compare(other: VersionVector): number | undefined; + setEnd(id: { peer: PeerID; counter: number }): void; + setLast(id: { peer: PeerID; counter: number }): void; + remove(peer: PeerID): void; + length(): number; +} + +// Frontiers represent a specific version +type Frontiers = OpId[]; + +// ID span for range queries +type IdSpan = { + peer: PeerID; + counter: number; + length: number; +}; +``` + +### Change Types + +```typescript no_run +// Change metadata +interface Change { + peer: PeerID; + counter: number; + lamport: number; + length: number; + timestamp: number; // Unix timestamp in seconds + deps: OpId[]; + message: string | undefined; +} + +// Change modifier for pre-commit hooks +interface ChangeModifier { + setMessage(message: string): this; + setTimestamp(timestamp: number): this; +} +``` + +### Cursor Types + +Stable cursors survive concurrent edits and resolve to absolute positions on demand. See [Cursor and Stable Positions](/docs/concepts/cursor_stable_positions) and [Cursor tutorial](/docs/tutorial/cursor). + +```typescript no_run +// Stable position in containers +class Cursor { + containerId(): ContainerID; + pos(): OpId | undefined; + side(): Side; // -1 | 0 | 1 + encode(): Uint8Array; + static decode(data: Uint8Array): Cursor; +} + +// Cursor side affinity +type Side = -1 | 0 | 1; +``` + +### Delta Type + +Delta is a popular rich-text operation format (e.g., Quill). LoroText can export/import Delta; see [Text](/docs/tutorial/text). + +```typescript no_run +// Rich text delta operations +type Delta = + | { + insert: T; + attributes?: { [key in string]: {} }; + retain?: undefined; + delete?: undefined; + } + | { + delete: number; + attributes?: undefined; + retain?: undefined; + insert?: undefined; + } + | { + retain: number; + attributes?: { [key in string]: {} }; + delete?: undefined; + insert?: undefined; + }; +``` + +--- + +## Utility Functions + +Small helpers for type checks and IDs. See [Container IDs](/docs/advanced/cid) and [Versioning Deep Dive](/docs/advanced/version_deep_dive) for frontiers/version encoding. + +### Frontier Encoding + + +```typescript no_run +encodeFrontiers(frontiers: OpId[]): Uint8Array +``` + + +Encode frontiers for efficient transmission. + +**Parameters:** + +- `frontiers` - Array of operation IDs representing frontiers + +**Returns:** Encoded bytes + +**Example:** + +```ts threeslash +import { LoroDoc, encodeFrontiers } from "loro-crdt"; + +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +const encoded = encodeFrontiers(frontiers); +// Send encoded to remote peers +``` + + + + +```typescript no_run +decodeFrontiers(bytes: Uint8Array): OpId[] +``` + + +Decode frontiers from bytes. + +**Parameters:** + +- `bytes` - Encoded frontier bytes + +**Returns:** Array of operation IDs + +**Example:** + +```typescript no_run threeslash +import { decodeFrontiers } from "loro-crdt"; + +declare const encodedData: Uint8Array; +const frontiers = decodeFrontiers(encodedData); +console.log(frontiers); // [{ peer: "1", counter: 10 }, ...] +``` + + + +### Debugging + + +```typescript no_run +setDebug(): void +``` + + +Enable debug mode for detailed logging. + +**Example:** + +```ts threeslash +import { setDebug } from "loro-crdt"; + +// Enable debug logging +setDebug(); +``` + + + + +```typescript no_run +LORO_VERSION(): string +``` + + +Get the current Loro version. + +**Returns:** Version string + +**Example:** + +```ts threeslash +import { LORO_VERSION } from "loro-crdt"; + +const version = LORO_VERSION(); +console.log("Loro version:", version); +``` + + + +### Import Blob Metadata + + +```typescript no_run +decodeImportBlobMeta(blob: Uint8Array, check_checksum: boolean): ImportBlobMetadata +``` + + +Decode metadata from an import blob. + +**Parameters:** + +- `blob` - The import blob bytes +- `check_checksum` - Whether to verify checksum + +**Returns:** Import blob metadata + +**Example:** + +```typescript no_run threeslash +import { decodeImportBlobMeta } from "loro-crdt"; + +declare const blob: Uint8Array; +const metadata = decodeImportBlobMeta(blob, true); +console.log("Blob metadata:", metadata); +``` + + + +--- + +## EphemeralStore + +Manages ephemeral state like cursor positions and user presence. See [Ephemeral Store](/docs/tutorial/ephemeral) for concepts and usage patterns. Each entry uses timestamp-based LWW (Last-Write-Wins) for conflict resolution. + +**⚠️ Important:** + +- EphemeralStore is a separate CRDT without history - history/operations are NOT persisted +- Perfect for temporary state: cursor positions, selections, typing indicators +- Each peer's state auto-expires after the timeout period +- Uses Last-Write-Wins + +### EphemeralStore + + +```typescript no_run +constructor(timeout?: number) +``` + + +Creates a new EphemeralStore instance. + +**Parameters:** + +- `timeout` - Duration in milliseconds. A peer's state is considered outdated if its last update is older than this timeout. Default is 30000ms (30 seconds). + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; + +// Create ephemeral store with 30 second timeout +const store = new EphemeralStore(30000); +``` + + + + +```typescript no_run +set(key: K, value: T[K]): void +``` + + +Set an ephemeral value. + +**Parameters:** + +- `key` - The key to set +- `value` - The value to store + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +store.set("cursor", { line: 10, column: 5 }); +store.set("selection", { start: 0, end: 10 }); +store.set("user", { name: "Alice", color: "#ff0000" }); +``` + + + + +```typescript no_run +get(key: K): T[K] | undefined +``` + + +Get an ephemeral value. + +**Parameters:** + +- `key` - The key to get + +**Returns:** The stored value or undefined + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); + +// Usage example: +const cursor = store.get("cursor"); +console.log(cursor); // { line: 10 } +``` + + + + +```typescript no_run +delete(key: K): void +``` + + +Delete an ephemeral value. + +**Parameters:** + +- `key` - The key to delete + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); + +// Usage example: +store.delete("cursor"); +``` + + + + +```typescript no_run +getAllStates(): Partial +``` + + +Get all ephemeral states. + +**Returns:** All stored states + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); +store.set("user", { name: "Alice" }); + +// Usage example: +const allStates = store.getAllStates(); +console.log(allStates); +``` + + + + +```typescript no_run +encode(key: K): Uint8Array +``` + + +Encode a specific key's state for transmission. + +**Parameters:** + +- `key` - The key to encode + +**Returns:** Encoded bytes + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); + +// Usage example: +const encoded = store.encode("cursor"); +// Send encoded to remote peers +``` + + + + +```typescript no_run +encodeAll(): Uint8Array +``` + + +Encode all states for transmission. + +**Returns:** Encoded bytes + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); +store.set("user", { name: "Alice" }); + +// Usage example: +const encoded = store.encodeAll(); +// Send encoded to remote peers +``` + + + + +```typescript no_run +apply(bytes: Uint8Array): void +``` + + +Apply remote updates. + +**Parameters:** + +- `bytes` - Encoded updates from remote peer + +**Example:** + +```typescript no_run threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +declare const remoteData: Uint8Array; +store.apply(remoteData); +``` + + + + +```typescript no_run +keys(): string[] +``` + + +Get all keys in the store. + +**Returns:** Array of keys + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); +store.set("cursor", { line: 10 }); +store.set("user", { name: "Alice" }); + +// Usage example: +const allKeys = store.keys(); +console.log(allKeys); // ["cursor", "user"] +``` + + + + +```typescript no_run +subscribe(listener: EphemeralListener): () => void +``` + + +Subscribe to all ephemeral state changes. + +**Parameters:** + +- `listener` - Callback function for state changes + +**Returns:** Unsubscribe function + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +const unsubscribe = store.subscribe((event) => { + console.log("Ephemeral state changed:", event); +}); + +// Later, unsubscribe +unsubscribe(); +``` + + + + +```typescript no_run +subscribeLocalUpdates(listener: EphemeralLocalListener): () => void +``` + + +Subscribe to local ephemeral updates for syncing to remote peers. + +**Parameters:** + +- `listener` - Callback function that receives encoded updates + +**Returns:** Unsubscribe function + +**Example:** + +```typescript no_run threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +declare const websocket: { send: (data: Uint8Array) => void }; + +const unsubscribe = store.subscribeLocalUpdates((data) => { + // Send to remote peers + websocket.send(data); +}); + +// Later, unsubscribe +unsubscribe(); +``` + + + + +```typescript no_run +destroy(): void +``` + + +Clean up and destroy the ephemeral store. + +**Example:** + +```ts threeslash +import { EphemeralStore } from "loro-crdt"; +const store = new EphemeralStore(30000); + +// Usage example: +store.destroy(); +``` + + + +### Complete Example + +```typescript no_run threeslash +import { EphemeralStore } from "loro-crdt"; + +// Assume we have: +declare const websocket: { + send: (data: Uint8Array) => void; + on: (event: string, handler: (data: any) => void) => void; +}; + +const store = new EphemeralStore(30000); +const store2 = new EphemeralStore(30000); + +// Subscribe to local updates +store.subscribeLocalUpdates((data) => { + store2.apply(data); +}); + +// Subscribe to all updates +store2.subscribe((event) => { + console.log("event:", event); +}); + +// Set a value +store.set("key", "value"); + +// Encode the value +const encoded = store.encode("key"); + +// Apply the encoded value +store2.apply(encoded); +``` + +--- + +
+ + +# FILE: pages/docs/tutorial/counter.mdx + +--- +keywords: "counter crdt" +description: "how to use loro counter crdt and show all APIs of loro counter crdt." +--- + +# Counter + +Loro's Counter will add up all the applied values, and supports integers and floating point numbers. + +Here is how to use it: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +const counter = doc.getCounter("counter"); +counter.increment(1); +counter.increment(2); +counter.decrement(1); +expect(counter.value).toBe(2); +``` + + +# FILE: pages/docs/tutorial/ephemeral.mdx + +--- +keywords: "ephemeral, awareness, presence, collaborative" +description: "How to use Loro's ephemeral store feature to implement user awareness and online status management in real-time collaboration." +--- + +# Ephemeral Store + +In real-time collaborative scenarios, Presence information is just as important as maintaining document consistency across peers through CRDTs. This includes information such as the current collaborator's username, mouse pointer position, or selected objects. We need a mechanism that doesn't persist in the CRDT Document but remains ephemeral, allowing collaborators to perceive each other's presence for better coordination and to avoid conflicts when multiple users edit the same object. This is why we've introduced the Ephemeral Store. + +![](/images/ephemeral.png) + +Since Ephemeral information is primarily used for real-time collaboration, we've chosen a simple yet effective approach. The Ephemeral Store is a timestamp-based, last-write-wins key-value store. Each entry maintains its own timestamp of the last update, enabling the system to send only the updated entry content rather than the complete current state. + +## Example + +```ts twoslash +import { EphemeralStore, EphemeralStoreEvent } from "loro-crdt"; +import { expect } from "expect"; + +const store = new EphemeralStore(); +// Set ephemeral data +store.set("loro-prosemirror", { + anchor: { pos: 0 }, + focus: { pos: 5 }, + user: "Alice", +}); +store.set("online-users", ["Alice", "Bob"]); + +expect(store.get("online-users")).toEqual(["Alice", "Bob"]); +// Encode only the data for `loro-prosemirror` +const encoded = store.encode("loro-prosemirror"); + +store.subscribe((e: EphemeralStoreEvent) => { + // Listen to changes from `local`, `remote`, or `timeout` events +}); +``` + +## API + +- `constructor(timeout)`: + Creates a new EphemeralStore instance with an optional timeout parameter (default: 30000ms). The timeout determines how long ephemeral data remains valid before being automatically removed. + +- `set(key, value)`: + Sets a value for the specified key in the ephemeral store. If the key already exists, its value will be updated. + +- `get(key)`: + Retrieves the current value for the specified key, or returns `undefined` if the key doesn't exist. + +- `delete(key)`: + Removes the specified key and its associated value from the ephemeral store. + +- `getAllStates()`: + Returns all current key-value pairs in the ephemeral store. + +- `keys()`: + Returns an array of all keys currently in the ephemeral store. + +- `encode(key)`: + Encodes the value associated with the specified key into a binary format that can be transmitted to other peers. + +- `encodeAll()`: + Encodes all key-value pairs in the ephemeral store into a binary format. + +- `apply(bytes)`: + Applies encoded ephemeral data received from other peers to the local ephemeral store. + +- `subscribe((event: EphemeralStoreEvent)=>void)`: + Registers a listener function that will be called whenever the ephemeral store is updated, either from local changes, remote changes, or timeout events. + + ```ts + interface EphemeralStoreEvent { + // The source of the event: local changes, imported from remote, or timeout expiration + by: "local" | "import" | "timeout"; + // Array of keys that were newly added + added: string[]; + // Array of keys that had their values updated + updated: string[]; + // Array of keys that were removed + removed: string[]; + } + ``` + +- `subscribeLocalUpdates((bytes: Uint8Array) => void)`: + Registers a listener that will be called only for local updates to the ephemeral store. + + ```ts + // you need maintain the Subscription to avoid gc + const _sub1 = ephemeral1.subscribeLocalUpdates((update) => { + ephemeral2.apply(update); + }); + + const _sub2 = ephemeral2.subscribeLocalUpdates((update) => { + ephemeral1.apply(update); + }); + ``` + + +# FILE: pages/docs/tutorial/loro_doc.mdx + +# Getting Started with LoroDoc + +LoroDoc is the main entry point for almost all Loro functionality. It serves as a container manager and coordinator that provides: + +1. **Container Management**: Create and manage different types of CRDT containers (Text, List, Map, Tree, MovableList) +2. **Version Control**: Track document history, checkout versions, and manage branches +3. **Event System**: Subscribe to changes at both document and container levels +4. **Import/Export**: Save and load documents/updates in various formats + +## Basic Usage + +First, let's create a new LoroDoc instance: + +```ts twoslash +import { + LoroDoc, + LoroText, + LoroList, + LoroMap, + LoroTree, + LoroMovableList, +} from "loro-crdt"; +// ---cut--- +// Create a new document with a random peer ID +const doc = new LoroDoc(); + +// Or set a specific peer ID +doc.setPeerId("1"); + +// Create containers +const text = doc.getText("text"); +const list = doc.getList("list"); +const map = doc.getMap("map"); +const tree = doc.getTree("tree"); +const movableList = doc.getMovableList("tasks"); +``` + +To model a document with the following format: + +```json +{ + "meta": { + "title": "Document Title", + "createdBy": "Author" + }, + "content": "Article", + "comments": [ + { + "user": "userId", + "comment": "comment" + } + ] +} +``` + +```ts twoslash +import { LoroDoc, LoroMap } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const meta = doc.getMap("meta"); +meta.set("title", "Document Title"); +meta.set("createdBy", "Author"); +doc.getText("content").insert(0, "Article"); +const comments = doc.getList("comments"); +const comment1 = comments.insertContainer(0, new LoroMap()); +comment1.set("user", "userId"); +comment1.set("comment", "comment"); +``` + +## Supported Data Types (LoroValue) + +Loro supports storing various data types in its containers. The `LoroValue` type represents all possible values that can be stored in a LoroDoc: + +```ts +type LoroValue = + | null // Null values + | boolean // true or false + | number // Numeric values (both integer and floating point) + | string // Text values + | Uint8Array // Binary data + | LoroValue[] // Arrays (nested) + | { [key: string]: LoroValue } // Objects/Maps (nested) +``` + +### Examples of Storing Different Data Types + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const map = doc.getMap("data"); + +// Primitive types +map.set("name", "Alice"); // string +map.set("age", 30); // integer number +map.set("score", 95.5); // floating point number +map.set("isActive", true); // boolean +map.set("metadata", null); // null + +// Binary data +const binaryData = new Uint8Array([1, 2, 3, 4, 5]); +map.set("bytes", binaryData); // Uint8Array + +// Nested objects (plain JavaScript objects) +map.set("profile", { + email: "alice@example.com", + settings: { + theme: "dark", + notifications: true + } +}); + +// Arrays +map.set("tags", ["javascript", "typescript", "loro"]); + +// Mixed nested structures +map.set("complex", { + items: [1, 2, { nested: true }], + binary: new Uint8Array([255, 128]), + active: false +}); +``` + +Note: When you need to store a CRDT container (like LoroText, LoroList, etc.) within another container, use `setContainer()` or `insertContainer()` methods instead of regular `set()` or `insert()`. This creates a proper sub-container relationship that maintains CRDT properties. If a child container under a Map may be created lazily by multiple peers at the same key, use the `ensureMergeable*` methods instead; see [Containers](/docs/concepts/container) and [Composing CRDTs](/docs/tutorial/composition). + +## Container Types + +LoroDoc supports several container types: + +1. **Text** - For rich text editing +2. **List** - For ordered collections +3. **Map** - For key-value pairs +4. **Tree** - For hierarchical data structures +5. **MovableList** - For lists with movable items + +Let's look at how to use each type: + +### Text Container + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello"); +text.insert(5, " World!"); +console.log(text.toString()); // "Hello World!" + +// Rich text support +doc.configTextStyle({ + bold: { expand: "after" }, + link: { expand: "none" }, +}); +text.mark({ start: 0, end: 5 }, "bold", true); +``` + +### List Container + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const list = doc.getList("list"); +list.insert(0, "first"); +list.insert(1, "second"); +console.log(list.toArray()); // ["first", "second"] + +// Nested containers +const nestedText = list.insertContainer(2, new LoroText()); +nestedText.insert(0, "nested text"); +``` + +### Map Container + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const map = doc.getMap("map"); +map.set("name", "John"); +map.set("age", 30); +console.log(map.get("name")); // "John" + +// Nested containers +const userText = map.setContainer("bio", new LoroText()); +userText.insert(0, "Software Engineer"); +``` + +### Tree Container + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const tree = doc.getTree("tree"); +const root = tree.createNode(); +root.data.set("name", "Root"); + +const child1 = root.createNode(); +child1.data.set("name", "Child 1"); + +const child2 = root.createNode(); +child2.data.set("name", "Child 2"); +``` + +### MovableList Container + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const movableList = doc.getMovableList("tasks"); +movableList.insert(0, "Task 1"); +movableList.insert(1, "Task 2"); +movableList.move(0, 1); // Move Task 1 after Task 2 +``` + +## Collaboration Features + +LoroDoc can be used for real-time collaboration. Here's how to sync changes between peers: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +declare const otherDoc: LoroDoc; +// ---cut--- +// First peer +const doc1 = new LoroDoc(); +doc1.setPeerId("1"); +const text1 = doc1.getText("text"); + +// Second peer +const doc2 = new LoroDoc(); +doc2.setPeerId("2"); +const text2 = doc2.getText("text"); + +// Set up two-way sync +doc1.subscribeLocalUpdates((updates) => { + doc2.import(updates); +}); + +doc2.subscribeLocalUpdates((updates) => { + doc1.import(updates); +}); + +// Now changes in doc1 will be reflected in doc2 and vice versa +text1.insert(0, "Hello"); +doc1.commit(); +await Promise.resolve(); // await for the event to be emitted +text2.insert(5, " World!"); +doc2.commit(); +``` + +## Undo/Redo Support + +Loro provides built-in undo/redo functionality: + +```ts twoslash +import { UndoManager, LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const undoManager = new UndoManager(doc, { + maxUndoSteps: 100, + mergeInterval: 1000, +}); + +const text = doc.getText("text"); +// Make some changes +text.insert(0, "Hello"); +doc.commit(); + +// Undo the changes +if (undoManager.canUndo()) { + undoManager.undo(); +} + +// Redo the changes +if (undoManager.canRedo()) { + undoManager.redo(); +} +``` + +## Exporting and Importing + +You can save and load the document state: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Export the document +const snapshot = doc.export({ mode: "snapshot" }); + +// Create a new document from the snapshot +const newDoc = LoroDoc.fromSnapshot(snapshot); + +const doc2 = new LoroDoc(); +// Or import into an existing document +doc2.import(snapshot); +``` + +### Shallow Import/Export + +Shallow import/export is a feature that allows you to create and share document snapshots without including the complete history. This is particularly useful for: + +1. Reducing the size of exported data +2. Sharing the document with others without revealing the complete history +3. Speedup the import/export process + +Here's how to use shallow export: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Export a shallow snapshot that only include the history since `doc.oplogFrontiers()` +// It works like `git clone --depth=1`, where the exported data only contain the most recent ops. +const shallowSnapshot = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.oplogFrontiers(), +}); + +// Check if a document is shallow +const isShallow = doc.isShallow(); + +// Get the version since which the history is available +const sinceVersion = doc.shallowSinceVV(); +// Or get it in frontiers format +const sinceFrontiers = doc.shallowSinceFrontiers(); +``` + +Note: A shallow document only contains history after a certain version point. Operations before the shallow start point are not included, but the document remains fully functional for collaboration. + +### Redacting Sensitive Content + +Loro allows you to redact specific segments of document history while preserving the rest. This is particularly useful when: + +1. A user accidentally pastes sensitive information (like passwords or API keys) into the document +2. You need to remove just the sensitive part of the history while keeping older and newer edits intact +3. You want to share document history with sensitive segments sanitized + +Here's how to use the redaction functionality: + +```ts no_run twoslash +import { LoroDoc, redactJsonUpdates } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("1"); + +// Create some content to be redacted +const text = doc.getText("text"); +text.insert(0, "Sensitive information"); +doc.commit(); + +const map = doc.getMap("map"); +map.set("password", "secret123"); +map.set("public", "public information"); +doc.commit(); + +// Export JSON updates +const jsonUpdates = doc.exportJsonUpdates(); + +// Define version range to redact (redact the text content) +const versionRange = { + "1": [0, 21], // Redact the "Sensitive information" +}; + +// Apply redaction +const redactedJson = redactJsonUpdates(jsonUpdates, versionRange); + +// Create a new document with redacted content +const redactedDoc = new LoroDoc(); +redactedDoc.importJsonUpdates(redactedJson); + +// The text content is now redacted with replacement characters +console.log(redactedDoc.getText("text").toString()); +// Outputs: "���������������������" + +// Map operations after the redacted range remain intact +console.log(redactedDoc.getMap("map").get("password")); // "secret123" +console.log(redactedDoc.getMap("map").get("public")); // "public information" +``` + +Redaction applies these rules to preserve document structure while removing sensitive content: + +- Preserves delete and move operations +- Replaces text insertion content with Unicode replacement characters '�' +- Substitutes list and map insert values with null +- Maintains structure of nested containers +- Replaces text mark values with null +- Preserves map keys and text annotation keys + +Note that redaction doesn't remove the operations completely - it just replaces the sensitive content with placeholders. If you need to completely remove portions of history, see the section on shallow snapshots in the [Tips](./tips.md) section. + +#### Important: Synchronization Considerations + +Both redaction and shallow snapshots maintain future synchronization consistency, but your application is responsible for ensuring all peers get the sanitized version. Otherwise, old instances of the document with sensitive information will still exist on other peers. + +## Event Subscription + +Subscribe to changes in the document: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.subscribe((event) => { + console.log("Document changed:", event); +}); + +const text = doc.getText("text"); +// Container-specific subscription +text.subscribe((event) => { + console.log("Text changed:", event); +}); +``` + +### Event Emission + +Events in LoroDoc are emitted only after a transaction is committed, and importantly, the events are emitted after a microtask. This means you need to await a microtask if you want to handle the events immediately after a commit. + +1. Explicitly calling `doc.commit()`: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); + +// Subscribe to changes +doc.subscribe((event) => { + console.log("Change event:", event); +}); + +text.insert(0, "Hello"); // No event emitted yet +doc.commit(); // Event will be emitted after a microtask + +// If you need to wait for the event: +await Promise.resolve(); // Now the event has been emitted +``` + +2. Implicitly through certain operations: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +declare const someData: Uint8Array; +declare const someVersion: any; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); + +// These operations trigger implicit commits: +doc.export({ mode: "snapshot" }); // Implicit commit +doc.import(someData); // Implicit commit +doc.checkout(someVersion); // Implicit commit +``` + +You can also specify additional information when committing: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.commit({ + origin: "user-edit", // Mark the event source + message: "Add greeting", // Like a git commit message + timestamp: Date.now(), // Custom timestamp +}); +await Promise.resolve(); // Wait for event if needed +``` + +Note: Multiple operations before a `commit` are batched into a single event. This helps reduce event overhead and provides atomic changes. The event will still be emitted after a microtask, regardless of whether the commit was explicit or implicit. + +## Version Control and History + +LoroDoc provides powerful version control features that allow you to track and manage document history: + +### Version Representation + +Loro uses two ways to represent versions: + +1. **Version Vector**: A map from peer ID to counter + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Get current version vector +const vv = doc.version(); + +// Get oplog version vector (latest known version) +const oplogVv = doc.oplogVersion(); +``` + +2. **Frontiers**: A list of operation IDs that represent the latest operations from each peer. This is compacter than version vector. In most of the cases, it only has 1 element. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("0"); +doc.getMap("map").set("text", "Hello"); +// Get current frontiers +const frontiers = doc.frontiers(); + +// Get oplog frontiers (latest known version) +const oplogFrontiers = doc.oplogFrontiers(); // { "0": 0 } +``` + +### Checkout and Time Travel + +You can navigate through document history using checkout: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Save current version +const frontiers = doc.frontiers(); +const text = doc.getText("text"); + +// Make some changes +text.insert(0, "Hello World!"); + +// Checkout to previous version +doc.checkout(frontiers); + +// Return to latest version +doc.checkoutToLatest(); +// or +doc.attach(); +``` + +Note: After checkout, the document enters "detached" mode. In this mode: + +- The document is not editable by default +- Import operations are recorded but not applied to the document state +- You need to call `attach()` or `checkoutToLatest()` to go back to the latest version and make it editable again + +### Detached Mode + +The document enters "detached" mode after a `checkout` operation or when explicitly calling `doc.detach()`. In detached mode, the document state is not synchronized with the latest version in the OpLog. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Check if document is in detached mode +console.log(doc.isDetached()); // false + +// Explicitly detach the document +doc.detach(); +console.log(doc.isDetached()); // true + +// Return to attached mode +doc.attach(); +console.log(doc.isDetached()); // false +``` + +By default, editing is disabled in detached mode. However, you can enable it: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Enable editing in detached mode +doc.setDetachedEditing(true); +console.log(doc.isDetachedEditingEnabled()); // true +``` + +#### Key Behaviors in Detached Mode + +1. **Import Operations** + - Operations imported via `doc.import()` are recorded in the OpLog + - These operations are not applied to the document state until checkout + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const oldDoc = new LoroDoc(); +oldDoc.getMap("map").set("name", "John"); +const updates = oldDoc.export({ mode: "update" }); +const doc = new LoroDoc(); +// In detached mode +doc.import(updates); // Updates are stored but not applied +doc.checkoutToLatest(); // Now updates are applied +``` + +2. **Version Management** + - Each checkout uses a different PeerID to prevent conflicts + - The document maintains two version states: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Current state version +const stateVersion = doc.version(); +// Latest known version in OpLog +const latestVersion = doc.oplogVersion(); +``` + +3. **Forking** + - You can create a new document at a specific version: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("0"); +doc.getText("text").insert(0, "Hello"); +// Fork at current frontiers +const forkedDoc = doc.fork(); +// Or fork at specific frontiers +const forkedAtVersion = doc.forkAt([{ peer: "0", counter: 1 }]); +console.log(forkedAtVersion.getText("text").toString()); // "He" +``` + +#### Common Use Cases + +1. **Time Travel and History Review** + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +declare const text: any; +// ---cut--- +const doc = new LoroDoc(); +// Save current version +const frontiers = doc.frontiers(); + +// Make changes +text.insert(0, "New content"); + +// Review previous version +doc.checkout(frontiers); + +// Return to latest version +doc.checkoutToLatest(); +``` + +2. **Branching** + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Enable detached editing +doc.setDetachedEditing(true); + +// Create a branch +const branch = doc.fork(); + +// Make changes in branch +const branchText = branch.getText("text"); +branchText.insert(0, "Branch changes"); +``` + +## Subscription and Sync + +### Local Updates Subscription + +Subscribe to local changes for syncing between peers: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +declare const otherDoc: LoroDoc; +// ---cut--- +const doc = new LoroDoc(); +// Subscribe to local updates +const unsubscribe = doc.subscribeLocalUpdates((updates) => { + // Send updates to other peers + otherDoc.import(updates); +}); + +// Later, unsubscribe when needed +unsubscribe(); +``` + +### Document Events + +Subscribe to all document changes. The event may be triggered by local operations, importing updates, or switching to another version. + +```ts no_run twoslash +import { LoroDoc, LoroEventBatch } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.subscribe((event: LoroEventBatch) => { + console.log("Event triggered by:", event.by); // "local" | "import" | "checkout" + console.log("Event origin:", event.origin); + + for (const e of event.events) { + console.log("Target container:", e.target); + console.log("Path:", e.path); + console.log("Changes:", e.diff); + } +}); +``` + +### Container-specific Events + +Subscribe to changes in specific containers: + +```ts no_run twoslash +import { LoroDoc, LoroEventBatch } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.subscribe((event: LoroEventBatch) => { + // Handle text-specific changes + console.log("Text changed:", event); +}); + +const list = doc.getList("list"); +list.subscribe((event: LoroEventBatch) => { + // Handle list-specific changes + console.log("List changed:", event); +}); +``` + +## Advanced Features + +### Cursor Support + +Loro provides stable cursor position tracking that remains valid across concurrent edits: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "123"); + +// Get cursor at position with side (-1, 0, or 1) +const cursor = text.getCursor(0, 0); +if (cursor) { + // Get current cursor position + const pos = doc.getCursorPos(cursor); + console.log(pos.offset); // Current position + console.log(pos.side); // Cursor side + + // Cursor position updates automatically with concurrent edits + text.insert(0, "abc"); + const newPos = doc.getCursorPos(cursor); + console.log(newPos.offset); // Position updated +} +``` + +### Change Tracking + +Track and analyze document changes: + +```ts twoslash +import { LoroDoc, OpId } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("1"); +doc.getText("text").insert(0, "Hello"); +doc.commit(); +// Get number of changes and operations +console.log(doc.changeCount()); // Number of changes +console.log(doc.opCount()); // Number of operations + +const changes = doc.getAllChanges(); +for (const [peer, peerChanges] of changes.entries()) { + for (const change of peerChanges) { + console.log("Change:", { + peer: change.peer, + counter: change.counter, + lamport: change.lamport, + timestamp: change.timestamp, + message: change.message, + }); + } +} + +// Get specific change + +const changeId = { peer: "1", counter: 0 } as OpId; +const change = doc.getChangeAt(changeId); + +// Get operations in a change +const ops = doc.getOpsInChange(changeId); + +doc.travelChangeAncestors([changeId], (change) => { + console.log("Ancestor change:", change); + return true; // continue traversal +}); + +// Get modified containers in a change +const modifiedContainers = doc.getChangedContainersIn(changeId, 1); +``` + +### Advanced Import/Export + +Loro supports various import and export modes: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +// Export modes +const doc = new LoroDoc(); +const previousVersion = doc.version(); +doc.getText("text").insert(0, "Hello"); +const snapshot = doc.export({ mode: "snapshot" }); +const updates = doc.export({ mode: "update", from: previousVersion }); +const shallowSnapshot = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.oplogFrontiers(), +}); +const rangeUpdates = doc.export({ + mode: "updates-in-range", + spans: [{ id: { peer: "1", counter: 0 }, len: 10 }], +}); + +// Import with status tracking +const status = doc.import(updates); +console.log("Successfully imported:", status.success); +console.log("Pending imports:", status.pending); + +// Batch import +const status2 = doc.importBatch([snapshot, updates]); + +// Import JSON updates +const jsonStatus = doc.importJsonUpdates({ + schema_version: 1, + start_version: new Map([["1", 0]]), + peers: ["1"], + changes: [], +}); +``` + +### Path and Value Access + +Access document content through paths: + +```ts no_run twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Get value or container by path +const value = doc.getByPath("map/key"); +const container = doc.getByPath("list"); + +// Get path to a container +const path = doc.getPathToContainer("cid:root-list:List"); + +// JSONPath support +const results = doc.JSONPath("$.list[*]"); + +// Get shallow values (container IDs instead of resolved values) +const shallowDoc = doc.getShallowValue(); +console.log(shallowDoc); // { list: 'cid:root-list:List', ... } + +// Custom JSON serialization +const json = doc.toJsonWithReplacer((key, value) => { + if (value instanceof LoroText) { + return value.toDelta(); + } + return value; +}); +``` + +### Debug and Metadata + +Access debug information and metadata: + +```ts no_run twoslash +import { setDebug, LoroDoc, decodeImportBlobMeta } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +// Enable debug info +setDebug(); +const blob = doc.export({ mode: "update" }); +// Get import blob metadata +const metadata = decodeImportBlobMeta(blob, true); +console.log({ + startTimestamp: metadata.startTimestamp, + endTimestamp: metadata.endTimestamp, + mode: metadata.mode, + changeNum: metadata.changeNum, +}); +``` + + +# FILE: pages/docs/tutorial/cursor.mdx + +--- +keywords: "cursor, crdts, multi-player, awareness, position, loro" +description: "How to use Loro to implement cursor position in real-time collaboration" +--- + +# Cursor + +Cursor is an independently storable entity, meaning it can store content separately from the Loro document. It is used to stably represent positions within structures such as Text, List, or MovableList. Cursors can be utilized to represent collaborative cursor positions, highlight ranges, or comment ranges. + +## Motivation + +Using "index" to denote cursor positions can be unstable, as positions may shift with document edits. To reliably represent a position or range within a document, it is more effective to leverage the unique ID of each item/character in a List CRDT or Text CRDT. + +## Updating Cursors + +Loro optimizes State metadata by not storing the IDs of deleted elements. This approach, while efficient, complicates tracking cursor positions since they rely on these IDs for precise locations within the document. The solution recalculates position by replaying relevant history to update stable positions accurately. To minimize the performance impact of history replay, the system updates cursor info to reference only the IDs of currently present elements, thereby reducing the need for replay. + +Each position has a "Side" information, indicating the actual cursor position is on the left, right, or directly in the center of the target ID. + +Note: In JavaScript, the offset returned when querying a Stable Position is based on the UTF-16 index. + +## Example + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const loro = new LoroDoc(); +const list = loro.getList("list"); +list.insert(0, "a"); +const pos0 = list.getCursor(0); +list.insert(1, "b"); +{ + const ans = loro.getCursorPos(pos0!); + expect(ans.offset).toEqual(0); + expect(ans.side).toEqual(0); + expect(ans.update).toBeUndefined(); +} +list.insert(0, "c"); +{ + const ans = loro.getCursorPos(pos0!); + expect(ans.offset).toEqual(1); + expect(ans.side).toEqual(0); + expect(ans.update).toBeUndefined(); +} +list.delete(1, 1); +{ + const ans = loro.getCursorPos(pos0!); + expect(ans.offset).toEqual(1); + expect(ans.side).toEqual(-1); + expect(ans.update).toBeDefined(); +} +``` + + +# FILE: pages/docs/tutorial/map.mdx + +--- +keywords: "map crdt, last write win, key value, conflict" +description: "how to use loro map crdt and show all APIs of loro map crdt." +--- + +# Map + +Loro's Map uses LWW (Last-Write-Wins) semantics. When concurrent edits conflict, it compares Lamport logic timestamps to determine the winner. + +Map slot conflicts are still resolved by LWW. If the value at a key is a child +container, regular child containers have operation-derived identities, so two +peers that create a child at the same key can create two different children. If +the child should be shared by logical key, use a mergeable child container: + +```ts no_run +const body = doc.getMap("notes").ensureMergeableText(noteId); +body.insert(0, "Hello"); +``` + +Here is how to use it: + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const docA = new LoroDoc(); +docA.setPeerId("0"); +const docB = new LoroDoc(); +docB.setPeerId("1"); + +const mapA = docA.getMap("map"); +const mapB = docB.getMap("map"); + +mapA.set("a", 1); +const textB = mapB.setContainer("a", new LoroText()); +textB.insert(0, "Hi"); + +console.log(docA.toJSON()); // OUTPUT: { map: { a: 1 } } +console.log(docB.toJSON()); // OUTPUT: { map: { a: "Hi" } } + +docA.import(docB.export({ mode: "snapshot" })); +docB.import(docA.export({ mode: "snapshot" })); + +// docB wins because it has the larger peerId, thus the larger logical timestamp +console.log(docA.toJSON()); // OUTPUT: { map: { a: "Hi" } } +console.log(docB.toJSON()); // OUTPUT: { map: { a: "Hi" } } +``` + +> **Note**: When calling `map.set(key, value)` on a LoroMap, if `map.get(key)` already returns `value`, the operation will be a no-op (no operation recorded). + + +# FILE: pages/docs/tutorial/persistence.mdx + +## Best Practices for Persisting Loro Documents + +The simplest approach is to use full snapshots for both import and export +operations. Here's how it works: + +1. Import the entire snapshot when loading the document. +2. Export a complete snapshot when saving changes. +3. Implement a debounce or throttle mechanism to trigger snapshot saves after a + certain number of edits. + +This method simplifies initial application development but has a drawback: user +edits are not immediately saved. Let's explore how to quickly save each user +edit while minimizing resource consumption. + +### Balancing Quick Saves and Resource Efficiency + +To achieve both quick saves and resource efficiency: + +- Use [`Snapshot Encoding`](./encoding#snapshot-encoding) to periodically store the entire document. +- Use [`Updates Encoding`](./encoding#updates-encoding) to export delta updates frequently (e.g., after each + keystroke or with debounce/throttle). Store these binary data in fast-write + storage like user disks or a key-value database. This ensures quick saves with + low resource cost. +- Loro validates the checksum embedded in every snapshot/update header during import, so corrupted binary payloads (for example, from bit flips in queues or object storage) are rejected before they touch your document state. +- When loading a document, import the snapshot and all related updates to get + the latest version. +- After importing, export a new snapshot to replace the old one and remove + imported updates for faster future loading. +- If your `LoroDoc` has grown large and older history can be safely recycled, + use `Shallow Snapshot Encoding` to reduce snapshot size. You can archive the + history before the shallow snapshot's start version in cold storage. + +For collaboration, the binary data generated by snapshot/updates can be +transmitted through any medium, such as WebRTC, WebSocket, or HTTP. + +The strong eventual consistency in CRDTs ensures that peers with identical sets +of operations will converge to the same document state, obviating concerns about +the order, duplication, or timing of operation delivery. + + +# FILE: pages/docs/tutorial/encoding.mdx + +--- +keywords: "crdts, schema, encoding, persist, snapshot, gc" +description: "Introduce how to encode and decode Loro documents, and how to persist data" +--- + +# Export Mode + +> Loro 1.0 has stabilized the data format and will not have any breaking +> changes. + +Loro introduces three encoding modes to meet the needs for different use cases: + +- **Updates Encoding**: Encodes all or a specified range of operations, mainly + used for delta updates of documents. +- **Snapshot Encoding**: Encodes the entire document, including both all the + operations and the current document state. +- **Shallow Snapshot Encoding**: Loro 1.0 provides a new encoding format for + discarding useless historical operations. It is a special snapshot that + encodes the most recent historical operations, the starting state of the + remaining history, and the latest state of the document. + +Different encoding formats are provided through the unified `doc.export(mode)`, +and all binary encoding formats can be imported via `doc.import(bytes)`. + +```ts no_run +export type ExportMode = + | { + mode: "update"; + from?: VersionVector; + } + | { + mode: "updates-in-range"; + spans: { + id: ID; + len: number; + }[]; + } + | { + mode: "snapshot"; + } + | { + mode: "shallow-snapshot"; + frontiers: Frontiers; + }; +``` + +## Updates Encoding + +There are two modes for updates encoding. `update` 和 `updates-in-range`. + +- `update` mode will encode all ops from the from version to the latest version + of the document. +- `updates-in-range` mode will encode all ops within the specified version + range. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc1 = new LoroDoc(); +const doc2 = new LoroDoc(); +doc2.setPeerId(2); + +doc2.getText("text").insert(0, "hello"); +const bytes = doc2.export({ mode: "update", from: doc1.version() }); // Uint8Array +// const bytes = doc2.export({ mode: "updates-in-range", spans: [{id: { peer: 2, counter: 0 }, len: 1}] }); +// These bytes can be stored in a database or transmitted over a network. +doc1.import(bytes); +console.log(doc1.toJSON()); // { text: "hello" } +``` + +Updates Encoding selectively encodes operations from a chosen initial version to +the most recent, enhancing support for real-time collaboration by focusing on +incremental changes rather than the entire document state. + +## Snapshot Encoding + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc1 = new LoroDoc(); +const doc2 = new LoroDoc(); + +doc2.getText("text").insert(0, "hello"); +const bytes = doc2.export({ mode: "snapshot" }); // Uint8Array +// These bytes can be stored in a database or transmitted over a network. +doc1.import(bytes); +console.log(doc1.toJSON()); // { text: "hello" } +``` + +Snapshot Encoding comprehensively captures a document's current state and its +historical operations. This mode can quickly obtain the latest document state. + +## Shallow Snapshot Encoding + +Loro will save all editing history to resolve conflicts and history +backtracking. However, for most scenes, most of the older history is useless, +taking up too much extra memory and requires more storage space for saving. + +Loro 1.0 provides a shallow snapshot encoding mode. You can specify the starting +historical version to be retained, and Loro will truncate all the history before +this version. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc1 = new LoroDoc(); +const doc2 = new LoroDoc(); +doc2.setPeerId(2); + +const text = doc2.getText("text"); +text.insert(0, "hello"); +const frontiers = doc2.frontiers(); +text.insert(0, "w"); +text.insert(0, "o"); +text.insert(0, "r"); +text.insert(0, "l"); +text.insert(0, "d"); +const bytes = doc2.export({ mode: "shallow-snapshot", frontiers }); // Uint8Array +// These bytes can be stored in a database or transmitted over a network. +doc1.import(bytes); +console.log(doc1.toJSON()); // { text: "hello" } +``` + +Note: When using shallow snapshots, you cannot import updates that are +concurrent to the snapshot's start version. For details, see +[shallow snapshot](/docs/advanced/shallow_snapshot). + +## Loro's Snapshot File Format + +To support lazy loading capabilities, we referenced common storage formats in +databases, introduced a simple LSM engine, and abstracted the encoding results +as simple key bytes and value bytes representations, preparing for future +integration with common key-value databases. + +### Encoding Details + +In Loro, multiple consecutive Ops of the same Container are merged into one +Change, i.e., one commit record. Each Change has additional metadata, such as +ID, Lamport, Timestamp, etc. In most scenarios, Changes are also consecutive, +and related metadata can be compressed to some extent. Therefore, we combine +multiple consecutive Changes into a ChangeBlock, which is the minimum unit for +encoding Op history. Detailed encoding content can be found in the +[documentation](https://github.com/loro-dev/loro/blob/main/crates/loro-internal/src/oplog/change_store/block_encode.rs). + +We use the first Change ID of the ChangeBlock as the query key, and the encoded +ChangeBlock as the value. This allows for quick querying of specified ID Changes +and their Ops without decoding all Changes. + +The complete State is composed of the states of all Containers starting from the +Root Container. The key is the bytes representation of the ContainerID, and the +value is composed of each Container's metadata and the bytes encoded according +to its own semantic state expression. + +These key-value pairs will be stored in our implemented simple LSM structure. + +The final exported encoding will be divided into four parts: encoding header, +oplog bytes, latest state bytes, and shallow state bytes. + +The Header consists of a 4-byte magic number, 16-byte checksum, and 2-byte +encode mode. + +The oplog bytes, latest state bytes, and shallow state bytes are all exported +using the +[LSM encoding structure](https://github.com/loro-dev/loro/blob/main/crates/kv-store/src/lib.rs). + +## Snapshot Encoding Layout + +![](../concepts/shallow-imgs/image-4.png) + + +# FILE: pages/docs/tutorial/sync.mdx + +## Sync + +Two documents with concurrent edits can be synchronized by just two message +exchanges. + +Below is an example of synchronization between two documents: + +```ts twoslash +import { LoroDoc, LoroList } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const docA = new LoroDoc(); +const docB = new LoroDoc(); +const listA: LoroList = docA.getList("list"); +listA.insert(0, "A"); +listA.insert(1, "B"); +listA.insert(2, "C"); +// B import the ops from A +const data: Uint8Array = docA.export({ mode: "update" }); +// The data can be sent to B through the network +docB.import(data); +expect(docB.toJSON()).toStrictEqual({ + list: ["A", "B", "C"], +}); + +const listB: LoroList = docB.getList("list"); +listB.delete(1, 1); + +// `doc.export({mode: "update", from: version})` can encode all the ops from the version to the latest version +// `version` is the version vector of another document +const missingOps = docB.export({ + mode: "update", + from: docA.oplogVersion(), +}); +docA.import(missingOps); + +expect(docA.toJSON()).toStrictEqual({ + list: ["A", "C"], +}); +expect(docA.toJSON()).toStrictEqual(docB.toJSON()); +``` + +## Real-time Collaboration + +Due to CRDT properties, document consistency is guaranteed when peers receive the same updates, regardless of order or duplicates. + +Consistency does not remove the need to choose the right child container +identity. If two offline peers create regular child containers under the same +Map key, sync will converge to one visible Map value and the other child can be +hidden by the Map conflict rule. For dynamic child containers that should be +shared by logical key, use `ensureMergeable*` when modeling the document. + +### Sync Strategies + +1. **First Sync** (Initial synchronization between peers): + - New peers can exchange their [Version Vectors](/docs/tutorial/version) to determine missing updates + - Use `doc.export({ mode: "update", from: versionVector })` to get updates since the peer's last known state. + You may as well send the whole history by `doc.export({ mode: "update" })` as shown in the example above. + - Example shows basic first sync scenario + +2. **Realtime Sync** (Continuous updates): + - Subscribe to local updates + - Broadcast updates directly to all other peers + - No need for version comparison after initial sync + - As long as updates reach all peers, consistency is maintained + +### Example + +Here's how two peers can establish realtime sync when one comes online with offline changes: + +1. Both peers exchange their version information +2. Each peer shares their missing updates: + - `doc2` gets updates it's missing from `doc1` + - `doc1` gets updates it's missing from `doc2` +3. Both peers establish realtime sync to stay connected + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc1 = new LoroDoc(); +doc1.getText("text").insert(0, "Hello"); +// Peer2 joins the network +const doc2 = new LoroDoc(); +// ... doc2 may import its local snapshot + +// 1. Exchange version information +const peer2Version = doc2.oplogVersion(); +const peer1Version = doc1.oplogVersion(); + +// 2. Request missing updates from existing peers +const missingOps = doc1.export({ + mode: "update", + from: peer2Version, +}); +doc2.import(missingOps); +const missingOps2 = doc2.export({ + mode: "update", + from: peer1Version, +}); +doc1.import(missingOps2); + +// 3. Establish realtime sync +doc2.subscribeLocalUpdates((update) => { + // websocket.send(update); +}); +doc1.subscribeLocalUpdates((update) => { + // websocket.send(update); +}); + +// Now both peers are in sync and can collaborate +``` + +## Understanding the `import()` Return Value + +The `import()` method in Loro's JavaScript/WASM binding returns an object that provides feedback on the import operation. This object, let's call it `ImportStatusJS`, has the following structure: + +```ts no_run twoslash +interface ImportStatusJS { + success: PeerVersionRange; + pending?: PeerVersionRange; // Optional: only present if there are pending operations +} + +interface PeerVersionRange { + [peerId: string]: { + start: number; // Start counter (inclusive) + end: number; // End counter (exclusive) + }; +} +``` + +### Fields Explained: + +1. **`success`** (Object, `PeerVersionRange`) + - **Description**: This field is always present and details the ranges of operations (changes) that were successfully imported and applied to the Loro document. + - **Structure**: It's an object where: + - Each **key** is a `string` representing a `PeerID` (the unique identifier of a collaborator or a source of changes). + - Each **value** is an object `{ start: number, end: number }` defining a continuous range of operation counters for that specific peer. + - `start`: The starting counter of the successfully imported range (inclusive). + - `end`: The ending counter of the successfully imported range (exclusive). This means operations from `start` up to, but not including, `end` were processed. + - **Purpose**: Helps understand which parts of the provided update data have been integrated into the local document's state. + - **Example**: + ```javascript + // Assuming importResult is the return value of doc.import(bytes) + console.log(importResult.success); + // Example output: + // { + // "clientA_peerId": { "start": 0, "end": 50 }, + // "server_peerId": { "start": 120, "end": 150 } + // } + // This means operations from clientA (counters 0-49) and + // operations from server (counters 120-149) were successfully imported. + ``` + +2. **`pending`** (Object, `PeerVersionRange`, optional) + - **Description**: This field is only present if some operations from the imported data could not be applied because they depend on other operations that Loro has not seen yet (i.e., their causal dependencies are missing). It details these "pending" operation ranges. + - **Structure**: Identical to the `success` field. An object mapping `PeerID` strings to `{ start: number, end: number }` counter ranges. + - **Purpose**: Informs the application that certain changes are known but are "on hold" awaiting their prerequisites. To apply these pending changes, the missing prerequisite operations must be imported first. This is crucial for maintaining data consistency in collaborative scenarios. + - **Example**: + ```javascript + // Assuming importResult is the return value of doc.import(bytes) + if (importResult.pending) { + console.log(importResult.pending); + // Example output: + // { + // "clientA_peerId": { "start": 50, "end": 60 }, + // "clientB_peerId": { "start": 10, "end": 25 } + // } + // This means operations from clientA (counters 50-59) and + // operations from clientB (counters 10-24) are pending due to missing dependencies. + } + ``` + +### How to Use This Information: + +- Check the `success` field to confirm which updates were applied. +- If the `pending` field exists and is not empty, it signals that further updates (dependencies) are required to fully integrate all known changes. Your application might need to fetch or request these missing updates from other peers or a central server. + + +# FILE: pages/docs/tutorial/list.mdx + +--- +keywords: "list crdt, movable list, move element, cursor, awareness" +description: "how to use loro list and movable list crdt and show all APIs of loro list and movable crdt." +--- + +# List and Movable List + +Loro supports two types of lists: `List` and `MovableList`. The `List` is a +standard list that supports Insert and Delete operations. In contrast, the +`MovableList` supports additional Set and Move operations. + +Using a combination of insert and delete operations, one can simulate set and +move operations on a `List`. However, this approach fails in concurrent editing +scenarios. For example, if the same element is set or moved concurrently, the +simulation would result in the deletion of the original element and the +insertion of two new elements, which does not meet expectations. + +The `MovableList` addresses these issues by ensuring that set and move +operations do not lead to such problems, though it incurs additional overheads. +Specifically, when performing only insertions/deletions, the `MovableList` is +approximately 80% slower in encode/decode and consumes about 50% more memory +compared to the `List`. + +Both `List` and `MovableList` utilize the +[_Fugue_](https://arxiv.org/abs/2305.00583) to achieve _maximal +non-interleaving_. Additionally, `MovableList` uses the algorithm from +[_Moving Elements in List CRDTs_](https://martin.kleppmann.com/2020/04/27/papoc-list-move.html) +to implement the move operation. + +## Basic Usage + +### List + +```ts twoslash +import { LoroDoc, Loro } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const docA = new LoroDoc(); +docA.setPeerId("1"); +const listA = docA.getList("list"); +listA.push(0); +listA.push(1); +listA.push(2); +const bytes: Uint8Array = docA.export({ mode: "snapshot" }); + +const docB = Loro.fromSnapshot(bytes); +docB.setPeerId("2"); +const listB = docB.getList("list"); +{ + // Concurrently docA and docB update element at index 2 + // docA updates it to 8 + // docB updates it to 9 + // docA.toJSON() should return { list: [0, 1, 8] } + // docB.toJSON() should return { list: [0, 1, 9] } + listB.delete(2, 1); + listB.insert(2, 9); + expect(docB.toJSON()).toStrictEqual({ list: [0, 1, 9] }); + listA.delete(2, 1); + listA.insert(2, 8); + expect(docA.toJSON()).toStrictEqual({ list: [0, 1, 8] }); +} + +{ + // Merge docA and docB + docA.import(docB.export({ mode: "update", from: docA.version() })); + docB.import(docA.export({ mode: "update", from: docB.version() })); +} + +expect(docA.toJSON()).toStrictEqual({ list: [0, 1, 8, 9] }); +expect(docB.toJSON()).toStrictEqual({ list: [0, 1, 8, 9] }); +``` + +### MovableList + +```ts twoslash +import { LoroDoc, Loro } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const docA = new LoroDoc(); +docA.setPeerId("1"); +const listA = docA.getMovableList("list"); +listA.push(0); +listA.push(1); +listA.push(2); +const bytes: Uint8Array = docA.export({ mode: "snapshot" }); + +const docB = Loro.fromSnapshot(bytes); +docB.setPeerId("2"); +const listB = docB.getMovableList("list"); +{ + // Concurrently docA and docB update element at index 2 + // docA updates it to 8 + // docB updates it to 9 + // docA.toJSON() should return { list: [0, 1, 8] } + // docB.toJSON() should return { list: [0, 1, 9] } + listA.set(2, 8); + expect(docA.toJSON()).toStrictEqual({ list: [0, 1, 8] }); + listB.set(2, 9); + expect(docB.toJSON()).toStrictEqual({ list: [0, 1, 9] }); +} + +{ + // Merge docA and docB + docA.import(docB.export({ mode: "update", from: docA.version() })); + docB.import(docA.export({ mode: "update", from: docB.version() })); +} + +// Converge to [0, 1, 9] because docB has larger peerId thus larger logical time +expect(docA.toJSON()).toStrictEqual({ list: [0, 1, 9] }); +expect(docB.toJSON()).toStrictEqual({ list: [0, 1, 9] }); + +{ + // Concurrently docA and docB move element at index 0 + // docA moves it to 2 + // docB moves it to 1 + // docA.toJSON() should return { list: [1, 9, 0] } + // docB.toJSON() should return { list: [1, 0, 9] } + listA.move(0, 2); + listB.move(0, 1); + expect(docA.toJSON()).toStrictEqual({ list: [1, 9, 0] }); + expect(docB.toJSON()).toStrictEqual({ list: [1, 0, 9] }); +} + +{ + // Merge docA and docB + docA.import(docB.export({ mode: "update", from: docA.version() })); + docB.import(docA.export({ mode: "update", from: docB.version() })); +} + +// Converge to [1, 0, 9] because docB has larger peerId thus larger logical time +expect(docA.toJSON()).toStrictEqual({ list: [1, 0, 9] }); +expect(docB.toJSON()).toStrictEqual({ list: [1, 0, 9] }); +``` + +## Using Cursor on List + +The Cursor on a List changes with the list's modifications. If new elements are +inserted in front of it, it moves to the right. If elements in front are +deleted, it moves to the left. If elements are inserted or deleted behind it, it +remains stationary. + +If you use a List to represent paragraphs in an article, you can use a Cursor to +stably represent the selection range on the paragraph. + +```ts twoslash +import { + Cursor, + LoroDoc, + LoroList, + LoroMovableList, + LoroText, +} from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("1"); +const list = doc.getList("list"); +list.push("Hello"); +list.push("World"); +const cursor = list.getCursor(1)!; +console.log(cursor.pos()); // OUTPUT: { peer: "1", counter: 1 } + +const encodedCursor: Uint8Array = cursor.encode(); +const exported: Uint8Array = doc.export({ mode: "snapshot" }); + +// Sending the exported snapshot and the encoded cursor to peer 2 +// Peer 2 will decode the cursor and get the position of the cursor in the list +// Peer 2 will then insert "Hello" at the beginning of the list + +const docB = new LoroDoc(); +docB.setPeerId("2"); +const listB = docB.getList("list"); +docB.import(exported); +listB.insert(0, "Foo"); +console.log(docB.toJSON()); // OUTPUT: { list: ["Foo", "Hello", "World"] } +const cursorB = Cursor.decode(encodedCursor); +{ + // The cursor position is shifted to the right by 1 + const pos = docB.getCursorPos(cursorB); + console.log(pos.offset); // OUTPUT: 2 +} +listB.insert(1, "Bar"); +console.log(docB.toJSON()); // OUTPUT: { list: ["Foo", "Bar", "Hello", "World"] } +{ + // The cursor position is shifted to the right by 1 + const pos = docB.getCursorPos(cursorB); + console.log(pos.offset); // OUTPUT: 3 +} +listB.delete(3, 1); +console.log(docB.toJSON()); // OUTPUT: { list: ["Foo", "Bar", "Hello"] } +{ + // The position the cursor points to is now deleted, + // but it should still get the position + const pos = docB.getCursorPos(cursorB); + console.log(pos.offset); // OUTPUT: 3 + + // It will also offer an update on the cursor position. + // + // Because the old cursor position is deleted, `doc.getCursorPos()` will slow down over time. + // Internally, it needs to traverse the related history to find the correct position for a deleted + // cursor position. + // + // After refreshing the cursor, the performance of `doc.getCursorPos()` will improve. + console.log(pos.update); // OUTPUT: { peer: "2", counter: 1 } + const newCursor: Cursor = pos.update!; + + // The new cursor position is undefined because the cursor is at the end of the list + console.log(newCursor.pos()); // OUTPUT: undefined + // The side is 1 because the cursor is at the right end of the list + console.log(newCursor.side()); // OUTPUT: 1 + + const newPos = docB.getCursorPos(newCursor); + // The offset doesn't change + console.log(newPos.offset); // OUTPUT: 3 + // The update is undefined because the cursor no longer needs to be updated + console.log(newPos.update); // OUTPUT: undefined +} +``` + + +# FILE: pages/docs/tutorial/tree.mdx + +--- +keywords: "tree crdt, move operation, fractional index, hierarchical, tree" +description: "how to use loro tree crdt and show all APIs of loro tree crdt." +--- + +# Tree + +When it comes to utilizing hierarchical data structures and even performing operations across different levels of data, tree structures are of paramount importance. Loro implements the concept of a movable tree CRDT, based on the algorithm created by Kleppmann et al. in **_[A highly-available move operation for replicated trees](https://martin.kleppmann.com/2021/10/07/crdt-tree-move-operation.html)_**. + +In addition to this, Loro further employs **Fractional Index** (an algorithm used in distributed systems to maintain the order of sequences) to sort each child node, ensuring that sibling nodes maintain an orderly sequence. As a result, Loro can provide APIs such as `moveAfter()` and `moveBefore()`. This is particularly useful when a custom-ordered hierarchical view is required. + +## Node Data + +Loro associates a [Map](https://www.loro.dev/docs/tutorial/map) with each tree node, serving as a data container for the node, allowing you to nest any data structure supported by Loro. + +> Note: The associated Map Container is considered a child of the Tree Container within the Loro documentation hierarchy. + +You can get the associated map by `.data` of the `LoroTreeNode` so you can invoke `set` or `setContainer` on the map to add some data or sub-containers to the tree node. For example: + +```ts twoslash +import { LoroDoc, LoroTree, LoroTreeNode, LoroMap, LoroList } from "loro-crdt"; +// ---cut--- +let doc = new LoroDoc(); +let tree: LoroTree = doc.getTree("tree"); +let node: LoroTreeNode = tree.createNode(); +node.data.set("key", "value"); +node.data.setContainer("list", new LoroList()); +``` + +`node.data` is a Map. If the child container under `node.data` may be lazily +created by multiple peers at the same key, use a mergeable child such as +`node.data.ensureMergeableList("list")` instead of `setContainer`. + +## Ordered Tree Nodes + +In certain scenarios such as graphic design or file systems, where sibling nodes may also require a sequential relationship, we have introduced `Fractional Index` in Loro to support this capability. You can read more about `Fractional Index` in the [Figma blog](https://www.figma.com/blog/realtime-editing-of-ordered-sequences). In simple terms, `Fractional Index` assigns a sortable value to each object, and if a new insertion occurs between two objects, the `Fractional Index` of the new object will be between the left and right values. The rust-based `Fractional Index` [implementation of Drifting-in-Space](https://github.com/drifting-in-space/fractional_index) has good design and minimal document size growth in most cases. We forked it and made a simple extension for use in Loro. + +Whenever a new tree node is created or a node is moved to a new position, Loro will generate a `Fractional Index` for the node based on its position to sort among sibling nodes. In collaborative environments, conflicts with `Fractional Indexes` can arise, such as different nodes having the same `Fractional Index` or the situation where the calculation of a new node's position results in the same `Fractional Index` on both sides. We will discuss the corresponding conflict resolution methods in detail in [the blog post](https://www.loro.dev/blog/movable-tree). + +You should call `enable_fractional_index(jitter: number)` to enable `Fractional Index`. + +> Note: Fractional Index has an interleaving issue, but we believe this is acceptable for tree structures. + +## Events + +There are three types of events in the tree structure: `Create`, `Move`, and `Delete`, as follows: + +```ts no_run twoslash +import { TreeID } from "loro-crdt"; +// ---cut--- +export type TreeDiffItem = + | { + target: TreeID; + action: "create"; + parent: TreeID | undefined; + index: number; + fractionalIndex: string; + } + | { + target: TreeID; + action: "delete"; + oldParent: TreeID | undefined; + oldIndex: number; + } + | { + target: TreeID; + action: "move"; + parent: TreeID | undefined; + index: number; + fractionalIndex: string; + oldParent: TreeID | undefined; + oldIndex: number; + }; +``` + +Here, `index` and `fractionalIndex` represent the node's index and the hex string representation of the `Fractional Index`, respectively. Loro will emit events in the order of causality. The deletion event signifies that the target node, along with the entire subtree rooted at the target node, has been deleted. Deletion events for the child nodes are not emitted. + +### Events of node's data + +Since the data of the tree nodes is represented using `MapContainer`, each `MapContainer` associated with a tree node is a child of the `TreeContainer` in the document. If you modify the data of a tree node, you will receive an event from the `MapContainer`. However, the event path contains a element of `TreeID` to indicate which node's data has been altered. + +## Retrieving All Nodes + +There are multiple ways to retrieve all nodes from a `LoroTree`: + +1. `nodes()`: Retrieves all `LoroTreeNode` instances from the current `LoroTree`, but this function does not guarantee the order of the nodes. +2. `roots()`: Retrieves all root nodes from the current `LoroTree`, with the root nodes being ordered. Subsequently, the `children()` method of `LoroTreeNode` can be used to perform a hierarchical traversal. +3. `toArray()`: Retrieves all node information in a hierarchical traversal order, where each node's data structure is as follows: + + ```ts no_run twoslash + import { TreeID, LoroMap } from "loro-crdt"; + // ---cut--- + { + id: TreeID; + parent: TreeID | undefined; + index: number; + fractionalIndex: string; + meta: LoroMap; + } + ``` + +4. `toJSON()`: The JSON representation of `toArray()`, where each node's `meta` is also recursively parsed into JSON format. + +## Basic Usage + +### Example + +```ts no_run twoslash +import { LoroDoc, LoroTree, LoroTreeNode, LoroMap } from "loro-crdt"; +// ---cut--- +let doc = new LoroDoc(); +let tree: LoroTree = doc.getTree("tree"); +let root: LoroTreeNode = tree.createNode(); +// By default, append to the end of the parent node's children list +let node = root.createNode(); +// Specify the child's position +let node2 = root.createNode(0); +// Move the node to become a child of another node +node.move(node2); +// Specify the child's position within the new parent +node.move(node2, 0); +// Move the node to become the root node +node.move(); +// Move the node to be positioned after another node +node.moveAfter(node2); +// Move the node to be positioned before another node +node.moveBefore(node2); +// Retrieve the index of the node within its parent's children +let index = node.index(); +// Get the fractional index of the node +let fractionalIndex = node.fractionalIndex(); +// Access the associated data map container +let nodeData: LoroMap = node.data; +``` + + +# FILE: pages/docs/tutorial/time_travel.mdx + +--- +keywords: "crdt, time travel, history, checkout, version control" +description: "time travel in Loro" +--- + +# How to Use Time Travel in Loro + +In Loro, you can call `doc.checkout(frontiers)` to jump to the version specified +by the +frontiers([Learn more about frontiers](/docs/advanced/version_deep_dive#frontiers)). + +Note that using `doc.checkout(frontiers)` to jump to a specific version places +the document in a detached state, preventing further edits. To learn more, see +[_Attached/Detached Status_](/docs/advanced/doc_state_and_oplog#attacheddetached-status). +To continue editing, reattach the document to the latest version using +`doc.attach()`. This design is temporary and will be phased out once we have a +more refined version control API in place. + +## Read-only Time Travel + +Below we demonstrate how to implement simple, read-only time-travel. You could, +for example, combine this with a slider in a UI to allow users to view the document +over time. + +### Enable Timestamps + +Before this example will work, it is important that the edits made to the document +have had [timestamp storage](/docs/advanced/timestamp) enabled: + +```ts no_run +doc.setRecordTimestamp(true); +``` + +This makes sure that all changes to the document will have a timestamp added to it. +We will use this timestamp to sort changes so that the ordering will match user +intuition. + +### Implementing Time Travel + +The first step is to load our document. Here we assume that you have a snapshot from your database +or API. + +```ts no_run +// Get the snapshot for your doc from your database / API +let snapshot = fetchSnapshot(); + +// Import into a new document +const doc = new LoroDoc(); +doc.import(snapshot); +``` + +Next we must collect and sort the timestamps for every change in the document. We want uesrs to be +able to drag a slider to select a timestamp out of this list. + +```ts no_run +// Collect all changes from the document +const changes = doc.getAllChanges(); + +// Get the timestamps for all changes +const timestamps = Array.from( + new Set( + [...changes.values()] + .flat() // Flatten changes from all peers into one list + .map((x) => x.timestamp) // Get the timestamp from each peer + .filter((x) => !!x), + ), +); + +// Sort the timestamps +timestamps.sort((a, b) => a - b); +``` + +Next we need to make a helper function that will return a list of +[Frontiers](/docs/advanced/version_deep_dive#frontiers) for any timestamp. + +For each peer that has edited a document, there is a list of changes by that peer. Each change has a +`counter`, and a `length`. That `counter` is like an always incrementing version number for the +changes made by that peer. + +A change's `counter` is the starting point of the change, and the `length` indicates how much the +change incremented the counter before the end of the change. + +The frontiers are the list of counters that we want to checkout from each peer. Since we are going +for a timeline view, we want to get the highest counter that we know happned before our timestamp +for each peer. + +Here we make a helper function to do that. + +```ts +const getFrontiersForTimestamp = ( + changes: Map, + ts: number, +): { peer: string; counter: number }[] => { + const frontiers = [] as { peer: string; counter: number }[]; + + // Record the highest counter for each peer where it's change is not later than + // our target timestamp. + changes.forEach((changes, peer) => { + let counter = -1; + for (const change of changes) { + if (change.timestamp <= ts) { + counter = Math.max(counter, change.counter + change.length - 1); + } + } + if (counter > -1) { + frontiers.push({ counter, peer }); + } + }); + return frontiers; +}; +``` + +Finally, all we can get the index from our slider, get the timestamp from our list, and then +checkout the calculated frontiers. + +```ts no_run +let sliderIdx = 3; +const timestamp = timestamps[sliderIdx - 1]; +const frontiers = getFrontiersForTimestamp(changes, timestamp); + +doc.checkout(frontiers); +``` + +## Time Travel With Editing + +Below is a more complete example demonstrating Time Travel functionality with a node editor. + + + + +# FILE: pages/docs/tutorial/composition.mdx + +--- +keywords: "crdts, json, data model, document state, semantics" +description: "Everyone can effectively model the states and the updates of documents that conform to the JSON schema." +--- + +# Composing CRDTs + +In Loro, you can build complex data structures using basic CRDTs such as List, MovableList, Map and Tree. These containers can include sub-containers, which in turn can contain more sub-containers, allowing for the composition of intricate data structures. + +It's important to note that documents in Loro must adhere to a tree structure. This means that while a parent can have multiple children, each child is restricted to only one parent. Therefore, the document forms a tree rather than a graph (like a DAG). + +By leveraging these fundamental CRDTs, you can effectively model the states and the updates of documents that conform to the JSON schema. + +```ts twoslash +import { LoroDoc, LoroList, LoroText } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +const map = doc.getMap("map"); +let callTimes = 0; +// Events from a child are propagated to all ancestor nodes. +map.subscribe((event) => { + console.log(event); + callTimes++; +}); + +// Create a sub container for map +// { map: { list: [] } } +const list = map.setContainer("list", new LoroList()); +list.push(0); +list.push(1); + +// Create a sub container for list +// { map: { list: [0, 1, LoroText] } } +const text = list.insertContainer(2, new LoroText()); +expect(doc.toJSON()).toStrictEqual({ map: { list: [0, 1, ""] } }); +{ + // Commit will trigger the event, because list is a sub container of map + doc.commit(); + await new Promise((resolve) => setTimeout(resolve, 1)); + expect(callTimes).toBe(1); +} + +text.insert(0, "Hello, "); +text.insert(7, "World!"); +expect(doc.toJSON()).toStrictEqual({ map: { list: [0, 1, "Hello, World!"] } }); +{ + // Commit will trigger the event, because text is a descendant of map + doc.commit(); + await new Promise((resolve) => setTimeout(resolve, 1)); + expect(callTimes).toBe(2); +} +``` + +`setContainer` creates a regular child container. This is fine for fixed +schemas, or when each creation should produce a distinct child object. If +multiple peers may lazily create the same child under the same Map key, use a +mergeable child container instead: + +```ts no_run +const sharedList = map.ensureMergeableList("list"); +``` + + +# FILE: pages/docs/tutorial/text.mdx + +--- +keywords: "text crdt, richtext, richtext editor" +description: "how to use loro richtext crdt and show all APIs of loro text crdt." +--- + +# Text + +Loro supports both plain text and rich text. When rich text features (like mark +and unmark) are not used, the text container operates as plain text without any +rich text overhead, making it efficient for simple text operations. + +LoroText offers excellent performance, particularly when handling large strings. +It significantly outperforms native JavaScript string operations due to its +internal B-tree structure. All basic operations like insert and delete have +O(log N) time complexity, making it highly efficient even when working with +documents containing several million characters. + +> To learn how rich text CRDT in Loro works under the hood, please refer to our +> blog: [Introduction to Loro's Rich Text CRDT](/blog/loro-richtext). + +LoroText also supports stable cursor and selection positions. Using the Cursor API (`text.getCursor`, `doc.getCursorPos`, `Side`), you can represent caret/selection endpoints that remain valid across concurrent edits—similar to Yjs's `RelativePosition`. See the [Cursor](/docs/tutorial/cursor) guide for details. + +## Editor Bindings + +Loro provides official bindings for popular editors to make it easier to integrate Loro's CRDT capabilities: + +### ProseMirror Binding + +The [loro-prosemirror](https://github.com/loro-dev/loro-prosemirror) package provides seamless integration between Loro and ProseMirror, a powerful rich text editor framework. It includes: + +- Document state synchronization with rich text support +- Cursor awareness and synchronization +- Undo/Redo support in collaborative editing + +The ProseMirror binding can also be used with [Tiptap](https://tiptap.dev/), a popular rich text editor built on top of ProseMirror. This means you can easily add collaborative editing capabilities to your Tiptap-based applications. + +```ts no_run +import { + CursorAwareness, + LoroCursorPlugin, + LoroSyncPlugin, + LoroUndoPlugin, + undo, + redo, +} from "loro-prosemirror"; +import { LoroDoc } from "loro-crdt"; +import { EditorView } from "prosemirror-view"; +import { EditorState } from "prosemirror-state"; +import { keymap } from "prosemirror-keymap"; + +const doc = new LoroDoc(); +const awareness = new CursorAwareness(doc.peerIdStr); +const plugins = [ + ...pmPlugins, + LoroSyncPlugin({ doc }), + LoroUndoPlugin({ doc }), + keymap({ + "Mod-z": undo, + "Mod-y": redo, + "Mod-Shift-z": redo, + }), + LoroCursorPlugin(awareness, {}), +]; +const editor = new EditorView(editorDom, { + state: EditorState.create({ doc, plugins }), +}); +``` + +### CodeMirror Binding + +The [loro-codemirror](https://github.com/loro-dev/loro-codemirror) package provides integration between Loro and CodeMirror 6, a versatile code editor. It supports: + +- Document state synchronization +- Cursor awareness +- Undo/Redo functionality + +```ts no_run +import { EditorState } from "@codemirror/state"; +import { EditorView } from "@codemirror/view"; +import { LoroExtensions } from "loro-codemirror"; +import { Awareness, LoroDoc, UndoManager } from "loro-crdt"; + +const doc = new LoroDoc(); +const awareness = new Awareness(doc.peerIdStr); +const undoManager = new UndoManager(doc, {}); + +new EditorView({ + state: EditorState.create({ + extensions: [ + // ... other extensions + LoroExtensions( + doc, + { + awareness: awareness, + user: { name: "Bob", colorClassName: "user1" }, + }, + undoManager, + ), + ], + }), + parent: document.querySelector("#editor")!, +}); +``` + +## LoroText vs String + +It's important to understand that LoroText is very different from using a regular string type. So the following code has different merge results: + +Using `LoroText`: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("0"); +doc.getText("text").insert(0, "Hello"); +const doc2 = new LoroDoc(); +doc2.setPeerId("1"); +doc2.getText("text").insert(0, "World"); +doc.import(doc2.export({ mode: "update" })); +console.log(doc.getText("text").toString()); // "HelloWorld" +``` + +Using `String`: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("0"); +doc.getMap("map").set("text", "Hello"); +const doc2 = new LoroDoc(); +doc2.setPeerId("1"); +doc2.getMap("map").set("text", "World"); +doc.import(doc2.export({ mode: "update" })); +console.log(doc.getMap("map").get("text")); // "World" +``` + +### Merge Semantics + +Unlike LoroMap which uses Last-Write-Wins (LWW) semantics, LoroText is designed to preserve edits. Here's how they differ: + +When user A and user B make concurrent edits to the same text: + +- LoroText will merge both users' edits in sequence, preserving both changes +- LoroMap will use LWW semantics, keeping only one user's changes + +### When to Use String in LoroMap + +There are specific scenarios where using a string in LoroMap (with LWW semantics) might be more appropriate than using LoroText: + +- **URLs**: When dealing with hyperlinks, automatic merging could result in invalid URLs. In this case, it's better to use LoroMap's LWW semantics to ensure the URL remains valid. +- **Hash String**: When handling hash string, LWW semantics are more appropriate to maintain data accuracy and consistency. + +## Rich Text Config + +To use rich text in Loro, you need to specify the expanding behaviors for each +format first. When we insert new text at the format boundaries, they define +whether the inserted text should inherit the format. + +There are four kinds of expansion behaviors: + +- `after`(default): when inserting text right after the given range, the mark + will be expanded to include the inserted text +- `before`: when inserting text right before the given range, the mark will be + expanded to include the inserted text +- `none`: the mark will not be expanded to include the inserted text at the + boundaries +- `both`: when inserting text either right before or right after the given + range, the mark will be expanded to include the inserted text + +### Example + +```ts twoslash +import { LoroDoc, Delta } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +doc.configTextStyle({ + bold: { expand: "after" }, + link: { expand: "before" }, +}); +const text = doc.getText("text"); +text.insert(0, "Hello World!"); +text.mark({ start: 0, end: 5 }, "bold", true); +expect(text.toDelta()).toStrictEqual([ + { + insert: "Hello", + attributes: { + bold: true, + }, + }, + { + insert: " World!", + }, +] as Delta[]); + +// " Test" will inherit the bold style because `bold` is configured to expand forward +text.insert(5, " Test"); +expect(text.toDelta()).toStrictEqual([ + { + insert: "Hello Test", + attributes: { + bold: true, + }, + }, + { + insert: " World!", + }, +] as Delta[]); +``` + +## Methods + +### `insert(pos: number, s: string)` + +Insert text at the given pos. + +### `delete(pos: number, len: number)` + +Delete text at the given range. + +### `slice(start: number, end: number): string` + +Get a string slice. + +### `sliceDelta(start: number, end: number): Delta[]` + +Get a Quill-style Delta slice for the given UTF-16 range. Use +`sliceDeltaUtf8` to slice by UTF-8 byte offsets instead. + +### `sliceDeltaUtf8(start: number, end: number): Delta[]` + +Get a Quill-style Delta slice for the given UTF-8 byte range. + +### `toString(): string` + +Get the plain text value. + +### `charAt(pos: number): char` + +Get the character at the given position. + +### `splice(pos: number, len: number, text: string): string` + +Delete and return the string at the given range and insert a string at the same +position. + +### `length: number` + +Get the length of text + +### `getCursor(pos: number, side?: Side): Cursor | undefined` + +Create a stable position handle for the given logical index. This is equivalent in purpose to Yjs's `RelativePosition` and is designed to stay valid across concurrent inserts/deletes. + +- A cursor can represent a caret or one end of a selection. Store two cursors (anchor/head) to represent a selection. +- Use `doc.getCursorPos(cursor)` to resolve the current absolute offset and side. It may also return an updated cursor you should persist to minimize future replays. +- Offsets for Text are UTF-16 indices for the WASM binding. +- `side` controls whether the cursor sits to the left (-1), center (0), or right (1) of the target ID at boundaries. It affects how the cursor behaves when text is inserted at its position. + +See the dedicated guide: [Cursor](/docs/tutorial/cursor). + +Example (caret): + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "123"); + +// Create a stable cursor at the beginning +const cur = text.getCursor(0, 0); +{ + const pos = doc.getCursorPos(cur!); + expect(pos.offset).toBe(0); +} + +// Insert before the cursor. The cursor now moves forward +text.insert(0, "1"); +{ + const pos = doc.getCursorPos(cur!); + expect(pos.offset).toBe(1); +} +``` + +Example (selection via two cursors): + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.update("Hello World"); + +// Anchor at index 0 (left side), head at index 5 (right side) +const anchor = text.getCursor(0, -1)!; +const head = text.getCursor(5, 1)!; + +// Resolve to absolute positions when needed +const a = doc.getCursorPos(anchor); +const h = doc.getCursorPos(head); +// selection = [a.offset, h.offset) +``` + +Persisting and restoring a cursor: + +```ts twoslash +import { Cursor, LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.update("Hi"); +const cur = text.getCursor(1, 0)!; + +// Serialize and share +const bytes = cur.encode(); + +// Later on another peer +const restored = Cursor.decode(bytes); +const pos = doc.getCursorPos(restored); +``` + +### `toJSON(): string` + +Returns the plain text content as a string. This method: +- Returns only the text content without any formatting marks +- Is not affected by any rich text attributes (bold, italic, links, etc.) +- Is equivalent to `toString()` for text containers + +If you need rich text information including formatting, use `toDelta()` instead. + +### `toDelta(): Delta[]` + +Get the rich text value with all formatting information. It's in +[Quill's Delta format](https://quilljs.com/docs/delta/). + +Unlike `toJSON()` which returns plain text, `toDelta()` preserves all rich text attributes like bold, italic, links, and custom marks. + +Example comparing `toJSON()` vs `toDelta()`: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.configTextStyle({ bold: { expand: "after" } }); +const text = doc.getText("text"); +text.insert(0, "Hello World!"); +text.mark({ start: 0, end: 5 }, "bold", true); + +// toJSON returns plain string without marks +console.log(text.toJSON()); // "Hello World!" + +// toDelta returns rich text with formatting +console.log(text.toDelta()); +// [ +// { insert: "Hello", attributes: { bold: true } }, +// { insert: " World!" } +// ] +``` + +### Slice a Delta snippet + +Use `sliceDelta(start, end)` when you only need a portion of the Delta (for example, to copy a styled snippet). It uses UTF-16 indices just like other text APIs; use `sliceDeltaUtf8` if you need to slice by UTF-8 byte offsets instead. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +doc.configTextStyle({ + bold: { expand: "after" }, + comment: { expand: "none" }, +}); +const text = doc.getText("text"); + +text.insert(0, "Hello World!"); +text.mark({ start: 0, end: 5 }, "bold", true); +text.mark({ start: 6, end: 11 }, "comment", "greeting"); + +const snippet = text.sliceDelta(1, 8); +expect(snippet).toStrictEqual([ + { insert: "ello", attributes: { bold: true } }, + { insert: " " }, + { insert: "Wo", attributes: { comment: "greeting" } }, +]); +``` + +### `mark(range: {start: number, end: number}, key: string, value: any): void` + +Mark the given range with a key-value pair. + +### `unmark(range: {start: number, end: number}, key: string): void` + +Remove key-value pairs in the given range with the given key. + +### `update(text: string)` + +Update the current text based on the provided text. + +### `applyDelta(delta: Delta[]): void` + +Change the state of this text by delta. + +If a delta item is `insert`, it should include all the attributes of the +inserted text. Loro's rich text CRDT may make the inserted text inherit some +styles when you use the `insert` method directly. However, when you use +`applyDelta` if some attributes are inherited from CRDT but not included in the +delta, they will be removed. + +Another special property of `applyDelta` is if you format an attribute for +ranges out of the text length, Loro will insert new lines to fill the gap first. +It's useful when you build the binding between Loro and rich text editors like +Quill, which might assume there is always a newline at the end of the text +implicitly. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +const text = doc.getText("text"); +doc.configTextStyle({ bold: { expand: "after" } }); + +text.insert(0, "Hello World!"); +text.mark({ start: 0, end: 5 }, "bold", true); +const delta = text.toDelta(); +const text2 = doc.getText("text2"); +text2.applyDelta(delta); +expect(text2.toDelta()).toStrictEqual(delta); +``` + +### `subscribe(f: (event: Listener)): number` + +This method returns a number that can be used to remove the subscription. + +The text event is in `Delta[]` format. It can be used to bind the rich +text editor. It has the same type as the arg of `applyDelta`, so the following +example works: + +```ts no_run twoslash +import { LoroDoc, TextDiff } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +(async () => { + const doc1 = new LoroDoc(); + doc1.configTextStyle({ + link: { expand: "none" }, + bold: { expand: "after" }, + }); + const text1 = doc1.getText("text"); + const doc2 = new LoroDoc(); + doc2.configTextStyle({ + link: { expand: "none" }, + bold: { expand: "after" }, + }); + const text2 = doc2.getText("text"); + text1.subscribe((e) => { + for (const event of e.events) { + const d = event.diff as TextDiff; + text2.applyDelta(d.diff); + } + }); + text1.insert(0, "foo"); + text1.mark({ start: 0, end: 3 }, "link", true); + doc1.commit(); + await new Promise((r) => setTimeout(r, 1)); + expect(text2.toDelta()).toStrictEqual(text1.toDelta()); + text1.insert(3, "baz"); + doc1.commit(); + await new Promise((r) => setTimeout(r, 1)); + expect(text2.toDelta()).toStrictEqual([ + { insert: "foo", attributes: { link: true } }, + { insert: "baz" }, + ]); + expect(text2.toDelta()).toStrictEqual(text1.toDelta()); + text1.mark({ start: 2, end: 5 }, "bold", true); + doc1.commit(); + await new Promise((r) => setTimeout(r, 1)); + expect(text2.toDelta()).toStrictEqual(text1.toDelta()); +})(); +``` + + +# FILE: pages/docs/tutorial/tips.mdx + +# Tips and Tricks + +##### `LoroDoc` will be initialized with a new random PeerID each time + +
+What if I need to set the initial state? + +If your document requires an initial state, you should not edit the document to achieve this state right +after creating it with new LoroDoc(). This approach can cause problems - each time someone opens the document, +new operations with different PeerIDs would be added just to set up the initial state. + +The better approach is to initialize your document by loading the same Snapshot. This ensures all users start +from an identical baseline without generating unnecessary operations. + +
+ +--- + +##### Be careful when using `doc.setPeerId(newId)` + +When using `setPeerId`, you must avoid having two parallel peers use the same PeerId. This can cause serious consistency problems in your application. + +If you plan to reuse a PeerId across sessions, make sure that you persist the document's local state together with that PeerId and load it before applying any remote updates. Otherwise, the same `peerId + counter` combination could end up referring to two different operations, producing divergence that cannot be resolved automatically. + +
+Why + +It's because Loro determines whether an operation has already been included by checking its operation ID. Since operation IDs are composed of `PeerId + Counter`, duplicate PeerIds can easily lead to duplicate operation IDs. During synchronization, Loro might incorrectly assume certain operations have already been processed, resulting in document inconsistency across peers. + +
+ +
+How to reuse PeerIds safely + +Be careful when reusing PeerIds (this optimization is often unnecessary). You should not assign a fixed PeerId to a user, as one user might use multiple devices. Similarly, you shouldn't assign a fixed PeerId to a device, because even on a single browser, multiple tabs might open the same document simultaneously. + +If you must reuse PeerIds, you need to carefully manage your local PeerId cache with proper locking mechanisms. This would allow only one tab to "take" a specific PeerId, while other tabs use random IDs. The PeerId should be returned to the cache when no longer in use. + +In addition to locking, avoid having multiple browser tabs (or apps on different processes) use the same PeerId concurrently. Coordinating access in this way ensures that a reused PeerId never emits two different operations with the same operation ID. + +
+ +--- + +##### Root containers don't need operations to be initialized + +Root Containers are created implicitly in Loro. This means that when you call `doc.getText("text")`, no new operations appear in the LoroDoc history, and there are no operations that need to be synchronized with other peers. + +This behavior contrasts with non-root containers. For example, when you execute `doc.getMap("meta").setContainer("text", new LoroText())`, it generates an operation to insert the LoroText container into the map. + +--- + +##### When initializing child containers of LoroMap in parallel, use mergeable containers when you want one shared child. + +
+Why this happens + +This happens because parallel creation of regular child containers results in different container IDs, preventing automatic merging of their contents. When a container holds substantial data or serves as the primary storage for document content, overwriting it can lead to unintended hiding or loss of critical information. + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const map = doc.getMap("map"); + +// Parallel initialization of child containers +const docB = doc.fork(); +const textA = doc.getMap("map").setContainer("text", new LoroText()); +textA.insert(0, "A"); +const textB = docB.getMap("map").setContainer("text", new LoroText()); +textB.insert(0, "B"); + +doc.import(docB.export({ mode: "update" })); +// Result: Either { "meta": { "text": "A" } } or { "meta": { "text": "B" } } +``` + +
+ +
+Best practices for container initialization + +1. If peers may lazily create the same child under the same Map key, use a mergeable child container: + + ```ts no_run + const entries = doc.getMap("days").ensureMergeableList("2026-06-08"); + entries.insert(0, "meeting notes"); + ``` + + The same pattern is available through `ensureMergeableText`, `ensureMergeableMap`, `ensureMergeableMovableList`, `ensureMergeableTree`, and `ensureMergeableCounter`. + +2. If the child structure is fixed and known ahead of time, initialize all child containers during the map container's initialization. + +3. If the child naturally has a unique global name, a root container is also safe. You can use `doc.getMap("user." + userId)` instead of `doc.getMap("user").getOrCreateContainer(userId, new LoroMap())`. + +4. Keep using `setContainer` or `insertContainer` when each creation should produce a distinct child object, or when you are modeling replacement rather than shared initialization. +
+ +--- + +##### Use `diff` + `applyDiff` to share squash-like change sets + +For PR-style workflows, compute a diff from the base version to the new version, then apply it back. The diff is compact: operations that cancel out (e.g., insert then delete) are compressed away. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const baseDoc = new LoroDoc(); +baseDoc.getText("text").insert(0, "hello world"); + +const newDoc = baseDoc.fork(); +const text = newDoc.getText("text"); +text.insert(0, "abc"); +text.delete(0, 4); // cancels most of the insert + +const diff = newDoc.diff(baseDoc.frontiers(), newDoc.frontiers()); +baseDoc.applyDiff(diff); + +console.log(diff); +// [ +// [ +// "cid:root-text:Text", +// { type: "text", diff: [ { delete: 1 } ] } +// ] +// ] +console.log(baseDoc.toJSON()); // { text: "ello world" } +``` + +--- + +##### Use redaction to safely share document history + +There are times when users might accidentally paste sensitive information (like API keys, passwords, or personal data) into a collaborative document. When this happens, you need a way to remove just that sensitive content from the document history without compromising the rest of the document's integrity. + +
+How to safely redact sensitive content + +Loro provides a `redactJsonUpdates` function that allows you to selectively redact operations within specific version ranges. + +For example, if a user accidentally pastes a password or API key into a document: + +```ts no_run twoslash +import { LoroDoc, redactJsonUpdates } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("1"); + +// Create some content to be redacted +const text = doc.getText("text"); +text.insert(0, "Sensitive information"); +doc.commit(); + +const map = doc.getMap("map"); +map.set("password", "secret123"); +map.set("public", "public information"); +doc.commit(); + +// Export JSON updates +const jsonUpdates = doc.exportJsonUpdates(); + +// Define version range to redact (redact the text content) +const versionRange = { + "1": [0, 21], // Redact the "Sensitive information" +}; + +// Apply redaction +const redactedJson = redactJsonUpdates(jsonUpdates, versionRange); + +// Create a new document with redacted content +const redactedDoc = new LoroDoc(); +redactedDoc.importJsonUpdates(redactedJson); + +// The text content is now redacted with replacement characters +console.log(redactedDoc.getText("text").toString()); +// Outputs: "���������������������" + +// You can also redact specific map entries +const versionRange2 = { + "1": [21, 22], // Redact the "secret123" password +}; + +const redactedJson2 = redactJsonUpdates(jsonUpdates, versionRange2); +const redactedDoc2 = new LoroDoc(); +redactedDoc2.importJsonUpdates(redactedJson2); + +console.log(redactedDoc2.getMap("map").get("password")); // null +console.log(redactedDoc2.getMap("map").get("public")); // "public information" +``` + +This approach is safer than manually editing document content because: + +1. It maintains document structure and CRDT consistency +2. It keeps key metadata like operation IDs and dependencies intact +3. It allows concurrent editing to continue working after redaction +4. It selectively redacts only specific operations, not the entire document + +The redaction process follows these rules: + +- Preserves delete, tree move, and list move operations +- Replaces text insertion content with Unicode replacement characters '�' +- Substitutes list and map insert values with null +- Maintains structure of child containers +- Replaces text mark values with null +- Preserves map keys and text annotation keys + +**Important**: Your application needs to ensure that all peers receive the redacted version, otherwise the original document with sensitive information will still exist on other peers. + +
+ +--- + +##### Use shallow snapshots to completely remove old history + +When you need to completely remove ALL history older than a certain version point, shallow snapshots provide the solution. + +
+How to remove old history with shallow snapshots + +Shallow snapshots create a new document that preserves the current state but completely eliminates all history before a specified point, similar to Git's shallow clone functionality. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId("1"); + +// Old history - will be completely removed +const text = doc.getText("text"); +text.insert(0, "This document has a long history with many edits"); +doc.commit(); +text.insert(0, "Including some potentially sensitive information. "); +doc.commit(); + +// More recent history - will be preserved +text.delete(11, 55); // Remove the middle part +text.insert(11, "with sanitized history"); +doc.commit(); + +// Create a sanitized version that removes ALL history before current point +const sanitizedSnapshot = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.oplogFrontiers(), +}); + +// Create a new document from the sanitized snapshot +const sanitizedDoc = new LoroDoc(); +sanitizedDoc.import(sanitizedSnapshot); + +// The document has the final state +console.log(sanitizedDoc.getText("text").toString()); +// Outputs: "Including with sanitized history" + +// But ALL history before the snapshot point is completely removed +console.log(sanitizedDoc.isShallow()); // true +console.log(sanitizedDoc.shallowSinceFrontiers()); // Shows the starting point +``` + +This approach is useful for: + +1. Completely removing all old history that might contain various sensitive information +2. Significantly reducing document size by eliminating unnecessary history +3. Creating clean document instances after certain milestones +4. Ensuring old operations cannot be recovered or examined + +Compared to redaction: + +- Shallow snapshots completely remove all operations before a version point +- Redaction selectively replaces just specific content with placeholders + +**Important**: While both methods maintain future synchronization consistency, your application must distribute the sanitized document to all peers. Otherwise, the original document with sensitive information will still exist on other clients. + +**When to use each approach**: + +- Use **redaction** when you need to sanitize specific operations (like an accidental password paste) while preserving older history +- Use **shallow snapshots** when you want to completely eliminate all history before a certain point + +
+ +--- + +##### You can store mappings between LoroDoc's peerIds and user IDs in the document itself + +Use `doc.subscribeFirstCommitFromPeer(listener)` to associate peer information with user identities when a peer first interacts with the document. + +
+How to track peer-to-user mappings + +This functionality is essential for building user-centric features in collaborative applications. You often need bidirectional mapping between user IDs and peer IDs: + +- **Finding all edits by a user**: When you need to retrieve all document edits made by a specific user ID, you must first find all peer IDs associated with that user +- **Showing edit attribution**: When displaying which user edited a piece of text, you need to map from the peer ID (stored in the operation) back to the user ID for display + +This hook provides an ideal point to associate peer information (such as author identity) with the document. The listener is triggered on the first commit from each peer, allowing you to store user metadata within the document itself. + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +doc.setPeerId(0); +doc.subscribeFirstCommitFromPeer((e) => { + doc.getMap("users").set(e.peer, "user-" + e.peer); +}); +doc.getList("list").insert(0, 100); +doc.commit(); +await Promise.resolve(); +expect(doc.getMap("users").get("0")).toBe("user-0"); +``` + +This approach allows you to: + +1. Automatically track which peers have contributed to the document +2. Store user metadata (names, emails, etc.) alongside the document +3. Build features like author attribution, presence indicators, or edit history + +The mapping is stored within the document, so it automatically synchronizes across all peers and persists with the document's state. + +
+ +--- + +##### You can use https://loro.dev/llms-full.txt to prompt your AI + +When working with AI assistants or language models on Loro-related tasks, you can use these URLs to provide comprehensive context about Loro's capabilities and API: + +- `https://loro.dev/llms-full.txt` - All the documentation in one file +- `https://loro.dev/llms.txt` - An overview of Loro website + + +# FILE: pages/docs/tutorial/event.mdx + +# Event Handling in Loro + +Loro implements an event system to track changes in the document. This section +explains when events are emitted and how transactions work in Loro. + +## Event Emission Points + +Events in Loro are emitted whenever the internal document state changes. This +mechanism allows application-level derived states to automatically synchronize +with changes in the document state. + +1. **Local Operations**: For local operations (like insertions or deletions on + text), the operations are first placed in a pending state within an internal + transaction. + +2. **Transaction Commit**: When a transaction is committed, all pending + operations collectively emit their corresponding events. This transaction + commit occurs in two scenarios: + - When `LoroDoc.commit()` is explicitly called + - Automatically before an import or export operation + +Starting from `loro-crdt@1.8.0`, events are emitted synchronously during the commit cycle. If you are using an older version (`<=1.7.x`), you will still need to await a microtask for the callbacks to fire. + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.subscribe((event) => { + console.log("Event:", event); +}); + +const text = doc.getText("text"); +text.insert(0, "Hello"); +doc.commit(); +// Event has already been emitted synchronously +``` + +3. **Import**: When importing changes from a remote source using the `import()` + method, respective events are emitted. This allows the local document to + react to changes made by other peers. + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +declare const remoteChanges: Uint8Array; +// ---cut--- +const doc = new LoroDoc(); +doc.subscribe((event) => { + console.log("Event:", event); +}); + +doc.import(remoteChanges); // This immediately triggers events (v1.8+) +``` + +4. **Version Checkout**: When you switch document state to a different version + using `doc.checkout(frontiers)`, Loro emits an event to reflect this change. + As of v1.8, checkout events fire synchronously alongside the state change. + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const frontiers = doc.frontiers(); +doc.checkout(frontiers); // This triggers events immediately (v1.8+) +``` + +## Transaction Behavior + +Transactions in Loro primarily serve to bundle related operations and emit their +events together as a cohesive unit. This is useful in several scenarios: + +1. **Related Local Operations**: When performing multiple local operations that + are logically connected, you may want them to: + - Share the same commit message + - Have the same timestamp + - Move together during undo/redo operations + +2. **Event Handling**: Applications often benefit from receiving related changes + as a single batch rather than individual updates. Transactions facilitate + this by: + - Allowing you to set an origin identifier during commit + - Including this origin value in the emitted events + - Enabling better event filtering and processing based on the origin + + + +## Triggering a Commit + +There are several ways to trigger a commit: + +1. **Explicit Commit**: Directly calling the `commit()` method on the Loro + document. + + ```ts twoslash + import { LoroDoc } from "loro-crdt"; + // ---cut--- + const doc = new LoroDoc(); + const text = doc.getText("myText"); + text.insert(0, "Hello, Loro!"); + doc.commit(); // This commits pending operations and emits events + ``` + +2. **Before Import/Export**: A commit is automatically triggered before + executing an import operation. + + ```ts twoslash + import { LoroDoc } from "loro-crdt"; + // ---cut--- + const doc1 = new LoroDoc(); + doc1.setPeerId(1); + const doc2 = new LoroDoc(); + doc2.setPeerId(2); + + // Some ops on doc1 and doc2 + doc1.getText("text").insert(0, "Alice"); + doc2.getText("text").insert(0, "Hello, Loro!"); + console.log(doc1.version().toJSON()); // Map(0) {} + console.log(doc2.version().toJSON()); // Map(0) {} + const updates = doc1.export({ mode: "snapshot" }); + doc2.import(updates); // This first commits any pending operations in doc2 + console.log(doc2.version().toJSON()); // Map(2) { "1" => 5, "2" => 12 } + console.log(doc1.version().toJSON()); // Map(2) { "1" => 5 } + ``` + +## Transactions in Loro + +It's important to note that Loro's concept of a transaction differs from +traditional database transactions: + +- Loro transactions do not have ACID properties. +- They primarily serve as event wrappers. +- There is no rollback mechanism if an operation fails. + + +# FILE: pages/docs/tutorial/version.mdx + +# Version + +In centralized environments, we can use linear version numbers to represent a version, such as incrementing a number each time or using timestamps. However, CRDTs can be used in decentralized environments, and their version representation is different. + +In Loro, you can express a document's version through a [Version Vector](/docs/concepts/version_vector) or [Frontiers](/docs/concepts/frontiers). + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.version(); // State Version vector +doc.oplogVersion(); // OpLog Version vector +doc.frontiers(); // State Frontiers +doc.oplogFrontiers(); // OpLog Frontiers +``` + +In most cases, you might only need the Version Vector, which can be used for data synchronization and version comparison. + +## Learn More + +- [Version Vector](/docs/concepts/version_vector) - Complete peer state tracking for synchronization +- [Frontiers](/docs/concepts/frontiers) - Compact version representation for checkpoints +- [Version Deep Dive](/docs/advanced/version_deep_dive) - Technical details about DAG, causality, and version implementations + + +# FILE: pages/docs/tutorial/get_started.mdx + +--- +keywords: "loro-crdt, build collaboration software, local-first, operation transform, crdts, ot" +description: "How to use Loro to build real-time or asynchronous collaboration software." +--- + +# Getting Started + +You can use Loro in your application by using: + +- [`loro-crdt`](https://www.npmjs.com/package/loro-crdt) NPM package +- [`loro`](https://crates.io/crates/loro) Rust crate +- [`loro-swift`](https://github.com/loro-dev/loro-swift) Swift package +- [`loro-py`](https://github.com/loro-dev/loro-py) Python package +- [`loro-cs`](https://github.com/sensslen/loro-cs) Community-maintained C# package +- You can also find a list of examples in + [Loro examples in Deno](https://github.com/loro-dev/loro-examples-deno). +- [`loro-go`](https://github.com/aholstenson/loro-go) Community-maintained Go package +- [`loro-go`](https://github.com/Deln0r/loro-go) Community-maintained pure-Go package (no cgo) + +You can use [Loro Inspector](/docs/advanced/inspector) to debug and visualize the state and history of Loro documents. + +The following guide will use `loro-crdt` js package as the example. + +[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/edit/loro-basic-test?file=test%2Floro-sync.test.ts) + +## Install + +```bash +npm install loro-crdt + +# Or +pnpm install loro-crdt + +# Or +yarn add loro-crdt +``` + +If you're using `Vite`, you should add the following to your vite.config.ts: + +```ts no-run +import wasm from "vite-plugin-wasm"; +import topLevelAwait from "vite-plugin-top-level-await"; + +export default defineConfig({ + plugins: [...otherConfigures, wasm(), topLevelAwait()], +}); +``` + +
+⚠️ DOMContentLoaded Timing Issue with Vite + +When using Loro with Vite, be aware of module loading timing issues with DOM events: + +**Problem:** The following code will cause nothing to load on the screen: + +```ts no_run +import { LoroDoc } from "loro-crdt"; + +document.addEventListener("DOMContentLoaded", () => { + const doc = new LoroDoc(); + // Your code here... +}); +``` + +**Reason:** This occurs because Vite loads ES modules asynchronously, and the WASM module initialization within `loro-crdt` also happens asynchronously. When you import at the top level but execute code inside `DOMContentLoaded`, the WASM module may not be fully initialized when the event fires, causing the application to fail silently. + +**Solutions:** + +1. **Remove the event listener** (recommended for most cases): + + ```ts no_run twoslash + import { LoroDoc } from "loro-crdt"; + + const doc = new LoroDoc(); + // Your code here... + ``` + +2. **Use dynamic import inside the event listener**: + ```ts no_run twoslash + document.addEventListener("DOMContentLoaded", async () => { + const { LoroDoc } = await import("loro-crdt"); + const doc = new LoroDoc(); + // Your code here... + }); + ``` + +The dynamic import ensures the module and its WASM dependencies are fully loaded before use. + +
+ +If you're using `Next.js`, you should add the following to your next.config.js: + +```js no-run +module.exports = { + webpack: function (config) { + config.experiments = { + layers: true, + asyncWebAssembly: true, + }; + return config; + }, +}; +``` + +You can also use Loro directly in the browser via ESM imports. Here's a minimal +example: + +```html + + + + + + ESM Module Example + + + +
+ + + +``` + +## Introduction + +It is well-known that syncing data/building realtime collaborative apps is +challenging, especially when devices can be offline or part of a peer-to-peer +network. Loro simplifies this process for you. + +After you model your app state by Loro, syncing is simple: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +const docA = new LoroDoc(); +const docB = new LoroDoc(); + +//...operations on docA and docB + +// Assume docA and docB are two Loro documents in two different devices +const bytesA = docA.export({ mode: "update" }); +// send bytes to docB by any method +docB.import(bytesA); +// docB is now updated with all the changes from docA + +const bytesB = docB.export({ mode: "update" }); +// send bytes to docA by any method +docA.import(bytesB); +// docA and docB are now in sync, they have the same state +``` + +Saving your app state is also straightforward: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").insert(0, "Hello world!"); +const bytes = doc.export({ mode: "snapshot" }); +// Bytes can be saved to local storage, database, or sent over the network +``` + +Snapshots and updates include a checksum in their headers, so any corruption from storage or transmission (like bit flips) is detected during import before it can affect your document. + +Loading your app state: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +const bytes = new Uint8Array(); +// ---cut--- +const newDoc = new LoroDoc(); +newDoc.import(bytes); +``` + +Loro also makes it easy for you to time travel the history and add version +control to your app. [Learn more about time travel](/docs/tutorial/time_travel). + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const version = doc.frontiers(); +// ---cut--- +doc.checkout(version); // Checkout the doc to the given version +``` + +Loro is compatible with the JSON schema. If you can model your app state with +JSON, you probably can sync your app with Loro. Because we need to adhere to the +JSON schema, using a number as a key in a Map is not permitted, and cyclic links +should be avoided. + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +// ---cut--- +doc.toJSON(); // Get the JSON representation of the doc +``` + +## Entry Point: LoroDoc + +LoroDoc is the entry point for using Loro. You must create a Doc to use Map, +List, Text, and other types and to complete data synchronization. + +```ts twoslash +import { LoroDoc, LoroText } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +const text: LoroText = doc.getText("text"); +text.insert(0, "Hello world!"); +console.log(doc.toJSON()); // { "text": "Hello world!" } +``` + +## Container + +We refer to CRDT types such as `List`, `Map`, `Tree`, `MovableList`, and `Text` +as `Container`s. + +Here are their basic operations: + +```ts twoslash +import { LoroDoc, LoroList, LoroMap, LoroText } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +const list: LoroList = doc.getList("list"); +list.insert(0, "A"); +list.insert(1, "B"); +list.insert(2, "C"); + +const map: LoroMap = doc.getMap("map"); +// map can only has string key +map.set("key", "value"); +expect(doc.toJSON()).toStrictEqual({ + list: ["A", "B", "C"], + map: { key: "value" }, +}); + +// delete 2 element at index 0 +list.delete(0, 2); +expect(doc.toJSON()).toStrictEqual({ + list: ["C"], + map: { key: "value" }, +}); + +// Insert a text container to the list +const text = list.insertContainer(0, new LoroText()); +text.insert(0, "Hello"); +text.insert(0, "Hi! "); + +expect(doc.toJSON()).toStrictEqual({ + list: ["Hi! Hello", "C"], + map: { key: "value" }, +}); + +// Insert a list container to the map +const list2 = map.setContainer("test", new LoroList()); +list2.insert(0, 1); +expect(doc.toJSON()).toStrictEqual({ + list: ["Hi! Hello", "C"], + map: { key: "value", test: [1] }, +}); +``` + +## Save and Load + +Loro is a pure library and does not handle network protocols or storage mechanisms. It is your responsibility to manage the storage and transmission of the binary data exported by Loro. + +To save the document, use `doc.export({mode: "snapshot"})` to get its binary +form. To open it again, use `doc.import(data)` to load this binary data. + +```ts twoslash +import { LoroDoc, LoroList, LoroMap, LoroText } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").insert(0, "Hello world!"); +const data = doc.export({ mode: "snapshot" }); + +const newDoc = new LoroDoc(); +newDoc.import(data); +expect(newDoc.toJSON()).toStrictEqual({ + text: "Hello world!", +}); +``` + +Exporting the entire document on each keypress is inefficient. Instead, use +`doc.export({mode: "update", from: VersionVector})` to obtain binary data for +operations since the last export. + +```ts twoslash +import { LoroDoc, LoroList, LoroMap, LoroText } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").insert(0, "Hello world!"); +const data = doc.export({ mode: "snapshot" }); +let lastSavedVersion = doc.version(); +doc.getText("text").insert(0, "✨"); +const update0 = doc.export({ mode: "update", from: lastSavedVersion }); +lastSavedVersion = doc.version(); +doc.getText("text").insert(0, "😶‍🌫️"); +const update1 = doc.export({ mode: "update", from: lastSavedVersion }); + +{ + /** + * You can import the snapshot and the updates to get the latest version of the document. + */ + + // import the snapshot + const newDoc = new LoroDoc(); + newDoc.import(data); + expect(newDoc.toJSON()).toStrictEqual({ + text: "Hello world!", + }); + + // import update0 + newDoc.import(update0); + expect(newDoc.toJSON()).toStrictEqual({ + text: "✨Hello world!", + }); + + // import update1 + newDoc.import(update1); + expect(newDoc.toJSON()).toStrictEqual({ + text: "😶‍🌫️✨Hello world!", + }); +} + +{ + /** + * You may also import them in a batch + */ + const newDoc = new LoroDoc(); + newDoc.importUpdateBatch([update1, update0, data]); + expect(newDoc.toJSON()).toStrictEqual({ + text: "😶‍🌫️✨Hello world!", + }); +} +``` + +If updates accumulate, exporting a new snapshot can quicken import times and +decrease the overall size of the exported data. + +You can store the binary data exported from Loro wherever you prefer. + +## Sync + +Two documents with concurrent edits can be synchronized by just two message +exchanges. + +Below is an example of synchronization between two documents: + +```ts twoslash +import { LoroDoc, LoroList, LoroMap, LoroText } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +const docA = new LoroDoc(); +const docB = new LoroDoc(); +const listA: LoroList = docA.getList("list"); +listA.insert(0, "A"); +listA.insert(1, "B"); +listA.insert(2, "C"); +// B import the ops from A +const data: Uint8Array = docA.export({ mode: "update" }); +// The data can be sent to B through the network +docB.import(data); +expect(docB.toJSON()).toStrictEqual({ + list: ["A", "B", "C"], +}); + +const listB: LoroList = docB.getList("list"); +listB.delete(1, 1); + +// `doc.export({mode: "update", from: version})` can encode all the ops from the version to the latest version +// `version` is the version vector of another document +const missingOps = docB.export({ + mode: "update", + from: docA.oplogVersion(), +}); +docA.import(missingOps); + +expect(docA.toJSON()).toStrictEqual({ + list: ["A", "C"], +}); +expect(docA.toJSON()).toStrictEqual(docB.toJSON()); +``` + +## Event + +You can subscribe to the event from `Container`s. + +`LoroText` and `LoroList` can receive updates in +[Quill Delta](https://quilljs.com/docs/delta/) format. + +The events will be emitted after a transaction is committed. A transaction is +committed when: + +- `doc.commit()` is called. +- `doc.export(mode)` is called. +- `doc.import(data)` is called. +- `doc.checkout(version)` is called. + +Below is an example of rich text event: + +```ts twoslash +import { LoroDoc, LoroList, LoroMap, LoroText } from "loro-crdt"; +import { expect } from "expect"; +// ---cut--- +// The code is from https://github.com/loro-dev/loro-examples-deno +const doc = new LoroDoc(); +const text = doc.getText("text"); +text.insert(0, "Hello world!"); +doc.commit(); +let ran = false; +text.subscribe((e) => { + for (const event of e.events) { + if (event.diff.type === "text") { + expect(event.diff.diff).toStrictEqual([ + { + retain: 5, + attributes: { bold: true }, + }, + ]); + ran = true; + } + } +}); +text.mark({ start: 0, end: 5 }, "bold", true); +doc.commit(); +await new Promise((r) => setTimeout(r, 1)); +expect(ran).toBeTruthy(); +``` + +The types of events are defined as follows: + +```ts twoslash +import { Path, Diff, Frontiers, ContainerID } from "loro-crdt"; + +export interface LoroEventBatch { + /** + * How the event is triggered. + * + * - `local`: The event is triggered by a local transaction. + * - `import`: The event is triggered by an import operation. + * - `checkout`: The event is triggered by a checkout operation. + */ + by: "local" | "import" | "checkout"; + origin?: string; + /** + * The container ID of the current event receiver. + * It's undefined if the subscriber is on the root document. + */ + currentTarget?: ContainerID; + events: LoroEvent[]; + from: Frontiers; + to: Frontiers; +} + +/** + * The concrete event of Loro. + */ +export interface LoroEvent { + /** + * The container ID of the event's target. + */ + target: ContainerID; + diff: Diff; + /** + * The absolute path of the event's emitter, which can be an index of a list container or a key of a map container. + */ + path: Path; +} +``` + + +# FILE: pages/docs/index.mdx + +## Introduction to Loro + +It is well-known that syncing data/building realtime collaborative apps is +challenging, especially when devices can be offline or part of a peer-to-peer +network. Loro simplifies this process for you. + +We want to provide better DevTools to make building +[local-first apps](https://www.inkandswitch.com/local-first/) easy and +enjoyable. + +Loro uses [Conflict-free Replicated Data Types (CRDTs)](/docs/concepts/crdt) to +resolve parallel edits. By utilizing Loro's data types, your applications can be +made collaborative and keep the editing history with low overhead. + +After you model your app state by Loro, syncing is simple: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +const docA = new LoroDoc(); +const docB = new LoroDoc(); +docA.getText("text").insert(0, "Hello world!"); +docB.getText("text").insert(0, "Hi!"); +// Assume docA and docB are two Loro documents in two different devices +const bytesA = docA.export({ mode: "update" }); +// send bytes to docB by any method +docB.import(bytesA); +// docB is now updated with all the changes from docA + +const bytesB = docB.export({ mode: "update" }); +// send bytes to docA by any method +docA.import(bytesB); +// docA and docB are now in sync, they have the same state +``` + +Saving your app state is also straightforward: + +```ts twoslash +import { LoroDoc } from "loro-crdt"; +// ---cut--- +const doc = new LoroDoc(); +doc.getText("text").insert(0, "Hello world!"); +const bytes = doc.export({ mode: "snapshot" }); +// Bytes can be saved to local storage, database, or sent over the network +``` + +Loading your app state: + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +const bytes = new Uint8Array([1, 2, 3]); +// ---cut--- +const newDoc = new LoroDoc(); +newDoc.import(bytes); +``` + +Loro also makes it easy for you to time travel the history and add version +control to your app. [Learn more about time travel](/docs/tutorial/time_travel). + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +const version = doc.frontiers(); +// ---cut--- +doc.checkout(version); // Checkout the doc to the given version +``` + +Loro is compatible with the JSON schema. If you can model your app state with +JSON, you probably can sync your app with Loro. Because we need to adhere to the +JSON schema, using a number as a key in a Map is not permitted, and cyclic links +should be avoided. + +```ts no_run twoslash +import { LoroDoc } from "loro-crdt"; +const doc = new LoroDoc(); +// ---cut--- +doc.toJSON(); // Get the JSON representation of the doc +``` + +import { Cards } from "nextra/components"; + + + + <>![Getting started](/images/GettingStarted.png) + + + +## Is Loro Right for You? + +### ✅ Use Loro when you need: + +- Real-time collaboration on documents +- Automatic conflict resolution for concurrent edits +- Offline editing with later synchronization +- Complete edit history and time travel +- P2P synchronization capabilities + +### ⚠️ Consider alternatives when: + +- Your application requires strong consistency +- Your data isn't JSON-like (e.g., large binary/media streaming) +- Simple client–server sync is sufficient (e.g., basic WebSockets) +- Your application is sensitive to bundle size (Loro WASM binary ~970KB gzipped) + +[Learn more about when not to use CRDTs →](/docs/concepts/when_not_crdt) + +## Differences from other CRDT libraries + +The table below summarizes Loro's features, which may not be present in other +CRDT libraries. + +| Features/Important design decisions | Loro | Diamond-types | Yjs | Automerge | +| :-------------------------------------------------------------------------- | :--- | :------------ | :---------- | :---------- | +| [Event Graph Walker](https://loro.dev/docs/advanced/replayable_event_graph) | ✅ | ✅ Inventor | ❌ | ❌ | +| Rich Text CRDT | ✅ | ❌ | ❌ | ✅ | +| [Movable Tree](https://ieeexplore.ieee.org/document/9563274) | ✅ | ❌ | ❌ | ❌ Inventor | +| [Movable List](https://loro.dev/docs/tutorial/list) | ✅ | ❌ | ❌ | ❌ Inventor | +| Time Travel | ✅ | ✅ | ✅[1] | ✅ | +| [Fugue](https://arxiv.org/abs/2305.00583) / Maximal non-interleaving | ✅ | ✅ | ❌ | ❌ | +| JSON Types | ✅ | ❓ | ✅ | ✅ | +| [Mergeable Containers](/blog/mergeable-containers) | ✅ | ❌ | ❌ | ❌ | +| Merging Elements in Memory by Run Length Encoding | ✅ | ✅ | ✅ Inventor | ❌ | +| Byzantine-fault-tolerance | ❌ | ❌ | ❌ | ✅ | +| Version Control | ✅ | ❌ | ❌ | ✅ | + +- [1] Unlike others, Yjs requires users to store a version vector and a delete + set, enabling time travel back to a specific point. +- [Fugue](https://arxiv.org/abs/2305.00583) is a text/list CRDTs that can + minimize the chance of the interleaving anomalies. + + +# FILE: pages/changelog/v1.8.0.mdx + +--- +version: "v1.8.0" +title: "Release Loro v1.8.0" +date: 2025/09/22 +--- + +Events now fire **synchronously** + +In the JavaScript package `loro-crdt@1.8.0`, we changed event emission from **“after a microtask”** to **synchronous** dispatch. This makes event handling simpler and less error‑prone. + +### Why we changed it + +Historically, the JS binding emitted events **after a microtask**. The reason was to avoid Rust borrow/aliasing issues: if an event triggered inside `doc.commit()` re‑entered the same `doc`, Rust could panic because `commit()` holds a borrow and reusing the object would violate borrowing rules. Deferring events to a microtask avoided that re‑entrancy. (Background: Loro’s JS API exposes subscriptions via `doc.subscribe(...)`; the docs note that events used to arrive after a microtask. ([Loro][1])) + +However, this deferral made app logic fragile. There was a window between the mutation and the event callback. Example: the app receives the event from CRDT “delete the 3rd character of `Hi!`”, but during the microtask gap the app state changed `Hi!` → `Hi`. When the deferred event arrives, applying a “delete index 2” delta can be wrong. In practice, users had to maintain an awkward invariant: **don’t mutate the app state during that microtask** if you consume delta updates. + +### What’s new in 1.8.0 + +Events are now **dispatched synchronously**—that is, your listeners run **before** the top‑level JS call (e.g., `doc.commit()`) returns. To keep this safe with Rust’s borrowing: + +- We keep a **global queue** of “pending (listener, event)” pairs. +- JS wrappers decorate APIs that can borrow the Rust doc (e.g., `commit`, import/export, checkout). +- The wrapper calls into WASM/Rust, **returns from the borrow**, then **flushes the queue** and clears it. +- Because callbacks run **outside** the borrowed region, listeners can freely call `doc` APIs without triggering borrow violations. + +### What this means for your app + +- **Simpler state updates.** No more microtask gap; event ordering matches your mutation order. +- **Fewer footguns with deltas.** You no longer need to uphold “don’t touch the doc during the microtask” when applying delta‑style updates. +- **Listeners can still safely use `doc`.** + + +# FILE: pages/changelog/v1.9.0.mdx + +--- +version: "v1.9.0" +title: "Release Loro v1.9.0" +date: 2025/11/10 +--- + +## Highlights + +- JSONPath queries are now RFC 9535 compliant across both Rust and WASM bindings. A new parser, evaluator, and conformance test suite (PR #848, commit `21d13218`) unlock richer filters like `in` and `contains` while delivering clearer error messages. Thanks @zolero! +- Source maps become first-class in the WASM toolchain. `loro-wasm-tools` now embeds symbols during bundling (#836) and the new `loro-wasm-map` package publishes maps next to the runtime bundle (#844), drastically simplifying debugging in browsers, Vite, and other build systems. + +## Breaking change + +- **Removed legacy v0.x encoding path (#849, `10a405b4`)**. Legacy serializers, compatibility fuzz tests, and decoding shims are gone. All runtimes now rely on the v1.* encoding format. Documents created with v0.x must be migrated via an intermediate ≤1.8.x release before installing 1.9.0. + +## New features & improvements + +- **Spec-compliant JSONPath (#848, `21d13218`)**. Ships a Pest-based parser, richer AST, the `in`/`contains` filter operators, better existence checks, and shared tests/benchmarks to keep Rust and WASM perfectly aligned. +- **WASM debugging experience (#836, `6f987022`; #844, `53f55331`)**. Adds `loro-wasm-tools` for embedding sourcemaps, automates sourcemap publishing in CI, and hosts the maps in a dedicated workspace package so source-level debugging works without manual steps. +- **Better bundler support (#852 `3af6a857`, #851 `366f0161`)**. Patches the JS entry points so esbuild and rsbuild can import the WASM bundle with zero loader overrides. +- **Bun runtime improvements (#829 `b8c070fd`, #828 `cf123453`, #827 `8b622619`, #834 `865bccba`)**. Streams the WASM binary during bundling, pre-seeds the externref table, integrates Bun into CI, and removes earlier hacks that hid loader bugs. +- **JavaScript performance (#820 `c2c535c1`)**. Event conversion now runs exactly once per callback invocation, trimming overhead during heavy real-time editing sessions. +- **Core dependency refresh (`e8f79de8`, `b2f5e107`)**. Upgrades `generic-btree` and the columnar storage crate to pick up the latest bug fixes and perf wins. + +## Bug fixes & stability + +- Fixed tree undo operations that moved nodes between siblings (#821 `76a8728e`). +- Resolved container ID bookkeeping when exporting shallow snapshots (#823 `b72a759a`) and ensured pending containers now return `None` instead of leaking state (#840 `35d9064d`). +- Prevented `LoroMap` entries from turning into `null` after `applyDiff` (#825 `3afc4d52`). +- Ensured undo manager callbacks fire without tripping Rust aliasing violations (#831 `a39daf85`). +- Guarded against panics when cursors point to deleted entries (#835 `e97e6056`) and when fetching unknown cursors in JS integration tests. +- Cleaned up WASM loader glue so we no longer rely on an extra patch (#834 `865bccba`) and added multiple Bun regression tests (#828 `cf123453`). +- Tightened CI release scripts (`74b78514`, `2383cdc1`, `4740a04c`, `3373e046`, `24f93249`) to keep `loro-wasm` artifacts and sourcemaps publishing in sync. + + +# FILE: pages/changelog/v1.2.0.mdx + +--- +version: "v1.2.0" +title: "Release Loro v1.2.0" +date: 2024/12/10 +--- + +## New + +- Add `isDeleted()` method to all container types (Text, Map, List, Tree, etc.) + +### LoroDoc + +- `changeCount()`: Get the number of changes in the oplog. +- `opCount()`: Get the number of ops in the oplog. + +### VersionVector + +- `setEnd(id: ID)`: Set the exclusive ending point. target id will NOT be included by self. +- `setLast(id: ID)`: Set the inclusive ending point. target id will be included. +- `remove(peer: PeerID)`: Remove the specific peer id. +- `length()`: Get the number of peers in the VersionVector. + +## Change + +- Return `ImportStatus` in the `importUpdateBatch` method. +- Fractional index is enabled by default now. + + +## Fix + +- fix: getOrCreateContainer should not throw if value is null [#576](https://github.com/loro-dev/loro/pull/576) +- fix: dead loop when importing updates [#570](https://github.com/loro-dev/loro/pull/570) + + +# FILE: pages/changelog/v1.4.7.mdx + +--- +version: "v1.4.7" +title: "Release Loro v1.4.7" +date: 2025/04/01 +--- + +## New + +- You can get the version of Loro by `LORO_VERSION` +- `setNextCommitOrigin(origin: string)`: Set the origin of the next commit. +- `setNextCommitTimestamp(timestamp: number)`: Set the timestamp of the next commit. +- `setNextCommitOptions(options: CommitOption)`: Set the options of the next commit. +- `clearNextCommitOptions()`: Clear the options of the next commit. +- `configDefaultTextStyle(style: TextStyle)`: Configures the default text style for the document. +- `getUncommittedOpsAsJson()`: Get the pending operations from the current transaction in JSON format + +## Fix + +- fix: memory leak issue [#647](https://github.com/loro-dev/loro/pull/647) +- fix: mark err on detached LoroText [#659](https://github.com/loro-dev/loro/pull/659) +- fix: detached loro text issues [#665](https://github.com/loro-dev/loro/pull/665) +- fix: entity index when the tree is empty [#670](https://github.com/loro-dev/loro/pull/670) + + +# FILE: pages/changelog/v1.5.0.mdx + +--- +version: "v1.5.0" +title: "Release Loro v1.5.0" +date: 2025/04/04 +--- + +## New + +### 1. New Hooks + +`doc.subscribePreCommit(listener)` - Modify commit options before processing: + +This hook is particularly useful because doc.commit() is often invoked implicitly in various methods such as doc.import, doc.export, doc.checkout, and doc.exportJsonUpdates. Without this hook, users attempting to add custom messages to each commit might miss these implicit commit triggers. + +```ts +const doc = new LoroDoc(); +doc.setPeerId(0); +doc.subscribePreCommit((e) => { + e.modifier.setMessage("test").setTimestamp(Date.now()); +}); +doc.getList("list").insert(0, 100); +doc.commit(); +expect(doc.getChangeAt({ peer: "0", counter: 0 }).message).toBe("test"); +``` + +Advanced Example - Creating a Merkle DAG: + +```ts no_run +const doc = new LoroDoc(); +doc.setPeerId(0); +doc.subscribePreCommit((e) => { + const changes = doc.exportJsonInIdSpan(e.changeMeta); + expect(changes).toHaveLength(1); + const hash = crypto.createHash("sha256"); + const change = { + ...changes[0], + deps: changes[0].deps.map((d) => { + const depChange = doc.getChangeAt(idStrToId(d)); + return depChange.message; + }), + }; + hash.update(JSON.stringify(change)); + const sha256Hash = hash.digest("hex"); + e.modifier.setMessage(sha256Hash); +}); + +doc.getList("list").insert(0, 100); +doc.commit(); + +expect(doc.getChangeAt({ peer: "0", counter: 0 }).message).toBe( + "2af99cf93869173984bcf6b1ce5412610b0413d027a5511a8f720a02a4432853", +); +``` + +`doc.subscribeFirstCommitFromPeer(listener)` - Triggers on first peer interaction: + +This hook provides an ideal point to associate peer information (such as author identity) with the document. + +```ts +const doc = new LoroDoc(); +doc.setPeerId(0); +doc.subscribeFirstCommitFromPeer((e) => { + doc.getMap("users").set(e.peer, "user-" + e.peer); +}); +doc.getList("list").insert(0, 100); +doc.commit(); +expect(doc.getMap("users").get("0")).toBe("user-0"); +``` + +### 2. EphemeralStore + +EphemeralStore is a better alternative to Awareness for ephemeral states: + +Awareness is commonly used as a state-based CRDT for handling ephemeral states in real-time collaboration scenarios, such as cursor positions and application component highlights. As application complexity grows, Awareness may be set in multiple places, from cursor positions to user presence. However, the current version of Awareness doesn't support partial state updates, which means even minor mouse movements require synchronizing the entire Awareness state. + +```ts no_run +awareness.setLocalState({ + ...awareness.getLocalState(), + x: 167, +}); +``` +Since Awareness is primarily used in real-time collaboration scenarios where consistency requirements are relatively low, we can make it more flexible. We've introduced EphemeralStore as an alternative to Awareness. Think of it as a simple key-value store that uses timestamp-based last-write-wins for conflict resolution. You can choose the appropriate granularity for your key-value pairs based on your application's needs, and only modified key-value pairs are synchronized. + + +```ts +import { + EphemeralStore, + EphemeralListener, + EphemeralStoreEvent, +} from "loro-crdt"; + +const store = new EphemeralStore(); +// Set ephemeral data +store.set("user-alice", { + anchor: 10, + focus: 20, + user: "Alice" +}); + +// Encode only the data for `loro-prosemirror` +const encoded = store.encode("user-alice") +const newStore = new EphemeralStore(); +newStore.subscribe((e: EphemeralStoreEvent) => { + // Listen to changes from `local`, `remote`, or `timeout` events +}); + +newStore.apply(encoded); +console.log(newStore.get("user-alice")) +// { +// anchor: 10, +// focus: 20, +// user: "Alice" +// } +``` + +## Fix + +- Fixed text styling at end "\n" character +- Added JSON support for current transaction operations +- For environments that support multi-threading such as Rust and Swift, LoroDoc can now be directly and safely + shared and accessed in parallel across multiple threads without triggering the previous WouldBlock panic. + + +# FILE: pages/changelog/v1.0.0-beta.mdx + +--- +version: "v1.0.0" +title: "Release Loro v1.0.0" +date: 2024/10/21 +# breakingChange: false +# category: ["Encoding", "Tree"] +--- + +We are very excited to announce the release of Loro v1.0, a major milestone. + +It has a stable encoding format, faster document import and export speed, better version control capabilities, and a shallow snapshot. For more information, please check [the blog](https://loro.dev/blog/v1.0). + +The following are the specific API changes: + +## New + +### LoroDoc + +- `getChange(id: ID)`: get `ChangeMeta` by `ID`. +- `setDetachedEditing(flag: boolean)`: Enables editing in detached mode, which is disabled by default. +- `isDetachedEditingEnabled()`: Whether the editing is enabled in detached mode. +- `setNextCommitMessage(msg: string)`: Set the commit message of the next commit. +- `shallowSinceVV()`: The doc only contains the history since this version. +- `shallowSinceFrontiers()`: The doc only contains the history since this version. +- `export(mode: ExportMode)`: Export the document based on the specified ExportMode. see more details [here](/docs/tutorial/encoding). +- `getDeepValueWithID()`: Get deep value of the document with container id. +- `subscribeLocalUpdates(callback:(bytes: Uint8Array) => void)`: Subscribe to updates from local edits. +- `getPathToContainer(id: ContainerID)`: Get the path from the root to the container. +- `JSONPath(jsonPath: string)`: Evaluate JSONPath against a LoroDoc. +- `forkAt(frontiers: Frontiers): LoroDoc`: Creates a new LoroDoc at a specified version (Frontiers) +- `getPendingTxnLength():number`: Get the number of operations in the pending transaction. +- `travelChangeAncestors(ids: ID[], callback: (meta: ChangeMeta)->bool)`: Iterate over all changes including the input id in order, and stop iterating if the callback returns false. + +### LoroText + +- `updateByLine(text: string)`: Update the current text based on the provided text line by line like git. + +### LoroList + +- `toArray(): ValueOrContainer[]`: Get elements of the list. If the value is a child container, the corresponding `Container` will be returned. +- `clear()`: Delete all elements in the list. + +### LoroMovableList + +- `toArray(): ValueOrContainer[]`: Get elements of the list. If the value is a child container, the corresponding `Container` will be returned. +- `clear()`: Delete all elements in the list. + +### LoroMap + +- `clear()`: Delete all key-value pairs in the map. + +### LoroTree + +- `enableFractionalIndex(jitter: number)`: Set whether to generate fractional index for Tree Position. +- `disableFractionalIndex()`: Disable the fractional index generation for Tree Position when + you don't need the Tree's siblings to be sorted. The fractional index will be always default. +- `isFractionalIndexEnabled()`: Whether the tree enables the fractional index generation. +- `isNodeDeleted(id: TreeID)`: Return `undefined` if the node is not exist, otherwise return `true` if the node is deleted. +- `getNodes(prop: getNodesProp): LoroTreeNode[]`: Get the flat array of the forest. If `with_deleted` is true, the deleted nodes will be included. + +### UndoManager + +- `clear()`: Clear the Undo and Redo stack of `UndoManager` + +## Changes + +### LoroDoc + +- Move `setFractionalIndexJitter()` to `LoroTree`, you can set whether to enable or disable it for each `Tree Container`. +- `import()`, `importWith()` and `importJsonUpdates` will return `ImportStatus` for indicating which ops have been successfully applied and which ops are pending. +- New Subscription for event. +- In Loro 1.0, `doc.version()` `doc.frontiers()` `doc.oplogVersion()` and `doc.oplogFrontiers()` even if ops has not been committed, it indicates the latest version of all operations. +- rename `Loro` to `LoroDoc`. + +### LoroTree + +- `contains(id: TreeID)`: Return true even if the node exists in the internal state and has been deleted. +- `nodes()`: deleted nodes will be included now, you can use `isDeleted()` to filter. +- `toJSON()`: Now use the hierarchical approach to express the tree structure. + +## Deprecation + +### LoroDoc + +- `exportFrom(version)` and `exportSnapshot()` are deprecated, use `export(mode: ExportMode)` instead. + + +# FILE: pages/changelog/v1.4.0.mdx + +--- +version: "v1.4.0" +title: "Release Loro v1.4.0" +date: 2025/02/13 +--- + +## New + +- add `unsubscribe()` for Subscription. + +## Fix + +- fix: getting values by path in LoroTree [#643](https://github.com/loro-dev/loro/pull/643) +- fix: should be able to call subscription after diffing [#637] +- fix: update long text may fail [#633](https://github.com/loro-dev/loro/pull/633) +- fix: map.keys() may return keys from deleted entries [#618](https://github.com/loro-dev/loro/pull/618) + + +# FILE: pages/changelog/inspector-v0.1.0.mdx + +--- +version: "v0.1.0" +title: "Release Loro Inspector v0.1.0" +date: 2025/04/30 +--- + +Try it here: [Loro Inspector](https://inspector.loro.dev/) + +Now you can directly browse the current state and complete edit history of your Loro +documents in the browser. You can also use this tool to time travel to any version +in the history of your Loro document. + +import { ReactPlayer } from "../../components/video"; + + + + +# FILE: pages/changelog/v1.6.0.mdx + +--- +version: "v1.6.0" +title: "Release Loro v1.6.0" +date: 2025/08/29 +--- + +Snapshot import speed is now 2x. + +This is implemented by skipping the scan when importing a snapshot, which avoids running the decompression during the import process (but we need to ensure the parent-child link is still accessible in Arena). It also skips the checksum check in the import_all method on MemKV because we already check the checksum in the header of the snapshot/update. + +## v1.0.0 vs v1.6.0 + +You can find the benchmark [here](https://github.com/loro-dev/latch-bench/tree/cmp-loro-160). + +| name | task | time | +| -------------------------- | ------------------------------- | ---------------------- | +| Shallow Snapshot on v1.0.0 | Import | 150.667µs +- 1.823µs | +| | Import+GetAllValues | 163.957µs +- 1.841µs | +| | Import+GetAllValues+Edit | 173.971µs +- 2.03µs | +| | Import+GetAllValues+Edit+Export | 488.848µs +- 3.621µs | +| Shallow Snapshot on v1.6.0 | Import | 82.82µs +- 507ns | +| | Import+GetAllValues | 90.376µs +- 393ns | +| | Import+GetAllValues+Edit | 103.358µs +- 1.916µs | +| | Import+GetAllValues+Edit+Export | 419.463µs +- 2.316µs | +| Snapshot on v1.0.0 | Import | 466.425µs +- 3.879µs | +| | Import+GetAllValues | 487.06µs +- 3.523µs | +| | Import+GetAllValues+Edit | 541.477µs +- 9.067µs | +| | Import+GetAllValues+Edit+Export | 2.98382ms +- 80.537µs | +| Snapshot on v1.6.0 | Import | 201.934µs +- 854ns | +| | Import+GetAllValues | 370.108µs +- 4.049µs | +| | Import+GetAllValues+Edit | 386.497µs +- 3.509µs | +| | Import+GetAllValues+Edit+Export | 2.362296ms +- 28.258µs | + + +# FILE: pages/changelog/v1.3.0.mdx + +--- +version: "v1.3.0" +title: "Release Loro v1.3.0" +date: 2025/01/09 +--- + +## New + +- UndoManager's `onPush` now can access the change event. +- add getShallowValue for each container. + +### LoroDoc + +- `toJsonWithReplacer(replacer: (k, v)=>Value)`: Convert the document to a JSON value with a custom replacer function. +- `revertTo(frontiers: Frontiers)`: Revert the document to the given frontiers. +- `findIdSpansBetween(from: Frontiers, to: Frontiers)`: Find the op id spans that between the `from` version and the `to` version. +- `exportJsonInIdSpan(idSpan: IdSpan)`: Export the readable [`Change`]s in the given [`IdSpan`]. + +## Fix + +- fix: prevent merging remote changes based on local `changeMergeInterval` config [#643](https://github.com/loro-dev/loro/pull/643) +- fix: should commit before travel_change_ancestors [#599](https://github.com/loro-dev/loro/pull/599) +- fix: panic when detach then attach [#592](https://github.com/loro-dev/loro/pull/592) +- fix: move child in current parent [#589](https://github.com/loro-dev/loro/pull/589) +- fix: panic when returned non-boolean value from text.iter(f) [#578](https://github.com/loro-dev/loro/pull/578) + + +# FILE: pages/changelog/v1.1.0.mdx + +--- +version: "v1.1.0" +title: "Release Loro v1.1.0" +date: 2024/11/09 +--- + +## New + +### LoroDoc + +- `forkAt(frontiers: Frontiers)`: Fork the document at the given frontiers. +- `getChangedContainersIn(id: ID, len: number)`: Gets container IDs modified in the given ID range. +- `` + +### LoroText + +- `getEditorOf(pos: number)`: Get the editor of the text at the given position. +- `push(s: string)`: Push a string to the end of the text. + + +### LoroMap + +- `getLastEditor(key: string)`: Get the peer id of the last editor on the given entry + +### LoroList + +- `getIdAt(pos: number)`: Get the ID of the list item at the given position. +- `pushContainer(child: Container)`: Push a container to the end of the list. + +### LoroMovableList + +- `getCreatorAt(pos: number)`: Get the creator of the list item at the given position. +- `getLastMoverAt(pos: number)`: Get the last mover of the list item at the given position. +- `getLastEditorAt(pos: number)`: Get the last editor of the list item at the given position. +- `pushContainer(child: Container)`: Push a container to the end of the list. + +### LoroTree + +- `toJSON()`: Get JSON format of the LoroTreeNode. + +## Fix + +- fix get correct encode blob info [#545](https://github.com/loro-dev/loro/pull/545) +- fix: avoid creating non-root containers that doesn't exist by get_container api [#541](https://github.com/loro-dev/loro/pull/541) +- fix: define the fork behavior when the doc is detached [#537](https://github.com/loro-dev/loro/pull/537) + + +# FILE: pages/blog/v1.0.mdx + +--- +title: "Loro 1.0" +date: 2024/10/23 +keywords: "loro, crdt, event graph walker" +description: "Announcing Loro 1.0: Introducing a stable encoding schema, 10-100x faster document import, advanced version control, and more for efficient real-time collaboration and local-first software development." +image: https://i.ibb.co/T1x1bSf/IMG-8191.jpg +--- + +# Loro 1.0 + +import Authors, { Author } from "../../components/authors"; + + + + +import Image from "next/image"; + +Loro is a [Conflict-free Replicated Data Type (CRDT)](https://crdt.tech/) +library that developers can use to implement real-time collaboration and version +control in their applications. You can use Loro to create +[local-first software](https://www.inkandswitch.com/local-first/). Loro 1.0 has +a stable data format, excellent performance, and rich features. You can use it +in Rust, JS (via WASM), and Swift. + +
+What is CRDT? What is it used for? + +Distributed states are now ubiquitous in multi-user collaborative applications +and applications that need multi-device synchronization. You need to ensure +consistency across devices. CRDTs provide a decentralized way to automatically +solve this problem. + +> CRDTs automatically resolve conflicts and ensure the consistency of the data. +> Some CRDT algorithms provide extra properties for merge results, which should align with user expectations as much as possible. + +CRDT provides a decentralized way to solve this problem. The decentralization here not only means that it can synchronize through P2P methods, but it also means: + +- It allows applications to naturally support offline editing +- It allows users to store and implement two-way synchronization of data in + multiple different locations +- It makes it easier for the backend to implement horizontal scaling +- It can easily support end-to-end encryption + +CRDTs were once considered unable to be used in serious and complex scenarios, +such as rich text, but optimizations in recent years have greatly expanded its +application scenarios, making it a practical and easy-to-use technology. + +Based on CRDT, we can create applications that +allow users to fully control data ownership. These applications can be like +Git-managed repositories, not relying on specific software service providers. +Users can switch between GitHub, GitLab, self-hosted Git servers, and the data +is always available locally. This is the vision of +[local-first software](https://www.inkandswitch.com/local-first/). + + + +CRDTs often also provides a simpler and easier-to-use sync method, +because for Op-based CRDTs like Loro, as long as the sets of CRDT operations +received by two peers are consistent, the CRDT document states of these two +peers are consistent. You don't have to worry about idempotency, the order of +operation application, and network exception handling. For Loro's CRDT document, +just two rounds of data exchange can transmit the missing operations between two +documents to achieve final consistency: + +> You can find all the code samples in this blog [here](https://github.com/zxch3n/loro-blog-examples) + +```jsx +import { LoroDoc, VersionVector } from "npm:loro-crdt@1.13.3"; + +const docA = new LoroDoc(); +const docB = new LoroDoc(); +docA.setPeerId(0); +docA.setPeerId(1); + +docA.getText("text").insert(0, "Hello!"); +docB.getText("text").insert(0, "Hi!"); +const versionA: Uint8Array = docA.version().encode(); +const versionB: Uint8Array = docB.version().encode(); + +// Exchange versionA and versionB Info +const bytesA: Uint8Array = docA.export({ + mode: "update", + from: VersionVector.decode(versionB), +}); +const bytesB: Uint8Array = docB.export({ + mode: "update", + from: VersionVector.decode(versionA), +}); + +// Exchange bytesA and bytesB +docB.import(bytesA); +docA.import(bytesB); + +console.log(docA.getText("text").toString()); // Hello!Hi! +console.log(docB.getText("text").toString()); // Hello!Hi! +``` + +
+ +A minimum of one round of data exchange can ensure consistency + + +```jsx +import { LoroDoc } from "npm:loro-crdt@1.13.3"; + +const docA = new LoroDoc(); +const docB = new LoroDoc(); +docA.setPeerId(0); +docA.setPeerId(1); +docA.getText("text").insert(0, "Hello!"); +docB.getText("text").insert(0, "Hi!"); + +// Exchange versionA and versionB Info +const bytesA: Uint8Array = docA.export({ + mode: "update", +}); +const bytesB: Uint8Array = docB.export({ + mode: "update", +}); + +// Exchange bytesA and bytesB +docB.import(bytesA); +docA.import(bytesB); + +console.log(docA.getText("text").toString()); // Hello!Hi! +console.log(docB.getText("text").toString()); // Hello!Hi! +``` + +If a nested child container may be created lazily by multiple peers under the +same Map key, use [Mergeable Containers](/blog/mergeable-containers) instead of +regular `setContainer`. + +
+
+ +## Features of Loro 1.0 + +### High-performance CRDTs + +High-performance, general-purpose CRDTs can significantly reduce data synchronization +complexity and are crucial for local-first development. + +However, large CRDT documents may face challenges with loading speed and memory consumption, +especially when dealing with those with extensive editing histories. +Loro 1.0 addresses this challenge through a new storage format, achieving a 10x improvement in +loading speed. In [benchmarks using Loro with real-world editing data](#document-import-speed-benchmarks), +we've reduced the loading time for a document with millions of operations from 16ms to 1ms. When utilizing the +shallow snapshot format (discussed later), the time can be further reduced to 0.37ms. +As a result, Loro will not become a bottleneck for applications dealing with such large documents. +It expands the potential use cases for CRDTs, making them viable for a wider range of applications. + +### Rich CRDT types + +Loro now supports +[rich text CRDT](https://loro.dev/blog/loro-richtext), +which enhances the merge result of rich text (text with formatting and styling) to better align with user expectations. +Our text/list CRDT is based on the [Fugue](https://arxiv.org/abs/2305.00583) algorithm. +It prevents interleaving issues when merging concurrent edits. For example, +it can avoid unintended merges like "1H2i3" when "123" and "Hi" are inserted concurrently. + +We also support: + +- Movable List: Supports set, insert, delete, and move operations. The algorithm ensures that after + merging concurrent moves, each element occupies only one position. +- Map: Similar to a JavaScript object. +- [Movable Tree](https://loro.dev/blog/movable-tree): Used to model file directories, outliners, and + other hierarchical structures that may need moving. It ensures no cyclic dependencies exist in the + tree after merging concurrent move operations. + +Loro also supports nesting between types, so you can model edits on JSON documents through them: + +> You can find all the code samples in this blog [here](https://github.com/zxch3n/loro-blog-examples) + +```tsx +import { + LoroDoc, + LoroList, + LoroMap, + LoroText, +} from "npm:loro-crdt@1.13.3"; + +// Create a JSON structure of +interface JsonStructure { + users: LoroList< + LoroMap<{ + name: string; + age: number; + }> + >; + notes: LoroList; +} + +const doc = new LoroDoc(); +const users = doc.getList("users"); +const user = users.insertContainer(0, new LoroMap()); +user.set("name", "Alice"); +user.set("age", 20); +const notes = doc.getList("notes"); +const firstNote = notes.insertContainer(0, new LoroText()); +firstNote.insert(0, "Hello, world!"); + +// { users: [ { age: 20, name: "Alice" } ], notes: [ "Hello, world!" ] } +console.log(doc.toJSON()); +``` + +### Version control + +Like Git, Loro saves a complete directed acyclic graph (DAG) of edit history. In Loro, the DAG is used to represent the dependencies between edits, similar to how Git represents commit history. + +Loro supports primitives that allow users to switch between different versions, fork new branches, edit on new branches, and merge branches. + +Based on this operation primitive, applications can build various Git-like capabilities: + +- You can merge multiple versions without needing to manually resolve conflicts +- Applications can build rebase and squash workflows on top of Loro's version primitives. + +```jsx +import { LoroDoc } from "npm:loro-crdt@1.13.3"; + +const doc = new LoroDoc(); +doc.setPeerId("0"); +doc.getText("text").insert(0, "Hello, world!"); +doc.checkout([{ peer: "0", counter: 1 }]); +console.log(doc.getText("text").toString()); // "He" +doc.checkout([{ peer: "0", counter: 5 }]); +console.log(doc.getText("text").toString()); // "Hello," +doc.checkoutToLatest(); +console.log(doc.getText("text").toString()); // "Hello, world!" + +// Simulate a concurrent edit +doc.checkout([{ peer: "0", counter: 5 }]); +doc.setDetachedEditing(true); +doc.setPeerId("1"); +doc.getText("text").insert(6, " Alice!"); +// ┌───────────────┐ ┌───────────────┐ +// │ Hello, │◀─┬──│ world! │ +// └───────────────┘ │ └───────────────┘ +// │ +// │ ┌───────────────┐ +// └──│ Alice! │ +// └───────────────┘ +doc.checkoutToLatest(); +console.log(doc.getText("text").toString()); // "Hello, world! Alice!" +``` + +You can also use `doc.fork()` to create a separate doc at the current version. It is independent of the current doc, and works like a fork: + +```tsx +import { LoroDoc } from "npm:loro-crdt@1.13.3"; + +const doc = new LoroDoc(); +doc.setPeerId("0"); +doc.getText("text").insert(0, "Hello, world!"); +doc.checkout([{ peer: "0", counter: 5 }]); +const newDoc = doc.fork(); +newDoc.setPeerId("1"); +newDoc.getText("text").insert(6, " Alice!"); +// ┌───────────────┐ ┌───────────────┐ +// │ Hello, │◀─┬──│ world! │ +// └───────────────┘ │ └───────────────┘ +// │ +// │ ┌───────────────┐ +// └──│ Alice! │ +// └───────────────┘ +doc.import(newDoc.export({ mode: "update" })); +doc.checkoutToLatest(); +console.log(doc.getText("text").toString()); // "Hello, world! Alice!" +``` + + + +### Leveraging the potential of the [Eg-walker](https://arxiv.org/abs/2409.14252) + +import { ReactPlayer } from "../../components/video"; +import Caption from "../../components/caption"; + + + +[Event Graph Walker (Eg-walker)](/docs/advanced/event_graph_walker) is a pioneering collaboration algorithm that combines the strengths of +Operational Transformation (OT) and CRDT, two widely used algorithms for real-time collaboration. + +While OT is centralized and CRDT is decentralized, OT traditionally had an advantage +in terms of lower document overhead. CRDTs initially had higher overhead, but recent +optimizations have significantly reduced this gap, making CRDTs increasingly competitive. +Eg-walker leverages the best aspects of both approaches. + +Not only have we use the idea of Eg-walker for Text and List CRDTs in Loro, but +Loro's overall architecture has also been greatly inspired by Eg-walker. As a +result, Loro closely resembles Eg-walker in terms of algorithmic properties. + + + +[The Eg-walker paper](https://arxiv.org/abs/2409.14252) was released in +September 2023. Prior to its official publication, Joseph Gentle shared an +initial version of the algorithm in the Diamond-Type repository. Excited by +the design, I implemented a similar algorithm in Loro two years ago. A brief +introduction to this algorithm can be found +[here](https://loro.dev/docs/advanced/event_graph_walker). + +The properties of Eg-walker includes: + +- It itself conforms to the definition of CRDT, so it has the strong eventual + consistency property of CRDT, thus can be used in distributed environments +- Fast local operation speed: compared to previous CRDTs, it processes + operations extremely fast because it doesn't need to generate corresponding + Operations based on CRDT data structures +- Fast merging of remote operations: The complexity of OT merging remote + operations is O(n^2), while Eg-walker, like mainstream CRDTs, is O(nlogn), + only reaching O(n^2) in extremely rare worst-case scenarios. This means that + when the number of concurrent operations reaches 10,000, OT will start to show + noticeable lag to users, while CRDTs can handle it easily. And in most + real-world scenario benchmarks, it's faster than other CRDTs. +- Lower memory usage: Because it doesn't need to persistently store CRDT + structures in memory, its memory usage is lower than general CRDTs +- Faster import speed: CRDT documents often take a long time to load because + they need to parse the corresponding CRDT structures or operations to build the CRDT + data structures. Without these structures, they cannot continue subsequent + editing, resulting in long import times. Eg-walker, like OT algorithms, only + needs the current document state and does not need to build these additional + structures to allow users to start editing the document directly, thus + achieving much faster import speed + + + +In the past quarter, we have made significant architectural adjustments to allow +Loro to further leverage the advantages of the Eg-walker algorithm. Here are our +achievements + +#### Shallow Snapshot + +By default, Loro stores the complete editing history of the document like Git, +because +[the Eg-walker algorithm needs to load edits that are parallel to them and to the least common ancestor when merging remote edits](https://loro.dev/docs/advanced/event_graph_walker). +Shallow Snapshot is like Git's Shallow Clone, which can remove old historical +operations that users don't need, greatly reducing document size and improving +document import and export speed. This allows you to cold store document history +that is too old and mainly use shallow doc for collaboration. Here's +an example usage: + +```jsx +import { LoroDoc } from "npm:loro-crdt@1.13.3"; + +const doc = new LoroDoc(); +for (let i = 0; i < 10_000; i++) { + doc.getText("text").insert(0, "Hello, world!"); +} +const snapshotBytes = doc.export({ mode: "snapshot" }); +const shallowSnapshotBytes = doc.export({ + mode: "shallow-snapshot", + frontiers: doc.frontiers(), +}); + +console.log(snapshotBytes.length); // 5421 +console.log(shallowSnapshotBytes.length); // 869 +``` + +For details on the implementation principle, see +[Shallow Snapshot](/docs/advanced/shallow_snapshot). + +#### Optimized Document Format + +Loro version 1.0 has achieved a 10x to 100x improvement in document import speed +compared to version 0.16, which already has a fast import speed. +It makes it possible to load a large text document with several million operations +in under a frame time. + +This is because we introduced a new snapshot format. +When a LoroDoc is initialized through this snapshot format, we don't +parse the corresponding document state and historical information until the user +actually needs that information. + + + +In Loro 1.0's snapshot format, without compression algorithms, its document size +is twice that of the old version (and other mainstream CRDTs). This additional +size mainly comes from encoding historical operations + document state in the +1.0 snapshot format, without reusing stored data between the two, while in the +old version we used the order of historical operations to encode the current +state of the document (the old version's encoding learned from +[Automerge encoding's Value Column](https://automerge.org/automerge-binary-format-spec/#_value_column)). + +Trading twice the document size for ten times the import speed is worthwhile +because import speed affects the performance of many aspects, and the import +speed of CRDT documents +[is often noticeable to users on large documents](https://loro.dev/docs/performance) +(> 16ms). It also leaves possibilities for more optimizations in the future. + + + +Inspired by the design of Key-Value Databases, we have also divided the storage +of document state and history into blocks, with each block roughly 4KB in size, +so that when users really need a piece of history, we only need to decompress +and read this 4KB of content, without parsing the entire document. This has led +to a qualitative improvement in import speed, and because the serialization +format can better compress history and state, memory usage is also lower than +before. + +The lazy loading optimization takes advantage of Eg-walker's property that "it +doesn't need to keep the complete CRDT data structure in memory at all times, +and only needs to access historical operations when parallel edits occur". + +
+How we implemented lazy loading + +In Loro 1.0, we implemented a simple [LSM (Log-structured merge-tree)](https://en.wikipedia.org/wiki/Log-structured_merge-tree) engine internally. LSM is a data structure often used to +implement Key-Value Databases, and Loro 1.0 is heavily inspired by its design. +At the time of Loro 1.0, Loro's storage implementation used get, set, and range operations of +Key-Value Database as primitives. For example, Loro stores history as a series of +ChangeBlocks, with each ChangeBlock serialized to about 4KB. Each ChangeBlock +uses its first Op Id as the Key, and the serialized binary data of the +ChangeBlock as the Value, stored in the internal LSM engine. + +In our simple LSM Engine implementation, each Block is compressed during +serialization, and decompression only occurs when the corresponding Value is +actually retrieved. This allows the import speed of the new data format to be up +to a hundred times faster than before, with even lower memory usage. So in Loro +1.0: + +1. Data integrity is checked during import +2. Loro internally stores history (History/OpLog) and state (DocState) in + blocks, loading the corresponding blocks as needed +3. The Eg-walker algorithm that Loro is based on allows documents to start + editing without complete CRDTs meta information, thus easily working with + lazy loading behavior + +Why is lazy loading valuable? Because in many use cases, we don't need to fully +load the document's history and state: + +- For example, when we receive a set of remote updates, but the Loro document + data is still in the database, and we want to know the latest state of the + document, we need to load the LoroDoc snapshot from the database, then import + the remote update set, and then get the latest document state. At this point, + most of the historical information won't be accessed. +- Sometimes in data synchronization scenarios, peer A needs to send historical + data that peer B doesn't have. It needs to import the snapshot and then + extract the historical information that B doesn't have. In this case, the + document's state doesn't need to be parsed, and the unused part of the history + doesn't need to be parsed either. +- Users don't need document history when initializing a document; only parsing + the State is necessary at this point + + ![When merging remote operations, only the modified containers and some of the related historical operations need to be visited](./v1/merge-edit.png) + + When merging remote operations, only the modified containers and some of the + related historical operations need to be visited + +What happens during import and export in the new version? Let's take a common +scenario as an example: + +In real-time collaboration sessions or local storage, we recommend developers +first store the operations from users, and then periodically perform compaction. +This compaction involves importing the old snapshot and all scattered updates +into the same LoroDoc, then exporting uniformly through the Snapshot format. In +the new version, this will involve the following: + +- First, the old version of the snapshot is imported +- The received updates may contain parallel edits, so a part of the related + parallel edit history from the old version needs to be loaded to construct the + CRDT and complete the diff calculation + - Loro internally loads and parses the data of the corresponding block to get + the corresponding history; at this point, complete document parsing does not + occur +- After the diff calculation is complete, it needs to be applied to the + corresponding States + - Loro will internally load and parse the corresponding state, and at this + point, complete document parsing does not occur either +- Export + - Unaffected history blocks or state blocks are exported as they are + - Affected blocks will be serialized to overwrite the original block, then + exported + - During export, we use a method similar to SSTable internally for the final + export + +The only data that needs to be parsed in this entire process are: + +- Meta information for each stored block +- Blocks that need to be read will be decompressed +- History Blocks / state Blocks that will be affected by Updates + + + +
+ +#### Benchmarks + +> All benchmarks results below were performed on a MacBook Pro M1 2020 + +Below is a historical comparison of Snapshot import and export speeds between +Loro versions 1.0.0-beta.1 and 0.16.12. The benchmark is based on document editing history +from the real world. Thanks to [latch.bio](http://latch.bio) for sharing the +document data. The benchmark code is available [here](https://github.com/loro-dev/latch-bench). +The document contains 1,659,541 operations. + +> In Loro, a Snapshot stores the document history along with its current state. +> The Shallow Snapshot format, similar to Git's Shallow Clone, can exclude +> history. In the benchmark below, the Shallow Snapshot has a depth=1 (only the +> most recent operation history is retained, other historical operations are +> removed) + +| task | Old Snapshot Format on 0.16.12 | New Snapshot Format | Shallow Snapshot Format | +| ------------------------------- | ------------------------------ | -------------------------- | ----------------------- | +| Import | 17.3ms +- 0.0298ms | 1.15ms +- 0.0101ms (15x) | 375µs +- 8.47µs (47x) | +| Import+GetAllValues | 17.4ms +- 0.0437ms | 1.19ms +- 0.0122ms (14.5x) | 375µs +- 1.60µs (46x) | +| Import+GetAllValues+Edit | 17.5ms +- 0.0263ms | 1.21ms +- 0.0120ms (14.5x) | 375µs +- 1.40µs (46.5x) | +| Import+GetAllValues+Edit+Export | 32.4ms +- 0.0560ms | 5.46ms +- 0.0772ms (6x) | 844µs +- 5.12µs (38.5x) | + +Here are the key points of this benchmark: + +- The Shallow Snapshot has a depth of 1, meaning it only contains the document + state and a single historical operation, which is why it's significantly + faster +- _GetAllValue_ refers to calling `doc.get_deep_value()` (in JS, it's + `doc.toJSON()` ). It loads the complete state of the document and + obtains the corresponding JSON-like structure. This represents the time spent + on CRDT parsing before a user loads a document. +- _Edit_ refers to making a local modification. As you can see, it has little + impact on the time taken because Loro doesn't need to load the complete CRDT + data structure for local operations. +- _Export_ refers to exporting the complete document data again. We expect to + further reduce the time spent here in the future, as we can continue to reuse + the encoding of unmodified Blocks from the import. + +The following shows the performance on a document after applying the editing +history from the +[Automerge Paper](https://github.com/automerge/automerge-perf/tree/master/edit-by-index) +**100 times**. You can reproduce the results [here](https://github.com/zxch3n/automerge-paper-bench). +The document contains: + +- 18,231,500 single-character insertion operations +- 7,746,300 single-character deletion operations +- 25,977,800 operations totally +- 10,485,200 characters in the final document + +| Snapshot Type | Size (bytes) | +| ---------------- | ------------ | +| Old snapshot | 27,347,374 | +| New snapshot | 23,433,380 | +| Shallow Snapshot | 4,388,215 | + +- The New snapshot data is smaller because it performs additional simple + compression on each Block during encoding internally + +| task | Old Snapshot | New Snapshot | Shallow Snapshot | +| -------------------------- | ---------------- | ---------------------- | ---------------------- | +| Parse | 538ms +- 3.23ms | 17.9ms +- 48.5µs (30x) | 14.4ms +- 114µs (37x) | +| Parse+ToString | 568ms +- 1.78ms | 20.2ms +- 57.2µs (28x) | 16.8ms +- 81.4µs (34x) | +| Parse+ToString+Edit | 561ms +- 940µs | 119ms +- 180µs (4.5x) | 113ms +- 185µs (5x) | +| Parse+ToString+Edit+Export | 1460ms +- 22.9ms | 251ms +- 1.60ms (6x) | 206ms +- 360µs (7x) | + +## Next Steps for Loro + +### Loro Version Controller + + + +Importing Loro Git Repo into Loro Version Controller + +Loro's performance on a single document is now sufficient to cover the real-time +collaboration and version management needs of most documents. So our next step +will be to explore real-time collaboration and version control across a +collection of documents. + +We believe that CRDTs can create a Git for Everyone and Everything: + +- It's for Everyone because by leveraging the power of CRDTs, we can make version + control much easier to reason about and use for the average person. +- It's (nearly) for Everything because Loro provides a rich set of data + synchronization types. We're no longer limited to synchronizing plain text + data, but can solve semantic automatic merging of JSON-like schema, which can meet + most needs of creative tools and collaborative tools. + +We've created a demo of the Loro version controller, which is based on our +sub-document implementation (implemented in the application layer) with Version information. +It can import the entire React repository (about 20,000 commits, thousands of +collaborators), and it supports real-time collaboration on such +repositories. However, how to better manage versions and seamlessly integrate with Git still needs to be explored. + + + +Loro CRDTs still have significant room for optimization in these scenarios. +At the time of Loro 1.0, the Loro CRDTs library didn't involve network or disk +I/O, which enhanced its ease of use but also constrained its capabilities and +potential optimizations. +For example, while we've implemented block-level storage, documents are still +imported and exported as whole units. Adding I/O capabilities to selectively +load/save blocks would enable significant performance optimizations. + +## Conclusion + +Loro 1.0 features great performance improvements, rich CRDT types, and advanced +version control features. Our optimized document format has yielded promising +results on the import speed and the memory usage. + +Now that Loro CRDTs are stable, we are able to develop a better ecosystem. +We're excited to see it being applied in various scenarios. +If you're interested in using Loro, welcome to join our +[Discord community](https://discord.gg/tUsBSVfqzf) for discussions. + +For current updates, follow the [changelog](/changelog) or join the +[Discord community](https://discord.gg/tUsBSVfqzf). + + +# FILE: pages/blog/mergeable-containers.mdx + +--- +title: "Mergeable Containers: Fixing Concurrent Child Creation" +date: 2026/06/09 +description: "Mergeable Containers let Loro peers concurrently create the same child container under a Map key and still merge into one shared child, by deriving identity from the logical parent/key/type instead of the creation OpID." +image: "/images/blog-mergeable-containers.png" +--- + +# Mergeable Containers: Fixing Concurrent Child Creation + +import Authors, { Author } from "../../components/authors"; + + + + + +![Mergeable Containers overview](/images/blog-mergeable-containers.png) + +Two users are offline. Both add content to the same empty note. They come back online, sync finishes, and one user's edits seem to disappear. + +There is no error, and the data is not actually gone from history. But `note.get("body")` can only return one Text container. The other container was created concurrently and still exists in history, but it is no longer visible in the current document state. From the application's point of view, this looks like data loss. + +This is a classic problem in JSON-like CRDTs. Users have run into versions of it in the Loro, Yjs, and Automerge communities. The [Appendix](#appendix-runnable-reproductions) has short scripts that reproduce it in all three. + +Loro now solves this with Mergeable Containers. They make a child container's identity come from its logical position in the `Map`, not from the ID of the operation that happened to create it. + +Special thanks to [Alexis Williams](https://github.com/typedrat) from [Synapdeck](https://synapdeck.com/) for the substantial implementation work and design discussion behind this feature. + +From the user's point of view, the API change is small. Instead of creating an on-demand child container like this: + +```ts no_run +// Peer A +doc.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "A"); + +// Peer B, offline +doc.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "B"); + +// after sync: only one List is visible at "2026-06-08" +``` + +you can use a mergeable child: + +```ts no_run +// Peer A +doc.getMap("days").ensureMergeableList("2026-06-08").insert(0, "A"); + +// Peer B, offline +doc.getMap("days").ensureMergeableList("2026-06-08").insert(0, "B"); + +// after sync: both peers edit the same List +``` + +As a rule of thumb, use `ensureMergeable*` when a child container should be identified by its logical position: + +```ts no_run +map.ensureMergeableText(key); +map.ensureMergeableMap(key); +map.ensureMergeableList(key); +map.ensureMergeableMovableList(key); +map.ensureMergeableTree(key); +map.ensureMergeableCounter(key); +``` + +Use them for fields that should behave like one shared child container for everyone: one shared Text, one shared List, one shared Map, and so on. It should not matter which peer creates that child first. The rest of this post walks through why the problem exists and how the new encoding works. + +## Why This Happens + +CRDTs are usually good at cases like "multiple users editing the same text at the same time" or "multiple users inserting into the same list concurrently." This issue happens one layer earlier: before the peers can edit the same List, Text, or Map, they first need to agree on which child container that key refers to. + +Before Mergeable Containers, the recommended workaround was to initialize all required child containers as soon as the parent `LoroMap` was created. For example, if every note always needs a `body` text, creating that `body` together with the note avoids the first-creation race. + +That workaround is useful, but it has limits. Some applications cannot know every child container ahead of time. A schema migration may add a new child container to existing documents. A calendar-like document may create child containers by date. A dynamic index may create one child container per user-defined key. In these cases, on-demand creation is natural, and concurrent first creation is hard to avoid. + +The root cause is the way regular child Container IDs are represented. A normal child Container ID includes the `OpID` that created it. Concurrent first creation therefore creates different Container IDs, and the Map conflict-resolution rule decides which one is visible. + +The issue is not that List insertion cannot merge. Once both peers are editing the same List, List edits merge normally. The issue is that the two peers created two different Lists at the same Map key. + +## Why Root Containers Are Naturally Mergeable + +In Loro and Yjs, top-level Root Containers are usually accessed by name: + +```ts no_run +doc.getMap("state"); +doc.getText("content"); +``` + +Here, `"state"` or `"content"` is already a stable identity. It does not depend on which peer created it or which operation created it. As long as multiple peers access the same root name, they naturally refer to the same logical Container. + +> Automerge has a different object identity model, so this root-container comparison is specifically about Loro and Yjs. The broader issue is still similar: when composite values are created concurrently at the same key, the system needs a rule for which object identity becomes visible. + +Regular child Containers are different. Their identity is tied to the operation that created them, so two concurrent "first creations" become two different objects. + +Mergeable Containers bring the useful part of Root Container identity to selected child Containers: the child identity comes from a deterministic name, not from the creation operation. + +## API: Explicitly Ensuring a Mergeable Child + +This feature does not change the existing `setContainer` / `insertContainer` behavior. It adds explicit `ensureMergeable*` APIs for the mergeable case. In Rust, the same methods use snake case: + +```rust +map.ensure_mergeable_text("body")?; +map.ensure_mergeable_map("profile")?; +``` + +The word `ensure` is intentional. It returns the child and, if needed, writes the marker that makes it visible at that key. Calling the same method again for the same type is idempotent. + +If the key already holds a regular scalar value or a regular child Container, the API returns an error instead of silently overwriting it. + +One subtle case is type changes. If one peer asks for a mergeable Text at `"field"` while another peer asks for a mergeable Map at the same key, Loro still needs one visible value at that key. The Map's normal conflict rule decides which type is visible. The non-visible mergeable child's state is still preserved under its deterministic ID, so switching back to that type can resurface it later. + +## Core Design: Deterministic CID + Map Slot Marker + +Mergeable Containers have two separate layers of representation: + +1. The child Container ID derived from the parent Container ID, key, and type. This decides whether peers address the same CRDT object. +2. The parent Map slot. This decides whether that object is currently visible at a key, and which mergeable child type is active there. + +Keeping these two layers separate makes the behavior easier to reason about. + +## 1. CID: A Synthetic Root Container ID + +A Mergeable Container uses a synthetic `ContainerID::Root` under an internal namespace. User-created root names cannot use this prefix, so ordinary roots cannot collide with mergeable CIDs: + +```text +🤝: +``` + +The payload is derived from the parent Map and the key. The Container type stays in `ContainerID::Root.container_type`, just like ordinary Root Containers. This lets all peers derive the same child ID without using the creation `OpID`. + +The current encoding keeps nested mergeable Map IDs linear in the logical path length. This change was made before release to avoid recursive CID growth for deeply nested mergeable maps. + +
+More details: the flattened CID encoding + +After [PR #1002](https://github.com/loro-dev/loro/pull/1002), the payload no longer recursively embeds the full parent CID. Instead, it uses a flattened path: + +```text +payload = base-parent ">" key-1 ">" key-2 ... +``` + +The `base-parent` is the nearest non-mergeable Map ancestor: + +```text +$ +@: +``` + +For example: + +```text +Root map "state", key "note-1", child map: +🤝:$state>note-1 type = Map + +Nested key "body" under that mergeable map, child text: +🤝:$state>note-1>body type = Text +``` + +Parsing the second CID gives: + +```text +parent = Root("🤝:$state>note-1", Map) +key = "body" +type = Text +``` + +
+ +## 2. Map Slot: A Binary Marker Controls Visibility + +A deterministic CID alone is not enough because Loro has multiple Container types. If one peer calls `ensureMergeableText("field")` while another peer concurrently calls `ensureMergeableMap("field")`, both deterministic child CIDs can exist. The parent Map still needs to decide which type is currently visible at `"field"`. That decision needs to be deterministic and reversible: switching the visible type should not destroy the state of the other mergeable child. + +So Loro stores a small activation marker in the parent Map slot. Its meaning is: + +```text +At this key of this parent Map, activate a mergeable child of this type. +``` + +When a new Loro client reads the slot, it uses the current `parent id + key + kind` to derive the deterministic mergeable CID, then presents it through the public API as a normal Container: + +```ts no_run +const body = map.get("body"); +// body is a LoroText, not the internal binary marker +``` + +When the key is deleted, only the marker is removed. The mergeable child state is not immediately destroyed, because the parent slot controls visibility rather than the child's stored history. Calling this again: + +```ts no_run +map.ensureMergeableText("body"); +``` + +resurfaces the same deterministic Text Container. + +The marker is also bound to its exact parent, key, and type. That keeps it from accidentally activating a mergeable child if the same binary value is copied somewhere else. + +
+More details: the binary marker format + +The marker is a compact binary value: + +```text +MAGIC[4] + KIND[1] + DIGEST[3] +``` + +`DIGEST` is the low 24 bits of CRC32 over `(parent_id, key, kind)`. So the marker is not a magic value that can be copied anywhere. + +If a user copies the marker binary from one key to another key, or from one parent Map to another, new Loro clients will not recognize it as a valid mergeable child marker. It remains an ordinary binary value. + +This matters because `LoroValue::Binary` is still valid user data. Without binding the marker to parent, key, and type, copying a binary value could accidentally activate a mergeable Container somewhere else. + +### Why Not Use a Reserved Keyword? + +One possible approach would be to store a special string or JSON object: + +```json +{ "__loro_mergeable_container__": "Text" } +``` + +or: + +```text +"__loro_mergeable_text__" +``` + +But that would take over part of the user data space. `LoroMap` is a general-purpose Map, and users may legitimately store such strings or objects. Reserved keywords would make ordinary user values suddenly have special meaning. + +They are also hard to bind safely to parent, key, and type. If a string marker is copied somewhere else, it still looks like a marker. Avoiding accidental activation would require extra validation fields, which would make the format longer and more fragile. + +A binary marker fits this role better: it is low-level structural metadata, not business data. Older clients that do not understand Mergeable Containers see it as an ordinary binary value, rather than misinterpreting it as a child Container reference. + +### Why Not Store the Full ContainerID in the Slot? + +Another possible design would be to store the full deterministic ContainerID directly in the parent Map slot. + +The problem is that older clients may interpret it as a regular child Container edge. That would give them the wrong view of the document structure. + +Mergeable Containers need more than "a pointer to a Container." The design also needs to preserve these rules: + +- The same `(parent, key, type)` deterministically produces the same CID. +- Deleting the key hides the child, but does not delete the child state. +- Conflicts between different mergeable child types still use the Map's normal LWW rule. +- The marker must only activate at the correct parent/key/type. +- Older clients must not mistake it for a normal child Container edge. + +The marker is better understood as an activation marker. New clients derive the actual child CID from the surrounding context. + +
+ +## What This Solves for Users + +Mergeable Containers are especially useful when eager initialization is not practical. + +For example, suppose an application stores one child List per date: + +```ts no_run +const days = doc.getMap("days"); +const entries = days.ensureMergeableList("2026-06-08"); +entries.insert(0, "meeting notes"); +``` + +Or suppose a schema migration lazily adds a new child Map to existing records: + +```ts no_run +const record = doc.getMap("records").ensureMergeableMap(recordId); +const metadata = record.ensureMergeableMap("metadata_v2"); +metadata.set("migrated", true); +``` + +In both cases, the child container identity no longer depends on which peer created it first. It depends on the logical position in the document structure. + +This makes Mergeable Containers especially useful for: + +- date-keyed child lists or maps +- schema migrations that add new child containers lazily +- dynamic per-user or per-entity subdocuments +- revision counters +- settings maps whose keys are discovered over time + +## Cost and Compatibility + +Mergeable Containers have some metadata cost. Their CIDs carry logical path information, so deeper paths and longer keys produce larger IDs. [PR #1002](https://github.com/loro-dev/loro/pull/1002) changed the encoding so nested mergeable Map IDs grow linearly instead of recursively, but very deep mergeable Map chains are still better to avoid. + +The compatibility story is intentionally conservative: + +- Existing `setContainer` / `insertContainer` behavior is unchanged. +- Existing documents can be read normally by new versions. +- Mergeable Containers are introduced through new APIs, without changing existing method signatures. +- Older clients that do not understand this feature see the parent slot marker as an ordinary binary value, not as a fake child Container edge. They can preserve and sync the data, but they will not display the mergeable child with the new semantics. +- User-created root names that start with the internal `🤝:` prefix are rejected by Loro's root-name validator, so they cannot collide with mergeable CIDs. + +## Summary + +Mergeable Containers are for child Containers whose identity should come from their logical position, not from whichever peer created them first. + +Use `ensureMergeable*` when: + +- the key is dynamic or lazily created +- different peers may initialize the same child while offline +- the child should behave like one shared Text, List, Map, Tree, or Counter +- deleting the key should hide the child without treating its internal history as immediately destroyed + +Keep using `setContainer` / `insertContainer` when: + +- each creation should produce a distinct child object +- the parent slot should point to exactly the Container created by that operation +- you are modeling replacement rather than shared initialization + +The short version: if two peers creating the same child at the same Map key should mean "we both found the same child," use a Mergeable Container. + +References: + +- Loro background: [issue #759](https://github.com/loro-dev/loro/issues/759) +- Loro implementation: [PR #991](https://github.com/loro-dev/loro/pull/991), [PR #1002](https://github.com/loro-dev/loro/pull/1002) +- Related Yjs discussions: [complex diagram page](https://discuss.yjs.dev/t/how-would-you-model-a-complex-diagram-page/2114), [losing data](https://discuss.yjs.dev/t/why-am-i-losing-data/2734), [nested `Y.Map`](https://discuss.yjs.dev/t/create-y-map-is-empty/1701) +- Related Automerge discussions: [#528: failing merge for text values](https://github.com/automerge/automerge/issues/528) is the closest match; [#526: conflict resolution for replaced arrays and objects](https://github.com/automerge/automerge/issues/526) is useful background on object identity and conflict handling; the historical [automerge-classic #4](https://github.com/automerge/automerge-classic/issues/4) also covers concurrently created objects under the same key. + +## Appendix: Runnable Reproductions + +The snippets below are self-contained and run directly on Node (tested on Node 24; any Node 18+ with ESM works). Install the three libraries once: + +```bash +npm install loro-crdt@^1.13 yjs @automerge/automerge +``` + +Save each block as a `.mjs` file and run it with `node file.mjs`. They all model the same scenario from this post: two offline peers concurrently create a child container under the same `Map` key, then sync. The Loro example also shows the `ensureMergeable*` fix. + +### Loro — the bug, and the fix + +```js no_run +// loro.mjs — node loro.mjs +// In plain Node import from "loro-crdt/nodejs"; the bare "loro-crdt" entry +// targets a bundler. With Vite/webpack, import from "loro-crdt" instead. +import { LoroDoc, LoroList } from "loro-crdt/nodejs"; + +function sync(a, b) { + const va = a.export({ mode: "update" }); + const vb = b.export({ mode: "update" }); + a.import(vb); + b.import(va); +} + +// 1. The bug: concurrent setContainer at the same key +{ + const a = new LoroDoc(); + const b = new LoroDoc(); + a.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "A"); + b.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "B"); + sync(a, b); + console.log( + "setContainer ->", + JSON.stringify(a.getMap("days").get("2026-06-08").toArray()), + ); + // -> ["A"] or ["B"], never both: only one peer's List survives. + // (which one wins depends on the randomly-assigned peer IDs) +} + +// 2. The fix: concurrent ensureMergeableList at the same key +{ + const a = new LoroDoc(); + const b = new LoroDoc(); + a.getMap("days").ensureMergeableList("2026-06-08").insert(0, "A"); + b.getMap("days").ensureMergeableList("2026-06-08").insert(0, "B"); + sync(a, b); + console.log( + "ensureMergeable ->", + JSON.stringify(a.getMap("days").get("2026-06-08").toArray()), + ); + // -> both entries, e.g. ["A","B"] (order may vary): both peers share one List. +} +``` + +### Yjs — the same problem + +```js no_run +// yjs.mjs — node yjs.mjs +import * as Y from "yjs"; + +const a = new Y.Doc(); +const b = new Y.Doc(); + +// Peer A and Peer B each create a Y.Array at the same key, offline. +{ + const l = new Y.Array(); + a.getMap("days").set("2026-06-08", l); + l.insert(0, ["A"]); +} +{ + const l = new Y.Array(); + b.getMap("days").set("2026-06-08", l); + l.insert(0, ["B"]); +} + +// Sync both ways. +Y.applyUpdate(a, Y.encodeStateAsUpdate(b)); +Y.applyUpdate(b, Y.encodeStateAsUpdate(a)); + +console.log( + "yjs ->", + JSON.stringify(a.getMap("days").get("2026-06-08").toArray()), +); +// -> ["A"] or ["B"], never both: one peer's child Y.Array wins, the other is dropped. +``` + +### Automerge — the same problem + +```js no_run +// automerge.mjs — node automerge.mjs +import * as A from "@automerge/automerge"; + +let base = A.from({ days: {} }); +let a = A.clone(base); +let b = A.clone(base); + +// Peer A and Peer B each create a list at the same key, offline. +a = A.change(a, (d) => { + d.days["2026-06-08"] = ["A"]; +}); +b = A.change(b, (d) => { + d.days["2026-06-08"] = ["B"]; +}); + +let merged = A.merge(A.clone(a), b); +console.log("automerge visible ->", JSON.stringify(merged.days["2026-06-08"])); +// -> ["A"] or ["B"], never both: one list wins. +console.log( + "automerge conflicts ->", + JSON.stringify(A.getConflicts(merged.days, "2026-06-08")), +); +// -> both lists keyed by op id: the losing list is retained but hidden, +// reachable only via getConflicts(). + +// Control: when the child is created ONCE up front, concurrent edits merge. +let shared = A.from({ days: { "2026-06-08": [] } }); +let c = A.clone(shared), + d = A.clone(shared); +c = A.change(c, (x) => { + x.days["2026-06-08"].push("A"); +}); +d = A.change(d, (x) => { + x.days["2026-06-08"].push("B"); +}); +let ok = A.merge(A.clone(c), d); +console.log("automerge pre-created ->", JSON.stringify(ok.days["2026-06-08"])); +// -> ["A","B"] (order may vary): both survive — this is the eager-init workaround. +``` + +Note one difference worth calling out: in Automerge the losing child is retained and can be recovered through `getConflicts()`, while Yjs overwrites the map key and drops the losing child outright. Either way, from the application's point of view it looks like data loss — which is exactly what Mergeable Containers avoid. + + +# FILE: pages/blog/crdt-richtext.mdx + +--- +title: crdt-richtext - Rust implementation of Peritext and Fugue +date: 2023/04/20 +keywords: crdt, richtext, peritext, fugue, loro +description: Presenting the early Rust crate that combined Peritext and Fugue's power with impressive performance, tailored specifically for rich text. This work later informed Loro's rich text CRDT. +tag: richtext +# ogImage: /images/blog/joining-vercel/x-card.png +--- + +# [crdt-richtext](https://github.com/loro-dev/crdt-richtext): Rust implementation of Peritext and Fugue + +import Authors, { Author } from "../../components/authors"; + + + + + +Presenting an early Rust crate that combines [Peritext](https://inkandswitch.com/peritext) and [Fugue](https://arxiv.org/abs/2305.00583)'s power with impressive performance, tailored specifically for rich text. This work later informed **[Loro](https://www.loro.dev/)**'s rich text CRDT. + +# What’s Peritext + +[Peritext: A CRDT for Rich-Text Collaboration](https://inkandswitch.com/peritext) + +Peritext is a novel rich-text CRDT (Conflict-free Replicated Data Type) algorithm. It is capable of merging concurrent edits in rich text format while [preserving users' intent as much as possible](https://www.inkandswitch.com/peritext/#preserving-the-authors-intent). Its primary focus is on merging the formats and annotations of rich text content, such as bold, italic, and comments. + +> 💡 The specific definition of user intent in the context of concurrent rich text editing can't be clearly explained in a few words. it's best understood through specific examples. + +Peritext is designed to solve a couple of significant challenges: + +Firstly, it addresses the anticipated problems arising from conflicting style edits. For instance, consider a text example, "The quick fox jumped." If User A highlights "The quick" in bold and User B highlights "quick fox jumped," the ideal merge should result in the entire sentence, "The quick fox jumped," being bold. However, existing algorithms might not meet this expectation, resulting in either "The quick fox" or "The" and "jumped" being bold instead. + +| Original Text | The quick fox jumped | +| -------------------------------------------- | ---------------------------- | +| Concurrent Edit from A | **The quick** fox jumped | +| Concurrent Edit from B | The **quick fox jumped** | +| Expected Merged Result | **The quick fox jumped** | +| Bad case from merging Markdown text directly | **The** quick **fox jumped** | +| Bad case from Yjs | **The quick** fox jumped | + +Additionally, Peritext manages conflicts between style and text edits. In the same example, if User A highlights "The quick" in bold, but User B changes the text to "The fast fox jumped," the ideal merge should result in "The fast" being bold. + +| Original Text | The quick fox jumped | +| ---------------------- | ------------------------ | +| Concurrent Edit from A | **The quick** fox jumped | +| Concurrent Edit from B | The fast fox jumped | +| Expected Merged Result | **The fast** fox jumped | + +What’s more, Peritext takes into account different expectations for expanding styles. For example, if you type after a bold text, you would typically want the new text to continue being bold. However, if you're typing after a hyperlink or a comment, you likely wouldn't want the new input to become part of the hyperlink or comment. + +# What’s Fugue + +Fugue is a new CRDT text algorithm, presented in _[The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing](https://arxiv.org/abs/2305.00583)_ by [Matthew Weidner](https://arxiv.org/search/cs?searchtype=author&query=Weidner%2C+M) et al., nicely solves **the interleaving problem**. + +## The interleaving problem + +The interleaving problem was proposed in the paper _[Interleaving anomalies in collaborative text editors](https://martin.kleppmann.com/2019/03/25/papoc-interleaving-anomalies.html)_ by Martin Kleppmann et al. + +An example of interleaving: + +- A type "Hello " from left to right/right to left +- B type "Hi " from left to right/right to left +- The expected result: "Hello Hi " or "Hi Hello " +- The interleaving result may look like: "HHeil lo" + - This happens when typing from right to left in RGA. + +![An example of an interleaving anomaly when using [fractional indexing](https://madebyevan.com/algos/crdt-fractional-indexing/) CRDT on text content. +Source: **Martin Kleppmann, Victor B. F. Gomes, Dominic P. Mulligan, and Alastair R. Beresford. 2019. Interleaving anomalies in collaborative text editors. [https://doi.org/10.1145/3301419.3323972](https://doi.org/10.1145/3301419.3323972)](./images/richtext0.png) + +An example of an interleaving anomaly when using [fractional indexing](https://madebyevan.com/algos/crdt-fractional-indexing/) CRDT on text content. +Source: \*\*Martin Kleppmann, Victor B. F. Gomes, Dominic P. Mulligan, and Alastair R. Beresford. 2019. Interleaving anomalies in collaborative text editors. [https://doi.org/10.1145/3301419.3323972](https://doi.org/10.1145/3301419.3323972) + +The [Fugue paper](https://arxiv.org/abs/2305.00583) summarizes the current state of the interleaving problems in the table. + +![Source: Weidner, M., Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing. *ArXiv*. /abs/2305.00583](./images/richtext1.png) + +Source: Weidner, M., Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing. _ArXiv_. /abs/2305.00583 + +The interleaving problem sometimes are unsolvable when there are more than 2 sites. See [Fugue](https://arxiv.org/abs/2305.00583) paper Appendix B, Proof of Theorem 5 for detailed explanation. + +![The case where the interleaving problem is unsolvable +Source: Weidner, M., Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing. *ArXiv*. /abs/2305.00583](./images/richtext2.png) + +The case where the interleaving problem is unsolvable +Source: Weidner, M., Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing. _ArXiv_. /abs/2305.00583 + +However, we can still minimize the chance of interleaving. Fugue introduces the concept of **maximal non-interleaving** and solves it with an elegant algorithm that is easy to optimize. The definition of _maximal non-interleaving_ makes a lot of sense to me and leaves little room for ambiguity. I won't reiterate the definition here. But the basic idea is first to solve forward interleaving by leftOrigin. If there is still ambiguity, then solve the backward interleaving by rightOrigin. (The leftOrigin and rightOrigin refer to the ids of the original neighbors when the character is inserted, just like Yjs) + +# CRDT-Richtext + +Based on the algorithms of Peritext and Fugue, we made `crdt-richtext`, a lib written in Rust that provides a wasm interface. It’s available on [crates.io](http://crates.io) and npm now. + +## Example + +```tsx +import { RichText } from "crdt-richtext-wasm"; + +const text = new RichText(BigInt(1)); +text.insert(0, "ä½ å¥½ï¼Œä¸–ç•Œï¼"); +text.insert(2, "å‘€"); +expect(text.toString()).toBe("ä½ å¥½å‘€ï¼Œä¸–ç•Œï¼"); +text.annotate(0, 3, "bold", AnnotateType.BoldLike); +const spans = text.getAnnSpans(); +expect(spans.length).toBe(2); +expect(spans[0].text).toBe("ä½ å¥½å‘€"); +expect(spans[0].annotations.size).toBe(1); +expect(spans[0].annotations.has("bold")).toBeTruthy(); +expect(spans[1].text.length).toBe(4); + +const b = new RichText(BigInt(2)); +b.import(text.export(new Uint8Array())); +expect(b.toString()).toBe("ä½ å¥½å‘€ï¼Œä¸–ç•Œï¼"); +``` + +## Data structure + +We heavily use B-Trees to optimize our algorithm. We made a library called [generic-btree](https://github.com/loro-dev/generic-btree), which is written in safe Rust code, which provides a flexible foundation for our optimization efforts. + +[https://github.com/loro-dev/generic-btree](https://github.com/loro-dev/generic-btree) + +![The cached content inside B-Tree](./images/richtext3.png) + +The cached content inside B-Tree + +There are several common tasks we need to address in Text CRDT, including: + +- Finding, inserting, or deleting content at a given index: + - We use a BTree to look up and update the content + - The time complexity is O(logN), where N is the length of the content +- Finding content with a given op ID: + - We use a combination of HashMap and BTree + - The time complexity if O(logN), where N is the number of operations +- Compressing content in memory: + - To reduce the amount of memory used by storing every operation in raw format, we compress the content using the RLE tricks from Yjs and DiamondTypes. + - The insight behind this compression is that neighboring inserts and deletions tend to be continuous, so we can merge them and store less metadata. + - Commonly, every leaf node in the diagram contains a dozen of characters +- Converting index between UTF-16 and UTF-8: + - In JS, the default encoding of a string is utf16, but in Rust, the default one is utf8. Although the WASM interface can help us convert the encoding of the string, we still need to convert the _index_ of the operation. + - To solve this, `crdt-richtext` also store the UTF-16 length of the content in B-Tree. So we can query the B-Tree with either the utf8 index or the utf16 index. +- Storing the boundary of style/format/comments: + - We use the same B-Tree to store the boundary, with each subtree corresponding to a span of text or tombstones. For each node in the tree, we store which annotations start before it, start after it, end before it, or end after it. + ```rust + #[derive(Debug, PartialEq, Eq, Default, Clone)] + pub struct ElemAnchorSet { + start_before: FxHashSet, + end_before: FxHashSet, + start_after: FxHashSet, + end_after: FxHashSet, + } + ``` + - This is basically the same optimization as Peritext, except we do it on the tree. + +## Encoding + +We use columnar encoding, which was first adopted to CRDTs by Martin Kelppmann [in automerge](https://github.com/automerge/automerge-classic/pull/253). To make it easier in Rust, we created the lib [Serde Columnar: Ergonomic columnar storage encoding crate](https://www.notion.so/Serde-Columnar-Ergonomic-columnar-storage-encoding-crate-7b0c86d6f8d24e4da45a1e2ebd86741c?pvs=21). + +## Heavily tested by libFuzzer + +Test-Driven Development (TDD) provides an amazing development experience. If possible, I always write unit tests for a standalone module before moving forward. However, for algorithms like CRDTs, it is infeasible to list all possible cases manually but is easy to generate test cases automatically. This is where fuzzing tests come into play. + +Some fuzzers can track coverage information and generate mutations on the input data to maximize code coverage. LibFuzzer can also identify memory leaks and UAF problems. + +`[cargo-fuzz`](https://www.notion.so/crdt-richtext-Rust-implementation-of-Peritext-and-Fugue-c49ef2a411c0404196170ac8daf066c0?pvs=21) provides a user-friendly API for writing fuzzing tests, and it supports two fuzzers: libFuzzer and AFL. It makes the unstructured libFuzzer feel structured. So we’re able to write fuzzing tests in this way + +```rust +use arbitrary::Arbitrary; + +#[derive(Arbitrary, Clone, Debug, Copy)] +pub enum Action { + Insert { + actor: u8, + pos: u8, + content: u16, + }, + Delete { + actor: u8, + pos: u8, + len: u8, + }, + Annotate { + actor: u8, + pos: u8, + len: u8, + annotation: AnnotationType, + }, + Sync(u8, u8), +} + +pub fn fuzzing(actions: Vec) { + // run tests based on actions + ... +} + +#![no_main] +use libfuzzer_sys::fuzz_target; + +fuzz_target!(|actions: [Action; 100]| { fuzzing(actions.to_vec()) }); +``` + +![We will run millions of Fuzzing Tests after making big changes. The fuzzer can help us extract the most useful thousands of tests to be included into the corpus. The minor changes can be verified by running the corpus.](./images/richtext4.png) + +We will run millions of Fuzzing Tests after making big changes. The fuzzer can help us extract the most useful thousands of tests to be included into the corpus. The minor changes can be verified by running the corpus. + +We use fuzzing tests in Loro's CRDTs too. This test suite is like our safety net when we're making big tweaks to the code. It's great at spotting all our little slip-ups. + +# Performance + +## Benchmark + +- Benchmark setup + ### **B4: Real-world editing dataset** + Replay a real-world editing dataset. This dataset contains the character-by-character editing trace of a large-ish text document, the LaTeX source of this paper: [https://arxiv.org/abs/1608.03960(opens in a new tab)](https://arxiv.org/abs/1608.03960) + Source: [https://github.com/automerge/automerge-perf/tree/master/edit-by-index(opens in a new tab)](https://github.com/automerge/automerge-perf/tree/master/edit-by-index) + - 182,315 single-character insertion operations + - 77,463 single-character deletion operations + - 259,778 operations totally + - 104,852 characters in the final document + We simulate one client replaying all changes and storing each update. We measure the time to replay the changes and the size of all update messages (`updateSize`), the size of the encoded document after the task is performed (`docSize`), the time to encode the document (`encodeTime`), the time to parse the encoded document (`parseTime`), and the memory used to hold the decoded document in memory (`memUsed`). + ### **[B4 x 100] Real-world editing dataset 100 times** + Replay the [B4] dataset one hundred times. The final document has a size of over 10 million characters. As comparison, the book "Game of Thrones: A Song of Ice and Fire" is only 1.6 million characters long (including whitespace). + - 18,231,500 single-character insertion operations + - 7,746,300 single-character deletion operations + - 25,977,800 operations totally + - 10,485,200 characters in the final document + +The benchmark was conducted on a 2020 M1 MacBook Pro 13-inch on 2023-05-11. + +| N=6000 | crdt-richtext-wasm | loro-wasm | automerge-wasm | tree-fugue | yjs | ywasm | +| ---------------------------------------------------------------- | ---------------------- | ----------------------- | ------------------- | --------------------------- | ---------------------------- | ------------------- | +| [B4] Apply real-world editing dataset (time) | 176 +/- 10 ms | 141 +/- 15 ms | 821 +/- 7 ms | 721 +/- 15 ms | 1,114 +/- 33 ms | 23,419 +/- 102 ms | +| [B4] Apply real-world editing dataset (memUsed) | skipped | skipped | skipped | 2,373,909 +/- 13725 bytes | 3,480,708 +/- 168887 bytes | skipped | +| [B4] Apply real-world editing dataset (encodeTime) | 8 +/- 1 ms | 8 +/- 1 ms | 115 +/- 2 ms | 12 +/- 0 ms | 12 +/- 1 ms | 6 +/- 1 ms | +| [B4] Apply real-world editing dataset (docSize) | 127,639 +/- 0 bytes | 255,603 +/- 8 bytes | 129,093 +/- 0 bytes | 167,873 +/- 0 bytes | 159,929 +/- 0 bytes | 159,929 +/- 0 bytes | +| [B4] Apply real-world editing dataset (parseTime) | 11 +/- 0 ms | 2 +/- 0 ms | 620 +/- 5 ms | 8 +/- 0 ms | 43 +/- 3 ms | 40 +/- 3 ms | +| [B4x100] Apply real-world editing dataset 100 times (time) | 15,324 +/- 3188 ms | 12,436 +/- 444 ms | skipped | 91,902 +/- 863 ms | 112,563 +/- 3861 ms | skipped | +| [B4x100] Apply real-world editing dataset 100 times (memUsed) | skipped | skipped | skipped | 224076566 +/- 2812359 bytes | 318807378 +/- 15737245 bytes | skipped | +| [B4x100] Apply real-world editing dataset 100 times (encodeTime) | 769 +/- 37 ms | 780 +/- 32 ms | skipped | 943 +/- 52 ms | 297 +/- 16 ms | skipped | +| [B4x100] Apply real-world editing dataset 100 times (docSize) | 12,667,753 +/- 0 bytes | 26,634,606 +/- 80 bytes | skipped | 17,844,936 +/- 0 bytes | 15,989,245 +/- 0 bytes | skipped | +| [B4x100] Apply real-world editing dataset 100 times (parseTime) | 1,252 +/- 14 ms | 170 +/- 15 ms | skipped | 368 +/- 13 ms | 1,335 +/- 238 ms | skipped | + +The complete benchmark result and code is available at https://github.com/zxch3n/fugue-bench. + +It is worth noting that: + +- The benchmark for Automerge is based on `automerge-wasm`, which is not the latest version of Automerge 2.0. +- `crdt-richtext` and `fugue` are special-purpose CRDTs that tend to be faster and have a smaller encoding size. +- The encoding of `yjs`, `ywasm`, and `loro-wasm` still contains redundancy that can be compressed significantly. For more details, see [the full report](https://loro.dev/docs/performance/docsize). +- At the time of this 2023 benchmark, `loro-wasm` and `fugue` only supported plain text. + +# Discussion + +[CRDT-richtext: Rust implementation of Peritext and Fugue | Hacker News](https://news.ycombinator.com/item?id=35988046) + + +# FILE: pages/blog/loro-richtext.mdx + +--- +title: "Introduction to Loro's Rich Text CRDT" +date: 2024/01/22 +keywords: "loro, crdt, rich text, peritext, event graph walker, eg-walker, fugue" +description: + This article presents the rich text CRDT algorithm implemented in Loro, + complying with Peritext's criteria for seamless rich text collaboration. + Furthermore, it can be built on top of any List CRDT algorithms and turn them + into rich text CRDTs. +image: https://i.ibb.co/rsX5vR6/cover-long.png +--- + +# Introduction to Loro's Rich Text CRDT + +import Authors, { Author } from "../../components/authors"; + + + + + +![](./loro-richtext/cover_long.png) + +import Caption from "../../components/caption"; +import Demo from "@components/richtextDemo"; + +This article presents the rich text CRDT algorithm implemented in Loro, +complying with [Peritext]'s criteria for seamless rich text collaboration. +Furthermore, it can be built on top of any List CRDT algorithms and turn them +into rich text CRDTs. + +
+ + + Above is an online demo of Loro's rich text CRDT, built with Quill. After the + replay, you can simulate real-time collaboration and concurrent editing while + offline. You can also drag on the history view to replay the editing history. + + +If CRDTs are new to you, our article [What are CRDTs](/docs/concepts/crdt) +provides a brief introduction. + +## Background + +Loro is based on the +[Event Graph Walker (Eg-walker)](/docs/advanced/replayable_event_graph) algorithm +proposed by Joseph Gentle, but this algorithm cannot integrate the original +version of Peritext. This motivates us to create a new rich text algorithm. It +is independent of the specific List CRDTs, thus working nicely with Eg-walker, and is +developed on top of them to establish a rich text CRDT. + +Before diving into the algorithm of Loro's rich text CRDT, I'd like to briefly +introduce Eg-walker and Peritext, and why Peritext cannot be used on Eg-walker. + +
+Recap on List CRDTs + +### Recap on List CRDTs + +Unlike OT, most List-oriented CRDTs assign a unique ID to each item or +character, often corresponding to the operation ID of its insertion. With unique +IDs for each character, we can reliably reference a character or position +through its ID. + +![](./loro-richtext/list_crdt_ids.png) + +The unique ID eliminates concerns about consistent position descriptions during +synchronization. For instance, deletions are straightforward by specifying the +deleted character's ID, and insertions are described using the IDs of adjacent +characters. In cases of concurrent insertions at the same location, List CRDT +algorithms resolve the consistency issues. + +![](./loro-richtext/list_crdt_insert.png) + +![](./loro-richtext/list_crdt_delete.png) + +However, a notable limitation of List CRDTs is the use of 'tombstones'. Upon +deletion of a character, it is not fully removed but replaced with a tombstone, +maintaining the ID's position. Depending on the algorithm, this tombstone may be +removed once all participating nodes acknowledge the deletion. However, it can +be challenging to determine if all peers have received the corresponding +deletion operation. This information often means additional overhead for many +CRDTs. Thus, the simplest solution is not to perform any tombstone collection. + +![](./loro-richtext/list_crdt_tombstone.png) + +
+ +### Brief Introduction to Event Graph Walker + +Eg-walker is a novel CRDT algorithm introduced in: + +> [Collaborative Text Editing with Eg-walker: Better, Faster, Smaller](https://arxiv.org/abs/2409.14252) +> By: Joseph Gentle, Martin Kleppmann + +Eg-walker is a novel CRDT algorithm that combines the strengths of both OT and CRDTs. +It has the distributed nature of CRDT that enables P2P collaboration and data +ownership. Moreover, it achieves minimal overhead in scenarios devoid of +concurrent edits, similar to OT. + +import { ReactPlayer } from "../../components/video"; + + + +Whether in real-time collaboration or multi-end synchronization, a directed +acyclic graph (DAG) forms over the history of these parallel edits, similar to +Git's history. The Eg-walker algorithm records the history of user edits on the DAG. + +Unlike conventional CRDTs, Eg-walker can record just the original description of +operations, not the metadata of CRDTs. For instance, in text editing scenarios, +the [RGA algorithm] needs the op ID and [Lamport timestamp][Lamport] of the +character to the left to determine the insertion point. [Yjs]/Fugue, however, +requires the op ID of both the left and right characters at insertion. In +contrast, Eg-walker simplifies this by only recording the index at the time of +insertion. Loro, which uses [Fugue] upon Eg-walker, inherits these advantages. + +An index is not a stable position descriptor, as the index of an operation can +be affected by other operations. For example, if you highlight content from +`index=x` to `index=y`, and concurrently someone inserts n characters at +`index=n` where `n +What is Fugue + +Fugue is a new CRDT text algorithm, presented in +_[The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing](https://arxiv.org/abs/2305.00583)_ +by +[Matthew Weidner](https://arxiv.org/search/cs?searchtype=author&query=Weidner%2C+M) +et al., nicely solves **the interleaving problem**. + +The interleaving problem was proposed in the paper +_[Interleaving anomalies in collaborative text editors](https://martin.kleppmann.com/2019/03/25/papoc-interleaving-anomalies.html)_ +by Martin Kleppmann et al. + +An example of interleaving: + +- A type "Hello " from left to right/right to left +- B type "Hi " from left to right/right to left +- The expected result: "Hello Hi " or "Hi Hello " +- The interleaving result may look like: "HHeil lo" + - This happens when typing from right to left in RGA. + +![An example of an interleaving anomaly when using [fractional indexing](https://madebyevan.com/algos/crdt-fractional-indexing/) CRDT on text content. +Source: **Martin Kleppmann, Victor B. F. Gomes, Dominic P. Mulligan, and Alastair R. Beresford. 2019. Interleaving anomalies in collaborative text editors. [https://doi.org/10.1145/3301419.3323972](https://doi.org/10.1145/3301419.3323972)](./images/richtext0.png) + +An example of an interleaving anomaly when using +[fractional indexing](https://madebyevan.com/algos/crdt-fractional-indexing/) +CRDT on text content. Source: \*\*Martin Kleppmann, Victor B. F. Gomes, Dominic +P. Mulligan, and Alastair R. Beresford. 2019. Interleaving anomalies in +collaborative text editors. +[https://doi.org/10.1145/3301419.3323972](https://doi.org/10.1145/3301419.3323972) + +The [Fugue paper](https://arxiv.org/abs/2305.00583) summarizes the current state +of the interleaving problems in the table. + +![Source: Weidner, M., Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing. *ArXiv*. /abs/2305.00583](./images/richtext1.png) + +Source: Weidner, M., Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: +Minimizing Interleaving in Collaborative Text Editing. _ArXiv_. /abs/2305.00583 + +The interleaving problem sometimes are unsolvable when there are more than 2 +sites. See [Fugue](https://arxiv.org/abs/2305.00583) paper Appendix B, Proof of +Theorem 5 for detailed explanation. + +![The case where the interleaving problem is unsolvable +Source: Weidner, M., Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: Minimizing Interleaving in Collaborative Text Editing. *ArXiv*. /abs/2305.00583](./images/richtext2.png) + +The case where the interleaving problem is unsolvable Source: Weidner, M., +Gentle, J., & Kleppmann, M. (2023). The Art of the Fugue: Minimizing +Interleaving in Collaborative Text Editing. _ArXiv_. /abs/2305.00583 + +However, we can still minimize the chance of interleaving. Fugue introduces the +concept of **maximal non-interleaving** and solves it with an elegant algorithm +that is easy to optimize. The definition of _maximal non-interleaving_ makes a +lot of sense to me and leaves little room for ambiguity. I won't reiterate the +definition here. But the basic idea is first to solve forward interleaving by +leftOrigin. If there is still ambiguity, then solve the backward interleaving by +rightOrigin. (The leftOrigin and rightOrigin refer to the ids of the original +neighbors when the character is inserted, just like Yjs) + + + +### Brief Introduction to Peritext + +[Peritext] was proposed by _Geoffrey Litt et al._ It's the first paper to +discuss rich text CRDTs. It can merge concurrent edits in rich text format while +[preserving users' intent as much as possible](https://www.inkandswitch.com/peritext/#preserving-the-authors-intent). +Its primary focus is merging the formats and annotations of rich text content, +such as bold, italic, and comments. It was implemented in [Automerge] and +[crdt-richtext]. + +> 💡 The specific definition of user intent in the context of concurrent rich +> text editing can't be clearly explained in a few words. It's best understood +> through particular examples. + +Peritext is designed to solve a couple of significant challenges: + +Firstly, it addresses the anticipated problems arising from conflicting style +edits. For instance, consider a text example, "The quick fox jumped." If User A +highlights "The quick" in bold and User B highlights "quick fox jumped," the +ideal merge should result in the entire sentence, "The quick fox jumped," being +bold. However, existing algorithms might not meet this expectation, resulting in +either "The quick fox" or "The" and "jumped" being bold instead. + +| Original Text | The quick fox jumped | +| -------------------------------------------- | ---------------------------- | +| Concurrent Edit from A | **The quick** fox jumped | +| Concurrent Edit from B | The **quick fox jumped** | +| Expected Merged Result | **The quick fox jumped** | +| Bad case from merging Markdown text directly | **The** quick **fox jumped** | +| Bad case from Yjs | **The quick** fox jumped | + +Additionally, Peritext manages conflicts between style and text edits. In the +same example, if User A highlights "The quick" in bold, but User B changes the +text to "The fast fox jumped," the ideal merge should result in "The fast" being +bold. + +| Original Text | The quick fox jumped | +| ---------------------- | ------------------------ | +| Concurrent Edit from A | **The quick** fox jumped | +| Concurrent Edit from B | The fast fox jumped | +| Expected Merged Result | **The fast** fox jumped | + +What’s more, Peritext takes into account different expectations for expanding +styles. For example, if you type after a bold text, you would typically want the +new text to continue being bold. However, if you're typing after a hyperlink or +a comment, you likely wouldn't want the new input to become part of the +hyperlink or comment. + +
+![Link style should not expand](./loro-richtext/Peritext.png) +
+ +Illustration of Peritext's internal state. It uses the IDs of the character's ops to record the style ranges. In the example, the bold mark has the range of `{ start: { type: "before", opId: "9@B" }, end: { type: "before", opId: "10@B" }}` + + +### Why Original Peritext Can't Be Directly Used with Eg-walker + +On the one hand, Peritext's algorithm expresses style ranges +[through character OpIDs](https://www.inkandswitch.com/peritext/#generating-inline-formatting-operations). +Without replaying history, CRDTs based on Eg-walker cannot determine the specific +positions corresponding to these OpIDs. + +On the other hand, it's not feasible to model Peritext on Eg-walker through replaying. +This is because Eg-walker's "local backtracking suffices" relies on the algorithm +satisfying "the same operation will produce the same effect, regardless of the +current state," which Peritext does not adhere to. For example, when inserting +the character "x" at position `p`, whether "x" is bold depends on "whether `p` +is surrounded by bold" and +"[whether the tombstones at `p` contain boundaries of bold and other styles](https://arc.net/l/quote/ifxpaand)." + +## Loro's Rich Text CRDT + +### Algorithm + +Loro implements rich text using special control characters called 'style +anchors'. Each matching pair of start anchor and end anchor contains the +following information: + +- The op ID of the style operation +- The style's key-value pair +- The style's [Lamport timestamp][Lamport] +- Style expansion behavior: Determines whether newly inserted text before or + after the style boundaries should inherit the style. + +The method to determine a character's style is as follows: + +- Find all style anchor pairs that include the character, where each pair is + created by the same style operation +- Aggregate pairs according to the key. There may be multiple style pairs with + the same key but different values. In such cases, the value with the greatest + Lamport timestamp is chosen (if Lamport timestamps are equal, then use the + peer ID to break the tie) + +Contrary to +[Yjs's method of using control characters](https://www.inkandswitch.com/peritext/#adding-control-characters-to-plain-text) +for rich text, our algorithm pairs start and end anchors when they originate +from the same style operation. This approach accurately handles the following +scenarios: + +![overlap_bold](./loro-richtext/overlap_bold.png) + +These special control characters are not exposed to the user; each control +character is effectively of zero length from the user's perspective. Our data +structure supports various methods of measuring text length for indexing text +content. Besides Unicode, UTF-16, and UTF-8, we also measure our rich text +length in `Entity length`. It treats each style anchor as an entity with a +length of 1 and measures plain text in Unicode length. + +![len](./loro-richtext/len.png) + +| Concept | Definition | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Style Anchors | Control characters used in Loro denote style boundaries' start and end. They are differentiated into start and end anchors, representing a style's beginning and end. | +| Rich Text Element | A rich text element is either a span of text or a style anchor. A list of rich text elements represents the internal state of Loro's rich text. | +| Unicode Index | A method of indexing text positions in rich text. In this method, the length of the text is measured in Unicode char length, and the length of style anchors is considered 0. | +| Entity Index | A method of indexing text positions in rich text. In this method, the length of the text is measured in Unicode char length, and the length of a style anchor is considered 1. | + +#### Local Behavior + +Multiple valid insertion points can exist when users insert text at a specific +Unicode index. It occurs due to style anchors, which are zero-length elements +from the user's perspective. + +![Two Different Kinds of Indexes](./loro-richtext/two_index.png) + +For example, in the case of `Hello world`, when a user inserts content at +Unicode `index=5`, they face the choice of inserting to the left or right of +``. If the user sets the expand behavior of bold to expand forward, the new +character will be inserted to the left of ``, making the inserted text bold +as well. + +When users delete text, Loro uses an additional mapping layer to avoid deleting +style anchors within the text range. + +To model the deletion of a style, a new style anchor pair with a null value is +added. + +We can implement the following optimizations to remove redundant style anchors: + +- The style anchors that include no text can be removed. +- When styles completely negate each other, like a span of bold is canceled by a + span of unbold, we can remove their style anchors. + +All these behaviors happen locally, and the algorithm is independent of the +specific List CRDT. + +##### Behavior When Inserting Text at Style Boundaries + +Most modern rich text editors (Google Doc, Microsoft Word, Notion) behave as +follows: when new text is entered right after bold text, the new text should +inherit the bold style; when entered after a hyperlink, the new content should +not inherit the hyperlink style. Different styles have varying preferences for +text insertion positions, leading to potential conflicts. This is reflected in +the degree of freedom we have when inserting new text. + +Users interact with rich text based on text-based indexes, like the Unicode +index. Since style anchors have a Unicode Length of 0, a Unicode index with n +style anchors presents n + 1 potential insertion positions. + +We select the insertion position based on the following rules: + +1. Insertions occur before a start anchor of a style that should not expand + backward. +2. Insertions occur before style anchors that signify the end of bold-like marks + (expand = "after" or expand = "both"). +3. Insertions occur after style anchors that signify the end of link-like marks + (expand = "none" or expand = "before"). + +Rule 1 should be prioritized over rules 2 and 3 to prevent +[the accidental creation of a new style](https://github.com/inkandswitch/peritext/issues/32). + +The current method first scans forward to find the last position satisfying +rules 1 and 2. + +Then, it scans backward to find the first position satisfying rule 3. + +#### Merging Remote Updates + +Loro treats style anchors as a special element and handles them using the same +List CRDT for resolving concurrent conflicts. The logic related to rich text is +independent of the particular List CRDT. Therefore, this algorithm can rely on +any List CRDT algorithm for merging remote operations. Loro utilizes the [Fugue] +List CRDT algorithm. + +When new style anchors are inserted by remote updates, new styles are added; if +old style anchors are deleted, the corresponding old styles are removed. + +#### Strong Eventual Consistency + +The internal state of this algorithm consists of a list of elements, each either +a text segment or a style anchor. The rich text document is derived from this +internal state. + +The internal state achieves strong eventual consistency through the upstream +List CRDT. + +Identical internal states result in identical rich text documents. Hence, the +same set of updates will produce the same rich text documents, evidencing the +strong eventual consistency of this algorithm. + +### Criteria in Peritext + +[The Peritext paper](https://www.inkandswitch.com/peritext/static/cscw-publication.pdf) +specifies the intent-preserving merge behavior for rich text inline format. +Loro's rich text algorithm successfully passes all test cases outlined therein. + +#### 1. Concurrent Formatting and Insertion + +| Name | Text | +| :-------------- | :------------------ | +| Origin | Hello World | +| Concurrent A | **Hello World** | +| Concurrent B | Hello New World | +| Expected Result | **Hello New World** | + +Loro easily supports this case by treating style anchors as special elements +alongside text. + +#### 2. Overlapping Formatting + +| Name | Text | +| :-------------- | :-------------- | +| Origin | Hello World | +| Concurrent A | **Hello** World | +| Concurrent B | Hel**lo World** | +| Expected Result | **Hello World** | + +This case has been analyzed earlier. Since our style anchors contain style op ID +information, we know there are two bold segments: one from 0 to 5 and another +from 3 to 11, allowing us to merge them. + +| Name | Text | +| :-------------- | :------------------ | +| Origin | Hello World | +| Concurrent A | **Hello** World | +| Concurrent B | Hel*lo World* | +| Expected Result | **Hel*lo*** _World_ | + +Multiple style types are easily supported. + +| Name | Text | Note | +| :-------------- | :--------------------------------------------------- | :------------------ | +| Origin | Hello World | | +| Concurrent A | **Hello World**
Then
**Hello** World | Bold, then unbold | +| Concurrent B | Hello Wor**ld** | | +| Expected Result | **Hello** Wor**ld**
Or
**Hello** World | Both are acceptable | + +Like Peritext, we model unbolding by adding a new style with the key `bold` and +the value `null`. The final value of each style key on each character is +determined by the style with the greatest [Lamport] timestamp that includes the +character. Thus, it easily supports this case. + +#### 3. Text Insertion at Span Boundaries + +Insertion right after a bold style should result in the newly inserted text also +being bold. + +
+ ![bold_expand](./loro-richtext/bold_expand.png) +
+ +However, insertion right after a link style should result in the newly inserted +text not having the hyperlink style. + +
+ ![Link style should not expand](./loro-richtext/link_expand.png) +
+ +#### 4. Styles that Support Overlapping + +
![](./loro-richtext/overlap_mark.png)
+ +The problem of overlapping styles is related to how we represent them. + +We represent the rich text using +[Quill's Delta](https://quilljs.com/docs/delta/) format. + +```ts no_run +[ + { insert: "Gandalf", attributes: { bold: true } }, + { insert: " the " }, + { insert: "Grey", attributes: { color: "#cccccc" } }, +]; +``` + +An example of Quill's Delta format + +However, it cannot handle cases with multiple values assigned to the same key. +So, it's a headache to handle the styles that support overlapping. + +![](./loro-richtext/overlap_comments.png) + +For example, in the above case, the text "fox" is commented on by both Alice and +Bob. We can't represent it with Quill's Delta format directly. So the possible +workaround includes: + +**Turn the attribute value into a list** + +```ts no-run +[ + { insert: "The ", attributes: { comment: [{ ...commentA }] } }, + { + insert: "fox", + attributes: { comment: [{ ...commentA }, { ...commentB }] }, + }, + { insert: " jumped", attributes: { comment: [{ ...commentB }] } }, +]; +``` + +**Use op ID that creates the op as the key of the attribute** + +```ts no-run +[ + { insert: "The ", attributes: { "id:0@A": { key: "comment", ...commentA } } }, + { + insert: "fox", + attributes: { + "id:0@A": { key: "comment", ...commentA }, + "id:0@B": { key: "comment", ...commentB }, + }, + }, + { + insert: " jumped", + attributes: { "id:0@B": { key: "comment", ...commentA } }, + }, +]; +``` + +But both require special behaviors for both CRDT lib and for application code, +which are painful to work with. + +Finally, we found that the optimal approach to represent an overlappable style +was to use `:` as a prefix and allow users to assign a unique suffix to +create a distinct style key. This method simplifies the CRDTs library code, as +it doesn't require handling special cases. It effectively addresses scenarios +where multiple comments overlap and is also user-friendly for application +coding. + +```ts no-run +[ + { insert: "The ", attributes: { "comment:alice": "Hi" } }, + { + insert: "fox", + attributes: { "comment:alice": "Hi", "comment:bob": "Jump" }, + }, + { insert: " jumped", attributes: { "comment:bob": "Jump" } }, +]; +``` + +Following is the example code in Loro: + +```ts +const doc = new Loro(); +doc.configTextStyle({ + comment: { expand: "none" }, +}); +const text = doc.getText("text"); +text.insert(0, "The fox jumped."); +text.mark({ start: 0, end: 7 }, "comment:alice", "Hi"); +text.mark({ start: 4, end: 14 }, "comment:bob", "Jump"); +expect(text.toDelta()).toStrictEqual([ + { + insert: "The ", + attributes: { "comment:alice": "Hi" }, + }, + { + insert: "fox", + attributes: { + "comment:alice": "Hi", + "comment:bob": "Jump", + }, + }, + { + insert: " jumped", + attributes: { "comment:bob": "Jump" }, + }, + { + insert: ".", + }, +]); +``` + +## Implementation of Loro's Rich Text Algorithm + +The following is an overview of Loro's implementation as of January, 2024. + +### Architecture of Loro + +In line with the properties of Event Graph Walker, Loro uses `OpLog` and +`DocState` as the internal state. + +`OpLog` is dedicated to recording history, while `DocState` only records the +current document state and does not include historical operation information. +When applying updates from remote sources, Loro uses the relevant operations +from `OpLog` and computes the diff through a `DiffCalculator`. This diff is then +applied to `DocState`. This architecture also makes time travel easier to +implement. + +For more details, see the documentation on +[DocState and OpLog](https://loro.dev/docs/advanced/doc_state_and_oplog). + +![](./loro-richtext/apply_updates.png) + +### Implementation of Loro's Rich Text CRDT + +For rich text, Loro reuses the same `DiffCalculator` as Loro List, based on the +[Fugue] algorithm. As a result, the primary logic related to rich text is +concentrated in `DocState`. This includes expressing styles, inserting new +characters, and representing multiple index formats. + +In the representation of rich text state, we distinguish between the data +structure `ContentTree`, which expresses the text (including style anchors), and +`StyleRangeMap`, which expresses styles. + +![](./loro-richtext/text_state_arch.png) + +Both structures are built on B+Trees. + +`ContentTree` is responsible for efficient text finding, insertion, and +deletion. It can index specific insertion/deletion positions using +Unicode/UTF-8/UTF-16/Entity index. It does not store what specific style each +text segment should have. + +We built the following B+Tree structure based on our +[generic-btree library](https://github.com/loro-dev/generic-btree) to express +text in memory: + +- Each internal node in the B+Tree stores the Unicode char length, UTF-16 + length, UTF-8 length, and Entity length of its subtree. The Entity length + considers the length of style anchors as 1, otherwise 0. +- The leaf nodes of the B+Tree are text or style anchors. + +`StyleRangeMap` is responsible for efficient updating/querying of style ranges. + +In the `StyleRangeMap` B+Tree expressing styles: + +- Each internal node stores the `entity length` of its subtree. +- Each leaf node stores the collection of style information for the + corresponding range and its `entity length. + +Separating the text `ContentTree` and style `StyleRangeMap` into two structures +aims for better performance optimization. On rich text, style information is +often not abundant and tends to have good continuity, such as several paragraphs +having the same format, which can be expressed with a single leaf node. However, +our structure for storing text is unsuitable for leaf nodes with large content, +as conversion time between different encoding formats would become excessively +long. + +When a user inserts a new character at `Unicode index` = i, the following +occurs: + +- Find the position at `Unicode index` = i in `ContentTree`. +- Check if there are any adjacent style anchors at this position. If not, + directly insert. +- If there are, decide whether to insert to the left or right of the + corresponding style anchor based on its type and properties. If there are + multiple such style anchors, insert them according to the previous section on + ["Behavior When Inserting Text at Style Boundaries"](#behavior-when-inserting-text-at-style-boundaries). + +### Testing + +We have written tests for the criteria proposed by Peritext and passed all of +them. + +To ensure the correctness of our CRDTs, we have added numerous fuzzing tests to +simulate different collaborative behaviors, synchronization behaviors, and +time-travel behaviors. These tests check for the strong eventual consistency and +the correctness of internal invariants. We run these fuzzing tests continuously +for several days after every critical modification to avoid oversights. + +## How to Use + +Before using the Loro's rich text module, it is necessary to define the +configuration for rich text styles, specifying the expand behavior for different +keys and whether overlap is allowed. + +Here is an example of using Loro's rich text in JavaScript: + +```typescript +const doc = new Loro(); +doc.configTextStyle({ + bold: { + expand: "after", + }, + comment: { + expand: "none", + overlap: true, + }, + link: { + expand: "none", + }, +}); + +const text = doc.getText("text"); +text.insert(0, "Hello world!"); +text.mark({ start: 0, end: 5 }, "bold", true); +expect(text.toDelta()).toStrictEqual([ + { + insert: "Hello", + attributes: { bold: true }, + }, + { + insert: " world!", + }, +] as Delta[]); + +text.insert(5, "!"); +expect(text.toDelta()).toStrictEqual([ + { + insert: "Hello!", + attributes: { bold: true }, + }, + { + insert: " world!", + }, +] as Delta[]); +``` + +## Summary + +This article presents Loro's rich text algorithm design and implementation. Its +correctness is readily demonstrable. It can be built upon any existing List CRDT +algorithm. It allows Loro to support rich text collaboration using +[Eg-walker](#brief-introduction-to-replayable-event-graph) and [Fugue], combining the +strengths of multiple CRDT algorithms. + +We are continuously refining its design and actively seeking design partners. We +are open to all forms of feedback and constructive criticism. Should you have +any proposals for collaboration, please reach out to zx@loro.dev + +[Peritext]: https://www.inkandswitch.com/peritext/ +[Fugue]: https://arxiv.org/abs/2305.00583 +[Lamport]: https://en.wikipedia.org/wiki/Lamport_timestamp +[RGA algorithm]: https://www.sciencedirect.com/science/article/abs/pii/S0743731510002716 +[Yjs]: https://github.com/yjs/yjs +[Automerge]: https://github.com/automerge/automerge +[crdt-richtext]: https://github.com/loro-dev/crdt-richtext + + +# FILE: pages/blog/loro-now-open-source.mdx + +--- +title: "Loro: Reimagine State Management with CRDTs" +date: 2023/11/13 +description: + "Loro, our high-performance CRDTs library, is now open source. In this + article, we share our vision for the local-first software development + paradigm, why we're excited about it, and the current status of Loro." +--- + +# Loro: Reimagine State Management with CRDTs + +import Caption from "../../components/caption"; +import GitHub from "../../components/github"; +import Authors, { Author } from "../../components/authors"; + + + + + + +
+
+ Loro, our high-performance CRDTs library, is now open source +
+ +
.
+
+ +In this article, we share our vision for the local-first software development +paradigm, explain why we're excited about it, and discuss the current status of +Loro. + +With better DevTools, documentation, and a friendly ecosystem, everyone can +easily build local-first software. + +![Loro's 'time machine' example](./loro-now-open-source/colab_and_travel.gif) + + + You can build collaborative apps with time travel features easily using Loro. + [Play the example online](https://loro-react-flow-example.vercel.app/). + + +## Envisioning the Local-First Development Paradigm + +Distributed states are commonly found in numerous scenarios, such as multiplayer +games, multi-device document synchronization, and edge networks. These scenarios +require synchronization to achieve consistency, usually entailing elaborate +design and coding. For instance, considerations for network issues or concurrent +write operations are necessary. However, for a wide range of applications CRDTs +can simplify the code significantly: + +- CRDTs can automatically merge concurrent writes without conflicts. +- Fewer abstractions. There's no need to design specific backend database + schemas, manually execute expected conflict merges, or implement interfaces to + memory and memory to persistent structure conversions. +- Offline supports are right out of the box + +
+What are CRDTs + +### What are Conflict-Free Replicated Data Types (CRDTs)? + +CRDTs are data structures used in distributed systems that allow updates to be +merged across multiple replicas without conflicts. In this context, "replicas" +refer to different independent data instances within the system, such as the +same collaborative document on various user devices. + +CRDTs enable users to operate independently on their replicas, like editing a +document, without needing real-time communication with other replicas. The CRDTs +merge these operations, ensuring all replicas achieve "strong eventual +consistency". As long as all nodes receive the same set of updates, regardless +of the order, their data states will eventually be consistent. + +> For more details, visit +> [What are CRDTs](https://www.loro.dev/docs/concepts/crdt) + +
+ +
+When you can't use CRDTs +### When you can't use CRDTs + +CRDTs only guarantee _Strong Eventual Consistency_. You have to make sure it's +suitable for your application. + +"Strong Eventual Consistency": As long as all nodes receive the same set of +updates, their data states will ultimately become consistent regardless of their +sequence. + +Strong eventual consistency may not be acceptable in scenarios requiring +immediate consistency or transactional integrity, such as financial +transactions, exclusive resource access, or allocation. + +
+ +Since the data resides locally, client applications can directly access and +manipulate local data, offering both speed and availability. Additionally, due +to CRDTs' nature, synchronization / real-time collaboration can be achieved +without relying on centralized servers (similar to Git, allowing migration to +other platforms without data loss). With performance improvements, CRDTs +increasingly replace traditional real-time collaboration solutions in various +contexts. + +This represents a new paradigm. Local-first not only empowers users with control +over their data, but also makes developers' lives easier. + +![Local-first](./loro-now-open-source/Untitled.png) + + + The annual growth rate of the *"local-first"* star count in GitHub has reached + 40%+. + + +### Integrating CRDTs with UI State Management + +![Loro's rich text collaboration example](./loro-now-open-source/richtext.gif) + +Loro's rich text collaboration example + +Since CRDTs enable conflict-free automatic merging, the challenge of managing +distributed states shifts to "how to express operations and states on CRDTs". + +Front-end state management libraries typically require developers to define how +to retrieve State and specify Actions, as illustrated by this example from Vue's +state management tool, Pinia: + +```ts no-run +export const useCartStore = defineStore({ + id: "cart", + state: () => ({ + rawItems: [] as string[], + }), + getters: { + items: (state): Array<{ name: string; amount: number }> => + state.rawItems.reduce( + (items, item) => { + const existingItem = items.find((it) => it.name === item); + + if (!existingItem) { + items.push({ name: item, amount: 1 }); + } else { + existingItem.amount++; + } + + return items; + }, + [] as Array<{ name: string; amount: number }>, + ), + }, + actions: { + addItem(name: string) { + this.rawItems.push(name); + }, + + removeItem(name: string) { + const i = this.rawItems.lastIndexOf(name); + if (i > -1) this.rawItems.splice(i, 1); + }, + + async purchaseItems() { + const user = useUserStore(); + if (!user.name) return; + + console.log("Purchasing", this.items); + const n = this.items.length; + this.rawItems = []; + + return n; + }, + }, +}); +``` + +This paradigm and CRDTs are easily compatible: The state in the state management +libraries corresponds to CRDT types, and Action corresponds to a set of CRDT +operations. + +Thus, implementing UI state management through CRDTs does not require users to +change their habits. It also has many advanced features: + +- Make states automatically synchronizable / support real-time collaboration. +- Like Git, maintain a complete distributed editing history. +- It can store an extensively large editing history with a low memory footprint + and a compact encoding size. Below is an example. + +With this, you can effortlessly implement products with real-time / async +collaboration and time machine features. + +![Tracing a document with 360,000 operations using Loro](./loro-now-open-source/Untitled.gif) + + +
+ Time travel a document with 360,000+ operations using Loro. To load the + whole history and playback, it only takes 8.4MB in memory. And the entire + history only takes 361KB in storage. The editing trace is from{" "} +
+ +
.
+ + +## Introduction to Loro + +Loro is our CRDTs library, now open-sourced under a permissive license. We +believe a cooperative and friendly open-source community is key to creating +outstanding developer experiences. + +We aim to make Loro simple to use, extensible, and maintain high performance. +The following is the latest status of Loro. + +### CRDTs + +We have explored extensively, supporting a range of CRDT algorithms that have +yet to be widely used. + +#### OT-like CRDTs + +> Update: This algorithm is now called Event Graph Walker (Eg-Walker) + +Our CRDTs library is built on the brilliant concept of OT-like CRDTs from Seph +Gentle's [Diamond-types](https://github.com/josephg/diamond-types). Joseph Gentle +and Martin Kleppmann later published the +[Eg-walker paper](https://arxiv.org/abs/2409.14252) on this line of work. Its +notable features include reducing the cost of local operations, easier +historical data reclamation, and sometimes lower storage and memory overhead. +However, it relies on high-performance algorithms to apply remote operations. +This design has great potential and we are excited about its future. + +
+Brief Introduction to OT-like CRDT algorithms + +To briefly introduce the concept of OT-like CRDTs, this part is complex and +requires some prior knowledge. I might not encapsulate it well. + +The general idea of OT-like CRDTs is that they do not retain the CRDTs' data +structure (e.g., originLeft originRight information). When merging remote +operations, they return to the lowest common ancestor in the directed acyclic +graph history of local and remote, and from there, reapply each operation. This +process reconstructs the CRDTs structure, resolving conflicts arising from +parallel editing. Its advantage is that, since it doesn't need to retain these +CRDTs Meta information, local operations are virtually cost-free, like OT, where +only the index at which insertions and deletions occur needs to be saved. The +trade-off is a longer time for merging remote operations, but this issue can be +significantly mitigated with well-designed data structures and algorithms. +Moreover, since most parallel edits last only a short time, the lowest common +ancestor is not far, making the merging process quick. + +The image below shows an example of merging versions 2@1 and 1@2 using this +algorithm on a DAG. The algorithm needs to revert to the lowest common ancestor +version 0@1 and apply all subsequent operations from there (a total of four +operations). For a better understanding of this image, refer to +[https://www.loro.dev/docs/advanced/version_deep_dive](https://www.loro.dev/docs/advanced/version_deep_dive) + +![Untitled](./loro-now-open-source/Untitled%201.png) + +
+ +#### Rich Text CRDTs + +In May of this year, we open-sourced the +[crdt-richtext](https://github.com/loro-dev/crdt-richtext) project, integrating +the algorithms of [the rich text CRDT](https://loro.dev/blog/loro-richtext) and the +sequence CRDT [Fugue by Matthew Weidner](https://arxiv.org/abs/2305.00583). A +brief introduction to these two algorithms can be found in +[our blog at the time](https://www.notion.so/crdt-richtext-Rust-implementation-of-Peritext-and-Fugue-c49ef2a411c0404196170ac8daf066c0?pvs=21). + +Based on our experience from previous projects, we have integrated a rich text +CRDT and Fugue into our framework in the current Loro. However, the biggest +challenge was that +[Peritext did not integrate well with OT-like CRDTs](https://github.com/inkandswitch/peritext/issues/31). +We have recently overcome this issue. We developed a new rich text CRDT +algorithm that can run on OT-like CRDTs and has passed the capabilities listed +in the Peritext paper's Criteria for rich text CRDTs, with no new issues +revealed in our current million fuzzing tests. We later introduced this work in +the [Loro rich text article](https://loro.dev/blog/loro-richtext). + +#### Movable Tree + +We have also supported a movable tree CRDT. Synchronizing tree movements is +often complex due to the potential for circular references. Addressing this +issue in the distributed environment is even more challenging. + +We implemented Martin Kleppmann's paper, +[_A Highly-Available Move Operation for Replicated Trees_](https://ieeexplore.ieee.org/document/9563274/). +The idea of this algorithm is to sort all move operations, ensuring the ordering +is consistent across the replicas. Then, each operation is applied sequentially. +If an operation would cause a circular reference, it has no effect. + +We found it to be elegant in design and also performant. The time complexity of +local operations is O(k) (k being the average tree depth, as circular reference +detection is required). For applying remote operations, which entails inserting +new operations into the sorted list, we must undo operations that are subsequent +in the ordering, apply the remote operation, and then redo the undone +operations, with a cost of O(km) (m being the number of operations to undo). + +![Untitled](./loro-now-open-source/Untitled%201.gif) + +Visualization of applying a remote op + +Our tests show that local operations involving ten thousand random movements +among a thousand nodes take less than 10ms (tested on an M2 MAX chip). Moreover, +the cost of merging remote operations in this algorithm is similar to applying +remote operations in OT-like CRDTs, making it adoptable. We've also experimented +with [log-spaced snapshots](https://madebyevan.com/algos/log-spaced-snapshots/) +and immutable data structure approaches in our +[movable-tree project](https://github.com/loro-dev/movable-tree), concluding +that the undo + redo method is the fastest and the most memory-efficient. + +### Data Structures + +Designing and experimenting with data structures is routine in Loro's +development process. + +We previously open-sourced +[generic-btree](https://github.com/loro-dev/generic-btree) and have redesigned +its structure for a more compact memory layout and cache-friendliness. Besides +its remarkable performance, its flexibility enables us to support various +information types required for Text, like utf16/Unicode code points/utf8, with +minimal code. We also extensively reuse it to fulfill various requirements, +highlighting Rust's impressive type expression capabilities. + +Internally, we've +[separated the document's state from its history](https://www.loro.dev/docs/advanced/doc_state_and_oplog). +The state represents the current form of the document, akin to Git's HEAD +pointer, while the document's history resembles the complete operation history +behind Git. Hence, multiple document states can correspond to the same history. +This structure simplifies our code and provides the foundation for Loro's +version-control primitives. + +Most of our optimizations thus far have focused on text manipulations, +historically one of the thorniest problems in CRDTs. Later releases expanded +these optimizations across more real-world scenarios. + +### Update + +![Untitled](./loro-now-open-source/Untitled%202.png) + +Loro has since reached 1.0 and stabilized its core data format. The project now +ships production-ready JavaScript/WASM and Rust packages, plus maintained +bindings and examples for other platforms. + +Documentation and developer tooling remain active areas of work. We welcome +ideas and suggestions in our +[community discussions](https://discord.gg/tUsBSVfqzf). + +Developing tools for developers is a challenging and exciting task. Many +developer tools and visualization methods in front-end development are +exceptionally good, and we hope to bring such experiences into the world of +CRDTs and local-first development. DevTools will reveal CRDTs' hidden states and +simplify control, making state maintenance and debugging a breeze. + +Loro now supports richer CRDT semantics, including movable data structures and +undo/redo APIs for more diverse application scenarios. + +## Seeking Collaborative Project Opportunities + +Our design and optimization efforts need feedback from real-world applications. +If you are excited about a local-first future and think Loro can help you, +please contact us directly at [zx@loro.dev](mailto:zx@loro.dev). We're open to +collaboration and ready to help. + + +# FILE: pages/blog/movable-tree.mdx + +--- +title: "Movable tree CRDTs and Loro's implementation" +date: 2024/07/18 +description: This article introduces the implementation difficulties and challenges of Movable Tree CRDTs when collaboration, and how Loro implements it and sorts child nodes. The algorithm has high performance and can be used in production. +image: https://i.ibb.co/nMrgzZJ/DALL-E-2024-01-31-21-29-16-Create-a-black-and-white-illustration-with-a-black-background-that-matche.png +--- + +# Movable tree CRDTs and Loro's implementation + +import Caption from "../../components/caption"; +import Authors, { Author } from "../../components/authors"; + + + + + +![](./movable-tree/movable-tree-cover.png) + +This article introduces the implementation difficulties and challenges of Movable Tree CRDTs when collaboration, and how Loro implements it and sorts child nodes. The algorithm has high performance and can be used in production. + +## Background + +In distributed systems and collaborative software, managing hierarchical relationships is difficult and complex. Challenges arise in resolving conflicts and meeting user expectations when working with the data structure that models movement by combining deletion and insertion. For instance, if a node is concurrently moved to different parents in replicas, it may lead to the unintended creation of duplicate nodes with the same content. Because the node is deleted twice and created under two parents. + +Currently, many software solutions offer different levels of support and functionality for managing hierarchical data structures in distributed environments. The key variation among these solutions lies in their approaches to handling potential conflicts. + +### Conflicts in Movable Trees + +A movable tree has 3 primary operations: creation, deletion, and movement. Consider a scenario where two peers independently execute various operations on their respective replicas of the same movable tree. Synchronizing these operations can lead to potential conflicts, such as: + +- The same node was deleted and moved +- The same node was moved under different nodes +- Different nodes were moved, resulting in a cycle +- The ancestor node is deleted while the descendant node is moved + +#### Deletion and Movement of the Same Node + +![Deletion and Movement of the Same Node](./movable-tree/move-delete-dark.png) + +This situation is relatively easy to resolve. It can be addressed by applying one of the operations while ignoring the other based on the timestamp in the distributed system or the application's specific requirements. Either approach yields an acceptable outcome. + +#### Moving the Same Node Under Different Parents + +![Moving the Same Node Under Different Parents](./movable-tree/move-same-node-dark.png) + +Merging concurrent movement operations of the same node is slightly more complex. Different approaches can be adopted depending on the application: + +- Delete the node and create copies of nodes under different parent nodes. Subsequent operations then treat these nodes independently. This approach is acceptable when node uniqueness is not critical. +- Allow the node have two edges pointing to different parents. However, this approach breaks the fundamental tree structure and is generally not considered acceptable. +- Sort all operations, then apply them one by one. The order can be determined by timestamps in a distributed system. Providing the system maintains a consistent operation sequence, it ensures uniform results across all peers. + +#### Movement of Different Nodes Resulting in a Cycle + +![cycle](./movable-tree/cycle-dark.png) + +Concurrent movement operations that cause cycles make the conflict resolution of movable trees complex. Matthew Weidner listed several solutions to resolve cycles in his [blog](https://mattweidner.com/2023/09/26/crdt-survey-2.html#forests-and-trees). + +> 1. Error. Some desktop file sync apps do this in practice ([Martin Kleppmann et al. (2022)](https://doi.org/10.1109/TPDS.2021.3118603) give an example). +> 2. Render the cycle nodes (and their descendants) in a special “time-out” zone. They will stay there until some user manually fixes the cycle. +> 3. Use a server to process move ops. When the server receives an op, if it would create a cycle in the server’s own state, the server rejects it and tells users to do likewise. This is [what Figma does](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/#syncing-trees-of-objects). Users can still process move ops optimistically, but they are tentative until confirmed by the server. (Optimistic updates can cause temporary cycles for users; in that case, Figma uses strategy (2): it hides the cycle nodes.) +> 4. Similar, but use a [topological sort](https://mattweidner.com/2023/09/26/crdt-survey-2.html#topological-sort) (below) instead of a server’s receipt order. When processing ops in the sort order, if an op would create a cycle, skip it [(Martin Kleppmann et al. 2022)](https://doi.org/10.1109/TPDS.2021.3118603). +> 5. For forests: Within each cycle, let `B.parent = A` be the edge whose `set` operation has the largest LWW timestamp. At render time, “hide” that edge, instead rendering `B.parent = "none"`, but don’t change the actual CRDT state. This hides one of the concurrent edges that created the cycle. +> • To prevent future surprises, users’ apps should follow the rule: before performing any operation that would create or destroy a cycle involving a hidden edge, first “affirm” that hidden edge, by performing an op that sets `B.parent = "none"`. +> 6. For trees: Similar, except instead of rendering `B.parent = "none"`, render the previous parent for `B` - as if the bad operation never happened. More generally, you might have to backtrack several operations. Both [Hall et al. (2018)](http://dx.doi.org/10.1145/3209280.3229110) and [Nair et al. (2022)](https://arxiv.org/abs/2103.04828) describe strategies along these lines. + +#### Ancestor Node Deletion and Descendant Node Movement + +![Ancestor Node Deletion and Descendant Node Movement](./movable-tree/move_chlid_delete_parent_dark.png) + +The most easily overlooked scenario is moving descendant nodes when deleting an ancestor node. If all descendant nodes of the ancestor are deleted directly, users may easily misunderstand that their data has been lost. + +### How Popular Applications Handle Conflicts + +Dropbox is a file data synchronization software. Initially, Dropbox treated file movement as a two-step process: deletion from the original location followed by creation at a new location. However, this method risked data loss, especially if a power outage or system crash occurred between the delete and create operations. + +Today, when multiple people move the same file concurrently and attempt to save their changes, Dropbox detects a conflict. In this scenario, it typically saves one version of the original file and creates a new ["conflicted copy"](https://help.dropbox.com/organize/conflicted-copy) for the changes made by one of the users. + +![Solution for conflicts when moving files with Dropbox](./movable-tree/dropbox_move.gif) + + + The image shows the conflict that occurs when A is moved to the B folder and B + is moved to the A folder concurrently. + + +Figma is a real-time collaborative prototyping tool. They consider tree structures as the most complex part of the collaborative system, as detailed in their [blog post about multiplayer technology](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/#syncing-trees-of-objects). To maintain consistency, each element in Figma has a "parent" attribute. The centralized server plays a crucial role in ensuring the integrity of these structures. It monitors updates from various users and checks if any operation would result in a cycle. If a potential cycle is detected, the server rejects the operation. + +However, due to network delays and similar issues, there can be instances where updates from users temporarily create a cycle before the server has the chance to reject them. Figma acknowledges that this situation is uncommon. Their [solution](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/#syncing-trees-of-objects) is straightforward yet effective: they temporarily preserve this state and hide the elements involved in the cycle. This approach lasts until the server formally rejects the operation, ensuring both the stability of the system and a seamless user experience. + +
+ ![An animation that demonstrates how Figma resolves + conflicts.](./movable-tree/figma-tree.gif) +
+ + + An animation that demonstrates how + [Figma](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/#syncing-trees-of-objects) + resolves conflicts. + + +## Movable Tree CRDTs + +The applications mentioned above use movable trees and resolve conflicts based on centralized solutions. Another alternative approach to collaborative tree structures is using Conflict-free Replicated Data Types (CRDTs). While initial CRDT-based algorithms were challenging to implement and incurred significant storage overhead as noted in prior research, such as [Abstract unordered and +ordered trees CRDT](https://arxiv.org/pdf/1201.1784.pdf) or [File system on CRDT](https://arxiv.org/pdf/1207.5990.pdf), but continual optimization and improvement have made several CRDT-based tree synchronization algorithms suitable for certain production environments. This article highlights two innovative CRDT-based approaches for movable trees. The first is presented by Martin Kleppmann et al. in their work **_[A highly-available move operation for replicated trees](https://martin.kleppmann.com/2021/10/07/crdt-tree-move-operation.html)_** and the second by Evan Wallace in his **_[CRDT: Mutable Tree Hierarchy](https://madebyevan.com/algos/crdt-mutable-tree-hierarchy/)_**. + +### A highly-available move operation for replicated trees + +This paper unifies the three operations used in trees (creating, deleting, and moving nodes) into a move operation. The move operation is defined as a four-tuple `Move t p m c`, where `t` is the operation's unique and ordered timestamp such as [`Lamport timestamp`](https://en.wikipedia.org/wiki/Lamport_timestamp), `p` is the parent node ID, `m` is the metadata associated with the node, and `c` is the child node ID. + +If all nodes of the tree do not contain `c`, this is a **creation** operation that creates a child node `c` under parent node `p`. Otherwise, it is a **move** operation that moves `c` from its original parent to the new parent `p`. Additionally, node deletion is elegantly handled by introducing a designated `TRASH` node; moving a node to `TRASH` implies its deletion, with all descendants of `TRASH` considered deleted. But they remain in memory to prevent concurrent editing from moving them to other nodes. In order to handle the previously mentioned situation of deleting ancestor nodes and moving descendant nodes concurrently. + +In the three potential conflicts mentioned earlier, since deletion is also defined as a move operation, **deleting and moving the same node** is transformed into two move operations, leaving only two remaining problems: + +- **Moving the same node under different parents** +- **Moving different nodes, creating a cycle** + +Logical timestamps are added so that all operations can be linearly ordered, thus the first conflict can be avoided as they can be expressed as two operations in sequence rather than concurrently for the same node. Therefore, in modeling a Tree using only move operations, the only exceptional case in concurrent editing would be creating a cycle, and operations causing a cycle are termed **unsafe operations**. + +This algorithm sorts all move operations according to their timestamps. It can then sequentially apply each operation. Before applying, the algorithm detects cycles to determine whether an operation is safe. If the operation creates a cycle, we ignore the unsafe operation to ensure the correct structure of the tree. + + +Based on the above approach, the consistency problem of movable trees becomes the following two questions: + +1. How to introduce global order to operations +2. How to apply a remote operation that should be inserted in the middle of an existing sorted sequence of operations + +#### Globally Ordered Logical Timestamps + +[Lamport Timestamp](https://en.wikipedia.org/wiki/Lamport_timestamp) can determine the causal order of events in a distributed system. Here's how they work: each peer starts with a counter initialized to `0`. When a local event occurs, the counter is increased by `1`, and this value becomes the event's Lamport Timestamp. When peer `A` sends a message to peer `B`, `A` attaches its Lamport Timestamp to the message. Upon receiving the message, peer `B` compares its current logical clock value with the timestamp in the message and updates its logical clock to the larger value. + +To globally sort events, we first look at the Lamport Timestamps: smaller numbers mean earlier events. If two events have the same timestamp, we use the unique ID of the peer serves as a tiebreaker. + + +#### Apply a Remote Operation + +An op's safety depends on the tree's state when applied, avoiding cycles. Insertion requires evaluating the state formed by all preceding ops. For remote updates, we may need to: + +1. Undo recent ops +2. Insert the new op +3. Reapply undone ops + +This ensures proper integration of new ops into the existing sequence. + +##### Undo Recent Ops + +Since we've modeled all operations on the tree as move operations, undoing a move operation involves either moving the node back to its old parent or undoing the operation that created this node. To enable quick undoing, we cache and record the **old parent** of the node before applying each move operation. + +##### Apply the Remote Op + +Upon encountering an unsafe operation, disregarding its effects prevents the creation of a cycle. Nevertheless, it's essential to record the operation, as the safety of an operation is determined **dynamically**. For instance, if we receive and sort an update that deletes another node causing the cycle prior to this operation, the operation that was initially unsafe becomes safe. Additionally, we need to mark this unsafe operation as ineffective, since during undo operations, it's necessary to query the **old parent** node, which is the target parent of the last effective operation in the sequence targeting this node. + +##### Reapply Undone Ops + +Cycles only occur when receiving updates from other peers, so the undo-do-redo process is also needed at this time. When receiving a new op: + +```jsx +function apply(newOp) + // Compare the ID of the new operation with existing operations + if largerThanExistingOpId(newOp.id, oplog) + // If the new operation's ID is greater, apply it directly + oplog.applyOp(newOp) + else + // If the new operation's ID is not the greatest, undo operations until it can be applied + undoneOps = oplog.undoUtilCanBeApplied(newOp) + oplog.applyOp(newOp) + // After applying the new operation, redo the undone operations to maintain sequence order + oplog.redoOps(undoneOps) +``` + +- If the new operation depends on an op that has not been encountered locally, indicating that some inter-version updates are still missing, it is necessary to temporarily cache the new op and wait to apply it until the missing updates are received. +- Compare the new operation with all existing operations. If the `opId` of the new operation is greater than that of all existing operations, it can be directly applied. If the new operation is safe, record the parent node of the target node as the old parent node, then apply the move operation to change the current state. If it is not safe, mark this operation as ineffective and ignore the operation's impact. +- If the new opId is sorted in the middle of the existing sequence, it is necessary to pop the operations that are sorted later from the sequence one by one, and undo the impact of this operation, which means moving back to the child of the old parent node, until the new operation can be applied. After applying the new operation, reapply the undone nodes in sequence order, ensuring that all operations are applied in order. + +The following animated GIF demonstrates the process executed by `Peer1`: + +1. Received `Peer0` creating node `A` with the `root` node as its parent. +2. Received `Peer0` creating node `B` with `A` as its parent. +3. Created node `C` with `A` as its parent and synchronized it with `Peer0`. +4. Moved `C` to have `B` as its parent. +5. Received `Peer0`'s moving `B` to have `C` as its parent. + +
+ ![](./movable-tree/undo-do-redo.gif) +
+ +The queue at the top right of the animation represents the order of local operations and newly received updates. The interpretation of each element in each `Block` is as follows: + +
+ ![](./movable-tree/explain.png) +
+ +A particular part of this process to note is the two operations with `lamport timestamps` of `0:3` and `1:3`. Initially, the `1:3` operation moving `C` to `B` was created and applied locally, followed by receiving `Peer0`'s `0:3` operation moving `B` to `C`. In `lamport timestamp` order, `0:3` is less than `1:3` but greater than `1:2` (with peer as the tiebreaker when counters are equal). To apply the new op, the `1:3` operation is undone first, moving `C` back to its old parent `A`, then `0:3` moving `B` to `C` is applied. After that, `1:3` is redone, attempting to move `C` to `B` again (the old parent remains `A`, omitted in the animation). However, a cycle is detected during this attempt, preventing the operation from taking effect, and the state of the tree remains unchanged. This completes an `undo-do-redo` process. + +### CRDT: Mutable Tree Hierarchy + +Evan Wallace has developed an innovative algorithm that enables each node to track all its historical parent nodes, attaching a counter to each recorded parent. The count value of a new parent node is 1 higher than that of all the node's historical parents, indicating the update sequence of the node's parents. The parent with the highest count is considered the current parent node. + +During synchronization, this parent node information is also synced. If a cycle occurs, a heuristic algorithm reattaches the nodes causing the cycle back to the nearest historical parent node that won't cause a cycle and is connected to the root node, thus updating the parent node record. This process is repeated until all nodes causing cycles are reattached to the tree, achieving all replica synchronization of the tree structure. The demo in [Evan's blog](https://madebyevan.com/algos/crdt-mutable-tree-hierarchy/) clearly illustrates this process. + +As Evan summarized at the end of the article, this algorithm does not require the expensive `undo-do-redo` process. However, each time a remote move is received, the algorithm needs to determine if all nodes are connected to the root node and reattach the nodes causing cycles back to the tree, which can perform poorly when there are too many nodes. + +I established a [benchmark](https://github.com/Leeeon233/movable-tree-crdt) to compare the performance of the movable tree algorithms. + +## Movable Tree CRDTs implementation in Loro + +Loro implements the algorithm proposed by Martin Kleppmann et al., **_[A highly-available move operation for replicated trees](https://martin.kleppmann.com/2021/10/07/crdt-tree-move-operation.html)_**. On one hand, this algorithm has high performance in most real world scenarios. On the other hand, the core `undo-do-redo` process of the algorithm is highly similar to how Eg-walker (Event Graph Walker) applies remote updates in Loro. Introduction about **Eg-walker** can be found in our previous [blog](https://www.loro.dev/blog/loro-richtext#brief-introduction-to-replayable-event-graph). + +Movable tree has been introduced in detail, but there is still another problem of tree structure that has not been solved. For movable tree, in some real use cases, we still need the capability to sort child nodes. This is necessary for outline notes or layer management in graphic design softwares. Users need to adjust node order and sync it to other collaborators or devices. + +We integrated the `Fractional Index` algorithm into Loro and combined it with the movable tree, making the child nodes of the movable tree sortable. + +There are many introductions to `Fractional Index` on the web, You can read more about `Fractional Index` in the [Figma blog](https://www.figma.com/blog/realtime-editing-of-ordered-sequences) or [Evan blog](https://madebyevan.com/algos/crdt-fractional-indexing/). In simple terms, `Fractional Index` assigns a sortable value to each object, and if a new insertion occurs between two objects, the `Fractional Index` of the new object will be between the left and right values. What we want to speak about more here is how to deal with potential conflicts brought by `Fractional Index` in CRDTs systems. + +### Potential Conflicts in Child Node Sorting + +As our applications are in a distributive condition, when multiple peers insert new nodes in the same position, the same `Fractional Index` would be assigned to these differing content but same position nodes. When updates from the remote are applied to local, conflicts arise as the same `Fractional Index` is encountered. + +In Loro, we retain these identical `Fractional Index` and use `PeerID` (unique ID of every Peer) as the tie-breaker for the relative order judgment of the same `Fractional Index`. + +![](./movable-tree/FI-and-PeerID-dark.png) + +Although this solved the sorting problem among the same `Fractional Index` nodes from different peers, it impacted the generation of new `Fractional Index` as we cannot generate a new `Fractional Index` between two same ones. We use two methods to solve this problem: + +1. The first method, as stated in Evan's blog, we could add a certain amount of jitter to each generated `Fractional Index`, (for the ease of explanation, all examples below take decimal fraction as the `Fractional Index`) for example, when generating a new `Fractional Index` between 0 and 1, it should have been 0.5, but through random jitters, it could be `0.52712`, `0.58312`, `0.52834`, etc., thus significantly reducing the chance of same `Fractional Index` appearing. +2. If the situation arises where the same `Fractional Index` is present on both sides, we can handle this problem by resetting these `Fractional Index`. For example, if we need to insert a new node between `0.7@A` and `0.7@B` (which indicates `Fractional Index` @ `PeerID`), instead of generating a new `Fractional Index` between 0.7 and 0.7, we could assign two new `Fractional Index` respectively for the new node and the `0.7@B` node between 0.7 and 1, which could be understood as an extra move operations. + +![](./movable-tree/same-FI-dark.png) + +### Implementation and Encoding Size + +Introducing `Fractional Index` brings the advantage of node sequence. What about encoding size? + +Loro uses [drifting-in-space](https://github.com/drifting-in-space/fractional_index) `Fractional Index` implementation based on `Vec`, which is base 256. In other words, you need to continuously insert 128 values forward or backward from the default value to increase the byte size of the `Fractional Index` by 1. The worst storage overhead case, such as inserting new values alternately each time. For example, the initial sequence is `ab`, insert `c` between `a` and `b`, then insert `d` between `c` and `b`, then `e` between `c` and `d`, like: + +```js no_run +ab // [128] [129, 128] +acb // [128] [129, 127, 128] [129, 128] +acdb // [128] [129, 127, 128] [129, 127, 129, 128] [129, 128] +acedb // [128] [129, 127, 128] [129, 127, 129, 127, 128] [129, 127, 129, 128] [129, 128] +``` +a new operation would cause an additional byte to be needed. But such a situation is very rare. + +Considering that potential conflicts wouldn't appear frequently in most applications, Loro simply extended the implementation, the original implementation produced new `Fractional Index` in `Vec` by only increasing or decreasing 1 in certain index to achieve relative sorting. The simple jitter solution was added, by appending random bytes in length of jitter value to `Fractional Index`. To enable jitter in js, you can use `doc.setFractionalIndexJitter(number)` with a positive value. But this will increase the encoding size slightly, but each `Fractional Index` only adds `jitter` bytes. If you want to generate `Fractional Index` at the same position with 99% probability without conflict, the relationship between `jitter` settings and the maximum number of concurrent edits `n` will be: + + + + + + + + + + + + + + + + + + + + + + +
+ jitter + + max num of concurrent edits +
+ 1 + + 3 +
+ 2 + + 37 +
+ 3 + + 582 +
+ +When there are numerous `Fractional Indexes`, there will be many common prefixes after being sorted, when Loro encodes these `Fractional Indexes`, prefix optimization would be implemented. Each `Fractional Index` only saves the amount of same prefix bits and remaining bytes with the previous one, which further downsizes the overall encoding size. + +### Related work + +Other than using Fractional Index, there are other movable list CRDT that can make sibling nodes of the tree in order. One of these algorithms is Martin Kleppmann's [Moving Elements in List CRDTs](https://martin.kleppmann.com/2020/04/27/papoc-list-move.html), which has been used in Loro's [Movable List](https://www.loro.dev/docs/tutorial/list). + + +In comparison, the implementation of `Fractional Index` solution is simpler, and no stable position representation is provided for child nodes when modeling nodes in a tree, otherwise, the overall tree structure would be too complex. However, the `Fractional Index` has the problem of [interleaving](https://vlcn.io/blog/fractional-indexing#interleaving), but this is acceptable when some only need relative order and do not require strict sequential semantics, such as figma layer items, multi-level bookmarks, etc. + +## Benchmark + +We conducted performance benchmarks on the Movable Tree implementation by Loro, including scenarios of random node movement, switching to historical versions, and performance under extreme conditions with significantly deep tree structures. The results indicate that it is capable of supporting real-time collaboration and enabling seamless historical version checkouts. + +| Task | Time | Setup | +| :---------------------------------------- | :----- | :------------------------------------------------ | +| Move 10000 times randomly | 28 ms | Create 1000 nodes first | +| Switch to different versions 1000 times | 153 ms | Create 1000 nodes and move 1000 times first | +| Switch to different versions 1000 times in a tree with depth of 300 | 701 ms | The new node is a child node of the previous node | + + + Test environment: M2 Max CPU, you can find the bench code + [here](https://github.com/loro-dev/loro/blob/main/crates/loro-internal/benches/tree.rs). + + +## Usage + +```tsx +import { Loro, LoroTree, LoroTreeNode, LoroMap } from "loro-crdt"; + +let doc = new Loro(); +let tree: LoroTree = doc.getTree("tree"); +let root: LoroTreeNode = tree.createNode(); +// By default, append to the end of the parent node's children list +let node = root.createNode(); +// Specify the child's position +let node2 = root.createNode(0); +// Move `node2` to be the last child of `node` +node2.move(node); +// Move `node` to be the first child of `node2` +node.move(node2, 0); +// Move the node to become the root node +node.move(); +// Move the node to be positioned after another node +node.moveAfter(node2); +// Move the node to be positioned before another node +node.moveBefore(node2); +// Retrieve the index of the node within its parent's children +let index = node.index(); +// Get the `Fractional Index` of the node +let fractionalIndex = node.fractionalIndex(); +// Access the associated data map container +let nodeData: LoroMap = node.data; +``` + +### Demo + +We developed a simulated Todo app with data synchronization among multiple peers using Loro, including the use of `Movable Tree` to represent subtask relationships, `Map` to represent various attributes of tasks, and `Text` to represent task titles, etc. In addition to basic creation, moving, modification, and deletion, we also implemented version switching based on Loro. You can drag the scrollbar to switch between all the historical versions that have been operated on. + +