Integer-proxy chunks: backend-agnostic join and reduce over (key_hash, value_id) by frankmcsherry · Pull Request #781 · TimelyDataflow/differential-dataflow

frankmcsherry · 2026-07-03T08:16:33Z

A boundary where only integers cross: a storage backend presents each record as ((key_hash, value_id), time, diff) — integer proxies for data it keeps in its own layout — the operators own all the lattice/time logic over those integers, and the backend supplies value semantics via callbacks. Any columnar (or otherwise opaque-to-DD) value store can then reuse join and reduce without materializing values as Rust types.

The two integers carry different contracts. key_hash is a content hash of the key, stable across the whole system with no registry: the same key hashes identically in every operator, including across the output→input boundary (a reduce output re-ingested downstream). value_id is an intra-key identifier, consistent within one operator computation — equal ids mean equal values there, which is all that consolidation and presence need. Output ids are minted by hashing the produced value, which is what lets a reduce output become a real arrangement whose values present with the same ids downstream. Hash collisions are an accepted risk (birthday bound; the module doc quantifies it, and the upgrade path is a wider id, never a registry).

Contents:

trace/chunk/int_proxy: ProxyChunk, a cursor-less Chunk of proxy columns (slots into the Chunk navigation capability #778 split; batches, fueled merging, and grading come free, entirely in integer space), plus from_unsorted — integer sort+consolidate with representative provenance — as the helper a backend uses to build presentations.
operators/int_proxy: ProxyJoinTactic / ProxyReduceTactic for the join_with_tactic and reduce_with_tactic seams (made pub here — these tactics are the first out-of-crate-shaped consumers), and the backend traits: present-as-proxies (read), value callback with hash-minted output ids (write), materialize (egress). The reduce tactic keeps its cross-retire pending keyed by the stable key_hash, and restricts presentations to changed keys so incremental cost tracks the delta, not the accumulation. The module doc carries the design notes, including why value_id is deliberately not order-preserving (min/max belong to the value callback).
operators/int_proxy/reference: an in-memory backend over VecChunk arrangements with fnv content hashes, so the framework is exercised without any columnar engine.
Tests: an equi-join and count/distinct/min reduces checked against the row-based operators over multi-round inputs with retractions, and a scripted Product-time retire sequence exercising synthetic time corrections both within an interval and pended across retires.

🤖 Generated with Claude Code

…flow#781) consumption plan Branch rebased onto int-proxy: the framework the SPIKE's boundary model called for now exists upstream (renamed ids -> int_proxy in review); the old in-branch copy is dropped. The hand-off section documents the current interfaces, the one-crossing-per-retire reduce_many contract, the assessment harness to replicate, the reduce-first plan, the traps already hit once (scan-presents, capacity leaks, giant-chunk settle), and the suitability questions the corgi agent should answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…C harness Swap the corgi path-dep (../../wip/corgi) for a git-dep pinned to frankmcsherry/WIP@9b41cdc (dd-arrange-api), so the branch builds standalone without a local wip checkout. Add int-proxy-columnar-findings.md (the field report tying TimelyDataflow#781 + this branch + the corgi kernel crate, with the SCC numbers and the columnar-Time gap) and the large-N SCC / vec-profiling harnesses the report's repro section references. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the two superseded modules (corgi_arrange.rs pre-Chunk arrangement, corgi_reduce.rs pre-int_proxy reduce) — nothing in the live path referenced them — plus the ~150-line dead stratum inside corgi_chunk.rs that only they used (from_rows/to_rows/batch_to_rows, rows_to_batch, CorgiChunkBuilder, group_offsets, semijoin_history) and the orphaned imports. Also delete the examples that only exercised the dead modules (corgi_arrange_smoke, corgi_join_mechanism/dataflow, corgi_reduce_dataflow/trace) and the SCC bug-isolation harnesses (corgi_scc_min/trace/stream, corgi_cc_min, corgi_bwd_min) for the interesting-times bug now fixed and pinned by tests + Lean. Live path unchanged: build clean (no warnings), corgi_chunk unit tests pass, corgi_progs matches vec. Net -2157 lines. Git history preserves the journey. Also correct int-proxy-columnar-findings.md: the DD subtree is NOT byte-identical to TimelyDataflow#781 — ~148 lines (MSD-bucket sort, reduce buffer-reuse, tests) sit ahead of the PR and should be pushed back to int-proxy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

G3 (done, stays in TimelyDataflow#781): annotate the proxy reduce tactic's deliberate divergences from `history_replay::compute`, all cited by symbol name so they don't rot. Added the two that were missing — the phase-A/B split at the single `reduce_many` crossing (with its S7 justification), and phase B as the compensating output-accumulation the reference does inline. The others (double-count avoidance, output-times-as-join-base / G1, seeds) were already inline. G2 (scoped out): documenting and asserting the `ReduceTactic`/`JoinTactic` contracts — including the `debug_assert!`s in the shared drivers `reduce_with_tactic`/`join_with_tactic` — hardens the tactic tier used by the cursor and reference tactics today, independent of the proxy. So it becomes its own master-next PR that TimelyDataflow#781 conforms to, not a TimelyDataflow#781 gate. DESIGN.md records the move and the enumerated invariants by enforcement class. Comments/docs only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…, value_id) A boundary where only integers cross: a storage backend presents each record as ((key_hash, value_id), time, diff) — integer proxies for data it keeps in its own layout — the operators own all the lattice/time logic over those integers, and the backend supplies value semantics via callbacks. Any columnar (or otherwise opaque-to-DD) value store can then reuse join and reduce without materializing values. - trace/chunk/int_proxy: ProxyChunk, a cursor-less Chunk of proxy columns, with from_unsorted (integer sort+consolidate with representative provenance) as the presentation-building helper. - operators/int_proxy: ProxyJoinTactic / ProxyReduceTactic for the join_with_tactic and reduce_with_tactic seams (made pub here), and the backend traits: present-as-proxies (read), value callback with hash-minted output ids (write), materialize (egress). Reduce output ids are content hashes, so an output arrangement re-presents with the same ids downstream with no registry; pending interesting times are keyed by the stable key_hash across retires. The module doc carries the boundary contract and design notes (why value_id is not order-preserving; collision risk); each tactic and the in-memory reference backend (VecChunk arrangements, fnv hashes) sit in their own file under the module. - Tests: join and count/distinct/min reduces against the row operators over multi-round retracting inputs, and a scripted Product-time retire sequence exercising synthetic corrections and pending. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…e, not a requirement value_id never outlives one computation (a join unit's presentations; one reduce retire, whose materialize resolves ids to real data before anything leaves). The actual contract is a per-computation bijection with value equality, plus within-retire agreement between the output presentation and minted ids. Content hashing discharges all of it statelessly (the reference backend's choice); exact schemes — dense ordinals from grouping, a per-retire value→id map — are equally valid and collision-free. Only key_hash must be a stable pure function of the key (cross-retire pending, changed-key filter), making the key side the irreducible collision exposure. Persisting ids into an output arrangement itself would force stable value ids; this design does not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…worse than the cursor tactics Two regressions found by asking exactly that question, both fixed: - The reference reduce backend implemented the changed-key restriction by scanning every batch and filtering — O(trace) per retire where the cursor tactic seeks, O(delta·log). Presents now seek the changed keys (novel keys resolve from this retire's delta-sized input batches; pending keys from the retire that pended them, which is exactly what the persistent hash→key map retains — pruned to the changed set each retire, so it is bounded by the delta, and the per-retire value map is cleared). - The join interface had no restriction at all, forcing ANY backend to present the entire accumulated side per fresh batch — O(trace·log) per unit where the cursor join seeks. present0/present1 now take an optional sorted key-hash filter; the tactic presents the fresh side first and passes its key set for the accumulated side, the join analogue of reduce's changed-key restriction. The check is mechanical, not wall-clock: counting backend wrappers measure presented records — with 20k arranged keys and five single-key rounds, reduce presents 37 records and join 10, where the scanning versions present Ω(rounds·N) (join: 100,005) and fail the gate. What remains above the cursor tactics is a log-factor sort of delta-sized presentations, and per-(key, wave) rescans of a changed key's presented range where the row replayer consolidates progressively — same worst-case order, different constants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…cle fuzz The minimal harness for the framework, closed over itself: the tactics demand only BatchReader of the trace, and ProxyChunk is already a Chunk, so a batch of proxy chunks is the minimal arrangement — the proxy data IS the data, no separate chunk class needed. The identity backend makes values u64s with the identity as the id function: no hashing, no resolution machinery, no collision possibility, and materialize emits the proxy records verbatim (incidentally exercising ChunkBatch<ProxyChunk> as a real output batch). What remains under test is exactly the framework's own contribution — interesting-time discovery, desired-vs-current deltas, pending, held routing — fuzzed over Product-time grids: 300 random inputs retired through random diagonal frontiers (so synthetic joins arise inside and across intervals and must pend), driven through an emulation of the reduce driver protocol, and checked against a brute-force oracle at every grid point: the accumulated output must equal the reduction of the accumulated input, everywhere, for count/distinct/min-shaped reducers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

An ignored benchmark (cargo test --release --test int_proxy -- --ignored --nocapture) compares full stacks — the stock row operators against chunk arrangements plus the proxy tactics over the reference backend — for a bulk load and a steady-state incremental phase (warmed past the post-load merge-amortization transient). The bench caught what the counting gates could not: the reference backend's hash→key map was pruned by retain (and the per-retire value map by clear), both of which keep the backing table — so after a million-key load, every retire walked a million-bucket table to visit one entry, ~130µs/round of pure capacity. shrink_to_fit after the prune and a fresh map per retire fix it. Steady-state single-key rounds after the fix, proxy vs row: reduce ~1.4x at every scale (flat from 10k to 1M keys — the delta- proportionality gates hold in wall clock too), join at parity below 1M (~1.0x). Bulk load carries the presentation layer's constant factor (per-record hashing, clones, by-hash sort): reduce 3-5x, join ~2x — the costs a columnar backend's bulk primitives are meant to attack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The longer-term goal for the boundary is fewer crossings into the backend's value logic (interpreted or columnar execution pays per-call overhead): each crossing should carry a list of keys and a longer bracketed list of value entries. ProxyReduceBackend gains reduce_many(keys, ends, input) — group_offsets -shaped brackets, one per key, each non-empty — returning concatenated per-key outputs with their own bracket ends. A default implementation loops the per-key reduce, so simple backends implement only that; backends with bulk value logic override reduce_many. The tactic now calls only reduce_many: retire's key-major loop becomes two passes — derive each changed key's active times, group the work into waves by time, then play the waves in ascending order (Ord extends the partial order, so a key's earlier deltas always precede a later time's reads) with at most one callback per wave, batching every key active at that time. The identity backend overrides reduce_many (asserting the bracket protocol from the backend's seat), so the grid-oracle fuzz exercises the batched path; the reference backend uses the default, so the row-comparison tests cover the loop. All gates and benches unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The desired output at a (key, time) moment is a function of the key's input accumulation at that time alone — no output-side state — so no time ordering constrains the batch: every moment of the retire can share one reduce_many call, a key contributing one bracket per active time. The order-sensitive part (subtracting the current output, which includes deltas emitted at earlier moments) is pure proxy-space arithmetic and moves to a separate pass that plays the moments in ascending time order. The bracket, not the key, is reduce_many's unit: keys may repeat. Contract docs updated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…linear The row suite's reduce_scaling/join_scaling shapes (one key, many distinct times, one batch) exposed quadratic behavior in both proxy tactics: reduce rescanned a key's full presented range per interesting time (and its interesting-time closure joined all pairs), and join cross-produced full matched histories pairwise. Measured: reduce 6.4s at scale 10k, 4.7x per doubling; join 9.8s at 10k, >90s at 20k. The robust versions are the cursor variants with integers in place of keys and values, as intended: - history.rs: IdHistory, the id-space ValueHistory — (value_id, time, diff) edits replayed in ascending time order into a buffer repeatedly advanced by the meet of the times still to come and consolidated. The advancement is the collapse that keeps a key with many distinct times linear: accumulations read the small buffer, never the raw history. The presentations serve as the fused per-key load. - reduce: discover_and_accumulate ports history_replay::compute — lazy interesting-time discovery (novel and pending seed; synthetics from joins with the advanced batch buffer and times_current) replaces the eager join-closure, and per-moment accumulation reads the advanced buffers. Phase B replays the output side per key over the discovered moments with the same machinery (suffix meets; emitted deltas advanced and consolidated). Still one batched reduce_many crossing per retire. - join: join_key ports JoinThinker::think — each side's edits replayed against the other side's advanced buffer (identical emitted times: t0 ∨ (t1 ∨ meet) = t0 ∨ t1), with the dead-simple cross product kept for small histories. proxy_reduce_scaling / proxy_join_scaling (scale 100k, the row tests' shapes) now pin this; scale 10k dropped from seconds to milliseconds and growth is ~4x per 4x scale. The grid-oracle fuzz over partially ordered times, the scripted pending test, the row comparisons, the delta-proportionality gates, and the steady-state bench all pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The row-vs-proxy comparison on the scaling shapes showed proxy reduce still superlinear (9x per 4x scale; timeout at 4M where row takes 0.5s). Profiling pinned it outside the tactic: the reference backend's materialize built ONE giant VecChunk and let the builder settle it, and settle's split path peels TARGET-sized pieces off the front with split_off, copying the remaining tail each iteration — O(m²/TARGET) in the batch size. Feeding the builder TARGET-sized chunks directly fixes it: 4M drops from >120s to 3.2s. With that, both operators are in the row implementations' complexity class on the scaling shapes (growth ~5x per 4x scale ≈ n·log n; the hot frames are the presentation sort and fnv hashing — constants, not structure): at scale 4M, join row 0.96s / proxy 3.6s, reduce row 0.48s / proxy 3.2s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ged view Field report from the corgi porting session (confirmed against Model.lean): reduce's Step-1 seed set was flawed. discover_and_accumulate recovered the batch by time-thresholding the CONSOLIDATED present_input (history ⊎ novel, net-zero records dropped). Legal compaction can advance a stored +1 onto a novel −1's exact (value, time); they cancel in that merged view, so the time drops out of the seeds — but it is in the batch's own support b.support, and a standing output change is still owed there. That is Model.lean's scenario1_cancels (the SCC deadlock) verbatim: int_proxy computed a strictly smaller seedSet than the model's seedSet = b.support ∪ pending. Fix (model-prescribed): seed from the batch's own support. The backend trait gains present_novel(novel) — the freshly arrived batches presented ALONE, before any merge with stored history — replacing key_hashes (which present_novel subsumes: its key set is the changed keys). The tactic seeds discovery and the synthetic-join closure from this run; the merged present_input is replayed only for ACCUMULATION (compaction there is accumulation-preserving, so consolidation is fine). The two are the row tactic's separate batch_cursor vs source-history split, in id space. Regression test proxy_reduce_seeds_survive_compaction_cancellation pins the scenario directly (failed before, passes after). The Product-grid oracle fuzz now also runs a COMPACTION ADVERSARY on alternate iterations (advance accumulated batches to the interval lower + reconsolidate before each retire — the model's acc_mapDomain), so the whole 300-case space is checked under the exact move that triggered the bug. Reported-by: corgi porting session Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The prior fix's present_novel did a full value-present of the delta (hash keys AND values, from_unsorted sort+consolidate) purely to seed interesting times, duplicating the value work present_input already does on novel — a redundancy the corgi porting session flagged. But seeds need far less. Two facts: interesting-time over-derivation is sound (a non-changing seed yields a zero delta and is discarded), so the seed set may be any superset of b.support; and the tactic reads only TIMES from the seed source (batch_replay's value_ids and diffs were never touched). So the batch's raw (key_hash, time) support — a superset of b.support, no value consolidation — is exactly enough. Zeroing value_id in a ProxyChunk would NOT work: from_unsorted then consolidates distinct values at one time into a dropped zero, reintroducing the seed- cancellation bug. The seed run must be raw and value-free. - Trait: present_novel becomes seed_times(&self, novel) -> Vec<(u64, T)>. It hashes keys only (no value hashing, no value_id, no value-sort, no consolidation), returns one (key_hash, time) per record sorted by key_hash, and is &self (stateless — seeds store no alignment). A full value-present of the delta happens once, in present_input. - history.rs: TimeHistory, the times-only replay (ValueHistory with no values/diffs) — same meet-advancement, keeping the *_scaling shapes linear. batch_replay is now a TimeHistory. - This resolves the redundancy without the fragile call-order caching contract: the seed call is intrinsically cheap, no backend caches. Load improves (n=10k reduce 1.67x to 1.22x row; n=1M ~4.4 to ~3.9); steady state and all correctness unchanged: compaction regression, compaction-adversary grid fuzz, scaling shapes, row comparisons, and delta-proportionality gates all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A living burndown in the manner of formal/DESIGN.md: the two extension tiers and the independence-marker key semantics as settled decisions; blocking gates (G1: restore output.support to phase A's join base per Model.lean baseSet + staleness_rederived, with a directed regression and a stronger compaction adversary; G2: tactic contracts on the traits; G3: model<->code linkage notes); mechanical landing items (replay-core unification, consumer-drift fold-back, reference->vec_backend rename, the join-half decision); and ordered follow-ups (columnar/ pivot, ProxyChunk time columns, chunk payload factoring, conformance kit). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The proxy reduce's interesting-time discovery omitted `output.support` from the join base, against the model's `baseSet = stored ∪ output ∪ seeds`. `discover_and_accumulate` now carries the key's output times as a `TimeHistory` (times only — accumulation is phase B's job): it enters the next-time min, folds its meet into the running collapse, and joins its buffer against interesting times. Output times are base, not seeds — they never mark a time interesting on their own. This mirrors the cursor tactic's output-history handling and is what lets pending coverage re-derive from current representatives each round under the fully general compaction adversary. Marks G1 done in DESIGN.md; records that no single-round adversary isolates it (the necessity is the cross-round staleness re-derivation), so it ships fix-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Two mechanical cleanups toward landing. - Rename the `int_proxy::reference` module to `vec_backend`: it is a backend over `VecChunk` (its types are already `Vec*Backend`), and "reference" collided with `operators::reduce`'s model-derived reference *tactic*, a different object. - Hoist `sort_dedup` to one `pub(crate)` copy in `operators::reduce`, shared by the cursor, reference, and proxy tactics, which each carried an identical private copy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…t backends Two related cleanups getting test-grade code off the shipped surface. - Relocate the reference backend (`vec_backend`) to `tests/support/vec_backend.rs`. The seam is tested by being consumed from test code, as an external backend crate would consume it — a blessed in-repo impl only adds a maintained impl on the shipped surface, it does not test out-of-repo implementability (DESIGN.md S5). - Un-fuse `ProxyChunk`. It was two things under one name: a seam *presentation* (the integer projection a backend `present`s, which DD reads positionally) and a trace-storage `Chunk`. The presentation becomes `ProxyBridge`, a plain owned struct in `operators/int_proxy/bridge.rs` — no `Rc`, because no shipped path ever cloned it (the lone clone was a test backend's shortcut), and no `Chunk`, because `value_id`s are ephemeral and arranging them would grant a lifetime they do not have (DESIGN.md S6). The `Chunk` impl — which only a test backend used, to store proxy records as an arrangement — moves verbatim to a test-local `IdentityChunk` in `tests/support/`; `src/trace/chunk/int_proxy.rs` is deleted. `pack`/`is_graded` were already `pub`, so no new public API. The shipped presentation type goes from a 506-line `Chunk` to an 85-line bridge. DESIGN.md: adds S4/S5/S6, closes L4 (join ships with reduce) and L6 (un-fusing), prunes the G1 narrative to its conclusion. Lib and the full `int_proxy` suite (14 tests, incl. the moved merger tests and the grid-oracle fuzz) pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… (L1) `IdHistory` was a near-verbatim second copy of the cursor tactics' `ValueHistory` replay (suffix-meet organize, step / advance-collapse) — the class of duplication where G1's divergence from the model lived. Because `int_proxy` is a child module of `operators`, the private `ValueHistory` is already visible to it, so the proxy can reuse it directly; no master-next precursor is needed, and the additions have their consumer in this same branch (no dead code). - Move the replay algorithm onto `ValueHistory` itself (`time`/`meet`/`edit`/`buffer`/ `step`/`step_while_time_is`/`advance_buffer_by`), and make `HistoryReplay` forward to it. The cursor's call surface is unchanged and behavior-identical. - Add `ValueHistory::load_iter` (cursor-free `(value_id, time, diff)` ingestion, the proxy presentation's shape) and `step_through` (step while `<=` in the total order). - `IdHistory` becomes a thin wrapper over `ValueHistory<u64, T, R>`. Its buffer type `((u64,T),R)` matches exactly, so the `reduce.rs` / `join.rs` call sites are untouched. `history.rs` shrinks 177 -> 135 lines; no new public API (`ValueHistory` stays private, visible only to the child module). `TimeHistory` (time-only, bare `&[T]` buffer) stays a separate small type with the reason recorded on it — a future clean-up, not a blocker. Validated: int_proxy suite (14 tests, incl. the grid-oracle fuzz, now over the shared machinery) and the cursor reduce path (reduce + reduce_reference, 16 tests) both green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…n contract The reduce seam requires an Abelian output diff, and the name should say so. Renamed `ProxyReduceBackend`/`ProxyReduceTactic` -> `AbelianReduceBackend`/`AbelianReduceTactic` (the `int_proxy` module already conveys the integer-proxy seam, so the `Proxy` prefix was redundant; `ProxyJoin*` stays — join carries no such constraint, so the asymmetry is accurate). Why Abelian: the bulk crossing (`reduce_many`) hands the backend only the input and takes back the desired output — never the current output — so the correction `desired - current` is computed in framework code (`negate()`), which needs negation. This is deliberate and likely temporary; the general cursor/reference reduce avoids it by presenting current output to the logic and letting the logic difference. Documents the constraint where it lives: expands the `ROut` doc, and adds DESIGN.md S7, which also records that the batching granularity (one crossing per retire) is a *tactic* choice, not a trait commitment — a future tactic can cross per-wave to cap peak state without a trait change; only the non-Abelian relaxation (present current output) needs one, and that would be a separate additive `ReduceBackend`, not a mutation of this trait. No behavior change; int_proxy suite (14 tests) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

G3 (done, stays in TimelyDataflow#781): annotate the proxy reduce tactic's deliberate divergences from `history_replay::compute`, all cited by symbol name so they don't rot. Added the two that were missing — the phase-A/B split at the single `reduce_many` crossing (with its S7 justification), and phase B as the compensating output-accumulation the reference does inline. The others (double-count avoidance, output-times-as-join-base / G1, seeds) were already inline. G2 (scoped out): documenting and asserting the `ReduceTactic`/`JoinTactic` contracts — including the `debug_assert!`s in the shared drivers `reduce_with_tactic`/`join_with_tactic` — hardens the tactic tier used by the cursor and reference tactics today, independent of the proxy. So it becomes its own master-next PR that TimelyDataflow#781 conforms to, not a TimelyDataflow#781 gate. DESIGN.md records the move and the enumerated invariants by enforcement class. Comments/docs only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`key_hash` is a content hash, so it is an independence marker, not identity — colliding keys share a cell. The framework stays collision-oblivious (sound, just coarse), but exactness is recoverable *backend-side*, because the backend always holds the real keys and values. Add that recipe to `mod.rs`'s collision-risk docs: - Join: verify real keys at `cross` and drop collision-fabricated pairs (their times/diffs are never emitted, so the output is exactly the real-key equijoin). - Reduce: partition the bracket by real key and mint key-qualified value ids (`hash(key, value)` or per-`(key, value)` ordinals) so `(key_hash, vid)` tracks `(key, value)` exactly. The unrecoverable residue is granularity only (colliding keys share `pending` and the changed-key restriction) — extra work, never wrong answers. Also document, on `vec_backend` itself, that the reference deliberately does NOT implement this — it hashes keys/values and accepts the birthday bound for simplicity. Also folds in pending DESIGN.md updates: G2 recorded as done/merged (TimelyDataflow#789) with TimelyDataflow#781 rebased and conforming; the S1-recipe task marked done. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Vetted the corgi-chunk drift on its own merit (it is pre-G1 and pre-refactor, so nothing merged — manual ports). - sort_perm (bridge.rs): `ProxyBridge::from_unsorted` now sorts via an MSD counting sort on the top byte of `key_hash` (a content hash, so uniform) into 256 ascending buckets, each finished by the full comparison. Output-identical to the previous full sort (reps stay valid — any member of a consolidated group serves); ~256x less compare work on large runs, and a single-bucket fallback that is never worse. - Phase-B delta (reduce.rs): replaced the per-moment `BTreeMap<u64, ROut>` (`cur`/`delta`) with a reused `Vec<(u64, ROut)>` + `consolidate`. corgi did this with an identity-hashed `IdMap`; `consolidate` gets the same no-per-moment-alloc win but matches the cursor's `update_buffer` and the sibling `emitted` accumulation — a keyed map is a proxy-ism with no cursor analog. This also moots L5a (there is no `IdHasher`/`IdMap` to export). - KeyView (reduce.rs, L5): `discover_and_accumulate`'s eleven arguments collapse to seven — the per-key input view (`p_in`, `i0`, `i1`, `rep`, `pending`) bundles into one `KeyView`. Skipped: the extended-tests claim was stale (corgi's tests are identical, and it never touched reduce_reference); the phase-A per-key scratch hoisting is a minor alloc nicety. int_proxy (14, oracle included), reduce (16), reduce_reference, join all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Restore the `[x]` checkbox on the G2 gate entry (lost when it was rewritten to "done as TimelyDataflow#789"), and fix "a AbelianReduceBackend" -> "an" in F1 (an article the ProxyReduce* -> AbelianReduce* rename left ungrammatical). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The columnar (struct-of-arrays) layout was inherited from ProxyBridge's former life as a `Chunk` (bulk merge/compaction wants SoA) and the un-fusing left it unexamined. As a transient *presentation* it earns nothing: DD reads the bridge one record at a time and consolidates it once, for which rows are simpler and cheaper — `from_unsorted` gathers into one output stream instead of four, and each record is one cache line instead of three. The only case SoA would serve, a vectorized backend processing whole columns (F1), doesn't apply: the in-tree columnar backend is trie-based and doesn't match this layout anyway. - ProxyBridge now stores `Vec<(key_hash, value_id, time, diff)>`. Column accessors become index accessors (`key_hash(i)`/`value_id(i)`/`time(i)`/`diff(i)`) plus `records()` for iteration; `into_parts` unzips on demand for the one consumer that stores columns. - `from_unsorted` still *takes* aligned input columns (the shape backends build) — only its output and storage changed — so the backends are untouched apart from the read accessors. - join's merge-join and rep-building move to the index accessors; a `rep_of` free fn replaces the slice-taking closure. Recorded as DESIGN.md S8. No behavior change; int_proxy (14, oracle incl.), reduce (11), reduce_reference, join all green, no warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`ProxyBridge<T,R>` becomes a type alias for `Vec<((key_hash, value_id), time, diff)>` — DD's standard `(data, time, diff)` update shape — so `consolidate_updates` sorts and consolidates it directly. The struct, index accessors, `from_unsorted`, and `sort_perm` were apparatus inherited from ProxyBridge's former life as a columnar `Chunk`, redundant with `consolidate_updates` for a transient presentation. `bridge.rs`: 117 -> 22 lines. - The reduce tactic consolidates its output buckets in place (they were already the bridge shape); the seam's read sites use tuple fields. - The representative-index `reps` alignment is now a *backend* concern, not the seam's: `vec_backend` keeps a private `sort_consolidate` (it wants exact index->real resolution), and the identity backend resolves through value_id and needs none of it — the existence proof that the seam works without the apparatus. - Being a plain `Vec`, the bridge structurally cannot be arranged/persisted, so S6's ephemeral-id guarantee now holds by type. Recorded as DESIGN.md S8. No behavior change; int_proxy (14, oracle incl.), reduce (11), reduce_reference, join all green, no warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A whole file for a one-line type alias is too much. Move `pub type ProxyBridge<T,R> = Vec<((u64,u64),T,R)>` into `mod.rs` and delete `bridge.rs`, and cut the doc-comment down to a few lines (the full reasoning lives in DESIGN.md S6/S8). Docs/placement only; int_proxy suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The L1 unification left `IdHistory` a newtype over `ValueHistory<u64, T, R>` whose ten methods all forwarded 1:1 — no logic of its own, the same critique we applied to `ProxyBridge`. Replace it with a plain type alias; call sites use `ValueHistory`'s methods directly (only `load` → `load_iter`). Visibility is `pub(in crate::operators)` to match the private `ValueHistory` it names (a `pub(crate)` alias would trip `private_interfaces`; the old newtype dodged this via its private field). `TimeHistory` stays a separate type. history.rs 135 → 115; no public API change; 14 int_proxy tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A ProxyBridge sed pass had introduced `rep_of` between `join_key`'s doc comment and its signature, so the "Match one key's records…" doc and the `#[allow(clippy::too_many_arguments)]` ended up attached to `rep_of` (2 args, needs neither), while `join_key` (8 args) was left with no doc and no allow — clippy would warn on it. It compiled only because `///` desugars to stackable `#[doc]` attributes and plain build skips clippy. Move each doc back to its function; no code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The join and reduce tactics handed the value callbacks (`cross`, `reduce`) positions into the presentation, then made every backend keep a positional alignment to resolve them. But the harness never indexes by the `value_id`s it consolidates on — it only groups and returns them — so the positions were a second addressing scheme, redundant with the ids and asymmetric with the already id-addressed output side. Hand the ids back unchanged instead. `cross` takes `&[u64]`, `reduce`/ `reduce_many` take `&[(u64, RIn)]`; the rep-list translation (`rep_of`/`find`, the per-key `rep`) is gone from both tactics — each emits the `value_id` it already grouped on. Backends resolve the ids however they minted them. The reference `vec_backend` now mints `value_id = hash((key, value))` (so an id identifies its record, and keys colliding on `key_hash` stay distinguished — per the value-id contract) and resolves through `value_id → (key, value)` maps, retaining no positional alignment. The identity backend drops `current_vals` entirely (ids are values). Docs: mod.rs one-line nits; DESIGN.md S6/S8 updated off the positional model, decision-log entry added. Net -23 lines of tactic/backend code; full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ith backends The value callbacks and presents received only fragments of the retire/unit context, so a backend could neither resolve value_ids against the source batches without copying data out at presentation, nor advance loaded times by the compaction frontier to collapse accumulated history. Both were artifacts of withholding context the tactic already holds. Introduce `ReduceInstance { source/input/output batches, lower }` and `JoinInstance { batches0, batches1, lower }`, built by the tactic and passed to `seed_times`/`present_*`/`reduce`/`reduce_many` (reduce) and `present0`/`present1`/`cross` (join). `materialize` is exempt (its values are produced, not drawn from a batch). The reference `vec_backend` advances `present_*` by `instance.lower` and consolidates, collapsing the historical tail; the join frontier is the unit's capability time, sound because every output is produced under it (`output ∨ cap = output`). The identity backend ignores `lower` (correct, less compact). Step 1 is the design/wiring: correct (14 tests green — reduce/join oracles, grid-oracle fuzz, delta-proportionality) and free (a no-op when there is no history to compact). No measurable bench change — the frontier's payoff is deep-history re-reads (iterative scopes / long-history steady state), which the standing bench doesn't stress, and a cold load carries no history. The structs also unlock step 2 (resolve value_ids by offset into the batches, dropping the copy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A value_id is intra-hash — it identifies a record only within a key_hash — so `cross`'s bare `left: &[u64]` was under-specified: it worked only because the reference mints globally-unique, key-qualified ids. Report the full bridge key instead — `cross(left: &[(u64, u64)], right: &[(u64, u64)])` — matching the bridge record shape `((key_hash, value_id), time, diff)`, and letting value ids be genuinely intra-hash (which unblocks cheaper per-hash id schemes later). `join_key` takes the matched `key_hash` (shared by every record it handles) and pushes `(kh, vid)` pairs. The reference resolves via `(key_hash, value_id)` maps: `Rows::present` keys its real-record map by the full bridge key, and the reduce backend's `in_vals`/`out_vals` follow suit (reduce resolves with its `key_hash` argument, materialize with the record's). No behavior change — ids stay `hash((key, value))` for now; 14 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Record what landed but wasn't yet in the burndown: - S6/S8: records are addressed across the seam by their full (key_hash, value_id) bridge key; a value_id is intra-hash, so the pair is the identity. - S9 (new): the seam shares the retire/unit context with the backend via ReduceInstance/JoinInstance (batches + lower frontier), passed to the read/value methods; materialize exempt. Frontier advancement is a backend option; seed_times must not apply it. - F6 (new): resolve value ids by offset into the Instance batches, dropping the present-time (key,value) copy. - F7 (new): hash-native storage (HashChunk) is the real perf path — the ~4x reduce load is the Ord-vs-hash order mismatch (re-sort on present), not the ~13% hashing; VecChunk is an anti-example; do F3 first. - Decision log: entries for the Instance sharing and the (kh, vid) change. Docs only; no code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The correctness gates (G1–G3) and landing items (L1–L6) are all closed, and their conclusions have graduated into the settled semantics, the module/trait docs, and the code; the decision log is a changelog git already carries. Drop those three sections, leaving §1 (what this is), §2 (settled semantics S1–S9), and the follow-ups (renumbered §3). Retitle, and fix the now-dangling G1/G2/L1 references. ~330 → 200 lines. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Doc/tracking fixes from review; no behavior change, 14 tests green. - seed_times: state the sorted-by-key_hash requirement in the trait doc (it was only enforced by the tactic's debug_assert!, invisible to an out-of-crate impl). - mod.rs typo: "difference hashes" -> "different hashes". - DESIGN.md F8 (new): join fuel/yielding — ProxyJoinTactic::work drains eagerly (fuel=0), the one known gap that lived nowhere in the burndown. A refinement. - DESIGN.md F4: the kit should add a join analog of the reduce's compaction- cancellation regression (advancing a fresh presentation to empty) — sound but untested, the sound-looking-cancellation shape that produced the TimelyDataflow#781 drift. - DESIGN.md F5: note join_key is likewise a second interesting-times copy (of the cursor JoinThinker); the fuzz/model net covers only reduce today. - DESIGN.md F7: record that system-wide key_hash stability is load-bearing only under hash-native shared arrangements (a deliberate commitment, not a current requirement), so S1's recovery-cost story weighs against hash-native storage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A corgi re-port atop TimelyDataflow#781 hit a real gap: the present_* methods' bridge output MUST be sorted and consolidated by ((key_hash, value_id), time) — the tactics merge-walk it — but the docs said only "may consolidate" (optional-sounding) and, unlike seed_times, nothing asserted it. vec_backend sorts internally so never hit it; a second, out-of-repo backend exposed it (exactly what S5 exists to catch). - mod.rs: debug_assert_sorted_bridge (mirrors the seed_times assert); the ProxyBridge alias doc now states the sorted+consolidated requirement. - reduce/join: call it after each present_input/present_output/present0/present1; the trait docs now say the sort/consolidate is mandatory, advance-by-lower is the only optional part. - DESIGN.md S5: record the validation. Docs + debug assert only; no behavior change. 14 tests green (asserts active). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Both tactics materialize the whole retire/unit before processing (present_* returns a Vec over all keys), so peak memory is O(keyspace) — breaks at ~100M keys, where the cursor operators stay bounded by streaming per key. Record the scope: Level 1 (windowed, today's Vec seam — reduce loops seed_times then (present*/reduce/materialize) per key-window, join windows a key-subset per work call, no signature changes, reduce self-contained, join fresh-side load needs F7) and Level 2 (streaming present, F7-gated). Mandatory before long; not this PR. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Per design call: the windowed version is the correct one — diverting bounded chunks of value work to the backend (a bulk crossing per window) is the seam's whole point; streaming through a cursor would reintroduce the per-record overhead the bulk seam removes. Drop the streaming "Level 2", and record the mechanism: elicit key hashes with per-key sizes (a cheap metadata pass), bracket into size-bounded windows, then bulk present/reduce(/cross)/materialize per window. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…low#791 pub); port ProxyJoinTactic to `prep` master-next reshaped `JoinTactic` from `defer`+fuel-metered `work` to a single `prep(input0, input1, fresh, meet) -> Box<dyn Iterator<Item = C>>` (TimelyDataflow#790), with the driver now owning the queues, capabilities, and fuel; and flipped the tactics to `pub` (TimelyDataflow#791). `operators/join.rs` resolves to master-next's version (our only change there was the same pub flip); `operators/reduce.rs` auto-merged. Port `ProxyJoinTactic` to the new trait: - The tactic is now pure data-to-data: `prep` maps two batch lists to output container(s), holding no queues/capabilities/fuel. Dropped `JoinUnit`, the two queues, and `defer`/`work`; `join_unit` becomes `join_prep` returning `Option<Bk::Output>`. `meet` (the driver-supplied capability time) feeds `JoinInstance.lower` directly. - This is the eager port: `prep` returns the whole unit's output as one container (`iter::once`). Correct and behaviour-preserving; the F8 fuel footgun is gone structurally (the driver meters fuel now). Making `prep` lazy/windowed is F9's join half — see the updated F8/F9. - Tests: `join_with_tactic` turbofish now takes the container `C`, not the builder. Full workspace green (incl. SCC/BFS and the int_proxy suite, 14 tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DAlperin mentioned this pull request Jul 3, 2026

UnloadChunk: bulk probe extraction for chunk batches #782

Open

frankmcsherry marked this pull request as draft July 5, 2026 22:26

frankmcsherry mentioned this pull request Jul 6, 2026

operators: document and assert the ReduceTactic / JoinTactic contracts #789

Merged

frankmcsherry and others added 18 commits July 5, 2026 22:05

frankmcsherry force-pushed the int-proxy branch from fd9ae6b to 9f6d7a0 Compare July 6, 2026 02:06

frankmcsherry and others added 4 commits July 5, 2026 22:22

frankmcsherry and others added 15 commits July 6, 2026 08:55

Improve presentation

e3da8eb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Integer-proxy chunks: backend-agnostic join and reduce over (key_hash, value_id)#781

Integer-proxy chunks: backend-agnostic join and reduce over (key_hash, value_id)#781
frankmcsherry wants to merge 37 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:int-proxy

frankmcsherry commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

frankmcsherry commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant