diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index b7e34ea08..aa83165f3 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,43 @@ +## 2026-09-18 — PR #1246 merged (`568965e9`): `EdgeBlock` is a `FacetCascade` on `main`, `Pred::Range` is in the mask-risc IR, and no manifest in this repo carries a `.0.0` pin + +The two entries below dated 2026-09-17 (3) and (2) describe what is now on +`main`, not what is in PR: the `EdgeBlock` contract-inventory delta and +`D-MRX-7`. Read them as shipped. + +- **New in the pin doctrine (operator, 2026-09-18):** *"never pin to x.00, + always float x.*"* / *"so no decimal .0.0"*. Eight exact-equals pins are + gone, one per DECLARATION: `lance` / `lance-linalg` / `lance-index` now + `11.*` and `lancedb` `0.38.*` in the workspace table (4), `lance` in + holograph (5), `lance-namespace` in `lance-graph-catalog` (6), and + `lance-namespace` + `lance-arrow` in `lance-graph` (7, 8). + `lance-namespace` is floated TWICE — it is declared in two manifests — + which is why an earlier wording that named it once read as seven. Resolution is UNCHANGED — arrow + 58.4.0, datafusion 54.1.0, lancedb 0.38.0, lance family 11.0.0 — verified + byte-identical on throwaway lockfiles before and after, then compiled and + tested end-to-end by CI on the merge parent. `arrow` and `datafusion` were + already caret. **CLAUDE.md § Key Dependencies is updated in the same + arc**: its `=11.0.0` / `=0.38.0` lines were stale the moment the manifest + changed, and the superseded 2026-09-05 half-sentence is struck in place + rather than deleted. +- **New contract surface:** `FacetCascade::as_bytes_mut`, plus a + `target_endian = "little"` compile-time assert on `FacetCascade`. The + assert exists because `as_bytes()` is a raw reinterpret while `to_bytes()` + is an explicit LE encode of `facet_classid` — identical on LE, divergent + on BE, and the struct's in-memory image IS the canonical stored row image. + Failing at compile time beats corrupting a row at runtime. The shape that + needs no guard — `edges` byte-backed the way `NodeGuid` is, with the lanes + projected — is named and NOT built. +- **State consumers should know:** nothing about byte positions moved. + `NODE_ROW_STRIDE`, `node_rows_from_le_bytes` and + `ENVELOPE_LAYOUT_VERSION` are all unchanged; `EdgeBlock::default()` and + equality still compile. +- **Gated ranges cost two passes:** `OpHistogram` gained `ranges`, and a + gated `Pred::Range` charges `two_input` as well, because `exec.rs` runs + `mask_set_range` and then `mask_and_assign`. Every lane predicate has a + fused `*_to_mask_under` kernel and stays free under a gate; Range is the + sole exception until `mask_set_range_under` exists upstream. Arc: + `PR_ARC_INVENTORY.md` 2026-09-18 #1246. + ## 2026-09-17 (3) — CONTRACT INVENTORY DELTA: `EdgeBlock` is now `pub type EdgeBlock = FacetCascade` (the V1 `12 + 4` struct is gone); `FacetCascade::as_bytes_mut` added - **State consumers should know:** bytes 16..32 of `NodeRow` are the same diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index f33b0580e..85ae62647 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -1,3 +1,73 @@ +## 2026-09-18 — lance-graph PR #1246 (merged `568965e9`, branch `claude/great-pascal-k96kok`) — `EdgeBlock` becomes `FacetCascade`, `Pred::Range` joins the mask-risc IR, and every `.0.0` exact pin floats + +- **Added:** `Pred::Range { lo: u32, hi: u32 }` + `ExecError::RangeOutOfBounds` + in `lance-graph-mask-risc` (D-MRX-7); `OpHistogram::ranges`; + `FacetCascade::as_bytes_mut`; a `target_endian = "little"` compile-time + assert on `FacetCascade` with a round-trip test proving `as_bytes()` and + `to_bytes()` agree on a non-zero `facet_classid`. +- **Retired:** the V1 `EdgeBlock { in_family: [u8; 12], out_family: [u8; 4] }` + struct. `EdgeBlock` is now `pub type EdgeBlock = FacetCascade` — bytes + 16..32 are the same content-blind `4 + 12` facet as the key, and the 12+4 + split no longer exists as a type. Byte positions, `NODE_ROW_STRIDE` and + `node_rows_from_le_bytes` are unchanged; no `ENVELOPE_LAYOUT_VERSION` bump. + Operator ruling `E-THE-SECOND-FACET-IS-NOT-AN-EDGE-BLOCK-1`: *"It's + forbidden for the edge block to even know it's an edge block."* +- **Pin doctrine changed (operator, 2026-09-18):** *"never pin to x.00, + always float x.*"* / *"so no decimal .0.0"*. Eight exact-equals pins + floated, one per DECLARATION: `lance` / `lance-linalg` / `lance-index` / + `lancedb` in the workspace table (4), `lance` in holograph (5), + `lance-namespace` in `lance-graph-catalog` (6), and `lance-namespace` + + `lance-arrow` in `lance-graph` (7, 8). `lance-namespace` is floated TWICE + because it is declared in two manifests — the earlier wording named it + once and so read as seven, which is what a reviewer counted. Verified a resolution NO-OP against throwaway + lockfiles before and after: arrow 58.4.0, datafusion 54.1.0, lancedb + 0.38.0, the whole lance family 11.0.0, byte-identical. `arrow` and + `datafusion` were already caret and were not touched. CLAUDE.md's pin + block carried `=11.0.0` / `=0.38.0` until this arc's hygiene entry; the + superseded half-sentence is struck in place there, not deleted. +- **Locked:** a gated `Pred::Range` costs TWO mask passes, not one. Every + LANE predicate has a fused `*_to_mask_under` kernel, so gating one is + free; Range is the sole exception — `exec.rs`'s `run_pred`, in its + `(Pred::Range { .. }, Some(u))` arm, runs `mask_set_range` and THEN + `mask_and_assign`. Charged to `two_input` (the second pass + literally IS an `and`), which leaves the asymmetry visible in the + histogram instead of hidden behind a range-specific name. The fused + primitive that would close it is named in the doc comment as + `mask_set_range_under`; it does not exist upstream. +- **Deferred / named, not built:** `mask_set_range_under`; the planner-side + ordering witness quack's `prefix_u64` lowering waits on (the IR does not + know a lane is address-ordered, and `Filter` carries no ordering + evidence); the readers that still split at 12 on their own authority + (`ISS-EDGE-BLOCK-WAS-A-SECOND-TYPE-FOR-THE-SAME-FACET`); making `edges` + byte-backed the way `NodeGuid` is, which would retire the new + `target_endian` guard entirely. +- **Withdrawn before landing:** `EdgeCodecFlavor::Refs16`, `EdgeRefs`, the + `T8` facet shape, `ColumnDescriptor::class_id`, and a `le-contract.md` §3c + ruling. None reached `main` or the branch. +- **Review:** codex P2 on the endian exposure — real, and the fix is the + compile-time assert plus two stale `canonical_node.rs` SAFETY comments + that still justified themselves with "EdgeBlock is plain `[u8; _]`". + CodeRabbit raised three; one (the gated-range undercount) was correct and + is the Locked item above, and CodeRabbit confirmed the fix. Two did NOT + survive checking and were declined with reasons: `mask_set_range` costs + ONE mask pass, not three (every word of `out_words` is written exactly + once across disjoint segments), and `STATUS_BOARD.md` is ascending by + D-id, so the prepend rule — which names the append-only ledgers — does not + reach it. CodeRabbit withdrew the second and recorded a learning. +- **CI note:** `member-tests` was CANCELLED on both attempts of run + 35330298662 with all 22 work steps green, dying in the + `Post Run Swatinem/rust-cache@v2` teardown. No test failed on either + attempt; `test`, `test-with-coverage`, Build and Style Check were green. + The one licensed re-run was spent on attempt 2; the operator merged during + attempt 2's teardown. +- **Confidence:** HIGH that the type change is byte-neutral (positions, + stride and the `from_le_bytes` path are untouched, and every in-tree field + site moved mechanically to the same bytes). HIGH that the pin change is a + resolution no-op (two throwaway lockfiles, byte-identical, plus a green + `Build` and a green `test` on `main`'s merge parent). MEDIUM on the + endian guard being the RIGHT shape rather than a correct stopgap — the + byte-backed `edges` above is the shape that needs no guard. + ## 2026-09-17 — lance-graph PR #1245 (merged `83369cad`, branch `claude/great-pascal-k96kok`) — the facet's per-axis LCP goes BACK to the `shared6` byte chain, verbatim; #1241's masked readout is the test oracle now; three prefix-fold carriers named - **Added:** `examples/facet_axis_lcp_probe.rs` — four-arm benchmark diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 5bbded313..586e6e1a8 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -58,7 +58,7 @@ evaluates one; `execute` stays the consumer's call on a scratch it owns. | D-MRX-4 | `ternlog_dispatch.rs` — generated 256-arm match, regenerate-and-diff gated | Shipped (#1226 `0b1ebaa`; PR3; CI `--check` line wired) | all 256 immediates equal the bit-serial reference | | D-MRX-5 | the differential suite at 8 row counts × every op × both CI arms | Shipped (#1226 `0b1ebaa`; PR3; `x86-64-v3` in CI, `-v4` locally on 2026-09-14 — NEON/WASM/scalar unexercised) | anti-vacuity `survivors * 3 < n_rows` for the 8 non-`Ne` predicates, UNGATED, at `n_rows >= 3` (the two `Ne` shapes are selective by complement, and `n_rows ∈ {0, 1}` are execution-only — the bound is `0 < 0` there); the gate itself is pinned selective (`gated < ungated`) at `n_rows >= 64` | | D-MRX-6 | `examples/count_probe.rs` — reference / interpreted / fused, 0 B per execute | Shipped (#1226 `0b1ebaa`; PR3; four arms agree on the COUNT — 4855 at 64k rows; `interpreted` and `fused` gated at 0 B; timings printed from one run, not pinned) | counting allocator reads 0 after warm-up; three arms agree | -| D-MRX-7 | `Pred::Range { lo, hi }` — the contiguous-range write (`ndarray::simd::mask_set_range`) as an IR op; `ExecError::RangeOutOfBounds`; gate-aware | **In PR** (branch `claude/great-pascal-k96kok`, `c3931a1a`). Closes the IR half of `ISS-MASK-RISC-HAD-NO-RANGE-OP`; quack's `prefix_u64` lowering still waits on lane-ordering evidence (planner side) | three differential tests at the 8 row counts: word-edge shapes incl. 63/64/65 ungated / under plane / under scratch (values AND scratch words); `== hi − lo` through the EXECUTOR (the oracle-counted first draft was vacuous — disable-run passed it); identical `RangeOutOfBounds` on both paths. Disable-run: off-by-one in the exec arm fails 2 of 3 | +| D-MRX-7 | `Pred::Range { lo, hi }` — the contiguous-range write (`ndarray::simd::mask_set_range`) as an IR op; `ExecError::RangeOutOfBounds`; gate-aware | Shipped (#1246 `568965e9`). Closes the IR half of `ISS-MASK-RISC-HAD-NO-RANGE-OP`; quack's `prefix_u64` lowering still waits on lane-ordering evidence (planner side) | three differential tests at the 8 row counts: word-edge shapes incl. 63/64/65 ungated / under plane / under scratch (values AND scratch words); `== hi − lo` through the EXECUTOR (the oracle-counted first draft was vacuous — disable-run passed it); identical `RangeOutOfBounds` on both paths. Disable-run: off-by-one in the exec arm fails 2 of 3. Cost accounting corrected post-review: a GATED range charges `ranges` AND `two_input` (`mask_set_range` then `mask_and_assign`), asserted two-sided against a gated lane predicate, which stays free | ## bbb-nars-lowering (D-id minted 2026-09-07 with the operator ruling) diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index 975b4f02f..ee7b9afeb 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -1,3 +1,75 @@ +## TD-JC-CLIPPY-RED-ON-BASE-2 (2026-09-18) — the 1.98 pre-bump lint sweep was WORKSPACE-scoped, and `jc` is workspace-EXCLUDED + +**`JC Substrate Proof` is RED on `main`** (run `35335429357`, head `568965e9`): +`clippy::chunks_exact_to_as_chunks` in `crates/jc/src/jirak.rs` and +`crates/jc/src/pearl.rs`, in each file's `deterministic_fingerprint`, both +spelled `chunks_exact_mut`, under the workflow's own `-D warnings`. Cited by +FUNCTION, not by line: the first draft of this entry wrote `jirak.rs:37` / +`pearl.rs:59` and the citation-decay gate failed it (one new decay, correctly +attributed to this entry) because the nearest backticked symbol to those +coordinates was the LINT NAME, which is nowhere near the code. The gate's +point, taken: a number is a coordinate in a moving frame. + +**It is NOT the merge that surfaced it.** Both lines are present verbatim at +`83369cad` (`main` before #1246), and #1246 touched no `jc` file. Verified by +checking out the base and grepping the two paths, not inferred from the diff. + +**Why the existing guards did not catch it, precisely:** + +- `jc-proof.yml` DOES run on pull_request — but path-filtered to + `crates/jc/**`, `crates/lance-graph-contract/src/cam.rs` and the workflow + itself. #1246 changed `facet.rs` / `canonical_node.rs`, so it correctly + never ran there. The filter is not the defect. +- `git log -S` dates both sites to `5959463f` (#1170, 2026-09-04) — ONE DAY + BEFORE `TD-JC-CLIPPY-RED-ON-BASE-1` was closed as "jc clippy clean" + (`13fdd587`). They survived that sweep because the lint did not exist yet: + `chunks_exact_to_as_chunks` is new in 1.98. +- `rust-toolchain.toml`'s own bump log says the 1.97.1 → 1.98.1 leg is + "ONE lint … at ten sites across four crates", closed separately in #1194 + and "verified clean on BOTH 1.97.1 and 1.98.1" — which is what licensed + the bump as a one-line channel edit. **That verification was + workspace-scoped, and `jc` is in `[workspace] exclude`.** A + workspace-wide `cargo clippy` cannot reach it — which is exactly the + reason `jc-proof.yml` carries its own `--manifest-path` clippy step in + the first place, as its comment says. The sweep and the gate knew about + the exclusion independently; the bump's measurement did not. + +**The fix (2 lines, mechanical):** `fp.chunks_exact_mut(8)` → +`fp.as_chunks_mut::<8>().0`, which is what clippy suggests and what #1194 +applied at its ten sites. NOT done here: it is code in a crate this PR does +not touch, and this PR is board hygiene. + +**Wider candidate surface — GREP, NOT VERDICT.** A `chunks_exact(_mut)\(\d+\)` +census over all **32** workspace-excluded members finds **118** candidate +sites: thinking-engine 98 (mostly `examples/`), highheelbgz 4, +lance-graph-cognitive 4, onebrc-probe 3, reader-lm 3, bge-m3 2, jc 2, +quasicryth-research 1, weather-poc 1. + +⊘ **The first version of this entry said 22 crates / 114 sites and missed +`onebrc-probe` and `weather-poc` entirely** (codex P2 on #1247, verified and +correct). Cause, which is the entry's own subject one level up: the census +parsed `[workspace].exclude` with a NON-GREEDY REGEX that stopped at the +first `]` and silently returned 22 of 32 members. An entry about a +workspace-scoped measurement missing workspace-excluded crates was itself +produced by a scope that silently truncated. Recounted with `tomllib` — +`len(exclude) == 32`, 118 sites across 9 crates — which reproduces codex's +numbers exactly. **Use a TOML parser for a TOML list; a regex over TOML is +the same class of error as a grep standing in for a read.** + +**Only `jc` is RED**, because only `jc` has a clippy +gate; the rest are ungated, so none of the other 116 has been compiled +against the lint at all. `weather-poc` has its own workflow, but it runs +`cargo test` only — no clippy step — so its one site is ungated like the +rest. Whether they would fire is UNVERIFIED — the regex +matches a call shape, not a lint verdict, and `--all-targets` is what pulls +`examples/` in. Do not quote 118 as a defect count. + +**Same shape as:** `TD-JC-CLIPPY-RED-ON-BASE-1` (resolved #1183) and +`TD-SIGKER-CLIPPY-RED-ON-BASE-1`. The recurring lesson is one line: **a +workspace-scoped measurement says nothing about a workspace-excluded +crate**, and this repo excludes 32 of them. A toolchain bump's "verified +clean" is only as wide as its manifest list. + ## 2026-09-15 — `TierFloors::stack_early_exit` promises a settled reading; that is true only for non-negative intensity, and the doc does not say so - **The premise, unstated.** `crates/perturbation-sim/src/rolling_floor.rs:220` diff --git a/CLAUDE.md b/CLAUDE.md index 22d768de3..fa95e10a7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,8 +25,16 @@ workspace is local — prefer the local/fork source over the registry, always. > finds them must not read the P0 above as a mandate to wire them in. Registry > pins for this family are therefore CORRECT, not a policy violation to > "fix". The version discipline that DOES apply: the family moves in exact -> lockstep (`=X.Y.Z` across every member, `lancedb` pinning its matching -> `lance`), so a bump is one deliberate, measured PR — never a drift. +> lockstep, so a bump is one deliberate, measured PR — never a drift. +> **⊘ The parenthetical that stood here — "(`=X.Y.Z` across every member, +> `lancedb` pinning its matching `lance`)" — is SUPERSEDED by the operator +> ruling 2026-09-18** (*"never pin to x.00, always float x.*"*): the family +> still moves in lockstep, but WE express that as a floating patch (`11.*`, +> `0.38.*`), never a `.0.0` exact pin. Lockstep is about the MAJOR moving +> together; it never required us to restate lancedb's own `=11.0.0`, and +> restating it can only narrow the graph. Full rule and the receipt: +> § Key Dependencies below — which is the ONE authoritative pin policy in +> this file. - If a fork's coordinates (git URL, branch/tag, feature flag) are unknown, **STOP and ask**. Do NOT fall back to crates.io as a convenience or to make a build pass. @@ -1208,8 +1216,16 @@ cd crates/lance-graph-python && maturin develop ## Key Dependencies ```toml -# THE PIN RULE (operator-ruled 2026-09-05). Exact-pin (`=X.Y.Z`) ONLY where an -# upstream crate itself demands exact-equals. Everywhere else the MAJOR, so the +# THE PIN RULE. ⊘ The 2026-09-05 half-sentence "Exact-pin (`=X.Y.Z`) ONLY +# where an upstream crate itself demands exact-equals" is SUPERSEDED by the +# operator ruling 2026-09-18: *"never pin to x.00, always float x.*"* / *"so no +# decimal .0.0"*. NOTHING in this workspace carries a `.0.0` exact pin any more; +# the patch FLOATS (`11.*`, `0.38.*`) even where lancedb spells its own +# requirement `=11.0.0`. Floating is strictly SAFER, not looser: our own pin can +# only ever be equal to or NARROWER than what the family asks for, so an exact +# one buys nothing and deadlocks the moment lancedb's transitive requirement +# moves by a patch (the Cargo.toml:141-144 receipt below is exactly that +# failure). What survives verbatim: everywhere else the MAJOR, so the # family can converge on the newest compatible patch. Nothing outside the # lance / lancedb / arrow / datafusion family is pinned at all, and NO repo # tracks a Cargo.lock (.gitignore; ISS-STALE-AUTHORITY-LOCKS, RESOLVED). @@ -1231,7 +1247,7 @@ cd crates/lance-graph-python && maturin develop # (Rust crate line is 0.33.0 -> 0.37.1 -> 0.38.0; 0.34/0.35/0.36 do not exist as # Rust crates -- those numbers are the independently-versioned PyPI package.) # -# The lance family moves in EXACT lockstep -- currently =11.0.0. Line of state: +# The lance family moves in EXACT lockstep -- currently the 11 line. Line of state: # lance 7 (2026-06-14) -> 9 (the lance-9 sweep, b2b08b07 / PR #896 arc) -> 10 # (#1187, D-LNC-1) -> 11 (#1190, D-LNC-3, probe-gated on the D-LNC-2 fragment-id # probe #1189). arrow/datafusion did NOT move with any of them. @@ -1266,10 +1282,10 @@ datafusion = "54" # OUR direct pin, in every crate that DEPENDS on it # 2026-08-18; crates.io releases still DF 53) — as its # own deliberate PR if a consumer needs Delta. # Probe: .claude/plans/lance9-datafusion54-upgrade-probe-v1.md -lance = "=11.0.0" # exact-pinned: lancedb 0.38.0 requires lance =11.0.0 -lance-linalg = "=11.0.0" -lance-index = "=11.0.0" -lancedb = "=0.38.0" # the lance-11 pairing (`default-features = false` in +lance = "11.*" # lancedb 0.38 requires `=11.0.0`; we float the patch +lance-linalg = "11.*" # and resolve to whatever it demands (today 11.0.0) +lance-index = "11.*" +lancedb = "0.38.*" # the lance-11 pairing (`default-features = false` in # the workspace table). NOTE: the Rust crate line is # 0.33.0 -> 0.37.1 -> 0.38.0; 0.34/0.35/0.36 exist only # as the independently-versioned PyPI package.