diff --git a/.claude/audits/nars-34-substrate-audit.md b/.claude/audits/nars-34-substrate-audit.md new file mode 100644 index 000000000..28e84ef2e --- /dev/null +++ b/.claude/audits/nars-34-substrate-audit.md @@ -0,0 +1,144 @@ +# Audit — the 34 NARS recipes against the CURRENT substrate + +> 2026-09-14. Read `recipes.rs` (all 34) against `ndarray::simd_masking_ops` +> (the masking algebra) and the workspace's own iron rules. The question asked: +> **what needs improving.** + +## ⊘ CORRECTION — `bucket` is NOT a dispatch key (read after writing this) + +The audit below says the stale `bucket` column matters because it routes. **It +does not route.** Measured: + +- `recipe_dispatch.rs` — **zero** uses of `bucket`. Dispatch is by + `RecipeInference` (deduction/induction/abduction/revision/counterfactual), + `rung` + `dispatch_order` (from `Tier` + `InferenceType::rung_delta`), and + the `nan_disqualifier` checklist. +- `recipe_kernels.rs` — **one** use, the default `Tactic::gate`: + `match bucket { Bucket::Gate => ctx.gate_state() != GateState::Flow, _ => true }`. + +So only the **`Gate`** arm is load-bearing, and `Gate` is the one bucket this +audit found ACCURATE (TCP/CAS/TCF/CUR are the prune/cascade family the masking +algebra genuinely expresses). `Datapath` vs `Control` is **inert descriptive +metadata that nothing reads.** + +Consequence: the 9 stale `Datapath` strings are **documentation debt, not live +mis-dispatch.** They still want fixing — a string naming a retired kernel reads +as a spec — but item 1 below ("re-derive bucket") is NOT urgent and NOT a +correctness risk. I claimed the opposite in session ("a routing key derived +from a retired substrate is currently deciding what executes") without reading +either module. That claim is withdrawn. + +## ⊘ ALSO: the loco↔34 bridge already exists — do not build it + +`lance-graph-ogar/src/recipe_vocab.rs` is the 34 as loco ops: +`RECIPE_OP_BASE = DOMAIN_FLOOR`, ids `1..=34` ↔ bytes `0x90..=0xB1`, +`op_of` / `recipe_of`, `impl Vocabulary for RecipeVocabulary`, and +`ladder_program() -> Vec` — the ladder already lowered to a loco +program in `dispatch_order`. It carries two gates that answer different +questions (awareness: the kanban census; epistemic: the NaN pothole) with +`refusal_of` reporting which spoke, "because an unwilling ladder and an unable +one are different diagnoses." + +Its header also settles the dependency question correctly: *"`ogar_loco` is +zero-dep by design and `lance_graph_contract` is zero-dep by charter. Neither +may import the other. A vocabulary needs both, so it lives in a consumer that +already depends on both."* An `ogar-loco/src/nars.rs` with a contract dep — +which this session started writing — is the wrong home for a reason already +recorded. + +## The finding that IS serious, and is not mine + +`E-RECIPE-SELECTOR-REACHABILITY-1`, measured and documented in +`recipe_dispatch.rs`'s own header: through the shipped saccade selector +(`materialize::select_tactic`) only **8 of 34 recipes are reachable**, **all 14 +Infrastructure recipes never win**, and **ICR #31 is permanently shadowed by +RCR #4** on a lowest-id tie. The `ladder` mode exists to cover exactly those +three gaps, and `dispatch_mode.rs` routes between the two modes. That is a +larger fact about the 34 than anything in the substrate-string audit below. + +## Headline: the `bucket` column was assigned against a substrate that is gone + +**9 of 9 `Datapath` recipes name a retired or forbidden realization.** The tier +that is supposed to BE the masking ops is the tier most contaminated: + +| id | code | substrate string | verdict | +|---|---|---|---| +| 19 | ARE | `ABBA unbind: A⊗B⊗B=A` | VSA — retired | +| 24 | ZCF | `VSA bind(A,B)` | VSA — retired | +| 25 | HPM | `fingerprint cosine/Hamming sweep` | Hamming — retired | +| 27 | MPC | `bundle = majority-vote-per-bit` | VSA bundle — retired | +| 28 | SSAM | `bind+similarity (Gentner)` | VSA — retired | +| 34 | HKF | `cross-domain bind(A,rel,B)` | VSA — retired | +| 14 | MCT | `… → one fingerprint` | fingerprint — retired | +| 12 | TCA | `Markov ±5` | **±5 was retired** by the whole-book finding (63.3% of same-subject links reach beyond ±5) | +| 32 | SDD | `Berry-Esseen noise floor` | **forbidden by `I-NOISE-FLOOR-JIRAK`** — classical Berry-Esseen is wrong under this system's weak dependence; Jirak 2016 is the rule | + +So the `Datapath` label currently means *"was a VSA kernel"*, not *"is a masking +op"*. **I asserted earlier in this session that `bucket` is still a valid +routing column and only `substrate` was stale. That was wrong** — the bucket was +derived FROM the substrate, so it inherited the staleness. + +## What the masking algebra DOES express today + +Cross-checking the 34 against the real op list (`eq/ne/lt/le/gt/ge_*_to_mask`, +`ternary_match_*`, each with a gated `_under` twin; `mask_and/or/xor/andnot/not`; +`mask_ternlog` over the full 256-table; `mask_any/all`; `masked_sum/min/max_i32`, +`masked_strided_group_sum`): + +| id | code | bucket | expressible as | note | +|---|---|---|---|---| +| 5 | TCP | Gate | a gated predicate — `*_under` | prune = don't evaluate where the gate is empty | +| 8 | CAS | Gate | a cascade of `_under` tiers | the INT1/4/8/32 ladder is the gate chain | +| 20 | TCF | Gate | N masks → agreement | `ternlog` majority + `popcount` | +| 26 | CUR | Gate | coarse-to-fine `_under` chain | same shape as CAS | +| 30 | SPP | Control | N independent masks → agreement | identical algebra to TCF; the ECC/RAID framing is the same majority | + +**The recipes the masking algebra can run TODAY are 4 Gate + 1 Control — and +zero Datapath.** The tier assignment is inverted relative to the substrate that +actually exists. + +## Other defects found + +- **22 ETD** — `"CLAM cluster geometry determines subtasks (no spec)"`. Says so + itself. Unspecified, not merely stale. +- **18 CWS** — `"persistent BindSpace"`. The singleton BindSpace was retired + (`E-MARKOV-TEMPORAL-STREAM-1`: *"in most cases the singleton-BindSpace VSA + substrate is NOT used"*). +- **31 ICR** — `"CausalEdge64 −6 mantissa"` cites the **v2** layout. Truth + (frequency/confidence) is moving to **v3**, so this reference needs re-pinning + with that change rather than after it. +- **1 RTE** — `"Berry-Esseen stop"`. Same iron-rule violation as SDD, in a + Control recipe. + +## What needs improving, in order + +1. **Re-derive `bucket` from the masking algebra**, not from the VSA substrate. + The five above are what the algebra can run today, and the table classifies + them **4 `Gate` + 1 `Control`, zero `Datapath`** — the current nine + `Datapath` labels are the ones that are not honest. ⊘ This read "the five + above are the honest `Datapath` set today", contradicting the table two + sections up; caught in review on PR #1235. +2. **Rewrite the 9 stale `substrate` strings** to name mask-algebra + compositions, or mark them `unrealized` — a string naming a retired kernel + reads as a spec and is worse than an empty one. +3. ⊘ **CLOSED in the same commit** — both strings now name shipped Jirak + surfaces (`recipes.rs:97` `SigmaTierBands::jirak_p`, `:408` `jc` Pillar 5), + and `no_recipe_cites_classical_berry_esseen` guards against a third. The + two rows above record the PRE-fix state and are kept as the finding, not + as outstanding work. Originally: **Fix the 2 Berry-Esseen citations** (1 RTE, 32 SDD) to Jirak 2016, per + `I-NOISE-FLOOR-JIRAK`. These are iron-rule violations sitting in shipped data. +4. **Re-pin 12 TCA** off the retired ±5 window onto the version-range read + (`QueryReference::at(v, rung)`), which is what replaced it. +5. **Re-pin 18 CWS** off the singleton BindSpace. +6. **31 ICR** rides the CausalEdge64 v3 change. +7. **22 ETD** needs a spec or an explicit `unrealized` mark. + +## What this audit does NOT claim + +It does not say the 34 are wrong as *tactics*. It says their recorded +realization is against a substrate that no longer exists, and that the routing +column derived from it cannot be trusted until re-derived. (⊘ this said +"as a dispatch key"; per the correction at the top of this file `bucket` is +NOT a dispatch key at all except for the `Bucket::Gate` arm, so the phrase +re-asserted the thing the correction retracted.) +Nothing here was executed; this is a read of the catalogue against the op list. diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index f8862f052..da91d3c62 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,176 @@ +## 2026-09-15 (4) — E-FUSING-FORFEITS-THE-SKIP-AND-ADAPTIVEFILTER-FAILS-IN-TWO-PLACES-NOT-ONE-1 — the fused lowering is order-independent BY CONSTRUCTION, two readers derived it from source because the crate doc does not say so, and DuckDB's A1 turns out to have a dead seed as well as an unrunnable loop + +**Status:** FINDING. Convergent — derived independently in two sessions from the same +line, neither having read the other, before comparing. +**Confidence:** HIGH — every claim is a source read or a row of §8a, all re-runnable. + +### The finding + +`lower` and `lower_fused` differ in a way neither name nor the crate doc suggested: +**only `lower` can skip.** + +| | gate on a predicate | order changes skipped words? | +|---|---|---| +| `lower` (in place) | first ungated, later ones on the running ACCUMULATOR | **yes** — up to 99.90 pts (§8a) | +| `lower_fused` | `under.map(\|m\| Operand::Plane(m.0))` — caller's plane ONLY | **no, ever** | + +`assign_slots` gives every predicate its own slot and never chains, because the Boolean +combination is deferred to the fuser — and `MaskOp::Ternlog { imm, a, b, c, dst }` has +**no `under` field at all**, where `MaskOp::Pred { pred, under, dst }` does. A ternlog +combines already-materialised masks; by the time it runs, every sweep is paid for. + +The saving the gated form buys is physical, not bookkeeping. `ndarray`'s `pack_under` +(`simd_masking_ops.rs:1541-1546`) is `if gate == 0 { out_words[w] = 0; continue; }` — +the 64 values are never loaded. That `continue` is the entire mechanism §8a measures. + +The crate doc framed the choice as *"a consumer picks by whether it is scratch-bound or +pass-bound"* and said nothing about the skip. Both readers fell into it and both had to +go to source. Fixed in the same commit. + +### Why it matters beyond the doc + +It is the whole of what survived DuckDB's `AdaptiveFilter`. §8a measures ordering worth +up to 99.90 percentage points of skipped words on a clustered conjunction — **under +`lower` only**. On the same query under `lower_fused` it is exactly zero. So +`Filter::and_by_skip`'s lever is alive in one configuration: gated lowering × plane-free +conjunction × contiguous survivors. Under a plane it is inert as well +(`ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE`). `adaptive_order_probe.rs` has no fused +arm, so its table cannot see this and does not claim to. + +### A1 fails in TWO places, and only one of them was recorded + +`AdaptiveFilter` is a seed plus a loop, and they die of different causes: + +- **the seed** — `GetInitialOrder`, from the optimizer's static SELECTIVITY heuristic. + §8a kills it independently of anything about V3's executor: selective (36 survivors, + 0.055 %) and clustered (31, 0.047 %) are indistinguishable by density and **19.24 + points apart best-vs-best**. A selectivity-seeded reorderer seeds on a statistic that + provably does not separate the two regimes where the lever exists. +- **the loop** — swap, measure RUNTIME, keep or revert, halve likeliness. This one would + find contiguity, because contiguity shows up in runtime whether or not you can name + it. It is the half quack structurally cannot run: `Program.ops` is a fixed `Vec` with + no measurement. + +**The half that would work cannot run; the half that can run measures the wrong thing.** +That is stronger than the matrix's "not DuckDB's algorithm" — it says why no amount of +porting reaches it. + +### The V3-native seed, and its one blocker + +The clustered regime IS an address prefix. `Filter::prefix_u32` builds +`care = u32::MAX << (32 - b)` — a contiguous high run — so a `Cmp::MatchU32/MatchU64` +whose care mask has that shape is prefix-shaped **at lowering time**: free, structural, +no measurement loop. That would feed an ordering score from structure instead of asking +the caller. + +The blocker is exact and is one bit: `prefix_u32`'s own doc says contiguous *"on an +address-ordered lane"* — a property of the DATA, not the query — and `Col(pub u16)` is +deliberately not a name ("name resolution is the catalogue's job"). quack carries no +catalogue by design, so the seed needs a schema fact the crate does not hold. Real +blocker, not a detail. + +### ⊘ A citation of mine that was under-qualified + +I cited D-GTM-0m as "22.4–22.8 µs, survivor-independent" without its stated limits. The +matrix carries them twice and I should have carried them too: R5 — *"measured on ONE +fixture (65 536 rows = 256×256 axial hex, 62 % permeable, one tile size, timing floor +50 ms, no `perf`)"*; E4 — it *"widened a u8 permeability column 4× to use +`gt_i32_to_mask`, and reported its `n_gen` and coal numbers as upper bounds because of +it."* + +The rescue offered for it — that survivor-independence is structural because +`gt_i32_to_mask` takes no gate parameter — is right about that function and must not be +read as a general claim about predicate generation, or it contradicts §8a. The accurate +form: **an UNGATED sweep is structurally survivor-independent; a GATED one structurally +is not**, `gt_i32_to_mask_under` exists (`simd_masking_ops.rs:1614`), and `pack_under`'s +`continue` is where the difference physically lives. The µs figure is one geometry; the +asymmetry is not. + +## 2026-09-15 (3) — E-A-CHECK-THAT-CANNOT-RUN-IS-INDISTINGUISHABLE-FROM-A-CHECK-THAT-PASSES-1 — "lots of CI errors" was stale red plus silence, and chasing it found `--all-features` broken since the lancedb 0.38 bump + +**Status:** FINDING. Two independent mechanisms, one shape. Both measured on PR #1235. +**Confidence:** HIGH — every claim below is an API read, a `cargo` exit code, or a line +number in a vendored crate. All re-runnable. + +### The shape + +A verdict you can see is not a verdict about the code you have. Two ways that breaks, and +both were live in this repo at the same moment: + +1. **The workflow could not run** — the PR was conflicted, so GitHub had no merge ref to + run `pull_request` workflows against. The last runnable SHA's red stayed on the page. +2. **The workflow does not run on push** — `rust-publish.yml` fires only on + `release: released` / `workflow_dispatch`, so what it checks rots between releases with + nothing to report it. + +Case 1 shows you a stale answer. Case 2 shows you no answer and you read it as "fine". +Neither is distinguishable from green by looking. + +### Mechanism 1 — a conflicted PR produces NO run, and the UI does not say so + +| what | measured | +|---|---| +| PR #1235 head | `947753d` (the aws-smithy fix) | +| workflow runs for `947753d` | **0** — none, in any state | +| newest runs on the branch | `36646a2`, the commit BEFORE the fix | +| PR `mergeable` / `mergeable_state` | `false` / **`dirty`** | + +`build.yml`, `rust-test.yml` and `style.yml` trigger on `pull_request`, which GitHub runs +against `refs/pull//merge` — a ref it can only synthesise when the PR merges cleanly. +A conflicted PR therefore produces no run at all. Five non-`pull_request` workflows +(`Supersession index`, `Append-only gate`, …) DID run and DID pass, so the page showed +five greens beside two stale reds and **not one of the seven described the head**. + +**The check, before believing any red on a PR:** compare the failing run's `head_sha` +against the PR's `head.sha`, and read `mergeable_state`. `dirty` means CI is silent, not +failing, and the remedy is to merge or rebase the base in — not to debug the code. + +### Mechanism 2 — `--all-features` was already broken, and only a release would have said so + +Chasing mechanism 1 turned up a second one. `.github/workflows/rust-publish.yml` +PASSED `args: "--all-features"` to `katyo/publish-crates@v2`, which runs a verification +build before publishing; `c2b4bc7` in this same PR replaced it with an explicit list. +Measured on the tree before that fix: + +| invocation | exit | +|---|---| +| `cargo check --workspace` | 0 | +| `cargo check --workspace --all-targets` | 0 | +| `cargo check -p lance-graph --all-features` | **101** | + +TWO independent causes, and the interesting one is not ours: + +- **`aws-sdk`** — ours, added this session, opt-in by design because + `aws-smithy-json 0.63.0` does not build against `aws-smithy-types 1.7.0`. +- **`lancedb-sdk`** — **not ours, and older.** `lancedb 0.38.0` declares `default = []`, + gates `Error::Http` behind `#[cfg(feature = "remote")]` (`src/error.rs:111`), but leaves + `pub mod job;` ungated (`src/lib.rs:188`) while `job.rs` uses `Error::Http` + unconditionally at `:56` and `:66`. **That crate cannot compile without `remote`.** The + workspace pin `lancedb = { version = "=0.38.0", default-features = false }` — the + `lancedb` key in the root `Cargo.toml`'s `[workspace.dependencies]` — is on `main` + unchanged and was untouched by `947753d`. + +So `--all-features` has been failing since the lancedb 0.38 bump (#1190), and **no +push-triggered run could ever have gone red for it**, because the only call site is a +workflow that fires on `release: released` / `workflow_dispatch` only. ⊘ An earlier +draft of this line said *no branch* could have gone red — too strong: +`workflow_dispatch` can be aimed at any branch, so the failure was reachable on +demand, just never by the ordinary push/PR cadence that makes a failure traceable to +a cause. The `aws-sdk` flag did not create this; it added a second reason to the same +silent failure. + +### What generalizes + +**Adding a feature you know to be broken is not a local act** — every `--all-features` +call site becomes a caller of it. Grep before landing one. Here there were two: +`rust-publish.yml` (fixed) and `style.yml:95`, which is scoped to `lance-graph-quack` — +one path dep, zero declared features — so it cannot reach `lance` or `lancedb`. + +**And the stronger one:** a workflow that only runs on release is not a gate, it is a +deferred assertion. Its failure is dated to whenever someone next cuts a release and will +be attributed to whatever PR happens to be adjacent. If a check matters, it has to run on +a cadence where its failure is still traceable to a cause. + ## 2026-09-15 — E-I-GRAFTED-HELIX-ONTO-HEXAGON-AND-THEN-DEPRECATED-THE-OPERATORS-TENANTS-ON-MY-OWN-AUTHORITY-1 — there is no residue in Hexagon; the guard that refused the graft was recorded as a missing feature; and then I called two of the operator's shipped tenants dead **Status:** CORRECTION. Operator-caught, same day, hours after @@ -155,6 +328,77 @@ the seam-6 promoter that does ship), `.claude/plans/hexagon-plasticity-v1.md`. --- +## 2026-09-15 — E-THE-ACCUMULATOR-GATE-OUTRANKED-THE-PLANE-AND-SILENTLY-DROPPED-IT-1 — a nested `Plane` vanished from the emitted program because the *enclosing* conjunction already had a gate + +**Status:** CORRECTION — a real correctness bug in shipped-in-PR code (`lance-graph-quack` +`emit_gated`), found by codex review on PR #1235, reproduced against the reference oracle +before the fix, fixed, and pinned by a permanent regression. +**Confidence:** HIGH — the reproduction is a number, not an argument: oracle **29**, emitted +program **204**, and the op list showed the plane's `FOCUS` operand absent entirely. + +### What happened + +`emit_gated(filter, under, acc_gate)` carries two gates. `under` is the plane the CALLER +handed down — `Filter::Plane(m)` encountered as a child, which must gate everything to its +right. `acc_gate` is the accumulator-so-far, the thing that makes the survivor skip work: +once some terms have been evaluated into a scratch, later comparisons can be evaluated +`Pred { under: Some(acc) }` and skip the 64-row words the accumulator already killed. + +The line was: + +```rust +let gate = acc_gate.or_else(|| under.map(|m| Operand::Plane(m.0))); +``` + +`Option::or_else` — the accumulator WINS whenever both are present. For a flat conjunction +that is right and is the whole point (the accumulator is strictly narrower than the plane +it already absorbed). For a **nested** one it is wrong, because the accumulator belongs to +the OUTER conjunction and has never seen the inner plane: + +```text +P1 AND (Plane(focus) AND P2) +``` + +The outer `AND` evaluates `P1` into the accumulator, recurses into the parenthesis with +`under = None, acc_gate = Some(acc)`; the inner `AND` meets `Plane(focus)`, sets +`under = Some(focus)`, and then for `P2` the `or_else` picks the accumulator and drops the +plane on the floor. `FOCUS` never appears in the program. Measured: 204 rows where the +oracle says 29. + +### The fix, and why it is the conservative direction + +```rust +let gate = under.map(|m| Operand::Plane(m.0)).or(acc_gate); +``` + +The plane always wins. This is conservative rather than optimal: where both are live the +accumulator may be the narrower gate, so preferring the plane can leave some skip on the +table. It can never be WRONG, because `Pred { under: g }` is exactly `g ∧ pred` +(`reference.rs:460`) and the `AND` that consumes the scratch re-applies the accumulator +anyway. The reverse — preferring the accumulator — drops a conjunct, which is a wrong +answer, not a slower one. **A gate you can only lose by choosing beats a gate you can drop +by choosing.** + +### The lesson that generalizes past this bug + +**`or_else` between two gates is a silent priority decision, and priority between gates is +only safe when one PROVABLY subsumes the other.** The accumulator subsumes `under` exactly +when the accumulator was built from a prefix that already included it — true for a flat +conjunction, false the moment recursion hands the accumulator across a nesting boundary. My +own hand-trace missed it because I traced the flat case, where the two are in fact ordered. +The shape that breaks it needs a plane that is BOTH nested and not first, which no existing +test had: `hoist_gate_subset` rotates a plane-subset child to the front precisely so the +accumulator starts inside the plane, and every fixture went down that path. + +### What the fix cost in the test suite, recorded because it is uncomfortable + +`the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` asserted the plane +rode the ACCUMULATOR (`assert!(on_acc)`). That assertion was a description of the bug, not +of the law — it is now inverted (`assert!(!on_acc)`) with a ⊘ note, and the permanent +regression `a_nested_plane_survives_an_outer_accumulator` is what actually pins the +behaviour: two-sided (`expected * 2 < without_plane`, so a fixture whose plane admits +everything cannot pass) plus a structural check that some emitted op reads `FOCUS`. Both +arms of the disable run are RED. ## 2026-09-15 — E-I-DECLARED-A-JOIN-ABSENT-BY-GREPPING-ONE-FILE-AND-COMPOSE-IS-THE-SAME-XOR-A-THIRD-TIME-1 — the canonical join shipped in `hhtl.rs` all along, `[a,b]:[b,c]` is `compose_chain`, and the Hexagon substrate is every organ shipped with no nerve between them @@ -248,7 +492,7 @@ already running. > Hexagon was tested exhaustively and never involved one. The real learning > surface is seam 6 of this session's own inventory, already hexagon-shaped: > `FrozenStyle`/`LearnedStyle`/`ExploreStyle`, `U8×12` = **6 × 2 × palette256** each — -> the shape the operator ruled and this board MEASURED (`EPIPHANIES:19221`, ρ_all 0.966, +> the shape the operator ruled and this board MEASURED (`E-CAM96-DISTRIBUTION-MEASURED-1`, ρ_all 0.966, > near-orth 170×) as better than the 48-bit class `HelixResidue` belongs to — with the > shipped held-out promote gate already on it. Full storno: > `E-I-GRAFTED-HELIX-ONTO-HEXAGON-AND-THEN-DEPRECATED-THE-OPERATORS-TENANTS-ON-MY-OWN-AUTHORITY-1`. @@ -1426,6 +1670,52 @@ the tile, seriation, LUT and spread contribute nothing that survives permutation **Scope.** Pure-Python lab, no Rust fingerprints — a shape proxy, never a measurement of `ndarray`/`bgz17`/`helix`. The cue reached **87.5 %** of the BPE incumbent (0.1579 vs 0.1805): PROCEED, not PASS. +## 2026-09-14 — E-A-FLOOR-PASSED-AT-ITS-BOUND-IS-A-DEAD-FIXTURE-1 + +**Status:** FINDING (measured twice in one file, both arms) +**Confidence:** High — the invisible-mis-map half is a direct red/green pair + +**An anti-vacuity bound written as a FLOOR (`>= N`) that passes at exactly +`N` is not a pass. It is the fixture telling you it is dead.** + +`lowering_convergence.rs` (lance-graph-java `native/lgj-abi`) arrived with +two: `>= 15` of 28 combine vectors non-degenerate, and `>= 7` of 9 opcode +seeds selecting a proper subset. Both measured **exactly** at their bound. +Both bounds had been reasoned to, honestly, from the fixture's DOCUMENTED +domain — and the documented domain was not the measured one. + +- The values lane is `-150..=361`. The arity-4 arm appended `LT_I32(500)`, + an always-true op, so the entire 16-vector arm was eight saturated + `n`-row answers plus eight verbatim copies of the arity-3 row: **zero** + additional discriminating power, clearing `>= 15` by sitting on it. + Measuring the lane and using `LT_I32(200)` took it **15 -> 21**. +- Worse, the same operand appeared in the per-opcode arm, so `LT_I32` and + `LE_I32` both selected every row. Measured: mis-mapping `LGJ_OP_LE_I32` + to `Pred::LtI32` — one token, and exactly the defect the file exists to + catch — is **RED at operand 300** (866 vs 865, a one-row difference) and + **GREEN at operand 500**. Two of nine opcodes were untested while the + file read as covering all nine. + +**Three rules, each with its own force:** + +1. **A floor is the wrong shape for an anti-vacuity bound.** Its whole job + is to notice the fixture going inert, and a floor cannot: inert is + exactly where it still passes. Write `assert_eq!` with the measured + count and the instruction not to relax it. +2. **Reason operands from the MEASURED lane, never from the documented + domain.** "The doc says the max is 361, so 500 is safely past it" is + sound arithmetic and produces a tautology. Probe the distribution. +3. **In a differential between two ARMS (not against ground truth), two + quantities that select the same number of rows hide a swap between + exactly those two.** Give the comparisons distinct counts on purpose. + This is specific to arm-vs-arm testing and does not arise when one side + is an oracle. + +Cross-ref: the falsifiability rule in `CLAUDE.md` already names +"a tolerance/threshold parameter needs an inertness test". This is its +sibling for the FIXTURE rather than the threshold — and the sharper case, +because a dead fixture leaves every assertion above it reading as green. +Board: `STATUS_BOARD.md` D-QCK-10; `LATEST_STATE.md` 2026-09-14 (6). ## 2026-09-14 (3) — E-THE-VOCABULARY-IS-THE-RECOGNITION-ORGAN-THE-LAW-IS-THE-TRANSFER-ORGAN-1 — F-MQ8 and H5b were filed as one null; they are a division of labour, and they say which half is weak diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 840d0d4a8..dc6a214e8 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,3 +1,257 @@ +## ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE (2026-09-15) — OPEN, upstream bug, our `lancedb-sdk` feature does not build + +`lancedb 0.38.0` does not compile with its own default feature set. Measured, reading the +vendored crate: + +| site | fact | +|---|---| +| `Cargo.toml` | `default = []` — so `default-features = false` on our side is a **no-op** | +| `src/error.rs:111` | `Error::Http` is `#[cfg(feature = "remote")]` | +| `src/lib.rs:188` | `pub mod job;` — **not** gated | +| `src/job.rs:56,66` | uses `Error::Http` unconditionally | + +Result: `error[E0599]: no variant named 'Http' found for enum 'error::Error'`, twice, +inside lancedb itself. Reproduce with +`cargo check -p lance-graph --features lancedb-sdk`. + +**Not ours and not new.** `lancedb = { version = "=0.38.0", default-features = false }` is +on `main` (the `lancedb` key in the root `Cargo.toml`'s `[workspace.dependencies]`), +landed with the lance-11 / lancedb-0.38 bump (#1190). Nothing +caught it because `lancedb-sdk` is optional, off by default, enabled by no workspace +member, and reachable in CI only through `rust-publish.yml`'s `--all-features` — a +workflow that never runs on push. See +`E-A-CHECK-THAT-CANNOT-RUN-IS-INDISTINGUISHABLE-FROM-A-CHECK-THAT-PASSES-1`. + +**Candidate fix, NOT applied here:** add `features = ["remote"]` to the workspace `lancedb` +entry. `remote = ["dep:reqwest", "dep:http", "dep:urlencoding", "lance-namespace-impls/rest", +"lance-namespace-impls/rest-adapter"]` — no aws, so it does not reintroduce the +`aws-smithy` breakage. Deliberately deferred: it adds `reqwest` to anyone enabling +`lancedb-sdk`, which is a dependency-graph decision that wants its own measured PR rather +than a tail-end change in a PR about something else. The alternative — report it upstream +and pin the fixed patch — is also open. + +**Blast radius, measured** — every crate that can reach the crate, and whether anything +turns it on: + +| crate | declaration | default-on? | +|---|---|---| +| `lance-graph` | `lancedb-sdk = ["dep:lancedb"]` | no — its `default` list omits it | +| `surreal_container` | `lancedb-sdk = ["dep:lancedb"]` | no (`default = []`) | +| `holograph` | a feature literally NAMED `lancedb` — but it maps to `["dep:lance"]`, i.e. it does **not** pull the lancedb crate | n/a | + +So **two** unbuildable `lancedb-sdk` features exist, not one, and a fix must clear both. +Nothing enables either, which is why `cargo check --workspace` is EXIT 0 and always has +been. The `holograph` entry is a naming trap worth knowing about separately: a feature +called `lancedb` that has nothing to do with lancedb. + +Until then `lancedb-sdk` is a declared-but-unbuildable feature, the same status +`aws-sdk` carries, and both are excluded from the publish verification list. + +## ISS-PUBLISH-FEATURE-LIST-CAN-DRIFT (2026-09-15) — OPEN, low severity + +`.github/workflows/rust-publish.yml` no longer passes `--all-features`; it passes an +explicit list of every `lance-graph` feature except the two that do not build — +`aws-sdk` (ISS-AWS-SMITHY-…) and `lancedb-sdk` (ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE). +That was forced: the publish step runs a verification build, and `--all-features` enables +both. Measured — `cargo check -p lance-graph --all-features` exits **101**; the explicit +list exits **0**. + +**The residue:** a feature added to `crates/lance-graph/Cargo.toml` later is silently NOT +covered by the publish verification, where `--all-features` would have picked it up for +free. The list was verified complete at landing by parsing both files and diffing the sets +(`declared - passed == {aws-sdk, lancedb-sdk}`, `passed - declared == {}`) — but that was a +one-off, not a gate. + +**The real fix, ~20 lines, not done:** a CI check that parses `[features]` out of +`crates/lance-graph/Cargo.toml`, parses the `args:` line out of `rust-publish.yml`, and +fails when the difference is anything other than the two known-broken names. Same shape as +the existing `append_only_gate.py` / `supersession_index.py` gates, and it would also make +the two exclusions expire loudly instead of silently outliving their cause. + +Severity is low: the publish workflow runs only on `release: released` / +`workflow_dispatch`, and a drift there costs one failed release job, not a bad artifact — +the verification build failing is precisely what *stops* the publish. + +## ISS-AWS-SMITHY-BREAKS-THE-WORKSPACE-BUILD-AND-THERE-ARE-TWO-REMEDIES (2026-09-15) — ⊘ RESOLVED SAME DAY, and the recorded cost was WRONG + +**Resolved by the operator's own framing — "make it optional so that later we fork 1.7 and fix +it if we ever want it"** — which is better than either remedy below, because it keeps the +capability addressable instead of deleting it. `lance` now takes `default-features = false` plus +its own default list MINUS `aws`, and `lance-graph` gains an opt-in `aws-sdk = ["lance/aws"]`. +**Measured after: `cargo check --workspace` EXIT=0 in 2m56s** — the job that had been red on +`main` and every branch. + +**⊘ The cost recorded below ("drop lance's `aws` feature → costs S3 object-store support") was +WRONG, and the operator caught it** by asking whether this was the native AWS library rather +than Tigris/Railway S3 slab hydration. It is the native SDK. `lance-io`'s `aws` feature bundles +TWO unrelated things — the AWS SDK (`aws-config`, broken) and `object_store/aws`, the generic +S3-COMPATIBLE backend — and only the first is dropped, because `object_store = { features = +["aws"] }` is declared DIRECTLY by the workspace and by `crates/lance-graph`. Verified: +`aws-smithy-json` ABSENT, `aws-config` ABSENT, `object_store` feature `aws` still ENABLED, and +`--features lance-graph/aws-sdk` brings `aws-config` back (so the switch is real, not +decoration). `lance-graph-hydrate`'s slab hydration is untouched — it drives `object_store` +with `aws_endpoint` + `aws_virtual_hosted_style_request = false`, an S3-compatible endpoint, +and never names the SDK. **Nothing in this workspace references `aws_config` / `aws_sdk_*` / +`aws_credential_types` at all.** What is actually lost: AWS-*native* credential machinery only +(IMDS, SSO, STS assume-role). + +**A past session had already built the insurance that makes this safe.** +`crates/lance-graph/Cargo.toml`'s `[dev-dependencies]` `object_store` entry declares +`object_store/aws` directly and says why: +*"slimming them to `default-features = false` is a plausible future move — and it would silently +remove S3 from THIS crate's own S3 callers … it makes the capability this crate USES a thing +this crate ASKS FOR."* That is exactly this move, anticipated. + +**Upstream has no fix, and checking told us the root cause.** This repo is a fork of +`lance-format/lance-graph`, which is fully green — because it **tracks a `Cargo.lock`** pinning +`aws-smithy-json 0.61.5` / `aws-smithy-types 1.3.2`, and sits on `lance 1.0.1` / +`object_store 0.12.4`, ten majors behind. It carries no aws-smithy pin, patch or workaround; +grep finds the string nowhere in its tree. So there was nothing to port — its immunity is the +tracked lock we deliberately removed (`ISS-STALE-AUTHORITY-LOCKS-RESIDUE`, 2026-09-04), which +is why our CI re-resolves into newly-published breakage on every run. That is a THIRD remedy +(restore a tracked lock) and the only one that reverses a prior ruling; not taken. + +The original entry follows unchanged. + +## ISS-AWS-SMITHY-BREAKS-THE-WORKSPACE-BUILD-AND-THERE-ARE-TWO-REMEDIES (2026-09-15) — OPEN, repo-wide, operator decision + +**`cargo build --workspace` fails on `main` and on every branch**, in a third-party crate, and +it is NOT a resolver problem — no version selection fixes it: + +```text +error[E0308] aws-smithy-json-0.63.0/src/codec/deserializer.rs:707 + expected `DocumentObject`, found `HashMap` +error[E0004] aws-smithy-json-0.63.0/src/serialize.rs:36 non-exhaustive patterns +``` + +`aws-smithy-types` 1.7.0 changed `Document::Object` to take a `DocumentObject` and marked +`Document` `#[non_exhaustive]`; `aws-smithy-json` 0.63.0 has not caught up. `aws-smithy-json` +declares `aws-smithy-types ^1.6.1`, so it ALWAYS resolves the breaking 1.7.0, and the newest +`aws-config` 1.12.0 requires `aws-smithy-json ^0.63.0` and cannot reach the fixed 0.64.0. This +repo tracks no `Cargo.lock`, so every CI run resolves fresh and picks the incompatible pair. +Jobs affected: `linux-build`, `test`, `member-tests`, `test-with-coverage` — all four die at the +same `cargo build --workspace` step, before any test body runs. + +**There are TWO remedies and both are operator decisions. Neither is a pin-vs-nothing choice.** + +| remedy | cost | touches the pin whitelist? | +|---|---|---| +| exact-pin `aws-smithy-types` | a FIFTH pinned coordinate | **yes** — `CLAUDE.md`: nothing outside lance/lancedb/arrow/datafusion is pinned at all | +| drop lance's `aws` feature | S3 object-store support | **no** | + +**The second one is the newly-established fact, and it is TESTED rather than inferred.** +`aws-config` is an OPTIONAL dependency of `lance-io` behind its `aws` feature; `lance` takes +`lance-io` with `default-features = false` and re-exposes `aws = ["lance-io/aws"]` inside its own +default. So the chain is feature-reachable from our side: + +```toml +lance = { version = "=11.0.0", default-features = false, features = [ + "azure", "gcp", "oss", "huggingface", "tencent", "tos", "goosefs", "geo", +] } +``` + +`cargo tree -i aws-smithy-json` then reports **"did not match any packages"** — gone from the +graph entirely, with every other lance default feature retained. (`default-features = false` +alone works too, but drops eight features instead of one.) + +**How real the S3 cost is, stated precisely.** No hard-coded AWS calls in this tree, but +`lance-graph-catalog`'s `storage_options` surface documents and passes through +`aws_access_key_id` / `aws_secret_access_key` / `aws_region`, and `DirNamespace` parses `s3://` +URIs. The capability is exposed to callers even though nothing here exercises it in CI. Whether +a deployment depends on it cannot be determined from the tree. + +**Correction recorded on purpose:** the stand-down comment on PR #1235 said the remedy "is a +version pin", naming one option as if it were the only one. That was incomplete, and the missing +option is the one that leaves the pin rule alone. Corrected on the PR. + +## ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE (2026-09-15) — OPEN, and it is the price of the P1 fix + +**`Filter::and_by_skip`'s ordering lever buys exactly zero on any conjunction that carries a +resident plane — which is this crate's own headline query shape.** + +Created deliberately by the codex-P1 fix (`b7e6cef`, +`E-THE-ACCUMULATOR-GATE-OUTRANKED-THE-PLANE-AND-SILENTLY-DROPPED-IT-1`): `emit_gated` now +prefers the caller's plane over the accumulator, so every `Pred` inside a planed `AND` gates on +the same FIXED plane regardless of position. Measured at HEAD, each entry `predicate <- its gate`: + +```text +PLANE-FREE And[A,B,C] : [GtI32<-none, EqU32<-ACC(s0), LtI32<-ACC(s0)] + And[C,B,A] : [LtI32<-none, EqU32<-ACC(s0), GtI32<-ACC(s0)] + +WITH A PLANE And[P,A,B,C]: [GtI32<-Plane(0), EqU32<-Plane(0), LtI32<-Plane(0)] + And[P,C,B,A]: [LtI32<-Plane(0), EqU32<-Plane(0), GtI32<-Plane(0)] +``` + +Order changes which predicate runs first and nothing else; the skipped-word count is +order-independent. Plane-free conjunctions are unaffected — those preds take the `None` arm and +still chain on the accumulator, so **A1's measured lever survives exactly where the probe +measured it and nowhere else.** + +**Why it is the right trade anyway.** The alternative is preferring the accumulator, which drops +a conjunct when the accumulator crossed a nesting boundary — a wrong answer, not a slower one +(oracle 29, emitted 204). A lost optimisation beats a lost row. + +**Why no gate catches it.** `examples/adaptive_order_probe.rs`'s four scenarios are all +plane-free, so its table cannot measure this and does not claim to. Its +`accumulator_gated == terms.len() - 1` assertion is what keeps it honest and would fire +immediately on a planed scenario — which is why the honest move is to say "not measured" rather +than to add a scenario that would simply assert the inertness. + +**The way out, if it is ever worth taking:** gate on `plane ∧ accumulator` rather than choosing +between them. That is strictly narrower than either, so it restores the lever without +reintroducing the hole — at the cost of one extra op per gated predicate, which is exactly the +measurement nobody has made. Not attempted here; recorded so the option is not re-derived. + +Found by an independent correctness review of the fix, not by the fix's author. + +## ISS-QUACK-LOWER-FUSED-IS-SUPERLINEAR-AND-DEEP-FILTERS-ABORT (2026-09-15) — OPEN, both caller-controlled + +Two pre-execution costs on `lance-graph-quack`'s public surface, measured in **release**, both +reachable from ordinary caller input and both hit before a single row is touched. + +**1. `lower_fused` is ~cubic in leaf count, reachable from `Filter::in_u32`.** + +| | `lower` | `lower_fused` | source | +|---|---|---|---| +| `IN(64)` | 6.02 µs | 97.20 µs | reproduced here (review: 0.12 ms) | +| `IN(256)` | 16.33 µs | 3.08 ms | reproduced here (review: 4.28 ms) | +| `IN(1024)` | 24.40 µs | **158.47 ms** | reproduced here (review: 192.70 ms) | +| `IN(2048)` | 0.13 ms | **1 443 ms** | review only, not re-run | +| flat `AND` w=4096 | — | **10 597 ms** | review only, not re-run | +| flat `AND` w=65 000 | **2.99 ms** | (not attempted) | review only, not re-run | + +The first three rows are first-hand, and they agree with the review within ~25 % (machine +variance, not a discrepancy). `lower` is linear across all of them; the blow-up is entirely +`lower_fused` — ×31.7 then ×51.5 for each 4× in leaves. Mechanism verified in source rather than +taken from the report: `distinct_leaves` (`fuse.rs:72`) is `out.contains(o)`, O(k) per leaf; +`leaf_count` (`:87`) allocates a fresh `Vec` and re-walks; `:173` calls it on **both** children +(`leaf_count(l) >= leaf_count(r)`) at every level. Mechanism sits in +`lance-graph-mask-risc/src/fuse.rs` — `Lowering::lower`'s reduction calls `leaf_count` on BOTH +children at every level, `leaf_count` calls `distinct_leaves`, and that uses `Vec::contains`, +O(k) per leaf; quack's `assign_slots` left-folds an n-ary junction into a depth-n binary +`BoolExpr`. Same class as the quadratic-`validate` stall `reference.rs:110-137` already records +and fixed on that side. Cheapest mitigation is a sorted set or `u64` bitset in `distinct_leaves`. +**Not fixed here** because it lives in a different crate and this PR is about the query surface — +widening it on my own is the thing the repo's own push rules forbid. + +**2. A deep `Filter` ABORTS the process; `LowerError::TooManySlots` is unreachable on the +in-place path.** Default 8 MiB stack, one process per depth: + +```text +depth=15000 -> LOWER OK slots=2 ops=30001 +depth=20000 -> fatal runtime error: stack overflow, aborting +``` + +Construction and `Drop` of the same tree survive to 30,000, so the recursion is +`gate_walk`/`emit_gated`'s, not the data structure's. It is an **abort**, not a catchable panic — +`catch_unwind` and `JoinHandle::join` cannot contain it. Meanwhile `emit_gated`'s +`dst.checked_add(1) → TooManySlots` needs depth 65,535, which the stack cannot reach: **the error +the crate defines for this condition can never fire on `lower`.** An explicit depth budget in +`gate_walk` returning `LowerError` would make the declared refusal real. + +Neither is a correctness defect and neither is reachable from the differential suite, which builds +trees to depth 6. Both were found by an independent correctness review. + ## ISS-FAMILY-IS-FOUR-WIDTHS-TWO-AT-OPPOSITE-ENDS (2026-09-15) — ⊘ RESOLVED SAME DAY: falsifier ran, hazard CONFIRMED (0 vs 3); see E-THE-TWO-FAMILY-NAMINGS-INVERT-AND-FROM-BE-BYTES-IS-THE-PLAUSIBLE-WRONG-JOIN-1 **"family" denotes four different things in this tree, and the two that share a width sit at diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 912dbb8f7..508d9169c 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -25,6 +25,51 @@ result `n ∈ [8, 47]` is **predicted, not certified**), width on `ISS-BOUNDED-K-NEVER-FAILS-ON-ANY-GRAPH-TESTED`. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1234. +## 2026-09-15 — the workspace build is GREEN again: the AWS SDK is now optional, and the cost I recorded for that was wrong + +`cargo check --workspace` **EXIT=0 in 2m56s** — the job that had been red on `main` and on every +branch since `aws-smithy-types 1.7.0` published. Not a pin, not a deletion: the operator's +framing, verbatim, was *"make it optional so that later we fork 1.7 and fix it if we ever want +it."* So `lance` takes `default-features = false` + its own default list minus `aws`, and +`lance-graph` gains `aws-sdk = ["lance/aws"]` — one documented switch instead of a silent +removal. + +**Two corrections came out of this, and both were prompted by the operator asking a question +rather than accepting my summary.** + +**1. "S3 support" was the wrong unit.** I had recorded the cost as "drops S3 object-store +support". The operator asked whether I meant the native AWS library, *"not to be confused with +Tigris RAILWAY S3 slab hydration"* — and that distinction is load-bearing, because `lance-io`'s +one `aws` feature bundles the AWS **SDK** (`aws-config`, the broken half) together with +`object_store/aws`, the generic **S3-compatible** backend. Only the first is dropped: the +workspace and `crates/lance-graph` declare `object_store = { features = ["aws"] }` directly, so +unification keeps it on. Measured — `aws-smithy-json` ABSENT, `aws-config` ABSENT, +`object_store` feature `aws` ENABLED, and `--features lance-graph/aws-sdk` brings `aws-config` +back, which is the anti-vacuity check that the switch is real. `lance-graph-hydrate`'s slab +hydration never touches the SDK; it drives `object_store` with `aws_endpoint` + +`aws_virtual_hosted_style_request = false`. Real cost: AWS-native credentials only — IMDS, SSO, +STS assume-role. + +**A past session had already paid for this.** The `object_store` entry in +`crates/lance-graph/Cargo.toml`'s `[dev-dependencies]` declares `object_store/aws` +directly and explains that slimming lance's defaults *"would silently remove +S3 from THIS crate's own S3 callers … it makes the capability this crate USES a thing this crate +ASKS FOR."* That defensive declaration is the entire reason today's change is safe. Worth +noticing as a pattern: the comment cost one paragraph then and saved a capability now. + +**2. "Check upstream" produced a root cause, not a patch.** This repo is a fork of +`lance-format/lance-graph`. Upstream is fully green and has NOTHING to port — no aws-smithy pin, +patch or workaround anywhere in its tree. It is green because it **tracks a `Cargo.lock`** +(`aws-smithy-json 0.61.5` / `aws-smithy-types 1.3.2`) and sits on `lance 1.0.1` / +`object_store 0.12.4`, ten majors back. So its immunity is precisely the mechanism we removed in +`ISS-STALE-AUTHORITY-LOCKS-RESIDUE` (2026-09-04): with no tracked lock, every CI run re-resolves +and can pick up a crate published minutes earlier. **The blocker was a symptom of the +no-lock ruling**, which is a third remedy nobody has taken and the only one that reverses a +prior decision. + +Scope note, stated rather than buried: this is a repo-wide dependency change riding inside a +feature PR (#1235, quack). It is here because it is what unblocks that PR's CI and the operator +asked for it in that context; it splits out cleanly if a reviewer would rather see it alone. ## 2026-09-15 — PR #1233 merged (`030ad80`): eleven measurement arcs are on `main` — docs and probes only, NO contract inventory delta @@ -76,6 +121,314 @@ census is measured and NOT yet on the board — see the arc entry's *Un-recorded* bullet. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1233. +## 2026-09-15 — #1235 round 2: an independent review re-found the P1, and my own fmt decline was WRONG + +An Opus correctness review ran against the branch while the fix was in flight and found the +same `emit_gated` defect independently — **2,021 wrong answers in 200,000 randomised trees** +against a per-row oracle that never builds a `Program`, smallest counterexample +`And([Not(Plane(2)), And([Cmp(Col(2), EqI32(39)), Plane(2)])])`. It then verified the fix +rather than taking it: 120,000 cases clean with the fix, **2,668 red** with only the pre-fix +gate line restored (the anti-vacuity control), plus 200,000-case fuzz, a 320,000-case sweep +across all eight `Agg`s, and 5,000 `GROUP BY` plans — all clean. Its verdict was HOLD on two +PROSE items, both now fixed. + +**I reversed my own review reply, and this is the uncomfortable half.** I declined CodeRabbit's +`cargo fmt --all` suggestion on the grounds that the scoped per-manifest line "already covers +everything `--all` would", and said I had measured it. CodeRabbit pushed back citing the +documented selection behaviour. **It was right and I was wrong**: mis-formatting +`mask-risc/src/ir.rs` and running the scoped quack line gives **exit 0** while mask-risc's own +line gives **exit 1** on the identical file. My earlier "measurement" must have been a +workspace-wide invocation misattributed to the scoped one — the same class of error as +measuring the wrong function. + +The CONCLUSION survives for a different reason than the one I gave: this fmt job is **per-crate +by design** (its own comments say so — `causal-edge` is workspace-excluded, so `--all` would +never reach it), and `mask-risc` has its own line directly above quack's. But checking the job +against the member list instead of reading it found the real defect the reviewer was circling: +**four workspace members had no rustfmt line at all** — `lance-graph-contract` (which THIS +branch edits), `lance-graph-callcenter`, `lance-graph-supervisor`, `bgz-tensor`. All four +measured clean and are now armed. A per-crate gate omits whatever nobody added a line for, and +nothing was checking that. + +**Fixed this round:** the three doc sites that still called `hoist_gate_subset`'s rotation a +"correctness requirement" and a "silent WRONG ANSWER" — true before the fix, false after, and +the reviewer disable-verified it (rotation removed, 120,000 cases, 65,919 planed, zero +divergences, on the harness ARM3 proves can see this regression). What the rotation still buys +is slot economy, 1 slot vs 2. Also the probe's `best/worst infx` print, which read +`spread 0.00 percentage points, best/worst infx` on the two regimes that skip nothing in any +order — self-contradictory, and those rows are cited in `and_by_skip`'s own doc table. + +**And a test I re-pinned this session was half-vacuous.** `assert!(!on_acc)` ran over both +lowerings, but `assign_slots` never emits a `Scratch` gate, so on the FUSED arm no input can +make it fail. It is now scoped to the in-place arm, where restoring the pre-fix line turns it +red, with the fused arm asserting the structural fact instead. One `assert!` over two arms +reads as twice the evidence and is once. + +**Recorded, not fixed** (`ISSUES.md`, two new entries): `and_by_skip`'s ordering lever is now +**inert on any conjunction carrying a plane** — the crate's own headline shape — which is a +genuine cost the P1 fix created and the right trade anyway (a lost optimisation beats a lost +row); and two caller-controlled pre-execution costs, `lower_fused` at ~7.6× per doubling +(`IN(2048)` = 1 443 ms against `lower`'s 0.13 ms, mechanism in another crate's +`distinct_leaves`) and a deep `Filter` **aborting the process** at depth 20,000, which also +makes `LowerError::TooManySlots` unreachable on the in-place path. + +**Also fixed:** the one review finding that never appeared inline. A §8a summary line said "the +signal is dead words rather than selectivity"; the probe permutes terms and measures skip and +never ranks by dead words against a selectivity ranking, so only the negative half is measured +(`selective` and `clustered` have near-identical survivor counts, 36 and 31, and differ by +94.24 points). The sort-key half is untested and §8a and §9 already said so — that one summary +line did not. + +## 2026-09-15 — PR #1235 review round: a codex P1 that was a real wrong answer, and three accepted doc/CI fixes + +Seven review threads on `claude/clone-repositories-71a5sw`. One was a genuine correctness +bug; two reviewer suggestions were DECLINED on measurement; three were accepted and applied. + +**The P1 (fixed).** `lance-graph-quack` `emit_gated` preferred the accumulator gate over the +caller-supplied plane (`acc_gate.or_else(|| under…)`). On `P1 AND (Plane(focus) AND P2)` the +outer accumulator — which has never seen the inner plane — won, and `FOCUS` never reached the +emitted program. Reproduced against the reference oracle: **oracle 29, emitted 204**. Fixed by +inverting the preference so the plane always wins (conservative: can cost skip, can never drop +a conjunct). One existing test asserted the buggy behaviour and is inverted in place with a ⊘ +note; `a_nested_plane_survives_an_outer_accumulator` is the permanent two-sided regression. +Full statement: `EPIPHANIES.md` +`E-THE-ACCUMULATOR-GATE-OUTRANKED-THE-PLANE-AND-SILENTLY-DROPPED-IT-1`. + +**Accepted.** `.claude/audits/nars-34-substrate-audit.md` item 1 contradicted its own table two +sections up ("the honest `Datapath` set" for five recipes the table classifies **4 `Gate` + 1 +`Control`, zero `Datapath`**) — corrected in place, plus the stale "as a dispatch key" phrase in +the closing section, which re-asserted exactly what the file's own top correction retracts. +`STATUS_BOARD.md` D-QCK-6's unescaped `|` inside `COUNT(alpha & ((A&B)|C))` split the row into a +sixth cell — escaped. `style.yml`'s quack clippy line gained `--all-features`. + +**Declined, both on measurement rather than preference.** (1) `cargo fmt --all` in place of the +per-manifest quack line: mis-formatting `crates/lance-graph-mask-risc/src/ir.rs` and re-running +shows the existing scoped line already reports it, so `--all` widens the blast radius without +adding coverage. (2) Reordering the new `EPIPHANIES.md` entry below the existing 09-15 block: +that file is reverse-chronological, and within a date the later entry goes first. + +**CI `linux-build` is red on this PR and the failure is the BASE's, not the branch's.** `main` +at `030ad80` fails identically. Root cause established rather than guessed: `aws-smithy-types +1.7.0` published 18 minutes after the last green run; `aws-smithy-json 0.63.0` declares +`^1.6.1`, so it always resolves the new one; the newest `aws-config 1.12.0` requires +`^0.63.0` and cannot reach the fixed `0.64.0`. **No resolver-only fix exists**, and none of the +four pinnable coordinates (lance / lancedb / arrow / datafusion) is in that chain — so nothing +in this PR's scope can clear it. Tracked as a repo-wide blocker, not a PR defect. + +## 2026-09-14 (6) — the convergence, answered: pin them equal, do not delegate + +The operator's hypothesis was *"wiring duckdb through lance-graph-java might +help both sides to converge."* It paid twice, and the second payment settles +what "converge" should mean here. + +**First payment (already landed, `0bc5e8d`):** quack gained the accumulator +gate `lgj-abi`'s `plan_lower` already had. That was a capability transfer in +one direction, and it exposed a real correctness bug (`hoist_gate_subset`) +that neither crate's own tests could have found, because quack had no +accumulator to be wrong about. + +**Second payment (lance-graph-java `8ad1a1b` + `e9bf3aa`):** the two +lowerings are now pinned equal by a differential — +`native/lgj-abi/src/exports/tests/lowering_convergence.rs`, 3 tests, 5 +disables red. + +### The verdict: shared LAW, not a shared DEPENDENCY + +`lgj-abi` could retire `plan_lower` and delegate to `quack::lower`. It does +not, and `lance-graph-quack` is wired as a **dev**-dependency to make that +impossible by construction. The membrane — the `cdylib` Java's `Linker` +loads — must not depend on a CONSUMER of the IR it serves; that dependency +points the wrong way. What the two share is the law, and the differential is +what keeps "one law, two implementations" a checkable statement instead of a +comment. + +The three rules they share are not equally placed, and the differential made +that legible: + +- **the accumulator gate** and **the AND/OR asymmetry** are facts about + `MaskOp::Pred { under }` — IR facts, which is why both lowerings must have + them and why a future shared helper would belong in `mask-risc` (which + already hosts `fuse`/`BoolExpr`) rather than in either consumer; +- **the prefix rewrite** is NOT an IR fact. It is a property of the + all-ones-seeded FOLD, and in tree form it is not a rule at all: + `all_ones | p == all_ones` means an OR before the first AND never becomes + a node. `plan_lower` scans for the least AND index; the tree reading gets + the same answer for free. A shared helper would have had to carry it as a + special case for one caller. + +So the duplication is smaller than it looked: one genuinely shared pair of +IR facts, and one artifact of `lgj-abi`'s own input shape. Not enough to +justify a helper today — pinned instead, and the pin is what will tell us if +that judgement ever stops holding. + +### Two dead fixtures, and one of them made a real defect invisible + +Both anti-vacuity bounds arrived as FLOORS (`>= 15` of 28, `>= 7` of 9) and +both passed **exactly at their bound**, which is what a floor looks like +when the fixture is dead. The values lane is `-150..=361`; the arity-4 arm +appended `LT_I32(500)`, an always-true op, and the whole 16-vector arm +collapsed to eight saturated 1000s plus eight verbatim copies of the +arity-3 row. `LT_I32(200)` took it 15 → 21. + +The sharper half: `LT_I32`/`LE_I32` carried `500` in the per-opcode arm too. +Measured, mis-mapping `LGJ_OP_LE_I32` to `Pred::LtI32` in `plan_lower` — one +token, and precisely the defect class the file exists to catch — is **red at +operand 300** (866 vs 865, one row) and **green at 500**, where both +readings select all 1000 and the two arms agree on an answer neither +computed correctly. Two of nine opcodes would have shipped untested while +the file read as covering all nine. Both bounds are `assert_eq!` now, with +the incident recorded at the assertion. + +## 2026-09-14 (5) — quack A1: the accumulator gate, and the falsifier the matrix asked for + +Two changes in `crates/lance-graph-quack`, one of them a correctness fix the +other one exposed. + +**The accumulator gate (D-QCK-8) — the convergence.** An `AND` gated its +children only under a resident PLANE. `lgj-abi`'s `plan_lower` already gated +each later conjunct under the ACCUMULATOR built so far, and the two lowerings +were implementing different laws for the same algebra. Now quack does both: +the first child establishes the accumulator, every later child is emitted with +`under = Scratch(acc)`. The asymmetry `plan_lower` documents carries over +unchanged and is the whole correctness question — an `OR` must NOT gate its +children on its own accumulator, because `acc | p` depends on `p` exactly +where `acc` is ZERO, which is what a gate under `acc` discards. + +**`hoist_gate_subset` — the bug the gate exposed.** With both gates available +a `Pred` can carry only one `under`, so the accumulator wins (it is strictly +narrower — the plane was folded into it). But the plane DROP decision depends +on the plane still gating the term that implies it, and on `alpha AND focus +AND v<50` the accumulator started as `focus`, which is not a subset of +`alpha` — the plane was elided while nothing constrained it, and rows outside +`alpha` were counted. Fix: rotate a child whose result is a subset of the +plane to the FRONT, so the accumulator starts inside the plane. Found once, +then found again one level down (a gated `AND` nested in a gated `AND` +reproduced it), which is why the helper is called from BOTH arms rather than +inlined in the one that first needed it. + +**A1 (D-QCK-9) — RUN, not ported.** `Filter::and_by_skip` takes the caller's +measured skip score per conjunct and orders the `AND` by it. The matrix's A1 +row was NEEDS FALSIFIER; `examples/adaptive_order_probe.rs` is that falsifier +— 65 536 rows, 5 conjuncts, all 120 permutations, 4 regimes, counting +64-row words a gated `Pred` does not evaluate: + +| regime | survivors | as written | worst | best | spread (best−worst) | +|---|---|---|---|---|---| +| selective | 36 (0.055%) | 5.66% | 5.66% | 80.66% | 75.00 pts, 14.2x | +| moderate | 14 311 (21.8%) | 0.00% | 0.00% | 0.00% | 0 | +| permissive | 61 777 (94.3%) | 0.00% | 0.00% | 0.00% | 0 | +| clustered (address prefix) | 31 (0.047%) | 99.90% | 0.00% | 99.90% | 99.90 pts | + +> ⊘ **CORRECTED 2026-09-15.** This table had no `worst` column and silently +> computed spread as `best − as-written`, so the clustered row published a +> spread of **0** — reading as "order does not matter here", the exact inverse +> of that row's whole point. The probe prints `best − worst` and the matrix +> uses that; clustered is **0.00 % → 99.90 %**, the widest spread of the four. + +Verdict: order MOVES the skip fraction, so A1 is **ADAPT, conditionally** — +it is exactly inert in two of the four regimes, and the pre-registered +falsifier's *representative predicate stream* half was never run (filed in the +matrix's §9). Selectivity cannot tell you WHETHER reordering is worth +anything: selective and clustered have near-identical survivor counts (36 vs +31) and differ by 19 points of achievable skip (best against best), because +one conjunct's survivors are contiguous and the other's are scattered. At +21.8% survival with survivors SCATTERED a 64-row word is all-dead with +probability ~1.4e-7, so no ordering can skip anything there. The score is the +CALLER's — the shipped surface builds programs and never evaluates one, so +there is no point at which it could measure a score. + +> ⊘ **THREE CORRECTIONS 2026-09-15, one of them a measured reversal.** +> (a) This read *"the control signal is **dead words, not selectivity**"* — a +> claim about the right SORT KEY. The probe enumerates permutations and +> reports min/max; it never ranks by either signal and never calls +> `and_by_skip`. Only the between-regime diagnostic is supported. +> (b) The inertness condition was stated as density ("at 21.8% survival … by +> arithmetic, not by implementation"); the condition is SCATTERING, the +> arithmetic is Bernoulli-independence and so conditional on this fixture's +> LCG, and the figure is 1.4e-7 not 2e-7. +> (c) **The hill-climb argument was FALSE.** It said "a local search over +> adjacent swaps explores the wrong landscape". Measured, instrumenting the +> probe's own model with the clustered prefix term at each index: +> `[4092, 3069, 2046, 1023, 0]`, adjacent deltas all exactly `−1023` — a +> monotone linear ramp, the friendliest hill-climb landscape there is. The +> real reason it is not ported is narrower: this crate never executes, so +> there is no runtime for a hill-climb to measure. (DuckDB adapts on measured +> RUNTIME seeded from a selectivity heuristic, not on measured selectivity — +> a second wording this entry had wrong.) + +Five disables, all red: gate-never-on-accumulator; an `OR` gating its children +on its own accumulator; `hoist_gate_subset` never rotating; `and_by_skip` +sorting ascending; `and_by_skip` not sorting. 13 tests, clippy `-D warnings` +and `fmt` clean. + +## 2026-09-14 (3) — `lance-graph-quack`: the DuckDB-shaped surface whose operators ARE masking ops + +New workspace member `crates/lance-graph-quack`, one dependency +(`lance-graph-mask-risc`, path). Inventory: `Col`, `Mask`, `Cmp` (9 +comparisons, one per `Pred` that produces a mask from a value lane), `Filter` +(`Cmp`/`Plane`/`And`/`Or`/`Not`, builders `cmp`/`plane`/`and`/`or`/`negate`/ +`in_u32`/`in_i32`), `Agg` (`Count`/`Any`/`All`/`SumI32`/`MinI32`/`MaxI32`/ +`Rows`/`BlendI32`), `Query`, `GroupBy`, `GroupPlan`, `LowerError` +(`EmptyJunction`/`TooManySlots`/`GroupedBlend`), and three lowerings — +`lower` (in-place), `lower_fused` (Boolean skeleton through +`mask_risc::fuse`), `lower_group_by` (two-phase). + +**The crate builds a `Program` and never evaluates one.** `execute` stays the +consumer's call on a scratch the consumer owns; a `match` here that computed +anything would be the duplicate evaluator the arc exists to avoid. No +`ndarray` dep for the same reason — the masking algebra is reached THROUGH +mask-risc, never beside it. + +Three things the scaffold owed and this closes: + +- **`Filter::Plane`** — a resident mask plane read as a predicate. The + scaffold's own doc recorded that "every row" had no honest spelling (no + `Fill` op; the draft's `NeU32{lane:0}` + constant-ternlog trick typechecked + only when lane 0 happened to be `U32`, a latent lane-kind bug). The plane + leaf is correct by construction, costs zero ops and zero slots, and is the + truer model: the table IS its validity plane, which is why there is no NULL + here to be three-valued about. +- **The survivor skip.** An `AND` with a plane child gates every comparison + beneath it (`MaskOp::Pred`'s `under`). Sound for any Boolean remainder + (`g ∧ rest(X) = g ∧ rest(g∧X)`), so the gate passes through `NOT` and `OR`; + what does not pass is the DROP — the plane leaf is elided only where the + gated remainder is identically zero wherever the gate is (a comparison is; + an `AND` if any child is; an `OR` if every child is; a `NOT` never). +- **`GROUP BY` with a mask plane where the hash table would be** — keep the + filter, bind it as a plane, one gated equality per key. K programs, not one, + and the doc says why: `masked_strided_group_sum` exists in the facade but + the IR names no strided operand and no group terminal. + +10 tests at the time of this entry. + +> ⊘ **CORRECTED 2026-09-15.** This read "10 tests, every one differential +> against a per-row oracle that never sees a `Program`". The "every one" was +> never true and the same overstatement reached two other places. Measured on +> the branch as it stands (13 tests): **9 differential**, **4 structural** — +> `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` +> (reads the emitted `ops`), `the_fused_lowering_trades_slots_for_passes` (op +> histogram), `a_wide_conjunction_costs_one_extra_slot_not_one_per_child` +> (slot count), and `an_empty_junction_is_refused_rather_than_folded_to_an_identity` +> (error shape). That is not a weakness — the structural four catch shapes no +> row count would notice — but "all differential" is the wrong summary, and in +> `rust-test.yml` it was the stated JUSTIFICATION for the CI line. + +The 64k vertical slice (`COUNT(alpha & ((A&B)|C))`) agrees across +five readings — oracle, both lowerings on the executor, both on mask-risc's +reference evaluator — with two-sided anti-vacuity (a proper subset of alpha +AND strictly below the ungated remainder, so a dropped gate fails even though +its count would still look plausible). + +Named absent: the join (`src_mask → hop → dst_mask` is mask-risc's PR5; there +is no `hop` op to lower to yet), a one-terminal grouped SUM, and everything +the IR itself excludes (strings, `ORDER BY`, three-valued NULL). + +Disable table: 8 arms, 7 load-bearing first try. The 8th is recorded as a +finding — the empty-junction refusal is spelled at three sites, so disabling +any one leaves the suite green while all three together turn it red. Both the +guard and the test now say so, because the failure mode is a future session +measuring one site, reading green, and deleting a guard as dead. Commits +`18c1d85`, `f1d41c8`. ## 2026-09-14 (2) — PR3 (branch `claude/clone-repositories-71a5sw`): `lance-graph-mask-risc` gains its executor, oracle, fuser and generated dispatch diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 89a1a1ea8..585b7d289 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -13,6 +13,25 @@ them **reverse a verdict published earlier in the same session**. | D-ELK-3 | does depth-rank over the meet reproduce `LensClosure::most_specific` without the pairwise test? | **Done — REVERSED TWICE.** Descending (my sign error) 6.7 %, published as *refuted*; ascending **84.6 %** (min-max) / **85.1 %** (min-sum), 3/3 on the clinical pairs. Then D-ELK-4 re-scoped it: taxonomy-shaped, not lens behaviour | RA/Still's/Felty's all returning `"disease"` (the ROOT — the node farthest from every seed) is what exposed the sign. A ranking that returns the root for every clinical pair is reporting the most GENERAL ancestor | | D-ELK-4 | **generality control** — same arm code on a second graph, so "the lens behaves like X" is separable from "a taxonomy behaves like X" | **Done — DIVERGES 48.6 pp, and the divergence survives both confounds.** MQ 36.0 % vs MONDO 84.6 %. A4 non-vacuous: identical (neither graph has a single `\|I\|==1` pair). A5 braided-only: 47.5 pp — braiding is directional inside MONDO (flat 13/13 vs braided 152/182) but explains almost none of the gap | **PRE-REGISTERED**: "if ASCENDING agrees across both, the result is the LENS's; if it diverges, it was the taxonomy's shape." A4 and A5 are the two confounds that could fake a divergence; a gap surviving both is structural. **Mechanism still unnamed** — see `ISS-ELK-DENSITY-UNISOLATED` | | D-ELK-5 | horizon width sweep — is an empty meet at width `w` one answer or two? | **Done — two, and code returning `∅` for both cannot tell them apart.** MONDO CUT 194→3→0 at `w`=1/7/14 (recall 0.1→84.5→100 %); MQ 178→85→21 (0.3→24.2→64.5 %). True EMPTY is width-invariant: MONDO **5**, MQ **0** | EMPTY must not move with `w` and CUT must — both hold, which is what makes the two causes separable at all. Same missing symbol `E-THE-BOXCAR-HORIZON-IS-NOT-A-DISCOUNT…` argued for from the other side | +## quack-duckdb-surface (D-ids minted 2026-09-14 with the crate) + +`crates/lance-graph-quack` — DuckDB's operator set with every operator lowering +to a `lance-graph-mask-risc` `Program`. The crate builds programs and never +evaluates one; `execute` stays the consumer's call on a scratch it owns. + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| D-QCK-0 | `Filter`/`Cmp`/`Agg`/`Query` + `lower` — predicates, AND/OR/NOT, COUNT/ANY/ALL/SUM/MIN/MAX | Shipped (`30924a8`, extended `18c1d85`) | every shape diffed against a per-row oracle that never sees a `Program`; anti-vacuity `0 < selected < n` per case | +| D-QCK-1 | `Filter::Plane` — a resident mask plane as a predicate; the honest "every row" the scaffold owed | Shipped (`18c1d85`). Zero ops, zero slots; projection of a plane returns the PLANE, not a copy | `a_resident_plane_alone_is_a_zero_op_program` pins ops empty + slots 0 + the terminal's operand | +| D-QCK-2 | the survivor skip — an `AND` with a plane child gates every comparison beneath it (`Pred{under}`); the gate passes through `NOT`/`OR`, the DROP does not | Shipped (`18c1d85`). Disable-verified twice: dropping under a negation, and dropping with no child vanishing, both redden | `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` — can-fire (slice: all gated, alpha implied) AND can-stay-silent (`alpha & !X`: still gated, alpha stays a leaf) | +| D-QCK-3 | `lower_fused` — the Boolean skeleton through `mask_risc::fuse` | Shipped (`18c1d85`). Both lowerings run against the same oracle on every fixture | `the_fused_lowering_trades_slots_for_passes` pins BOTH sides (2 passes/2 slots vs 1 pass/4 slots), so a fuser that stopped fusing and an emitter that started allocating per leaf each fail their own line | +| D-QCK-4 | projection — `Agg::Rows` (`Terminal::Keep`, the mask IS the result) and `Agg::BlendI32` (the CASE shape) | Shipped (`18c1d85`) | the kept mask materialises to exactly the oracle's rows; the blend must actually pick from `els` somewhere | +| D-QCK-5 | `lower_group_by` — two-phase, a mask plane where the hash table would be; K gated equalities over the kept filter | Shipped (`18c1d85`). K programs not one, because the IR names no strided operand and no group terminal — `masked_strided_group_sum` exists in the facade and cannot be reached from here yet | per-group counts and sums equal the oracle's AND sum to the ungrouped count (the partition identity); a grouped blend is refused, not answered wrongly | +| D-QCK-6 | the 64k vertical slice `COUNT(alpha & ((A&B)\|C))` across five arms | Shipped (`18c1d85`) — closes Phase 3+6 | oracle, both lowerings on the executor, both on the reference evaluator, one number; two-sided anti-vacuity (subset of alpha AND strictly below the ungated remainder) | +| D-QCK-7 | the join — `src_mask → hop → dst_mask` | **Blocked** on mask-risc's PR5: there is no `hop` op to lower to | — | +| D-QCK-10 | the lowering convergence — `lgj-abi`'s `plan_lower` (a flat op list) and `quack::lower` (a Boolean tree) pinned equal. **A differential, deliberately NOT a delegation:** quack is a DEV-dependency of `lgj-abi`, because the membrane must not depend on a consumer of the IR it serves | Shipped (lance-graph-java `8ad1a1b` + `e9bf3aa`, `native/lgj-abi/src/exports/tests/lowering_convergence.rs`). Five disables red. Two dead fixtures found: `LT_I32(500)` on a `-150..=361` lane made the whole 16-vector arity-4 arm inert (15 → 21 of 28) and made a real `LE_I32 → LtI32` mis-map INVISIBLE (measured: red at operand 300, green at 500) | `the_two_lowerings_agree_on_every_combine_vector` (28 vectors, `assert_eq!(non_degenerate, 21)`), `the_all_rows_shortcut_is_the_same_condition_on_both_sides` (two-sided: 3 `AllRows`, 25 not), `every_opcode_maps_to_the_same_predicate_in_both_lowerings` (`assert_eq!(proper_subset, 9)` + the TCAM half-swap) | +| D-QCK-8 | accumulator gating — an `AND` gates each later child under the accumulator built so far, not only under a resident plane. The same rule `lgj-abi`'s `plan_lower` already had; the two lowerings now implement one law | Shipped. Disable-verified 3×: gate-never-on-accumulator, an `OR` gating its children on its own accumulator, and `hoist_gate_subset` never rotating — each reddens | `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` via the new `gate_shape` helper (all-gated / on-accumulator / dropped, three facts not two); `every_lowered_filter_agrees_with_an_independent_per_row_reading` is what catches the DROP going unsound | +| D-QCK-9 | A1 `Filter::and_by_skip` — the caller's measured skip score orders an `AND`'s conjuncts. DuckDB's hill-climb NOT ported (see the corrected reason) | Shipped, with the matrix's A1 falsifier RUN (`examples/adaptive_order_probe.rs`, 65 536 rows × 5 conjuncts × all 120 permutations × 4 regimes). Order moves the skip fraction, so A1 is **ADAPT conditionally** — inert in 2 of the 4 regimes, and the pre-registered *representative predicate stream* half was never run (matrix §9). ⊘ **Corrected 2026-09-15:** this row read "the control signal is DEAD WORDS, not selectivity" — the probe never ranks by either signal and never calls `and_by_skip`, so only the between-regime diagnostic is supported; and the "hill-climb explores the wrong landscape" reason was MEASURED FALSE (`[4092, 3069, 2046, 1023, 0]`, deltas all −1023: a monotone ramp). The real reason is that this crate never executes | `skip_ordering_moves_the_work_and_never_the_answer`: the reordering agrees with the oracle (not with the other lowering), the emitted program's first predicate is the highest-scored, and descending/no-sort disables both redden. **The probe itself is run by no gate** — `cargo test` compiles an example and never executes it — so the four-row table is a measured-once observation, not a pinned one | ## mask-risc-executor (PR3 — D-ids minted 2026-09-14, `.claude/plans/mask-risc-executor-v1.md`) diff --git a/.claude/harvest/duckdb-headers/README.md b/.claude/harvest/duckdb-headers/README.md new file mode 100644 index 000000000..6e02048e5 --- /dev/null +++ b/.claude/harvest/duckdb-headers/README.md @@ -0,0 +1,158 @@ +# DuckDB HEADER harvest — the repair of the matrix's §6 + +The `ruff_cpp_spo` harvest behind `duckdb-to-v3-translation-matrix-v1.md` was +pointed at 22 `.cpp` translation units. Its own §6 records the result honestly: +**seven TUs harvested to 100 % `Empty`**, because DuckDB's execution is +template-dispatched and lives in HEADERS — so "no row in this matrix cites a +harvest TSV as evidence", and §6 names the fix: point it at the headers. + +Done, 2026-09-14, same harvester (`ruff/examples/harvest_events`, libclang 18), +same `args.txt`. Per-header counts: + +| header | methods | events | +|---|---|---| +| `common_operator_comparison_operators` | 5 | 39 | +| `common_operator_constant_operators` | 0 | 0 | +| `common_row_operations_row_matcher` | 0 | 0 | +| `common_types_selection_vector` | 26 | 310 | +| `common_types_validity_mask` | 61 | 836 | +| `common_types_vector` | 6 | 42 | +| `common_vector_operations_scalar_executor` | 10 | 286 | +| `common_vector_unified_vector_format` | 1 | 6 | +| `execution_expression_executor_state` | 2 | 11 | +| `execution_ht_entry` | 12 | 92 | + +**123 methods, 1,622 events** where the `.cpp` pass yielded none for the same +concepts. + +> ⊘ **HONESTY NOTE, added 2026-09-15 after review.** Two things this file +> claimed by implication and does not deliver. +> +> **The numbers are not re-derivable from this repo.** The TSV is not +> committed, there is no `args.txt` here, and no DuckDB source is in this tree +> — a repo-wide grep finds `harvest_events` and `args.txt` only inside this +> README. Treat the counts as a recorded observation of a run that happened +> against a local checkout, not as evidence a reader can verify. +> +> **§6's complaint is therefore NOT discharged.** §6 says *"no row in this +> matrix cites a harvest TSV as evidence."* That is still true: what exists +> now is a README quoting counts from a TSV that is absent. What the harvest +> genuinely changed in the consumer crate is two things — `Cmp::MatchU64` and +> the range-write note — and those stand on their own. +> +> **The headline and the inventory disagree.** The per-header table sums to +> 123 and the method inventory below lists 92 distinct names; the difference +> is overloads. Read 123 as definitions, 92 as names. Two headers yield nothing and that is information too: +`row_matcher.hpp` and `constant_operators.hpp` are pure declarations whose +bodies are templates no TU instantiates here — the .cpp pass already covered +`row_matcher.cpp`, which did produce events. + +## The two findings that changed code + +1. **`TemplatedValidityMask::SetRangeInvalid`** — DuckDB's own bit-plane, the + same packed-`u64` carrier V3 uses (matrix R6), carries a RANGE write. The + matrix files `mask_set_range` as T1 gap **G6** on the strength of V3's own + trie-reveal measurement; this is the same operation on the other side, and + it is why G6 is a real primitive rather than a wish. `lance-graph-quack` + now spells the address-PREFIX predicate and says plainly that it lowers to + a sweep, not a range write, until G6 lands. +2. **`ScalarExecutor::{Runtime,Static}SelectionSink`** — `Append`, + `AppendInvalidRange`, `FillConstant`, `Result`. This is matrix row **C2** + ("the sink", ELIMINATE: the mask word IS the sink), and it was entirely + invisible to the `.cpp` pass. + +## Method inventory +``` +ConsecutiveChildListInfo.ConsecutiveChildListInfo +Equals.Operation +ExecuteFunctionState.GetFunctionState +ExpressionState.~ExpressionState +GreaterThan.Operation +ScalarExecutor::RuntimeSelectionSink.Append +ScalarExecutor::RuntimeSelectionSink.AppendInvalidRange +ScalarExecutor::RuntimeSelectionSink.FillConstant +ScalarExecutor::RuntimeSelectionSink.Result +ScalarExecutor::RuntimeSelectionSink.RuntimeSelectionSink +ScalarExecutor::StaticSelectionSink.Append +ScalarExecutor::StaticSelectionSink.AppendInvalidRange +ScalarExecutor::StaticSelectionSink.FillConstant +ScalarExecutor::StaticSelectionSink.Result +ScalarExecutor::StaticSelectionSink.StaticSelectionSink +SelectionVector.Capacity +SelectionVector.Incremental +SelectionVector.Initialize +SelectionVector.Inverted +SelectionVector.IsSet +SelectionVector.SelectionVector +SelectionVector.data +SelectionVector.get_index +SelectionVector.get_index_unsafe +SelectionVector.operator= +SelectionVector.operator[] +SelectionVector.sel_data +SelectionVector.set_index +SelectionVector.swap +TemplatedValidityData.EntryCount +TemplatedValidityData.TemplatedValidityData +TemplatedValidityMask.AllValid +TemplatedValidityMask.CanHaveNull +TemplatedValidityMask.CannotHaveNull +TemplatedValidityMask.CheckAllInvalid +TemplatedValidityMask.CheckAllValid +TemplatedValidityMask.Copy +TemplatedValidityMask.CountValid +TemplatedValidityMask.EnsureWritable +TemplatedValidityMask.EntryCount +TemplatedValidityMask.EntryWithValidBits +TemplatedValidityMask.GetAllocationSize +TemplatedValidityMask.GetData +TemplatedValidityMask.GetEntryIndex +TemplatedValidityMask.GetValidityEntry +TemplatedValidityMask.GetValidityEntryUnsafe +TemplatedValidityMask.Initialize +TemplatedValidityMask.IsMaskSet +TemplatedValidityMask.NoneValid +TemplatedValidityMask.Reset +TemplatedValidityMask.RowIsValid +TemplatedValidityMask.RowIsValidUnsafe +TemplatedValidityMask.Set +TemplatedValidityMask.SetAllInvalid +TemplatedValidityMask.SetAllValid +TemplatedValidityMask.SetInvalid +TemplatedValidityMask.SetInvalidUnsafe +TemplatedValidityMask.SetRangeInvalid +TemplatedValidityMask.SetValid +TemplatedValidityMask.SetValidUnsafe +TemplatedValidityMask.SizeInBytes +TemplatedValidityMask.TemplatedValidityMask +TemplatedValidityMask.ValidityMaskSize +ValidityArray.AllValid +ValidityArray.CanHaveNull +ValidityArray.CannotHaveNull +ValidityArray.Capacity +ValidityArray.Initialize +ValidityArray.InitializeEmpty +ValidityArray.Pack +ValidityArray.RowIsValid +ValidityArray.RowIsValidUnsafe +ValidityArray.SetValid +ValidityArray.SetValidUnsafe +ValidityArray.ValidityArray +ValidityMask.ValidityMask +Vector.Buffer +Vector.BufferMutable +Vector.GetBufferRef +Vector.GetType +Vector.GetVectorType +Vector.SetBuffer +duckdb.IncrementAndWrap +ht_entry_t.ExtractSalt +ht_entry_t.GetPointer +ht_entry_t.GetPointerOrNull +ht_entry_t.GetSalt +ht_entry_t.GetSaltWithNulls +ht_entry_t.IsOccupied +ht_entry_t.SetPointer +ht_entry_t.SetSalt +ht_entry_t.ht_entry_t +``` diff --git a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md index fc74fb0b3..9c912529b 100644 --- a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md +++ b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md @@ -145,7 +145,7 @@ Verdict vocabulary, used strictly: | # | DuckDB concept | verdict | DuckDB file:line | V3 / ndarray counterpart | mechanism | FALSIFIER | |---|---|---|---|---|---|---| -| A1 | `AdaptiveFilter` — runtime permutation of conjunction terms by measured selectivity | **NEEDS FALSIFIER — the one row where DuckDB has something V3 does not** | `adaptive_filter.cpp:17-29` (ctor), `:113-186` (`AdaptRuntimeStatistics`); the swap `:127,163`; likeliness decay `:132-134`; intervals `observe=10 / execute=20 / warmup=5` (`:17,31,180`) | **absent** — `Program.ops` is a fixed `Vec` (`ir.rs:133`) executed in order, with no reordering and no measurement | An adjacent-transposition hill-climb: swap two neighbouring terms, measure 10 iterations, keep if mean runtime dropped else revert and **halve** that position's swap likeliness (floor 1, so exploration never dies). `GetInitialOrder` seeds from the optimizer's static heuristic. V3 has no analog at any layer. | **The falsifier is prior to the port, and it may kill the whole idea.** DuckDB's reordering pays because term k runs only on survivors of 1..k−1, so a selective term first *shrinks the input*. In V3 a predicate sweep costs the **full column** regardless of position (`gt_i32_to_mask` writes every word; D-GTM-0m: 22.4–22.8 µs, survivor-independent) — so **reordering saves nothing on generation**. It can only save by *avoidance*: `mask_any` says empty, skip the rest (E5), or `MaskOp::Pred { under }` skips chunks (`ir.rs:84-86`). **Measure:** on a representative predicate stream, what fraction of `Pred` ops are skippable by `under`, and does term order change that fraction? If order does not move the skip fraction, A1 is ELIMINATE and the machinery must not be ported. If it does, the port is a reordering of *skip opportunities*, not of costs — a different algorithm than DuckDB's, and its swap-likeliness decay is not obviously the right control law for it. | +| A1 | `AdaptiveFilter` — runtime permutation of conjunction terms by measured selectivity | **⊘ FALSIFIER RUN 2026-09-14 → ADAPT (not ELIMINATE), and the control law is DIFFERENT — see §8a. Was: NEEDS FALSIFIER — the one row where DuckDB has something V3 does not** | `adaptive_filter.cpp:17-29` (ctor), `:113-186` (`AdaptRuntimeStatistics`); the swap `:127,163`; likeliness decay `:132-134`; intervals `observe=10 / execute=20 / warmup=5` (`:17,31,180`) | **absent** — `Program.ops` is a fixed `Vec` (`ir.rs:133`) executed in order, with no reordering and no measurement | An adjacent-transposition hill-climb: swap two neighbouring terms, measure 10 iterations, keep if mean runtime dropped else revert and **halve** that position's swap likeliness (floor 1, so exploration never dies). `GetInitialOrder` seeds from the optimizer's static heuristic. V3 has no analog at any layer. | **The falsifier is prior to the port, and it may kill the whole idea.** DuckDB's reordering pays because term k runs only on survivors of 1..k−1, so a selective term first *shrinks the input*. In V3 a predicate sweep costs the **full column** regardless of position (`gt_i32_to_mask` writes every word; D-GTM-0m: 22.4–22.8 µs, survivor-independent) — so **reordering saves nothing on generation**. It can only save by *avoidance*: `mask_any` says empty, skip the rest (E5), or `MaskOp::Pred { under }` skips chunks (`ir.rs:84-86`). **Measure:** on a representative predicate stream, what fraction of `Pred` ops are skippable by `under`, and does term order change that fraction? If order does not move the skip fraction, A1 is ELIMINATE and the machinery must not be ported. If it does, the port is a reordering of *skip opportunities*, not of costs — a different algorithm than DuckDB's, and its swap-likeliness decay is not obviously the right control law for it. | | A2 | `VectorHash` / `CombineHash` (`TightLoopHash`, `CombineHashScalar`) | **NEEDS FALSIFIER** | `vector_hash.cpp:50-67` (`TightLoopHash`); `:43-47` (`CombineHashScalar`); NULL_HASH `:24` | **absent** from `simd_masking_ops` by design (a hash is a value, not a mask) | Hashing is not a mask operation and does not belong in T1. The open question is whether V3 needs hashing **at all** on the addressed path: a minted classid prefix is already a *semantic* bucket, so the hash's job (map a key to a bucket) is done by the address. | Where does V3 need a hash that the address does not already give? Candidate: joining on a **non-address column**. Enumerate real consumers; if every join key is a minted address, A2 is ELIMINATE for this substrate and hashing stays outside T1 entirely. If a non-address join key exists, A2 is a real gap but belongs to a value-kernel family, **not** to the mask vocabulary — do not let it widen `simd_masking_ops`. | | A3 | `JoinHashTable` — probe, salt prefilter, linear-probe chains | **V3 BETTER for the addressed case; NEEDS FALSIFIER otherwise** | `join_hashtable.cpp:248-296` (`ProbeForPointersInternal`); `ht_entry.hpp:34-37` (salt/pointer split), `:49-51` (`IsOccupied`) | prefix range on the minted address; `ternary_match_u32_to_mask` with a care mask over the prefix (`simd_masking_ops.rs:1285`) | `ht_entry_t` packs **16 bits of salt + 48 bits of pointer** in one u64 and prefilters on the salt to avoid a full key compare (`:271-277`). That is a *probabilistic* prefix derived from a hash. **The V3 classid prefix is the real thing**: matching it is not a filter that may be wrong, it is a **contiguous row range** (R5, D-GTM-0m: 49–99 ns vs 22.4 µs). Also note `IsOccupied() == (value != 0)` (`ht_entry.hpp:50`) — DuckDB independently arrived at zero-is-absence, the same convention as `CLAUDE.md:1657`'s ladder. | The comparison only holds when **both sides are minted into the same address space**. A join between a minted V3 population and an external, unminted key set has no shared prefix and falls back to A2's hash question. Falsifier: exhibit the intended join workload. If either side is unminted, "V3 BETTER" is false for it and the honest verdict is NEEDS FALSIFIER. Second: `join_hashtable.cpp` is 6,986 harvested events, by far the largest TU read — this row is a **reading of two functions, not of the join**; spilling, radix partitioning, and chain building are not assessed. | | A4 | `RowMatcher` (multi-column match by sequential in-place sel compaction) | **ADAPT — the single cleanest mapping in the matrix** | `row_matcher.cpp:19-62` (`TemplatedMatchLoop`); the in-place compaction `:56`; the 4-way validity specialization `:64-89`; `row_matcher.hpp:47` (`Match`) | `ternary_match_strided_to_mask(bytes, first_offset, stride_bytes, count, &[u8;12] pattern, &[u8;12] care, out)` (`simd_masking_ops.rs:1415-1424`) | DuckDB matches **one key column at a time**, narrowing `sel` in place each round (`sel.set_index(match_count++, idx)`, `:56`) — k columns, k passes, k compactions. The V3 12-byte register **is** the multi-column key, and the `care` mask names which rails participate, so k columns become **one pass** with no compaction. The strided form is shaped for exactly the V3 facet: `stride_bytes = 16`, `first_offset = 4`, pattern/care = `[u8;12]` — the 4+12 atom of `le-contract.md` §1. | **Scope, and it is narrow.** `ternary_match` is *equality with don't-cares only*. `RowMatcher` dispatches a per-column `ExpressionType` predicate (`row_matcher.hpp:52-56`) and NULL-semantics variants (DISTINCT FROM). So A4 covers the **equi-match** case and nothing else. Falsifier: take a real multi-column match from a V3 consumer; if any column needs an ordered or distinct-from predicate, the one-pass claim fails for it and it degrades to per-column masks + `mask_and` — still allocation-free, but k passes, not one. | @@ -526,14 +526,14 @@ rounding error dressed as a finding.* | verdict | whole rows | split halves | rows | |---|---|---|---| -| **ADAPT** | **14** | — | R5, R7, R8, E1, E2, E4, E5, E7, E10, E11, C1, C3, C5, A4 | +| **ADAPT** | **15** | — | R5, R7, R8, E1, E2, E4, E5, E7, E10, E11, C1, C3, C5, A4, **A1** (moved 2026-09-14 — **conditional**, see §8a: inert in 2 of the 4 regimes, and the pre-registered *representative predicate stream* half was never run) | | **ELIMINATE** | **8** | **+2** | R1, R3, R4, E3, E8, E9, C2, C4 · *plus* R6(the NULL **role**), A5(composition) | | **V3 BETTER** | **3** | **+1** | E6, C6, C7 · *plus* A3(addressed case) | | **KEEP** | **2** | **+2** | R2, C8 · *plus* R6(the mask **representation**), A5(egress) | -| **NEEDS FALSIFIER** | **2** | **+1** | A1, A2 · *plus* A3(unaddressed case) | +| **NEEDS FALSIFIER** | **1** | **+1** | A2 · *plus* A3(unaddressed case) — ⊘ A1 left this column 2026-09-14 | | | **29** | **+6** | = 35 entries / 32 rows | -Read the shape rather than the totals: **ADAPT dominates (14/32)**, which is the +Read the shape rather than the totals: **ADAPT dominates (15/32)**, which is the non-obvious result. The tempting summary — "DuckDB's machinery is an artifact of index lists, delete it" — is contradicted by the matrix's own count: only 8 rows eliminate cleanly, and they are concentrated in *representation* (R1, R3, R4) and @@ -543,6 +543,18 @@ does not. And exactly **one** DuckDB mechanism has no V3 counterpart at all (A1, `AdaptiveFilter`) — which is why its falsifier is one of the three that gate Phase 2. +> **⊘ A1 UPDATE 2026-09-14 — the falsifier ran (§8a) and moved the row to +> ADAPT, which strengthens the shape above rather than dulling it.** Term order +> does change the skip fraction (up to 99.90 percentage points in the clustered +> regime; 75.00 points and 14.2× are the SELECTIVE regime's spread), so the +> mechanism is not deleted — but what transferred is the INTENT (order the +> conjuncts) and what did not is the CARRIER (an adjacent-transposition +> hill-climb over measured runtimes). That is the same intent-transfers / +> carrier-does-not pattern the paragraph above describes for representation, +> arriving from a third direction: the optimised quantity here is a step +> function of clustering, so the control law has to change even though the +> goal does not. + **Phase 2's ordering falls out of the falsifiers, not from the verdict counts.** Three measurements gate the largest number of downstream rows and should run before any design: @@ -559,10 +571,22 @@ before any design: 75 µs M1b generation. The substrate's native width is the byte (`le-contract.md` §3); i32 is the foreign one, and D-GTM-0m's numbers are explicitly **upper bounds** because of it. -3. **A1's prior question** — does term order change the *skip fraction*? If not, +3. ~~**A1's prior question** — does term order change the *skip fraction*? If not, `AdaptiveFilter` is ELIMINATE and the only DuckDB mechanism V3 lacks turns out not to be needed. That is a cheap answer with a large consequence, and it is - answered by measurement (1) plus one sweep. + answered by measurement (1) plus one sweep.~~ **ANSWERED 2026-09-14 — see + §8a.** Order moves it (up to 99.90 percentage points, clustered — 75.00 + points and 14.2× are the selective regime), so A1 is not + ELIMINATE; selectivity alone does **not** predict skip — `selective` and + `clustered` have near-identical survivor counts (36 and 31) and differ by + 94.24 points of as-written skip — and the hill-climb is not ported. + ⊘ This read "the signal is dead words rather than selectivity", which + overstates the probe: it permutes terms and measures skip, and never + ranks by dead words against a selectivity ranking, so the SORT-KEY half + is untested (§8a and §9 already say so; this summary line did not). + Note this did NOT need measurement (1): the skip + fraction is a count of avoided word-evaluations, not a timing, so it was + answerable on its own. **What Phase 2 must NOT do**: build G5 (masked compaction) before A5's egress count exists — it is the most expensive gap on the list (five backends, BACKEND @@ -571,8 +595,137 @@ counted. --- +## §8a — A1's falsifier, RUN (2026-09-14) + +The queue item above asks one question: *does term order change the skip +fraction?* It does. `crates/lance-graph-quack/examples/adaptive_order_probe.rs` +is the measurement — 65 536 rows (1024 words), five conjuncts, **all 120 +permutations**, four regimes, counting the 64-row words a gated +`MaskOp::Pred { under }` does not evaluate, summed over the 4096 gated +positions of one ordering (term 0 is the ungated seed). + +| regime | survivors | as written | worst | best | spread | +|---|---|---|---|---|---| +| selective (a long tail of survivors) | 36 (0.055 %) | 5.66 % | 5.66 % | **80.66 %** | 75.00 pts, 14.2× | +| moderate | 14 311 (21.8 %) | 0.00 % | 0.00 % | 0.00 % | 0 | +| permissive | 61 777 (94.3 %) | 0.00 % | 0.00 % | 0.00 % | 0 | +| clustered (one conjunct is an ADDRESS PREFIX) | 31 (0.047 %) | **99.90 %** | 0.00 % | 99.90 % | 99.90 pts | + +**Verdict: ADAPT, not ELIMINATE — but not DuckDB's algorithm.** Three findings, +each of which changes what should be built: + +1. **The knob is inert wherever survivors are SCATTERED, whatever the + density.** At 21.8 % survival with independent placement a 64-row word is + all-dead with probability `0.78163⁶⁴ ≈ 1.4·10⁻⁷`, so *no* ordering skips + anything and the best and worst permutations are identical. Reordering is + not "a small win" in the moderate and permissive regimes; it is exactly + zero. Any cost model that spends on ordering there is spending on nothing. + + > ⊘ **CORRECTED 2026-09-15.** This read *"inert wherever the population is + > not sparse, by arithmetic rather than by implementation"*, and both + > halves were wrong. **Density is not the condition** — finding 2 below + > says so two paragraphs later, and a dense-but-contiguous population has + > plenty of dead words. The condition is scattering. And the arithmetic is + > a Bernoulli-INDEPENDENCE computation, so it is conditional on a + > uniformity that is a property of this fixture's LCG, not of the + > substrate — "by arithmetic, not by implementation" claimed a hardness it + > does not have. The figure was also stated as `2·10⁻⁷`; recomputed, it is + > `1.419·10⁻⁷`. + +2. **Selectivity cannot tell you WHETHER reordering is worth anything.** The + selective and the clustered regimes have almost identical survivor counts — + 36 and 31 — and differ by **19 percentage points of achievable skip** + (best-vs-best: 80.66 % against 99.90 %), because one conjunct's survivors + are contiguous and the other's are scattered. A selectivity-only cost model + cannot separate those two cases. This is also why V3 has the lever at all: + an address prefix selects a contiguous subtree (`Filter::prefix_u64`, + R5/R8), which is the clustered row. + + > ⊘ **CORRECTED 2026-09-15, twice.** (a) The 19 points were attributed *as + > written*; as written the two regimes differ by **94.24** points (5.66 vs + > 99.90) and 19.24 is the best-vs-best figure. The italic made the sentence + > false against its own table. (b) The heading read *"the control signal is + > DEAD WORDS, not selectivity"* and closed *"rank by selectivity and those + > two look the same; rank by dead words and they do not"* — a claim about + > the right SORT KEY. **That is not what was measured.** The probe + > enumerates all 120 permutations and reports min/max; it never computes a + > selectivity-ranked order, never computes a dead-word-ranked order, never + > compares two ranking rules, and never calls `and_by_skip`. What the + > evidence supports is the diagnostic above — a between-regime statement — + > not a between-orderings one. Whether dead words is also the better sort + > key is **untested**. + +3. **DuckDB's adjacent-transposition hill-climb is not ported — but not + because its search would fail.** ⊘ **CORRECTED 2026-09-15: the original + argument here was measured FALSE.** It read: *"the optimised quantity is a + step function of clustering … so a local search over adjacent swaps is + exploring the wrong landscape. In the clustered regime … a hill-climb + starting from the worst has no adjacent swap that improves anything until + it happens to move the prefix term to the front."* + + Measured, instrumenting the probe's own `skipped_words` model with the + prefix term at each index: **`[4092, 3069, 2046, 1023, 0]`, adjacent deltas + all exactly `−1023`.** The prefix term's mask is one live word of 1024, and + `skipped_words` charges `dead_words(acc)` once per gated position, so skip + is `(4 − p) · 1023` in the term's index `p` — a monotone linear ramp. Every + single forward adjacent swap improves it by the same amount. That is the + *most* hill-climb-friendly landscape possible, not the wrong one, and the + quoted sentence is the exact opposite of the truth. + + The step-like behaviour is BETWEEN the four regimes (does this conjunction + contain a clustered term at all); the search space is WITHIN one + permutation set, where the objective is the ramp above. The original + reasoned from the first to the second. + + **The real reason it is not ported** is narrower and is not about search + quality: this crate never executes, so there is no runtime for a hill-climb + to measure. Whether a consumer that DOES execute should run one is + **unanswered here** — and, per the note on `[H]` below, DuckDB's own loop + adapts on measured RUNTIME rather than on selectivity, so a comparison + would have to be against that. + +**What was built instead:** `Filter::and_by_skip(impl IntoIterator)` — the caller supplies a measured skip score per conjunct from a +previous execution and the builder orders the `AND` by it, descending, stably. +No measurement, no decay, no intervals, no `observe=10 / execute=20 / warmup=5`. +The crate builds programs and never evaluates one, so it cannot measure +anything; putting the score at the boundary is the whole adaptation. + +**One override — and since the `emit_gated` fix it buys SLOT ECONOMY, not +correctness (⊘ corrected 2026-09-15; it WAS a correctness requirement before +that fix — see the ⊘ note on `hoist_gate_subset` in `lib.rs`):** when the `AND` also carries a resident plane the gate walk DROPS +as implied, a child whose result is a subset of that plane is rotated to the +front regardless of score (`hoist_gate_subset`). The ordering applies among the +children that rotation leaves alone. + +Board: `STATUS_BOARD.md` D-QCK-9; the probe is committed, not a one-off. + +--- + ## §9 — Claims in this document that are NOT verified in code +> **⊘ ADDED 2026-09-15 — two A1 claims belong here and were not filed.** §8a's +> verdict was written as settled and this register, which exists for exactly +> this, was not updated in the same pass. +> +> - **A1's own falsifier is half-run.** The pre-registered question (the A1 +> row's FALSIFIER cell) is *"on a **representative predicate stream**, what +> fraction of `Pred` ops are skippable by `under`, and does term order +> change that fraction?"* What ran is one LCG seed, 65 536 rows, five +> conjuncts, and **four hand-designed regimes** — one of them constructed +> specifically to contain a contiguous address-prefix term. That answers the +> ORDER half on a fixture and answers the FRACTION half only as "between +> 0 % and 99.9 %, pick a regime". `[H]` Nothing in this repo samples a +> workload V3 actually runs. +> - **"Dead words is the better sort key" is untested**, per §8a finding 2's +> own correction: the probe never compares ranking rules. `[H]` +> +> A third, related: the pre-registered text ends *"its swap-likeliness decay +> is **not obviously** the right control law"* — a hedge. §8a first cited that +> as having "anticipated the reason" and upgraded it to "must NOT be ported". +> A hedge quoted as a prediction is not a confirmation; §8a finding 3 is now +> corrected to say so. + Collected so they are not quoted as findings. Everything else in §2–§5 carries a `file:line` read this session. diff --git a/.github/workflows/rust-publish.yml b/.github/workflows/rust-publish.yml index 157a61570..f1b82d608 100644 --- a/.github/workflows/rust-publish.yml +++ b/.github/workflows/rust-publish.yml @@ -52,6 +52,30 @@ jobs: - uses: katyo/publish-crates@v2 with: registry-token: ${{ secrets.CARGO_REGISTRY_TOKEN }} - args: "--all-features" + # Every `lance-graph` feature EXCEPT the TWO that do not build. `--all-features` + # turns both on and fails the verification build this step runs before + # publishing. Measured on this tree: `cargo check -p lance-graph --all-features` + # exits 101; this list exits 0. + # + # aws-sdk — ours, opt-in by design. `aws-smithy-json 0.63.0` does not + # compile against `aws-smithy-types 1.7.0` (full reasoning at + # the `lance` entry in the workspace `Cargo.toml`). The feature + # exists so the capability stays addressable by name. + # lancedb-sdk — NOT ours, and it predates the `aws-sdk` flag: `lancedb 0.38.0` + # declares `default = []`, gates `Error::Http` behind + # `#[cfg(feature = "remote")]` (`src/error.rs:111`), but leaves + # `pub mod job;` ungated (`src/lib.rs:188`) while `job.rs` uses + # `Error::Http` unconditionally. The crate therefore cannot + # compile at all without `remote`. So `--all-features` has been + # broken here since the lancedb 0.38 bump — invisibly, because + # this is the only workflow that passes it and it never runs on + # push (`release: released` / `workflow_dispatch` only). + # See ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE. + # + # The cost of spelling the list out, stated rather than hidden: a feature added + # to `crates/lance-graph/Cargo.toml` later is NOT covered here, where + # `--all-features` would have picked it up for free. Add it in the same PR. + # See ISS-PUBLISH-FEATURE-LIST-CAN-DRIFT. + args: "--features unity-catalog,ndarray-hpc,with-cam-pq,planner,bgz17-codec,tensor-codec,lite-unified" path: crates/lance-graph dry-run: ${{ github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && github.event.inputs.mode == 'dry_run') }} diff --git a/.github/workflows/rust-test.yml b/.github/workflows/rust-test.yml index a18376208..30f1e0eb3 100644 --- a/.github/workflows/rust-test.yml +++ b/.github/workflows/rust-test.yml @@ -124,6 +124,15 @@ jobs: # exits non-zero when the committed region differs from its output. - name: Ternlog dispatch table is regenerated-and-diffed run: python3 crates/lance-graph-mask-risc/tools/gen_ternlog_dispatch.py --check + # lance-graph-quack: the DuckDB-shaped surface that LOWERS to mask-risc. + # Same reason as the mask-risc line above — a member's tests are not run + # by `cargo build --workspace`, so the line lands the day the crate + # joins. 9 of its 13 tests are differential against a per-row oracle + # (the other 4 are structural: gate shape, the pass/slot trade, + # width-vs-depth, the empty-junction refusal), so an un-run suite is an + # un-checked LOWERING, not merely an un-run test. + - name: Run quack tests + run: cargo test --manifest-path crates/lance-graph-quack/Cargo.toml # weather-poc is workspace-EXCLUDED. A new workflow introduced by the # same pull request cannot gate that pull request because `pull_request` # workflow selection comes from the base branch. Run both surfaces here diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml index fb6fc57b7..4837ca08d 100644 --- a/.github/workflows/style.yml +++ b/.github/workflows/style.yml @@ -87,6 +87,12 @@ jobs: # clean so it cannot silently regress — the deepnsm rule below. - name: Clippy lance-graph-mask-risc (mandatory) run: cargo clippy --manifest-path crates/lance-graph-mask-risc/Cargo.toml --all-targets -- -D warnings + # lance-graph-quack joined clean; gated while clean, same rule. + # `--all-targets` is load-bearing here rather than habitual: the A1 + # measurement lives in `examples/adaptive_order_probe.rs`, and `--lib` + # would leave the probe unlinted. + - name: Clippy lance-graph-quack (mandatory) + run: cargo clippy --manifest-path crates/lance-graph-quack/Cargo.toml --all-targets --all-features -- -D warnings # Tier B (advisory until incrementally cleaned, non-gating): # lance-graph core has ~91 pre-existing clippy violations to be paid down # in subsequent PRs (TD-CLIPPY-LG-1). Don't auto-fix — each violation @@ -184,6 +190,24 @@ jobs: run: cargo fmt --manifest-path crates/lance-graph-planner/Cargo.toml -- --check - name: Rustfmt lance-graph-mask-risc run: cargo fmt --manifest-path crates/lance-graph-mask-risc/Cargo.toml -- --check + # The four workspace members that had NO rustfmt line at all. Found by + # measuring the job against the member list rather than reading it: this + # job is per-crate by design (`causal-edge` is workspace-excluded, so + # `--all` would never reach it), and a per-crate job silently omits any + # member nobody added a line for. `lance-graph-contract` is the one that + # matters here — this branch edits `recipes.rs` and nothing was gating it. + # All four verified rustfmt-clean on the pinned toolchain before arming, + # the same rule the block below this one records. + - name: Rustfmt lance-graph-contract (previously ungated) + run: cargo fmt --manifest-path crates/lance-graph-contract/Cargo.toml -- --check + - name: Rustfmt lance-graph-callcenter (previously ungated) + run: cargo fmt --manifest-path crates/lance-graph-callcenter/Cargo.toml -- --check + - name: Rustfmt lance-graph-supervisor (previously ungated) + run: cargo fmt --manifest-path crates/lance-graph-supervisor/Cargo.toml -- --check + - name: Rustfmt bgz-tensor (previously ungated) + run: cargo fmt --manifest-path crates/bgz-tensor/Cargo.toml -- --check + - name: Rustfmt lance-graph-quack + run: cargo fmt --manifest-path crates/lance-graph-quack/Cargo.toml -- --check - name: Rustfmt lance-graph-ontology (previously ungated) run: cargo fmt --manifest-path crates/lance-graph-ontology/Cargo.toml -- --check - name: Rustfmt lance-graph-rbac (previously ungated) diff --git a/Cargo.toml b/Cargo.toml index b8d6b9283..6152562cf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -6,6 +6,7 @@ members = [ "crates/lance-graph-planner", "crates/lance-graph-contract", "crates/lance-graph-mask-risc", + "crates/lance-graph-quack", "crates/neural-debug", "crates/lance-graph-callcenter", "crates/lance-graph-archetype", @@ -259,7 +260,50 @@ resolver = "2" # deltalake was the only thing that pulled 53, and it is gone (0 hits in the # lock). [workspace.dependencies] -lance = "=11.0.0" +# `default-features = false` + lance's own default list MINUS `aws`. +# +# WHY: `lance/aws` -> `lance-io/aws` -> `dep:aws-config` -> `aws-smithy-json +# 0.63.0`, which does not compile against `aws-smithy-types 1.7.0` +# (`Document::Object` took a `DocumentObject`, `Document` went +# `#[non_exhaustive]`). 0.63.0 declares `aws-smithy-types ^1.6.1`, so it ALWAYS +# resolves the breaking 1.7.0, and the newest `aws-config` requires `^0.63.0` +# and cannot reach the fixed 0.64.0. No version selection fixes it; this repo +# tracks no lock, so every CI run re-resolves into the break. It is upstream's +# bug, reproducing on our `main` and on every branch. +# +# WHAT THIS DOES *NOT* COST -- measured, not assumed. `lance-io`'s `aws` +# feature bundles TWO unrelated things: the AWS *SDK* (`aws-config`, the broken +# half) and `object_store/aws`, the generic S3-COMPATIBLE backend. Only the +# first is dropped here: `object_store/aws` is declared DIRECTLY by this +# workspace and by `crates/lance-graph`, so cargo unification keeps it on. +# Verified with the change applied: `cargo tree -i aws-config` -> "did not +# match any packages", while `cargo tree -e features -i object_store` still +# lists feature "aws". So S3-compatible object storage is UNAFFECTED -- +# including `lance-graph-hydrate`'s slab hydration, which drives `object_store` +# with `aws_endpoint` + `aws_virtual_hosted_style_request = false` (an +# S3-compatible endpoint, not an AWS-native one) and never touches the SDK. +# Nothing in this workspace names `aws_config` / `aws_sdk_*` / +# `aws_credential_types` at all -- the SDK was pure transitive weight. +# +# WHAT IT DOES COST: AWS-*native* credential machinery only -- IMDS, SSO, +# STS assume-role -- which a deployment carrying explicit key/secret/endpoint +# does not use. +# +# TO RESTORE: `--features lance-graph/aws-sdk` (see that crate's [features]). +# If upstream is still broken when someone wants it, the escape hatch is a +# fork of `aws-smithy-json` fixed against `aws-smithy-types 1.7` -- the two +# call sites need `.into()` and one wildcard arm; upstream's own compiler +# hints spell both out. +lance = { version = "=11.0.0", default-features = false, features = [ + "azure", + "gcp", + "oss", + "huggingface", + "tencent", + "tos", + "goosefs", + "geo", +] } lance-linalg = "=11.0.0" lance-index = "=11.0.0" lancedb = { version = "=0.38.0", default-features = false } diff --git a/crates/lance-graph-contract/src/recipes.rs b/crates/lance-graph-contract/src/recipes.rs index 6897fb432..b948e3bd8 100644 --- a/crates/lance-graph-contract/src/recipes.rs +++ b/crates/lance-graph-contract/src/recipes.rs @@ -93,7 +93,8 @@ pub const RECIPES: [Recipe; 34] = [ mechanism: ParallelIndependence, bucket: Control, spo2cubed: NotCovered, - substrate: "rung depth × Expand/Compress; Berry-Esseen stop", + substrate: + "rung depth × Expand/Compress; Jirak-derived Σ-band stop (SigmaTierBands::jirak_p)", }, Recipe { id: 2, @@ -404,7 +405,7 @@ pub const RECIPES: [Recipe; 34] = [ mechanism: Infrastructure, bucket: Datapath, spo2cubed: NotCovered, - substrate: "Berry-Esseen noise floor + reciprocal A→B,B→A validation", + substrate: "Jirak weak-dep noise floor (jc Pillar 5) + reciprocal A→B,B→A validation", }, Recipe { id: 33, @@ -908,4 +909,69 @@ mod tests { assert_eq!(count(Mechanism::StructuralDivergence), 8); // #4,6,9,13,23,28,31,34 assert_eq!(count(Mechanism::Infrastructure), 14); } + + /// Does this substrate string cite the CLASSICAL (IID) Berry-Esseen bound + /// without naming Jirak's weak-dependence form? + /// + /// Kept as a named function so the guard below can be shown to FIRE — a + /// check that only ever runs over already-clean data proves nothing about + /// itself. + fn cites_classical_berry_esseen(substrate: &str) -> bool { + substrate.contains("Berry-Esseen") && !substrate.contains("Jirak") + } + + /// FAILS IF: a recipe cites classical Berry-Esseen. + /// + /// `I-NOISE-FLOOR-JIRAK` is an iron rule, not a preference: this system's + /// bits are weakly dependent BY CONSTRUCTION (correlated embedding + /// projections, overlapping role-key slices, a shared 4096-centroid + /// codebook, XOR bundle accumulation), so the classical IID bound is the + /// wrong theorem and understates the error. `jc`'s Pillar 5 measures + /// exactly that inflation and its own result line says so: *"Jirak's + /// weak-dep rate is the correct citation for this substrate."* + /// + /// Two recipes violated it in shipped data — #1 RTE ("Berry-Esseen stop") + /// and #32 SDD ("Berry-Esseen noise floor") — and neither was caught by + /// any test, because nothing read the `substrate` column. Both now name + /// the SHIPPED Jirak surfaces (`SigmaTierBands::jirak_p`, which replaced + /// the hand-tuned bands, and `jc` Pillar 5) rather than the paper alone, + /// so the citation points at something a reader can run. + #[test] + fn no_recipe_cites_classical_berry_esseen() { + let bad: Vec<&str> = RECIPES + .iter() + .filter(|r| cites_classical_berry_esseen(r.substrate)) + .map(|r| r.code) + .collect(); + assert!( + bad.is_empty(), + "I-NOISE-FLOOR-JIRAK: these cite classical Berry-Esseen: {bad:?}" + ); + + // CAN-FIRE half: the guard must actually reject the shape it names. + // Without this the assertion above would hold for a predicate that + // never returns true — including one with a typo'd needle. + assert!( + cites_classical_berry_esseen("Berry-Esseen noise floor"), + "the guard must reject a classical citation" + ); + // ...and must NOT reject the corrected form, or the fix would be + // indistinguishable from deleting the concept. + assert!( + !cites_classical_berry_esseen("Jirak weak-dep noise floor (jc Pillar 5)"), + "naming Jirak is the correction, not a second violation" + ); + + // Anti-vacuity: the corpus really does still discuss the noise floor, + // so this is a check over live citations rather than over a column + // someone emptied. + assert!( + RECIPES + .iter() + .filter(|r| r.substrate.contains("Jirak")) + .count() + >= 2, + "both corrected recipes must still carry their citation" + ); + } } diff --git a/crates/lance-graph-quack/Cargo.toml b/crates/lance-graph-quack/Cargo.toml new file mode 100644 index 000000000..87bd0786d --- /dev/null +++ b/crates/lance-graph-quack/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "lance-graph-quack" +version = "0.1.0" +edition = "2021" +publish = false +description = "A columnar query surface whose OPERATORS ARE MASKING OPS over the V3 SoA. DuckDB's shape — scan, filter, aggregate — with every operator lowering to a lance-graph-mask-risc Program and, under it, ndarray::simd's masking algebra. No expression interpreter, no row iterator, no second evaluator." + +# The ONE evaluator. This crate LOWERS to `Program`; it never evaluates one — +# a `match` here that computed anything would be the duplicate evaluator the +# whole arc exists to avoid. No ndarray dep for the same reason: the masking +# algebra is reached THROUGH mask-risc, never beside it. +[dependencies] +lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } + +[dev-dependencies] diff --git a/crates/lance-graph-quack/examples/adaptive_order_probe.rs b/crates/lance-graph-quack/examples/adaptive_order_probe.rs new file mode 100644 index 000000000..7579d73b4 --- /dev/null +++ b/crates/lance-graph-quack/examples/adaptive_order_probe.rs @@ -0,0 +1,418 @@ +//! A1 — does conjunction term ORDER change how much work the survivor skip +//! avoids? The measurement the translation matrix demands before any port. +//! +//! # Why this probe exists, and why it comes before the code +//! +//! `duckdb-to-v3-translation-matrix-v1.md` row A1 is the one place DuckDB has +//! something V3 does not: `AdaptiveFilter` permutes a conjunction's terms at +//! runtime by measured selectivity, via an adjacent-transposition hill-climb +//! with a swap-likeliness decay. The row's falsifier is unusually blunt about +//! what to do with that: +//! +//! > **The falsifier is prior to the port, and it may kill the whole idea.** +//! > DuckDB's reordering pays because term k runs only on survivors of +//! > 1..k−1, so a selective term first *shrinks the input*. In V3 a predicate +//! > sweep costs the **full column** regardless of position — so **reordering +//! > saves nothing on generation**. It can only save by *avoidance*. +//! > **Measure:** what fraction of `Pred` ops are skippable by `under`, and +//! > does term order change that fraction? If order does not move the skip +//! > fraction, A1 is ELIMINATE and the machinery must not be ported. +//! +//! So this prints a number and draws no conclusion the number does not force. +//! +//! # What is counted, and what that models +//! +//! `MaskOp::Pred { under }` skips at 64-row WORD granularity: a word where the +//! gate has no survivor is not evaluated. Under a conjunction lowered in place, +//! term `i + 1` is gated on the accumulated result of terms `1..i`, so the +//! skippable fraction for term `i + 1` is the fraction of gate words that are +//! entirely zero. +//! +//! That is an exact model of WHICH words are skipped — the same quantity the +//! facade's `*_to_mask_under` decides on — and it is deliberately not a timing. +//! A wall-clock number here would be dominated by the fixture's cache +//! behaviour and would not answer the row's question, which is about skip +//! OPPORTUNITIES. The honest reading of the output is "how much work becomes +//! avoidable", not "how much time is saved". +//! +//! Run: `cargo run -p lance-graph-quack --example adaptive_order_probe --release` + +use lance_graph_mask_risc::{MaskOp, Operand}; +use lance_graph_quack::{Agg, Cmp, Col, Filter, Query}; + +const VALS: Col = Col(0); +const CLASS: Col = Col(1); +const ADDR: Col = Col(2); + +const N: usize = 1 << 16; +const WORDS: usize = N / 64; + +/// One conjunct: a label, the filter, and the rows it selects. +struct Term { + label: &'static str, + filter: Filter, + mask: Vec, + selectivity: f64, +} + +/// Knuth's LCG constants. Deterministic on purpose and seeded from a +/// constant: this probe's numbers are quoted in +/// `duckdb-to-v3-translation-matrix-v1.md` §8a and in `Query::and_by_skip`'s +/// doc table, so a run that did not reproduce them would silently invalidate +/// a recorded measurement rather than fail. The `>> 11` drops the low bits, +/// which are the weakest in an LCG. +fn lcg(seed: &mut u64) -> u64 { + *seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *seed >> 11 +} + +/// Words of `mask` that are entirely zero — exactly the words a `Pred` gated +/// on this mask does not evaluate. +fn dead_words(mask: &[u64]) -> usize { + mask.iter().filter(|w| **w == 0).count() +} + +fn and_into(acc: &mut [u64], other: &[u64]) { + for (a, b) in acc.iter_mut().zip(other) { + *a &= *b; + } +} + +fn popcount(mask: &[u64]) -> u32 { + mask.iter().map(|w| w.count_ones()).sum() +} + +/// Total words SKIPPED across a conjunction evaluated in this order. +/// +/// Term 0 is ungated and skips nothing — it is the seed. Term `i` for `i > 0` +/// is gated on the accumulation of `0..i`, and skips that accumulation's dead +/// words. +fn skipped_words(terms: &[&Term]) -> (usize, u32) { + let mut acc = vec![u64::MAX; WORDS]; + and_into(&mut acc, &terms[0].mask); + let mut skipped = 0usize; + for t in &terms[1..] { + skipped += dead_words(&acc); + and_into(&mut acc, &t.mask); + } + (skipped, popcount(&acc)) +} + +/// EXHAUSTIVE, not sampled — which is what lets the probe report a true +/// best and worst order rather than the best and worst it happened to try. +/// Factorial in the term count, so it is only viable because the fixtures +/// are deliberately small (4 terms = 24 orders). +fn permutations<'a>(items: &[&'a Term]) -> Vec> { + if items.len() <= 1 { + return vec![items.to_vec()]; + } + let mut out = Vec::new(); + for i in 0..items.len() { + let mut rest = items.to_vec(); + let head = rest.remove(i); + for mut p in permutations(&rest) { + p.insert(0, head); + out.push(p); + } + } + out +} + +/// One regime: a set of conjuncts, and the spread its orderings produce. +struct Scenario { + name: &'static str, + terms: Vec, +} + +fn main() { + let mut seed = 0x51ED_C0DEu64; + let vals: Vec = (0..N) + .map(|_| (lcg(&mut seed) % 2000) as i32 - 1000) + .collect(); + let classes: Vec = (0..N).map(|_| (lcg(&mut seed) % 64) as u32).collect(); + // An ADDRESS-ORDERED lane, the shape the V3 substrate actually has. + let addr: Vec = (0..N).map(|i| (i as u64) << 8).collect(); + + let build = |label: &'static str, filter: Filter, keep: &dyn Fn(usize) -> bool| { + let mut mask = vec![0u64; WORDS]; + let mut live = 0usize; + for r in 0..N { + if keep(r) { + mask[r / 64] |= 1u64 << (r % 64); + live += 1; + } + } + Term { + label, + filter, + mask, + selectivity: live as f64 / N as f64, + } + }; + + // THREE regimes, not one. A single fixture would answer the row's question + // only for that fixture, and the two ends behave differently for a reason + // that matters: a conjunction whose accumulator collapses to nearly empty + // makes almost every later word skippable regardless of order, while a + // permissive one leaves little to skip at all. The interesting number is + // what order buys in between. + // + // The first draft of this probe had ONE regime and it was degenerate: + // `v < 500` and `v > 900` are disjoint, so the conjunction selected ZERO + // rows and the "best" ordering was just measuring how fast the accumulator + // died. Every scenario below is asserted to select a proper subset. + let scenarios = vec![ + Scenario { + name: "selective (a long tail of survivors)", + terms: vec![ + build( + "v > -900 (0.95)", + Filter::cmp(VALS, Cmp::GtI32(-900)), + &|r| vals[r] > -900, + ), + build( + "v < 500 (0.75)", + Filter::cmp(VALS, Cmp::LtI32(500)), + &|r| vals[r] < 500, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "v < -850 (selective)", + Filter::cmp(VALS, Cmp::LtI32(-850)), + &|r| vals[r] < -850, + ), + build( + "class == 7 (very selective)", + Filter::cmp(CLASS, Cmp::EqU32(7)), + &|r| classes[r] == 7, + ), + ], + }, + Scenario { + name: "moderate", + terms: vec![ + build( + "v > -900 (0.95)", + Filter::cmp(VALS, Cmp::GtI32(-900)), + &|r| vals[r] > -900, + ), + build( + "v < 500 (0.75)", + Filter::cmp(VALS, Cmp::LtI32(500)), + &|r| vals[r] < 500, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "v > -400 (0.70)", + Filter::cmp(VALS, Cmp::GtI32(-400)), + &|r| vals[r] > -400, + ), + build( + "class < 32 (0.50)", + Filter::cmp( + CLASS, + Cmp::MatchU32 { + pattern: 0, + care: 32, + }, + ), + &|r| classes[r] & 32 == 0, + ), + ], + }, + Scenario { + name: "permissive (little to skip at all)", + terms: vec![ + build( + "v > -990 (0.995)", + Filter::cmp(VALS, Cmp::GtI32(-990)), + &|r| vals[r] > -990, + ), + build( + "v < 990 (0.995)", + Filter::cmp(VALS, Cmp::LtI32(990)), + &|r| vals[r] < 990, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "class != 1 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(1)), + &|r| classes[r] != 1, + ), + build( + "class != 2 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(2)), + &|r| classes[r] != 2, + ), + ], + }, + Scenario { + name: "clustered (one conjunct is an ADDRESS PREFIX)", + terms: vec![ + build( + "addr prefix /50 (contiguous)", + Filter::prefix_u64(ADDR, addr[N / 4], 50), + &|r| { + let care = u64::MAX << (64 - 50); + (addr[r] ^ addr[N / 4]) & care == 0 + }, + ), + build( + "v > -900 (0.95)", + Filter::cmp(VALS, Cmp::GtI32(-900)), + &|r| vals[r] > -900, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "v < 500 (0.75)", + Filter::cmp(VALS, Cmp::LtI32(500)), + &|r| vals[r] < 500, + ), + build( + "v > -400 (0.70)", + Filter::cmp(VALS, Cmp::GtI32(-400)), + &|r| vals[r] > -400, + ), + ], + }, + ]; + + println!("N = {N} rows, {WORDS} words per mask, 5 conjuncts per scenario"); + println!( + "\nskipped = 64-row words a gated `Pred` does not evaluate, summed over the\n {} gated positions of one ordering (term 0 is the ungated seed).\n", + 4 * WORDS + ); + + for sc in &scenarios { + let refs: Vec<&Term> = sc.terms.iter().collect(); + let gated_positions = (sc.terms.len() - 1) * WORDS; + + // The model is tied to the SHIPPED lowering rather than assumed: the + // conjunction is really lowered, and every term after the first must + // come back gated on a scratch slot. If that stops being true the + // probe is measuring something the executor does not do. + let program = lance_graph_quack::lower(&Query { + filter: Filter::and(refs.iter().map(|t| t.filter.clone())), + agg: Agg::Count, + }) + .expect("lowers"); + let accumulator_gated = program + .ops + .iter() + .filter(|op| { + matches!( + op, + MaskOp::Pred { + under: Some(Operand::Scratch(_)), + .. + } + ) + }) + .count(); + assert_eq!( + accumulator_gated, + sc.terms.len() - 1, + "{}: the lowering gated {accumulator_gated} of {} later terms on the \ + accumulator; this probe measures a skip the executor would not perform", + sc.name, + sc.terms.len() - 1 + ); + + let perms = permutations(&refs); + let mut results: Vec<(usize, u32, Vec<&'static str>)> = perms + .iter() + .map(|p| { + let (skipped, count) = skipped_words(p); + (skipped, count, p.iter().map(|t| t.label).collect()) + }) + .collect(); + + let answer = results[0].1; + assert!( + results.iter().all(|r| r.1 == answer), + "{}: orderings disagree on the count; the probe is measuring a defect", + sc.name + ); + assert!( + answer > 0 && (answer as usize) < N, + "{}: selects {answer}/{N} — a degenerate conjunction proves nothing \ + about order", + sc.name + ); + + results.sort_by_key(|r| r.0); + let worst = &results[0]; + let best = results.last().expect("non-empty"); + let (written, _) = skipped_words(&refs); + let pct = |s: usize| 100.0 * s as f64 / gated_positions as f64; + + println!( + "=== {} — {answer} survivors ({:.3}%)", + sc.name, + 100.0 * answer as f64 / N as f64 + ); + for t in &sc.terms { + println!(" {:<30} sel {:.4}", t.label, t.selectivity); + } + println!( + " {:<12} {:>9} {:>9}", + "ordering", "skipped", "of gated" + ); + println!( + " {:<12} {:>9} {:>8.2}%", + "as written", + written, + pct(written) + ); + println!( + " {:<12} {:>9} {:>8.2}%", + "worst", + worst.0, + pct(worst.0) + ); + println!(" {:<12} {:>9} {:>8.2}%", "best", best.0, pct(best.0)); + // 0/0 is not an infinite ratio, it is NO SPREAD — the `moderate` and + // `permissive` regimes skip nothing in ANY order, and printing `infx` + // beside `spread 0.00` made the line contradict itself. Those two rows + // are cited in `Query::and_by_skip`'s doc table, so the print is the + // evidence a reader sees. + match (best.0, worst.0) { + (0, 0) => println!(" spread 0.00 percentage points, no skip in any order"), + (_, 0) => println!( + " spread {:.2} percentage points, best/worst unbounded (worst skips 0)", + pct(best.0) - pct(worst.0) + ), + (_, w) => println!( + " spread {:.2} percentage points, best/worst {:.2}x", + pct(best.0) - pct(worst.0), + best.0 as f64 / w as f64 + ), + } + println!(" best order: {}", best.2.join(" < ")); + println!(); + } + + println!( + "A1's falsifier: if term order does not move the skipped fraction, the row is\n\ + ELIMINATE and DuckDB's hill-climb must not be ported. These are counts of\n\ + AVOIDABLE word-evaluations, not timings." + ); +} diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs new file mode 100644 index 000000000..829dc6779 --- /dev/null +++ b/crates/lance-graph-quack/src/lib.rs @@ -0,0 +1,2232 @@ +//! A columnar query surface whose **operators are masking ops**. +//! +//! # What this is, and the shape it deliberately does NOT have +//! +//! DuckDB's operator set — scan, filter, project, aggregate, group — expressed +//! so that every operator LOWERS to a [`Program`] and is executed by the one +//! evaluator in `lance-graph-mask-risc`, which sits on `ndarray::simd`'s +//! masking algebra. +//! +//! What a columnar engine normally grows, and what is absent here on purpose: +//! +//! | the usual shape | why it is absent | +//! |---|---| +//! | an expression interpreter (`Expr` tree walked per batch) | a filter IS a `Pred`; walking a tree per batch is the second evaluator | +//! | a row iterator / `next()` volcano loop | the unit is a mask over `n_rows`, never a row | +//! | a per-operator kernel library | every operator is a `MaskOp` composition; a new operator is a new LOWERING, never a new kernel | +//! | a physical-plan `dyn Operator` chain | a plan is a `Program` — one flat op list, one terminal | +//! | a validity bitmap beside the data | the table's validity IS a resident mask plane ([`Filter::Plane`]); there is no separate NULL | +//! | a hash table for GROUP BY | a group is a mask; K groups are K gated equalities over the kept filter ([`lower_group_by`]) | +//! +//! The rule that keeps it honest: **this crate may build a [`Program`] and +//! must never evaluate one.** `execute` is called by the consumer, on a +//! scratch the consumer owns. A single `match` over operators here that +//! computed anything would be the duplicate evaluator the whole arc exists to +//! avoid. +//! +//! # The survivor skip — where a scan under a validity plane gets its cost model +//! +//! An `AND` whose children include a resident plane `g` (alpha, focus, a class +//! mask) is lowered so that every comparison beneath it is evaluated `under +//! g`: the predicate runs only over the 64-row words where `g` has a survivor +//! ([`MaskOp::Pred`]'s `under`). The rewrite is sound for ANY Boolean +//! remainder — `g ∧ rest(X₁..Xₙ) = g ∧ rest(g∧X₁, .., g∧Xₙ)`, since a row with +//! `g = 0` gives 0 on both sides and a row with `g = 1` leaves every leaf +//! unchanged — so gating passes through `NOT` and `OR` alike as long as `g` +//! itself stays a leaf. The plane leaf is DROPPED only when the gated +//! remainder is identically zero wherever `g` is zero: a gated comparison is; +//! an `AND` is if any child is; an `OR` is if every child is; a `NOT` never +//! is. `alpha & ((A & B) | C)` therefore costs three gated predicates and one +//! Boolean pass, with alpha never read as an operand at all. +//! +//! # Two lowerings, one meaning +//! +//! [`lower`] evaluates predicates IN PLACE and folds a junction's children +//! into its first child's slot — a filter of depth `d` costs `d + 1` slots, +//! width is free. [`lower_fused`] gives every predicate its own slot and hands +//! the Boolean skeleton to the fuser, which turns any subtree over three +//! leaves into one [`MaskOp::Ternlog`] — fewer mask passes, more slots. The +//! differential suite runs both against the same per-row oracle; a consumer +//! picks by whether it is scratch-bound or pass-bound. +//! +//! **They also differ in a way neither name suggests: only [`lower`] can +//! skip.** [`lower`] chains — the first predicate is ungated and each later +//! one gates on the running accumulator, so a conjunct narrows the input its +//! successors read. [`lower_fused`] gives every predicate its own slot and +//! gates it on the caller's resident plane ONLY (`under.map(Operand::Plane)`), +//! never on an accumulator, because the Boolean combination is deferred to the +//! ternlog — and [`MaskOp::Ternlog`] has no `under` field at all. So the fused +//! program has no progressive narrowing, and **term order cannot change its +//! skipped-word count.** The saving the gated form buys is physical, not +//! bookkeeping: `pack_under` does `if gate == 0 { continue }` before loading +//! the 64 values, so a dead word's column read never happens. +//! +//! This matters because it is the whole of what survives from DuckDB's +//! `AdaptiveFilter` (matrix row A1 / §8a). Ordering is worth up to 99.90 +//! percentage points of skipped words on a clustered conjunction under +//! [`lower`] — and exactly zero under [`lower_fused`], on the same query. +//! [`Filter::and_by_skip`]'s lever is therefore alive in one configuration: +//! gated lowering, plane-free conjunction, contiguous survivors. Under a +//! plane it is inert too (`ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE`). +//! +//! # Provenance — this is harvest-driven, not remembered +//! +//! Every operator below answers to a row of +//! `.claude/plans/duckdb-to-v3-translation-matrix-v1.md`, which reads DuckDB's +//! own source with `file:line` and rules each concept KEEP / ADAPT / +//! ELIMINATE / V3 BETTER / NEEDS FALSIFIER. The matrix is the specification; +//! this crate is one reading of it. +//! +//! | operator here | matrix row | verdict there | +//! |---|---|---| +//! | no `SelectionVector`, anywhere | R1 | ELIMINATE — an index list is the materialisation the mask-native invariant forbids | +//! | [`Col`] over a borrowed lane | R2 | KEEP — a flat vector IS an SoA lane | +//! | [`Cmp`] taking a scalar | R3 | ELIMINATE `ConstantVector` — a constant never acquires a representation | +//! | [`Filter::Plane`] | R6 | KEEP the representation, ELIMINATE the role — same packed `u64`, no NULL plane | +//! | [`Filter::prefix_u32`] / [`Filter::prefix_u64`] | R5 | ADAPT — *"the closest DuckDB comes to the V3 address"* | +//! | [`Query`] → [`Program`] | E1 | ADAPT — a state tree with per-node scratch becomes straight-line code over numbered slots | +//! | [`Agg`] as one terminal | E2 | ADAPT — Select-vs-Execute's two carriers collapse to one mask + one terminal tag | +//! | the six `i32` comparisons + two `u32` | E4 | ADAPT — DuckDB's 14-way physical-type switch narrows to what the lanes actually hold | +//! | `And` / `Or` | E5, E6 | ADAPT / **V3 BETTER** — DuckDB must SORT after OR to restore row order (`execute_conjunction.cpp:139`); a mask never lost it | +//! | [`Agg::BlendI32`] | E7 | ADAPT — CASE's narrowing false-set becomes a blend | +//! | [`Agg::Any`] / [`Agg::All`] | C7 | V3 BETTER — `HasNull`/`HasNotNull` are literally `mask_any`/`mask_all` | +//! | [`lower_group_by`] | A3 | V3 BETTER for the addressed case — a mask plane where the hash table would be | +//! +//! The harvest itself was re-run on 2026-09-14 and its first pass was +//! **repaired**, which is why the table above can cite what it cites. The +//! matrix's own §6 recorded the failure honestly: the `ruff_cpp_spo` harvest +//! had been pointed at 22 `.cpp` translation units, **seven of which came back +//! 100 % `Empty`**, because DuckDB's execution is template-dispatched and lives +//! in headers — so *"no row in this matrix cites a harvest TSV as evidence"*. +//! Pointing the same harvester at the headers (`scalar_executor.hpp`, +//! `comparison_operators.hpp`, `validity_mask.hpp`, `selection_vector.hpp`, +//! `ht_entry.hpp`, `vector.hpp`) yields **123 methods and 1,622 events** where +//! the `.cpp` pass yielded none. +//! +//! # What the header harvest changed here +//! +//! Two things, and both are in the code rather than only in this comment. +//! +//! **`Pred::MatchU64` was reachable from nothing.** It had been in the IR since +//! PR3, and this crate had no spelling for it, so a borrowed `LaneRef::U64` — +//! edge targets, ids, addresses — was queryable by no query. [`Cmp::MatchU64`] +//! closes that, and it is what the prefix operator over a 64-bit address needs. +//! +//! **The range primitive is real, and DuckDB's own bit-plane has it.** The +//! matrix files `mask_set_range` as T1 gap G6 on the strength of V3's +//! trie-reveal measurement. The header harvest shows the same operation on the +//! other side: `TemplatedValidityMask::SetRangeInvalid`, on the same packed-`u64` +//! carrier V3 uses. So [`Filter::prefix_u32`] lowers to a ternary-match SWEEP +//! today and says so plainly; the range WRITE waits on the primitive, per the +//! missing-capability STOP rule, rather than being hand-rolled one layer up. +//! +//! # What DuckDB has that this does not: the measurement loop +//! +//! `AdaptiveFilter` (matrix row A1) permutes a conjunction's terms at RUNTIME, +//! seeded from a static selectivity heuristic and then adapted by measured +//! RUNTIME. This crate cannot do that, and the reason is structural rather +//! than unfinished: it never executes, so there is nothing for it to measure. +//! +//! Order is still a real cost lever here — under the survivor skip a conjunct +//! whose survivors die in whole WORDS shrinks every later predicate's live +//! count — so [`Filter::and_by_skip`] takes the ordering decision as an INPUT. +//! The measurement that licensed even that much is +//! `examples/adaptive_order_probe.rs`; read [`Filter::and_by_skip`] for what it +//! does and does not establish. +//! +//! ⊘ This section previously called A1 "its only NEEDS-FALSIFIER of this kind" +//! and said "the caller currently owns it with no help". Both were true when +//! written and stopped being true in the same branch: A1 left that column and +//! `and_by_skip` is the help. +//! +//! # Status +//! +//! Filter (`=`/`<>`/`<`/`<=`/`>`/`>=`/ternary match/`IN`, `AND`/`OR`/`NOT`, +//! resident planes), the aggregates `COUNT`/`EXISTS`/`ALL`/`SUM`/`MIN`/`MAX`, +//! projection ([`Agg::Rows`] keeps the mask, [`Agg::BlendI32`] is the `CASE` +//! shape), and a two-phase `GROUP BY` over a categorical key. Absent, named: +//! the join (`src_mask → hop → dst_mask` is `lance-graph-mask-risc`'s PR5 — +//! there is no `hop` op to lower to yet); a one-terminal `GROUP BY SUM` +//! (`ndarray::simd` ships `masked_strided_group_sum`, but the IR names no +//! strided operand or group terminal, so K programs is the honest spelling +//! today); and everything the IR itself excludes — strings, `ORDER BY`, +//! three-valued NULL. + +#![forbid(unsafe_code)] + +use std::cmp::Reverse; + +use lance_graph_mask_risc::{ + fuse, BoolExpr, FuseError, MaskOp, Operand, Pred, Program, Terminal, MAX_SCRATCH_SLOTS, +}; + +/// A column reference — an index into [`Planes::lanes`](lance_graph_mask_risc::Planes). +/// +/// Not a name: name resolution is the catalogue's job and this crate has no +/// catalogue. A consumer that has names resolves them before it gets here. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Col(pub u16); + +/// A resident mask plane — an index into +/// [`Planes::masks`](lance_graph_mask_risc::Planes): alpha, focus, a class +/// mask, a kept filter. The table's validity lives here, not in a NULL bit. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Mask(pub u16); + +/// A predicate over one column — the filter's whole vocabulary. +/// +/// One variant per masking op that produces a mask from a value lane. That is +/// not a coincidence and not a coding convenience: **the query language's +/// predicate set IS the masking algebra's predicate set**, so a predicate this +/// crate cannot spell is one the substrate cannot run, and adding one here +/// without adding it below would be the first crack. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Cmp { + /// `col == v` over a signed lane. + EqI32(i32), + /// `col != v`. + NeI32(i32), + /// `col < v`. + LtI32(i32), + /// `col <= v`. + LeI32(i32), + /// `col > v`. + GtI32(i32), + /// `col >= v`. + GeI32(i32), + /// `col == v` over an unsigned lane, exact bitwise. + EqU32(u32), + /// `col != v`. + NeU32(u32), + /// `(col ^ pattern) & care == 0` — the ternary match, which SQL has no + /// spelling for and the substrate has had all along. + MatchU32 { + /// The bits to compare. + pattern: u32, + /// Which bits participate; zero means "don't care". + care: u32, + }, + /// The ternary match over a 64-bit lane — edge targets, ids, addresses. + /// + /// `Pred::MatchU64` has been in the IR since PR3 and was unreachable from + /// this crate, so a `LaneRef::U64` lane could be borrowed and never + /// queried. That is the whole of the gap this closes; there is no new + /// primitive underneath. + MatchU64 { + /// The bits to compare. + pattern: u64, + /// Which bits participate; zero means "don't care". + care: u64, + }, +} + +/// A filter expression: predicates over columns and resident planes, composed +/// with AND/OR/NOT. +/// +/// Deliberately a tree HERE and never at run time — it is lowered once into a +/// flat [`Program`] and the tree is gone before anything executes. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Filter { + /// A leaf comparison on one column. + Cmp(Col, Cmp), + /// A resident mask plane read as a predicate — the scan's validity, a + /// focus, a class mask. `Filter::Plane(alpha)` alone is `SELECT … FROM t` + /// with no `WHERE`: the table is its validity plane, and a query over + /// every row of it lowers to zero ops. + Plane(Mask), + /// Every child must hold. + And(Vec), + /// Some child must hold. + Or(Vec), + /// The child must not hold. + Not(Box), +} + +impl Filter { + /// `col ` — the leaf builder, so call sites read as the query does. + pub fn cmp(col: Col, cmp: Cmp) -> Self { + Filter::Cmp(col, cmp) + } + + /// A resident plane as a predicate. + pub fn plane(mask: Mask) -> Self { + Filter::Plane(mask) + } + + /// Conjunction. + pub fn and(parts: impl IntoIterator) -> Self { + Filter::And(parts.into_iter().collect()) + } + + /// Disjunction. + pub fn or(parts: impl IntoIterator) -> Self { + Filter::Or(parts.into_iter().collect()) + } + + /// Negation. + /// + /// Named `negate` rather than `not`: an inherent `not` shadows + /// `std::ops::Not` and reads ambiguously at a call site that also uses the + /// operator. Clippy's `should_implement_trait` is right here. + pub fn negate(inner: Filter) -> Self { + Filter::Not(Box::new(inner)) + } + + /// `col IN (set)` over an unsigned lane — a disjunction of equalities, + /// which is exactly what it is; there is no IN-list kernel because none + /// is needed. An empty set is `x IN ()`, refused at lowering for the same + /// reason an empty `OR` is: it is that `OR`. + pub fn in_u32(col: Col, set: impl IntoIterator) -> Self { + Filter::Or( + set.into_iter() + .map(|v| Filter::Cmp(col, Cmp::EqU32(v))) + .collect(), + ) + } + + /// `col IN (set)` over a signed lane. See [`Filter::in_u32`]. + pub fn in_i32(col: Col, set: impl IntoIterator) -> Self { + Filter::Or( + set.into_iter() + .map(|v| Filter::Cmp(col, Cmp::EqI32(v))) + .collect(), + ) + } + + /// An `AND` whose children are ordered by how much of the gate each one + /// KILLS — the V3 answer to DuckDB's `AdaptiveFilter`, and deliberately + /// not DuckDB's algorithm. + /// + /// `parts` is `(skip_score, child)`; children sort by DESCENDING score, + /// stably, so equal scores keep the caller's order. + /// + /// # Why the score is dead WORDS and not selectivity + /// + /// DuckDB orders conjunction terms by a static selectivity heuristic and + /// then adapts by measured RUNTIME (`adaptive_filter.cpp`: swap two + /// neighbours, measure 10 iterations, keep if mean runtime dropped). It + /// can rank that way because there term `k` runs only on the survivors of + /// `1..k-1` — a selective term + /// first literally shrinks the input. In V3 a predicate sweep costs the + /// full column wherever it sits, so ordering can only pay by AVOIDANCE: + /// the survivor skip drops a 64-row WORD when the gate has no survivor in + /// it. The matrix (row A1) said so and required the measurement before any + /// port. `examples/adaptive_order_probe.rs` is that measurement, over + /// 65,536 rows, five conjuncts, all 120 orderings, four regimes: + /// + /// | regime | survivors | skipped, worst → best order | + /// |---|---|---| + /// | permissive | 94.3 % | **0.00 % → 0.00 %** | + /// | moderate | 21.8 % | **0.00 % → 0.00 %** | + /// | selective | 0.055 % | 5.66 % → **80.66 %** (14.2×) | + /// | clustered (an address prefix) | 0.047 % | 0.00 % → **99.90 %** | + /// + /// Two findings, and a third that is a correction rather than a result. + /// + /// **Order does move the skip fraction, so A1 is not ELIMINATE** — but + /// only where there is anything to skip. At 21.8 % survival with + /// survivors SCATTERED, a 64-row word is all-dead with probability + /// `0.78163^64 ≈ 1.4e-7`, so NO ordering skips anything and the whole + /// question is moot. Word granularity needs the accumulator to die in + /// whole words, not merely to be small — scattering is the condition, + /// not density, and a dense-but-contiguous population has plenty of dead + /// words. + /// + /// **Which means selectivity cannot tell you WHETHER reordering is worth + /// anything.** The selective and the clustered regimes have almost + /// identical survivor counts — 36 and 31 rows — and differ by 19 + /// percentage points of achievable skip (best against best: 80.66 % vs + /// 99.90 %), because one conjunct's survivors are contiguous and the + /// other's are scattered. A selectivity-only cost model cannot separate + /// those two cases. That is also why V3 has this lever at all: an address + /// prefix selects a contiguous subtree ([`Filter::prefix_u64`]), which is + /// the clustered row of that table. + /// + /// Note what that does NOT say. An earlier version of this paragraph read + /// *"rank by selectivity and those two look the same; rank by DEAD WORDS + /// and they do not"* — a claim about the right SORT KEY. The probe does + /// not support it: it enumerates all 120 permutations and reports + /// min/max, never computing a selectivity-ranked order, never computing a + /// dead-word-ranked order, never comparing two ranking rules, and never + /// calling this function. Whether dead words is the better key to sort by + /// is **untested**; what is measured is the between-regime diagnostic + /// above. + /// + /// # Why the score is the CALLER's, and why there is no hill-climb + /// + /// The shipped surface of this crate builds programs and never evaluates + /// one, so there is no point at which it could measure a score; it comes + /// from a previous execution the caller ran. + /// + /// DuckDB's adjacent-transposition hill-climb is not ported — **and the + /// reason first given here was measured FALSE, so it is worth stating + /// correctly.** The claim was that "the quantity being optimised is a step + /// function of clustering … so a local search over adjacent swaps is + /// exploring the wrong landscape". Instrumenting the probe's own + /// `skipped_words` model with the clustered regime's prefix term at each + /// index gives `[4092, 3069, 2046, 1023, 0]` — adjacent deltas all + /// exactly `-1023`, a monotone linear ramp, because the prefix term's + /// mask is one live word of 1024 and each gated position past it skips + /// the other 1023. Every forward adjacent swap improves it by the same + /// amount. That is the friendliest possible hill-climb landscape, not the + /// wrong one. + /// + /// The step-like behaviour is BETWEEN regimes (does this conjunction + /// contain a clustered term at all); the search space is WITHIN one + /// permutation set. The original argument reasoned from the first to the + /// second. + /// + /// The real reason is narrower: this crate never executes, so there is no + /// runtime for a hill-climb to measure. Whether a consumer that DOES + /// execute should run one is unanswered here — and DuckDB's loop adapts + /// on measured RUNTIME, seeded from a static selectivity heuristic, not + /// on measured selectivity, so the comparison would have to be against + /// that. + /// + /// # This lever is INERT on a conjunction that carries a resident plane + /// + /// Measured, and it is the price of the `emit_gated` fix in `b7e6cef`: + /// once an `AND` carries a plane, every `Pred` inside it gates on that + /// same FIXED plane regardless of position, so the skipped-word count is + /// order-independent and this ordering buys exactly zero. Plane-free + /// conjunctions are unaffected — their preds still chain on the running + /// accumulator, which is where `adaptive_order_probe` measured the lever + /// and the only place it claims one. Since the crate's own headline shape + /// (`Filter::Plane(alpha)` = `SELECT … FROM t`) IS planed, that is most + /// real queries. Recorded as `ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE` + /// with the measurement and the way out. + /// + /// # One caveat, because it is a real override + /// + /// When this `AND` also carries a resident plane that the gate walk + /// DROPS as implied, a child whose result is a subset of that plane is + /// rotated to the front regardless of score, so the ordering here applies + /// among the children the rotation leaves alone. + /// + /// ⊘ That rotation was called "a correctness requirement, not a + /// preference" until 2026-09-15. It WAS one before the `emit_gated` fix; + /// afterwards the plane gates every child wherever it sits, so what the + /// rotation still buys is SLOT ECONOMY (a plane first makes the + /// accumulator a plane operand, costing no scratch slot: 1 vs 2 on + /// `alpha AND focus AND v < 50`). Disable-verified — rotation removed, + /// 120,000 differential cases, 65,919 of them planed, zero divergences, on + /// a harness proven able to see this class of regression by restoring the + /// pre-fix gate line and watching it fail 2,668 of the same cases. + pub fn and_by_skip(parts: impl IntoIterator) -> Self { + let mut scored: Vec<(u32, Filter)> = parts.into_iter().collect(); + // `sort_by_key` is stable, so equal scores keep the caller's order + // rather than being permuted by an implementation detail. `Reverse` + // rather than a reversed comparator: a stable sort over a reversed KEY + // keeps ties in caller order, while reversing the COMPARISON of a + // stable sort would too — but clippy rejects the latter spelling, and + // the two are only equivalent because the key is `Copy`. + scored.sort_by_key(|&(score, _)| Reverse(score)); + Filter::And(scored.into_iter().map(|(_, f)| f).collect()) + } + + /// Rows whose 32-bit address lane starts with the top `bits` of `prefix` — + /// the operator SQL has no name for and the V3 address was built around. + /// + /// # Why this is not just another equality + /// + /// On an address-ordered lane a prefix names a CONTIGUOUS RANGE: every row + /// under one trie node is `2^(32 - bits)` consecutive addresses. DuckDB + /// arrives at the same shape and then discards it — `SequenceVector` + /// compresses a range to three scalars (`vector.cpp:498-500`) and + /// `ToUnifiedFormat` flattens it to N materialised values before any kernel + /// runs (`vector.cpp:461-465`), after which `DataChunk::Slice` re-manufactures + /// the range as a per-row index loop (`data_chunk.cpp:394-397`). That round + /// trip is the matrix's R5/R8, and it is what "better than faithful" means + /// here: the range is kept, not rebuilt. + /// + /// # What this actually lowers to today, stated exactly + /// + /// A ternary match — a full sweep of the lane, one pass, no allocation. It + /// is NOT yet a range WRITE. The range write is `mask_set_range`, the + /// matrix's T1 gap G6, and it is absent from `ndarray::simd`; per the + /// missing-capability STOP rule a consumer does not hand-roll it one layer + /// up, so this crate spells the PREDICATE and waits for the primitive. + /// + /// That the primitive is real rather than wished for is the harvest's + /// evidence, not this crate's opinion: DuckDB's own bit-plane carries + /// `TemplatedValidityMask::SetRangeInvalid` (`validity_mask.hpp`, harvested + /// 2026-09-14 from the headers) — the same packed-`u64` representation V3 + /// uses, with the range operation already on it. + /// + /// `bits` is clamped to 32; `bits == 0` matches every row (care is empty), + /// which is the honest reading of "no significant bits" rather than an + /// error, and `bits == 32` matches exactly one address. + pub fn prefix_u32(col: Col, prefix: u32, bits: u32) -> Self { + let care = match bits.min(32) { + 0 => 0, + b => u32::MAX << (32 - b), + }; + Filter::Cmp( + col, + Cmp::MatchU32 { + pattern: prefix & care, + care, + }, + ) + } + + /// [`Filter::prefix_u32`] over a 64-bit address lane. + /// + /// This is the one that reaches the canonical GUID's own prefix: classid, + /// then HEEL/HIP/TWIG, each a nibble-addressed tier of the cascade. A + /// `bits` that lands on a tier boundary selects exactly that subtree. + pub fn prefix_u64(col: Col, prefix: u64, bits: u32) -> Self { + let care = match bits.min(64) { + 0 => 0, + b => u64::MAX << (64 - b), + }; + Filter::Cmp( + col, + Cmp::MatchU64 { + pattern: prefix & care, + care, + }, + ) + } +} + +/// Why a query could not be lowered. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum LowerError { + /// An `And`/`Or` with no children. Refused rather than folded to a + /// constant: an empty conjunction is `true` and an empty disjunction is + /// `false`, and a caller that built one by accident wants to hear about it + /// rather than receive whichever identity this crate happened to pick. + EmptyJunction, + /// The program would need more scratch slots than a `u16` can name. + TooManySlots { + /// The count that overflowed. + needed: usize, + }, + /// A `GROUP BY` asked for [`Agg::BlendI32`]. Every group program writes + /// the WHOLE `out` slice, so K groups would leave the last group's blend + /// and silently discard K − 1 — refused rather than answered wrongly. + GroupedBlend, +} + +impl core::fmt::Display for LowerError { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + match self { + LowerError::EmptyJunction => { + write!(f, "an AND/OR with no children has no non-arbitrary meaning") + } + LowerError::TooManySlots { needed } => { + write!(f, "needs {needed} scratch slots; the address space is u16") + } + LowerError::GroupedBlend => { + write!(f, "a blend writes the whole output; it cannot be grouped") + } + } + } +} + +/// `LowerError` is public, so it must be usable as an error: without this a +/// caller cannot `?` it into `Box` and no wrapper can surface it +/// through `source()`. Empty body — the variants carry no nested cause — which +/// is the shape eight sibling crates already use (`cognitive-shader-driver`, +/// `elixir-template`, `lance-graph-callcenter`, …). +impl std::error::Error for LowerError {} + +/// What a query asks for. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Agg { + /// How many rows survive the filter. + Count, + /// Whether any row survives. + Any, + /// Whether every row survives. + All, + /// Σ over a signed lane, restricted to surviving rows. + SumI32(Col), + /// min over surviving rows. + MinI32(Col), + /// max over surviving rows. + MaxI32(Col), + /// The surviving rows themselves — projection. Nothing is reduced and + /// nothing is copied: the result is the final mask + /// ([`Terminal::Keep`]), and the projected columns are the resident + /// lanes the caller already holds. The one materialiser is + /// `lance_graph_mask_risc::materialize_rows`, and it is the caller's + /// to invoke. + Rows, + /// `CASE WHEN filter THEN then ELSE els END` — written per row into a + /// caller-supplied `out` slice, no compaction ([`Terminal::BlendI32`]). + BlendI32 { + /// The lane read where the filter holds. + then: Col, + /// The lane read where it does not. + els: Col, + }, +} + +/// One query: a filter and what to ask of the rows that pass it. +/// +/// The filter is required. There is no unfiltered table in the substrate — +/// every table IS its validity plane — so "every row" is spelled +/// [`Filter::Plane`]`(alpha)`, which lowers to zero ops and reads the plane +/// straight from the terminal. (A first draft of this crate tried to spell it +/// without a plane, as a predicate over lane 0 followed by a constant +/// ternlog; that typechecked only when lane 0 happened to be `U32` — a latent +/// lane-kind bug. The plane leaf is the spelling that is correct by +/// construction.) +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Query { + /// The filter. + pub filter: Filter, + /// The aggregate. + pub agg: Agg, +} + +/// `GROUP BY key` over a low-cardinality unsigned key lane whose values are +/// `0..groups` — the dictionary-encoded / categorical shape. +/// +/// The filter is required for the same reason [`Query`]'s is; the whole +/// table is `Filter::Plane(alpha)`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GroupBy { + /// Rows admitted to any group. + pub filter: Filter, + /// The key lane (`u32`). + pub key: Col, + /// Number of key values; group `g` is the rows whose key equals `g`. + pub groups: u32, + /// The aggregate computed per group. [`Agg::BlendI32`] is refused. + pub agg: Agg, +} + +/// The two-phase plan a [`GroupBy`] lowers to — DuckDB's pipeline break, +/// with a mask plane where the hash table would be. +/// +/// The caller runs `filter` (a [`Terminal::Keep`]), reads the mask its +/// `Value::Mask(op)` names — a scratch slot, or for a bare-plane filter the +/// plane itself — presents it as `planes.masks[filter_plane]`, and runs each +/// program in `groups` over the widened planes. Each group program is ONE +/// gated equality and a terminal: the key lane is compared once per group, +/// but only over the live words of the kept filter (the survivor skip), which +/// is the bitmap-index cost model rather than the hash-table one. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GroupPlan { + /// Phase 1: the filter, kept. + pub filter: Program, + /// The plane index phase 2 reads the kept filter at. + pub filter_plane: u16, + /// Phase 2: one program per key value, in key order. + pub groups: Vec, +} + +/// Lower a query to a [`Program`], evaluating predicates in place. +/// +/// # The allocation rule +/// +/// Slots are assigned by a strict post-order walk, and a junction's children +/// are folded left-to-right into the FIRST child's slot. So a filter of depth +/// `d` costs `d + 1` slots, never one per leaf — which is what keeps a wide +/// conjunction from asking for a scratch arena proportional to its width. A +/// resident plane costs nothing: it is read as an operand where it stands. +/// +/// # Errors +/// +/// [`LowerError::EmptyJunction`] for an `And`/`Or` with no children; +/// [`LowerError::TooManySlots`] if the walk needs more than `u16::MAX + 1`. +pub fn lower(q: &Query) -> Result { + lower_with(&q.filter, q.agg, |node, ops| emit_inplace(node, 0, ops)) +} + +/// Lower a query to a [`Program`] whose Boolean skeleton is fused into +/// ternlogs. +/// +/// Every comparison gets its own slot, then the fuser reduces the skeleton — +/// any subtree over three leaves is one [`MaskOp::Ternlog`]. Fewer mask passes +/// than [`lower`]; more slots, proportional to the number of comparisons. +/// +/// # Errors +/// +/// As [`lower`]. +pub fn lower_fused(q: &Query) -> Result { + lower_with(&q.filter, q.agg, emit_fused) +} + +/// Lower a `GROUP BY` to its two-phase [`GroupPlan`]; `filter_plane` is the +/// plane index the caller will bind the kept filter at. +/// +/// # Errors +/// +/// As [`lower`], plus [`LowerError::GroupedBlend`]. +pub fn lower_group_by(g: &GroupBy, filter_plane: u16) -> Result { + if matches!(g.agg, Agg::BlendI32 { .. }) { + return Err(LowerError::GroupedBlend); + } + let filter = lower_with(&g.filter, Agg::Rows, |node, ops| emit_inplace(node, 0, ops))?; + let groups = (0..g.groups) + .map(|v| { + Program::new( + vec![MaskOp::Pred { + pred: Pred::EqU32 { lane: g.key.0, v }, + under: Some(Operand::Plane(filter_plane)), + dst: 0, + }], + terminal_of(g.agg, Operand::Scratch(0)), + ) + }) + .collect(); + Ok(GroupPlan { + filter, + filter_plane, + groups, + }) +} + +/// The shared front half of every lowering: gate the tree, emit it, read the +/// aggregate off wherever the result landed. +fn lower_with( + filter: &Filter, + agg: Agg, + emit: impl FnOnce(&Node, &mut Vec) -> Result, +) -> Result { + let (node, _) = gate_walk(filter, None)?; + let mut ops = Vec::new(); + let mask = emit(&node, &mut ops)?; + Ok(Program::new(ops, terminal_of(agg, mask))) +} + +/// The filter after the survivor-skip pass, before any slot is assigned. +enum Node { + /// A comparison, evaluated under `under` when there is a gate. + Pred { + pred: Pred, + under: Option, + }, + /// A resident plane read as an operand. + Plane(Mask), + And(Vec), + Or(Vec), + Not(Box), +} + +/// Rotate a child whose result is a SUBSET of the gate to the front, and say +/// whether one was found. +/// +/// `flags[i]` is `gate_walk`'s "vanishes with the gate", which is exactly +/// "this child's result is a subset of the gate". Putting such a child first +/// makes the running accumulator a subset of the gate from the first fold +/// onward, which keeps the accumulator a plane operand and costs no scratch +/// slot. +/// +/// ⊘ This read "that is the precondition [`emit_gated`] needs before it may +/// narrow later comparisons onto the accumulator instead of onto the gate". +/// Since `b7e6cef` `emit_gated` does not narrow onto the accumulator at all +/// while a plane gate is live — the plane always wins — so the rotation is an +/// optimisation, not a precondition. See the disable run recorded on +/// [`Filter::and_by_skip`]. +/// +/// Shared by BOTH `AND` arms of [`gate_walk`] — the already-gated one and the +/// one that establishes a gate — because a first version rotated only in the +/// second, and a gated `AND` nested inside a gated `AND` reproduced the same +/// wrong answer one level down. Two spellings of one rule is how that happens +/// twice. +fn hoist_gate_subset(nodes: &mut [Node], flags: &mut [bool]) -> bool { + match flags.iter().position(|&v| v) { + Some(i) => { + nodes.swap(0, i); + flags.swap(0, i); + true + } + None => false, + } +} + +/// Walk `f` under `gate`, establishing at most one gate per `AND` path. +/// +/// Returns the node and whether it is identically zero wherever the gate is +/// zero — the condition under which an `AND` may drop its gate plane as a +/// leaf (see the crate doc's soundness argument). +fn gate_walk(f: &Filter, gate: Option) -> Result<(Node, bool), LowerError> { + Ok(match f { + Filter::Cmp(col, cmp) => ( + Node::Pred { + pred: pred_of(*col, *cmp), + under: gate, + }, + gate.is_some(), + ), + Filter::Plane(m) => (Node::Plane(*m), gate == Some(*m)), + // Gating passes THROUGH a negation (the rewrite is sound for any + // remainder while the gate stays a leaf), but `!x` is 1 where the + // gate is 0, so a negation never lets the gate be dropped. + Filter::Not(inner) => (Node::Not(Box::new(gate_walk(inner, gate)?.0)), false), + Filter::Or(parts) => { + let (nodes, flags) = walk_all(parts, gate)?; + (Node::Or(nodes), flags.iter().all(|&v| v)) + } + Filter::And(parts) => { + if let Some(g) = gate { + // Already gated from above: this AND's own planes are plain + // leaves; one gate per comparison is all `under` can carry. + let (mut nodes, mut flags) = walk_all(parts, Some(g))?; + let subset = hoist_gate_subset(&mut nodes, &mut flags); + (Node::And(nodes), subset) + } else { + let found = parts.iter().find_map(|p| match p { + Filter::Plane(m) => Some(*m), + _ => None, + }); + let Some(g) = found else { + let (nodes, _) = walk_all(parts, None)?; + return Ok((Node::And(nodes), false)); + }; + let mut nodes = Vec::with_capacity(parts.len()); + let mut flags = Vec::with_capacity(parts.len()); + let mut gate_taken = false; + for p in parts { + if !gate_taken && matches!(p, Filter::Plane(m) if *m == g) { + gate_taken = true; + continue; + } + let (n, v) = gate_walk(p, Some(g))?; + nodes.push(n); + flags.push(v); + } + // The gate is implied by any child that vanishes with it; + // otherwise it must be read as a leaf of its own. + // + // When it IS implied, the vanishing child is rotated to the + // FRONT, and that is load-bearing rather than tidy. `vanishes` + // means exactly "this child's result is a subset of the gate", + // so putting one first makes the running accumulator a subset + // of the gate from the first fold onward — which is what lets + // [`emit_gated`] narrow later comparisons onto the accumulator + // without losing the gate. + // + // ⊘ This said "Without the rotation this is a silent WRONG + // ANSWER, not a missed optimisation". TRUE BEFORE `b7e6cef`, + // FALSE AFTER: the plane now gates every child wherever it + // sits, so the `AND` emits `⋂ children` where each flagged + // child is `plane & pred ⊆ plane`, and `⋂ children ⊆ plane` + // holds for ANY ordering. What survives is slot economy. + // The original measurement, kept because it is what the + // rotation was built from: + // `alpha AND focus AND v < 50`, the foreign plane `focus` sat + // first, the accumulator therefore started as `focus` (which is + // NOT a subset of `alpha`), the comparison was gated on that + // instead of on `alpha`, and `alpha` — already dropped as + // "implied" — was nowhere in the program. The per-row oracle + // caught it immediately. + // + // AND is commutative, so the rotation costs nothing + // semantically. It is also the first place in this crate where + // term ORDER changes the emitted program, which is the + // mechanism row A1 is about. + if !hoist_gate_subset(&mut nodes, &mut flags) { + nodes.insert(0, Node::Plane(g)); + } + (Node::And(nodes), false) + } + } + }) +} + +/// [`gate_walk`] over every child of a junction, refusing an empty one. +/// +/// One of THREE spellings of that refusal — the others are the +/// `acc.ok_or(EmptyJunction)` in [`emit_inplace`] and in [`assign_slots`]. +/// Measured: disabling any ONE leaves the suite green, because the other two +/// still catch it; disabling all three turns +/// `an_empty_junction_is_refused_rather_than_folded_to_an_identity` red. So +/// the refusal is load-bearing and the redundancy is deliberate — each +/// spelling guards a different stage (the gate walk, the in-place emitter, +/// the fused emitter) and none may be removed as "obviously dead" on the +/// strength of its own disable run coming back green. +/// +/// The early one is not merely belt-and-braces: without it an empty `Or` +/// reaches `flags.iter().all(..)` over an EMPTY vector, which is vacuously +/// `true`, and reports that it vanishes with the gate — letting a parent +/// `AND` drop a gate plane it should have kept. The program is refused +/// downstream either way, so nothing observable changes today; it is a +/// vacuous-truth corner not worth leaving open. +fn walk_all(parts: &[Filter], gate: Option) -> Result<(Vec, Vec), LowerError> { + if parts.is_empty() { + return Err(LowerError::EmptyJunction); + } + let mut nodes = Vec::with_capacity(parts.len()); + let mut flags = Vec::with_capacity(parts.len()); + for p in parts { + let (n, v) = gate_walk(p, gate)?; + nodes.push(n); + flags.push(v); + } + Ok((nodes, flags)) +} + +/// Emit `n` with its result in `dst` (or, for a plane, where it already is), +/// folding junction children into the first child's slot. +fn emit_inplace(n: &Node, dst: u16, ops: &mut Vec) -> Result { + emit_gated(n, dst, None, ops) +} + +/// [`emit_inplace`] with a RUNNING gate: an operand every comparison beneath +/// `n` is evaluated `under`. +/// +/// # The accumulator gate, and why it is the one that makes order matter +/// +/// The plane gate ([`gate_walk`]) skips words where a RESIDENT mask is empty. +/// This is the other one: inside an `AND`, once the first `k` conjuncts have +/// been folded into `dst`, conjunct `k + 1` is only consulted where that +/// partial result still has a survivor. So the gate NARROWS as the conjunction +/// proceeds, and a selective term early shrinks the live-word count of every +/// term after it. +/// +/// Soundness is the same identity the plane gate uses — `g ∧ rest(X) = +/// g ∧ rest(g ∧ X)` — so it passes through `OR` and `NOT` alike. What it may +/// NOT do is let anything be dropped: the accumulator is a real operand of the +/// `AND`, never elided, so there is no analog here of the plane's +/// vanishes-with-the-gate rule. +/// +/// It applies to `AND` only. Under an `OR`, `acc | p` depends on `p` exactly +/// where `acc` is ZERO — gating there would discard the bits that matter and +/// quietly answer `acc`. That asymmetry is the same one `lgj-abi`'s +/// `plan_lower` documents, and it is the correctness question in both. +fn emit_gated( + n: &Node, + dst: u16, + acc_gate: Option, + ops: &mut Vec, +) -> Result { + match n { + Node::Pred { pred, under } => { + // A `Pred` carries exactly ONE `under`, so when both a plane gate + // and an accumulator are available this is a choice, not a union + // — and the choice is the PLANE, always. + // + // An earlier version preferred the accumulator on the reasoning + // that it is strictly narrower, "because the plane was the AND's + // first conjunct, so it is already folded into the accumulator". + // That holds only inside the AND that ESTABLISHED the plane, + // where [`hoist_gate_subset`] arranges it. It is false the moment + // an OUTER conjunction with no plane of its own wraps an inner one + // that has: + // + // ```text + // P1 AND (Plane(focus) AND P2) + // ``` + // + // The outer AND finds no plane among its own parts, so it + // establishes no gate and simply folds; the inner AND establishes + // `focus`, gates P2 under it, sees P2 vanish and DROPS the plane + // as implied. Then the outer emitter hands its accumulator `P1` + // down, this line preferred it over `focus`, and `focus` — already + // elided — appeared nowhere in the program. Measured on a 512-row + // fixture: the oracle selects 29 rows, the emitted program + // selected 204, and the op list was `GtI32 AND NeU32` with no + // trace of the plane. A silent wrong answer, not a slow one. + // Pinned by `a_nested_plane_survives_an_outer_accumulator`. + // + // Preferring the plane is sound unconditionally: the emitted mask + // is `plane & pred`, and the enclosing junction intersects the + // accumulator afterwards anyway, so `acc & (plane & pred)` is + // exactly the wanted value. What it costs is the EXTRA narrowing + // the accumulator would have given inside a plane's own AND — the + // plane still gates there, just less tightly than it could. That + // cost is named in `ISSUES.md` rather than traded against a + // correctness hole, and A1's lever is untouched because a + // conjunction of plain comparisons carries no plane at all: those + // preds take the `None` arm below and gate on the accumulator. + let gate = under.map(|m| Operand::Plane(m.0)).or(acc_gate); + ops.push(MaskOp::Pred { + pred: *pred, + under: gate, + dst, + }); + Ok(Operand::Scratch(dst)) + } + Node::Plane(m) => Ok(Operand::Plane(m.0)), + Node::Not(inner) => { + let a = emit_gated(inner, dst, acc_gate, ops)?; + ops.push(MaskOp::Not { a, dst }); + Ok(Operand::Scratch(dst)) + } + Node::And(parts) | Node::Or(parts) => { + let is_and = matches!(n, Node::And(_)); + let mut acc: Option = None; + for part in parts { + // The first child lands in `dst`; every sibling after it + // borrows the next slot up in turn, so width costs one slot, + // not one per child. + let slot = if acc.is_none() { + dst + } else { + dst.checked_add(1).ok_or(LowerError::TooManySlots { + needed: usize::from(dst) + 2, + })? + }; + // Inside an AND, every child after the first is gated on the + // partial result. Inside an OR the inherited gate is passed + // through unchanged — an OR may not gate on its own + // accumulator, but it is still inside whatever AND encloses + // it, and that gate remains sound. + let child_gate = match (is_and, acc) { + (true, Some(a)) => Some(a), + _ => acc_gate, + }; + let b = emit_gated(part, slot, child_gate, ops)?; + acc = Some(match acc { + None => b, + Some(a) => { + ops.push(if is_and { + MaskOp::And { a, b, dst } + } else { + MaskOp::Or { a, b, dst } + }); + Operand::Scratch(dst) + } + }); + } + acc.ok_or(LowerError::EmptyJunction) + } + } +} + +/// Emit `n` with every comparison in its own slot and the skeleton fused. +fn emit_fused(n: &Node, ops: &mut Vec) -> Result { + let expr = assign_slots(n, ops)?; + let first_free = u16::try_from(ops.len()).map_err(|_| LowerError::TooManySlots { + needed: ops.len() + 1, + })?; + match fuse(&expr, first_free) { + Ok(fused) => { + ops.extend(fused.ops); + Ok(fused.result) + } + Err(FuseError::SlotOverflow) => Err(LowerError::TooManySlots { + needed: MAX_SCRATCH_SLOTS as usize + 1, + }), + } +} + +/// Give each comparison the next slot and mirror the skeleton as a +/// [`BoolExpr`] over the resulting operands (n-ary junctions left-folded). +fn assign_slots(n: &Node, ops: &mut Vec) -> Result { + match n { + Node::Pred { pred, under } => { + let dst = u16::try_from(ops.len()).map_err(|_| LowerError::TooManySlots { + needed: ops.len() + 1, + })?; + ops.push(MaskOp::Pred { + pred: *pred, + under: under.map(|m| Operand::Plane(m.0)), + dst, + }); + Ok(BoolExpr::Leaf(Operand::Scratch(dst))) + } + Node::Plane(m) => Ok(BoolExpr::Leaf(Operand::Plane(m.0))), + Node::Not(inner) => Ok(BoolExpr::Not(Box::new(assign_slots(inner, ops)?))), + Node::And(parts) | Node::Or(parts) => { + let is_and = matches!(n, Node::And(_)); + let mut acc: Option = None; + for part in parts { + let e = assign_slots(part, ops)?; + acc = Some(match acc { + None => e, + Some(a) => { + if is_and { + BoolExpr::And(Box::new(a), Box::new(e)) + } else { + BoolExpr::Or(Box::new(a), Box::new(e)) + } + } + }); + } + acc.ok_or(LowerError::EmptyJunction) + } + } +} + +/// `agg` read over `mask`. +fn terminal_of(agg: Agg, mask: Operand) -> Terminal { + match agg { + Agg::Count => Terminal::Count { mask }, + Agg::Any => Terminal::Any { mask }, + Agg::All => Terminal::All { mask }, + Agg::SumI32(c) => Terminal::MaskedSumI32 { mask, lane: c.0 }, + Agg::MinI32(c) => Terminal::MaskedMinI32 { mask, lane: c.0 }, + Agg::MaxI32(c) => Terminal::MaskedMaxI32 { mask, lane: c.0 }, + Agg::Rows => Terminal::Keep { mask }, + Agg::BlendI32 { then, els } => Terminal::BlendI32 { + mask, + then: then.0, + els: els.0, + }, + } +} + +/// One comparison → one `Pred`. Exhaustive, and the compiler keeps it so. +fn pred_of(col: Col, cmp: Cmp) -> Pred { + let lane = col.0; + match cmp { + Cmp::EqI32(v) => Pred::EqI32 { lane, v }, + Cmp::NeI32(v) => Pred::NeI32 { lane, v }, + Cmp::LtI32(t) => Pred::LtI32 { lane, t }, + Cmp::LeI32(t) => Pred::LeI32 { lane, t }, + Cmp::GtI32(t) => Pred::GtI32 { lane, t }, + Cmp::GeI32(t) => Pred::GeI32 { lane, t }, + Cmp::EqU32(v) => Pred::EqU32 { lane, v }, + Cmp::NeU32(v) => Pred::NeU32 { lane, v }, + Cmp::MatchU32 { pattern, care } => Pred::MatchU32 { + lane, + pattern, + care, + }, + Cmp::MatchU64 { pattern, care } => Pred::MatchU64 { + lane, + pattern, + care, + }, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use lance_graph_mask_risc::{ + execute, materialize_rows, reference_execute, scratch_words_for, words_for, LaneRef, + Planes, Scratch, Value, + }; + + const N: usize = 1000; + + const VALS: Col = Col(0); + const CLASS: Col = Col(1); + const ALT: Col = Col(2); + const ADDR: Col = Col(3); + const ALPHA: Mask = Mask(0); + const FOCUS: Mask = Mask(1); + + /// Three lanes and two resident planes, every tail bit clean. + struct Fx { + vals: Vec, + classes: Vec, + alt: Vec, + /// An ADDRESS-ORDERED lane: row `i` holds address `i << 8`, so the + /// top bits are a trie prefix and a prefix predicate must select a + /// contiguous row range. That ordering is the fixture's whole point — + /// on an unordered lane a prefix is just an equality with holes, and + /// the range claim would be untestable. + addr: Vec, + masks: Vec>, + } + + fn plane(n: usize, set: impl Fn(usize) -> bool) -> Vec { + let mut words = vec![0u64; words_for(n)]; + for r in (0..n).filter(|&r| set(r)) { + words[r / 64] |= 1u64 << (r % 64); + } + words + } + + impl Fx { + fn new(n: usize) -> Self { + let vals = (0..n) + .map(|i| ((i as i64 * 7) % 401 - 200) as i32) + .collect(); + let classes = (0..n).map(|i| (i % 5) as u32).collect(); + let alt = (0..n).map(|i| ((i as i64 * 13) % 89 - 44) as i32).collect(); + let addr = (0..n).map(|i| (i as u64) << 8).collect(); + let masks = vec![plane(n, |r| r % 3 != 0), plane(n, |r| r % 7 == 0)]; + Fx { + vals, + classes, + alt, + addr, + masks, + } + } + + fn n(&self) -> usize { + self.vals.len() + } + + fn i32_at(&self, col: Col, row: usize) -> i32 { + match col { + VALS => self.vals[row], + ALT => self.alt[row], + other => panic!("{other:?} is not a signed lane of the fixture"), + } + } + + fn u32_at(&self, col: Col, row: usize) -> u32 { + match col { + CLASS => self.classes[row], + other => panic!("{other:?} is not an unsigned lane of the fixture"), + } + } + + fn u64_at(&self, col: Col, row: usize) -> u64 { + match col { + ADDR => self.addr[row], + other => panic!("{other:?} is not a 64-bit lane of the fixture"), + } + } + + fn bit(&self, m: Mask, row: usize) -> bool { + (self.masks[usize::from(m.0)][row / 64] >> (row % 64)) & 1 == 1 + } + + /// The INDEPENDENT reading — a plain row loop that shares no code + /// with the lowering. This is the oracle: a filter tree walked per + /// row, which is exactly the shape this crate refuses to ship, + /// written here because a test oracle is the one place it is + /// licensed. + fn oracle(&self, f: &Filter, row: usize) -> bool { + match f { + Filter::Cmp(col, cmp) => match *cmp { + Cmp::EqI32(x) => self.i32_at(*col, row) == x, + Cmp::NeI32(x) => self.i32_at(*col, row) != x, + Cmp::LtI32(x) => self.i32_at(*col, row) < x, + Cmp::LeI32(x) => self.i32_at(*col, row) <= x, + Cmp::GtI32(x) => self.i32_at(*col, row) > x, + Cmp::GeI32(x) => self.i32_at(*col, row) >= x, + Cmp::EqU32(x) => self.u32_at(*col, row) == x, + Cmp::NeU32(x) => self.u32_at(*col, row) != x, + Cmp::MatchU32 { pattern, care } => { + (self.u32_at(*col, row) ^ pattern) & care == 0 + } + Cmp::MatchU64 { pattern, care } => { + (self.u64_at(*col, row) ^ pattern) & care == 0 + } + }, + Filter::Plane(m) => self.bit(*m, row), + Filter::And(ps) => ps.iter().all(|p| self.oracle(p, row)), + Filter::Or(ps) => ps.iter().any(|p| self.oracle(p, row)), + Filter::Not(p) => !self.oracle(p, row), + } + } + + fn rows(&self, f: &Filter) -> Vec { + (0..self.n()).filter(|&r| self.oracle(f, r)).collect() + } + + fn with_planes(&self, extra: &[Vec], f: impl FnOnce(&Planes<'_>) -> R) -> R { + let lanes = [ + LaneRef::I32(&self.vals), + LaneRef::U32(&self.classes), + LaneRef::I32(&self.alt), + LaneRef::U64(&self.addr), + ]; + let masks: Vec<&[u64]> = self.masks.iter().chain(extra).map(Vec::as_slice).collect(); + f(&Planes { + n_rows: self.n(), + masks: &masks, + lanes: &lanes, + }) + } + + /// The executor, on a scratch sized exactly from the program. + fn exec(&self, program: &Program, extra: &[Vec], out: Option<&mut [i32]>) -> Value { + self.with_planes(extra, |planes| { + let words = words_for(self.n()); + let slots = program.scratch_slots as usize; + let mut buf = vec![0u64; scratch_words_for(words, slots).expect("sized")]; + let mut scratch = Scratch::over(&mut buf, words, slots).expect("carves"); + execute(program, planes, &mut scratch, out).expect("runs") + }) + } + + /// The executor on a `Keep` program, copying the kept mask out of + /// wherever `Value::Mask` says it landed. + fn exec_mask(&self, program: &Program, extra: &[Vec]) -> Vec { + self.with_planes(extra, |planes| { + let words = words_for(self.n()); + let slots = program.scratch_slots as usize; + let mut buf = vec![0u64; scratch_words_for(words, slots).expect("sized")]; + let mut scratch = Scratch::over(&mut buf, words, slots).expect("carves"); + match execute(program, planes, &mut scratch, None).expect("runs") { + Value::Mask(Operand::Scratch(i)) => scratch.slot(i).expect("written").to_vec(), + Value::Mask(Operand::Plane(p)) => planes.masks[usize::from(p)].to_vec(), + other => panic!("not a kept mask: {other:?}"), + } + }) + } + + /// mask-risc's own row-at-a-time oracle — a second independent arm. + fn reference(&self, program: &Program, out: Option<&mut [i32]>) -> Value { + self.with_planes(&[], |planes| { + reference_execute(program, planes, out).expect("runs") + }) + } + + fn count(&self, f: &Filter) -> usize { + let q = Query { + filter: f.clone(), + agg: Agg::Count, + }; + match self.exec(&lower(&q).expect("lowers"), &[], None) { + Value::Count(c) => c, + other => panic!("not a count: {other:?}"), + } + } + } + + /// FAILS IF: an outer conjunction's accumulator replaces a plane gate + /// established by an INNER conjunction, so the plane — already elided as + /// implied — vanishes from the program entirely. + /// + /// The shape is `P1 AND (Plane(focus) AND P2)`. It is the one the survivor + /// skip's own machinery cannot reach by the route the other gate tests + /// take: the OUTER `AND` finds no plane among its parts, so it establishes + /// no gate, while the INNER one establishes `focus`, sees `P2` vanish + /// under it and drops it. Nothing in the outer scope then carries `focus`. + /// + /// Reported as a P1 by codex on PR #1235 and reproduced before it was + /// believed: the oracle selected **29** rows of 512 and the emitted + /// program selected **204**, its op list `GtI32 AND NeU32` with no trace + /// of the plane. That is the failure mode this asserts against, and the + /// op-list half is what distinguishes "the answer happened to match" from + /// "the gate is actually there". + #[test] + fn a_nested_plane_survives_an_outer_accumulator() { + let fx = Fx::new(512); + let f = Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::and([Filter::plane(FOCUS), Filter::cmp(CLASS, Cmp::NeU32(0))]), + ]); + + // Anti-vacuity, two-sided: the plane must actually exclude rows the + // rest admits, or a lost gate would be invisible in the count. + let without_plane = fx + .rows(&Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ])) + .len(); + let expected = fx.rows(&f).len(); + assert!( + expected > 0 && expected * 2 < without_plane, + "the fixture must make the plane load-bearing: {expected} with it, \ + {without_plane} without" + ); + + assert_eq!( + fx.count(&f), + expected, + "the plane was lost from the program" + ); + + // ...and structurally: SOME operand must still read `FOCUS`. A count + // that happens to agree is not evidence the gate survived. + let prog = lower(&Query { + filter: f, + agg: Agg::Count, + }) + .expect("lowers"); + let reads_focus = prog.ops.iter().any( + |op| matches!(op, MaskOp::Pred { under: Some(Operand::Plane(p)), .. } if *p == FOCUS.0), + ); + assert!( + reads_focus, + "no op reads FOCUS — the nested plane is gone: {:?}", + prog.ops + ); + } + + /// `alpha & ((A & B) | C)` — the vertical slice. + fn slice_filter() -> Filter { + Filter::and([ + Filter::plane(ALPHA), + Filter::or([ + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ]), + Filter::cmp(VALS, Cmp::LtI32(-190)), + ]), + ]) + } + + /// FAILS IF: either lowering and an independent per-row reading of the + /// same filter disagree. + /// + /// This is the whole claim of the crate — that a query expressed as + /// masking ops answers what the query means — so it is checked against an + /// oracle that never sees a `Program`, for the in-place AND the fused + /// lowering, over shapes that exercise every gate rule: a gate over a + /// nested `OR`, a gate that must survive a negation, a gate that must + /// survive a foreign plane, two planes, a plane alone, an IN-list. + #[test] + fn every_lowered_filter_agrees_with_an_independent_per_row_reading() { + let fx = Fx::new(N); + let cases: Vec<(&str, Filter)> = vec![ + ("one leaf", Filter::cmp(VALS, Cmp::GtI32(0))), + ("u32 leaf", Filter::cmp(CLASS, Cmp::EqU32(2))), + ("a plane alone", Filter::plane(ALPHA)), + ( + "and of two", + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(-50)), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ]), + ), + ( + "or of two", + Filter::or([ + Filter::cmp(VALS, Cmp::LtI32(-150)), + Filter::cmp(CLASS, Cmp::EqU32(4)), + ]), + ), + ("not", Filter::negate(Filter::cmp(VALS, Cmp::GeI32(0)))), + ( + "and of four — the width case", + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(-180)), + Filter::cmp(VALS, Cmp::LtI32(180)), + Filter::cmp(CLASS, Cmp::NeU32(3)), + Filter::cmp(CLASS, Cmp::NeU32(1)), + ]), + ), + ( + "nested — or inside and, with a not", + Filter::and([ + Filter::cmp(VALS, Cmp::GeI32(-100)), + Filter::or([ + Filter::cmp(CLASS, Cmp::EqU32(1)), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(100))), + ]), + ]), + ), + ( + "ternary match — no SQL spelling, the substrate had it all along", + Filter::cmp( + CLASS, + Cmp::MatchU32 { + pattern: 0b100, + care: 0b110, + }, + ), + ), + ("gate over a comparison", { + Filter::and([Filter::plane(ALPHA), Filter::cmp(VALS, Cmp::GtI32(0))]) + }), + ("gate over a nested or — the slice", slice_filter()), + ( + "gate over a negation — the gate must stay a leaf", + Filter::and([ + Filter::plane(ALPHA), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(0))), + ]), + ), + ( + "gate over an or with a negated arm", + Filter::and([ + Filter::plane(ALPHA), + Filter::or([ + Filter::cmp(CLASS, Cmp::EqU32(3)), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(-100))), + ]), + ]), + ), + ( + "gate beside a foreign plane — the foreign plane is a leaf", + Filter::and([ + Filter::plane(ALPHA), + Filter::plane(FOCUS), + Filter::cmp(VALS, Cmp::LtI32(50)), + ]), + ), + ( + "two planes or'd, then a comparison", + Filter::and([ + Filter::or([Filter::plane(ALPHA), Filter::plane(FOCUS)]), + Filter::cmp(VALS, Cmp::NeI32(0)), + ]), + ), + ("plane after the comparisons", { + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(-100)), + Filter::cmp(VALS, Cmp::LtI32(100)), + Filter::plane(FOCUS), + ]) + }), + ( + "a gated and nested under a gated and", + Filter::and([ + Filter::plane(ALPHA), + Filter::and([Filter::plane(FOCUS), Filter::cmp(VALS, Cmp::GtI32(-100))]), + ]), + ), + ("not of a plane", Filter::negate(Filter::plane(ALPHA))), + ( + "in-list over the class lane", + Filter::in_u32(CLASS, [1, 3, 3]), + ), + ( + "in-list over the value lane", + Filter::in_i32(VALS, [-200, 5, 100, 12]), + ), + ]; + + for (label, f) in cases { + let expected = fx.rows(&f).len(); + let q = Query { + filter: f, + agg: Agg::Count, + }; + let inplace = lower(&q).expect("lowers in place"); + let fused = lower_fused(&q).expect("lowers fused"); + assert_eq!( + fx.exec(&inplace, &[], None), + Value::Count(expected), + "{label}: the in-place program and the per-row oracle disagree" + ); + assert_eq!( + fx.exec(&fused, &[], None), + Value::Count(expected), + "{label}: the fused program and the per-row oracle disagree" + ); + // Anti-vacuity: agreement on "nothing" or "everything" would hold + // for a lowering that ignored the filter entirely. + assert!( + expected > 0 && expected < N, + "{label} selects {expected}/{N} — a degenerate case proves nothing" + ); + } + } + + /// How `p` gates its comparisons, and whether `g` survives as an operand. + /// + /// Returns `(every comparison gated, some comparison gated on the + /// ACCUMULATOR, g is never read as a Boolean operand)`. + /// + /// The middle field did not exist before accumulator gating: a gate that is + /// a scratch slot is the partial result of the conjunction so far, so it + /// NARROWS as the conjunction proceeds, where a plane gate is fixed. That + /// narrowing is the mechanism row A1 is about. + fn gate_shape(p: &Program, g: Mask) -> (bool, bool, bool) { + let gate = Operand::Plane(g.0); + let mut all_gated = true; + let mut on_accumulator = false; + let mut read_as_leaf = false; + for op in &p.ops { + match *op { + MaskOp::Pred { under, .. } => match under { + Some(Operand::Scratch(_)) => on_accumulator = true, + Some(_) => {} + None => all_gated = false, + }, + MaskOp::And { a, b, .. } + | MaskOp::Or { a, b, .. } + | MaskOp::Xor { a, b, .. } + | MaskOp::AndNot { a, b, .. } => read_as_leaf |= a == gate || b == gate, + MaskOp::Not { a, .. } => read_as_leaf |= a == gate, + MaskOp::Ternlog { a, b, c, .. } => { + read_as_leaf |= a == gate || b == gate || c == gate; + } + } + } + let terminal_reads_gate = matches!( + p.terminal, + Terminal::Count { mask } + | Terminal::Any { mask } + | Terminal::All { mask } + | Terminal::MaskedSumI32 { mask, .. } + | Terminal::MaskedMinI32 { mask, .. } + | Terminal::MaskedMaxI32 { mask, .. } + | Terminal::BlendI32 { mask, .. } + | Terminal::Keep { mask } if mask == gate + ); + ( + all_gated, + on_accumulator, + !read_as_leaf && !terminal_reads_gate, + ) + } + + /// FAILS IF: the survivor skip gates the wrong leaves, drops the gate where + /// the drop is unsound, or stops narrowing onto the accumulator. + /// + /// - **can-fire.** Under `alpha & ((A & B) | C)` every comparison is gated + /// and alpha is never read as a Boolean operand; under the in-place + /// lowering at least one comparison is gated on the ACCUMULATOR. + /// - **can-stay-silent.** Under `alpha & !X` the comparison is still gated, + /// but alpha IS read as a leaf — `!(alpha & X)` is 1 exactly where alpha + /// is 0, so dropping it would be unsound. + /// - **the foreign plane.** A second plane beside the gate stays an + /// ordinary leaf; it does not become a second gate. + /// + /// The differential holds either way; this pins the SHAPE, so a walk that + /// dropped the gate under a negation — or one that quietly stopped gating + /// on the accumulator, which no count would ever notice — fails here. + #[test] + fn the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes() { + for (is_inplace, lowering) in [(true, lower as fn(&Query) -> _), (false, lower_fused)] { + let slice = lowering(&Query { + filter: slice_filter(), + agg: Agg::Count, + }) + .expect("lowers"); + let (gated, on_acc, dropped) = gate_shape(&slice, ALPHA); + assert!( + gated && dropped, + "the slice: every predicate gated, alpha implied: {:?}", + slice.ops + ); + // ⊘ RE-PINNED. This asserted `on_acc` for the in-place lowering: + // inside a plane's own AND, later comparisons narrowed onto the + // running accumulator rather than onto the plane. That override + // is what the codex P1 on PR #1235 showed to be UNSOUND once an + // outer conjunction supplies an unrelated accumulator, so + // [`emit_gated`] now always prefers the plane and the assertion + // became false. It is inverted rather than deleted, because the + // property is load-bearing in the other direction: a plane gate + // must never be silently replaced. + // + // A1's lever is untouched and is pinned separately by + // `skip_ordering_moves_the_work_and_never_the_answer`, whose + // conjunction carries no plane — those preds still gate on the + // accumulator. + // + // ⊘ AND IT IS A REAL FALSIFIER ON ONE ARM ONLY. `assign_slots` + // never emits a `Scratch` gate at all, so on the FUSED arm no + // input can make this fail — it passes structurally, not because + // the property holds. The in-place arm is where it bites, and + // where restoring the pre-fix gate line turns it red. Asserting + // the two separately is the point: a single `assert!` run over + // both arms reads as twice the evidence and is once. + if is_inplace { + assert!( + !on_acc, + "a plane-gated comparison narrowed onto the accumulator \ + instead of the plane — the codex-P1 override is back: {:?}", + slice.ops + ); + } else { + assert!( + !slice.ops.iter().any(|op| matches!( + op, + MaskOp::Pred { + under: Some(Operand::Scratch(_)), + .. + } + )), + "the fused arm is expected to carry no scratch gate at \ + all; if that ever changes, the in-place assertion above \ + stops being the only place this is tested: {:?}", + slice.ops + ); + } + + let negated = lowering(&Query { + filter: Filter::and([ + Filter::plane(ALPHA), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(0))), + ]), + agg: Agg::Count, + }) + .expect("lowers"); + let (gated, _, dropped) = gate_shape(&negated, ALPHA); + assert!( + gated && !dropped, + "under a negation: still gated, but alpha stays a leaf: {:?}", + negated.ops + ); + + let foreign = lowering(&Query { + filter: Filter::and([ + Filter::plane(ALPHA), + Filter::plane(FOCUS), + Filter::cmp(VALS, Cmp::LtI32(50)), + ]), + agg: Agg::Count, + }) + .expect("lowers"); + let (gated, _, dropped) = gate_shape(&foreign, ALPHA); + assert!(gated && dropped, "the first plane gates: {:?}", foreign.ops); + assert!( + !gate_shape(&foreign, FOCUS).2, + "the foreign plane is read as a leaf, not gated away: {:?}", + foreign.ops + ); + } + } + + /// FAILS IF: a resident plane costs an op or a slot. + /// + /// `SELECT count(*) FROM t` is the plane's population; the program is + /// empty, the terminal reads the plane, and projection returns the plane + /// itself rather than a copy of it. + #[test] + fn a_resident_plane_alone_is_a_zero_op_program() { + let fx = Fx::new(N); + let expected = fx.rows(&Filter::plane(ALPHA)).len(); + assert!(expected > 0 && expected < N); + for lowering in [lower, lower_fused] { + let count = lowering(&Query { + filter: Filter::plane(ALPHA), + agg: Agg::Count, + }) + .expect("lowers"); + assert!(count.ops.is_empty(), "{:?}", count.ops); + assert_eq!(count.scratch_slots, 0); + assert_eq!( + count.terminal, + Terminal::Count { + mask: Operand::Plane(0) + } + ); + assert_eq!(fx.exec(&count, &[], None), Value::Count(expected)); + + let rows = lowering(&Query { + filter: Filter::plane(ALPHA), + agg: Agg::Rows, + }) + .expect("lowers"); + assert_eq!( + fx.exec(&rows, &[], None), + Value::Mask(Operand::Plane(0)), + "projection of a plane is the plane, not a copy" + ); + } + } + + /// FAILS IF: the fused lowering does not buy passes with slots — or buys + /// nothing. + /// + /// On the slice, in place is two Boolean passes (an AND, then an OR) in + /// two slots; fused is one ternlog in four. Both are pinned, so a fuser + /// that stopped fusing OR an in-place emitter that started allocating per + /// leaf would each fail their own line. + #[test] + fn the_fused_lowering_trades_slots_for_passes() { + let q = Query { + filter: slice_filter(), + agg: Agg::Count, + }; + let inplace = lower(&q).expect("lowers"); + let fused = lower_fused(&q).expect("lowers"); + + assert_eq!(inplace.op_histogram().predicates, 3); + assert_eq!(fused.op_histogram().predicates, 3); + assert_eq!(inplace.op_histogram().mask_passes(), 2); + assert_eq!(fused.op_histogram().mask_passes(), 1); + assert_eq!(fused.op_histogram().ternlog, 1); + assert_eq!(inplace.scratch_slots, 2); + assert_eq!(fused.scratch_slots, 4); + } + + /// FAILS IF: a junction allocates a slot per child in the in-place form. + /// + /// Children fold left-to-right into the first child's slot, so width is + /// free and only DEPTH costs. Without this a 64-wide conjunction would ask + /// for a 64-slot arena — a scratch allocation proportional to the query's + /// text rather than to its shape. The fused form DOES pay width, and that + /// is pinned too: it is the trade, not a defect. + #[test] + fn a_wide_conjunction_costs_one_extra_slot_not_one_per_child() { + let wide = Query { + filter: Filter::and((0..32).map(|i| Filter::cmp(VALS, Cmp::NeI32(i)))), + agg: Agg::Count, + }; + let p = lower(&wide).expect("lowers"); + assert_eq!( + p.scratch_slots, 2, + "32 children must cost depth-2, not 32 slots" + ); + let pf = lower_fused(&wide).expect("lowers"); + assert!( + pf.scratch_slots >= 32, + "fused pays one slot per comparison: {}", + pf.scratch_slots + ); + + // Paired: DEPTH does cost, or the constant above would be vacuous. + let deep = Query { + filter: Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::or([ + Filter::cmp(CLASS, Cmp::EqU32(1)), + Filter::and([ + Filter::cmp(VALS, Cmp::LtI32(50)), + Filter::cmp(CLASS, Cmp::NeU32(2)), + ]), + ]), + ]), + agg: Agg::Count, + }; + let pd = lower(&deep).expect("lowers"); + assert!( + pd.scratch_slots > p.scratch_slots, + "depth must cost more than width: deep={} wide={}", + pd.scratch_slots, + p.scratch_slots + ); + } + + /// FAILS IF: an empty AND/OR — or the `IN ()` that is one — is folded to + /// a constant instead of refused. + /// + /// An empty conjunction is `true` and an empty disjunction is `false`, so + /// whichever identity the code picked would silently be the answer to a + /// query the caller built by accident. + /// + /// Disable note, because a single-site run misreads: the refusal is + /// spelled at THREE sites ([`walk_all`], [`emit_inplace`], + /// [`assign_slots`]) and disabling any one leaves this green — not + /// because the test is vacuous but because the other two still catch it. + /// Disabling all three together is what turns it red. A future session + /// measuring one site and concluding "not load-bearing" would be reading + /// the redundancy, not the guard. + #[test] + fn an_empty_junction_is_refused_rather_than_folded_to_an_identity() { + for f in [ + Filter::And(vec![]), + Filter::Or(vec![]), + Filter::in_u32(CLASS, []), + Filter::in_i32(VALS, []), + Filter::and([Filter::plane(ALPHA), Filter::Or(vec![])]), + ] { + let q = Query { + filter: f.clone(), + agg: Agg::Count, + }; + assert_eq!(lower(&q), Err(LowerError::EmptyJunction), "{f:?}"); + assert_eq!(lower_fused(&q), Err(LowerError::EmptyJunction), "{f:?}"); + } + } + + /// FAILS IF: an IN-list is anything other than the disjunction it claims + /// to be. + /// + /// Structural equality with the hand-built OR, and agreement with the + /// oracle; duplicates in the set are harmless because they are harmless + /// in an OR. + #[test] + fn an_in_list_is_a_disjunction_of_equalities() { + let fx = Fx::new(N); + let by_hand = Filter::or([ + Filter::cmp(CLASS, Cmp::EqU32(1)), + Filter::cmp(CLASS, Cmp::EqU32(3)), + Filter::cmp(CLASS, Cmp::EqU32(3)), + ]); + let in_list = Filter::in_u32(CLASS, [1, 3, 3]); + assert_eq!(in_list, by_hand); + let expected = (0..N).filter(|&r| [1, 3].contains(&fx.classes[r])).count(); + assert_eq!(fx.count(&in_list), expected); + assert!(expected > 0 && expected < N); + } + + /// FAILS IF: the aggregates and the projections do not read the filter's + /// mask. + /// + /// `Count`/`Any`/`All`/`Sum`/`Min`/`Max`/`Rows`/`Blend` over the SAME + /// filter must be mutually consistent with the per-row oracle — an + /// aggregate wired to the wrong operand would still return a plausible + /// number, and a blend wired to the wrong lane a plausible column. + #[test] + fn every_aggregate_and_projection_reads_the_same_filter_mask() { + let fx = Fx::new(N); + let f = Filter::and([ + Filter::plane(ALPHA), + Filter::cmp(VALS, Cmp::GtI32(-100)), + Filter::cmp(CLASS, Cmp::EqU32(3)), + ]); + let rows = fx.rows(&f); + assert!( + !rows.is_empty() && rows.len() < N, + "fixture must be a proper subset" + ); + + let sum: i64 = rows.iter().map(|&r| i64::from(fx.vals[r])).sum(); + let min = rows.iter().map(|&r| fx.vals[r]).min(); + let max = rows.iter().map(|&r| fx.vals[r]).max(); + + let q = |agg| Query { + filter: f.clone(), + agg, + }; + let run = |agg| fx.exec(&lower(&q(agg)).expect("lowers"), &[], None); + assert_eq!(run(Agg::Count), Value::Count(rows.len())); + assert_eq!(run(Agg::Any), Value::Bool(true)); + assert_eq!(run(Agg::All), Value::Bool(false)); + assert_eq!(run(Agg::SumI32(VALS)), Value::SumI64(sum)); + assert_eq!(run(Agg::MinI32(VALS)), Value::OptI32(min)); + assert_eq!(run(Agg::MaxI32(VALS)), Value::OptI32(max)); + + // Projection: the kept mask materialises to exactly the oracle's rows. + let kept = fx.exec_mask(&lower(&q(Agg::Rows)).expect("lowers"), &[]); + assert_eq!(materialize_rows(&kept, N), rows); + + // CASE: every row reads `then` where the filter holds, `els` where not. + let mut out = vec![0i32; N]; + assert_eq!( + fx.exec( + &lower(&q(Agg::BlendI32 { + then: VALS, + els: ALT + })) + .expect("lowers"), + &[], + Some(&mut out), + ), + Value::Blended + ); + let expected: Vec = (0..N) + .map(|r| { + if fx.oracle(&f, r) { + fx.vals[r] + } else { + fx.alt[r] + } + }) + .collect(); + assert_eq!(out, expected); + assert!( + out.iter().zip(&fx.vals).any(|(o, v)| o != v), + "the blend must actually pick from `els` somewhere" + ); + } + + /// FAILS IF: the two-phase GROUP BY does not partition the filtered rows + /// by key. + /// + /// Per-group counts and sums equal the oracle's, and the counts sum to + /// the ungrouped filtered count: a group program that ignored the key + /// would report the total K times, one that ignored the kept filter + /// would report the unfiltered class sizes, and either breaks the + /// partition identity. + #[test] + fn a_group_by_partitions_the_filtered_rows_by_key() { + let fx = Fx::new(N); + let filter = Filter::and([Filter::plane(ALPHA), Filter::cmp(VALS, Cmp::GtI32(0))]); + let rows = fx.rows(&filter); + let groups = 5u32; + let filter_plane = u16::try_from(fx.masks.len()).expect("fits"); + + for agg in [Agg::Count, Agg::SumI32(VALS)] { + let plan = lower_group_by( + &GroupBy { + filter: filter.clone(), + key: CLASS, + groups, + agg, + }, + filter_plane, + ) + .expect("lowers"); + assert_eq!(plan.groups.len(), groups as usize); + assert_eq!(plan.filter_plane, filter_plane); + + let kept = fx.exec_mask(&plan.filter, &[]); + let extra = vec![kept]; + let results: Vec = plan + .groups + .iter() + .map(|p| fx.exec(p, &extra, None)) + .collect(); + + for (g, value) in results.iter().enumerate() { + let members: Vec = rows + .iter() + .copied() + .filter(|&r| fx.classes[r] == g as u32) + .collect(); + let expected = match agg { + Agg::Count => Value::Count(members.len()), + Agg::SumI32(_) => { + Value::SumI64(members.iter().map(|&r| i64::from(fx.vals[r])).sum()) + } + other => unreachable!("{other:?}"), + }; + assert_eq!(*value, expected, "group {g}"); + assert!(!members.is_empty(), "group {g} must be non-empty to count"); + } + if agg == Agg::Count { + let total: usize = results + .iter() + .map(|v| match v { + Value::Count(c) => *c, + other => panic!("{other:?}"), + }) + .sum(); + assert_eq!(total, rows.len(), "the groups partition the filter"); + assert!(rows.len() < N); + } + } + + assert_eq!( + lower_group_by( + &GroupBy { + filter, + key: CLASS, + groups, + agg: Agg::BlendI32 { + then: VALS, + els: ALT + }, + }, + filter_plane, + ), + Err(LowerError::GroupedBlend) + ); + } + + /// FAILS IF: an address prefix does not select a CONTIGUOUS row range, or + /// the range is not the size the prefix arithmetic says it is. + /// + /// This is the matrix's §5.1 claim — *the address IS the trie, so a prefix + /// predicate is a range rather than a sweep* — made checkable instead of + /// asserted. It matters because the whole "better than faithful" argument + /// rests on it: DuckDB has the compressed range in `SequenceVector` and + /// throws it away at `ToUnifiedFormat`, then rebuilds it as a per-row index + /// loop in `DataChunk::Slice`. If V3's prefix did not actually select a + /// range, keeping the range would be keeping nothing. + /// + /// Two-sided, because "selects a contiguous run" alone would hold for a + /// predicate that selected everything: each additional significant bit must + /// HALVE the run, and the widest prefix must select exactly one row. + #[test] + fn an_address_prefix_selects_exactly_a_contiguous_trie_subtree() { + // A POWER-OF-TWO population, deliberately: a subtree can only halve + // cleanly while it still fits inside the population. At N = 1000 the + // widest prefix here selects 1000 rather than 1024, and the halving + // claim then reads as a defect when it is a fixture artifact — which + // is exactly what the first version of this test measured. + const POW2: usize = 1024; + let fx = Fx::new(POW2); + // The lane is `row << 8`, so bit 8 + k of the address is bit k of the + // row index: a prefix of `24 + b` significant bits pins the top `b` + // bits of the row index and leaves `24 - 8 = 16`... stated the way the + // arithmetic actually runs, `care = !0 << (64 - bits)`, and a row + // survives when its address agrees on those bits. + let base_row = 384usize; + let base = fx.addr[base_row]; + // The arithmetic, stated so the expected sizes are DERIVED and not + // fitted to what the code happened to return: row `r` sits at address + // `r << 8`, so row bit `k` is address bit `8 + k`. A prefix of `bits` + // significant bits pins address bits `[64 - bits, 63]`, hence row bits + // `k >= 56 - bits`, leaving `clamp(56 - bits, 0, 10)` of them free — a + // subtree of `2^free` consecutive rows. + let subtree_of = |bits: u32| 1usize << (56u32.saturating_sub(bits).min(10)); + + let mut previous: Option = None; + for bits in [46u32, 47, 48, 49, 50] { + let f = Filter::prefix_u64(ADDR, base, bits); + let rows = fx.rows(&f); + assert!(!rows.is_empty(), "bits={bits} selected nothing"); + + // Contiguous: the selected rows are consecutive, with no holes. + // A sweep over an unordered lane could not satisfy this, which is + // exactly the property being claimed. + let first = rows[0]; + assert!( + rows.iter().enumerate().all(|(i, &r)| r == first + i), + "bits={bits} selected a non-contiguous set: {:?}..", + &rows[..rows.len().min(8)] + ); + assert!( + rows.contains(&base_row), + "bits={bits} must contain the row the prefix was taken from" + ); + + // Each extra significant bit halves the subtree. That is the + // arithmetic the prefix IS; a care mask built wrongly would still + // select a contiguous run, just the wrong one. + assert_eq!( + rows.len(), + subtree_of(bits), + "bits={bits} selected {} rows; the prefix arithmetic says {}", + rows.len(), + subtree_of(bits) + ); + if let Some(prev) = previous { + assert_eq!( + rows.len() * 2, + prev, + "bits={bits} selected {} rows against {prev} for one bit fewer \ + — a trie level must halve", + rows.len() + ); + } + previous = Some(rows.len()); + } + assert_eq!( + previous, + Some(64), + "the 50-bit prefix pins the row index down to a 64-row subtree" + ); + + // The widest prefix is a single address; the empty one is every row. + assert_eq!(fx.count(&Filter::prefix_u64(ADDR, base, 64)), 1); + assert_eq!(fx.count(&Filter::prefix_u64(ADDR, base, 0)), POW2); + } + + /// FAILS IF: the 64-bit ternary match disagrees with an independent per-row + /// reading, or the lowered program does not reach the `U64` lane at all. + /// + /// `Pred::MatchU64` shipped with the IR in PR3 and this crate could not + /// spell it, so a borrowed `LaneRef::U64` — edge targets, ids, addresses — + /// was queryable by nothing. The anti-vacuity half matters more than usual + /// here: a care mask of zero matches every row, which is a real answer and + /// a useless test. + #[test] + fn the_64_bit_ternary_match_agrees_with_a_per_row_reading() { + let fx = Fx::new(N); + let cases: [(&str, u64, u64); 3] = [ + ("one low nibble of the address", 0x300, 0xF00), + ("a sparse care mask", 0x2_0100, 0x3_0100), + ("every bit — exact equality", fx.addr[7], u64::MAX), + ]; + for (label, pattern, care) in cases { + let f = Filter::cmp(ADDR, Cmp::MatchU64 { pattern, care }); + let expected = fx.rows(&f).len(); + assert_eq!(fx.count(&f), expected, "{label}"); + assert!( + expected > 0 && expected < N, + "{label} selects {expected}/{N} — a degenerate case proves nothing" + ); + } + // Composed with the rest of the vocabulary, so the U64 lane is not a + // second world that only works alone. + // 48 significant bits, not 56: at 56 the subtree is a SINGLE row, and + // whether the conjunction is non-empty then turns on whether that one + // row happens to satisfy the other two conjuncts — which it does not, + // so the first version measured 0/1000 and looked like a defect. A + // 256-row subtree is what makes this a composition test rather than a + // coin flip. + let mixed = Filter::and([ + Filter::plane(ALPHA), + Filter::prefix_u64(ADDR, fx.addr[500], 48), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ]); + let expected = fx.rows(&mixed).len(); + assert!(expected > 0 && expected < N, "{expected}/{N}"); + assert_eq!(fx.count(&mixed), expected); + } + + /// FAILS IF: skip-ordering changes the ANSWER, or does not change the + /// ORDER, or stops being stable for equal scores. + /// + /// The safety property is the one that matters most: `AND` is commutative, + /// so reordering may move work and must never move the result. A lowering + /// that reordered children but got the gate chain wrong would produce a + /// different count, and the per-row oracle would catch it — which is why + /// the answer is checked against the oracle and not merely against the + /// unordered lowering. + /// + /// Then the can-fire half: the emitted program must actually differ. A + /// builder that sorted and then lost the order somewhere in `gate_walk` + /// would pass the safety half perfectly. + #[test] + fn skip_ordering_moves_the_work_and_never_the_answer() { + let fx = Fx::new(N); + // Written worst-first on purpose: the permissive term leads, which is + // the shape DuckDB's AdaptiveFilter exists to fix. + let permissive = Filter::cmp(VALS, Cmp::GtI32(-900)); + let moderate = Filter::cmp(CLASS, Cmp::NeU32(0)); + let selective = Filter::cmp(CLASS, Cmp::EqU32(3)); + + let written = Filter::and([permissive.clone(), moderate.clone(), selective.clone()]); + // Scores are the caller's evidence — here, dead words each term would + // leave behind, highest first. + let ordered = Filter::and_by_skip([ + (1, permissive.clone()), + (7, moderate.clone()), + (900, selective.clone()), + ]); + + // Same answer, and the oracle — not the other lowering — is the judge. + let expected = fx.rows(&written).len(); + assert!( + expected > 0 && expected < N, + "{expected}/{N} — a degenerate conjunction proves nothing" + ); + assert_eq!(fx.rows(&ordered).len(), expected); + assert_eq!(fx.count(&written), expected); + assert_eq!(fx.count(&ordered), expected); + + // The order really moved: the most selective term now leads. + assert_eq!( + ordered, + Filter::and([selective.clone(), moderate.clone(), permissive.clone()]), + "and_by_skip must sort descending by score" + ); + assert_ne!(ordered, written, "the fixture must not already be ordered"); + + // ...and the emitted program reflects it: the FIRST predicate is the + // one that was scored highest. Without this the sort could be undone + // between the builder and the lowering and nothing would notice. + let program = lower(&Query { + filter: ordered.clone(), + agg: Agg::Count, + }) + .expect("lowers"); + assert_eq!( + program.ops.first(), + Some(&MaskOp::Pred { + pred: Pred::EqU32 { + lane: CLASS.0, + v: 3 + }, + under: None, + dst: 0, + }), + "the highest-scored term must be the ungated seed: {:?}", + program.ops + ); + + // Stability: equal scores keep the caller's order, so a scorer with no + // information cannot silently permute a hand-tuned conjunction. + assert_eq!( + Filter::and_by_skip([ + (5, permissive.clone()), + (5, moderate.clone()), + (5, selective.clone()), + ]), + written, + "equal scores must be stable" + ); + } + + /// FAILS IF: any arm of the vertical slice disagrees on a 64k slab. + /// + /// `COUNT(alpha & ((A & B) | C))` over 65,536 rows through the per-row + /// oracle, the in-place program on the executor, the fused program on the + /// executor, and both programs on mask-risc's reference evaluator — five + /// readings, one number. Anti-vacuity: the gate binds (the count is + /// strictly below the ungated remainder's) and the answer is a proper + /// subset of alpha. + #[test] + fn the_vertical_slice_agrees_across_every_arm_on_a_64k_slab() { + let fx = Fx::new(1 << 16); + let f = slice_filter(); + let expected = fx.rows(&f).len(); + let alpha = fx.rows(&Filter::plane(ALPHA)).len(); + let ungated = fx + .rows(&Filter::or([ + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ]), + Filter::cmp(VALS, Cmp::LtI32(-190)), + ])) + .len(); + assert!(expected > 0 && expected < alpha, "{expected} of {alpha}"); + assert!( + expected < ungated, + "the gate must bind: {expected} vs {ungated}" + ); + + let q = Query { + filter: f, + agg: Agg::Count, + }; + let inplace = lower(&q).expect("lowers"); + let fused = lower_fused(&q).expect("lowers"); + assert_eq!(fx.exec(&inplace, &[], None), Value::Count(expected)); + assert_eq!(fx.exec(&fused, &[], None), Value::Count(expected)); + assert_eq!(fx.reference(&inplace, None), Value::Count(expected)); + assert_eq!(fx.reference(&fused, None), Value::Count(expected)); + } +} diff --git a/crates/lance-graph/Cargo.toml b/crates/lance-graph/Cargo.toml index 030e642a8..b3201d398 100644 --- a/crates/lance-graph/Cargo.toml +++ b/crates/lance-graph/Cargo.toml @@ -102,6 +102,19 @@ unity-catalog = ["lance-graph-catalog/unity-catalog"] # lancedb: high-level table/connection API (Python/JS SDK equivalent in Rust). # Enables LanceDB-compatible connection strings, table management, and the # vector search API surface that third-party tooling (DuckDB ext, Polars) expects. +# The AWS-NATIVE SDK path (IMDS / SSO / STS assume-role), OFF by default. +# +# Not a capability judgement -- it is off because `aws-smithy-json 0.63.0` does +# not compile against `aws-smithy-types 1.7.0` (full reasoning at the `lance` +# entry in the workspace `Cargo.toml`). S3-COMPATIBLE storage is NOT behind +# this flag: `object_store/aws` is declared directly below and stays on, so +# Tigris/Railway-style endpoints and `lance-graph-hydrate` are unaffected. +# +# Turning this on today re-introduces the broken build. It exists so the +# capability stays addressable by name instead of being silently deleted: +# when upstream fixes the pair -- or when someone forks `aws-smithy-json` +# against 1.7 -- this is the one switch to flip. +aws-sdk = ["lance/aws"] lancedb-sdk = ["dep:lancedb"] # ndarray-hpc: use AdaWorldAPI/ndarray for Fingerprint<256>, CAM-PQ, CLAM, BLAS, ZeckF64. # Disable for minimal builds (CI, wasm, embedded) — falls back to standalone ndarray_bridge.rs.