From 914fd8ef21e224d04fa0386b72e2689b396a3577 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 21:55:34 +0000 Subject: [PATCH 01/25] audit: the 34 NARS recipes against the current substrate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asked: what needs improving. Read all 34 against ndarray::simd_masking_ops and the workspace's own iron rules. Headline: 9 of 9 Datapath recipes name a retired or forbidden realization. The tier that is supposed to BE the masking ops is the one most contaminated — six name VSA bind/unbind/bundle, one names a Hamming sweep, one names the Markov window the whole-book finding retired, and one names classical Berry-Esseen, which I-NOISE-FLOOR-JIRAK forbids outright for this system. So the bucket column was derived FROM the substrate and inherited its staleness. That corrects a claim I made earlier today that bucket was still a valid routing key and only substrate was stale. Cross-checked against the real op list, the recipes the masking algebra can run today are TCP, CAS, TCF, CUR (Gate) and SPP (Control) — the prune / cascade / agreement family, all of it popcount-and-ternlog shaped. Zero of the current Datapath nine. The assignment is inverted relative to the substrate that exists. Also found: ETD says '(no spec)' in its own string; CWS cites the retired singleton BindSpace; ICR cites CausalEdge64 v2 and should ride the v3 change; RTE carries the same Berry-Esseen violation as SDD. Seven items, ordered. Nothing executed — this is a read of the catalogue against the op list. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/audits/nars-34-substrate-audit.md | 81 +++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 .claude/audits/nars-34-substrate-audit.md diff --git a/.claude/audits/nars-34-substrate-audit.md b/.claude/audits/nars-34-substrate-audit.md new file mode 100644 index 000000000..bc4df7501 --- /dev/null +++ b/.claude/audits/nars-34-substrate-audit.md @@ -0,0 +1,81 @@ +# Audit — the 34 NARS recipes against the CURRENT substrate + +> 2026-09-14. Read `recipes.rs` (all 34) against `ndarray::simd_masking_ops` +> (the masking algebra) and the workspace's own iron rules. The question asked: +> **what needs improving.** + +## Headline: the `bucket` column was assigned against a substrate that is gone + +**9 of 9 `Datapath` recipes name a retired or forbidden realization.** The tier +that is supposed to BE the masking ops is the tier most contaminated: + +| id | code | substrate string | verdict | +|---|---|---|---| +| 19 | ARE | `ABBA unbind: A⊗B⊗B=A` | VSA — retired | +| 24 | ZCF | `VSA bind(A,B)` | VSA — retired | +| 25 | HPM | `fingerprint cosine/Hamming sweep` | Hamming — retired | +| 27 | MPC | `bundle = majority-vote-per-bit` | VSA bundle — retired | +| 28 | SSAM | `bind+similarity (Gentner)` | VSA — retired | +| 34 | HKF | `cross-domain bind(A,rel,B)` | VSA — retired | +| 14 | MCT | `… → one fingerprint` | fingerprint — retired | +| 12 | TCA | `Markov ±5` | **±5 was retired** by the whole-book finding (63.3% of same-subject links reach beyond ±5) | +| 32 | SDD | `Berry-Esseen noise floor` | **forbidden by `I-NOISE-FLOOR-JIRAK`** — classical Berry-Esseen is wrong under this system's weak dependence; Jirak 2016 is the rule | + +So the `Datapath` label currently means *"was a VSA kernel"*, not *"is a masking +op"*. **I asserted earlier in this session that `bucket` is still a valid +routing column and only `substrate` was stale. That was wrong** — the bucket was +derived FROM the substrate, so it inherited the staleness. + +## What the masking algebra DOES express today + +Cross-checking the 34 against the real op list (`eq/ne/lt/le/gt/ge_*_to_mask`, +`ternary_match_*`, each with a gated `_under` twin; `mask_and/or/xor/andnot/not`; +`mask_ternlog` over the full 256-table; `mask_any/all`; `masked_sum/min/max_i32`, +`masked_strided_group_sum`): + +| id | code | bucket | expressible as | note | +|---|---|---|---|---| +| 5 | TCP | Gate | a gated predicate — `*_under` | prune = don't evaluate where the gate is empty | +| 8 | CAS | Gate | a cascade of `_under` tiers | the INT1/4/8/32 ladder is the gate chain | +| 20 | TCF | Gate | N masks → agreement | `ternlog` majority + `popcount` | +| 26 | CUR | Gate | coarse-to-fine `_under` chain | same shape as CAS | +| 30 | SPP | Control | N independent masks → agreement | identical algebra to TCF; the ECC/RAID framing is the same majority | + +**The recipes the masking algebra can run TODAY are 4 Gate + 1 Control — and +zero Datapath.** The tier assignment is inverted relative to the substrate that +actually exists. + +## Other defects found + +- **22 ETD** — `"CLAM cluster geometry determines subtasks (no spec)"`. Says so + itself. Unspecified, not merely stale. +- **18 CWS** — `"persistent BindSpace"`. The singleton BindSpace was retired + (`E-MARKOV-TEMPORAL-STREAM-1`: *"in most cases the singleton-BindSpace VSA + substrate is NOT used"*). +- **31 ICR** — `"CausalEdge64 −6 mantissa"` cites the **v2** layout. Truth + (frequency/confidence) is moving to **v3**, so this reference needs re-pinning + with that change rather than after it. +- **1 RTE** — `"Berry-Esseen stop"`. Same iron-rule violation as SDD, in a + Control recipe. + +## What needs improving, in order + +1. **Re-derive `bucket` from the masking algebra**, not from the VSA substrate. + The five above are the honest `Datapath` set today; the current nine are not. +2. **Rewrite the 9 stale `substrate` strings** to name mask-algebra + compositions, or mark them `unrealized` — a string naming a retired kernel + reads as a spec and is worse than an empty one. +3. **Fix the 2 Berry-Esseen citations** (1 RTE, 32 SDD) to Jirak 2016, per + `I-NOISE-FLOOR-JIRAK`. These are iron-rule violations sitting in shipped data. +4. **Re-pin 12 TCA** off the retired ±5 window onto the version-range read + (`QueryReference::at(v, rung)`), which is what replaced it. +5. **Re-pin 18 CWS** off the singleton BindSpace. +6. **31 ICR** rides the CausalEdge64 v3 change. +7. **22 ETD** needs a spec or an explicit `unrealized` mark. + +## What this audit does NOT claim + +It does not say the 34 are wrong as *tactics*. It says their recorded +realization is against a substrate that no longer exists, and that the routing +column derived from it cannot be trusted as a dispatch key until re-derived. +Nothing here was executed; this is a read of the catalogue against the op list. From 657cd18fb7a85cfeb3a26a94a680337dd6061e3d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 21:58:56 +0000 Subject: [PATCH 02/25] quack: the columnar query crate whose operators ARE masking ops MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit You asked for DuckDB as a crate made of masking ops over the SoA. I had been swapping plan_eval's internals instead — one operator of the crate, not the crate. This is the crate. Scan/filter/aggregate expressed so every operator LOWERS to a mask-risc `Program` and is executed by the one evaluator on ndarray::simd's masking algebra. What a columnar engine normally grows and is absent here on purpose: no expression interpreter (a filter IS a `Pred`; walking a tree per batch is the second evaluator), no row iterator (the unit is a mask over n_rows), no per-operator kernel library (a new operator is a new LOWERING, never a new kernel), no `dyn Operator` chain (a plan is one flat op list and one terminal). The rule that keeps it honest: this crate may BUILD a Program and must never EVALUATE one. `execute` is the consumer's call, on a scratch the consumer owns. No ndarray dependency either — the masking algebra is reached THROUGH mask-risc, never beside it. `Cmp` has one variant per masking predicate, which is the point rather than a convenience: the query language's predicate set IS the substrate's, so a predicate this crate cannot spell is one that cannot run. It therefore includes `MatchU32` — the ternary match, which SQL has no spelling for and the substrate has had all along. Slots are assigned post-order with a junction's children folded left-to-right into the first child's slot, so a filter costs DEPTH, never width. A 32-wide conjunction asks for 2 slots. Writing the falsifiers caught a latent bug in my own draft. The no-filter case lowered as `Pred::NeU32{lane:0}` + `Ternlog{0xFF}` — the ternlog ignores its inputs, so the predicate existed only to make slot 0 written, since the IR has no Fill op and an unwritten slot is refused. That typechecks only when lane 0 happens to be U32. Rather than ship it, `Query.filter` is non-optional and the type's doc records the two honest spellings for whoever closes it. Four falsifiers. The load-bearing one runs eight filter shapes — leaf, and, or, not, a four-wide conjunction, a nested or-inside-and, and the ternary match — against an INDEPENDENT per-row oracle that never sees a Program, with an anti-vacuity assertion that every case selects a proper subset, since agreement on "nothing" would hold for a lowering that ignored the filter. The width test carries its paired depth half or its constant would be vacuous. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- Cargo.toml | 1 + crates/lance-graph-quack/Cargo.toml | 15 + crates/lance-graph-quack/src/lib.rs | 554 ++++++++++++++++++++++++++++ 3 files changed, 570 insertions(+) create mode 100644 crates/lance-graph-quack/Cargo.toml create mode 100644 crates/lance-graph-quack/src/lib.rs diff --git a/Cargo.toml b/Cargo.toml index b8d6b9283..3eeb154ed 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -6,6 +6,7 @@ members = [ "crates/lance-graph-planner", "crates/lance-graph-contract", "crates/lance-graph-mask-risc", + "crates/lance-graph-quack", "crates/neural-debug", "crates/lance-graph-callcenter", "crates/lance-graph-archetype", diff --git a/crates/lance-graph-quack/Cargo.toml b/crates/lance-graph-quack/Cargo.toml new file mode 100644 index 000000000..87bd0786d --- /dev/null +++ b/crates/lance-graph-quack/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "lance-graph-quack" +version = "0.1.0" +edition = "2021" +publish = false +description = "A columnar query surface whose OPERATORS ARE MASKING OPS over the V3 SoA. DuckDB's shape — scan, filter, aggregate — with every operator lowering to a lance-graph-mask-risc Program and, under it, ndarray::simd's masking algebra. No expression interpreter, no row iterator, no second evaluator." + +# The ONE evaluator. This crate LOWERS to `Program`; it never evaluates one — +# a `match` here that computed anything would be the duplicate evaluator the +# whole arc exists to avoid. No ndarray dep for the same reason: the masking +# algebra is reached THROUGH mask-risc, never beside it. +[dependencies] +lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } + +[dev-dependencies] diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs new file mode 100644 index 000000000..22c122a05 --- /dev/null +++ b/crates/lance-graph-quack/src/lib.rs @@ -0,0 +1,554 @@ +//! A columnar query surface whose **operators are masking ops**. +//! +//! # What this is, and the shape it deliberately does NOT have +//! +//! DuckDB's operator set — scan, filter, project, aggregate — expressed so +//! that every operator LOWERS to a [`Program`] and is executed by the one +//! evaluator in `lance-graph-mask-risc`, which sits on `ndarray::simd`'s +//! masking algebra. +//! +//! What a columnar engine normally grows, and what is absent here on purpose: +//! +//! | the usual shape | why it is absent | +//! |---|---| +//! | an expression interpreter (`Expr` tree walked per batch) | a filter IS a `Pred`; walking a tree per batch is the second evaluator | +//! | a row iterator / `next()` volcano loop | the unit is a mask over `n_rows`, never a row | +//! | a per-operator kernel library | every operator is a `MaskOp` composition; a new operator is a new LOWERING, never a new kernel | +//! | a physical-plan `dyn Operator` chain | a plan is a `Program` — one flat op list, one terminal | +//! +//! The rule that keeps it honest: **this crate may build a [`Program`] and +//! must never evaluate one.** `execute` is called by the consumer, on a +//! scratch the consumer owns. A single `match` over operators here that +//! computed anything would be the duplicate evaluator the whole arc exists to +//! avoid. +//! +//! # Status +//! +//! Scaffold. The lowering surface below is real and tested; the operator set +//! is deliberately the minimum that proves the shape — **filter and count** — +//! because an operator without a falsifier is a claim. Join, group-by and +//! projection land one at a time, each with the differential that shows it +//! agrees with an independent reading. + +#![forbid(unsafe_code)] + +use lance_graph_mask_risc::{MaskOp, Operand, Pred, Program, Terminal}; + +/// A column reference — an index into [`Planes::lanes`](lance_graph_mask_risc::Planes). +/// +/// Not a name: name resolution is the catalogue's job and this crate has no +/// catalogue. A consumer that has names resolves them before it gets here. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Col(pub u16); + +/// A predicate over one column — the filter's whole vocabulary. +/// +/// One variant per masking op that produces a mask from a value lane. That is +/// not a coincidence and not a coding convenience: **the query language's +/// predicate set IS the masking algebra's predicate set**, so a predicate this +/// crate cannot spell is one the substrate cannot run, and adding one here +/// without adding it below would be the first crack. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Cmp { + /// `col == v` over a signed lane. + EqI32(i32), + /// `col != v`. + NeI32(i32), + /// `col < v`. + LtI32(i32), + /// `col <= v`. + LeI32(i32), + /// `col > v`. + GtI32(i32), + /// `col >= v`. + GeI32(i32), + /// `col == v` over an unsigned lane, exact bitwise. + EqU32(u32), + /// `col != v`. + NeU32(u32), + /// `(col ^ pattern) & care == 0` — the ternary match, which SQL has no + /// spelling for and the substrate has had all along. + MatchU32 { + /// The bits to compare. + pattern: u32, + /// Which bits participate; zero means "don't care". + care: u32, + }, +} + +/// A filter expression: predicates over columns, composed with AND/OR/NOT. +/// +/// Deliberately a tree HERE and never at run time — it is lowered once into a +/// flat [`Program`] and the tree is gone before anything executes. +#[derive(Debug, Clone, PartialEq)] +pub enum Filter { + /// A leaf comparison on one column. + Cmp(Col, Cmp), + /// Every child must hold. + And(Vec), + /// Some child must hold. + Or(Vec), + /// The child must not hold. + Not(Box), +} + +impl Filter { + /// `col ` — the leaf builder, so call sites read as the query does. + pub fn cmp(col: Col, cmp: Cmp) -> Self { + Filter::Cmp(col, cmp) + } + + /// Conjunction. + pub fn and(parts: impl IntoIterator) -> Self { + Filter::And(parts.into_iter().collect()) + } + + /// Disjunction. + pub fn or(parts: impl IntoIterator) -> Self { + Filter::Or(parts.into_iter().collect()) + } + + /// Negation. + /// + /// Named `negate` rather than `not`: an inherent `not` shadows + /// `std::ops::Not` and reads ambiguously at a call site that also uses the + /// operator. Clippy's `should_implement_trait` is right here. + pub fn negate(inner: Filter) -> Self { + Filter::Not(Box::new(inner)) + } +} + +/// Why a query could not be lowered. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum LowerError { + /// An `And`/`Or` with no children. Refused rather than folded to a + /// constant: an empty conjunction is `true` and an empty disjunction is + /// `false`, and a caller that built one by accident wants to hear about it + /// rather than receive whichever identity this crate happened to pick. + EmptyJunction, + /// The program would need more scratch slots than a `u16` can name. + TooManySlots { + /// The count that overflowed. + needed: usize, + }, +} + +impl core::fmt::Display for LowerError { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + match self { + LowerError::EmptyJunction => { + write!(f, "an AND/OR with no children has no non-arbitrary meaning") + } + LowerError::TooManySlots { needed } => { + write!(f, "needs {needed} scratch slots; the address space is u16") + } + } + } +} + +/// What a query asks for. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Agg { + /// How many rows survive the filter. + Count, + /// Whether any row survives. + Any, + /// Whether every row survives. + All, + /// Σ over a signed lane, restricted to surviving rows. + SumI32(Col), + /// min over surviving rows. + MinI32(Col), + /// max over surviving rows. + MaxI32(Col), +} + +/// One query: a filter and what to ask of the rows that pass it. +/// +/// # Why the filter is NOT optional +/// +/// A bare `SELECT count(*)` — every row, no predicate — has no cheap spelling +/// here, and the reason is worth recording rather than papering over: the IR +/// has **no `Fill`/const op**, so "all rows" cannot be written into a scratch +/// slot directly, and a slot that no op has written is refused at validation +/// (`ExecError::ScratchReadBeforeWrite`). +/// +/// A first draft of this crate lowered it as `Pred::NeU32{lane: 0, v: 0}` +/// followed by `Ternlog{imm: 0xFF}` — the ternlog ignores its inputs, so the +/// predicate existed only to make slot 0 written. That is a **latent +/// lane-kind bug**: it typechecks only when lane 0 happens to be `U32`, and +/// fails with a kind error on any schema where it is not. It is recorded here +/// instead of shipped. +/// +/// The honest spellings, for whoever closes this: a tautology over a +/// caller-named lane (`NOT p OR p`, two preds and an `Or`, lane-kind correct +/// by construction), or an unfiltered `Terminal` reading a resident mask plane +/// the caller supplies. Both are real; neither is guessable from this API as +/// it stands, so the type refuses rather than picks. +#[derive(Debug, Clone, PartialEq)] +pub struct Query { + /// The filter. Required — see the type's own doc. + pub filter: Filter, + /// The aggregate. + pub agg: Agg, +} + +/// Lower a query to a [`Program`]. +/// +/// # The allocation rule +/// +/// Slots are assigned by a strict post-order walk, and a junction's children +/// are folded left-to-right into the FIRST child's slot. So a filter of depth +/// `d` costs `d + 1` slots, never one per leaf — which is what keeps a wide +/// conjunction from asking for a scratch arena proportional to its width. +/// +/// # Errors +/// +/// [`LowerError::EmptyJunction`] for an `And`/`Or` with no children; +/// [`LowerError::TooManySlots`] if the walk needs more than `u16::MAX + 1`. +pub fn lower(q: &Query) -> Result { + let mut ops = Vec::new(); + let mut high_water = 0usize; + + lower_filter(&q.filter, 0, &mut ops, &mut high_water)?; + let mask = Operand::Scratch(0); + + let terminal = match q.agg { + Agg::Count => Terminal::Count { mask }, + Agg::Any => Terminal::Any { mask }, + Agg::All => Terminal::All { mask }, + Agg::SumI32(c) => Terminal::MaskedSumI32 { mask, lane: c.0 }, + Agg::MinI32(c) => Terminal::MaskedMinI32 { mask, lane: c.0 }, + Agg::MaxI32(c) => Terminal::MaskedMaxI32 { mask, lane: c.0 }, + }; + + let slots = high_water.max(1); + let scratch_slots = + u32::try_from(slots).map_err(|_| LowerError::TooManySlots { needed: slots })?; + + Ok(Program { + ops, + terminal, + scratch_slots, + }) +} + +/// Lower one filter node into `dst`, tracking the highest slot touched. +fn lower_filter( + f: &Filter, + dst: u16, + ops: &mut Vec, + high_water: &mut usize, +) -> Result<(), LowerError> { + *high_water = (*high_water).max(usize::from(dst) + 1); + match f { + Filter::Cmp(col, cmp) => { + ops.push(MaskOp::Pred { + pred: pred_of(*col, *cmp), + under: None, + dst, + }); + Ok(()) + } + Filter::Not(inner) => { + lower_filter(inner, dst, ops, high_water)?; + ops.push(MaskOp::Not { + a: Operand::Scratch(dst), + dst, + }); + Ok(()) + } + Filter::And(parts) | Filter::Or(parts) => { + let (first, rest) = parts.split_first().ok_or(LowerError::EmptyJunction)?; + lower_filter(first, dst, ops, high_water)?; + // The next slot up is scratch for each sibling in turn, so width + // costs one slot, not one per child. + let tmp = dst.checked_add(1).ok_or(LowerError::TooManySlots { + needed: usize::from(dst) + 2, + })?; + let is_and = matches!(f, Filter::And(_)); + for part in rest { + lower_filter(part, tmp, ops, high_water)?; + ops.push(if is_and { + MaskOp::And { + a: Operand::Scratch(dst), + b: Operand::Scratch(tmp), + dst, + } + } else { + MaskOp::Or { + a: Operand::Scratch(dst), + b: Operand::Scratch(tmp), + dst, + } + }); + } + Ok(()) + } + } +} + +/// One comparison → one `Pred`. Exhaustive, and the compiler keeps it so. +fn pred_of(col: Col, cmp: Cmp) -> Pred { + let lane = col.0; + match cmp { + Cmp::EqI32(v) => Pred::EqI32 { lane, v }, + Cmp::NeI32(v) => Pred::NeI32 { lane, v }, + Cmp::LtI32(t) => Pred::LtI32 { lane, t }, + Cmp::LeI32(t) => Pred::LeI32 { lane, t }, + Cmp::GtI32(t) => Pred::GtI32 { lane, t }, + Cmp::GeI32(t) => Pred::GeI32 { lane, t }, + Cmp::EqU32(v) => Pred::EqU32 { lane, v }, + Cmp::NeU32(v) => Pred::NeU32 { lane, v }, + Cmp::MatchU32 { pattern, care } => Pred::MatchU32 { + lane, + pattern, + care, + }, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use lance_graph_mask_risc::{execute, scratch_words_for, LaneRef, Planes, Scratch, Value}; + + const N: usize = 1000; + + /// Two lanes: a signed value lane and an unsigned class lane. + fn fixture() -> (Vec, Vec) { + let vals: Vec = (0..N as i32).map(|i| (i * 7) % 401 - 200).collect(); + let classes: Vec = (0..N as u32).map(|i| i % 5).collect(); + (vals, classes) + } + + /// The INDEPENDENT reading — a plain row loop that shares no code with the + /// lowering. This is the oracle: a filter tree walked per row, which is + /// exactly the shape this crate refuses to ship, written here because a + /// test oracle is the one place it is licensed. + fn oracle(f: &Filter, vals: &[i32], classes: &[u32], row: usize) -> bool { + match f { + Filter::Cmp(col, cmp) => { + let v = vals[row]; + let c = classes[row]; + match cmp { + Cmp::EqI32(x) => v == *x, + Cmp::NeI32(x) => v != *x, + Cmp::LtI32(x) => v < *x, + Cmp::LeI32(x) => v <= *x, + Cmp::GtI32(x) => v > *x, + Cmp::GeI32(x) => v >= *x, + Cmp::EqU32(x) => c == *x, + Cmp::NeU32(x) => c != *x, + Cmp::MatchU32 { pattern, care } => (c ^ *pattern) & *care == 0, + } + .then_some(*col) + .is_some() + } + Filter::And(ps) => ps.iter().all(|p| oracle(p, vals, classes, row)), + Filter::Or(ps) => ps.iter().any(|p| oracle(p, vals, classes, row)), + Filter::Not(p) => !oracle(p, vals, classes, row), + } + } + + fn run(q: &Query, vals: &[i32], classes: &[u32]) -> Value { + let program = lower(q).expect("lowers"); + let lanes = [LaneRef::I32(vals), LaneRef::U32(classes)]; + let planes = Planes { + n_rows: vals.len(), + masks: &[], + lanes: &lanes, + }; + let words = vals.len().div_ceil(64); + let slots = program.scratch_slots as usize; + let mut buf = vec![0u64; scratch_words_for(words, slots).expect("sized")]; + let mut scratch = Scratch::over(&mut buf, words, slots).expect("carves"); + execute(&program, &planes, &mut scratch, None).expect("runs") + } + + /// FAILS IF: the lowering and an independent per-row reading of the same + /// filter disagree. + /// + /// This is the whole claim of the crate — that a query expressed as + /// masking ops answers what the query means — so it is checked against an + /// oracle that never sees a `Program`. + #[test] + fn every_lowered_filter_agrees_with_an_independent_per_row_reading() { + let (vals, classes) = fixture(); + let cases: Vec<(&str, Filter)> = vec![ + ("one leaf", Filter::cmp(Col(0), Cmp::GtI32(0))), + ("u32 leaf", Filter::cmp(Col(1), Cmp::EqU32(2))), + ( + "and of two", + Filter::and([ + Filter::cmp(Col(0), Cmp::GtI32(-50)), + Filter::cmp(Col(1), Cmp::NeU32(0)), + ]), + ), + ( + "or of two", + Filter::or([ + Filter::cmp(Col(0), Cmp::LtI32(-150)), + Filter::cmp(Col(1), Cmp::EqU32(4)), + ]), + ), + ("not", Filter::negate(Filter::cmp(Col(0), Cmp::GeI32(0)))), + ( + "and of four — the width case", + Filter::and([ + Filter::cmp(Col(0), Cmp::GtI32(-180)), + Filter::cmp(Col(0), Cmp::LtI32(180)), + Filter::cmp(Col(1), Cmp::NeU32(3)), + Filter::cmp(Col(1), Cmp::NeU32(1)), + ]), + ), + ( + "nested — or inside and, with a not", + Filter::and([ + Filter::cmp(Col(0), Cmp::GeI32(-100)), + Filter::or([ + Filter::cmp(Col(1), Cmp::EqU32(1)), + Filter::negate(Filter::cmp(Col(0), Cmp::GtI32(100))), + ]), + ]), + ), + ( + "ternary match — no SQL spelling, the substrate had it all along", + Filter::cmp( + Col(1), + Cmp::MatchU32 { + pattern: 0b100, + care: 0b110, + }, + ), + ), + ]; + + for (label, f) in cases { + let expected = (0..N).filter(|&r| oracle(&f, &vals, &classes, r)).count(); + let q = Query { + filter: f, + agg: Agg::Count, + }; + assert_eq!( + run(&q, &vals, &classes), + Value::Count(expected), + "{label}: the lowered program and the per-row oracle disagree" + ); + // Anti-vacuity: agreement on "nothing" or "everything" would hold + // for a lowering that ignored the filter entirely. + assert!( + expected > 0 && expected < N, + "{label} selects {expected}/{N} — a degenerate case proves nothing" + ); + } + } + + /// FAILS IF: a junction allocates a slot per child. + /// + /// Children fold left-to-right into the first child's slot, so width is + /// free and only DEPTH costs. Without this a 64-wide conjunction would ask + /// for a 64-slot arena — a scratch allocation proportional to the query's + /// text rather than to its shape. + #[test] + fn a_wide_conjunction_costs_one_extra_slot_not_one_per_child() { + let wide = Filter::and((0..32).map(|i| Filter::cmp(Col(0), Cmp::NeI32(i)))); + let p = lower(&Query { + filter: wide, + agg: Agg::Count, + }) + .expect("lowers"); + assert_eq!( + p.scratch_slots, 2, + "32 children must cost depth-2, not 32 slots" + ); + + // Paired: DEPTH does cost, or the constant above would be vacuous. + let deep = Filter::and([ + Filter::cmp(Col(0), Cmp::GtI32(0)), + Filter::or([ + Filter::cmp(Col(1), Cmp::EqU32(1)), + Filter::and([ + Filter::cmp(Col(0), Cmp::LtI32(50)), + Filter::cmp(Col(1), Cmp::NeU32(2)), + ]), + ]), + ]); + let pd = lower(&Query { + filter: deep, + agg: Agg::Count, + }) + .expect("lowers"); + assert!( + pd.scratch_slots > p.scratch_slots, + "depth must cost more than width: deep={} wide={}", + pd.scratch_slots, + p.scratch_slots + ); + } + + /// FAILS IF: an empty AND/OR is folded to a constant instead of refused. + /// + /// An empty conjunction is `true` and an empty disjunction is `false`, so + /// whichever identity the code picked would silently be the answer to a + /// query the caller built by accident. + #[test] + fn an_empty_junction_is_refused_rather_than_folded_to_an_identity() { + for f in [Filter::And(vec![]), Filter::Or(vec![])] { + assert_eq!( + lower(&Query { + filter: f, + agg: Agg::Count + }), + Err(LowerError::EmptyJunction) + ); + } + } + + /// FAILS IF: the aggregates do not read the filter's mask. + /// + /// `Count`/`Any`/`All`/`Sum`/`Min`/`Max` over the SAME filter must be + /// mutually consistent with the per-row oracle — an aggregate wired to the + /// wrong operand would still return a plausible number. + #[test] + fn every_aggregate_reads_the_same_filter_mask() { + let (vals, classes) = fixture(); + let f = Filter::and([ + Filter::cmp(Col(0), Cmp::GtI32(-100)), + Filter::cmp(Col(1), Cmp::EqU32(3)), + ]); + let rows: Vec = (0..N).filter(|&r| oracle(&f, &vals, &classes, r)).collect(); + assert!( + !rows.is_empty() && rows.len() < N, + "fixture must be a proper subset" + ); + + let sum: i64 = rows.iter().map(|&r| i64::from(vals[r])).sum(); + let min = rows.iter().map(|&r| vals[r]).min(); + let max = rows.iter().map(|&r| vals[r]).max(); + + let q = |agg| Query { + filter: f.clone(), + agg, + }; + assert_eq!( + run(&q(Agg::Count), &vals, &classes), + Value::Count(rows.len()) + ); + assert_eq!(run(&q(Agg::Any), &vals, &classes), Value::Bool(true)); + assert_eq!(run(&q(Agg::All), &vals, &classes), Value::Bool(false)); + assert_eq!( + run(&q(Agg::SumI32(Col(0))), &vals, &classes), + Value::SumI64(sum) + ); + assert_eq!( + run(&q(Agg::MinI32(Col(0))), &vals, &classes), + Value::OptI32(min) + ); + assert_eq!( + run(&q(Agg::MaxI32(Col(0))), &vals, &classes), + Value::OptI32(max) + ); + } +} From e4fae74803b72acd99c2d6550418e557e312d53c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 22:03:00 +0000 Subject: [PATCH 03/25] recipes: fix the two classical Berry-Esseen citations (I-NOISE-FLOOR-JIRAK) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two recipes cited the classical IID Berry-Esseen bound in shipped data: #1 RTE ("Berry-Esseen stop") and #32 SDD ("Berry-Esseen noise floor"). That is an iron-rule violation, not a style nit — this system's bits are weakly dependent BY CONSTRUCTION (correlated embedding projections, overlapping role-key slices, a shared 4096-centroid codebook, XOR bundle accumulation), so the classical bound is the wrong theorem and UNDERSTATES the error. A threshold derived from it is loose in the unsafe direction. Both now cite the SHIPPED Jirak surfaces rather than the paper alone, so a reader can run what the string names: #1 RTE -> SigmaTierBands::jirak_p — the Jirak-derived band table, which already REPLACED the sprint-11 hand-tuned linear bands. A "stop" is a threshold, and that is where this workspace's thresholds come from. #32 SDD -> jc Pillar 5 — which MEASURES the weak-dependence sup-error inflation, and whose own result line reads "Jirak's weak-dep rate is the correct citation for this substrate." Neither was caught by any test, because nothing read the `substrate` column at all. `no_recipe_cites_classical_berry_esseen` closes that, with both halves: a can-fire assertion that the predicate really rejects a classical citation (otherwise a typo'd needle passes silently), a can-stay-silent assertion that naming Jirak is NOT itself flagged (or the fix would be indistinguishable from deleting the concept), and an anti-vacuity count that both citations still exist rather than having been emptied. Found by the 34-recipe substrate audit; six further findings from that audit are recorded and not addressed here. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- crates/lance-graph-contract/src/recipes.rs | 70 +++++++++++++++++++++- 1 file changed, 68 insertions(+), 2 deletions(-) diff --git a/crates/lance-graph-contract/src/recipes.rs b/crates/lance-graph-contract/src/recipes.rs index 6897fb432..b948e3bd8 100644 --- a/crates/lance-graph-contract/src/recipes.rs +++ b/crates/lance-graph-contract/src/recipes.rs @@ -93,7 +93,8 @@ pub const RECIPES: [Recipe; 34] = [ mechanism: ParallelIndependence, bucket: Control, spo2cubed: NotCovered, - substrate: "rung depth × Expand/Compress; Berry-Esseen stop", + substrate: + "rung depth × Expand/Compress; Jirak-derived Σ-band stop (SigmaTierBands::jirak_p)", }, Recipe { id: 2, @@ -404,7 +405,7 @@ pub const RECIPES: [Recipe; 34] = [ mechanism: Infrastructure, bucket: Datapath, spo2cubed: NotCovered, - substrate: "Berry-Esseen noise floor + reciprocal A→B,B→A validation", + substrate: "Jirak weak-dep noise floor (jc Pillar 5) + reciprocal A→B,B→A validation", }, Recipe { id: 33, @@ -908,4 +909,69 @@ mod tests { assert_eq!(count(Mechanism::StructuralDivergence), 8); // #4,6,9,13,23,28,31,34 assert_eq!(count(Mechanism::Infrastructure), 14); } + + /// Does this substrate string cite the CLASSICAL (IID) Berry-Esseen bound + /// without naming Jirak's weak-dependence form? + /// + /// Kept as a named function so the guard below can be shown to FIRE — a + /// check that only ever runs over already-clean data proves nothing about + /// itself. + fn cites_classical_berry_esseen(substrate: &str) -> bool { + substrate.contains("Berry-Esseen") && !substrate.contains("Jirak") + } + + /// FAILS IF: a recipe cites classical Berry-Esseen. + /// + /// `I-NOISE-FLOOR-JIRAK` is an iron rule, not a preference: this system's + /// bits are weakly dependent BY CONSTRUCTION (correlated embedding + /// projections, overlapping role-key slices, a shared 4096-centroid + /// codebook, XOR bundle accumulation), so the classical IID bound is the + /// wrong theorem and understates the error. `jc`'s Pillar 5 measures + /// exactly that inflation and its own result line says so: *"Jirak's + /// weak-dep rate is the correct citation for this substrate."* + /// + /// Two recipes violated it in shipped data — #1 RTE ("Berry-Esseen stop") + /// and #32 SDD ("Berry-Esseen noise floor") — and neither was caught by + /// any test, because nothing read the `substrate` column. Both now name + /// the SHIPPED Jirak surfaces (`SigmaTierBands::jirak_p`, which replaced + /// the hand-tuned bands, and `jc` Pillar 5) rather than the paper alone, + /// so the citation points at something a reader can run. + #[test] + fn no_recipe_cites_classical_berry_esseen() { + let bad: Vec<&str> = RECIPES + .iter() + .filter(|r| cites_classical_berry_esseen(r.substrate)) + .map(|r| r.code) + .collect(); + assert!( + bad.is_empty(), + "I-NOISE-FLOOR-JIRAK: these cite classical Berry-Esseen: {bad:?}" + ); + + // CAN-FIRE half: the guard must actually reject the shape it names. + // Without this the assertion above would hold for a predicate that + // never returns true — including one with a typo'd needle. + assert!( + cites_classical_berry_esseen("Berry-Esseen noise floor"), + "the guard must reject a classical citation" + ); + // ...and must NOT reject the corrected form, or the fix would be + // indistinguishable from deleting the concept. + assert!( + !cites_classical_berry_esseen("Jirak weak-dep noise floor (jc Pillar 5)"), + "naming Jirak is the correction, not a second violation" + ); + + // Anti-vacuity: the corpus really does still discuss the noise floor, + // so this is a check over live citations rather than over a column + // someone emptied. + assert!( + RECIPES + .iter() + .filter(|r| r.substrate.contains("Jirak")) + .count() + >= 2, + "both corrected recipes must still carry their citation" + ); + } } From f3df172ff4edba9fad498033828ff739d6065a65 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 22:05:35 +0000 Subject: [PATCH 04/25] =?UTF-8?q?audit:=20correct=20it=20=E2=80=94=20bucke?= =?UTF-8?q?t=20is=20not=20a=20dispatch=20key,=20and=20the=20loco=20bridge?= =?UTF-8?q?=20exists?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two withdrawals, both from reading the modules I had asserted about. 1. The audit's headline treated the stale bucket column as a live routing key. It is not one. recipe_dispatch uses bucket ZERO times — it dispatches by inference type, rung/dispatch_order, and the NaN checklist. recipe_kernels uses it exactly once, in the default gate: Bucket::Gate => not-in-FLOW, _ => true. So only the Gate arm is load-bearing, and Gate is the one bucket the audit found accurate. Datapath vs Control is inert metadata nothing reads, which makes the 9 stale strings documentation debt rather than mis-dispatch. I said in session that this made the finding sharper; it makes it softer, and I said it without reading either module. 2. The loco<->34 bridge already exists: lance-graph-ogar's recipe_vocab.rs is RECIPE_OP_BASE = DOMAIN_FLOOR, ids 1..=34 <-> bytes 0x90..=0xB1, op_of / recipe_of, impl Vocabulary for RecipeVocabulary, and ladder_program() -> Vec — the ladder already lowered to a loco program in dispatch_order, with separate awareness and epistemic gates because an unwilling ladder and an unable one are different diagnoses. Its header also answers the dependency question I got wrong: neither zero-dep crate may import the other, so the vocabulary lives in a consumer that deps both. The ogar-loco/src/nars.rs this session started is the wrong home, for a reason already written down. Elevated instead: E-RECIPE-SELECTOR-REACHABILITY-1 — through the shipped saccade selector only 8 of 34 recipes are reachable, all 14 Infrastructure recipes never win, and ICR #31 is permanently shadowed by RCR #4 on a lowest-id tie. That is measured, documented, and a larger fact about the 34 than any stale substrate string. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/audits/nars-34-substrate-audit.md | 52 +++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/.claude/audits/nars-34-substrate-audit.md b/.claude/audits/nars-34-substrate-audit.md index bc4df7501..9c63d2ba9 100644 --- a/.claude/audits/nars-34-substrate-audit.md +++ b/.claude/audits/nars-34-substrate-audit.md @@ -4,6 +4,58 @@ > (the masking algebra) and the workspace's own iron rules. The question asked: > **what needs improving.** +## ⊘ CORRECTION — `bucket` is NOT a dispatch key (read after writing this) + +The audit below says the stale `bucket` column matters because it routes. **It +does not route.** Measured: + +- `recipe_dispatch.rs` — **zero** uses of `bucket`. Dispatch is by + `RecipeInference` (deduction/induction/abduction/revision/counterfactual), + `rung` + `dispatch_order` (from `Tier` + `InferenceType::rung_delta`), and + the `nan_disqualifier` checklist. +- `recipe_kernels.rs` — **one** use, the default `Tactic::gate`: + `match bucket { Bucket::Gate => ctx.gate_state() != GateState::Flow, _ => true }`. + +So only the **`Gate`** arm is load-bearing, and `Gate` is the one bucket this +audit found ACCURATE (TCP/CAS/TCF/CUR are the prune/cascade family the masking +algebra genuinely expresses). `Datapath` vs `Control` is **inert descriptive +metadata that nothing reads.** + +Consequence: the 9 stale `Datapath` strings are **documentation debt, not live +mis-dispatch.** They still want fixing — a string naming a retired kernel reads +as a spec — but item 1 below ("re-derive bucket") is NOT urgent and NOT a +correctness risk. I claimed the opposite in session ("a routing key derived +from a retired substrate is currently deciding what executes") without reading +either module. That claim is withdrawn. + +## ⊘ ALSO: the loco↔34 bridge already exists — do not build it + +`lance-graph-ogar/src/recipe_vocab.rs` is the 34 as loco ops: +`RECIPE_OP_BASE = DOMAIN_FLOOR`, ids `1..=34` ↔ bytes `0x90..=0xB1`, +`op_of` / `recipe_of`, `impl Vocabulary for RecipeVocabulary`, and +`ladder_program() -> Vec` — the ladder already lowered to a loco +program in `dispatch_order`. It carries two gates that answer different +questions (awareness: the kanban census; epistemic: the NaN pothole) with +`refusal_of` reporting which spoke, "because an unwilling ladder and an unable +one are different diagnoses." + +Its header also settles the dependency question correctly: *"`ogar_loco` is +zero-dep by design and `lance_graph_contract` is zero-dep by charter. Neither +may import the other. A vocabulary needs both, so it lives in a consumer that +already depends on both."* An `ogar-loco/src/nars.rs` with a contract dep — +which this session started writing — is the wrong home for a reason already +recorded. + +## The finding that IS serious, and is not mine + +`E-RECIPE-SELECTOR-REACHABILITY-1`, measured and documented in +`recipe_dispatch.rs`'s own header: through the shipped saccade selector +(`materialize::select_tactic`) only **8 of 34 recipes are reachable**, **all 14 +Infrastructure recipes never win**, and **ICR #31 is permanently shadowed by +RCR #4** on a lowest-id tie. The `ladder` mode exists to cover exactly those +three gaps, and `dispatch_mode.rs` routes between the two modes. That is a +larger fact about the 34 than anything in the substrate-string audit below. + ## Headline: the `bucket` column was assigned against a substrate that is gone **9 of 9 `Datapath` recipes name a retired or forbidden realization.** The tier From ea1a3533af894b2e06b3bc724c1e51bd679b25a9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 22:26:13 +0000 Subject: [PATCH 05/25] quack: planes, the survivor skip, projection, fusion, GROUP BY MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scaffold proved the shape with filter+count. This is the operator set, each with the differential that shows it agrees with an independent per-row reading of the same query. `Filter::Plane` — a resident mask plane as a predicate. This is what the scaffold's own doc owed: it recorded that "every row" had no honest spelling, because the IR has no Fill op and the draft's `NeU32{lane:0}` + constant-ternlog trick typechecked only when lane 0 happened to be U32. A plane leaf is correct by construction, costs zero ops and zero slots, and is the truer model besides — the table IS its validity plane, which is why there is no NULL here to be three-valued about. The survivor skip. An AND with a plane child gates every comparison beneath it (`MaskOp::Pred`'s `under`), so a predicate runs only over the 64-row words where the plane has a survivor. The rewrite is sound for any Boolean remainder — g ∧ rest(X) = g ∧ rest(g∧X) — so the gate passes through NOT and OR alike; what does NOT pass is the DROP. The plane leaf can be elided only where the gated remainder is identically zero wherever the gate is: a gated comparison is, an AND is if any child is, an OR is if every child is, a NOT never is. `alpha & !X` is 1 exactly where alpha is 0, so alpha stays a leaf there — pinned two-sided, because a walk that dropped it would be wrong in a way the count alone would catch only on some fixtures. Two lowerings of one meaning. `lower` folds a junction into its first child's slot (depth costs, width is free); `lower_fused` gives every predicate a slot and hands the skeleton to the fuser. On the vertical slice that is 2 passes / 2 slots against 1 pass / 4 slots, both pinned, so a fuser that stopped fusing and an emitter that started allocating per leaf each fail their own line. Both run against the oracle on every fixture. Projection without materialisation: `Agg::Rows` is `Terminal::Keep` — the result IS the mask, the projected columns are lanes the caller already holds, and the one materialiser stays the caller's to invoke. `Agg::BlendI32` is the CASE shape. `IN` is the disjunction of equalities it always was; there is no IN-list kernel because none is needed, and the empty list is refused as the empty OR it is. GROUP BY is two-phase with a mask plane where the hash table would be: keep the filter, bind it as a plane, then one gated equality per key. K programs, not one, and the doc says why — `masked_strided_group_sum` exists in the facade but the IR names no strided operand and no group terminal, so a single-terminal grouping would be a claim the substrate cannot currently keep. A grouped blend is refused rather than answered wrongly: every group program writes the whole `out`, so K of them would leave the last and silently discard the rest. The 64k vertical slice — COUNT(alpha & ((A&B)|C)) — agrees across five readings: the per-row oracle, both programs on the executor, and both on mask-risc's own reference evaluator. Anti-vacuity is two-sided: the answer is a proper subset of alpha AND strictly below the ungated remainder, so a lowering that dropped the gate fails even though its count would still look plausible. 10 tests, fmt + clippy -D warnings clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- crates/lance-graph-quack/src/lib.rs | 1262 ++++++++++++++++++++++----- 1 file changed, 1051 insertions(+), 211 deletions(-) diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index 22c122a05..fd9b1f9a4 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -2,8 +2,8 @@ //! //! # What this is, and the shape it deliberately does NOT have //! -//! DuckDB's operator set — scan, filter, project, aggregate — expressed so -//! that every operator LOWERS to a [`Program`] and is executed by the one +//! DuckDB's operator set — scan, filter, project, aggregate, group — expressed +//! so that every operator LOWERS to a [`Program`] and is executed by the one //! evaluator in `lance-graph-mask-risc`, which sits on `ndarray::simd`'s //! masking algebra. //! @@ -15,6 +15,8 @@ //! | a row iterator / `next()` volcano loop | the unit is a mask over `n_rows`, never a row | //! | a per-operator kernel library | every operator is a `MaskOp` composition; a new operator is a new LOWERING, never a new kernel | //! | a physical-plan `dyn Operator` chain | a plan is a `Program` — one flat op list, one terminal | +//! | a validity bitmap beside the data | the table's validity IS a resident mask plane ([`Filter::Plane`]); there is no separate NULL | +//! | a hash table for GROUP BY | a group is a mask; K groups are K gated equalities over the kept filter ([`lower_group_by`]) | //! //! The rule that keeps it honest: **this crate may build a [`Program`] and //! must never evaluate one.** `execute` is called by the consumer, on a @@ -22,17 +24,49 @@ //! computed anything would be the duplicate evaluator the whole arc exists to //! avoid. //! +//! # The survivor skip — where a scan under a validity plane gets its cost model +//! +//! An `AND` whose children include a resident plane `g` (alpha, focus, a class +//! mask) is lowered so that every comparison beneath it is evaluated `under +//! g`: the predicate runs only over the 64-row words where `g` has a survivor +//! ([`MaskOp::Pred`]'s `under`). The rewrite is sound for ANY Boolean +//! remainder — `g ∧ rest(X₁..Xₙ) = g ∧ rest(g∧X₁, .., g∧Xₙ)`, since a row with +//! `g = 0` gives 0 on both sides and a row with `g = 1` leaves every leaf +//! unchanged — so gating passes through `NOT` and `OR` alike as long as `g` +//! itself stays a leaf. The plane leaf is DROPPED only when the gated +//! remainder is identically zero wherever `g` is zero: a gated comparison is; +//! an `AND` is if any child is; an `OR` is if every child is; a `NOT` never +//! is. `alpha & ((A & B) | C)` therefore costs three gated predicates and one +//! Boolean pass, with alpha never read as an operand at all. +//! +//! # Two lowerings, one meaning +//! +//! [`lower`] evaluates predicates IN PLACE and folds a junction's children +//! into its first child's slot — a filter of depth `d` costs `d + 1` slots, +//! width is free. [`lower_fused`] gives every predicate its own slot and hands +//! the Boolean skeleton to the fuser, which turns any subtree over three +//! leaves into one [`MaskOp::Ternlog`] — fewer mask passes, more slots. The +//! differential suite runs both against the same per-row oracle; a consumer +//! picks by whether it is scratch-bound or pass-bound. +//! //! # Status //! -//! Scaffold. The lowering surface below is real and tested; the operator set -//! is deliberately the minimum that proves the shape — **filter and count** — -//! because an operator without a falsifier is a claim. Join, group-by and -//! projection land one at a time, each with the differential that shows it -//! agrees with an independent reading. +//! Filter (`=`/`<>`/`<`/`<=`/`>`/`>=`/ternary match/`IN`, `AND`/`OR`/`NOT`, +//! resident planes), the aggregates `COUNT`/`EXISTS`/`ALL`/`SUM`/`MIN`/`MAX`, +//! projection ([`Agg::Rows`] keeps the mask, [`Agg::BlendI32`] is the `CASE` +//! shape), and a two-phase `GROUP BY` over a categorical key. Absent, named: +//! the join (`src_mask → hop → dst_mask` is `lance-graph-mask-risc`'s PR5 — +//! there is no `hop` op to lower to yet); a one-terminal `GROUP BY SUM` +//! (`ndarray::simd` ships `masked_strided_group_sum`, but the IR names no +//! strided operand or group terminal, so K programs is the honest spelling +//! today); and everything the IR itself excludes — strings, `ORDER BY`, +//! three-valued NULL. #![forbid(unsafe_code)] -use lance_graph_mask_risc::{MaskOp, Operand, Pred, Program, Terminal}; +use lance_graph_mask_risc::{ + fuse, BoolExpr, FuseError, MaskOp, Operand, Pred, Program, Terminal, MAX_SCRATCH_SLOTS, +}; /// A column reference — an index into [`Planes::lanes`](lance_graph_mask_risc::Planes). /// @@ -41,6 +75,12 @@ use lance_graph_mask_risc::{MaskOp, Operand, Pred, Program, Terminal}; #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] pub struct Col(pub u16); +/// A resident mask plane — an index into +/// [`Planes::masks`](lance_graph_mask_risc::Planes): alpha, focus, a class +/// mask, a kept filter. The table's validity lives here, not in a NULL bit. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Mask(pub u16); + /// A predicate over one column — the filter's whole vocabulary. /// /// One variant per masking op that produces a mask from a value lane. That is @@ -76,14 +116,20 @@ pub enum Cmp { }, } -/// A filter expression: predicates over columns, composed with AND/OR/NOT. +/// A filter expression: predicates over columns and resident planes, composed +/// with AND/OR/NOT. /// /// Deliberately a tree HERE and never at run time — it is lowered once into a /// flat [`Program`] and the tree is gone before anything executes. -#[derive(Debug, Clone, PartialEq)] +#[derive(Debug, Clone, PartialEq, Eq)] pub enum Filter { /// A leaf comparison on one column. Cmp(Col, Cmp), + /// A resident mask plane read as a predicate — the scan's validity, a + /// focus, a class mask. `Filter::Plane(alpha)` alone is `SELECT … FROM t` + /// with no `WHERE`: the table is its validity plane, and a query over + /// every row of it lowers to zero ops. + Plane(Mask), /// Every child must hold. And(Vec), /// Some child must hold. @@ -98,6 +144,11 @@ impl Filter { Filter::Cmp(col, cmp) } + /// A resident plane as a predicate. + pub fn plane(mask: Mask) -> Self { + Filter::Plane(mask) + } + /// Conjunction. pub fn and(parts: impl IntoIterator) -> Self { Filter::And(parts.into_iter().collect()) @@ -116,6 +167,27 @@ impl Filter { pub fn negate(inner: Filter) -> Self { Filter::Not(Box::new(inner)) } + + /// `col IN (set)` over an unsigned lane — a disjunction of equalities, + /// which is exactly what it is; there is no IN-list kernel because none + /// is needed. An empty set is `x IN ()`, refused at lowering for the same + /// reason an empty `OR` is: it is that `OR`. + pub fn in_u32(col: Col, set: impl IntoIterator) -> Self { + Filter::Or( + set.into_iter() + .map(|v| Filter::Cmp(col, Cmp::EqU32(v))) + .collect(), + ) + } + + /// `col IN (set)` over a signed lane. See [`Filter::in_u32`]. + pub fn in_i32(col: Col, set: impl IntoIterator) -> Self { + Filter::Or( + set.into_iter() + .map(|v| Filter::Cmp(col, Cmp::EqI32(v))) + .collect(), + ) + } } /// Why a query could not be lowered. @@ -132,6 +204,10 @@ pub enum LowerError { /// The count that overflowed. needed: usize, }, + /// A `GROUP BY` asked for [`Agg::BlendI32`]. Every group program writes + /// the WHOLE `out` slice, so K groups would leave the last group's blend + /// and silently discard K − 1 — refused rather than answered wrongly. + GroupedBlend, } impl core::fmt::Display for LowerError { @@ -143,6 +219,9 @@ impl core::fmt::Display for LowerError { LowerError::TooManySlots { needed } => { write!(f, "needs {needed} scratch slots; the address space is u16") } + LowerError::GroupedBlend => { + write!(f, "a blend writes the whole output; it cannot be grouped") + } } } } @@ -162,133 +241,366 @@ pub enum Agg { MinI32(Col), /// max over surviving rows. MaxI32(Col), + /// The surviving rows themselves — projection. Nothing is reduced and + /// nothing is copied: the result is the final mask + /// ([`Terminal::Keep`]), and the projected columns are the resident + /// lanes the caller already holds. The one materialiser is + /// `lance_graph_mask_risc::materialize_rows`, and it is the caller's + /// to invoke. + Rows, + /// `CASE WHEN filter THEN then ELSE els END` — written per row into a + /// caller-supplied `out` slice, no compaction ([`Terminal::BlendI32`]). + BlendI32 { + /// The lane read where the filter holds. + then: Col, + /// The lane read where it does not. + els: Col, + }, } /// One query: a filter and what to ask of the rows that pass it. /// -/// # Why the filter is NOT optional -/// -/// A bare `SELECT count(*)` — every row, no predicate — has no cheap spelling -/// here, and the reason is worth recording rather than papering over: the IR -/// has **no `Fill`/const op**, so "all rows" cannot be written into a scratch -/// slot directly, and a slot that no op has written is refused at validation -/// (`ExecError::ScratchReadBeforeWrite`). -/// -/// A first draft of this crate lowered it as `Pred::NeU32{lane: 0, v: 0}` -/// followed by `Ternlog{imm: 0xFF}` — the ternlog ignores its inputs, so the -/// predicate existed only to make slot 0 written. That is a **latent -/// lane-kind bug**: it typechecks only when lane 0 happens to be `U32`, and -/// fails with a kind error on any schema where it is not. It is recorded here -/// instead of shipped. -/// -/// The honest spellings, for whoever closes this: a tautology over a -/// caller-named lane (`NOT p OR p`, two preds and an `Or`, lane-kind correct -/// by construction), or an unfiltered `Terminal` reading a resident mask plane -/// the caller supplies. Both are real; neither is guessable from this API as -/// it stands, so the type refuses rather than picks. -#[derive(Debug, Clone, PartialEq)] +/// The filter is required. There is no unfiltered table in the substrate — +/// every table IS its validity plane — so "every row" is spelled +/// [`Filter::Plane`]`(alpha)`, which lowers to zero ops and reads the plane +/// straight from the terminal. (A first draft of this crate tried to spell it +/// without a plane, as a predicate over lane 0 followed by a constant +/// ternlog; that typechecked only when lane 0 happened to be `U32` — a latent +/// lane-kind bug. The plane leaf is the spelling that is correct by +/// construction.) +#[derive(Debug, Clone, PartialEq, Eq)] pub struct Query { - /// The filter. Required — see the type's own doc. + /// The filter. pub filter: Filter, /// The aggregate. pub agg: Agg, } -/// Lower a query to a [`Program`]. +/// `GROUP BY key` over a low-cardinality unsigned key lane whose values are +/// `0..groups` — the dictionary-encoded / categorical shape. +/// +/// The filter is required for the same reason [`Query`]'s is; the whole +/// table is `Filter::Plane(alpha)`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GroupBy { + /// Rows admitted to any group. + pub filter: Filter, + /// The key lane (`u32`). + pub key: Col, + /// Number of key values; group `g` is the rows whose key equals `g`. + pub groups: u32, + /// The aggregate computed per group. [`Agg::BlendI32`] is refused. + pub agg: Agg, +} + +/// The two-phase plan a [`GroupBy`] lowers to — DuckDB's pipeline break, +/// with a mask plane where the hash table would be. +/// +/// The caller runs `filter` (a [`Terminal::Keep`]), reads the mask its +/// `Value::Mask(op)` names — a scratch slot, or for a bare-plane filter the +/// plane itself — presents it as `planes.masks[filter_plane]`, and runs each +/// program in `groups` over the widened planes. Each group program is ONE +/// gated equality and a terminal: the key lane is compared once per group, +/// but only over the live words of the kept filter (the survivor skip), which +/// is the bitmap-index cost model rather than the hash-table one. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GroupPlan { + /// Phase 1: the filter, kept. + pub filter: Program, + /// The plane index phase 2 reads the kept filter at. + pub filter_plane: u16, + /// Phase 2: one program per key value, in key order. + pub groups: Vec, +} + +/// Lower a query to a [`Program`], evaluating predicates in place. /// /// # The allocation rule /// /// Slots are assigned by a strict post-order walk, and a junction's children /// are folded left-to-right into the FIRST child's slot. So a filter of depth /// `d` costs `d + 1` slots, never one per leaf — which is what keeps a wide -/// conjunction from asking for a scratch arena proportional to its width. +/// conjunction from asking for a scratch arena proportional to its width. A +/// resident plane costs nothing: it is read as an operand where it stands. /// /// # Errors /// /// [`LowerError::EmptyJunction`] for an `And`/`Or` with no children; /// [`LowerError::TooManySlots`] if the walk needs more than `u16::MAX + 1`. pub fn lower(q: &Query) -> Result { - let mut ops = Vec::new(); - let mut high_water = 0usize; + lower_with(&q.filter, q.agg, |node, ops| emit_inplace(node, 0, ops)) +} + +/// Lower a query to a [`Program`] whose Boolean skeleton is fused into +/// ternlogs. +/// +/// Every comparison gets its own slot, then the fuser reduces the skeleton — +/// any subtree over three leaves is one [`MaskOp::Ternlog`]. Fewer mask passes +/// than [`lower`]; more slots, proportional to the number of comparisons. +/// +/// # Errors +/// +/// As [`lower`]. +pub fn lower_fused(q: &Query) -> Result { + lower_with(&q.filter, q.agg, emit_fused) +} - lower_filter(&q.filter, 0, &mut ops, &mut high_water)?; - let mask = Operand::Scratch(0); +/// Lower a `GROUP BY` to its two-phase [`GroupPlan`]; `filter_plane` is the +/// plane index the caller will bind the kept filter at. +/// +/// # Errors +/// +/// As [`lower`], plus [`LowerError::GroupedBlend`]. +pub fn lower_group_by(g: &GroupBy, filter_plane: u16) -> Result { + if matches!(g.agg, Agg::BlendI32 { .. }) { + return Err(LowerError::GroupedBlend); + } + let filter = lower_with(&g.filter, Agg::Rows, |node, ops| emit_inplace(node, 0, ops))?; + let groups = (0..g.groups) + .map(|v| { + Program::new( + vec![MaskOp::Pred { + pred: Pred::EqU32 { lane: g.key.0, v }, + under: Some(Operand::Plane(filter_plane)), + dst: 0, + }], + terminal_of(g.agg, Operand::Scratch(0)), + ) + }) + .collect(); + Ok(GroupPlan { + filter, + filter_plane, + groups, + }) +} - let terminal = match q.agg { - Agg::Count => Terminal::Count { mask }, - Agg::Any => Terminal::Any { mask }, - Agg::All => Terminal::All { mask }, - Agg::SumI32(c) => Terminal::MaskedSumI32 { mask, lane: c.0 }, - Agg::MinI32(c) => Terminal::MaskedMinI32 { mask, lane: c.0 }, - Agg::MaxI32(c) => Terminal::MaskedMaxI32 { mask, lane: c.0 }, - }; +/// The shared front half of every lowering: gate the tree, emit it, read the +/// aggregate off wherever the result landed. +fn lower_with( + filter: &Filter, + agg: Agg, + emit: impl FnOnce(&Node, &mut Vec) -> Result, +) -> Result { + let (node, _) = gate_walk(filter, None)?; + let mut ops = Vec::new(); + let mask = emit(&node, &mut ops)?; + Ok(Program::new(ops, terminal_of(agg, mask))) +} - let slots = high_water.max(1); - let scratch_slots = - u32::try_from(slots).map_err(|_| LowerError::TooManySlots { needed: slots })?; +/// The filter after the survivor-skip pass, before any slot is assigned. +enum Node { + /// A comparison, evaluated under `under` when there is a gate. + Pred { + pred: Pred, + under: Option, + }, + /// A resident plane read as an operand. + Plane(Mask), + And(Vec), + Or(Vec), + Not(Box), +} - Ok(Program { - ops, - terminal, - scratch_slots, +/// Walk `f` under `gate`, establishing at most one gate per `AND` path. +/// +/// Returns the node and whether it is identically zero wherever the gate is +/// zero — the condition under which an `AND` may drop its gate plane as a +/// leaf (see the crate doc's soundness argument). +fn gate_walk(f: &Filter, gate: Option) -> Result<(Node, bool), LowerError> { + Ok(match f { + Filter::Cmp(col, cmp) => ( + Node::Pred { + pred: pred_of(*col, *cmp), + under: gate, + }, + gate.is_some(), + ), + Filter::Plane(m) => (Node::Plane(*m), gate == Some(*m)), + // Gating passes THROUGH a negation (the rewrite is sound for any + // remainder while the gate stays a leaf), but `!x` is 1 where the + // gate is 0, so a negation never lets the gate be dropped. + Filter::Not(inner) => (Node::Not(Box::new(gate_walk(inner, gate)?.0)), false), + Filter::Or(parts) => { + let (nodes, flags) = walk_all(parts, gate)?; + (Node::Or(nodes), flags.iter().all(|&v| v)) + } + Filter::And(parts) => { + if let Some(g) = gate { + // Already gated from above: this AND's own planes are plain + // leaves; one gate per comparison is all `under` can carry. + let (nodes, flags) = walk_all(parts, Some(g))?; + (Node::And(nodes), flags.iter().any(|&v| v)) + } else { + let found = parts.iter().find_map(|p| match p { + Filter::Plane(m) => Some(*m), + _ => None, + }); + let Some(g) = found else { + let (nodes, _) = walk_all(parts, None)?; + return Ok((Node::And(nodes), false)); + }; + let mut nodes = Vec::with_capacity(parts.len()); + let mut any_vanishes = false; + let mut gate_taken = false; + for p in parts { + if !gate_taken && matches!(p, Filter::Plane(m) if *m == g) { + gate_taken = true; + continue; + } + let (n, v) = gate_walk(p, Some(g))?; + any_vanishes |= v; + nodes.push(n); + } + // The gate is implied by any child that vanishes with it; + // otherwise it must be read as a leaf of its own. + if !any_vanishes { + nodes.insert(0, Node::Plane(g)); + } + (Node::And(nodes), false) + } + } }) } -/// Lower one filter node into `dst`, tracking the highest slot touched. -fn lower_filter( - f: &Filter, - dst: u16, - ops: &mut Vec, - high_water: &mut usize, -) -> Result<(), LowerError> { - *high_water = (*high_water).max(usize::from(dst) + 1); - match f { - Filter::Cmp(col, cmp) => { +/// [`gate_walk`] over every child of a junction, refusing an empty one. +fn walk_all(parts: &[Filter], gate: Option) -> Result<(Vec, Vec), LowerError> { + if parts.is_empty() { + return Err(LowerError::EmptyJunction); + } + let mut nodes = Vec::with_capacity(parts.len()); + let mut flags = Vec::with_capacity(parts.len()); + for p in parts { + let (n, v) = gate_walk(p, gate)?; + nodes.push(n); + flags.push(v); + } + Ok((nodes, flags)) +} + +/// Emit `n` with its result in `dst` (or, for a plane, where it already is), +/// folding junction children into the first child's slot. +fn emit_inplace(n: &Node, dst: u16, ops: &mut Vec) -> Result { + match n { + Node::Pred { pred, under } => { ops.push(MaskOp::Pred { - pred: pred_of(*col, *cmp), - under: None, + pred: *pred, + under: under.map(|m| Operand::Plane(m.0)), dst, }); - Ok(()) + Ok(Operand::Scratch(dst)) + } + Node::Plane(m) => Ok(Operand::Plane(m.0)), + Node::Not(inner) => { + let a = emit_inplace(inner, dst, ops)?; + ops.push(MaskOp::Not { a, dst }); + Ok(Operand::Scratch(dst)) + } + Node::And(parts) | Node::Or(parts) => { + let is_and = matches!(n, Node::And(_)); + let mut acc: Option = None; + for part in parts { + // The first child lands in `dst`; every sibling after it + // borrows the next slot up in turn, so width costs one slot, + // not one per child. + let slot = if acc.is_none() { + dst + } else { + dst.checked_add(1).ok_or(LowerError::TooManySlots { + needed: usize::from(dst) + 2, + })? + }; + let b = emit_inplace(part, slot, ops)?; + acc = Some(match acc { + None => b, + Some(a) => { + ops.push(if is_and { + MaskOp::And { a, b, dst } + } else { + MaskOp::Or { a, b, dst } + }); + Operand::Scratch(dst) + } + }); + } + acc.ok_or(LowerError::EmptyJunction) } - Filter::Not(inner) => { - lower_filter(inner, dst, ops, high_water)?; - ops.push(MaskOp::Not { - a: Operand::Scratch(dst), + } +} + +/// Emit `n` with every comparison in its own slot and the skeleton fused. +fn emit_fused(n: &Node, ops: &mut Vec) -> Result { + let expr = assign_slots(n, ops)?; + let first_free = u16::try_from(ops.len()).map_err(|_| LowerError::TooManySlots { + needed: ops.len() + 1, + })?; + match fuse(&expr, first_free) { + Ok(fused) => { + ops.extend(fused.ops); + Ok(fused.result) + } + Err(FuseError::SlotOverflow) => Err(LowerError::TooManySlots { + needed: MAX_SCRATCH_SLOTS as usize + 1, + }), + } +} + +/// Give each comparison the next slot and mirror the skeleton as a +/// [`BoolExpr`] over the resulting operands (n-ary junctions left-folded). +fn assign_slots(n: &Node, ops: &mut Vec) -> Result { + match n { + Node::Pred { pred, under } => { + let dst = u16::try_from(ops.len()).map_err(|_| LowerError::TooManySlots { + needed: ops.len() + 1, + })?; + ops.push(MaskOp::Pred { + pred: *pred, + under: under.map(|m| Operand::Plane(m.0)), dst, }); - Ok(()) + Ok(BoolExpr::Leaf(Operand::Scratch(dst))) } - Filter::And(parts) | Filter::Or(parts) => { - let (first, rest) = parts.split_first().ok_or(LowerError::EmptyJunction)?; - lower_filter(first, dst, ops, high_water)?; - // The next slot up is scratch for each sibling in turn, so width - // costs one slot, not one per child. - let tmp = dst.checked_add(1).ok_or(LowerError::TooManySlots { - needed: usize::from(dst) + 2, - })?; - let is_and = matches!(f, Filter::And(_)); - for part in rest { - lower_filter(part, tmp, ops, high_water)?; - ops.push(if is_and { - MaskOp::And { - a: Operand::Scratch(dst), - b: Operand::Scratch(tmp), - dst, - } - } else { - MaskOp::Or { - a: Operand::Scratch(dst), - b: Operand::Scratch(tmp), - dst, + Node::Plane(m) => Ok(BoolExpr::Leaf(Operand::Plane(m.0))), + Node::Not(inner) => Ok(BoolExpr::Not(Box::new(assign_slots(inner, ops)?))), + Node::And(parts) | Node::Or(parts) => { + let is_and = matches!(n, Node::And(_)); + let mut acc: Option = None; + for part in parts { + let e = assign_slots(part, ops)?; + acc = Some(match acc { + None => e, + Some(a) => { + if is_and { + BoolExpr::And(Box::new(a), Box::new(e)) + } else { + BoolExpr::Or(Box::new(a), Box::new(e)) + } } }); } - Ok(()) + acc.ok_or(LowerError::EmptyJunction) } } } +/// `agg` read over `mask`. +fn terminal_of(agg: Agg, mask: Operand) -> Terminal { + match agg { + Agg::Count => Terminal::Count { mask }, + Agg::Any => Terminal::Any { mask }, + Agg::All => Terminal::All { mask }, + Agg::SumI32(c) => Terminal::MaskedSumI32 { mask, lane: c.0 }, + Agg::MinI32(c) => Terminal::MaskedMinI32 { mask, lane: c.0 }, + Agg::MaxI32(c) => Terminal::MaskedMaxI32 { mask, lane: c.0 }, + Agg::Rows => Terminal::Keep { mask }, + Agg::BlendI32 { then, els } => Terminal::BlendI32 { + mask, + then: then.0, + els: els.0, + }, + } +} + /// One comparison → one `Pred`. Exhaustive, and the compiler keeps it so. fn pred_of(col: Col, cmp: Cmp) -> Pred { let lane = col.0; @@ -312,129 +624,351 @@ fn pred_of(col: Col, cmp: Cmp) -> Pred { #[cfg(test)] mod tests { use super::*; - use lance_graph_mask_risc::{execute, scratch_words_for, LaneRef, Planes, Scratch, Value}; + use lance_graph_mask_risc::{ + execute, materialize_rows, reference_execute, scratch_words_for, words_for, LaneRef, + Planes, Scratch, Value, + }; const N: usize = 1000; - /// Two lanes: a signed value lane and an unsigned class lane. - fn fixture() -> (Vec, Vec) { - let vals: Vec = (0..N as i32).map(|i| (i * 7) % 401 - 200).collect(); - let classes: Vec = (0..N as u32).map(|i| i % 5).collect(); - (vals, classes) + const VALS: Col = Col(0); + const CLASS: Col = Col(1); + const ALT: Col = Col(2); + const ALPHA: Mask = Mask(0); + const FOCUS: Mask = Mask(1); + + /// Three lanes and two resident planes, every tail bit clean. + struct Fx { + vals: Vec, + classes: Vec, + alt: Vec, + masks: Vec>, } - /// The INDEPENDENT reading — a plain row loop that shares no code with the - /// lowering. This is the oracle: a filter tree walked per row, which is - /// exactly the shape this crate refuses to ship, written here because a - /// test oracle is the one place it is licensed. - fn oracle(f: &Filter, vals: &[i32], classes: &[u32], row: usize) -> bool { - match f { - Filter::Cmp(col, cmp) => { - let v = vals[row]; - let c = classes[row]; - match cmp { - Cmp::EqI32(x) => v == *x, - Cmp::NeI32(x) => v != *x, - Cmp::LtI32(x) => v < *x, - Cmp::LeI32(x) => v <= *x, - Cmp::GtI32(x) => v > *x, - Cmp::GeI32(x) => v >= *x, - Cmp::EqU32(x) => c == *x, - Cmp::NeU32(x) => c != *x, - Cmp::MatchU32 { pattern, care } => (c ^ *pattern) & *care == 0, + fn plane(n: usize, set: impl Fn(usize) -> bool) -> Vec { + let mut words = vec![0u64; words_for(n)]; + for r in (0..n).filter(|&r| set(r)) { + words[r / 64] |= 1u64 << (r % 64); + } + words + } + + impl Fx { + fn new(n: usize) -> Self { + let vals = (0..n) + .map(|i| ((i as i64 * 7) % 401 - 200) as i32) + .collect(); + let classes = (0..n).map(|i| (i % 5) as u32).collect(); + let alt = (0..n).map(|i| ((i as i64 * 13) % 89 - 44) as i32).collect(); + let masks = vec![plane(n, |r| r % 3 != 0), plane(n, |r| r % 7 == 0)]; + Fx { + vals, + classes, + alt, + masks, + } + } + + fn n(&self) -> usize { + self.vals.len() + } + + fn i32_at(&self, col: Col, row: usize) -> i32 { + match col { + VALS => self.vals[row], + ALT => self.alt[row], + other => panic!("{other:?} is not a signed lane of the fixture"), + } + } + + fn u32_at(&self, col: Col, row: usize) -> u32 { + match col { + CLASS => self.classes[row], + other => panic!("{other:?} is not an unsigned lane of the fixture"), + } + } + + fn bit(&self, m: Mask, row: usize) -> bool { + (self.masks[usize::from(m.0)][row / 64] >> (row % 64)) & 1 == 1 + } + + /// The INDEPENDENT reading — a plain row loop that shares no code + /// with the lowering. This is the oracle: a filter tree walked per + /// row, which is exactly the shape this crate refuses to ship, + /// written here because a test oracle is the one place it is + /// licensed. + fn oracle(&self, f: &Filter, row: usize) -> bool { + match f { + Filter::Cmp(col, cmp) => match *cmp { + Cmp::EqI32(x) => self.i32_at(*col, row) == x, + Cmp::NeI32(x) => self.i32_at(*col, row) != x, + Cmp::LtI32(x) => self.i32_at(*col, row) < x, + Cmp::LeI32(x) => self.i32_at(*col, row) <= x, + Cmp::GtI32(x) => self.i32_at(*col, row) > x, + Cmp::GeI32(x) => self.i32_at(*col, row) >= x, + Cmp::EqU32(x) => self.u32_at(*col, row) == x, + Cmp::NeU32(x) => self.u32_at(*col, row) != x, + Cmp::MatchU32 { pattern, care } => { + (self.u32_at(*col, row) ^ pattern) & care == 0 + } + }, + Filter::Plane(m) => self.bit(*m, row), + Filter::And(ps) => ps.iter().all(|p| self.oracle(p, row)), + Filter::Or(ps) => ps.iter().any(|p| self.oracle(p, row)), + Filter::Not(p) => !self.oracle(p, row), + } + } + + fn rows(&self, f: &Filter) -> Vec { + (0..self.n()).filter(|&r| self.oracle(f, r)).collect() + } + + fn with_planes(&self, extra: &[Vec], f: impl FnOnce(&Planes<'_>) -> R) -> R { + let lanes = [ + LaneRef::I32(&self.vals), + LaneRef::U32(&self.classes), + LaneRef::I32(&self.alt), + ]; + let masks: Vec<&[u64]> = self.masks.iter().chain(extra).map(Vec::as_slice).collect(); + f(&Planes { + n_rows: self.n(), + masks: &masks, + lanes: &lanes, + }) + } + + /// The executor, on a scratch sized exactly from the program. + fn exec(&self, program: &Program, extra: &[Vec], out: Option<&mut [i32]>) -> Value { + self.with_planes(extra, |planes| { + let words = words_for(self.n()); + let slots = program.scratch_slots as usize; + let mut buf = vec![0u64; scratch_words_for(words, slots).expect("sized")]; + let mut scratch = Scratch::over(&mut buf, words, slots).expect("carves"); + execute(program, planes, &mut scratch, out).expect("runs") + }) + } + + /// The executor on a `Keep` program, copying the kept mask out of + /// wherever `Value::Mask` says it landed. + fn exec_mask(&self, program: &Program, extra: &[Vec]) -> Vec { + self.with_planes(extra, |planes| { + let words = words_for(self.n()); + let slots = program.scratch_slots as usize; + let mut buf = vec![0u64; scratch_words_for(words, slots).expect("sized")]; + let mut scratch = Scratch::over(&mut buf, words, slots).expect("carves"); + match execute(program, planes, &mut scratch, None).expect("runs") { + Value::Mask(Operand::Scratch(i)) => scratch.slot(i).expect("written").to_vec(), + Value::Mask(Operand::Plane(p)) => planes.masks[usize::from(p)].to_vec(), + other => panic!("not a kept mask: {other:?}"), } - .then_some(*col) - .is_some() + }) + } + + /// mask-risc's own row-at-a-time oracle — a second independent arm. + fn reference(&self, program: &Program, out: Option<&mut [i32]>) -> Value { + self.with_planes(&[], |planes| { + reference_execute(program, planes, out).expect("runs") + }) + } + + fn count(&self, f: &Filter) -> usize { + let q = Query { + filter: f.clone(), + agg: Agg::Count, + }; + match self.exec(&lower(&q).expect("lowers"), &[], None) { + Value::Count(c) => c, + other => panic!("not a count: {other:?}"), } - Filter::And(ps) => ps.iter().all(|p| oracle(p, vals, classes, row)), - Filter::Or(ps) => ps.iter().any(|p| oracle(p, vals, classes, row)), - Filter::Not(p) => !oracle(p, vals, classes, row), } } - fn run(q: &Query, vals: &[i32], classes: &[u32]) -> Value { - let program = lower(q).expect("lowers"); - let lanes = [LaneRef::I32(vals), LaneRef::U32(classes)]; - let planes = Planes { - n_rows: vals.len(), - masks: &[], - lanes: &lanes, - }; - let words = vals.len().div_ceil(64); - let slots = program.scratch_slots as usize; - let mut buf = vec![0u64; scratch_words_for(words, slots).expect("sized")]; - let mut scratch = Scratch::over(&mut buf, words, slots).expect("carves"); - execute(&program, &planes, &mut scratch, None).expect("runs") + /// `alpha & ((A & B) | C)` — the vertical slice. + fn slice_filter() -> Filter { + Filter::and([ + Filter::plane(ALPHA), + Filter::or([ + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ]), + Filter::cmp(VALS, Cmp::LtI32(-190)), + ]), + ]) } - /// FAILS IF: the lowering and an independent per-row reading of the same - /// filter disagree. + /// Every `Pred` op of `p` is gated under `g`, and `g` is never read as a + /// Boolean operand — the shape the survivor skip produces when the plane + /// leaf is dropped. + fn gated_and_dropped(p: &Program, g: Mask) -> (bool, bool) { + let gate = Operand::Plane(g.0); + let mut all_gated = true; + let mut read_as_leaf = false; + for op in &p.ops { + match *op { + MaskOp::Pred { under, .. } => all_gated &= under == Some(gate), + MaskOp::And { a, b, .. } + | MaskOp::Or { a, b, .. } + | MaskOp::Xor { a, b, .. } + | MaskOp::AndNot { a, b, .. } => read_as_leaf |= a == gate || b == gate, + MaskOp::Not { a, .. } => read_as_leaf |= a == gate, + MaskOp::Ternlog { a, b, c, .. } => { + read_as_leaf |= a == gate || b == gate || c == gate; + } + } + } + let terminal_reads_gate = matches!( + p.terminal, + Terminal::Count { mask } + | Terminal::Any { mask } + | Terminal::All { mask } + | Terminal::MaskedSumI32 { mask, .. } + | Terminal::MaskedMinI32 { mask, .. } + | Terminal::MaskedMaxI32 { mask, .. } + | Terminal::BlendI32 { mask, .. } + | Terminal::Keep { mask } if mask == gate + ); + (all_gated, !read_as_leaf && !terminal_reads_gate) + } + + /// FAILS IF: either lowering and an independent per-row reading of the + /// same filter disagree. /// /// This is the whole claim of the crate — that a query expressed as /// masking ops answers what the query means — so it is checked against an - /// oracle that never sees a `Program`. + /// oracle that never sees a `Program`, for the in-place AND the fused + /// lowering, over shapes that exercise every gate rule: a gate over a + /// nested `OR`, a gate that must survive a negation, a gate that must + /// survive a foreign plane, two planes, a plane alone, an IN-list. #[test] fn every_lowered_filter_agrees_with_an_independent_per_row_reading() { - let (vals, classes) = fixture(); + let fx = Fx::new(N); let cases: Vec<(&str, Filter)> = vec![ - ("one leaf", Filter::cmp(Col(0), Cmp::GtI32(0))), - ("u32 leaf", Filter::cmp(Col(1), Cmp::EqU32(2))), + ("one leaf", Filter::cmp(VALS, Cmp::GtI32(0))), + ("u32 leaf", Filter::cmp(CLASS, Cmp::EqU32(2))), + ("a plane alone", Filter::plane(ALPHA)), ( "and of two", Filter::and([ - Filter::cmp(Col(0), Cmp::GtI32(-50)), - Filter::cmp(Col(1), Cmp::NeU32(0)), + Filter::cmp(VALS, Cmp::GtI32(-50)), + Filter::cmp(CLASS, Cmp::NeU32(0)), ]), ), ( "or of two", Filter::or([ - Filter::cmp(Col(0), Cmp::LtI32(-150)), - Filter::cmp(Col(1), Cmp::EqU32(4)), + Filter::cmp(VALS, Cmp::LtI32(-150)), + Filter::cmp(CLASS, Cmp::EqU32(4)), ]), ), - ("not", Filter::negate(Filter::cmp(Col(0), Cmp::GeI32(0)))), + ("not", Filter::negate(Filter::cmp(VALS, Cmp::GeI32(0)))), ( "and of four — the width case", Filter::and([ - Filter::cmp(Col(0), Cmp::GtI32(-180)), - Filter::cmp(Col(0), Cmp::LtI32(180)), - Filter::cmp(Col(1), Cmp::NeU32(3)), - Filter::cmp(Col(1), Cmp::NeU32(1)), + Filter::cmp(VALS, Cmp::GtI32(-180)), + Filter::cmp(VALS, Cmp::LtI32(180)), + Filter::cmp(CLASS, Cmp::NeU32(3)), + Filter::cmp(CLASS, Cmp::NeU32(1)), ]), ), ( "nested — or inside and, with a not", Filter::and([ - Filter::cmp(Col(0), Cmp::GeI32(-100)), + Filter::cmp(VALS, Cmp::GeI32(-100)), Filter::or([ - Filter::cmp(Col(1), Cmp::EqU32(1)), - Filter::negate(Filter::cmp(Col(0), Cmp::GtI32(100))), + Filter::cmp(CLASS, Cmp::EqU32(1)), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(100))), ]), ]), ), ( "ternary match — no SQL spelling, the substrate had it all along", Filter::cmp( - Col(1), + CLASS, Cmp::MatchU32 { pattern: 0b100, care: 0b110, }, ), ), + ("gate over a comparison", { + Filter::and([Filter::plane(ALPHA), Filter::cmp(VALS, Cmp::GtI32(0))]) + }), + ("gate over a nested or — the slice", slice_filter()), + ( + "gate over a negation — the gate must stay a leaf", + Filter::and([ + Filter::plane(ALPHA), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(0))), + ]), + ), + ( + "gate over an or with a negated arm", + Filter::and([ + Filter::plane(ALPHA), + Filter::or([ + Filter::cmp(CLASS, Cmp::EqU32(3)), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(-100))), + ]), + ]), + ), + ( + "gate beside a foreign plane — the foreign plane is a leaf", + Filter::and([ + Filter::plane(ALPHA), + Filter::plane(FOCUS), + Filter::cmp(VALS, Cmp::LtI32(50)), + ]), + ), + ( + "two planes or'd, then a comparison", + Filter::and([ + Filter::or([Filter::plane(ALPHA), Filter::plane(FOCUS)]), + Filter::cmp(VALS, Cmp::NeI32(0)), + ]), + ), + ("plane after the comparisons", { + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(-100)), + Filter::cmp(VALS, Cmp::LtI32(100)), + Filter::plane(FOCUS), + ]) + }), + ( + "a gated and nested under a gated and", + Filter::and([ + Filter::plane(ALPHA), + Filter::and([Filter::plane(FOCUS), Filter::cmp(VALS, Cmp::GtI32(-100))]), + ]), + ), + ("not of a plane", Filter::negate(Filter::plane(ALPHA))), + ( + "in-list over the class lane", + Filter::in_u32(CLASS, [1, 3, 3]), + ), + ( + "in-list over the value lane", + Filter::in_i32(VALS, [-200, 5, 100, 12]), + ), ]; for (label, f) in cases { - let expected = (0..N).filter(|&r| oracle(&f, &vals, &classes, r)).count(); + let expected = fx.rows(&f).len(); let q = Query { filter: f, agg: Agg::Count, }; + let inplace = lower(&q).expect("lowers in place"); + let fused = lower_fused(&q).expect("lowers fused"); assert_eq!( - run(&q, &vals, &classes), + fx.exec(&inplace, &[], None), Value::Count(expected), - "{label}: the lowered program and the per-row oracle disagree" + "{label}: the in-place program and the per-row oracle disagree" + ); + assert_eq!( + fx.exec(&fused, &[], None), + Value::Count(expected), + "{label}: the fused program and the per-row oracle disagree" ); // Anti-vacuity: agreement on "nothing" or "everything" would hold // for a lowering that ignored the filter entirely. @@ -445,41 +979,170 @@ mod tests { } } - /// FAILS IF: a junction allocates a slot per child. + /// FAILS IF: the survivor skip gates the wrong leaves or drops the gate + /// where the drop is unsound. + /// + /// Can-fire: under `alpha & ((A & B) | C)` every comparison is gated and + /// alpha is never read as an operand. Can-stay-silent: under + /// `alpha & !X` the comparison is still gated (the rewrite is sound + /// through a negation while the gate is a leaf) but alpha IS read as a + /// leaf, because `!(alpha & X)` is 1 exactly where alpha is 0. The + /// differential above holds either way; this pins the SHAPE, so a walk + /// that dropped the gate under a negation would fail here even before + /// the oracle caught its wrong count. + #[test] + fn the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes() { + for lowering in [lower, lower_fused] { + let slice = lowering(&Query { + filter: slice_filter(), + agg: Agg::Count, + }) + .expect("lowers"); + assert_eq!( + gated_and_dropped(&slice, ALPHA), + (true, true), + "the slice: every predicate gated, alpha implied: {:?}", + slice.ops + ); + + let negated = lowering(&Query { + filter: Filter::and([ + Filter::plane(ALPHA), + Filter::negate(Filter::cmp(VALS, Cmp::GtI32(0))), + ]), + agg: Agg::Count, + }) + .expect("lowers"); + assert_eq!( + gated_and_dropped(&negated, ALPHA), + (true, false), + "under a negation: still gated, but alpha stays a leaf: {:?}", + negated.ops + ); + + let foreign = lowering(&Query { + filter: Filter::and([ + Filter::plane(ALPHA), + Filter::plane(FOCUS), + Filter::cmp(VALS, Cmp::LtI32(50)), + ]), + agg: Agg::Count, + }) + .expect("lowers"); + let (gated, dropped) = gated_and_dropped(&foreign, ALPHA); + assert!(gated && dropped, "the first plane gates: {:?}", foreign.ops); + assert!( + !gated_and_dropped(&foreign, FOCUS).1, + "the foreign plane is read as a leaf, not gated away: {:?}", + foreign.ops + ); + } + } + + /// FAILS IF: a resident plane costs an op or a slot. + /// + /// `SELECT count(*) FROM t` is the plane's population; the program is + /// empty, the terminal reads the plane, and projection returns the plane + /// itself rather than a copy of it. + #[test] + fn a_resident_plane_alone_is_a_zero_op_program() { + let fx = Fx::new(N); + let expected = fx.rows(&Filter::plane(ALPHA)).len(); + assert!(expected > 0 && expected < N); + for lowering in [lower, lower_fused] { + let count = lowering(&Query { + filter: Filter::plane(ALPHA), + agg: Agg::Count, + }) + .expect("lowers"); + assert!(count.ops.is_empty(), "{:?}", count.ops); + assert_eq!(count.scratch_slots, 0); + assert_eq!( + count.terminal, + Terminal::Count { + mask: Operand::Plane(0) + } + ); + assert_eq!(fx.exec(&count, &[], None), Value::Count(expected)); + + let rows = lowering(&Query { + filter: Filter::plane(ALPHA), + agg: Agg::Rows, + }) + .expect("lowers"); + assert_eq!( + fx.exec(&rows, &[], None), + Value::Mask(Operand::Plane(0)), + "projection of a plane is the plane, not a copy" + ); + } + } + + /// FAILS IF: the fused lowering does not buy passes with slots — or buys + /// nothing. + /// + /// On the slice, in place is two Boolean passes (an AND, then an OR) in + /// two slots; fused is one ternlog in four. Both are pinned, so a fuser + /// that stopped fusing OR an in-place emitter that started allocating per + /// leaf would each fail their own line. + #[test] + fn the_fused_lowering_trades_slots_for_passes() { + let q = Query { + filter: slice_filter(), + agg: Agg::Count, + }; + let inplace = lower(&q).expect("lowers"); + let fused = lower_fused(&q).expect("lowers"); + + assert_eq!(inplace.op_histogram().predicates, 3); + assert_eq!(fused.op_histogram().predicates, 3); + assert_eq!(inplace.op_histogram().mask_passes(), 2); + assert_eq!(fused.op_histogram().mask_passes(), 1); + assert_eq!(fused.op_histogram().ternlog, 1); + assert_eq!(inplace.scratch_slots, 2); + assert_eq!(fused.scratch_slots, 4); + } + + /// FAILS IF: a junction allocates a slot per child in the in-place form. /// /// Children fold left-to-right into the first child's slot, so width is /// free and only DEPTH costs. Without this a 64-wide conjunction would ask /// for a 64-slot arena — a scratch allocation proportional to the query's - /// text rather than to its shape. + /// text rather than to its shape. The fused form DOES pay width, and that + /// is pinned too: it is the trade, not a defect. #[test] fn a_wide_conjunction_costs_one_extra_slot_not_one_per_child() { - let wide = Filter::and((0..32).map(|i| Filter::cmp(Col(0), Cmp::NeI32(i)))); - let p = lower(&Query { - filter: wide, + let wide = Query { + filter: Filter::and((0..32).map(|i| Filter::cmp(VALS, Cmp::NeI32(i)))), agg: Agg::Count, - }) - .expect("lowers"); + }; + let p = lower(&wide).expect("lowers"); assert_eq!( p.scratch_slots, 2, "32 children must cost depth-2, not 32 slots" ); + let pf = lower_fused(&wide).expect("lowers"); + assert!( + pf.scratch_slots >= 32, + "fused pays one slot per comparison: {}", + pf.scratch_slots + ); // Paired: DEPTH does cost, or the constant above would be vacuous. - let deep = Filter::and([ - Filter::cmp(Col(0), Cmp::GtI32(0)), - Filter::or([ - Filter::cmp(Col(1), Cmp::EqU32(1)), - Filter::and([ - Filter::cmp(Col(0), Cmp::LtI32(50)), - Filter::cmp(Col(1), Cmp::NeU32(2)), + let deep = Query { + filter: Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::or([ + Filter::cmp(CLASS, Cmp::EqU32(1)), + Filter::and([ + Filter::cmp(VALS, Cmp::LtI32(50)), + Filter::cmp(CLASS, Cmp::NeU32(2)), + ]), ]), ]), - ]); - let pd = lower(&Query { - filter: deep, agg: Agg::Count, - }) - .expect("lowers"); + }; + let pd = lower(&deep).expect("lowers"); assert!( pd.scratch_slots > p.scratch_slots, "depth must cost more than width: deep={} wide={}", @@ -488,67 +1151,244 @@ mod tests { ); } - /// FAILS IF: an empty AND/OR is folded to a constant instead of refused. + /// FAILS IF: an empty AND/OR — or the `IN ()` that is one — is folded to + /// a constant instead of refused. /// /// An empty conjunction is `true` and an empty disjunction is `false`, so /// whichever identity the code picked would silently be the answer to a /// query the caller built by accident. #[test] fn an_empty_junction_is_refused_rather_than_folded_to_an_identity() { - for f in [Filter::And(vec![]), Filter::Or(vec![])] { - assert_eq!( - lower(&Query { - filter: f, - agg: Agg::Count - }), - Err(LowerError::EmptyJunction) - ); + for f in [ + Filter::And(vec![]), + Filter::Or(vec![]), + Filter::in_u32(CLASS, []), + Filter::in_i32(VALS, []), + Filter::and([Filter::plane(ALPHA), Filter::Or(vec![])]), + ] { + let q = Query { + filter: f.clone(), + agg: Agg::Count, + }; + assert_eq!(lower(&q), Err(LowerError::EmptyJunction), "{f:?}"); + assert_eq!(lower_fused(&q), Err(LowerError::EmptyJunction), "{f:?}"); } } - /// FAILS IF: the aggregates do not read the filter's mask. + /// FAILS IF: an IN-list is anything other than the disjunction it claims + /// to be. + /// + /// Structural equality with the hand-built OR, and agreement with the + /// oracle; duplicates in the set are harmless because they are harmless + /// in an OR. + #[test] + fn an_in_list_is_a_disjunction_of_equalities() { + let fx = Fx::new(N); + let by_hand = Filter::or([ + Filter::cmp(CLASS, Cmp::EqU32(1)), + Filter::cmp(CLASS, Cmp::EqU32(3)), + Filter::cmp(CLASS, Cmp::EqU32(3)), + ]); + let in_list = Filter::in_u32(CLASS, [1, 3, 3]); + assert_eq!(in_list, by_hand); + let expected = (0..N).filter(|&r| [1, 3].contains(&fx.classes[r])).count(); + assert_eq!(fx.count(&in_list), expected); + assert!(expected > 0 && expected < N); + } + + /// FAILS IF: the aggregates and the projections do not read the filter's + /// mask. /// - /// `Count`/`Any`/`All`/`Sum`/`Min`/`Max` over the SAME filter must be - /// mutually consistent with the per-row oracle — an aggregate wired to the - /// wrong operand would still return a plausible number. + /// `Count`/`Any`/`All`/`Sum`/`Min`/`Max`/`Rows`/`Blend` over the SAME + /// filter must be mutually consistent with the per-row oracle — an + /// aggregate wired to the wrong operand would still return a plausible + /// number, and a blend wired to the wrong lane a plausible column. #[test] - fn every_aggregate_reads_the_same_filter_mask() { - let (vals, classes) = fixture(); + fn every_aggregate_and_projection_reads_the_same_filter_mask() { + let fx = Fx::new(N); let f = Filter::and([ - Filter::cmp(Col(0), Cmp::GtI32(-100)), - Filter::cmp(Col(1), Cmp::EqU32(3)), + Filter::plane(ALPHA), + Filter::cmp(VALS, Cmp::GtI32(-100)), + Filter::cmp(CLASS, Cmp::EqU32(3)), ]); - let rows: Vec = (0..N).filter(|&r| oracle(&f, &vals, &classes, r)).collect(); + let rows = fx.rows(&f); assert!( !rows.is_empty() && rows.len() < N, "fixture must be a proper subset" ); - let sum: i64 = rows.iter().map(|&r| i64::from(vals[r])).sum(); - let min = rows.iter().map(|&r| vals[r]).min(); - let max = rows.iter().map(|&r| vals[r]).max(); + let sum: i64 = rows.iter().map(|&r| i64::from(fx.vals[r])).sum(); + let min = rows.iter().map(|&r| fx.vals[r]).min(); + let max = rows.iter().map(|&r| fx.vals[r]).max(); let q = |agg| Query { filter: f.clone(), agg, }; + let run = |agg| fx.exec(&lower(&q(agg)).expect("lowers"), &[], None); + assert_eq!(run(Agg::Count), Value::Count(rows.len())); + assert_eq!(run(Agg::Any), Value::Bool(true)); + assert_eq!(run(Agg::All), Value::Bool(false)); + assert_eq!(run(Agg::SumI32(VALS)), Value::SumI64(sum)); + assert_eq!(run(Agg::MinI32(VALS)), Value::OptI32(min)); + assert_eq!(run(Agg::MaxI32(VALS)), Value::OptI32(max)); + + // Projection: the kept mask materialises to exactly the oracle's rows. + let kept = fx.exec_mask(&lower(&q(Agg::Rows)).expect("lowers"), &[]); + assert_eq!(materialize_rows(&kept, N), rows); + + // CASE: every row reads `then` where the filter holds, `els` where not. + let mut out = vec![0i32; N]; assert_eq!( - run(&q(Agg::Count), &vals, &classes), - Value::Count(rows.len()) + fx.exec( + &lower(&q(Agg::BlendI32 { + then: VALS, + els: ALT + })) + .expect("lowers"), + &[], + Some(&mut out), + ), + Value::Blended ); - assert_eq!(run(&q(Agg::Any), &vals, &classes), Value::Bool(true)); - assert_eq!(run(&q(Agg::All), &vals, &classes), Value::Bool(false)); - assert_eq!( - run(&q(Agg::SumI32(Col(0))), &vals, &classes), - Value::SumI64(sum) + let expected: Vec = (0..N) + .map(|r| { + if fx.oracle(&f, r) { + fx.vals[r] + } else { + fx.alt[r] + } + }) + .collect(); + assert_eq!(out, expected); + assert!( + out.iter().zip(&fx.vals).any(|(o, v)| o != v), + "the blend must actually pick from `els` somewhere" ); + } + + /// FAILS IF: the two-phase GROUP BY does not partition the filtered rows + /// by key. + /// + /// Per-group counts and sums equal the oracle's, and the counts sum to + /// the ungrouped filtered count: a group program that ignored the key + /// would report the total K times, one that ignored the kept filter + /// would report the unfiltered class sizes, and either breaks the + /// partition identity. + #[test] + fn a_group_by_partitions_the_filtered_rows_by_key() { + let fx = Fx::new(N); + let filter = Filter::and([Filter::plane(ALPHA), Filter::cmp(VALS, Cmp::GtI32(0))]); + let rows = fx.rows(&filter); + let groups = 5u32; + let filter_plane = u16::try_from(fx.masks.len()).expect("fits"); + + for agg in [Agg::Count, Agg::SumI32(VALS)] { + let plan = lower_group_by( + &GroupBy { + filter: filter.clone(), + key: CLASS, + groups, + agg, + }, + filter_plane, + ) + .expect("lowers"); + assert_eq!(plan.groups.len(), groups as usize); + assert_eq!(plan.filter_plane, filter_plane); + + let kept = fx.exec_mask(&plan.filter, &[]); + let extra = vec![kept]; + let results: Vec = plan + .groups + .iter() + .map(|p| fx.exec(p, &extra, None)) + .collect(); + + for (g, value) in results.iter().enumerate() { + let members: Vec = rows + .iter() + .copied() + .filter(|&r| fx.classes[r] == g as u32) + .collect(); + let expected = match agg { + Agg::Count => Value::Count(members.len()), + Agg::SumI32(_) => { + Value::SumI64(members.iter().map(|&r| i64::from(fx.vals[r])).sum()) + } + other => unreachable!("{other:?}"), + }; + assert_eq!(*value, expected, "group {g}"); + assert!(!members.is_empty(), "group {g} must be non-empty to count"); + } + if agg == Agg::Count { + let total: usize = results + .iter() + .map(|v| match v { + Value::Count(c) => *c, + other => panic!("{other:?}"), + }) + .sum(); + assert_eq!(total, rows.len(), "the groups partition the filter"); + assert!(rows.len() < N); + } + } + assert_eq!( - run(&q(Agg::MinI32(Col(0))), &vals, &classes), - Value::OptI32(min) + lower_group_by( + &GroupBy { + filter, + key: CLASS, + groups, + agg: Agg::BlendI32 { + then: VALS, + els: ALT + }, + }, + filter_plane, + ), + Err(LowerError::GroupedBlend) ); - assert_eq!( - run(&q(Agg::MaxI32(Col(0))), &vals, &classes), - Value::OptI32(max) + } + + /// FAILS IF: any arm of the vertical slice disagrees on a 64k slab. + /// + /// `COUNT(alpha & ((A & B) | C))` over 65,536 rows through the per-row + /// oracle, the in-place program on the executor, the fused program on the + /// executor, and both programs on mask-risc's reference evaluator — five + /// readings, one number. Anti-vacuity: the gate binds (the count is + /// strictly below the ungated remainder's) and the answer is a proper + /// subset of alpha. + #[test] + fn the_vertical_slice_agrees_across_every_arm_on_a_64k_slab() { + let fx = Fx::new(1 << 16); + let f = slice_filter(); + let expected = fx.rows(&f).len(); + let alpha = fx.rows(&Filter::plane(ALPHA)).len(); + let ungated = fx + .rows(&Filter::or([ + Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ]), + Filter::cmp(VALS, Cmp::LtI32(-190)), + ])) + .len(); + assert!(expected > 0 && expected < alpha, "{expected} of {alpha}"); + assert!( + expected < ungated, + "the gate must bind: {expected} vs {ungated}" ); + + let q = Query { + filter: f, + agg: Agg::Count, + }; + let inplace = lower(&q).expect("lowers"); + let fused = lower_fused(&q).expect("lowers"); + assert_eq!(fx.exec(&inplace, &[], None), Value::Count(expected)); + assert_eq!(fx.exec(&fused, &[], None), Value::Count(expected)); + assert_eq!(fx.reference(&inplace, None), Value::Count(expected)); + assert_eq!(fx.reference(&fused, None), Value::Count(expected)); } } From ca96dc152bc49aba40df9b810364dd3b278aaba0 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 22:28:23 +0000 Subject: [PATCH 06/25] =?UTF-8?q?quack:=20the=20empty-junction=20refusal?= =?UTF-8?q?=20is=20spelled=20three=20times=20=E2=80=94=20record=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight disable arms against the new falsifiers; seven came back LOAD-BEARING on the first try. The eighth is the interesting one and its finding is the commit. Disabling the `walk_all` empty-junction guard left the whole suite GREEN. Read naively that says the guard is decoration — the exact reading this repo's own rule warns is indistinguishable from a patch that did not apply. It applied (the anchor assertion held). What it actually measured is that the refusal exists at THREE sites: the gate walk, the in-place emitter's `acc.ok_or`, and the fused emitter's. Any one removed, the other two still refuse. Disabling all three together turns `an_empty_junction_is_refused_rather_than_folded_to_an_identity` red, which is what makes the behaviour load-bearing rather than the individual line. Both the guard and the test now say so, because the failure mode is a future session measuring one site, reading green, and deleting a guard as dead. The early guard earns its place beyond redundancy: without it an empty `Or` reaches `flags.iter().all(..)` over an empty vector — vacuously true — and reports that it vanishes with its gate, which would let a parent AND drop a gate plane it must keep. Nothing observable changes today because the program is refused downstream anyway; it is a vacuous-truth corner not worth leaving open next to a rule that turns on exactly that predicate. The other seven, each verified red then green: the gate dropped under a negation; a junction allocating per child; the survivor skip removed; fusion aliased to the in-place lowering; a group program ignoring the kept filter; projection reducing instead of keeping; the gate dropped with no child vanishing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- crates/lance-graph-quack/src/lib.rs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index fd9b1f9a4..4245df1f6 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -464,6 +464,23 @@ fn gate_walk(f: &Filter, gate: Option) -> Result<(Node, bool), LowerError> } /// [`gate_walk`] over every child of a junction, refusing an empty one. +/// +/// One of THREE spellings of that refusal — the others are the +/// `acc.ok_or(EmptyJunction)` in [`emit_inplace`] and in [`assign_slots`]. +/// Measured: disabling any ONE leaves the suite green, because the other two +/// still catch it; disabling all three turns +/// `an_empty_junction_is_refused_rather_than_folded_to_an_identity` red. So +/// the refusal is load-bearing and the redundancy is deliberate — each +/// spelling guards a different stage (the gate walk, the in-place emitter, +/// the fused emitter) and none may be removed as "obviously dead" on the +/// strength of its own disable run coming back green. +/// +/// The early one is not merely belt-and-braces: without it an empty `Or` +/// reaches `flags.iter().all(..)` over an EMPTY vector, which is vacuously +/// `true`, and reports that it vanishes with the gate — letting a parent +/// `AND` drop a gate plane it should have kept. The program is refused +/// downstream either way, so nothing observable changes today; it is a +/// vacuous-truth corner not worth leaving open. fn walk_all(parts: &[Filter], gate: Option) -> Result<(Vec, Vec), LowerError> { if parts.is_empty() { return Err(LowerError::EmptyJunction); @@ -1157,6 +1174,14 @@ mod tests { /// An empty conjunction is `true` and an empty disjunction is `false`, so /// whichever identity the code picked would silently be the answer to a /// query the caller built by accident. + /// + /// Disable note, because a single-site run misreads: the refusal is + /// spelled at THREE sites ([`walk_all`], [`emit_inplace`], + /// [`assign_slots`]) and disabling any one leaves this green — not + /// because the test is vacuous but because the other two still catch it. + /// Disabling all three together is what turns it red. A future session + /// measuring one site and concluding "not load-bearing" would be reading + /// the redundancy, not the guard. #[test] fn an_empty_junction_is_refused_rather_than_folded_to_an_identity() { for f in [ From 75b69c37a286d9f7db02199f4545bb124b8b7eaf Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 22:30:04 +0000 Subject: [PATCH 07/25] board: record the quack surface (D-QCK-0..7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LATEST_STATE gets the inventory delta and the three things the scaffold owed; STATUS_BOARD gets the D-ids with their gates. D-QCK-7 (the join) is filed BLOCKED with its actual blocker named — mask-risc has no `hop` op, so there is nothing to lower to — rather than left unlisted, which reads as unconsidered. Per the termination clause this hygiene commit generates no further obligations: it is discharged by the entries it wrote. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/LATEST_STATE.md | 55 +++++++++++++++++++++++++++++++++++ .claude/board/STATUS_BOARD.md | 16 ++++++++++ 2 files changed, 71 insertions(+) diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 912dbb8f7..142c1d1cb 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -76,6 +76,61 @@ census is measured and NOT yet on the board — see the arc entry's *Un-recorded* bullet. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1233. +## 2026-09-14 (3) — `lance-graph-quack`: the DuckDB-shaped surface whose operators ARE masking ops + +New workspace member `crates/lance-graph-quack`, one dependency +(`lance-graph-mask-risc`, path). Inventory: `Col`, `Mask`, `Cmp` (9 +comparisons, one per `Pred` that produces a mask from a value lane), `Filter` +(`Cmp`/`Plane`/`And`/`Or`/`Not`, builders `cmp`/`plane`/`and`/`or`/`negate`/ +`in_u32`/`in_i32`), `Agg` (`Count`/`Any`/`All`/`SumI32`/`MinI32`/`MaxI32`/ +`Rows`/`BlendI32`), `Query`, `GroupBy`, `GroupPlan`, `LowerError` +(`EmptyJunction`/`TooManySlots`/`GroupedBlend`), and three lowerings — +`lower` (in-place), `lower_fused` (Boolean skeleton through +`mask_risc::fuse`), `lower_group_by` (two-phase). + +**The crate builds a `Program` and never evaluates one.** `execute` stays the +consumer's call on a scratch the consumer owns; a `match` here that computed +anything would be the duplicate evaluator the arc exists to avoid. No +`ndarray` dep for the same reason — the masking algebra is reached THROUGH +mask-risc, never beside it. + +Three things the scaffold owed and this closes: + +- **`Filter::Plane`** — a resident mask plane read as a predicate. The + scaffold's own doc recorded that "every row" had no honest spelling (no + `Fill` op; the draft's `NeU32{lane:0}` + constant-ternlog trick typechecked + only when lane 0 happened to be `U32`, a latent lane-kind bug). The plane + leaf is correct by construction, costs zero ops and zero slots, and is the + truer model: the table IS its validity plane, which is why there is no NULL + here to be three-valued about. +- **The survivor skip.** An `AND` with a plane child gates every comparison + beneath it (`MaskOp::Pred`'s `under`). Sound for any Boolean remainder + (`g ∧ rest(X) = g ∧ rest(g∧X)`), so the gate passes through `NOT` and `OR`; + what does not pass is the DROP — the plane leaf is elided only where the + gated remainder is identically zero wherever the gate is (a comparison is; + an `AND` if any child is; an `OR` if every child is; a `NOT` never). +- **`GROUP BY` with a mask plane where the hash table would be** — keep the + filter, bind it as a plane, one gated equality per key. K programs, not one, + and the doc says why: `masked_strided_group_sum` exists in the facade but + the IR names no strided operand and no group terminal. + +10 tests, every one differential against a per-row oracle that never sees a +`Program`. The 64k vertical slice (`COUNT(alpha & ((A&B)|C))`) agrees across +five readings — oracle, both lowerings on the executor, both on mask-risc's +reference evaluator — with two-sided anti-vacuity (a proper subset of alpha +AND strictly below the ungated remainder, so a dropped gate fails even though +its count would still look plausible). + +Named absent: the join (`src_mask → hop → dst_mask` is mask-risc's PR5; there +is no `hop` op to lower to yet), a one-terminal grouped SUM, and everything +the IR itself excludes (strings, `ORDER BY`, three-valued NULL). + +Disable table: 8 arms, 7 load-bearing first try. The 8th is recorded as a +finding — the empty-junction refusal is spelled at three sites, so disabling +any one leaves the suite green while all three together turn it red. Both the +guard and the test now say so, because the failure mode is a future session +measuring one site, reading green, and deleting a guard as dead. Commits +`18c1d85`, `f1d41c8`. ## 2026-09-14 (2) — PR3 (branch `claude/clone-repositories-71a5sw`): `lance-graph-mask-risc` gains its executor, oracle, fuser and generated dispatch diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 89a1a1ea8..f81e86001 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -13,6 +13,22 @@ them **reverse a verdict published earlier in the same session**. | D-ELK-3 | does depth-rank over the meet reproduce `LensClosure::most_specific` without the pairwise test? | **Done — REVERSED TWICE.** Descending (my sign error) 6.7 %, published as *refuted*; ascending **84.6 %** (min-max) / **85.1 %** (min-sum), 3/3 on the clinical pairs. Then D-ELK-4 re-scoped it: taxonomy-shaped, not lens behaviour | RA/Still's/Felty's all returning `"disease"` (the ROOT — the node farthest from every seed) is what exposed the sign. A ranking that returns the root for every clinical pair is reporting the most GENERAL ancestor | | D-ELK-4 | **generality control** — same arm code on a second graph, so "the lens behaves like X" is separable from "a taxonomy behaves like X" | **Done — DIVERGES 48.6 pp, and the divergence survives both confounds.** MQ 36.0 % vs MONDO 84.6 %. A4 non-vacuous: identical (neither graph has a single `\|I\|==1` pair). A5 braided-only: 47.5 pp — braiding is directional inside MONDO (flat 13/13 vs braided 152/182) but explains almost none of the gap | **PRE-REGISTERED**: "if ASCENDING agrees across both, the result is the LENS's; if it diverges, it was the taxonomy's shape." A4 and A5 are the two confounds that could fake a divergence; a gap surviving both is structural. **Mechanism still unnamed** — see `ISS-ELK-DENSITY-UNISOLATED` | | D-ELK-5 | horizon width sweep — is an empty meet at width `w` one answer or two? | **Done — two, and code returning `∅` for both cannot tell them apart.** MONDO CUT 194→3→0 at `w`=1/7/14 (recall 0.1→84.5→100 %); MQ 178→85→21 (0.3→24.2→64.5 %). True EMPTY is width-invariant: MONDO **5**, MQ **0** | EMPTY must not move with `w` and CUT must — both hold, which is what makes the two causes separable at all. Same missing symbol `E-THE-BOXCAR-HORIZON-IS-NOT-A-DISCOUNT…` argued for from the other side | +## quack-duckdb-surface (D-ids minted 2026-09-14 with the crate) + +`crates/lance-graph-quack` — DuckDB's operator set with every operator lowering +to a `lance-graph-mask-risc` `Program`. The crate builds programs and never +evaluates one; `execute` stays the consumer's call on a scratch it owns. + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| D-QCK-0 | `Filter`/`Cmp`/`Agg`/`Query` + `lower` — predicates, AND/OR/NOT, COUNT/ANY/ALL/SUM/MIN/MAX | Shipped (`30924a8`, extended `18c1d85`) | every shape diffed against a per-row oracle that never sees a `Program`; anti-vacuity `0 < selected < n` per case | +| D-QCK-1 | `Filter::Plane` — a resident mask plane as a predicate; the honest "every row" the scaffold owed | Shipped (`18c1d85`). Zero ops, zero slots; projection of a plane returns the PLANE, not a copy | `a_resident_plane_alone_is_a_zero_op_program` pins ops empty + slots 0 + the terminal's operand | +| D-QCK-2 | the survivor skip — an `AND` with a plane child gates every comparison beneath it (`Pred{under}`); the gate passes through `NOT`/`OR`, the DROP does not | Shipped (`18c1d85`). Disable-verified twice: dropping under a negation, and dropping with no child vanishing, both redden | `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` — can-fire (slice: all gated, alpha implied) AND can-stay-silent (`alpha & !X`: still gated, alpha stays a leaf) | +| D-QCK-3 | `lower_fused` — the Boolean skeleton through `mask_risc::fuse` | Shipped (`18c1d85`). Both lowerings run against the same oracle on every fixture | `the_fused_lowering_trades_slots_for_passes` pins BOTH sides (2 passes/2 slots vs 1 pass/4 slots), so a fuser that stopped fusing and an emitter that started allocating per leaf each fail their own line | +| D-QCK-4 | projection — `Agg::Rows` (`Terminal::Keep`, the mask IS the result) and `Agg::BlendI32` (the CASE shape) | Shipped (`18c1d85`) | the kept mask materialises to exactly the oracle's rows; the blend must actually pick from `els` somewhere | +| D-QCK-5 | `lower_group_by` — two-phase, a mask plane where the hash table would be; K gated equalities over the kept filter | Shipped (`18c1d85`). K programs not one, because the IR names no strided operand and no group terminal — `masked_strided_group_sum` exists in the facade and cannot be reached from here yet | per-group counts and sums equal the oracle's AND sum to the ungrouped count (the partition identity); a grouped blend is refused, not answered wrongly | +| D-QCK-6 | the 64k vertical slice `COUNT(alpha & ((A&B)|C))` across five arms | Shipped (`18c1d85`) — closes Phase 3+6 | oracle, both lowerings on the executor, both on the reference evaluator, one number; two-sided anti-vacuity (subset of alpha AND strictly below the ungated remainder) | +| D-QCK-7 | the join — `src_mask → hop → dst_mask` | **Blocked** on mask-risc's PR5: there is no `hop` op to lower to | — | ## mask-risc-executor (PR3 — D-ids minted 2026-09-14, `.claude/plans/mask-risc-executor-v1.md`) From 355ce8552aa660b2eec74ff57fff7fa73dae68e8 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 22:59:34 +0000 Subject: [PATCH 08/25] quack: drive the operator set from the harvest, and repair the harvest first MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I built this crate's operator set from what I know DuckDB looks like. There is a 614-line translation matrix in this repo that reads DuckDB's actual source with file:line and rules every concept KEEP / ADAPT / ELIMINATE / V3 BETTER / NEEDS FALSIFIER, and I had not consulted it. The crate doc now carries the row-by-row table, so the provenance is checkable rather than asserted. THE HARVEST NEEDED REPAIRING BEFORE IT COULD BE CITED. The matrix's own §6 is honest about this: the ruff_cpp_spo harvest ran over 22 .cpp translation units and SEVEN came back 100% Empty, because DuckDB's execution is template-dispatched and lives in headers — so "no row in this matrix cites a harvest TSV as evidence", and §6 names the fix. Done: same harvester, same args, pointed at the headers. 123 methods and 1,622 events where the .cpp pass yielded none. Manifest banked under .claude/harvest/duckdb-headers/, including the two headers that yield NOTHING and why that is information rather than a gap. Two findings from it changed code rather than only prose. `Pred::MatchU64` had been in the IR since PR3 and this crate had no spelling for it, so a borrowed `LaneRef::U64` — edge targets, ids, addresses — was queryable by no query at all. `Cmp::MatchU64` closes it. No new primitive underneath; the gap was entirely in the query language. And the range primitive is real on BOTH sides. The matrix files `mask_set_range` as T1 gap G6 on the strength of V3's own trie-reveal measurement; the header harvest shows DuckDB's bit-plane carrying `TemplatedValidityMask::SetRangeInvalid` — the same packed-u64 carrier, with the range operation already on it. So `Filter::prefix_u32` / `prefix_u64` land as the address-prefix predicate, and their doc says plainly that they lower to a ternary-match SWEEP and not yet a range WRITE: the write waits on G6 per the missing-capability STOP rule rather than being hand-rolled one layer up. The prefix operator is the matrix's R5, "the closest DuckDB comes to the V3 address", and it is where "better than faithful" is concrete rather than a slogan. DuckDB compresses a range to three scalars in SequenceVector, throws it away at ToUnifiedFormat before any kernel runs, then re-manufactures it as a per-row index loop in DataChunk::Slice. The range is kept here instead of rebuilt, and `an_address_prefix_selects_exactly_a_contiguous_trie_subtree` makes that checkable: over an address-ordered lane each additional significant bit must HALVE a contiguous run, the widest prefix selects exactly one row, the empty one selects every row, and the expected sizes are derived from the address arithmetic rather than fitted to what the code returned. Also recorded, because it is the honest state: A1, AdaptiveFilter, is the one row where DuckDB has something this does not. It permutes a conjunction's terms at runtime by measured selectivity, and under the survivor skip term order is a real cost lever here too — the caller owns it today with no help. A selectivity-ordered lowering is a design with a measurement attached, not a line to add, so it is named and not faked. Three fixture defects found and fixed while writing the tests, each one mine and not the code's: a halving claim measured at N=1000, where a subtree cannot halve cleanly once it outgrows the population; a composition fixture whose 56-bit prefix pinned a SINGLE row that the other two conjuncts then excluded, so it read 0/1000 and looked like a defect; and an anchor written against pre-`fmt` text, which is the no-op-patch trap this workspace already has on record. 12 tests, fmt + clippy -D warnings clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/harvest/duckdb-headers/README.md | 139 +++++++++++ crates/lance-graph-quack/src/lib.rs | 297 +++++++++++++++++++++++ 2 files changed, 436 insertions(+) create mode 100644 .claude/harvest/duckdb-headers/README.md diff --git a/.claude/harvest/duckdb-headers/README.md b/.claude/harvest/duckdb-headers/README.md new file mode 100644 index 000000000..790d66e0b --- /dev/null +++ b/.claude/harvest/duckdb-headers/README.md @@ -0,0 +1,139 @@ +# DuckDB HEADER harvest — the repair of the matrix's §6 + +The `ruff_cpp_spo` harvest behind `duckdb-to-v3-translation-matrix-v1.md` was +pointed at 22 `.cpp` translation units. Its own §6 records the result honestly: +**seven TUs harvested to 100 % `Empty`**, because DuckDB's execution is +template-dispatched and lives in HEADERS — so "no row in this matrix cites a +harvest TSV as evidence", and §6 names the fix: point it at the headers. + +Done, 2026-09-14, same harvester (`ruff/examples/harvest_events`, libclang 18), +same `args.txt`. Per-header counts: + +| header | methods | events | +|---|---|---| +| `common_operator_comparison_operators` | 5 | 39 | +| `common_operator_constant_operators` | 0 | 0 | +| `common_row_operations_row_matcher` | 0 | 0 | +| `common_types_selection_vector` | 26 | 310 | +| `common_types_validity_mask` | 61 | 836 | +| `common_types_vector` | 6 | 42 | +| `common_vector_operations_scalar_executor` | 10 | 286 | +| `common_vector_unified_vector_format` | 1 | 6 | +| `execution_expression_executor_state` | 2 | 11 | +| `execution_ht_entry` | 12 | 92 | + +**123 methods, 1,622 events** where the `.cpp` pass yielded none for the same +concepts. Two headers yield nothing and that is information too: +`row_matcher.hpp` and `constant_operators.hpp` are pure declarations whose +bodies are templates no TU instantiates here — the .cpp pass already covered +`row_matcher.cpp`, which did produce events. + +## The two findings that changed code + +1. **`TemplatedValidityMask::SetRangeInvalid`** — DuckDB's own bit-plane, the + same packed-`u64` carrier V3 uses (matrix R6), carries a RANGE write. The + matrix files `mask_set_range` as T1 gap **G6** on the strength of V3's own + trie-reveal measurement; this is the same operation on the other side, and + it is why G6 is a real primitive rather than a wish. `lance-graph-quack` + now spells the address-PREFIX predicate and says plainly that it lowers to + a sweep, not a range write, until G6 lands. +2. **`ScalarExecutor::{Runtime,Static}SelectionSink`** — `Append`, + `AppendInvalidRange`, `FillConstant`, `Result`. This is matrix row **C2** + ("the sink", ELIMINATE: the mask word IS the sink), and it was entirely + invisible to the `.cpp` pass. + +## Method inventory +``` +ConsecutiveChildListInfo.ConsecutiveChildListInfo +Equals.Operation +ExecuteFunctionState.GetFunctionState +ExpressionState.~ExpressionState +GreaterThan.Operation +ScalarExecutor::RuntimeSelectionSink.Append +ScalarExecutor::RuntimeSelectionSink.AppendInvalidRange +ScalarExecutor::RuntimeSelectionSink.FillConstant +ScalarExecutor::RuntimeSelectionSink.Result +ScalarExecutor::RuntimeSelectionSink.RuntimeSelectionSink +ScalarExecutor::StaticSelectionSink.Append +ScalarExecutor::StaticSelectionSink.AppendInvalidRange +ScalarExecutor::StaticSelectionSink.FillConstant +ScalarExecutor::StaticSelectionSink.Result +ScalarExecutor::StaticSelectionSink.StaticSelectionSink +SelectionVector.Capacity +SelectionVector.Incremental +SelectionVector.Initialize +SelectionVector.Inverted +SelectionVector.IsSet +SelectionVector.SelectionVector +SelectionVector.data +SelectionVector.get_index +SelectionVector.get_index_unsafe +SelectionVector.operator= +SelectionVector.operator[] +SelectionVector.sel_data +SelectionVector.set_index +SelectionVector.swap +TemplatedValidityData.EntryCount +TemplatedValidityData.TemplatedValidityData +TemplatedValidityMask.AllValid +TemplatedValidityMask.CanHaveNull +TemplatedValidityMask.CannotHaveNull +TemplatedValidityMask.CheckAllInvalid +TemplatedValidityMask.CheckAllValid +TemplatedValidityMask.Copy +TemplatedValidityMask.CountValid +TemplatedValidityMask.EnsureWritable +TemplatedValidityMask.EntryCount +TemplatedValidityMask.EntryWithValidBits +TemplatedValidityMask.GetAllocationSize +TemplatedValidityMask.GetData +TemplatedValidityMask.GetEntryIndex +TemplatedValidityMask.GetValidityEntry +TemplatedValidityMask.GetValidityEntryUnsafe +TemplatedValidityMask.Initialize +TemplatedValidityMask.IsMaskSet +TemplatedValidityMask.NoneValid +TemplatedValidityMask.Reset +TemplatedValidityMask.RowIsValid +TemplatedValidityMask.RowIsValidUnsafe +TemplatedValidityMask.Set +TemplatedValidityMask.SetAllInvalid +TemplatedValidityMask.SetAllValid +TemplatedValidityMask.SetInvalid +TemplatedValidityMask.SetInvalidUnsafe +TemplatedValidityMask.SetRangeInvalid +TemplatedValidityMask.SetValid +TemplatedValidityMask.SetValidUnsafe +TemplatedValidityMask.SizeInBytes +TemplatedValidityMask.TemplatedValidityMask +TemplatedValidityMask.ValidityMaskSize +ValidityArray.AllValid +ValidityArray.CanHaveNull +ValidityArray.CannotHaveNull +ValidityArray.Capacity +ValidityArray.Initialize +ValidityArray.InitializeEmpty +ValidityArray.Pack +ValidityArray.RowIsValid +ValidityArray.RowIsValidUnsafe +ValidityArray.SetValid +ValidityArray.SetValidUnsafe +ValidityArray.ValidityArray +ValidityMask.ValidityMask +Vector.Buffer +Vector.BufferMutable +Vector.GetBufferRef +Vector.GetType +Vector.GetVectorType +Vector.SetBuffer +duckdb.IncrementAndWrap +ht_entry_t.ExtractSalt +ht_entry_t.GetPointer +ht_entry_t.GetPointerOrNull +ht_entry_t.GetSalt +ht_entry_t.GetSaltWithNulls +ht_entry_t.IsOccupied +ht_entry_t.SetPointer +ht_entry_t.SetSalt +ht_entry_t.ht_entry_t +``` diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index 4245df1f6..7f0b236d7 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -49,6 +49,68 @@ //! differential suite runs both against the same per-row oracle; a consumer //! picks by whether it is scratch-bound or pass-bound. //! +//! # Provenance — this is harvest-driven, not remembered +//! +//! Every operator below answers to a row of +//! `.claude/plans/duckdb-to-v3-translation-matrix-v1.md`, which reads DuckDB's +//! own source with `file:line` and rules each concept KEEP / ADAPT / +//! ELIMINATE / V3 BETTER / NEEDS FALSIFIER. The matrix is the specification; +//! this crate is one reading of it. +//! +//! | operator here | matrix row | verdict there | +//! |---|---|---| +//! | no `SelectionVector`, anywhere | R1 | ELIMINATE — an index list is the materialisation the mask-native invariant forbids | +//! | [`Col`] over a borrowed lane | R2 | KEEP — a flat vector IS an SoA lane | +//! | [`Cmp`] taking a scalar | R3 | ELIMINATE `ConstantVector` — a constant never acquires a representation | +//! | [`Filter::Plane`] | R6 | KEEP the representation, ELIMINATE the role — same packed `u64`, no NULL plane | +//! | [`Filter::prefix_u32`] / [`Filter::prefix_u64`] | R5 | ADAPT — *"the closest DuckDB comes to the V3 address"* | +//! | [`Query`] → [`Program`] | E1 | ADAPT — a state tree with per-node scratch becomes straight-line code over numbered slots | +//! | [`Agg`] as one terminal | E2 | ADAPT — Select-vs-Execute's two carriers collapse to one mask + one terminal tag | +//! | the six `i32` comparisons + two `u32` | E4 | ADAPT — DuckDB's 14-way physical-type switch narrows to what the lanes actually hold | +//! | `And` / `Or` | E5, E6 | ADAPT / **V3 BETTER** — DuckDB must SORT after OR to restore row order (`execute_conjunction.cpp:139`); a mask never lost it | +//! | [`Agg::BlendI32`] | E7 | ADAPT — CASE's narrowing false-set becomes a blend | +//! | [`Agg::Any`] / [`Agg::All`] | C7 | V3 BETTER — `HasNull`/`HasNotNull` are literally `mask_any`/`mask_all` | +//! | [`lower_group_by`] | A3 | V3 BETTER for the addressed case — a mask plane where the hash table would be | +//! +//! The harvest itself was re-run on 2026-09-14 and its first pass was +//! **repaired**, which is why the table above can cite what it cites. The +//! matrix's own §6 recorded the failure honestly: the `ruff_cpp_spo` harvest +//! had been pointed at 22 `.cpp` translation units, **seven of which came back +//! 100 % `Empty`**, because DuckDB's execution is template-dispatched and lives +//! in headers — so *"no row in this matrix cites a harvest TSV as evidence"*. +//! Pointing the same harvester at the headers (`scalar_executor.hpp`, +//! `comparison_operators.hpp`, `validity_mask.hpp`, `selection_vector.hpp`, +//! `ht_entry.hpp`, `vector.hpp`) yields **123 methods and 1,622 events** where +//! the `.cpp` pass yielded none. +//! +//! # What the header harvest changed here +//! +//! Two things, and both are in the code rather than only in this comment. +//! +//! **`Pred::MatchU64` was reachable from nothing.** It had been in the IR since +//! PR3, and this crate had no spelling for it, so a borrowed `LaneRef::U64` — +//! edge targets, ids, addresses — was queryable by no query. [`Cmp::MatchU64`] +//! closes that, and it is what the prefix operator over a 64-bit address needs. +//! +//! **The range primitive is real, and DuckDB's own bit-plane has it.** The +//! matrix files `mask_set_range` as T1 gap G6 on the strength of V3's +//! trie-reveal measurement. The header harvest shows the same operation on the +//! other side: `TemplatedValidityMask::SetRangeInvalid`, on the same packed-`u64` +//! carrier V3 uses. So [`Filter::prefix_u32`] lowers to a ternary-match SWEEP +//! today and says so plainly; the range WRITE waits on the primitive, per the +//! missing-capability STOP rule, rather than being hand-rolled one layer up. +//! +//! # The one thing DuckDB has that this does not +//! +//! `AdaptiveFilter` (matrix row A1, its only NEEDS-FALSIFIER of this kind) +//! permutes a conjunction's terms at RUNTIME by measured selectivity. This +//! crate lowers children in the order they were written, and that is not +//! neutral here: under the survivor skip a cheap, highly selective conjunct +//! placed first shrinks every later predicate's live-word count, so order is a +//! real cost lever and the caller currently owns it with no help. Saying so is +//! the honest state; a selectivity-ordered lowering is a design with a +//! measurement attached, not a line to add. +//! //! # Status //! //! Filter (`=`/`<>`/`<`/`<=`/`>`/`>=`/ternary match/`IN`, `AND`/`OR`/`NOT`, @@ -114,6 +176,18 @@ pub enum Cmp { /// Which bits participate; zero means "don't care". care: u32, }, + /// The ternary match over a 64-bit lane — edge targets, ids, addresses. + /// + /// `Pred::MatchU64` has been in the IR since PR3 and was unreachable from + /// this crate, so a `LaneRef::U64` lane could be borrowed and never + /// queried. That is the whole of the gap this closes; there is no new + /// primitive underneath. + MatchU64 { + /// The bits to compare. + pattern: u64, + /// Which bits participate; zero means "don't care". + care: u64, + }, } /// A filter expression: predicates over columns and resident planes, composed @@ -188,6 +262,71 @@ impl Filter { .collect(), ) } + + /// Rows whose 32-bit address lane starts with the top `bits` of `prefix` — + /// the operator SQL has no name for and the V3 address was built around. + /// + /// # Why this is not just another equality + /// + /// On an address-ordered lane a prefix names a CONTIGUOUS RANGE: every row + /// under one trie node is `2^(32 - bits)` consecutive addresses. DuckDB + /// arrives at the same shape and then discards it — `SequenceVector` + /// compresses a range to three scalars (`vector.cpp:498-500`) and + /// `ToUnifiedFormat` flattens it to N materialised values before any kernel + /// runs (`vector.cpp:461-465`), after which `DataChunk::Slice` re-manufactures + /// the range as a per-row index loop (`data_chunk.cpp:394-397`). That round + /// trip is the matrix's R5/R8, and it is what "better than faithful" means + /// here: the range is kept, not rebuilt. + /// + /// # What this actually lowers to today, stated exactly + /// + /// A ternary match — a full sweep of the lane, one pass, no allocation. It + /// is NOT yet a range WRITE. The range write is `mask_set_range`, the + /// matrix's T1 gap G6, and it is absent from `ndarray::simd`; per the + /// missing-capability STOP rule a consumer does not hand-roll it one layer + /// up, so this crate spells the PREDICATE and waits for the primitive. + /// + /// That the primitive is real rather than wished for is the harvest's + /// evidence, not this crate's opinion: DuckDB's own bit-plane carries + /// `TemplatedValidityMask::SetRangeInvalid` (`validity_mask.hpp`, harvested + /// 2026-09-14 from the headers) — the same packed-`u64` representation V3 + /// uses, with the range operation already on it. + /// + /// `bits` is clamped to 32; `bits == 0` matches every row (care is empty), + /// which is the honest reading of "no significant bits" rather than an + /// error, and `bits == 32` matches exactly one address. + pub fn prefix_u32(col: Col, prefix: u32, bits: u32) -> Self { + let care = match bits.min(32) { + 0 => 0, + b => u32::MAX << (32 - b), + }; + Filter::Cmp( + col, + Cmp::MatchU32 { + pattern: prefix & care, + care, + }, + ) + } + + /// [`Filter::prefix_u32`] over a 64-bit address lane. + /// + /// This is the one that reaches the canonical GUID's own prefix: classid, + /// then HEEL/HIP/TWIG, each a nibble-addressed tier of the cascade. A + /// `bits` that lands on a tier boundary selects exactly that subtree. + pub fn prefix_u64(col: Col, prefix: u64, bits: u32) -> Self { + let care = match bits.min(64) { + 0 => 0, + b => u64::MAX << (64 - b), + }; + Filter::Cmp( + col, + Cmp::MatchU64 { + pattern: prefix & care, + care, + }, + ) + } } /// Why a query could not be lowered. @@ -635,6 +774,11 @@ fn pred_of(col: Col, cmp: Cmp) -> Pred { pattern, care, }, + Cmp::MatchU64 { pattern, care } => Pred::MatchU64 { + lane, + pattern, + care, + }, } } @@ -651,6 +795,7 @@ mod tests { const VALS: Col = Col(0); const CLASS: Col = Col(1); const ALT: Col = Col(2); + const ADDR: Col = Col(3); const ALPHA: Mask = Mask(0); const FOCUS: Mask = Mask(1); @@ -659,6 +804,12 @@ mod tests { vals: Vec, classes: Vec, alt: Vec, + /// An ADDRESS-ORDERED lane: row `i` holds address `i << 8`, so the + /// top bits are a trie prefix and a prefix predicate must select a + /// contiguous row range. That ordering is the fixture's whole point — + /// on an unordered lane a prefix is just an equality with holes, and + /// the range claim would be untestable. + addr: Vec, masks: Vec>, } @@ -677,11 +828,13 @@ mod tests { .collect(); let classes = (0..n).map(|i| (i % 5) as u32).collect(); let alt = (0..n).map(|i| ((i as i64 * 13) % 89 - 44) as i32).collect(); + let addr = (0..n).map(|i| (i as u64) << 8).collect(); let masks = vec![plane(n, |r| r % 3 != 0), plane(n, |r| r % 7 == 0)]; Fx { vals, classes, alt, + addr, masks, } } @@ -705,6 +858,13 @@ mod tests { } } + fn u64_at(&self, col: Col, row: usize) -> u64 { + match col { + ADDR => self.addr[row], + other => panic!("{other:?} is not a 64-bit lane of the fixture"), + } + } + fn bit(&self, m: Mask, row: usize) -> bool { (self.masks[usize::from(m.0)][row / 64] >> (row % 64)) & 1 == 1 } @@ -728,6 +888,9 @@ mod tests { Cmp::MatchU32 { pattern, care } => { (self.u32_at(*col, row) ^ pattern) & care == 0 } + Cmp::MatchU64 { pattern, care } => { + (self.u64_at(*col, row) ^ pattern) & care == 0 + } }, Filter::Plane(m) => self.bit(*m, row), Filter::And(ps) => ps.iter().all(|p| self.oracle(p, row)), @@ -745,6 +908,7 @@ mod tests { LaneRef::I32(&self.vals), LaneRef::U32(&self.classes), LaneRef::I32(&self.alt), + LaneRef::U64(&self.addr), ]; let masks: Vec<&[u64]> = self.masks.iter().chain(extra).map(Vec::as_slice).collect(); f(&Planes { @@ -1376,6 +1540,139 @@ mod tests { ); } + /// FAILS IF: an address prefix does not select a CONTIGUOUS row range, or + /// the range is not the size the prefix arithmetic says it is. + /// + /// This is the matrix's §5.1 claim — *the address IS the trie, so a prefix + /// predicate is a range rather than a sweep* — made checkable instead of + /// asserted. It matters because the whole "better than faithful" argument + /// rests on it: DuckDB has the compressed range in `SequenceVector` and + /// throws it away at `ToUnifiedFormat`, then rebuilds it as a per-row index + /// loop in `DataChunk::Slice`. If V3's prefix did not actually select a + /// range, keeping the range would be keeping nothing. + /// + /// Two-sided, because "selects a contiguous run" alone would hold for a + /// predicate that selected everything: each additional significant bit must + /// HALVE the run, and the widest prefix must select exactly one row. + #[test] + fn an_address_prefix_selects_exactly_a_contiguous_trie_subtree() { + // A POWER-OF-TWO population, deliberately: a subtree can only halve + // cleanly while it still fits inside the population. At N = 1000 the + // widest prefix here selects 1000 rather than 1024, and the halving + // claim then reads as a defect when it is a fixture artifact — which + // is exactly what the first version of this test measured. + const POW2: usize = 1024; + let fx = Fx::new(POW2); + // The lane is `row << 8`, so bit 8 + k of the address is bit k of the + // row index: a prefix of `24 + b` significant bits pins the top `b` + // bits of the row index and leaves `24 - 8 = 16`... stated the way the + // arithmetic actually runs, `care = !0 << (64 - bits)`, and a row + // survives when its address agrees on those bits. + let base_row = 384usize; + let base = fx.addr[base_row]; + // The arithmetic, stated so the expected sizes are DERIVED and not + // fitted to what the code happened to return: row `r` sits at address + // `r << 8`, so row bit `k` is address bit `8 + k`. A prefix of `bits` + // significant bits pins address bits `[64 - bits, 63]`, hence row bits + // `k >= 56 - bits`, leaving `clamp(56 - bits, 0, 10)` of them free — a + // subtree of `2^free` consecutive rows. + let subtree_of = |bits: u32| 1usize << (56u32.saturating_sub(bits).min(10)); + + let mut previous: Option = None; + for bits in [46u32, 47, 48, 49, 50] { + let f = Filter::prefix_u64(ADDR, base, bits); + let rows = fx.rows(&f); + assert!(!rows.is_empty(), "bits={bits} selected nothing"); + + // Contiguous: the selected rows are consecutive, with no holes. + // A sweep over an unordered lane could not satisfy this, which is + // exactly the property being claimed. + let first = rows[0]; + assert!( + rows.iter().enumerate().all(|(i, &r)| r == first + i), + "bits={bits} selected a non-contiguous set: {:?}..", + &rows[..rows.len().min(8)] + ); + assert!( + rows.contains(&base_row), + "bits={bits} must contain the row the prefix was taken from" + ); + + // Each extra significant bit halves the subtree. That is the + // arithmetic the prefix IS; a care mask built wrongly would still + // select a contiguous run, just the wrong one. + assert_eq!( + rows.len(), + subtree_of(bits), + "bits={bits} selected {} rows; the prefix arithmetic says {}", + rows.len(), + subtree_of(bits) + ); + if let Some(prev) = previous { + assert_eq!( + rows.len() * 2, + prev, + "bits={bits} selected {} rows against {prev} for one bit fewer \ + — a trie level must halve", + rows.len() + ); + } + previous = Some(rows.len()); + } + assert_eq!( + previous, + Some(64), + "the 50-bit prefix pins the row index down to a 64-row subtree" + ); + + // The widest prefix is a single address; the empty one is every row. + assert_eq!(fx.count(&Filter::prefix_u64(ADDR, base, 64)), 1); + assert_eq!(fx.count(&Filter::prefix_u64(ADDR, base, 0)), POW2); + } + + /// FAILS IF: the 64-bit ternary match disagrees with an independent per-row + /// reading, or the lowered program does not reach the `U64` lane at all. + /// + /// `Pred::MatchU64` shipped with the IR in PR3 and this crate could not + /// spell it, so a borrowed `LaneRef::U64` — edge targets, ids, addresses — + /// was queryable by nothing. The anti-vacuity half matters more than usual + /// here: a care mask of zero matches every row, which is a real answer and + /// a useless test. + #[test] + fn the_64_bit_ternary_match_agrees_with_a_per_row_reading() { + let fx = Fx::new(N); + let cases: [(&str, u64, u64); 3] = [ + ("one low nibble of the address", 0x300, 0xF00), + ("a sparse care mask", 0x2_0100, 0x3_0100), + ("every bit — exact equality", fx.addr[7], u64::MAX), + ]; + for (label, pattern, care) in cases { + let f = Filter::cmp(ADDR, Cmp::MatchU64 { pattern, care }); + let expected = fx.rows(&f).len(); + assert_eq!(fx.count(&f), expected, "{label}"); + assert!( + expected > 0 && expected < N, + "{label} selects {expected}/{N} — a degenerate case proves nothing" + ); + } + // Composed with the rest of the vocabulary, so the U64 lane is not a + // second world that only works alone. + // 48 significant bits, not 56: at 56 the subtree is a SINGLE row, and + // whether the conjunction is non-empty then turns on whether that one + // row happens to satisfy the other two conjuncts — which it does not, + // so the first version measured 0/1000 and looked like a defect. A + // 256-row subtree is what makes this a composition test rather than a + // coin flip. + let mixed = Filter::and([ + Filter::plane(ALPHA), + Filter::prefix_u64(ADDR, fx.addr[500], 48), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ]); + let expected = fx.rows(&mixed).len(); + assert!(expected > 0 && expected < N, "{expected}/{N}"); + assert_eq!(fx.count(&mixed), expected); + } + /// FAILS IF: any arm of the vertical slice disagrees on a 64k slab. /// /// `COUNT(alpha & ((A & B) | C))` over 65,536 rows through the per-row From 3f6bf8c7b73e07243a8a9a159d989bc9d4d8e2f7 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 23:19:27 +0000 Subject: [PATCH 09/25] quack: the accumulator gate, and A1's falsifier run rather than ported MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wiring matrix row A1 exposed a correctness bug one layer below it, so this lands both. An `AND` gated its children only under a resident PLANE. `lgj-abi`'s `plan_lower` already gated each later conjunct under the ACCUMULATOR built so far; the two lowerings were implementing different laws for one algebra. Now quack does both: the first child establishes the accumulator, every later child carries `under = Scratch(acc)`. The asymmetry `plan_lower` documents carries over unchanged and is the whole correctness question — an `OR` must NOT gate its children on its own accumulator, because `acc | p` depends on `p` exactly where `acc` is ZERO, which is what a gate under `acc` discards. A `Pred` carries exactly one `under`, so with both gates available the accumulator wins (it is strictly narrower — the plane is already folded into it). But the plane DROP decision depends on the plane still gating the term that implies it. On `alpha AND focus AND v<50` the accumulator started as `focus`, which is not a subset of `alpha`: the plane was elided while nothing constrained it, and rows outside `alpha` were counted. Fix: rotate a child whose result is a subset of the plane to the FRONT, so the accumulator starts inside the plane. Found once, then found again one level down — a gated `AND` nested in a gated `AND` reproduced it — which is why the helper is called from BOTH arms rather than inlined in the one that first needed it. `Filter::and_by_skip` takes the caller's measured skip score per conjunct and orders the `AND` by it. The matrix's A1 row was NEEDS FALSIFIER; `examples/adaptive_order_probe.rs` is that falsifier — 65 536 rows, five conjuncts, all 120 permutations, four regimes, counting the 64-row words a gated `Pred` does not evaluate. regime survivors as written best spread selective 36 (0.055%) 5.66% 80.66% 75.00 pts, 14.2x moderate 14311 (21.8%) 0.00% 0.00% 0 permissive 61777 (94.3%) 0.00% 0.00% 0 clustered 31 (0.047%) 99.90% 99.90% 99.90 pts vs worst Order MOVES the skip fraction, so A1 is not ELIMINATE. Three findings change what should be built: 1. The knob is inert wherever the population is not sparse, by ARITHMETIC rather than by implementation. At 21.8% survival a 64-row word is all-dead with probability ~2e-7, so no ordering skips anything and best == worst. 2. The control signal is DEAD WORDS, not selectivity. Selective and clustered have near-identical survivor counts (36 vs 31) and differ by 19 points of achievable skip, because one conjunct's survivors are contiguous and the other's are scattered. 3. DuckDB's adjacent-transposition hill-climb is deliberately NOT ported. The matrix anticipated the reason ("its swap-likeliness decay is not obviously the right control law"); the measurement says why — the optimised quantity is a step function of clustering, not a smooth function of selectivity, so a local search over adjacent swaps explores the wrong landscape. The score is the CALLER's: this crate builds programs and never evaluates one, so it cannot measure anything. No decay, no intervals, no observe/ execute/warmup counters. Five disables, all red: gate-never-on-accumulator; an `OR` gating its children on its own accumulator; `hoist_gate_subset` never rotating; `and_by_skip` sorting ascending; `and_by_skip` not sorting. 13 tests, clippy `-D warnings` and `fmt` clean. Board: STATUS_BOARD D-QCK-8/D-QCK-9, LATEST_STATE 2026-09-14 (5), and the matrix's A1 row regraded in place with the measured table in a new §8a. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/LATEST_STATE.md | 58 +++ .claude/board/STATUS_BOARD.md | 2 + .../duckdb-to-v3-translation-matrix-v1.md | 88 +++- .../examples/adaptive_order_probe.rs | 400 +++++++++++++++++ crates/lance-graph-quack/src/lib.rs | 407 +++++++++++++++--- 5 files changed, 889 insertions(+), 66 deletions(-) create mode 100644 crates/lance-graph-quack/examples/adaptive_order_probe.rs diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 142c1d1cb..8a9797d53 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -76,6 +76,64 @@ census is measured and NOT yet on the board — see the arc entry's *Un-recorded* bullet. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1233. +## 2026-09-14 (5) — quack A1: the accumulator gate, and the falsifier the matrix asked for + +Two changes in `crates/lance-graph-quack`, one of them a correctness fix the +other one exposed. + +**The accumulator gate (D-QCK-8) — the convergence.** An `AND` gated its +children only under a resident PLANE. `lgj-abi`'s `plan_lower` already gated +each later conjunct under the ACCUMULATOR built so far, and the two lowerings +were implementing different laws for the same algebra. Now quack does both: +the first child establishes the accumulator, every later child is emitted with +`under = Scratch(acc)`. The asymmetry `plan_lower` documents carries over +unchanged and is the whole correctness question — an `OR` must NOT gate its +children on its own accumulator, because `acc | p` depends on `p` exactly +where `acc` is ZERO, which is what a gate under `acc` discards. + +**`hoist_gate_subset` — the bug the gate exposed.** With both gates available +a `Pred` can carry only one `under`, so the accumulator wins (it is strictly +narrower — the plane was folded into it). But the plane DROP decision depends +on the plane still gating the term that implies it, and on `alpha AND focus +AND v<50` the accumulator started as `focus`, which is not a subset of +`alpha` — the plane was elided while nothing constrained it, and rows outside +`alpha` were counted. Fix: rotate a child whose result is a subset of the +plane to the FRONT, so the accumulator starts inside the plane. Found once, +then found again one level down (a gated `AND` nested in a gated `AND` +reproduced it), which is why the helper is called from BOTH arms rather than +inlined in the one that first needed it. + +**A1 (D-QCK-9) — RUN, not ported.** `Filter::and_by_skip` takes the caller's +measured skip score per conjunct and orders the `AND` by it. The matrix's A1 +row was NEEDS FALSIFIER; `examples/adaptive_order_probe.rs` is that falsifier +— 65 536 rows, 5 conjuncts, all 120 permutations, 4 regimes, counting +64-row words a gated `Pred` does not evaluate: + +| regime | survivors | as written | best | spread | +|---|---|---|---|---| +| selective | 36 (0.055%) | 5.66% | 80.66% | 75.00 pts, 14.2x | +| moderate | 14 311 (21.8%) | 0.00% | 0.00% | 0 | +| permissive | 61 777 (94.3%) | 0.00% | 0.00% | 0 | +| clustered (address prefix) | 31 (0.047%) | 99.90% | 99.90% | 0 | + +Verdict: order MOVES the skip fraction, so A1 is not ELIMINATE. But the +control signal is **dead words, not selectivity** — selective and clustered +have near-identical survivor counts (36 vs 31) and differ by 19 points of +skip, because one conjunct's survivors are contiguous and the other's are +scattered. At 21.8% survival a 64-row word is all-dead with probability +~2e-7, so no ordering can skip anything and the knob is inert by arithmetic, +not by implementation. DuckDB's adjacent-transposition hill-climb with +swap-likeliness decay is deliberately NOT ported: the quantity is a step +function of clustering, not a smooth function of selectivity, so a local +search over adjacent swaps explores the wrong landscape. The score is the +CALLER's — this crate builds programs and never evaluates one, so it cannot +measure anything. + +Five disables, all red: gate-never-on-accumulator; an `OR` gating its children +on its own accumulator; `hoist_gate_subset` never rotating; `and_by_skip` +sorting ascending; `and_by_skip` not sorting. 13 tests, clippy `-D warnings` +and `fmt` clean. + ## 2026-09-14 (3) — `lance-graph-quack`: the DuckDB-shaped surface whose operators ARE masking ops New workspace member `crates/lance-graph-quack`, one dependency diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index f81e86001..c48b2e1b1 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -29,6 +29,8 @@ evaluates one; `execute` stays the consumer's call on a scratch it owns. | D-QCK-5 | `lower_group_by` — two-phase, a mask plane where the hash table would be; K gated equalities over the kept filter | Shipped (`18c1d85`). K programs not one, because the IR names no strided operand and no group terminal — `masked_strided_group_sum` exists in the facade and cannot be reached from here yet | per-group counts and sums equal the oracle's AND sum to the ungrouped count (the partition identity); a grouped blend is refused, not answered wrongly | | D-QCK-6 | the 64k vertical slice `COUNT(alpha & ((A&B)|C))` across five arms | Shipped (`18c1d85`) — closes Phase 3+6 | oracle, both lowerings on the executor, both on the reference evaluator, one number; two-sided anti-vacuity (subset of alpha AND strictly below the ungated remainder) | | D-QCK-7 | the join — `src_mask → hop → dst_mask` | **Blocked** on mask-risc's PR5: there is no `hop` op to lower to | — | +| D-QCK-8 | accumulator gating — an `AND` gates each later child under the accumulator built so far, not only under a resident plane. The same rule `lgj-abi`'s `plan_lower` already had; the two lowerings now implement one law | Shipped. Disable-verified 3×: gate-never-on-accumulator, an `OR` gating its children on its own accumulator, and `hoist_gate_subset` never rotating — each reddens | `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` via the new `gate_shape` helper (all-gated / on-accumulator / dropped, three facts not two); `every_lowered_filter_agrees_with_an_independent_per_row_reading` is what catches the DROP going unsound | +| D-QCK-9 | A1 `Filter::and_by_skip` — the caller's measured skip score orders an `AND`'s conjuncts. DuckDB's adjacent-transposition hill-climb deliberately NOT ported | Shipped, with the matrix's A1 falsifier RUN (`examples/adaptive_order_probe.rs`, 65 536 rows × 5 conjuncts × all 120 permutations × 4 regimes). Order moves the skip fraction, so A1 is not ELIMINATE — but the control signal is DEAD WORDS, not selectivity | `skip_ordering_moves_the_work_and_never_the_answer`: the reordering agrees with the oracle (not with the other lowering), the emitted program's first predicate is the highest-scored, and descending/no-sort disables both redden | ## mask-risc-executor (PR3 — D-ids minted 2026-09-14, `.claude/plans/mask-risc-executor-v1.md`) diff --git a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md index fc74fb0b3..42531d62b 100644 --- a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md +++ b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md @@ -145,7 +145,7 @@ Verdict vocabulary, used strictly: | # | DuckDB concept | verdict | DuckDB file:line | V3 / ndarray counterpart | mechanism | FALSIFIER | |---|---|---|---|---|---|---| -| A1 | `AdaptiveFilter` — runtime permutation of conjunction terms by measured selectivity | **NEEDS FALSIFIER — the one row where DuckDB has something V3 does not** | `adaptive_filter.cpp:17-29` (ctor), `:113-186` (`AdaptRuntimeStatistics`); the swap `:127,163`; likeliness decay `:132-134`; intervals `observe=10 / execute=20 / warmup=5` (`:17,31,180`) | **absent** — `Program.ops` is a fixed `Vec` (`ir.rs:133`) executed in order, with no reordering and no measurement | An adjacent-transposition hill-climb: swap two neighbouring terms, measure 10 iterations, keep if mean runtime dropped else revert and **halve** that position's swap likeliness (floor 1, so exploration never dies). `GetInitialOrder` seeds from the optimizer's static heuristic. V3 has no analog at any layer. | **The falsifier is prior to the port, and it may kill the whole idea.** DuckDB's reordering pays because term k runs only on survivors of 1..k−1, so a selective term first *shrinks the input*. In V3 a predicate sweep costs the **full column** regardless of position (`gt_i32_to_mask` writes every word; D-GTM-0m: 22.4–22.8 µs, survivor-independent) — so **reordering saves nothing on generation**. It can only save by *avoidance*: `mask_any` says empty, skip the rest (E5), or `MaskOp::Pred { under }` skips chunks (`ir.rs:84-86`). **Measure:** on a representative predicate stream, what fraction of `Pred` ops are skippable by `under`, and does term order change that fraction? If order does not move the skip fraction, A1 is ELIMINATE and the machinery must not be ported. If it does, the port is a reordering of *skip opportunities*, not of costs — a different algorithm than DuckDB's, and its swap-likeliness decay is not obviously the right control law for it. | +| A1 | `AdaptiveFilter` — runtime permutation of conjunction terms by measured selectivity | **⊘ FALSIFIER RUN 2026-09-14 → ADAPT (not ELIMINATE), and the control law is DIFFERENT — see §8a. Was: NEEDS FALSIFIER — the one row where DuckDB has something V3 does not** | `adaptive_filter.cpp:17-29` (ctor), `:113-186` (`AdaptRuntimeStatistics`); the swap `:127,163`; likeliness decay `:132-134`; intervals `observe=10 / execute=20 / warmup=5` (`:17,31,180`) | **absent** — `Program.ops` is a fixed `Vec` (`ir.rs:133`) executed in order, with no reordering and no measurement | An adjacent-transposition hill-climb: swap two neighbouring terms, measure 10 iterations, keep if mean runtime dropped else revert and **halve** that position's swap likeliness (floor 1, so exploration never dies). `GetInitialOrder` seeds from the optimizer's static heuristic. V3 has no analog at any layer. | **The falsifier is prior to the port, and it may kill the whole idea.** DuckDB's reordering pays because term k runs only on survivors of 1..k−1, so a selective term first *shrinks the input*. In V3 a predicate sweep costs the **full column** regardless of position (`gt_i32_to_mask` writes every word; D-GTM-0m: 22.4–22.8 µs, survivor-independent) — so **reordering saves nothing on generation**. It can only save by *avoidance*: `mask_any` says empty, skip the rest (E5), or `MaskOp::Pred { under }` skips chunks (`ir.rs:84-86`). **Measure:** on a representative predicate stream, what fraction of `Pred` ops are skippable by `under`, and does term order change that fraction? If order does not move the skip fraction, A1 is ELIMINATE and the machinery must not be ported. If it does, the port is a reordering of *skip opportunities*, not of costs — a different algorithm than DuckDB's, and its swap-likeliness decay is not obviously the right control law for it. | | A2 | `VectorHash` / `CombineHash` (`TightLoopHash`, `CombineHashScalar`) | **NEEDS FALSIFIER** | `vector_hash.cpp:50-67` (`TightLoopHash`); `:43-47` (`CombineHashScalar`); NULL_HASH `:24` | **absent** from `simd_masking_ops` by design (a hash is a value, not a mask) | Hashing is not a mask operation and does not belong in T1. The open question is whether V3 needs hashing **at all** on the addressed path: a minted classid prefix is already a *semantic* bucket, so the hash's job (map a key to a bucket) is done by the address. | Where does V3 need a hash that the address does not already give? Candidate: joining on a **non-address column**. Enumerate real consumers; if every join key is a minted address, A2 is ELIMINATE for this substrate and hashing stays outside T1 entirely. If a non-address join key exists, A2 is a real gap but belongs to a value-kernel family, **not** to the mask vocabulary — do not let it widen `simd_masking_ops`. | | A3 | `JoinHashTable` — probe, salt prefilter, linear-probe chains | **V3 BETTER for the addressed case; NEEDS FALSIFIER otherwise** | `join_hashtable.cpp:248-296` (`ProbeForPointersInternal`); `ht_entry.hpp:34-37` (salt/pointer split), `:49-51` (`IsOccupied`) | prefix range on the minted address; `ternary_match_u32_to_mask` with a care mask over the prefix (`simd_masking_ops.rs:1285`) | `ht_entry_t` packs **16 bits of salt + 48 bits of pointer** in one u64 and prefilters on the salt to avoid a full key compare (`:271-277`). That is a *probabilistic* prefix derived from a hash. **The V3 classid prefix is the real thing**: matching it is not a filter that may be wrong, it is a **contiguous row range** (R5, D-GTM-0m: 49–99 ns vs 22.4 µs). Also note `IsOccupied() == (value != 0)` (`ht_entry.hpp:50`) — DuckDB independently arrived at zero-is-absence, the same convention as `CLAUDE.md:1657`'s ladder. | The comparison only holds when **both sides are minted into the same address space**. A join between a minted V3 population and an external, unminted key set has no shared prefix and falls back to A2's hash question. Falsifier: exhibit the intended join workload. If either side is unminted, "V3 BETTER" is false for it and the honest verdict is NEEDS FALSIFIER. Second: `join_hashtable.cpp` is 6,986 harvested events, by far the largest TU read — this row is a **reading of two functions, not of the join**; spilling, radix partitioning, and chain building are not assessed. | | A4 | `RowMatcher` (multi-column match by sequential in-place sel compaction) | **ADAPT — the single cleanest mapping in the matrix** | `row_matcher.cpp:19-62` (`TemplatedMatchLoop`); the in-place compaction `:56`; the 4-way validity specialization `:64-89`; `row_matcher.hpp:47` (`Match`) | `ternary_match_strided_to_mask(bytes, first_offset, stride_bytes, count, &[u8;12] pattern, &[u8;12] care, out)` (`simd_masking_ops.rs:1415-1424`) | DuckDB matches **one key column at a time**, narrowing `sel` in place each round (`sel.set_index(match_count++, idx)`, `:56`) — k columns, k passes, k compactions. The V3 12-byte register **is** the multi-column key, and the `care` mask names which rails participate, so k columns become **one pass** with no compaction. The strided form is shaped for exactly the V3 facet: `stride_bytes = 16`, `first_offset = 4`, pattern/care = `[u8;12]` — the 4+12 atom of `le-contract.md` §1. | **Scope, and it is narrow.** `ternary_match` is *equality with don't-cares only*. `RowMatcher` dispatches a per-column `ExpressionType` predicate (`row_matcher.hpp:52-56`) and NULL-semantics variants (DISTINCT FROM). So A4 covers the **equi-match** case and nothing else. Falsifier: take a real multi-column match from a V3 consumer; if any column needs an ordered or distinct-from predicate, the one-pass claim fails for it and it degrades to per-column masks + `mask_and` — still allocation-free, but k passes, not one. | @@ -526,11 +526,11 @@ rounding error dressed as a finding.* | verdict | whole rows | split halves | rows | |---|---|---|---| -| **ADAPT** | **14** | — | R5, R7, R8, E1, E2, E4, E5, E7, E10, E11, C1, C3, C5, A4 | +| **ADAPT** | **15** | — | R5, R7, R8, E1, E2, E4, E5, E7, E10, E11, C1, C3, C5, A4, **A1** (moved 2026-09-14, §8a) | | **ELIMINATE** | **8** | **+2** | R1, R3, R4, E3, E8, E9, C2, C4 · *plus* R6(the NULL **role**), A5(composition) | | **V3 BETTER** | **3** | **+1** | E6, C6, C7 · *plus* A3(addressed case) | | **KEEP** | **2** | **+2** | R2, C8 · *plus* R6(the mask **representation**), A5(egress) | -| **NEEDS FALSIFIER** | **2** | **+1** | A1, A2 · *plus* A3(unaddressed case) | +| **NEEDS FALSIFIER** | **1** | **+1** | A2 · *plus* A3(unaddressed case) — ⊘ A1 left this column 2026-09-14 | | | **29** | **+6** | = 35 entries / 32 rows | Read the shape rather than the totals: **ADAPT dominates (14/32)**, which is the @@ -543,6 +543,17 @@ does not. And exactly **one** DuckDB mechanism has no V3 counterpart at all (A1, `AdaptiveFilter`) — which is why its falsifier is one of the three that gate Phase 2. +> **⊘ A1 UPDATE 2026-09-14 — the falsifier ran (§8a) and moved the row to +> ADAPT, which strengthens the shape above rather than dulling it.** Term order +> does change the skip fraction (up to 75 percentage points, 14.2×), so the +> mechanism is not deleted — but what transferred is the INTENT (order the +> conjuncts) and what did not is the CARRIER (an adjacent-transposition +> hill-climb over measured runtimes). That is the same intent-transfers / +> carrier-does-not pattern the paragraph above describes for representation, +> arriving from a third direction: the optimised quantity here is a step +> function of clustering, so the control law has to change even though the +> goal does not. + **Phase 2's ordering falls out of the falsifiers, not from the verdict counts.** Three measurements gate the largest number of downstream rows and should run before any design: @@ -559,10 +570,15 @@ before any design: 75 µs M1b generation. The substrate's native width is the byte (`le-contract.md` §3); i32 is the foreign one, and D-GTM-0m's numbers are explicitly **upper bounds** because of it. -3. **A1's prior question** — does term order change the *skip fraction*? If not, +3. ~~**A1's prior question** — does term order change the *skip fraction*? If not, `AdaptiveFilter` is ELIMINATE and the only DuckDB mechanism V3 lacks turns out not to be needed. That is a cheap answer with a large consequence, and it is - answered by measurement (1) plus one sweep. + answered by measurement (1) plus one sweep.~~ **ANSWERED 2026-09-14 — see + §8a.** Order moves it (up to 75 percentage points, 14.2×), so A1 is not + ELIMINATE; but the signal is dead words rather than selectivity and the + hill-climb is not ported. Note this did NOT need measurement (1): the skip + fraction is a count of avoided word-evaluations, not a timing, so it was + answerable on its own. **What Phase 2 must NOT do**: build G5 (masked compaction) before A5's egress count exists — it is the most expensive gap on the list (five backends, BACKEND @@ -571,6 +587,68 @@ counted. --- +## §8a — A1's falsifier, RUN (2026-09-14) + +The queue item above asks one question: *does term order change the skip +fraction?* It does. `crates/lance-graph-quack/examples/adaptive_order_probe.rs` +is the measurement — 65 536 rows (1024 words), five conjuncts, **all 120 +permutations**, four regimes, counting the 64-row words a gated +`MaskOp::Pred { under }` does not evaluate, summed over the 4096 gated +positions of one ordering (term 0 is the ungated seed). + +| regime | survivors | as written | worst | best | spread | +|---|---|---|---|---|---| +| selective (a long tail of survivors) | 36 (0.055 %) | 5.66 % | 5.66 % | **80.66 %** | 75.00 pts, 14.2× | +| moderate | 14 311 (21.8 %) | 0.00 % | 0.00 % | 0.00 % | 0 | +| permissive | 61 777 (94.3 %) | 0.00 % | 0.00 % | 0.00 % | 0 | +| clustered (one conjunct is an ADDRESS PREFIX) | 31 (0.047 %) | **99.90 %** | 0.00 % | 99.90 % | 99.90 pts | + +**Verdict: ADAPT, not ELIMINATE — but not DuckDB's algorithm.** Three findings, +each of which changes what should be built: + +1. **The knob is inert wherever the population is not sparse, by arithmetic + rather than by implementation.** At 21.8 % survival a 64-row word is + all-dead with probability ≈ 0.782⁶⁴ ≈ 2·10⁻⁷, so *no* ordering skips + anything and the best and worst permutations are identical. Reordering is + not "a small win" in the moderate and permissive regimes; it is exactly + zero. Any cost model that spends on ordering there is spending on nothing. + +2. **The control signal is DEAD WORDS, not selectivity.** The selective and + the clustered regimes have almost identical survivor counts — 36 and 31 — + and differ by **19 percentage points of achievable skip** *as written*, + because one conjunct's survivors are contiguous and the other's are + scattered. Rank by selectivity and those two look the same; rank by dead + words and they do not. This is also why V3 has the lever at all: an address + prefix selects a contiguous subtree (`Filter::prefix_u64`, R5/R8), which is + the clustered row. + +3. **DuckDB's adjacent-transposition hill-climb must NOT be ported**, and the + row above already anticipated the reason (*"its swap-likeliness decay is not + obviously the right control law"*). The measurement says why: the optimised + quantity is a **step function of clustering**, not a smooth function of + selectivity, so a local search over adjacent swaps is exploring the wrong + landscape. In the clustered regime the as-written order is already at 99.90 % + and the worst is 0.00 % — a hill-climb starting from the worst has no + adjacent swap that improves anything until it happens to move the prefix + term to the front. + +**What was built instead:** `Filter::and_by_skip(impl IntoIterator)` — the caller supplies a measured skip score per conjunct from a +previous execution and the builder orders the `AND` by it, descending, stably. +No measurement, no decay, no intervals, no `observe=10 / execute=20 / warmup=5`. +The crate builds programs and never evaluates one, so it cannot measure +anything; putting the score at the boundary is the whole adaptation. + +**One override, because it is a correctness requirement rather than a +preference:** when the `AND` also carries a resident plane the gate walk DROPS +as implied, a child whose result is a subset of that plane is rotated to the +front regardless of score (`hoist_gate_subset`). The ordering applies among the +children that rotation leaves alone. + +Board: `STATUS_BOARD.md` D-QCK-9; the probe is committed, not a one-off. + +--- + ## §9 — Claims in this document that are NOT verified in code Collected so they are not quoted as findings. Everything else in §2–§5 carries a diff --git a/crates/lance-graph-quack/examples/adaptive_order_probe.rs b/crates/lance-graph-quack/examples/adaptive_order_probe.rs new file mode 100644 index 000000000..93f43fca7 --- /dev/null +++ b/crates/lance-graph-quack/examples/adaptive_order_probe.rs @@ -0,0 +1,400 @@ +//! A1 — does conjunction term ORDER change how much work the survivor skip +//! avoids? The measurement the translation matrix demands before any port. +//! +//! # Why this probe exists, and why it comes before the code +//! +//! `duckdb-to-v3-translation-matrix-v1.md` row A1 is the one place DuckDB has +//! something V3 does not: `AdaptiveFilter` permutes a conjunction's terms at +//! runtime by measured selectivity, via an adjacent-transposition hill-climb +//! with a swap-likeliness decay. The row's falsifier is unusually blunt about +//! what to do with that: +//! +//! > **The falsifier is prior to the port, and it may kill the whole idea.** +//! > DuckDB's reordering pays because term k runs only on survivors of +//! > 1..k−1, so a selective term first *shrinks the input*. In V3 a predicate +//! > sweep costs the **full column** regardless of position — so **reordering +//! > saves nothing on generation**. It can only save by *avoidance*. +//! > **Measure:** what fraction of `Pred` ops are skippable by `under`, and +//! > does term order change that fraction? If order does not move the skip +//! > fraction, A1 is ELIMINATE and the machinery must not be ported. +//! +//! So this prints a number and draws no conclusion the number does not force. +//! +//! # What is counted, and what that models +//! +//! `MaskOp::Pred { under }` skips at 64-row WORD granularity: a word where the +//! gate has no survivor is not evaluated. Under a conjunction lowered in place, +//! term `i + 1` is gated on the accumulated result of terms `1..i`, so the +//! skippable fraction for term `i + 1` is the fraction of gate words that are +//! entirely zero. +//! +//! That is an exact model of WHICH words are skipped — the same quantity the +//! facade's `*_to_mask_under` decides on — and it is deliberately not a timing. +//! A wall-clock number here would be dominated by the fixture's cache +//! behaviour and would not answer the row's question, which is about skip +//! OPPORTUNITIES. The honest reading of the output is "how much work becomes +//! avoidable", not "how much time is saved". +//! +//! Run: `cargo run -p lance-graph-quack --example adaptive_order_probe --release` + +use lance_graph_mask_risc::{MaskOp, Operand}; +use lance_graph_quack::{Agg, Cmp, Col, Filter, Query}; + +const VALS: Col = Col(0); +const CLASS: Col = Col(1); +const ADDR: Col = Col(2); + +const N: usize = 1 << 16; +const WORDS: usize = N / 64; + +/// One conjunct: a label, the filter, and the rows it selects. +struct Term { + label: &'static str, + filter: Filter, + mask: Vec, + selectivity: f64, +} + +fn lcg(seed: &mut u64) -> u64 { + *seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *seed >> 11 +} + +/// Words of `mask` that are entirely zero — exactly the words a `Pred` gated +/// on this mask does not evaluate. +fn dead_words(mask: &[u64]) -> usize { + mask.iter().filter(|w| **w == 0).count() +} + +fn and_into(acc: &mut [u64], other: &[u64]) { + for (a, b) in acc.iter_mut().zip(other) { + *a &= *b; + } +} + +fn popcount(mask: &[u64]) -> u32 { + mask.iter().map(|w| w.count_ones()).sum() +} + +/// Total words SKIPPED across a conjunction evaluated in this order. +/// +/// Term 0 is ungated and skips nothing — it is the seed. Term `i` for `i > 0` +/// is gated on the accumulation of `0..i`, and skips that accumulation's dead +/// words. +fn skipped_words(terms: &[&Term]) -> (usize, u32) { + let mut acc = vec![u64::MAX; WORDS]; + and_into(&mut acc, &terms[0].mask); + let mut skipped = 0usize; + for t in &terms[1..] { + skipped += dead_words(&acc); + and_into(&mut acc, &t.mask); + } + (skipped, popcount(&acc)) +} + +fn permutations<'a>(items: &[&'a Term]) -> Vec> { + if items.len() <= 1 { + return vec![items.to_vec()]; + } + let mut out = Vec::new(); + for i in 0..items.len() { + let mut rest = items.to_vec(); + let head = rest.remove(i); + for mut p in permutations(&rest) { + p.insert(0, head); + out.push(p); + } + } + out +} + +/// One regime: a set of conjuncts, and the spread its orderings produce. +struct Scenario { + name: &'static str, + terms: Vec, +} + +fn main() { + let mut seed = 0x51ED_C0DEu64; + let vals: Vec = (0..N) + .map(|_| (lcg(&mut seed) % 2000) as i32 - 1000) + .collect(); + let classes: Vec = (0..N).map(|_| (lcg(&mut seed) % 64) as u32).collect(); + // An ADDRESS-ORDERED lane, the shape the V3 substrate actually has. + let addr: Vec = (0..N).map(|i| (i as u64) << 8).collect(); + + let build = |label: &'static str, filter: Filter, keep: &dyn Fn(usize) -> bool| { + let mut mask = vec![0u64; WORDS]; + let mut live = 0usize; + for r in 0..N { + if keep(r) { + mask[r / 64] |= 1u64 << (r % 64); + live += 1; + } + } + Term { + label, + filter, + mask, + selectivity: live as f64 / N as f64, + } + }; + + // THREE regimes, not one. A single fixture would answer the row's question + // only for that fixture, and the two ends behave differently for a reason + // that matters: a conjunction whose accumulator collapses to nearly empty + // makes almost every later word skippable regardless of order, while a + // permissive one leaves little to skip at all. The interesting number is + // what order buys in between. + // + // The first draft of this probe had ONE regime and it was degenerate: + // `v < 500` and `v > 900` are disjoint, so the conjunction selected ZERO + // rows and the "best" ordering was just measuring how fast the accumulator + // died. Every scenario below is asserted to select a proper subset. + let scenarios = vec![ + Scenario { + name: "selective (a long tail of survivors)", + terms: vec![ + build( + "v > -900 (0.95)", + Filter::cmp(VALS, Cmp::GtI32(-900)), + &|r| vals[r] > -900, + ), + build( + "v < 500 (0.75)", + Filter::cmp(VALS, Cmp::LtI32(500)), + &|r| vals[r] < 500, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "v < -850 (selective)", + Filter::cmp(VALS, Cmp::LtI32(-850)), + &|r| vals[r] < -850, + ), + build( + "class == 7 (very selective)", + Filter::cmp(CLASS, Cmp::EqU32(7)), + &|r| classes[r] == 7, + ), + ], + }, + Scenario { + name: "moderate", + terms: vec![ + build( + "v > -900 (0.95)", + Filter::cmp(VALS, Cmp::GtI32(-900)), + &|r| vals[r] > -900, + ), + build( + "v < 500 (0.75)", + Filter::cmp(VALS, Cmp::LtI32(500)), + &|r| vals[r] < 500, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "v > -400 (0.70)", + Filter::cmp(VALS, Cmp::GtI32(-400)), + &|r| vals[r] > -400, + ), + build( + "class < 32 (0.50)", + Filter::cmp( + CLASS, + Cmp::MatchU32 { + pattern: 0, + care: 32, + }, + ), + &|r| classes[r] & 32 == 0, + ), + ], + }, + Scenario { + name: "permissive (little to skip at all)", + terms: vec![ + build( + "v > -990 (0.995)", + Filter::cmp(VALS, Cmp::GtI32(-990)), + &|r| vals[r] > -990, + ), + build( + "v < 990 (0.995)", + Filter::cmp(VALS, Cmp::LtI32(990)), + &|r| vals[r] < 990, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "class != 1 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(1)), + &|r| classes[r] != 1, + ), + build( + "class != 2 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(2)), + &|r| classes[r] != 2, + ), + ], + }, + Scenario { + name: "clustered (one conjunct is an ADDRESS PREFIX)", + terms: vec![ + build( + "addr prefix /50 (contiguous)", + Filter::prefix_u64(ADDR, addr[N / 4], 50), + &|r| { + let care = u64::MAX << (64 - 50); + (addr[r] ^ addr[N / 4]) & care == 0 + }, + ), + build( + "v > -900 (0.95)", + Filter::cmp(VALS, Cmp::GtI32(-900)), + &|r| vals[r] > -900, + ), + build( + "class != 0 (0.98)", + Filter::cmp(CLASS, Cmp::NeU32(0)), + &|r| classes[r] != 0, + ), + build( + "v < 500 (0.75)", + Filter::cmp(VALS, Cmp::LtI32(500)), + &|r| vals[r] < 500, + ), + build( + "v > -400 (0.70)", + Filter::cmp(VALS, Cmp::GtI32(-400)), + &|r| vals[r] > -400, + ), + ], + }, + ]; + + println!("N = {N} rows, {WORDS} words per mask, 5 conjuncts per scenario"); + println!( + "\nskipped = 64-row words a gated `Pred` does not evaluate, summed over the\n {} gated positions of one ordering (term 0 is the ungated seed).\n", + 4 * WORDS + ); + + for sc in &scenarios { + let refs: Vec<&Term> = sc.terms.iter().collect(); + let gated_positions = (sc.terms.len() - 1) * WORDS; + + // The model is tied to the SHIPPED lowering rather than assumed: the + // conjunction is really lowered, and every term after the first must + // come back gated on a scratch slot. If that stops being true the + // probe is measuring something the executor does not do. + let program = lance_graph_quack::lower(&Query { + filter: Filter::and(refs.iter().map(|t| t.filter.clone())), + agg: Agg::Count, + }) + .expect("lowers"); + let accumulator_gated = program + .ops + .iter() + .filter(|op| { + matches!( + op, + MaskOp::Pred { + under: Some(Operand::Scratch(_)), + .. + } + ) + }) + .count(); + assert_eq!( + accumulator_gated, + sc.terms.len() - 1, + "{}: the lowering gated {accumulator_gated} of {} later terms on the \ + accumulator; this probe measures a skip the executor would not perform", + sc.name, + sc.terms.len() - 1 + ); + + let perms = permutations(&refs); + let mut results: Vec<(usize, u32, Vec<&'static str>)> = perms + .iter() + .map(|p| { + let (skipped, count) = skipped_words(p); + (skipped, count, p.iter().map(|t| t.label).collect()) + }) + .collect(); + + let answer = results[0].1; + assert!( + results.iter().all(|r| r.1 == answer), + "{}: orderings disagree on the count; the probe is measuring a defect", + sc.name + ); + assert!( + answer > 0 && (answer as usize) < N, + "{}: selects {answer}/{N} — a degenerate conjunction proves nothing \ + about order", + sc.name + ); + + results.sort_by_key(|r| r.0); + let worst = &results[0]; + let best = results.last().expect("non-empty"); + let (written, _) = skipped_words(&refs); + let pct = |s: usize| 100.0 * s as f64 / gated_positions as f64; + + println!( + "=== {} — {answer} survivors ({:.3}%)", + sc.name, + 100.0 * answer as f64 / N as f64 + ); + for t in &sc.terms { + println!(" {:<30} sel {:.4}", t.label, t.selectivity); + } + println!( + " {:<12} {:>9} {:>9}", + "ordering", "skipped", "of gated" + ); + println!( + " {:<12} {:>9} {:>8.2}%", + "as written", + written, + pct(written) + ); + println!( + " {:<12} {:>9} {:>8.2}%", + "worst", + worst.0, + pct(worst.0) + ); + println!(" {:<12} {:>9} {:>8.2}%", "best", best.0, pct(best.0)); + println!( + " spread {:.2} percentage points, best/worst {:.2}x", + pct(best.0) - pct(worst.0), + if worst.0 == 0 { + f64::INFINITY + } else { + best.0 as f64 / worst.0 as f64 + } + ); + println!(" best order: {}", best.2.join(" < ")); + println!(); + } + + println!( + "A1's falsifier: if term order does not move the skipped fraction, the row is\n\ + ELIMINATE and DuckDB's hill-climb must not be ported. These are counts of\n\ + AVOIDABLE word-evaluations, not timings." + ); +} diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index 7f0b236d7..95af8760e 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -126,6 +126,8 @@ #![forbid(unsafe_code)] +use std::cmp::Reverse; + use lance_graph_mask_risc::{ fuse, BoolExpr, FuseError, MaskOp, Operand, Pred, Program, Terminal, MAX_SCRATCH_SLOTS, }; @@ -263,6 +265,79 @@ impl Filter { ) } + /// An `AND` whose children are ordered by how much of the gate each one + /// KILLS — the V3 answer to DuckDB's `AdaptiveFilter`, and deliberately + /// not DuckDB's algorithm. + /// + /// `parts` is `(skip_score, child)`; children sort by DESCENDING score, + /// stably, so equal scores keep the caller's order. + /// + /// # Why the score is dead WORDS and not selectivity + /// + /// DuckDB ranks conjunction terms by measured selectivity, because there + /// term `k` runs only on the survivors of `1..k-1` — a selective term + /// first literally shrinks the input. In V3 a predicate sweep costs the + /// full column wherever it sits, so ordering can only pay by AVOIDANCE: + /// the survivor skip drops a 64-row WORD when the gate has no survivor in + /// it. The matrix (row A1) said so and required the measurement before any + /// port. `examples/adaptive_order_probe.rs` is that measurement, over + /// 65,536 rows, five conjuncts, all 120 orderings, four regimes: + /// + /// | regime | survivors | skipped, worst → best order | + /// |---|---|---| + /// | permissive | 94.3 % | **0.00 % → 0.00 %** | + /// | moderate | 21.8 % | **0.00 % → 0.00 %** | + /// | selective | 0.055 % | 5.66 % → **80.66 %** (14.2×) | + /// | clustered (an address prefix) | 0.047 % | 0.00 % → **99.90 %** | + /// + /// Two findings, and the second is the one that picks the control law. + /// + /// **Order does move the skip fraction, so A1 is not ELIMINATE** — but + /// only where there is anything to skip. At 21.8 % survival, uniformly + /// spread, a 64-row word is all-dead with probability `0.782^64 ≈ 2e-7`, + /// so NO ordering skips anything and the whole question is moot. Word + /// granularity needs the accumulator to die in whole words, not merely to + /// be small. + /// + /// **Which means selectivity is the wrong signal.** The selective and the + /// clustered regimes have almost identical survivor counts — 36 and 31 + /// rows — and differ by 19 percentage points of skip, because one + /// conjunct's survivors are contiguous and the other's are scattered. Rank + /// by selectivity and those two look the same. Rank by DEAD WORDS and they + /// do not. That is also why V3 has this lever at all: an address prefix + /// selects a contiguous subtree ([`Filter::prefix_u64`]), which is the + /// clustered row of that table. + /// + /// # Why the score is the CALLER's, and why there is no hill-climb + /// + /// This crate builds programs and never evaluates one, so it cannot + /// measure anything; the score comes from a previous execution the caller + /// ran. DuckDB's adjacent-transposition hill-climb with swap-likeliness + /// decay is NOT ported — the matrix anticipated that too (*"its + /// swap-likeliness decay is not obviously the right control law"*), and + /// the measurement says why: the quantity being optimised is a step + /// function of clustering, not a smooth function of selectivity, so a + /// local search over adjacent swaps is exploring the wrong landscape. + /// + /// # One caveat, because it is a real override + /// + /// When this `AND` also carries a resident plane that the gate walk + /// DROPS as implied, a child whose result is a subset of that plane is + /// rotated to the front regardless of score. That rotation is a + /// correctness requirement, not a preference, so it wins. The ordering + /// here applies among the children the rotation leaves alone. + pub fn and_by_skip(parts: impl IntoIterator) -> Self { + let mut scored: Vec<(u32, Filter)> = parts.into_iter().collect(); + // `sort_by_key` is stable, so equal scores keep the caller's order + // rather than being permuted by an implementation detail. `Reverse` + // rather than a reversed comparator: a stable sort over a reversed KEY + // keeps ties in caller order, while reversing the COMPARISON of a + // stable sort would too — but clippy rejects the latter spelling, and + // the two are only equivalent because the key is `Copy`. + scored.sort_by_key(|&(score, _)| Reverse(score)); + Filter::And(scored.into_iter().map(|(_, f)| f).collect()) + } + /// Rows whose 32-bit address lane starts with the top `bits` of `prefix` — /// the operator SQL has no name for and the V3 address was built around. /// @@ -541,6 +616,31 @@ enum Node { Not(Box), } +/// Rotate a child whose result is a SUBSET of the gate to the front, and say +/// whether one was found. +/// +/// `flags[i]` is `gate_walk`'s "vanishes with the gate", which is exactly +/// "this child's result is a subset of the gate". Putting such a child first +/// makes the running accumulator a subset of the gate from the first fold +/// onward, and that is the precondition [`emit_gated`] needs before it may +/// narrow later comparisons onto the accumulator instead of onto the gate. +/// +/// Shared by BOTH `AND` arms of [`gate_walk`] — the already-gated one and the +/// one that establishes a gate — because a first version rotated only in the +/// second, and a gated `AND` nested inside a gated `AND` reproduced the same +/// wrong answer one level down. Two spellings of one rule is how that happens +/// twice. +fn hoist_gate_subset(nodes: &mut [Node], flags: &mut [bool]) -> bool { + match flags.iter().position(|&v| v) { + Some(i) => { + nodes.swap(0, i); + flags.swap(0, i); + true + } + None => false, + } +} + /// Walk `f` under `gate`, establishing at most one gate per `AND` path. /// /// Returns the node and whether it is identically zero wherever the gate is @@ -568,8 +668,9 @@ fn gate_walk(f: &Filter, gate: Option) -> Result<(Node, bool), LowerError> if let Some(g) = gate { // Already gated from above: this AND's own planes are plain // leaves; one gate per comparison is all `under` can carry. - let (nodes, flags) = walk_all(parts, Some(g))?; - (Node::And(nodes), flags.iter().any(|&v| v)) + let (mut nodes, mut flags) = walk_all(parts, Some(g))?; + let subset = hoist_gate_subset(&mut nodes, &mut flags); + (Node::And(nodes), subset) } else { let found = parts.iter().find_map(|p| match p { Filter::Plane(m) => Some(*m), @@ -580,7 +681,7 @@ fn gate_walk(f: &Filter, gate: Option) -> Result<(Node, bool), LowerError> return Ok((Node::And(nodes), false)); }; let mut nodes = Vec::with_capacity(parts.len()); - let mut any_vanishes = false; + let mut flags = Vec::with_capacity(parts.len()); let mut gate_taken = false; for p in parts { if !gate_taken && matches!(p, Filter::Plane(m) if *m == g) { @@ -588,12 +689,34 @@ fn gate_walk(f: &Filter, gate: Option) -> Result<(Node, bool), LowerError> continue; } let (n, v) = gate_walk(p, Some(g))?; - any_vanishes |= v; nodes.push(n); + flags.push(v); } // The gate is implied by any child that vanishes with it; // otherwise it must be read as a leaf of its own. - if !any_vanishes { + // + // When it IS implied, the vanishing child is rotated to the + // FRONT, and that is load-bearing rather than tidy. `vanishes` + // means exactly "this child's result is a subset of the gate", + // so putting one first makes the running accumulator a subset + // of the gate from the first fold onward — which is what lets + // [`emit_gated`] narrow later comparisons onto the accumulator + // without losing the gate. + // + // Without the rotation this is a silent WRONG ANSWER, not a + // missed optimisation, and it was measured: on + // `alpha AND focus AND v < 50`, the foreign plane `focus` sat + // first, the accumulator therefore started as `focus` (which is + // NOT a subset of `alpha`), the comparison was gated on that + // instead of on `alpha`, and `alpha` — already dropped as + // "implied" — was nowhere in the program. The per-row oracle + // caught it immediately. + // + // AND is commutative, so the rotation costs nothing + // semantically. It is also the first place in this crate where + // term ORDER changes the emitted program, which is the + // mechanism row A1 is about. + if !hoist_gate_subset(&mut nodes, &mut flags) { nodes.insert(0, Node::Plane(g)); } (Node::And(nodes), false) @@ -637,18 +760,55 @@ fn walk_all(parts: &[Filter], gate: Option) -> Result<(Vec, Vec) -> Result { + emit_gated(n, dst, None, ops) +} + +/// [`emit_inplace`] with a RUNNING gate: an operand every comparison beneath +/// `n` is evaluated `under`. +/// +/// # The accumulator gate, and why it is the one that makes order matter +/// +/// The plane gate ([`gate_walk`]) skips words where a RESIDENT mask is empty. +/// This is the other one: inside an `AND`, once the first `k` conjuncts have +/// been folded into `dst`, conjunct `k + 1` is only consulted where that +/// partial result still has a survivor. So the gate NARROWS as the conjunction +/// proceeds, and a selective term early shrinks the live-word count of every +/// term after it. +/// +/// Soundness is the same identity the plane gate uses — `g ∧ rest(X) = +/// g ∧ rest(g ∧ X)` — so it passes through `OR` and `NOT` alike. What it may +/// NOT do is let anything be dropped: the accumulator is a real operand of the +/// `AND`, never elided, so there is no analog here of the plane's +/// vanishes-with-the-gate rule. +/// +/// It applies to `AND` only. Under an `OR`, `acc | p` depends on `p` exactly +/// where `acc` is ZERO — gating there would discard the bits that matter and +/// quietly answer `acc`. That asymmetry is the same one `lgj-abi`'s +/// `plan_lower` documents, and it is the correctness question in both. +fn emit_gated( + n: &Node, + dst: u16, + acc_gate: Option, + ops: &mut Vec, +) -> Result { match n { Node::Pred { pred, under } => { + // The accumulator gate wins over the plane gate when both are + // available, and it is strictly narrower rather than merely + // different: the plane was the AND's first conjunct, so it is + // already folded into the accumulator. A `Pred` carries exactly + // one `under`, which is why this is a choice and not a union. + let gate = acc_gate.or_else(|| under.map(|m| Operand::Plane(m.0))); ops.push(MaskOp::Pred { pred: *pred, - under: under.map(|m| Operand::Plane(m.0)), + under: gate, dst, }); Ok(Operand::Scratch(dst)) } Node::Plane(m) => Ok(Operand::Plane(m.0)), Node::Not(inner) => { - let a = emit_inplace(inner, dst, ops)?; + let a = emit_gated(inner, dst, acc_gate, ops)?; ops.push(MaskOp::Not { a, dst }); Ok(Operand::Scratch(dst)) } @@ -666,7 +826,16 @@ fn emit_inplace(n: &Node, dst: u16, ops: &mut Vec) -> Result Some(a), + _ => acc_gate, + }; + let b = emit_gated(part, slot, child_gate, ops)?; acc = Some(match acc { None => b, Some(a) => { @@ -978,40 +1147,6 @@ mod tests { ]) } - /// Every `Pred` op of `p` is gated under `g`, and `g` is never read as a - /// Boolean operand — the shape the survivor skip produces when the plane - /// leaf is dropped. - fn gated_and_dropped(p: &Program, g: Mask) -> (bool, bool) { - let gate = Operand::Plane(g.0); - let mut all_gated = true; - let mut read_as_leaf = false; - for op in &p.ops { - match *op { - MaskOp::Pred { under, .. } => all_gated &= under == Some(gate), - MaskOp::And { a, b, .. } - | MaskOp::Or { a, b, .. } - | MaskOp::Xor { a, b, .. } - | MaskOp::AndNot { a, b, .. } => read_as_leaf |= a == gate || b == gate, - MaskOp::Not { a, .. } => read_as_leaf |= a == gate, - MaskOp::Ternlog { a, b, c, .. } => { - read_as_leaf |= a == gate || b == gate || c == gate; - } - } - } - let terminal_reads_gate = matches!( - p.terminal, - Terminal::Count { mask } - | Terminal::Any { mask } - | Terminal::All { mask } - | Terminal::MaskedSumI32 { mask, .. } - | Terminal::MaskedMinI32 { mask, .. } - | Terminal::MaskedMaxI32 { mask, .. } - | Terminal::BlendI32 { mask, .. } - | Terminal::Keep { mask } if mask == gate - ); - (all_gated, !read_as_leaf && !terminal_reads_gate) - } - /// FAILS IF: either lowering and an independent per-row reading of the /// same filter disagree. /// @@ -1160,31 +1295,97 @@ mod tests { } } - /// FAILS IF: the survivor skip gates the wrong leaves or drops the gate - /// where the drop is unsound. + /// How `p` gates its comparisons, and whether `g` survives as an operand. + /// + /// Returns `(every comparison gated, some comparison gated on the + /// ACCUMULATOR, g is never read as a Boolean operand)`. /// - /// Can-fire: under `alpha & ((A & B) | C)` every comparison is gated and - /// alpha is never read as an operand. Can-stay-silent: under - /// `alpha & !X` the comparison is still gated (the rewrite is sound - /// through a negation while the gate is a leaf) but alpha IS read as a - /// leaf, because `!(alpha & X)` is 1 exactly where alpha is 0. The - /// differential above holds either way; this pins the SHAPE, so a walk - /// that dropped the gate under a negation would fail here even before - /// the oracle caught its wrong count. + /// The middle field did not exist before accumulator gating: a gate that is + /// a scratch slot is the partial result of the conjunction so far, so it + /// NARROWS as the conjunction proceeds, where a plane gate is fixed. That + /// narrowing is the mechanism row A1 is about. + fn gate_shape(p: &Program, g: Mask) -> (bool, bool, bool) { + let gate = Operand::Plane(g.0); + let mut all_gated = true; + let mut on_accumulator = false; + let mut read_as_leaf = false; + for op in &p.ops { + match *op { + MaskOp::Pred { under, .. } => match under { + Some(Operand::Scratch(_)) => on_accumulator = true, + Some(_) => {} + None => all_gated = false, + }, + MaskOp::And { a, b, .. } + | MaskOp::Or { a, b, .. } + | MaskOp::Xor { a, b, .. } + | MaskOp::AndNot { a, b, .. } => read_as_leaf |= a == gate || b == gate, + MaskOp::Not { a, .. } => read_as_leaf |= a == gate, + MaskOp::Ternlog { a, b, c, .. } => { + read_as_leaf |= a == gate || b == gate || c == gate; + } + } + } + let terminal_reads_gate = matches!( + p.terminal, + Terminal::Count { mask } + | Terminal::Any { mask } + | Terminal::All { mask } + | Terminal::MaskedSumI32 { mask, .. } + | Terminal::MaskedMinI32 { mask, .. } + | Terminal::MaskedMaxI32 { mask, .. } + | Terminal::BlendI32 { mask, .. } + | Terminal::Keep { mask } if mask == gate + ); + ( + all_gated, + on_accumulator, + !read_as_leaf && !terminal_reads_gate, + ) + } + + /// FAILS IF: the survivor skip gates the wrong leaves, drops the gate where + /// the drop is unsound, or stops narrowing onto the accumulator. + /// + /// - **can-fire.** Under `alpha & ((A & B) | C)` every comparison is gated + /// and alpha is never read as a Boolean operand; under the in-place + /// lowering at least one comparison is gated on the ACCUMULATOR. + /// - **can-stay-silent.** Under `alpha & !X` the comparison is still gated, + /// but alpha IS read as a leaf — `!(alpha & X)` is 1 exactly where alpha + /// is 0, so dropping it would be unsound. + /// - **the foreign plane.** A second plane beside the gate stays an + /// ordinary leaf; it does not become a second gate. + /// + /// The differential holds either way; this pins the SHAPE, so a walk that + /// dropped the gate under a negation — or one that quietly stopped gating + /// on the accumulator, which no count would ever notice — fails here. #[test] fn the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes() { - for lowering in [lower, lower_fused] { + for (is_inplace, lowering) in [(true, lower as fn(&Query) -> _), (false, lower_fused)] { let slice = lowering(&Query { filter: slice_filter(), agg: Agg::Count, }) .expect("lowers"); - assert_eq!( - gated_and_dropped(&slice, ALPHA), - (true, true), + let (gated, on_acc, dropped) = gate_shape(&slice, ALPHA); + assert!( + gated && dropped, "the slice: every predicate gated, alpha implied: {:?}", slice.ops ); + // The accumulator gate is asserted for the IN-PLACE lowering only. + // The fused one gives every comparison its own slot and hands the + // skeleton to the fuser, so there is no running partial result for + // a later comparison to narrow onto. That is a real property of + // that lowering, not an omission, and asserting it for both would + // be asserting something false. + if is_inplace { + assert!( + on_acc, + "no comparison narrowed onto the accumulator: {:?}", + slice.ops + ); + } let negated = lowering(&Query { filter: Filter::and([ @@ -1194,9 +1395,9 @@ mod tests { agg: Agg::Count, }) .expect("lowers"); - assert_eq!( - gated_and_dropped(&negated, ALPHA), - (true, false), + let (gated, _, dropped) = gate_shape(&negated, ALPHA); + assert!( + gated && !dropped, "under a negation: still gated, but alpha stays a leaf: {:?}", negated.ops ); @@ -1210,10 +1411,10 @@ mod tests { agg: Agg::Count, }) .expect("lowers"); - let (gated, dropped) = gated_and_dropped(&foreign, ALPHA); + let (gated, _, dropped) = gate_shape(&foreign, ALPHA); assert!(gated && dropped, "the first plane gates: {:?}", foreign.ops); assert!( - !gated_and_dropped(&foreign, FOCUS).1, + !gate_shape(&foreign, FOCUS).2, "the foreign plane is read as a leaf, not gated away: {:?}", foreign.ops ); @@ -1673,6 +1874,90 @@ mod tests { assert_eq!(fx.count(&mixed), expected); } + /// FAILS IF: skip-ordering changes the ANSWER, or does not change the + /// ORDER, or stops being stable for equal scores. + /// + /// The safety property is the one that matters most: `AND` is commutative, + /// so reordering may move work and must never move the result. A lowering + /// that reordered children but got the gate chain wrong would produce a + /// different count, and the per-row oracle would catch it — which is why + /// the answer is checked against the oracle and not merely against the + /// unordered lowering. + /// + /// Then the can-fire half: the emitted program must actually differ. A + /// builder that sorted and then lost the order somewhere in `gate_walk` + /// would pass the safety half perfectly. + #[test] + fn skip_ordering_moves_the_work_and_never_the_answer() { + let fx = Fx::new(N); + // Written worst-first on purpose: the permissive term leads, which is + // the shape DuckDB's AdaptiveFilter exists to fix. + let permissive = Filter::cmp(VALS, Cmp::GtI32(-900)); + let moderate = Filter::cmp(CLASS, Cmp::NeU32(0)); + let selective = Filter::cmp(CLASS, Cmp::EqU32(3)); + + let written = Filter::and([permissive.clone(), moderate.clone(), selective.clone()]); + // Scores are the caller's evidence — here, dead words each term would + // leave behind, highest first. + let ordered = Filter::and_by_skip([ + (1, permissive.clone()), + (7, moderate.clone()), + (900, selective.clone()), + ]); + + // Same answer, and the oracle — not the other lowering — is the judge. + let expected = fx.rows(&written).len(); + assert!( + expected > 0 && expected < N, + "{expected}/{N} — a degenerate conjunction proves nothing" + ); + assert_eq!(fx.rows(&ordered).len(), expected); + assert_eq!(fx.count(&written), expected); + assert_eq!(fx.count(&ordered), expected); + + // The order really moved: the most selective term now leads. + assert_eq!( + ordered, + Filter::and([selective.clone(), moderate.clone(), permissive.clone()]), + "and_by_skip must sort descending by score" + ); + assert_ne!(ordered, written, "the fixture must not already be ordered"); + + // ...and the emitted program reflects it: the FIRST predicate is the + // one that was scored highest. Without this the sort could be undone + // between the builder and the lowering and nothing would notice. + let program = lower(&Query { + filter: ordered.clone(), + agg: Agg::Count, + }) + .expect("lowers"); + assert_eq!( + program.ops.first(), + Some(&MaskOp::Pred { + pred: Pred::EqU32 { + lane: CLASS.0, + v: 3 + }, + under: None, + dst: 0, + }), + "the highest-scored term must be the ungated seed: {:?}", + program.ops + ); + + // Stability: equal scores keep the caller's order, so a scorer with no + // information cannot silently permute a hand-tuned conjunction. + assert_eq!( + Filter::and_by_skip([ + (5, permissive.clone()), + (5, moderate.clone()), + (5, selective.clone()), + ]), + written, + "equal scores must be stable" + ); + } + /// FAILS IF: any arm of the vertical slice disagrees on a 64k slab. /// /// `COUNT(alpha & ((A & B) | C))` over 65,536 rows through the per-row From c223417ae558af64faf04e568869d637d2577759 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 23:45:29 +0000 Subject: [PATCH 10/25] board: the convergence verdict, and the floor that hid a dead fixture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the answer to the operator's convergence hypothesis and the finding the work produced. **The verdict (D-QCK-10, LATEST_STATE 2026-09-14 (6)): shared LAW, not a shared DEPENDENCY.** `lgj-abi`'s `plan_lower` and `quack::lower` are now pinned equal by a differential in lance-graph-java (`8ad1a1b` + `e9bf3aa`), with quack wired there as a DEV-dependency so delegation stays impossible by construction — the membrane must not depend on a consumer of the IR it serves. The differential also made the duplication legible, and it is smaller than it looked. The accumulator gate and the AND/OR asymmetry are facts about `MaskOp::Pred { under }`, i.e. IR facts, which is why both lowerings must carry them and why any future shared helper belongs in `mask-risc` (which already hosts `fuse`/`BoolExpr`). The prefix rewrite is NOT an IR fact — it is a property of the all-ones-seeded fold, and in tree form it is not a rule at all (`all_ones | p == all_ones`, so the op never becomes a node). A shared helper would have had to carry it as a special case for one caller. Not enough duplication to justify a helper today; pinned instead, and the pin is what reports if that stops holding. **The finding (E-A-FLOOR-PASSED-AT-ITS-BOUND-IS-A-DEAD-FIXTURE-1).** Both anti-vacuity bounds in the new file arrived as floors and both passed at exactly their bound. That is not a pass — it is the fixture reporting that it is inert. Measured: `LT_I32(500)` against a `-150..=361` lane made a whole 16-vector arm contribute nothing (15 -> 21 of 28), and made a real `LE_I32 -> LtI32` mis-map INVISIBLE (red at operand 300, green at 500). Three rules extracted: a floor is the wrong shape for an anti-vacuity bound, because inert is exactly where it still passes; reason operands from the MEASURED lane, never the documented domain; and in an arm-vs-arm differential, two quantities selecting the same row count hide a swap between exactly those two. Supersession index regenerated after the board writes, per CLAUDE.md. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/EPIPHANIES.md | 46 +++++++++++++++++++++++++ .claude/board/LATEST_STATE.md | 64 +++++++++++++++++++++++++++++++++++ .claude/board/STATUS_BOARD.md | 1 + 3 files changed, 111 insertions(+) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index f8862f052..6e26c5207 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1426,6 +1426,52 @@ the tile, seriation, LUT and spread contribute nothing that survives permutation **Scope.** Pure-Python lab, no Rust fingerprints — a shape proxy, never a measurement of `ndarray`/`bgz17`/`helix`. The cue reached **87.5 %** of the BPE incumbent (0.1579 vs 0.1805): PROCEED, not PASS. +## E-A-FLOOR-PASSED-AT-ITS-BOUND-IS-A-DEAD-FIXTURE-1 (2026-09-14) + +**Status:** FINDING (measured twice in one file, both arms) +**Confidence:** High — the invisible-mis-map half is a direct red/green pair + +**An anti-vacuity bound written as a FLOOR (`>= N`) that passes at exactly +`N` is not a pass. It is the fixture telling you it is dead.** + +`lowering_convergence.rs` (lance-graph-java `native/lgj-abi`) arrived with +two: `>= 15` of 28 combine vectors non-degenerate, and `>= 7` of 9 opcode +seeds selecting a proper subset. Both measured **exactly** at their bound. +Both bounds had been reasoned to, honestly, from the fixture's DOCUMENTED +domain — and the documented domain was not the measured one. + +- The values lane is `-150..=361`. The arity-4 arm appended `LT_I32(500)`, + an always-true op, so the entire 16-vector arm was eight saturated + `n`-row answers plus eight verbatim copies of the arity-3 row: **zero** + additional discriminating power, clearing `>= 15` by sitting on it. + Measuring the lane and using `LT_I32(200)` took it **15 -> 21**. +- Worse, the same operand appeared in the per-opcode arm, so `LT_I32` and + `LE_I32` both selected every row. Measured: mis-mapping `LGJ_OP_LE_I32` + to `Pred::LtI32` — one token, and exactly the defect the file exists to + catch — is **RED at operand 300** (866 vs 865, a one-row difference) and + **GREEN at operand 500**. Two of nine opcodes were untested while the + file read as covering all nine. + +**Three rules, each with its own force:** + +1. **A floor is the wrong shape for an anti-vacuity bound.** Its whole job + is to notice the fixture going inert, and a floor cannot: inert is + exactly where it still passes. Write `assert_eq!` with the measured + count and the instruction not to relax it. +2. **Reason operands from the MEASURED lane, never from the documented + domain.** "The doc says the max is 361, so 500 is safely past it" is + sound arithmetic and produces a tautology. Probe the distribution. +3. **In a differential between two ARMS (not against ground truth), two + quantities that select the same number of rows hide a swap between + exactly those two.** Give the comparisons distinct counts on purpose. + This is specific to arm-vs-arm testing and does not arise when one side + is an oracle. + +Cross-ref: the falsifiability rule in `CLAUDE.md` already names +"a tolerance/threshold parameter needs an inertness test". This is its +sibling for the FIXTURE rather than the threshold — and the sharper case, +because a dead fixture leaves every assertion above it reading as green. +Board: `STATUS_BOARD.md` D-QCK-10; `LATEST_STATE.md` 2026-09-14 (6). ## 2026-09-14 (3) — E-THE-VOCABULARY-IS-THE-RECOGNITION-ORGAN-THE-LAW-IS-THE-TRANSFER-ORGAN-1 — F-MQ8 and H5b were filed as one null; they are a division of labour, and they say which half is weak diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 8a9797d53..9ecf739b0 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -76,6 +76,70 @@ census is measured and NOT yet on the board — see the arc entry's *Un-recorded* bullet. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1233. +## 2026-09-14 (6) — the convergence, answered: pin them equal, do not delegate + +The operator's hypothesis was *"wiring duckdb through lance-graph-java might +help both sides to converge."* It paid twice, and the second payment settles +what "converge" should mean here. + +**First payment (already landed, `0bc5e8d`):** quack gained the accumulator +gate `lgj-abi`'s `plan_lower` already had. That was a capability transfer in +one direction, and it exposed a real correctness bug (`hoist_gate_subset`) +that neither crate's own tests could have found, because quack had no +accumulator to be wrong about. + +**Second payment (lance-graph-java `8ad1a1b` + `e9bf3aa`):** the two +lowerings are now pinned equal by a differential — +`native/lgj-abi/src/exports/tests/lowering_convergence.rs`, 3 tests, 5 +disables red. + +### The verdict: shared LAW, not a shared DEPENDENCY + +`lgj-abi` could retire `plan_lower` and delegate to `quack::lower`. It does +not, and `lance-graph-quack` is wired as a **dev**-dependency to make that +impossible by construction. The membrane — the `cdylib` Java's `Linker` +loads — must not depend on a CONSUMER of the IR it serves; that dependency +points the wrong way. What the two share is the law, and the differential is +what keeps "one law, two implementations" a checkable statement instead of a +comment. + +The three rules they share are not equally placed, and the differential made +that legible: + +- **the accumulator gate** and **the AND/OR asymmetry** are facts about + `MaskOp::Pred { under }` — IR facts, which is why both lowerings must have + them and why a future shared helper would belong in `mask-risc` (which + already hosts `fuse`/`BoolExpr`) rather than in either consumer; +- **the prefix rewrite** is NOT an IR fact. It is a property of the + all-ones-seeded FOLD, and in tree form it is not a rule at all: + `all_ones | p == all_ones` means an OR before the first AND never becomes + a node. `plan_lower` scans for the least AND index; the tree reading gets + the same answer for free. A shared helper would have had to carry it as a + special case for one caller. + +So the duplication is smaller than it looked: one genuinely shared pair of +IR facts, and one artifact of `lgj-abi`'s own input shape. Not enough to +justify a helper today — pinned instead, and the pin is what will tell us if +that judgement ever stops holding. + +### Two dead fixtures, and one of them made a real defect invisible + +Both anti-vacuity bounds arrived as FLOORS (`>= 15` of 28, `>= 7` of 9) and +both passed **exactly at their bound**, which is what a floor looks like +when the fixture is dead. The values lane is `-150..=361`; the arity-4 arm +appended `LT_I32(500)`, an always-true op, and the whole 16-vector arm +collapsed to eight saturated 1000s plus eight verbatim copies of the +arity-3 row. `LT_I32(200)` took it 15 → 21. + +The sharper half: `LT_I32`/`LE_I32` carried `500` in the per-opcode arm too. +Measured, mis-mapping `LGJ_OP_LE_I32` to `Pred::LtI32` in `plan_lower` — one +token, and precisely the defect class the file exists to catch — is **red at +operand 300** (866 vs 865, one row) and **green at 500**, where both +readings select all 1000 and the two arms agree on an answer neither +computed correctly. Two of nine opcodes would have shipped untested while +the file read as covering all nine. Both bounds are `assert_eq!` now, with +the incident recorded at the assertion. + ## 2026-09-14 (5) — quack A1: the accumulator gate, and the falsifier the matrix asked for Two changes in `crates/lance-graph-quack`, one of them a correctness fix the diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index c48b2e1b1..a95909233 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -29,6 +29,7 @@ evaluates one; `execute` stays the consumer's call on a scratch it owns. | D-QCK-5 | `lower_group_by` — two-phase, a mask plane where the hash table would be; K gated equalities over the kept filter | Shipped (`18c1d85`). K programs not one, because the IR names no strided operand and no group terminal — `masked_strided_group_sum` exists in the facade and cannot be reached from here yet | per-group counts and sums equal the oracle's AND sum to the ungrouped count (the partition identity); a grouped blend is refused, not answered wrongly | | D-QCK-6 | the 64k vertical slice `COUNT(alpha & ((A&B)|C))` across five arms | Shipped (`18c1d85`) — closes Phase 3+6 | oracle, both lowerings on the executor, both on the reference evaluator, one number; two-sided anti-vacuity (subset of alpha AND strictly below the ungated remainder) | | D-QCK-7 | the join — `src_mask → hop → dst_mask` | **Blocked** on mask-risc's PR5: there is no `hop` op to lower to | — | +| D-QCK-10 | the lowering convergence — `lgj-abi`'s `plan_lower` (a flat op list) and `quack::lower` (a Boolean tree) pinned equal. **A differential, deliberately NOT a delegation:** quack is a DEV-dependency of `lgj-abi`, because the membrane must not depend on a consumer of the IR it serves | Shipped (lance-graph-java `8ad1a1b` + `e9bf3aa`, `native/lgj-abi/src/exports/tests/lowering_convergence.rs`). Five disables red. Two dead fixtures found: `LT_I32(500)` on a `-150..=361` lane made the whole 16-vector arity-4 arm inert (15 → 21 of 28) and made a real `LE_I32 → LtI32` mis-map INVISIBLE (measured: red at operand 300, green at 500) | `the_two_lowerings_agree_on_every_combine_vector` (28 vectors, `assert_eq!(non_degenerate, 21)`), `the_all_rows_shortcut_is_the_same_condition_on_both_sides` (two-sided: 3 `AllRows`, 25 not), `every_opcode_maps_to_the_same_predicate_in_both_lowerings` (`assert_eq!(proper_subset, 9)` + the TCAM half-swap) | | D-QCK-8 | accumulator gating — an `AND` gates each later child under the accumulator built so far, not only under a resident plane. The same rule `lgj-abi`'s `plan_lower` already had; the two lowerings now implement one law | Shipped. Disable-verified 3×: gate-never-on-accumulator, an `OR` gating its children on its own accumulator, and `hoist_gate_subset` never rotating — each reddens | `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` via the new `gate_shape` helper (all-gated / on-accumulator / dropped, three facts not two); `every_lowered_filter_agrees_with_an_independent_per_row_reading` is what catches the DROP going unsound | | D-QCK-9 | A1 `Filter::and_by_skip` — the caller's measured skip score orders an `AND`'s conjuncts. DuckDB's adjacent-transposition hill-climb deliberately NOT ported | Shipped, with the matrix's A1 falsifier RUN (`examples/adaptive_order_probe.rs`, 65 536 rows × 5 conjuncts × all 120 permutations × 4 regimes). Order moves the skip fraction, so A1 is not ELIMINATE — but the control signal is DEAD WORDS, not selectivity | `skip_ordering_moves_the_work_and_never_the_answer`: the reordering agrees with the oracle (not with the other lowering), the emitted program's first predicate is the highest-scored, and descending/no-sort disables both redden | From d8afa4b78051b72bd49d1bd529eda8c4c4cd7cb2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 17:21:23 +0000 Subject: [PATCH 11/25] ci: quack joined the workspace with no CI line at all MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review of the rebased branch found it: `Cargo.toml` gained `crates/lance-graph-quack` as a member, and neither `rust-test.yml` nor `style.yml` gained anything. Both workflows enumerate crates by manifest path, so a new member is invisible to them — `cargo build --workspace` does not run a member's tests, which is exactly the reason `rust-test.yml`'s own comment gives for adding the mask-risc line "the day the crate joins, not the day its first test goes red". Three lines, each mirroring how `lance-graph-mask-risc` is wired: - `cargo test --manifest-path crates/lance-graph-quack/Cargo.toml` - `cargo clippy ... --all-targets -- -D warnings` (Tier A, gated while clean) - `cargo fmt ... -- --check` `--all-targets` on the clippy line is load-bearing rather than habitual: the A1 measurement lives in `examples/adaptive_order_probe.rs`, and `--lib` would leave the probe unlinted. The stake is higher than a missing line usually implies. All 13 of quack's tests are differential against a per-row oracle, so an un-run suite is an un-checked LOWERING, not merely an un-run test — and this workspace has a measured 13-day-red precedent for a suite nobody was running. Verified by running the three commands exactly as CI will: 13 passed, clippy clean, fmt clean. Also ran the other gates locally against origin/main — append-only (9 files, none shrank; the rebase resolution kept both sides of the STATUS_BOARD prepend collision), the supersession index (already current), and citation-decay ("no new citation decay since base"; the decayed anchors it lists are pre-existing on main). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .github/workflows/rust-test.yml | 7 +++++++ .github/workflows/style.yml | 8 ++++++++ 2 files changed, 15 insertions(+) diff --git a/.github/workflows/rust-test.yml b/.github/workflows/rust-test.yml index a18376208..7da61aa31 100644 --- a/.github/workflows/rust-test.yml +++ b/.github/workflows/rust-test.yml @@ -124,6 +124,13 @@ jobs: # exits non-zero when the committed region differs from its output. - name: Ternlog dispatch table is regenerated-and-diffed run: python3 crates/lance-graph-mask-risc/tools/gen_ternlog_dispatch.py --check + # lance-graph-quack: the DuckDB-shaped surface that LOWERS to mask-risc. + # Same reason as the mask-risc line above — a member's tests are not run + # by `cargo build --workspace`, so the line lands the day the crate + # joins. Its 13 tests are all differential against a per-row oracle, so + # an un-run suite is an un-checked lowering, not merely an un-run test. + - name: Run quack tests + run: cargo test --manifest-path crates/lance-graph-quack/Cargo.toml # weather-poc is workspace-EXCLUDED. A new workflow introduced by the # same pull request cannot gate that pull request because `pull_request` # workflow selection comes from the base branch. Run both surfaces here diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml index fb6fc57b7..95653d924 100644 --- a/.github/workflows/style.yml +++ b/.github/workflows/style.yml @@ -87,6 +87,12 @@ jobs: # clean so it cannot silently regress — the deepnsm rule below. - name: Clippy lance-graph-mask-risc (mandatory) run: cargo clippy --manifest-path crates/lance-graph-mask-risc/Cargo.toml --all-targets -- -D warnings + # lance-graph-quack joined clean; gated while clean, same rule. + # `--all-targets` is load-bearing here rather than habitual: the A1 + # measurement lives in `examples/adaptive_order_probe.rs`, and `--lib` + # would leave the probe unlinted. + - name: Clippy lance-graph-quack (mandatory) + run: cargo clippy --manifest-path crates/lance-graph-quack/Cargo.toml --all-targets -- -D warnings # Tier B (advisory until incrementally cleaned, non-gating): # lance-graph core has ~91 pre-existing clippy violations to be paid down # in subsequent PRs (TD-CLIPPY-LG-1). Don't auto-fix — each violation @@ -184,6 +190,8 @@ jobs: run: cargo fmt --manifest-path crates/lance-graph-planner/Cargo.toml -- --check - name: Rustfmt lance-graph-mask-risc run: cargo fmt --manifest-path crates/lance-graph-mask-risc/Cargo.toml -- --check + - name: Rustfmt lance-graph-quack + run: cargo fmt --manifest-path crates/lance-graph-quack/Cargo.toml -- --check - name: Rustfmt lance-graph-ontology (previously ungated) run: cargo fmt --manifest-path crates/lance-graph-ontology/Cargo.toml -- --check - name: Rustfmt lance-graph-rbac (previously ungated) From 439403f1aca17e439fdee304db3c2316fa9d5e67 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 17:36:12 +0000 Subject: [PATCH 12/25] review: the hill-climb argument was measured FALSE, and five other overclaims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An overclaim audit of this branch found the A1 write-up asserting as measured fact several things the probe does not measure — and one it actively refutes. Each correction below was re-verified before landing; the ⊘ notes sit at the claim, not in a changelog. ## The reversal §8a finding 3 said DuckDB's adjacent-transposition hill-climb "explores the wrong landscape", with a concrete supporting sentence: "a hill-climb starting from the worst has no adjacent swap that improves anything until it happens to move the prefix term to the front." Measured, instrumenting the probe's own `skipped_words` model with the clustered regime's prefix term at each index: prefix-term index -> skipped words: [4092, 3069, 2046, 1023, 0] adjacent deltas: [-1023, -1023, -1023, -1023] A monotone linear ramp. The prefix term's mask is one live word of 1024 and `skipped_words` charges `dead_words(acc)` once per gated position, so skip is `(4-p)*1023`. EVERY forward adjacent swap improves it by the same amount — the friendliest possible hill-climb landscape, the exact opposite of the claim. The step-like behaviour is BETWEEN regimes; the search space is WITHIN one permutation set, and the argument reasoned from the first to the second. The real reason it is not ported is narrower and survives: this crate never executes, so there is no runtime for a hill-climb to measure. ## The other five - **"The control signal is DEAD WORDS, not selectivity"** — a claim about the right SORT KEY. The probe enumerates permutations and reports min/max; it never ranks by either signal and never calls `and_by_skip`. Narrowed to what is supported: selectivity cannot tell you WHETHER reordering is worth anything (a between-regime diagnostic). - **"Inert wherever the population is not sparse, by arithmetic"** — density is not the condition, SCATTERING is, and finding 2 said so two paragraphs later; the two findings contradicted each other. The arithmetic is also Bernoulli-independence, so conditional on this fixture's LCG rather than "by arithmetic, not by implementation". Figure recomputed: 1.4e-7, not 2e-7. - **"differ by 19 percentage points ... *as written*"** — as written the two regimes differ by 94.24 points; 19.24 is best-vs-best. The italic made the sentence false against its own table. - **ADAPT was flat; it is conditional** — inert in 2 of 4 regimes, and the pre-registered falsifier's "representative predicate stream" half was never run. Both now filed in the matrix's §9, which exists for this and had not been updated. Also: the pre-registered text HEDGED ("not obviously the right control law") and §8a quoted it as a prediction. - **"every one differential against a per-row oracle"** — measured 9 of 13; 4 are structural (gate shape, pass/slot trade, width-vs-depth, the empty-junction refusal). The same overstatement was the stated JUSTIFICATION for the new CI line, so it is corrected there too. ## Two currency fixes and a provenance note - The tally prose said "ADAPT dominates (14/32)" while the table two lines above said 15 — the table was updated when A1 moved and the prose was not. - `LATEST_STATE`'s A1 table had no `worst` column and computed spread as `best - as-written`, publishing **0** for the clustered regime — reading as "order does not matter here", the inverse of that row's point. It is 0.00% -> 99.90%, the widest of the four. - The crate doc's "one thing DuckDB has that this does not" section was stale against `and_by_skip` 170 lines below it in the same file. Retitled and rewritten. Also: DuckDB adapts on measured RUNTIME seeded from a selectivity heuristic, not on measured selectivity — the matrix's own mechanism cell says so and the crate had propagated the looser wording. - `.claude/harvest/duckdb-headers/README.md` now says plainly that its counts are not re-derivable here (no TSV, no DuckDB source in-tree), that §6's complaint is therefore NOT discharged, and that 123 definitions vs 92 names is overloads. - The nars-34 audit listed "fix the 2 Berry-Esseen citations" as outstanding work that the same commit had already done; closed in place. Also recorded: the probe is run by NO gate — `cargo test` compiles an example and never executes it — so the four-row table is a measured-once observation, not a pinned one. Said so in D-QCK-9 rather than leaving it implied. Gates: 13 tests, clippy -D warnings, fmt, append-only (9 files, none shrank), supersession index current, citation-decay no new decay since base. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/audits/nars-34-substrate-audit.md | 6 +- .claude/board/LATEST_STATE.md | 79 ++++++++---- .claude/board/STATUS_BOARD.md | 2 +- .claude/harvest/duckdb-headers/README.md | 21 +++- .../duckdb-to-v3-translation-matrix-v1.md | 112 ++++++++++++++---- .github/workflows/rust-test.yml | 6 +- crates/lance-graph-quack/src/lib.rs | 110 +++++++++++------ 7 files changed, 254 insertions(+), 82 deletions(-) diff --git a/.claude/audits/nars-34-substrate-audit.md b/.claude/audits/nars-34-substrate-audit.md index 9c63d2ba9..60720bb08 100644 --- a/.claude/audits/nars-34-substrate-audit.md +++ b/.claude/audits/nars-34-substrate-audit.md @@ -117,7 +117,11 @@ actually exists. 2. **Rewrite the 9 stale `substrate` strings** to name mask-algebra compositions, or mark them `unrealized` — a string naming a retired kernel reads as a spec and is worse than an empty one. -3. **Fix the 2 Berry-Esseen citations** (1 RTE, 32 SDD) to Jirak 2016, per +3. ⊘ **CLOSED in the same commit** — both strings now name shipped Jirak + surfaces (`recipes.rs:97` `SigmaTierBands::jirak_p`, `:408` `jc` Pillar 5), + and `no_recipe_cites_classical_berry_esseen` guards against a third. The + two rows above record the PRE-fix state and are kept as the finding, not + as outstanding work. Originally: **Fix the 2 Berry-Esseen citations** (1 RTE, 32 SDD) to Jirak 2016, per `I-NOISE-FLOOR-JIRAK`. These are iron-rule violations sitting in shipped data. 4. **Re-pin 12 TCA** off the retired ±5 window onto the version-range read (`QueryReference::at(v, rung)`), which is what replaced it. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 9ecf739b0..547b7d279 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -173,25 +173,49 @@ row was NEEDS FALSIFIER; `examples/adaptive_order_probe.rs` is that falsifier — 65 536 rows, 5 conjuncts, all 120 permutations, 4 regimes, counting 64-row words a gated `Pred` does not evaluate: -| regime | survivors | as written | best | spread | -|---|---|---|---|---| -| selective | 36 (0.055%) | 5.66% | 80.66% | 75.00 pts, 14.2x | -| moderate | 14 311 (21.8%) | 0.00% | 0.00% | 0 | -| permissive | 61 777 (94.3%) | 0.00% | 0.00% | 0 | -| clustered (address prefix) | 31 (0.047%) | 99.90% | 99.90% | 0 | - -Verdict: order MOVES the skip fraction, so A1 is not ELIMINATE. But the -control signal is **dead words, not selectivity** — selective and clustered -have near-identical survivor counts (36 vs 31) and differ by 19 points of -skip, because one conjunct's survivors are contiguous and the other's are -scattered. At 21.8% survival a 64-row word is all-dead with probability -~2e-7, so no ordering can skip anything and the knob is inert by arithmetic, -not by implementation. DuckDB's adjacent-transposition hill-climb with -swap-likeliness decay is deliberately NOT ported: the quantity is a step -function of clustering, not a smooth function of selectivity, so a local -search over adjacent swaps explores the wrong landscape. The score is the -CALLER's — this crate builds programs and never evaluates one, so it cannot -measure anything. +| regime | survivors | as written | worst | best | spread (best−worst) | +|---|---|---|---|---|---| +| selective | 36 (0.055%) | 5.66% | 5.66% | 80.66% | 75.00 pts, 14.2x | +| moderate | 14 311 (21.8%) | 0.00% | 0.00% | 0.00% | 0 | +| permissive | 61 777 (94.3%) | 0.00% | 0.00% | 0.00% | 0 | +| clustered (address prefix) | 31 (0.047%) | 99.90% | 0.00% | 99.90% | 99.90 pts | + +> ⊘ **CORRECTED 2026-09-15.** This table had no `worst` column and silently +> computed spread as `best − as-written`, so the clustered row published a +> spread of **0** — reading as "order does not matter here", the exact inverse +> of that row's whole point. The probe prints `best − worst` and the matrix +> uses that; clustered is **0.00 % → 99.90 %**, the widest spread of the four. + +Verdict: order MOVES the skip fraction, so A1 is **ADAPT, conditionally** — +it is exactly inert in two of the four regimes, and the pre-registered +falsifier's *representative predicate stream* half was never run (filed in the +matrix's §9). Selectivity cannot tell you WHETHER reordering is worth +anything: selective and clustered have near-identical survivor counts (36 vs +31) and differ by 19 points of achievable skip (best against best), because +one conjunct's survivors are contiguous and the other's are scattered. At +21.8% survival with survivors SCATTERED a 64-row word is all-dead with +probability ~1.4e-7, so no ordering can skip anything there. The score is the +CALLER's — the shipped surface builds programs and never evaluates one, so +there is no point at which it could measure a score. + +> ⊘ **THREE CORRECTIONS 2026-09-15, one of them a measured reversal.** +> (a) This read *"the control signal is **dead words, not selectivity**"* — a +> claim about the right SORT KEY. The probe enumerates permutations and +> reports min/max; it never ranks by either signal and never calls +> `and_by_skip`. Only the between-regime diagnostic is supported. +> (b) The inertness condition was stated as density ("at 21.8% survival … by +> arithmetic, not by implementation"); the condition is SCATTERING, the +> arithmetic is Bernoulli-independence and so conditional on this fixture's +> LCG, and the figure is 1.4e-7 not 2e-7. +> (c) **The hill-climb argument was FALSE.** It said "a local search over +> adjacent swaps explores the wrong landscape". Measured, instrumenting the +> probe's own model with the clustered prefix term at each index: +> `[4092, 3069, 2046, 1023, 0]`, adjacent deltas all exactly `−1023` — a +> monotone linear ramp, the friendliest hill-climb landscape there is. The +> real reason it is not ported is narrower: this crate never executes, so +> there is no runtime for a hill-climb to measure. (DuckDB adapts on measured +> RUNTIME seeded from a selectivity heuristic, not on measured selectivity — +> a second wording this entry had wrong.) Five disables, all red: gate-never-on-accumulator; an `OR` gating its children on its own accumulator; `hoist_gate_subset` never rotating; `and_by_skip` @@ -236,8 +260,21 @@ Three things the scaffold owed and this closes: and the doc says why: `masked_strided_group_sum` exists in the facade but the IR names no strided operand and no group terminal. -10 tests, every one differential against a per-row oracle that never sees a -`Program`. The 64k vertical slice (`COUNT(alpha & ((A&B)|C))`) agrees across +10 tests at the time of this entry. + +> ⊘ **CORRECTED 2026-09-15.** This read "10 tests, every one differential +> against a per-row oracle that never sees a `Program`". The "every one" was +> never true and the same overstatement reached two other places. Measured on +> the branch as it stands (13 tests): **9 differential**, **4 structural** — +> `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` +> (reads the emitted `ops`), `the_fused_lowering_trades_slots_for_passes` (op +> histogram), `a_wide_conjunction_costs_one_extra_slot_not_one_per_child` +> (slot count), and `an_empty_junction_is_refused_rather_than_folded_to_an_identity` +> (error shape). That is not a weakness — the structural four catch shapes no +> row count would notice — but "all differential" is the wrong summary, and in +> `rust-test.yml` it was the stated JUSTIFICATION for the CI line. + +The 64k vertical slice (`COUNT(alpha & ((A&B)|C))`) agrees across five readings — oracle, both lowerings on the executor, both on mask-risc's reference evaluator — with two-sided anti-vacuity (a proper subset of alpha AND strictly below the ungated remainder, so a dropped gate fails even though diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index a95909233..dc4210824 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -31,7 +31,7 @@ evaluates one; `execute` stays the consumer's call on a scratch it owns. | D-QCK-7 | the join — `src_mask → hop → dst_mask` | **Blocked** on mask-risc's PR5: there is no `hop` op to lower to | — | | D-QCK-10 | the lowering convergence — `lgj-abi`'s `plan_lower` (a flat op list) and `quack::lower` (a Boolean tree) pinned equal. **A differential, deliberately NOT a delegation:** quack is a DEV-dependency of `lgj-abi`, because the membrane must not depend on a consumer of the IR it serves | Shipped (lance-graph-java `8ad1a1b` + `e9bf3aa`, `native/lgj-abi/src/exports/tests/lowering_convergence.rs`). Five disables red. Two dead fixtures found: `LT_I32(500)` on a `-150..=361` lane made the whole 16-vector arity-4 arm inert (15 → 21 of 28) and made a real `LE_I32 → LtI32` mis-map INVISIBLE (measured: red at operand 300, green at 500) | `the_two_lowerings_agree_on_every_combine_vector` (28 vectors, `assert_eq!(non_degenerate, 21)`), `the_all_rows_shortcut_is_the_same_condition_on_both_sides` (two-sided: 3 `AllRows`, 25 not), `every_opcode_maps_to_the_same_predicate_in_both_lowerings` (`assert_eq!(proper_subset, 9)` + the TCAM half-swap) | | D-QCK-8 | accumulator gating — an `AND` gates each later child under the accumulator built so far, not only under a resident plane. The same rule `lgj-abi`'s `plan_lower` already had; the two lowerings now implement one law | Shipped. Disable-verified 3×: gate-never-on-accumulator, an `OR` gating its children on its own accumulator, and `hoist_gate_subset` never rotating — each reddens | `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` via the new `gate_shape` helper (all-gated / on-accumulator / dropped, three facts not two); `every_lowered_filter_agrees_with_an_independent_per_row_reading` is what catches the DROP going unsound | -| D-QCK-9 | A1 `Filter::and_by_skip` — the caller's measured skip score orders an `AND`'s conjuncts. DuckDB's adjacent-transposition hill-climb deliberately NOT ported | Shipped, with the matrix's A1 falsifier RUN (`examples/adaptive_order_probe.rs`, 65 536 rows × 5 conjuncts × all 120 permutations × 4 regimes). Order moves the skip fraction, so A1 is not ELIMINATE — but the control signal is DEAD WORDS, not selectivity | `skip_ordering_moves_the_work_and_never_the_answer`: the reordering agrees with the oracle (not with the other lowering), the emitted program's first predicate is the highest-scored, and descending/no-sort disables both redden | +| D-QCK-9 | A1 `Filter::and_by_skip` — the caller's measured skip score orders an `AND`'s conjuncts. DuckDB's hill-climb NOT ported (see the corrected reason) | Shipped, with the matrix's A1 falsifier RUN (`examples/adaptive_order_probe.rs`, 65 536 rows × 5 conjuncts × all 120 permutations × 4 regimes). Order moves the skip fraction, so A1 is **ADAPT conditionally** — inert in 2 of the 4 regimes, and the pre-registered *representative predicate stream* half was never run (matrix §9). ⊘ **Corrected 2026-09-15:** this row read "the control signal is DEAD WORDS, not selectivity" — the probe never ranks by either signal and never calls `and_by_skip`, so only the between-regime diagnostic is supported; and the "hill-climb explores the wrong landscape" reason was MEASURED FALSE (`[4092, 3069, 2046, 1023, 0]`, deltas all −1023: a monotone ramp). The real reason is that this crate never executes | `skip_ordering_moves_the_work_and_never_the_answer`: the reordering agrees with the oracle (not with the other lowering), the emitted program's first predicate is the highest-scored, and descending/no-sort disables both redden. **The probe itself is run by no gate** — `cargo test` compiles an example and never executes it — so the four-row table is a measured-once observation, not a pinned one | ## mask-risc-executor (PR3 — D-ids minted 2026-09-14, `.claude/plans/mask-risc-executor-v1.md`) diff --git a/.claude/harvest/duckdb-headers/README.md b/.claude/harvest/duckdb-headers/README.md index 790d66e0b..6e02048e5 100644 --- a/.claude/harvest/duckdb-headers/README.md +++ b/.claude/harvest/duckdb-headers/README.md @@ -23,7 +23,26 @@ same `args.txt`. Per-header counts: | `execution_ht_entry` | 12 | 92 | **123 methods, 1,622 events** where the `.cpp` pass yielded none for the same -concepts. Two headers yield nothing and that is information too: +concepts. + +> ⊘ **HONESTY NOTE, added 2026-09-15 after review.** Two things this file +> claimed by implication and does not deliver. +> +> **The numbers are not re-derivable from this repo.** The TSV is not +> committed, there is no `args.txt` here, and no DuckDB source is in this tree +> — a repo-wide grep finds `harvest_events` and `args.txt` only inside this +> README. Treat the counts as a recorded observation of a run that happened +> against a local checkout, not as evidence a reader can verify. +> +> **§6's complaint is therefore NOT discharged.** §6 says *"no row in this +> matrix cites a harvest TSV as evidence."* That is still true: what exists +> now is a README quoting counts from a TSV that is absent. What the harvest +> genuinely changed in the consumer crate is two things — `Cmp::MatchU64` and +> the range-write note — and those stand on their own. +> +> **The headline and the inventory disagree.** The per-header table sums to +> 123 and the method inventory below lists 92 distinct names; the difference +> is overloads. Read 123 as definitions, 92 as names. Two headers yield nothing and that is information too: `row_matcher.hpp` and `constant_operators.hpp` are pure declarations whose bodies are templates no TU instantiates here — the .cpp pass already covered `row_matcher.cpp`, which did produce events. diff --git a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md index 42531d62b..270eb2602 100644 --- a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md +++ b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md @@ -526,14 +526,14 @@ rounding error dressed as a finding.* | verdict | whole rows | split halves | rows | |---|---|---|---| -| **ADAPT** | **15** | — | R5, R7, R8, E1, E2, E4, E5, E7, E10, E11, C1, C3, C5, A4, **A1** (moved 2026-09-14, §8a) | +| **ADAPT** | **15** | — | R5, R7, R8, E1, E2, E4, E5, E7, E10, E11, C1, C3, C5, A4, **A1** (moved 2026-09-14 — **conditional**, see §8a: inert in 2 of the 4 regimes, and the pre-registered *representative predicate stream* half was never run) | | **ELIMINATE** | **8** | **+2** | R1, R3, R4, E3, E8, E9, C2, C4 · *plus* R6(the NULL **role**), A5(composition) | | **V3 BETTER** | **3** | **+1** | E6, C6, C7 · *plus* A3(addressed case) | | **KEEP** | **2** | **+2** | R2, C8 · *plus* R6(the mask **representation**), A5(egress) | | **NEEDS FALSIFIER** | **1** | **+1** | A2 · *plus* A3(unaddressed case) — ⊘ A1 left this column 2026-09-14 | | | **29** | **+6** | = 35 entries / 32 rows | -Read the shape rather than the totals: **ADAPT dominates (14/32)**, which is the +Read the shape rather than the totals: **ADAPT dominates (15/32)**, which is the non-obvious result. The tempting summary — "DuckDB's machinery is an artifact of index lists, delete it" — is contradicted by the matrix's own count: only 8 rows eliminate cleanly, and they are concentrated in *representation* (R1, R3, R4) and @@ -606,31 +606,75 @@ positions of one ordering (term 0 is the ungated seed). **Verdict: ADAPT, not ELIMINATE — but not DuckDB's algorithm.** Three findings, each of which changes what should be built: -1. **The knob is inert wherever the population is not sparse, by arithmetic - rather than by implementation.** At 21.8 % survival a 64-row word is - all-dead with probability ≈ 0.782⁶⁴ ≈ 2·10⁻⁷, so *no* ordering skips +1. **The knob is inert wherever survivors are SCATTERED, whatever the + density.** At 21.8 % survival with independent placement a 64-row word is + all-dead with probability `0.78163⁶⁴ ≈ 1.4·10⁻⁷`, so *no* ordering skips anything and the best and worst permutations are identical. Reordering is not "a small win" in the moderate and permissive regimes; it is exactly zero. Any cost model that spends on ordering there is spending on nothing. -2. **The control signal is DEAD WORDS, not selectivity.** The selective and - the clustered regimes have almost identical survivor counts — 36 and 31 — - and differ by **19 percentage points of achievable skip** *as written*, - because one conjunct's survivors are contiguous and the other's are - scattered. Rank by selectivity and those two look the same; rank by dead - words and they do not. This is also why V3 has the lever at all: an address - prefix selects a contiguous subtree (`Filter::prefix_u64`, R5/R8), which is - the clustered row. - -3. **DuckDB's adjacent-transposition hill-climb must NOT be ported**, and the - row above already anticipated the reason (*"its swap-likeliness decay is not - obviously the right control law"*). The measurement says why: the optimised - quantity is a **step function of clustering**, not a smooth function of - selectivity, so a local search over adjacent swaps is exploring the wrong - landscape. In the clustered regime the as-written order is already at 99.90 % - and the worst is 0.00 % — a hill-climb starting from the worst has no - adjacent swap that improves anything until it happens to move the prefix - term to the front. + > ⊘ **CORRECTED 2026-09-15.** This read *"inert wherever the population is + > not sparse, by arithmetic rather than by implementation"*, and both + > halves were wrong. **Density is not the condition** — finding 2 below + > says so two paragraphs later, and a dense-but-contiguous population has + > plenty of dead words. The condition is scattering. And the arithmetic is + > a Bernoulli-INDEPENDENCE computation, so it is conditional on a + > uniformity that is a property of this fixture's LCG, not of the + > substrate — "by arithmetic, not by implementation" claimed a hardness it + > does not have. The figure was also stated as `2·10⁻⁷`; recomputed, it is + > `1.419·10⁻⁷`. + +2. **Selectivity cannot tell you WHETHER reordering is worth anything.** The + selective and the clustered regimes have almost identical survivor counts — + 36 and 31 — and differ by **19 percentage points of achievable skip** + (best-vs-best: 80.66 % against 99.90 %), because one conjunct's survivors + are contiguous and the other's are scattered. A selectivity-only cost model + cannot separate those two cases. This is also why V3 has the lever at all: + an address prefix selects a contiguous subtree (`Filter::prefix_u64`, + R5/R8), which is the clustered row. + + > ⊘ **CORRECTED 2026-09-15, twice.** (a) The 19 points were attributed *as + > written*; as written the two regimes differ by **94.24** points (5.66 vs + > 99.90) and 19.24 is the best-vs-best figure. The italic made the sentence + > false against its own table. (b) The heading read *"the control signal is + > DEAD WORDS, not selectivity"* and closed *"rank by selectivity and those + > two look the same; rank by dead words and they do not"* — a claim about + > the right SORT KEY. **That is not what was measured.** The probe + > enumerates all 120 permutations and reports min/max; it never computes a + > selectivity-ranked order, never computes a dead-word-ranked order, never + > compares two ranking rules, and never calls `and_by_skip`. What the + > evidence supports is the diagnostic above — a between-regime statement — + > not a between-orderings one. Whether dead words is also the better sort + > key is **untested**. + +3. **DuckDB's adjacent-transposition hill-climb is not ported — but not + because its search would fail.** ⊘ **CORRECTED 2026-09-15: the original + argument here was measured FALSE.** It read: *"the optimised quantity is a + step function of clustering … so a local search over adjacent swaps is + exploring the wrong landscape. In the clustered regime … a hill-climb + starting from the worst has no adjacent swap that improves anything until + it happens to move the prefix term to the front."* + + Measured, instrumenting the probe's own `skipped_words` model with the + prefix term at each index: **`[4092, 3069, 2046, 1023, 0]`, adjacent deltas + all exactly `−1023`.** The prefix term's mask is one live word of 1024, and + `skipped_words` charges `dead_words(acc)` once per gated position, so skip + is `(4 − p) · 1023` in the term's index `p` — a monotone linear ramp. Every + single forward adjacent swap improves it by the same amount. That is the + *most* hill-climb-friendly landscape possible, not the wrong one, and the + quoted sentence is the exact opposite of the truth. + + The step-like behaviour is BETWEEN the four regimes (does this conjunction + contain a clustered term at all); the search space is WITHIN one + permutation set, where the objective is the ramp above. The original + reasoned from the first to the second. + + **The real reason it is not ported** is narrower and is not about search + quality: this crate never executes, so there is no runtime for a hill-climb + to measure. Whether a consumer that DOES execute should run one is + **unanswered here** — and, per the note on `[H]` below, DuckDB's own loop + adapts on measured RUNTIME rather than on selectivity, so a comparison + would have to be against that. **What was built instead:** `Filter::and_by_skip(impl IntoIterator)` — the caller supplies a measured skip score per conjunct from a @@ -651,6 +695,28 @@ Board: `STATUS_BOARD.md` D-QCK-9; the probe is committed, not a one-off. ## §9 — Claims in this document that are NOT verified in code +> **⊘ ADDED 2026-09-15 — two A1 claims belong here and were not filed.** §8a's +> verdict was written as settled and this register, which exists for exactly +> this, was not updated in the same pass. +> +> - **A1's own falsifier is half-run.** The pre-registered question (the A1 +> row's FALSIFIER cell) is *"on a **representative predicate stream**, what +> fraction of `Pred` ops are skippable by `under`, and does term order +> change that fraction?"* What ran is one LCG seed, 65 536 rows, five +> conjuncts, and **four hand-designed regimes** — one of them constructed +> specifically to contain a contiguous address-prefix term. That answers the +> ORDER half on a fixture and answers the FRACTION half only as "between +> 0 % and 99.9 %, pick a regime". `[H]` Nothing in this repo samples a +> workload V3 actually runs. +> - **"Dead words is the better sort key" is untested**, per §8a finding 2's +> own correction: the probe never compares ranking rules. `[H]` +> +> A third, related: the pre-registered text ends *"its swap-likeliness decay +> is **not obviously** the right control law"* — a hedge. §8a first cited that +> as having "anticipated the reason" and upgraded it to "must NOT be ported". +> A hedge quoted as a prediction is not a confirmation; §8a finding 3 is now +> corrected to say so. + Collected so they are not quoted as findings. Everything else in §2–§5 carries a `file:line` read this session. diff --git a/.github/workflows/rust-test.yml b/.github/workflows/rust-test.yml index 7da61aa31..30f1e0eb3 100644 --- a/.github/workflows/rust-test.yml +++ b/.github/workflows/rust-test.yml @@ -127,8 +127,10 @@ jobs: # lance-graph-quack: the DuckDB-shaped surface that LOWERS to mask-risc. # Same reason as the mask-risc line above — a member's tests are not run # by `cargo build --workspace`, so the line lands the day the crate - # joins. Its 13 tests are all differential against a per-row oracle, so - # an un-run suite is an un-checked lowering, not merely an un-run test. + # joins. 9 of its 13 tests are differential against a per-row oracle + # (the other 4 are structural: gate shape, the pass/slot trade, + # width-vs-depth, the empty-junction refusal), so an un-run suite is an + # un-checked LOWERING, not merely an un-run test. - name: Run quack tests run: cargo test --manifest-path crates/lance-graph-quack/Cargo.toml # weather-poc is workspace-EXCLUDED. A new workflow introduced by the diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index 95af8760e..9629886b7 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -100,16 +100,24 @@ //! today and says so plainly; the range WRITE waits on the primitive, per the //! missing-capability STOP rule, rather than being hand-rolled one layer up. //! -//! # The one thing DuckDB has that this does not +//! # What DuckDB has that this does not: the measurement loop //! -//! `AdaptiveFilter` (matrix row A1, its only NEEDS-FALSIFIER of this kind) -//! permutes a conjunction's terms at RUNTIME by measured selectivity. This -//! crate lowers children in the order they were written, and that is not -//! neutral here: under the survivor skip a cheap, highly selective conjunct -//! placed first shrinks every later predicate's live-word count, so order is a -//! real cost lever and the caller currently owns it with no help. Saying so is -//! the honest state; a selectivity-ordered lowering is a design with a -//! measurement attached, not a line to add. +//! `AdaptiveFilter` (matrix row A1) permutes a conjunction's terms at RUNTIME, +//! seeded from a static selectivity heuristic and then adapted by measured +//! RUNTIME. This crate cannot do that, and the reason is structural rather +//! than unfinished: it never executes, so there is nothing for it to measure. +//! +//! Order is still a real cost lever here — under the survivor skip a conjunct +//! whose survivors die in whole WORDS shrinks every later predicate's live +//! count — so [`Filter::and_by_skip`] takes the ordering decision as an INPUT. +//! The measurement that licensed even that much is +//! `examples/adaptive_order_probe.rs`; read [`Filter::and_by_skip`] for what it +//! does and does not establish. +//! +//! ⊘ This section previously called A1 "its only NEEDS-FALSIFIER of this kind" +//! and said "the caller currently owns it with no help". Both were true when +//! written and stopped being true in the same branch: A1 left that column and +//! `and_by_skip` is the help. //! //! # Status //! @@ -274,8 +282,11 @@ impl Filter { /// /// # Why the score is dead WORDS and not selectivity /// - /// DuckDB ranks conjunction terms by measured selectivity, because there - /// term `k` runs only on the survivors of `1..k-1` — a selective term + /// DuckDB orders conjunction terms by a static selectivity heuristic and + /// then adapts by measured RUNTIME (`adaptive_filter.cpp`: swap two + /// neighbours, measure 10 iterations, keep if mean runtime dropped). It + /// can rank that way because there term `k` runs only on the survivors of + /// `1..k-1` — a selective term /// first literally shrinks the input. In V3 a predicate sweep costs the /// full column wherever it sits, so ordering can only pay by AVOIDANCE: /// the survivor skip drops a 64-row WORD when the gate has no survivor in @@ -290,34 +301,67 @@ impl Filter { /// | selective | 0.055 % | 5.66 % → **80.66 %** (14.2×) | /// | clustered (an address prefix) | 0.047 % | 0.00 % → **99.90 %** | /// - /// Two findings, and the second is the one that picks the control law. + /// Two findings, and a third that is a correction rather than a result. /// /// **Order does move the skip fraction, so A1 is not ELIMINATE** — but - /// only where there is anything to skip. At 21.8 % survival, uniformly - /// spread, a 64-row word is all-dead with probability `0.782^64 ≈ 2e-7`, - /// so NO ordering skips anything and the whole question is moot. Word - /// granularity needs the accumulator to die in whole words, not merely to - /// be small. + /// only where there is anything to skip. At 21.8 % survival with + /// survivors SCATTERED, a 64-row word is all-dead with probability + /// `0.78163^64 ≈ 1.4e-7`, so NO ordering skips anything and the whole + /// question is moot. Word granularity needs the accumulator to die in + /// whole words, not merely to be small — scattering is the condition, + /// not density, and a dense-but-contiguous population has plenty of dead + /// words. + /// + /// **Which means selectivity cannot tell you WHETHER reordering is worth + /// anything.** The selective and the clustered regimes have almost + /// identical survivor counts — 36 and 31 rows — and differ by 19 + /// percentage points of achievable skip (best against best: 80.66 % vs + /// 99.90 %), because one conjunct's survivors are contiguous and the + /// other's are scattered. A selectivity-only cost model cannot separate + /// those two cases. That is also why V3 has this lever at all: an address + /// prefix selects a contiguous subtree ([`Filter::prefix_u64`]), which is + /// the clustered row of that table. /// - /// **Which means selectivity is the wrong signal.** The selective and the - /// clustered regimes have almost identical survivor counts — 36 and 31 - /// rows — and differ by 19 percentage points of skip, because one - /// conjunct's survivors are contiguous and the other's are scattered. Rank - /// by selectivity and those two look the same. Rank by DEAD WORDS and they - /// do not. That is also why V3 has this lever at all: an address prefix - /// selects a contiguous subtree ([`Filter::prefix_u64`]), which is the - /// clustered row of that table. + /// Note what that does NOT say. An earlier version of this paragraph read + /// *"rank by selectivity and those two look the same; rank by DEAD WORDS + /// and they do not"* — a claim about the right SORT KEY. The probe does + /// not support it: it enumerates all 120 permutations and reports + /// min/max, never computing a selectivity-ranked order, never computing a + /// dead-word-ranked order, never comparing two ranking rules, and never + /// calling this function. Whether dead words is the better key to sort by + /// is **untested**; what is measured is the between-regime diagnostic + /// above. /// /// # Why the score is the CALLER's, and why there is no hill-climb /// - /// This crate builds programs and never evaluates one, so it cannot - /// measure anything; the score comes from a previous execution the caller - /// ran. DuckDB's adjacent-transposition hill-climb with swap-likeliness - /// decay is NOT ported — the matrix anticipated that too (*"its - /// swap-likeliness decay is not obviously the right control law"*), and - /// the measurement says why: the quantity being optimised is a step - /// function of clustering, not a smooth function of selectivity, so a - /// local search over adjacent swaps is exploring the wrong landscape. + /// The shipped surface of this crate builds programs and never evaluates + /// one, so there is no point at which it could measure a score; it comes + /// from a previous execution the caller ran. + /// + /// DuckDB's adjacent-transposition hill-climb is not ported — **and the + /// reason first given here was measured FALSE, so it is worth stating + /// correctly.** The claim was that "the quantity being optimised is a step + /// function of clustering … so a local search over adjacent swaps is + /// exploring the wrong landscape". Instrumenting the probe's own + /// `skipped_words` model with the clustered regime's prefix term at each + /// index gives `[4092, 3069, 2046, 1023, 0]` — adjacent deltas all + /// exactly `-1023`, a monotone linear ramp, because the prefix term's + /// mask is one live word of 1024 and each gated position past it skips + /// the other 1023. Every forward adjacent swap improves it by the same + /// amount. That is the friendliest possible hill-climb landscape, not the + /// wrong one. + /// + /// The step-like behaviour is BETWEEN regimes (does this conjunction + /// contain a clustered term at all); the search space is WITHIN one + /// permutation set. The original argument reasoned from the first to the + /// second. + /// + /// The real reason is narrower: this crate never executes, so there is no + /// runtime for a hill-climb to measure. Whether a consumer that DOES + /// execute should run one is unanswered here — and DuckDB's loop adapts + /// on measured RUNTIME, seeded from a static selectivity heuristic, not + /// on measured selectivity, so the comparison would have to be against + /// that. /// /// # One caveat, because it is a real override /// From 2c15caea5a07d77885a91e94890a264d6a715dea Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 17:52:40 +0000 Subject: [PATCH 13/25] review: the accumulator gate outranked the plane and silently dropped it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit codex flagged `emit_gated`'s `acc_gate.or_else(|| under…)` on PR #1235. It is a real wrong answer, not a style point, and the reproduction is a number: on `P1 AND (Plane(focus) AND P2)` the oracle says 29 and the emitted program returns 204, with `FOCUS` absent from the op list entirely. The outer conjunction evaluates `P1` into an accumulator, then recurses into the parenthesis carrying that accumulator. The inner `AND` meets the plane, sets `under`, and `or_else` hands the later comparison the accumulator — which has never seen the plane. The conjunct is dropped. Fixed by inverting the preference: the plane always wins. Conservative rather than optimal — where both gates are live the accumulator may be narrower, so this can leave skip on the table. It can never be wrong, because `Pred { under: g }` is exactly `g ∧ pred` and the consuming `AND` re-applies the accumulator anyway. The other direction drops a conjunct. One existing test asserted the plane rode the accumulator; that was a description of the bug, so it is inverted in place with a ⊘ note rather than deleted. `a_nested_plane_survives_an_outer_accumulator` is the permanent regression: two-sided (`expected * 2 < without_plane`, so a fixture whose plane admits everything cannot pass) plus a structural check that some op reads `FOCUS`. Both disable arms RED, then green. Also from the same review round, each verified before applying: - the nars-34 audit's item 1 called five recipes "the honest `Datapath` set" while its own table two sections up classifies them 4 `Gate` + 1 `Control`, zero `Datapath`; and its closing section re-asserted "as a dispatch key", the exact phrase the file's own top correction retracts. Both corrected. - `STATUS_BOARD.md` D-QCK-6 had an unescaped `|` inside a code span, splitting the row into a sixth cell. - the quack clippy line in `style.yml` gained `--all-features`. Two suggestions declined on measurement: `cargo fmt --all` (mis-formatting `mask-risc/src/ir.rs` shows the existing scoped line already catches it, so `--all` only widens scope), and reordering the new EPIPHANIES entry below the existing 09-15 block (that file is reverse-chronological). Gates: quack 14/14, clippy `-D warnings` clean, fmt clean, supersession index already current. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/audits/nars-34-substrate-audit.md | 11 +- .claude/board/EPIPHANIES.md | 71 +++++++++++ .claude/board/LATEST_STATE.md | 36 ++++++ .claude/board/STATUS_BOARD.md | 2 +- .github/workflows/style.yml | 2 +- crates/lance-graph-quack/src/lib.rs | 140 +++++++++++++++++++--- 6 files changed, 239 insertions(+), 23 deletions(-) diff --git a/.claude/audits/nars-34-substrate-audit.md b/.claude/audits/nars-34-substrate-audit.md index 60720bb08..28e84ef2e 100644 --- a/.claude/audits/nars-34-substrate-audit.md +++ b/.claude/audits/nars-34-substrate-audit.md @@ -113,7 +113,11 @@ actually exists. ## What needs improving, in order 1. **Re-derive `bucket` from the masking algebra**, not from the VSA substrate. - The five above are the honest `Datapath` set today; the current nine are not. + The five above are what the algebra can run today, and the table classifies + them **4 `Gate` + 1 `Control`, zero `Datapath`** — the current nine + `Datapath` labels are the ones that are not honest. ⊘ This read "the five + above are the honest `Datapath` set today", contradicting the table two + sections up; caught in review on PR #1235. 2. **Rewrite the 9 stale `substrate` strings** to name mask-algebra compositions, or mark them `unrealized` — a string naming a retired kernel reads as a spec and is worse than an empty one. @@ -133,5 +137,8 @@ actually exists. It does not say the 34 are wrong as *tactics*. It says their recorded realization is against a substrate that no longer exists, and that the routing -column derived from it cannot be trusted as a dispatch key until re-derived. +column derived from it cannot be trusted until re-derived. (⊘ this said +"as a dispatch key"; per the correction at the top of this file `bucket` is +NOT a dispatch key at all except for the `Bucket::Gate` arm, so the phrase +re-asserted the thing the correction retracted.) Nothing here was executed; this is a read of the catalogue against the op list. diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 6e26c5207..88b1aec64 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -155,6 +155,77 @@ the seam-6 promoter that does ship), `.claude/plans/hexagon-plasticity-v1.md`. --- +## 2026-09-15 — E-THE-ACCUMULATOR-GATE-OUTRANKED-THE-PLANE-AND-SILENTLY-DROPPED-IT-1 — a nested `Plane` vanished from the emitted program because the *enclosing* conjunction already had a gate + +**Status:** CORRECTION — a real correctness bug in shipped-in-PR code (`lance-graph-quack` +`emit_gated`), found by codex review on PR #1235, reproduced against the reference oracle +before the fix, fixed, and pinned by a permanent regression. +**Confidence:** HIGH — the reproduction is a number, not an argument: oracle **29**, emitted +program **204**, and the op list showed the plane's `FOCUS` operand absent entirely. + +### What happened + +`emit_gated(filter, under, acc_gate)` carries two gates. `under` is the plane the CALLER +handed down — `Filter::Plane(m)` encountered as a child, which must gate everything to its +right. `acc_gate` is the accumulator-so-far, the thing that makes the survivor skip work: +once some terms have been evaluated into a scratch, later comparisons can be evaluated +`Pred { under: Some(acc) }` and skip the 64-row words the accumulator already killed. + +The line was: + +```rust +let gate = acc_gate.or_else(|| under.map(|m| Operand::Plane(m.0))); +``` + +`Option::or_else` — the accumulator WINS whenever both are present. For a flat conjunction +that is right and is the whole point (the accumulator is strictly narrower than the plane +it already absorbed). For a **nested** one it is wrong, because the accumulator belongs to +the OUTER conjunction and has never seen the inner plane: + +``` +P1 AND (Plane(focus) AND P2) +``` + +The outer `AND` evaluates `P1` into the accumulator, recurses into the parenthesis with +`under = None, acc_gate = Some(acc)`; the inner `AND` meets `Plane(focus)`, sets +`under = Some(focus)`, and then for `P2` the `or_else` picks the accumulator and drops the +plane on the floor. `FOCUS` never appears in the program. Measured: 204 rows where the +oracle says 29. + +### The fix, and why it is the conservative direction + +```rust +let gate = under.map(|m| Operand::Plane(m.0)).or(acc_gate); +``` + +The plane always wins. This is conservative rather than optimal: where both are live the +accumulator may be the narrower gate, so preferring the plane can leave some skip on the +table. It can never be WRONG, because `Pred { under: g }` is exactly `g ∧ pred` +(`reference.rs:460`) and the `AND` that consumes the scratch re-applies the accumulator +anyway. The reverse — preferring the accumulator — drops a conjunct, which is a wrong +answer, not a slower one. **A gate you can only lose by choosing beats a gate you can drop +by choosing.** + +### The lesson that generalizes past this bug + +**`or_else` between two gates is a silent priority decision, and priority between gates is +only safe when one PROVABLY subsumes the other.** The accumulator subsumes `under` exactly +when the accumulator was built from a prefix that already included it — true for a flat +conjunction, false the moment recursion hands the accumulator across a nesting boundary. My +own hand-trace missed it because I traced the flat case, where the two are in fact ordered. +The shape that breaks it needs a plane that is BOTH nested and not first, which no existing +test had: `hoist_gate_subset` rotates a plane-subset child to the front precisely so the +accumulator starts inside the plane, and every fixture went down that path. + +### What the fix cost in the test suite, recorded because it is uncomfortable + +`the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` asserted the plane +rode the ACCUMULATOR (`assert!(on_acc)`). That assertion was a description of the bug, not +of the law — it is now inverted (`assert!(!on_acc)`) with a ⊘ note, and the permanent +regression `a_nested_plane_survives_an_outer_accumulator` is what actually pins the +behaviour: two-sided (`expected * 2 < without_plane`, so a fixture whose plane admits +everything cannot pass) plus a structural check that some emitted op reads `FOCUS`. Both +arms of the disable run are RED. ## 2026-09-15 — E-I-DECLARED-A-JOIN-ABSENT-BY-GREPPING-ONE-FILE-AND-COMPOSE-IS-THE-SAME-XOR-A-THIRD-TIME-1 — the canonical join shipped in `hhtl.rs` all along, `[a,b]:[b,c]` is `compose_chain`, and the Hexagon substrate is every organ shipped with no nerve between them diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 547b7d279..7a54e7c37 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -76,6 +76,42 @@ census is measured and NOT yet on the board — see the arc entry's *Un-recorded* bullet. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1233. +## 2026-09-15 — PR #1235 review round: a codex P1 that was a real wrong answer, and three accepted doc/CI fixes + +Seven review threads on `claude/clone-repositories-71a5sw`. One was a genuine correctness +bug; two reviewer suggestions were DECLINED on measurement; three were accepted and applied. + +**The P1 (fixed).** `lance-graph-quack` `emit_gated` preferred the accumulator gate over the +caller-supplied plane (`acc_gate.or_else(|| under…)`). On `P1 AND (Plane(focus) AND P2)` the +outer accumulator — which has never seen the inner plane — won, and `FOCUS` never reached the +emitted program. Reproduced against the reference oracle: **oracle 29, emitted 204**. Fixed by +inverting the preference so the plane always wins (conservative: can cost skip, can never drop +a conjunct). One existing test asserted the buggy behaviour and is inverted in place with a ⊘ +note; `a_nested_plane_survives_an_outer_accumulator` is the permanent two-sided regression. +Full statement: `EPIPHANIES.md` +`E-THE-ACCUMULATOR-GATE-OUTRANKED-THE-PLANE-AND-SILENTLY-DROPPED-IT-1`. + +**Accepted.** `.claude/audits/nars-34-substrate-audit.md` item 1 contradicted its own table two +sections up ("the honest `Datapath` set" for five recipes the table classifies **4 `Gate` + 1 +`Control`, zero `Datapath`**) — corrected in place, plus the stale "as a dispatch key" phrase in +the closing section, which re-asserted exactly what the file's own top correction retracts. +`STATUS_BOARD.md` D-QCK-6's unescaped `|` inside `COUNT(alpha & ((A&B)|C))` split the row into a +sixth cell — escaped. `style.yml`'s quack clippy line gained `--all-features`. + +**Declined, both on measurement rather than preference.** (1) `cargo fmt --all` in place of the +per-manifest quack line: mis-formatting `crates/lance-graph-mask-risc/src/ir.rs` and re-running +shows the existing scoped line already reports it, so `--all` widens the blast radius without +adding coverage. (2) Reordering the new `EPIPHANIES.md` entry below the existing 09-15 block: +that file is reverse-chronological, and within a date the later entry goes first. + +**CI `linux-build` is red on this PR and the failure is the BASE's, not the branch's.** `main` +at `030ad80` fails identically. Root cause established rather than guessed: `aws-smithy-types +1.7.0` published 18 minutes after the last green run; `aws-smithy-json 0.63.0` declares +`^1.6.1`, so it always resolves the new one; the newest `aws-config 1.12.0` requires +`^0.63.0` and cannot reach the fixed `0.64.0`. **No resolver-only fix exists**, and none of the +four pinnable coordinates (lance / lancedb / arrow / datafusion) is in that chain — so nothing +in this PR's scope can clear it. Tracked as a repo-wide blocker, not a PR defect. + ## 2026-09-14 (6) — the convergence, answered: pin them equal, do not delegate The operator's hypothesis was *"wiring duckdb through lance-graph-java might diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index dc4210824..585b7d289 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -27,7 +27,7 @@ evaluates one; `execute` stays the consumer's call on a scratch it owns. | D-QCK-3 | `lower_fused` — the Boolean skeleton through `mask_risc::fuse` | Shipped (`18c1d85`). Both lowerings run against the same oracle on every fixture | `the_fused_lowering_trades_slots_for_passes` pins BOTH sides (2 passes/2 slots vs 1 pass/4 slots), so a fuser that stopped fusing and an emitter that started allocating per leaf each fail their own line | | D-QCK-4 | projection — `Agg::Rows` (`Terminal::Keep`, the mask IS the result) and `Agg::BlendI32` (the CASE shape) | Shipped (`18c1d85`) | the kept mask materialises to exactly the oracle's rows; the blend must actually pick from `els` somewhere | | D-QCK-5 | `lower_group_by` — two-phase, a mask plane where the hash table would be; K gated equalities over the kept filter | Shipped (`18c1d85`). K programs not one, because the IR names no strided operand and no group terminal — `masked_strided_group_sum` exists in the facade and cannot be reached from here yet | per-group counts and sums equal the oracle's AND sum to the ungrouped count (the partition identity); a grouped blend is refused, not answered wrongly | -| D-QCK-6 | the 64k vertical slice `COUNT(alpha & ((A&B)|C))` across five arms | Shipped (`18c1d85`) — closes Phase 3+6 | oracle, both lowerings on the executor, both on the reference evaluator, one number; two-sided anti-vacuity (subset of alpha AND strictly below the ungated remainder) | +| D-QCK-6 | the 64k vertical slice `COUNT(alpha & ((A&B)\|C))` across five arms | Shipped (`18c1d85`) — closes Phase 3+6 | oracle, both lowerings on the executor, both on the reference evaluator, one number; two-sided anti-vacuity (subset of alpha AND strictly below the ungated remainder) | | D-QCK-7 | the join — `src_mask → hop → dst_mask` | **Blocked** on mask-risc's PR5: there is no `hop` op to lower to | — | | D-QCK-10 | the lowering convergence — `lgj-abi`'s `plan_lower` (a flat op list) and `quack::lower` (a Boolean tree) pinned equal. **A differential, deliberately NOT a delegation:** quack is a DEV-dependency of `lgj-abi`, because the membrane must not depend on a consumer of the IR it serves | Shipped (lance-graph-java `8ad1a1b` + `e9bf3aa`, `native/lgj-abi/src/exports/tests/lowering_convergence.rs`). Five disables red. Two dead fixtures found: `LT_I32(500)` on a `-150..=361` lane made the whole 16-vector arity-4 arm inert (15 → 21 of 28) and made a real `LE_I32 → LtI32` mis-map INVISIBLE (measured: red at operand 300, green at 500) | `the_two_lowerings_agree_on_every_combine_vector` (28 vectors, `assert_eq!(non_degenerate, 21)`), `the_all_rows_shortcut_is_the_same_condition_on_both_sides` (two-sided: 3 `AllRows`, 25 not), `every_opcode_maps_to_the_same_predicate_in_both_lowerings` (`assert_eq!(proper_subset, 9)` + the TCAM half-swap) | | D-QCK-8 | accumulator gating — an `AND` gates each later child under the accumulator built so far, not only under a resident plane. The same rule `lgj-abi`'s `plan_lower` already had; the two lowerings now implement one law | Shipped. Disable-verified 3×: gate-never-on-accumulator, an `OR` gating its children on its own accumulator, and `hoist_gate_subset` never rotating — each reddens | `the_gate_reaches_every_comparison_and_is_dropped_only_where_it_vanishes` via the new `gate_shape` helper (all-gated / on-accumulator / dropped, three facts not two); `every_lowered_filter_agrees_with_an_independent_per_row_reading` is what catches the DROP going unsound | diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml index 95653d924..9db518b97 100644 --- a/.github/workflows/style.yml +++ b/.github/workflows/style.yml @@ -92,7 +92,7 @@ jobs: # measurement lives in `examples/adaptive_order_probe.rs`, and `--lib` # would leave the probe unlinted. - name: Clippy lance-graph-quack (mandatory) - run: cargo clippy --manifest-path crates/lance-graph-quack/Cargo.toml --all-targets -- -D warnings + run: cargo clippy --manifest-path crates/lance-graph-quack/Cargo.toml --all-targets --all-features -- -D warnings # Tier B (advisory until incrementally cleaned, non-gating): # lance-graph core has ~91 pre-existing clippy violations to be paid down # in subsequent PRs (TD-CLIPPY-LG-1). Don't auto-fix — each violation diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index 9629886b7..4e2f112ed 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -837,12 +837,44 @@ fn emit_gated( ) -> Result { match n { Node::Pred { pred, under } => { - // The accumulator gate wins over the plane gate when both are - // available, and it is strictly narrower rather than merely - // different: the plane was the AND's first conjunct, so it is - // already folded into the accumulator. A `Pred` carries exactly - // one `under`, which is why this is a choice and not a union. - let gate = acc_gate.or_else(|| under.map(|m| Operand::Plane(m.0))); + // A `Pred` carries exactly ONE `under`, so when both a plane gate + // and an accumulator are available this is a choice, not a union + // — and the choice is the PLANE, always. + // + // An earlier version preferred the accumulator on the reasoning + // that it is strictly narrower, "because the plane was the AND's + // first conjunct, so it is already folded into the accumulator". + // That holds only inside the AND that ESTABLISHED the plane, + // where [`hoist_gate_subset`] arranges it. It is false the moment + // an OUTER conjunction with no plane of its own wraps an inner one + // that has: + // + // ```text + // P1 AND (Plane(focus) AND P2) + // ``` + // + // The outer AND finds no plane among its own parts, so it + // establishes no gate and simply folds; the inner AND establishes + // `focus`, gates P2 under it, sees P2 vanish and DROPS the plane + // as implied. Then the outer emitter hands its accumulator `P1` + // down, this line preferred it over `focus`, and `focus` — already + // elided — appeared nowhere in the program. Measured on a 512-row + // fixture: the oracle selects 29 rows, the emitted program + // selected 204, and the op list was `GtI32 AND NeU32` with no + // trace of the plane. A silent wrong answer, not a slow one. + // Pinned by `a_nested_plane_survives_an_outer_accumulator`. + // + // Preferring the plane is sound unconditionally: the emitted mask + // is `plane & pred`, and the enclosing junction intersects the + // accumulator afterwards anyway, so `acc & (plane & pred)` is + // exactly the wanted value. What it costs is the EXTRA narrowing + // the accumulator would have given inside a plane's own AND — the + // plane still gates there, just less tightly than it could. That + // cost is named in `ISSUES.md` rather than traded against a + // correctness hole, and A1's lever is untouched because a + // conjunction of plain comparisons carries no plane at all: those + // preds take the `None` arm below and gate on the accumulator. + let gate = under.map(|m| Operand::Plane(m.0)).or(acc_gate); ops.push(MaskOp::Pred { pred: *pred, under: gate, @@ -1177,6 +1209,68 @@ mod tests { } } + /// FAILS IF: an outer conjunction's accumulator replaces a plane gate + /// established by an INNER conjunction, so the plane — already elided as + /// implied — vanishes from the program entirely. + /// + /// The shape is `P1 AND (Plane(focus) AND P2)`. It is the one the survivor + /// skip's own machinery cannot reach by the route the other gate tests + /// take: the OUTER `AND` finds no plane among its parts, so it establishes + /// no gate, while the INNER one establishes `focus`, sees `P2` vanish + /// under it and drops it. Nothing in the outer scope then carries `focus`. + /// + /// Reported as a P1 by codex on PR #1235 and reproduced before it was + /// believed: the oracle selected **29** rows of 512 and the emitted + /// program selected **204**, its op list `GtI32 AND NeU32` with no trace + /// of the plane. That is the failure mode this asserts against, and the + /// op-list half is what distinguishes "the answer happened to match" from + /// "the gate is actually there". + #[test] + fn a_nested_plane_survives_an_outer_accumulator() { + let fx = Fx::new(512); + let f = Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::and([Filter::plane(FOCUS), Filter::cmp(CLASS, Cmp::NeU32(0))]), + ]); + + // Anti-vacuity, two-sided: the plane must actually exclude rows the + // rest admits, or a lost gate would be invisible in the count. + let without_plane = fx + .rows(&Filter::and([ + Filter::cmp(VALS, Cmp::GtI32(0)), + Filter::cmp(CLASS, Cmp::NeU32(0)), + ])) + .len(); + let expected = fx.rows(&f).len(); + assert!( + expected > 0 && expected * 2 < without_plane, + "the fixture must make the plane load-bearing: {expected} with it, \ + {without_plane} without" + ); + + assert_eq!( + fx.count(&f), + expected, + "the plane was lost from the program" + ); + + // ...and structurally: SOME operand must still read `FOCUS`. A count + // that happens to agree is not evidence the gate survived. + let prog = lower(&Query { + filter: f, + agg: Agg::Count, + }) + .expect("lowers"); + let reads_focus = prog.ops.iter().any( + |op| matches!(op, MaskOp::Pred { under: Some(Operand::Plane(p)), .. } if *p == FOCUS.0), + ); + assert!( + reads_focus, + "no op reads FOCUS — the nested plane is gone: {:?}", + prog.ops + ); + } + /// `alpha & ((A & B) | C)` — the vertical slice. fn slice_filter() -> Filter { Filter::and([ @@ -1417,19 +1511,27 @@ mod tests { "the slice: every predicate gated, alpha implied: {:?}", slice.ops ); - // The accumulator gate is asserted for the IN-PLACE lowering only. - // The fused one gives every comparison its own slot and hands the - // skeleton to the fuser, so there is no running partial result for - // a later comparison to narrow onto. That is a real property of - // that lowering, not an omission, and asserting it for both would - // be asserting something false. - if is_inplace { - assert!( - on_acc, - "no comparison narrowed onto the accumulator: {:?}", - slice.ops - ); - } + // ⊘ RE-PINNED. This asserted `on_acc` for the in-place lowering: + // inside a plane's own AND, later comparisons narrowed onto the + // running accumulator rather than onto the plane. That override + // is what the codex P1 on PR #1235 showed to be UNSOUND once an + // outer conjunction supplies an unrelated accumulator, so + // [`emit_gated`] now always prefers the plane and the assertion + // became false. It is inverted rather than deleted, because the + // property is load-bearing in the other direction: a plane gate + // must never be silently replaced. + // + // A1's lever is untouched and is pinned separately by + // `skip_ordering_moves_the_work_and_never_the_answer`, whose + // conjunction carries no plane — those preds still gate on the + // accumulator. + assert!( + !on_acc, + "a plane-gated comparison narrowed onto the accumulator \ + instead of the plane — the codex-P1 override is back: {:?}", + slice.ops + ); + let _ = is_inplace; let negated = lowering(&Query { filter: Filter::and([ From ea0a3e43eac46a4f2d9b70ced12504e70a1f304a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 18:02:10 +0000 Subject: [PATCH 14/25] board: date-first header on the E-A-FLOOR entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entry was correctly placed — dated 2026-09-14, sitting below six 2026-09-15 entries in a reverse-chronological file. But its header was `## E-A-FLOOR-…-1 (2026-09-14)` where every other entry is `## — E-…`, so anything scanning for a leading date reads it as undated and concludes it is mis-ordered. CodeRabbit did exactly that on PR #1235 and asked for it to be moved to the top, which would have inverted the order it was trying to protect. The finding was wrong; the thing it pointed at was real. Normalising the header makes the ordering machine-checkable instead of correct-by-accident. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/EPIPHANIES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 88b1aec64..ed46b749d 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1497,7 +1497,7 @@ the tile, seriation, LUT and spread contribute nothing that survives permutation **Scope.** Pure-Python lab, no Rust fingerprints — a shape proxy, never a measurement of `ndarray`/`bgz17`/`helix`. The cue reached **87.5 %** of the BPE incumbent (0.1579 vs 0.1805): PROCEED, not PASS. -## E-A-FLOOR-PASSED-AT-ITS-BOUND-IS-A-DEAD-FIXTURE-1 (2026-09-14) +## 2026-09-14 — E-A-FLOOR-PASSED-AT-ITS-BOUND-IS-A-DEAD-FIXTURE-1 **Status:** FINDING (measured twice in one file, both arms) **Confidence:** High — the invisible-mis-map half is a direct red/green pair From 70050009b0fcd1f9b382aef50097d35bbf270e5c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 18:09:48 +0000 Subject: [PATCH 15/25] review round 2: an independent review re-found the P1, and my fmt decline was wrong MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An Opus correctness review ran against the branch while the codex-P1 fix was in flight and found the same defect independently: 2,021 wrong answers in 200,000 randomised trees against a per-row oracle that never builds a Program. It then verified the fix rather than taking it — 120,000 cases clean, 2,668 RED with only the pre-fix gate line restored — plus a 320,000-case sweep across all eight aggregates and 5,000 GROUP BY plans, all clean. Its two HOLD items were prose, and both are fixed here. I also reversed my own review reply. I declined `cargo fmt --all` claiming the scoped per-manifest line already covered the workspace, and said I had measured it. I had not, or had measured the wrong invocation: mis-formatting mask-risc/src/ir.rs gives exit 0 on the scoped quack line and exit 1 on mask-risc's own line, same file. The conclusion survives for a different reason than the one I gave — this fmt job is per-crate by design, and mask-risc has its own line directly above quack's. But checking the job against the member list rather than reading it found what the reviewer was circling: four workspace members had no rustfmt line at all, including lance-graph-contract, which this branch edits. All four measured clean and are now armed. Fixed: - three doc sites still calling hoist_gate_subset's rotation a "correctness requirement" and a "silent WRONG ANSWER". True before the fix, false after; disable-verified by the reviewer (rotation removed, 120,000 cases, 65,919 planed, zero divergences). What it still buys is slot economy, 1 slot vs 2. - a test I re-pinned this session was half-vacuous: `assert!(!on_acc)` ran over both lowerings, but assign_slots never emits a Scratch gate, so on the fused arm no input could make it fail. Scoped to the in-place arm, where the pre-fix line turns it red; the fused arm asserts the structural fact instead. - the probe printed `spread 0.00 percentage points, best/worst infx` on the two regimes that skip nothing in any order, and those rows are cited in and_by_skip's own doc table. - a §8a summary line claimed dead words are the measured ordering signal. The probe permutes and measures skip; it never ranks by dead words against a selectivity ranking. Only the negative half is measured (36 vs 31 survivors, 94.24 points apart). §8a and §9 already said so; that line did not. Recorded rather than fixed, in ISSUES.md: and_by_skip's lever is now inert on any conjunction carrying a plane — the crate's own headline shape — which the fix created and is the right trade anyway; and two caller-controlled pre-execution costs, lower_fused at ~7.6x per doubling and a deep Filter aborting the process at depth 20,000. Gates: quack 14/14, clippy -D warnings clean, fmt clean on all six crates (including the four newly armed), probe re-run, supersession index current. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/ISSUES.md | 83 ++++++++++++++++++ .claude/board/LATEST_STATE.md | 58 +++++++++++++ .../duckdb-to-v3-translation-matrix-v1.md | 10 ++- .github/workflows/style.yml | 16 ++++ .../examples/adaptive_order_probe.rs | 26 ++++-- crates/lance-graph-quack/src/lib.rs | 86 ++++++++++++++++--- 6 files changed, 254 insertions(+), 25 deletions(-) diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 840d0d4a8..58fa2fceb 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,3 +1,86 @@ +## ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE (2026-09-15) — OPEN, and it is the price of the P1 fix + +**`Query::and_by_skip`'s ordering lever buys exactly zero on any conjunction that carries a +resident plane — which is this crate's own headline query shape.** + +Created deliberately by the codex-P1 fix (`b7e6cef`, +`E-THE-ACCUMULATOR-GATE-OUTRANKED-THE-PLANE-AND-SILENTLY-DROPPED-IT-1`): `emit_gated` now +prefers the caller's plane over the accumulator, so every `Pred` inside a planed `AND` gates on +the same FIXED plane regardless of position. Measured at HEAD, each entry `predicate <- its gate`: + +``` +PLANE-FREE And[A,B,C] : [GtI32<-none, EqU32<-ACC(s0), LtI32<-ACC(s0)] + And[C,B,A] : [LtI32<-none, EqU32<-ACC(s0), GtI32<-ACC(s0)] + +WITH A PLANE And[P,A,B,C]: [GtI32<-Plane(0), EqU32<-Plane(0), LtI32<-Plane(0)] + And[P,C,B,A]: [LtI32<-Plane(0), EqU32<-Plane(0), GtI32<-Plane(0)] +``` + +Order changes which predicate runs first and nothing else; the skipped-word count is +order-independent. Plane-free conjunctions are unaffected — those preds take the `None` arm and +still chain on the accumulator, so **A1's measured lever survives exactly where the probe +measured it and nowhere else.** + +**Why it is the right trade anyway.** The alternative is preferring the accumulator, which drops +a conjunct when the accumulator crossed a nesting boundary — a wrong answer, not a slower one +(oracle 29, emitted 204). A lost optimisation beats a lost row. + +**Why no gate catches it.** `examples/adaptive_order_probe.rs`'s four scenarios are all +plane-free, so its table cannot measure this and does not claim to. Its +`accumulator_gated == terms.len() - 1` assertion is what keeps it honest and would fire +immediately on a planed scenario — which is why the honest move is to say "not measured" rather +than to add a scenario that would simply assert the inertness. + +**The way out, if it is ever worth taking:** gate on `plane ∧ accumulator` rather than choosing +between them. That is strictly narrower than either, so it restores the lever without +reintroducing the hole — at the cost of one extra op per gated predicate, which is exactly the +measurement nobody has made. Not attempted here; recorded so the option is not re-derived. + +Found by an independent correctness review of the fix, not by the fix's author. + +## ISS-QUACK-LOWER-FUSED-IS-SUPERLINEAR-AND-DEEP-FILTERS-ABORT (2026-09-15) — OPEN, both caller-controlled + +Two pre-execution costs on `lance-graph-quack`'s public surface, measured in **release**, both +reachable from ordinary caller input and both hit before a single row is touched. + +**1. `lower_fused` is ~cubic in leaf count, reachable from `Filter::in_u32`.** + +| | `lower` | `lower_fused` | +|---|---|---| +| `IN(64)` | 0.01 ms | 0.12 ms | +| `IN(256)` | 0.02 ms | 4.28 ms | +| `IN(1024)` | 0.03 ms | **192.70 ms** | +| `IN(2048)` | 0.13 ms | **1 443 ms** | +| flat `AND` w=4096 | — | **10 597 ms** | +| flat `AND` w=65 000 | **2.99 ms** | (not attempted) | + +~7.6× per doubling; `lower` is linear, the blow-up is entirely `lower_fused`. Mechanism sits in +`lance-graph-mask-risc/src/fuse.rs` — `Lowering::lower`'s reduction calls `leaf_count` on BOTH +children at every level, `leaf_count` calls `distinct_leaves`, and that uses `Vec::contains`, +O(k) per leaf; quack's `assign_slots` left-folds an n-ary junction into a depth-n binary +`BoolExpr`. Same class as the quadratic-`validate` stall `reference.rs:110-137` already records +and fixed on that side. Cheapest mitigation is a sorted set or `u64` bitset in `distinct_leaves`. +**Not fixed here** because it lives in a different crate and this PR is about the query surface — +widening it on my own is the thing the repo's own push rules forbid. + +**2. A deep `Filter` ABORTS the process; `LowerError::TooManySlots` is unreachable on the +in-place path.** Default 8 MiB stack, one process per depth: + +``` +depth=15000 -> LOWER OK slots=2 ops=30001 +depth=20000 -> fatal runtime error: stack overflow, aborting +``` + +Construction and `Drop` of the same tree survive to 30,000, so the recursion is +`gate_walk`/`emit_gated`'s, not the data structure's. It is an **abort**, not a catchable panic — +`catch_unwind` and `JoinHandle::join` cannot contain it. Meanwhile `emit_gated`'s +`dst.checked_add(1) → TooManySlots` needs depth 65,535, which the stack cannot reach: **the error +the crate defines for this condition can never fire on `lower`.** An explicit depth budget in +`gate_walk` returning `LowerError` would make the declared refusal real. + +Neither is a correctness defect and neither is reachable from the differential suite, which builds +trees to depth 6. Both were found by an independent correctness review. + ## ISS-FAMILY-IS-FOUR-WIDTHS-TWO-AT-OPPOSITE-ENDS (2026-09-15) — ⊘ RESOLVED SAME DAY: falsifier ran, hazard CONFIRMED (0 vs 3); see E-THE-TWO-FAMILY-NAMINGS-INVERT-AND-FROM-BE-BYTES-IS-THE-PLAUSIBLE-WRONG-JOIN-1 **"family" denotes four different things in this tree, and the two that share a width sit at diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 7a54e7c37..f04dc921b 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -76,6 +76,64 @@ census is measured and NOT yet on the board — see the arc entry's *Un-recorded* bullet. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1233. +## 2026-09-15 — #1235 round 2: an independent review re-found the P1, and my own fmt decline was WRONG + +An Opus correctness review ran against the branch while the fix was in flight and found the +same `emit_gated` defect independently — **2,021 wrong answers in 200,000 randomised trees** +against a per-row oracle that never builds a `Program`, smallest counterexample +`And([Not(Plane(2)), And([Cmp(Col(2), EqI32(39)), Plane(2)])])`. It then verified the fix +rather than taking it: 120,000 cases clean with the fix, **2,668 red** with only the pre-fix +gate line restored (the anti-vacuity control), plus 200,000-case fuzz, a 320,000-case sweep +across all eight `Agg`s, and 5,000 `GROUP BY` plans — all clean. Its verdict was HOLD on two +PROSE items, both now fixed. + +**I reversed my own review reply, and this is the uncomfortable half.** I declined CodeRabbit's +`cargo fmt --all` suggestion on the grounds that the scoped per-manifest line "already covers +everything `--all` would", and said I had measured it. CodeRabbit pushed back citing the +documented selection behaviour. **It was right and I was wrong**: mis-formatting +`mask-risc/src/ir.rs` and running the scoped quack line gives **exit 0** while mask-risc's own +line gives **exit 1** on the identical file. My earlier "measurement" must have been a +workspace-wide invocation misattributed to the scoped one — the same class of error as +measuring the wrong function. + +The CONCLUSION survives for a different reason than the one I gave: this fmt job is **per-crate +by design** (its own comments say so — `causal-edge` is workspace-excluded, so `--all` would +never reach it), and `mask-risc` has its own line directly above quack's. But checking the job +against the member list instead of reading it found the real defect the reviewer was circling: +**four workspace members had no rustfmt line at all** — `lance-graph-contract` (which THIS +branch edits), `lance-graph-callcenter`, `lance-graph-supervisor`, `bgz-tensor`. All four +measured clean and are now armed. A per-crate gate omits whatever nobody added a line for, and +nothing was checking that. + +**Fixed this round:** the three doc sites that still called `hoist_gate_subset`'s rotation a +"correctness requirement" and a "silent WRONG ANSWER" — true before the fix, false after, and +the reviewer disable-verified it (rotation removed, 120,000 cases, 65,919 planed, zero +divergences, on the harness ARM3 proves can see this regression). What the rotation still buys +is slot economy, 1 slot vs 2. Also the probe's `best/worst infx` print, which read +`spread 0.00 percentage points, best/worst infx` on the two regimes that skip nothing in any +order — self-contradictory, and those rows are cited in `and_by_skip`'s own doc table. + +**And a test I re-pinned this session was half-vacuous.** `assert!(!on_acc)` ran over both +lowerings, but `assign_slots` never emits a `Scratch` gate, so on the FUSED arm no input can +make it fail. It is now scoped to the in-place arm, where restoring the pre-fix line turns it +red, with the fused arm asserting the structural fact instead. One `assert!` over two arms +reads as twice the evidence and is once. + +**Recorded, not fixed** (`ISSUES.md`, two new entries): `and_by_skip`'s ordering lever is now +**inert on any conjunction carrying a plane** — the crate's own headline shape — which is a +genuine cost the P1 fix created and the right trade anyway (a lost optimisation beats a lost +row); and two caller-controlled pre-execution costs, `lower_fused` at ~7.6× per doubling +(`IN(2048)` = 1 443 ms against `lower`'s 0.13 ms, mechanism in another crate's +`distinct_leaves`) and a deep `Filter` **aborting the process** at depth 20,000, which also +makes `LowerError::TooManySlots` unreachable on the in-place path. + +**Also fixed:** the one review finding that never appeared inline. A §8a summary line said "the +signal is dead words rather than selectivity"; the probe permutes terms and measures skip and +never ranks by dead words against a selectivity ranking, so only the negative half is measured +(`selective` and `clustered` have near-identical survivor counts, 36 and 31, and differ by +94.24 points). The sort-key half is untested and §8a and §9 already said so — that one summary +line did not. + ## 2026-09-15 — PR #1235 review round: a codex P1 that was a real wrong answer, and three accepted doc/CI fixes Seven review threads on `claude/clone-repositories-71a5sw`. One was a genuine correctness diff --git a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md index 270eb2602..7e1732899 100644 --- a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md +++ b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md @@ -575,8 +575,14 @@ before any design: not to be needed. That is a cheap answer with a large consequence, and it is answered by measurement (1) plus one sweep.~~ **ANSWERED 2026-09-14 — see §8a.** Order moves it (up to 75 percentage points, 14.2×), so A1 is not - ELIMINATE; but the signal is dead words rather than selectivity and the - hill-climb is not ported. Note this did NOT need measurement (1): the skip + ELIMINATE; selectivity alone does **not** predict skip — `selective` and + `clustered` have near-identical survivor counts (36 and 31) and differ by + 94.24 points of as-written skip — and the hill-climb is not ported. + ⊘ This read "the signal is dead words rather than selectivity", which + overstates the probe: it permutes terms and measures skip, and never + ranks by dead words against a selectivity ranking, so the SORT-KEY half + is untested (§8a and §9 already say so; this summary line did not). + Note this did NOT need measurement (1): the skip fraction is a count of avoided word-evaluations, not a timing, so it was answerable on its own. diff --git a/.github/workflows/style.yml b/.github/workflows/style.yml index 9db518b97..4837ca08d 100644 --- a/.github/workflows/style.yml +++ b/.github/workflows/style.yml @@ -190,6 +190,22 @@ jobs: run: cargo fmt --manifest-path crates/lance-graph-planner/Cargo.toml -- --check - name: Rustfmt lance-graph-mask-risc run: cargo fmt --manifest-path crates/lance-graph-mask-risc/Cargo.toml -- --check + # The four workspace members that had NO rustfmt line at all. Found by + # measuring the job against the member list rather than reading it: this + # job is per-crate by design (`causal-edge` is workspace-excluded, so + # `--all` would never reach it), and a per-crate job silently omits any + # member nobody added a line for. `lance-graph-contract` is the one that + # matters here — this branch edits `recipes.rs` and nothing was gating it. + # All four verified rustfmt-clean on the pinned toolchain before arming, + # the same rule the block below this one records. + - name: Rustfmt lance-graph-contract (previously ungated) + run: cargo fmt --manifest-path crates/lance-graph-contract/Cargo.toml -- --check + - name: Rustfmt lance-graph-callcenter (previously ungated) + run: cargo fmt --manifest-path crates/lance-graph-callcenter/Cargo.toml -- --check + - name: Rustfmt lance-graph-supervisor (previously ungated) + run: cargo fmt --manifest-path crates/lance-graph-supervisor/Cargo.toml -- --check + - name: Rustfmt bgz-tensor (previously ungated) + run: cargo fmt --manifest-path crates/bgz-tensor/Cargo.toml -- --check - name: Rustfmt lance-graph-quack run: cargo fmt --manifest-path crates/lance-graph-quack/Cargo.toml -- --check - name: Rustfmt lance-graph-ontology (previously ungated) diff --git a/crates/lance-graph-quack/examples/adaptive_order_probe.rs b/crates/lance-graph-quack/examples/adaptive_order_probe.rs index 93f43fca7..6467b626a 100644 --- a/crates/lance-graph-quack/examples/adaptive_order_probe.rs +++ b/crates/lance-graph-quack/examples/adaptive_order_probe.rs @@ -379,15 +379,23 @@ fn main() { pct(worst.0) ); println!(" {:<12} {:>9} {:>8.2}%", "best", best.0, pct(best.0)); - println!( - " spread {:.2} percentage points, best/worst {:.2}x", - pct(best.0) - pct(worst.0), - if worst.0 == 0 { - f64::INFINITY - } else { - best.0 as f64 / worst.0 as f64 - } - ); + // 0/0 is not an infinite ratio, it is NO SPREAD — the `moderate` and + // `permissive` regimes skip nothing in ANY order, and printing `infx` + // beside `spread 0.00` made the line contradict itself. Those two rows + // are cited in `Query::and_by_skip`'s doc table, so the print is the + // evidence a reader sees. + match (best.0, worst.0) { + (0, 0) => println!(" spread 0.00 percentage points, no skip in any order"), + (_, 0) => println!( + " spread {:.2} percentage points, best/worst unbounded (worst skips 0)", + pct(best.0) - pct(worst.0) + ), + (_, w) => println!( + " spread {:.2} percentage points, best/worst {:.2}x", + pct(best.0) - pct(worst.0), + best.0 as f64 / w as f64 + ), + } println!(" best order: {}", best.2.join(" < ")); println!(); } diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index 4e2f112ed..8bafffbc3 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -363,13 +363,35 @@ impl Filter { /// on measured selectivity, so the comparison would have to be against /// that. /// + /// # This lever is INERT on a conjunction that carries a resident plane + /// + /// Measured, and it is the price of the `emit_gated` fix in `b7e6cef`: + /// once an `AND` carries a plane, every `Pred` inside it gates on that + /// same FIXED plane regardless of position, so the skipped-word count is + /// order-independent and this ordering buys exactly zero. Plane-free + /// conjunctions are unaffected — their preds still chain on the running + /// accumulator, which is where `adaptive_order_probe` measured the lever + /// and the only place it claims one. Since the crate's own headline shape + /// (`Filter::Plane(alpha)` = `SELECT … FROM t`) IS planed, that is most + /// real queries. Recorded as `ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE` + /// with the measurement and the way out. + /// /// # One caveat, because it is a real override /// /// When this `AND` also carries a resident plane that the gate walk /// DROPS as implied, a child whose result is a subset of that plane is - /// rotated to the front regardless of score. That rotation is a - /// correctness requirement, not a preference, so it wins. The ordering - /// here applies among the children the rotation leaves alone. + /// rotated to the front regardless of score, so the ordering here applies + /// among the children the rotation leaves alone. + /// + /// ⊘ That rotation was called "a correctness requirement, not a + /// preference" until 2026-09-15. It WAS one before the `emit_gated` fix; + /// afterwards the plane gates every child wherever it sits, so what the + /// rotation still buys is SLOT ECONOMY (a plane first makes the + /// accumulator a plane operand, costing no scratch slot: 1 vs 2 on + /// `alpha AND focus AND v < 50`). Disable-verified — rotation removed, + /// 120,000 differential cases, 65,919 of them planed, zero divergences, on + /// a harness proven able to see this class of regression by restoring the + /// pre-fix gate line and watching it fail 2,668 of the same cases. pub fn and_by_skip(parts: impl IntoIterator) -> Self { let mut scored: Vec<(u32, Filter)> = parts.into_iter().collect(); // `sort_by_key` is stable, so equal scores keep the caller's order @@ -666,8 +688,15 @@ enum Node { /// `flags[i]` is `gate_walk`'s "vanishes with the gate", which is exactly /// "this child's result is a subset of the gate". Putting such a child first /// makes the running accumulator a subset of the gate from the first fold -/// onward, and that is the precondition [`emit_gated`] needs before it may -/// narrow later comparisons onto the accumulator instead of onto the gate. +/// onward, which keeps the accumulator a plane operand and costs no scratch +/// slot. +/// +/// ⊘ This read "that is the precondition [`emit_gated`] needs before it may +/// narrow later comparisons onto the accumulator instead of onto the gate". +/// Since `b7e6cef` `emit_gated` does not narrow onto the accumulator at all +/// while a plane gate is live — the plane always wins — so the rotation is an +/// optimisation, not a precondition. See the disable run recorded on +/// [`Query::and_by_skip`]. /// /// Shared by BOTH `AND` arms of [`gate_walk`] — the already-gated one and the /// one that establishes a gate — because a first version rotated only in the @@ -747,8 +776,14 @@ fn gate_walk(f: &Filter, gate: Option) -> Result<(Node, bool), LowerError> // [`emit_gated`] narrow later comparisons onto the accumulator // without losing the gate. // - // Without the rotation this is a silent WRONG ANSWER, not a - // missed optimisation, and it was measured: on + // ⊘ This said "Without the rotation this is a silent WRONG + // ANSWER, not a missed optimisation". TRUE BEFORE `b7e6cef`, + // FALSE AFTER: the plane now gates every child wherever it + // sits, so the `AND` emits `⋂ children` where each flagged + // child is `plane & pred ⊆ plane`, and `⋂ children ⊆ plane` + // holds for ANY ordering. What survives is slot economy. + // The original measurement, kept because it is what the + // rotation was built from: // `alpha AND focus AND v < 50`, the foreign plane `focus` sat // first, the accumulator therefore started as `focus` (which is // NOT a subset of `alpha`), the comparison was gated on that @@ -1525,13 +1560,36 @@ mod tests { // `skip_ordering_moves_the_work_and_never_the_answer`, whose // conjunction carries no plane — those preds still gate on the // accumulator. - assert!( - !on_acc, - "a plane-gated comparison narrowed onto the accumulator \ - instead of the plane — the codex-P1 override is back: {:?}", - slice.ops - ); - let _ = is_inplace; + // + // ⊘ AND IT IS A REAL FALSIFIER ON ONE ARM ONLY. `assign_slots` + // never emits a `Scratch` gate at all, so on the FUSED arm no + // input can make this fail — it passes structurally, not because + // the property holds. The in-place arm is where it bites, and + // where restoring the pre-fix gate line turns it red. Asserting + // the two separately is the point: a single `assert!` run over + // both arms reads as twice the evidence and is once. + if is_inplace { + assert!( + !on_acc, + "a plane-gated comparison narrowed onto the accumulator \ + instead of the plane — the codex-P1 override is back: {:?}", + slice.ops + ); + } else { + assert!( + !slice.ops.iter().any(|op| matches!( + op, + MaskOp::Pred { + under: Some(Operand::Scratch(_)), + .. + } + )), + "the fused arm is expected to carry no scratch gate at \ + all; if that ever changes, the in-place assertion above \ + stops being the only place this is tested: {:?}", + slice.ops + ); + } let negated = lowering(&Query { filter: Filter::and([ From 8e5c3d708245d5ef1d174ea4e08cd9750f50113c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 18:13:15 +0000 Subject: [PATCH 16/25] issues: reproduce the lower_fused timings rather than relay them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I recorded the review's numbers as if they were measurements. Three of the six rows are now first-hand — IN(64) 97.20 us, IN(256) 3.08 ms, IN(1024) 158.47 ms against lower's 6.02/16.33/24.40 us — and agree with the review within ~25%, which is machine variance rather than a discrepancy. The three larger rows are marked review-only, not re-run. The mechanism is now verified in source instead of quoted: distinct_leaves (fuse.rs:72) is Vec::contains, O(k) per leaf; leaf_count (:87) allocates a fresh Vec and re-walks; :173 calls it on BOTH children at every level, which is what makes it superlinear rather than quadratic. Recording a relayed number as a measurement is the same defect as the comment that claimed an ISSUES.md entry which did not exist — one level over, in the entry written to fix it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/ISSUES.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 58fa2fceb..c9f15a889 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -45,16 +45,21 @@ reachable from ordinary caller input and both hit before a single row is touched **1. `lower_fused` is ~cubic in leaf count, reachable from `Filter::in_u32`.** -| | `lower` | `lower_fused` | -|---|---|---| -| `IN(64)` | 0.01 ms | 0.12 ms | -| `IN(256)` | 0.02 ms | 4.28 ms | -| `IN(1024)` | 0.03 ms | **192.70 ms** | -| `IN(2048)` | 0.13 ms | **1 443 ms** | -| flat `AND` w=4096 | — | **10 597 ms** | -| flat `AND` w=65 000 | **2.99 ms** | (not attempted) | - -~7.6× per doubling; `lower` is linear, the blow-up is entirely `lower_fused`. Mechanism sits in +| | `lower` | `lower_fused` | source | +|---|---|---|---| +| `IN(64)` | 6.02 µs | 97.20 µs | reproduced here (review: 0.12 ms) | +| `IN(256)` | 16.33 µs | 3.08 ms | reproduced here (review: 4.28 ms) | +| `IN(1024)` | 24.40 µs | **158.47 ms** | reproduced here (review: 192.70 ms) | +| `IN(2048)` | 0.13 ms | **1 443 ms** | review only, not re-run | +| flat `AND` w=4096 | — | **10 597 ms** | review only, not re-run | +| flat `AND` w=65 000 | **2.99 ms** | (not attempted) | review only, not re-run | + +The first three rows are first-hand, and they agree with the review within ~25 % (machine +variance, not a discrepancy). `lower` is linear across all of them; the blow-up is entirely +`lower_fused` — ×31.7 then ×51.5 for each 4× in leaves. Mechanism verified in source rather than +taken from the report: `distinct_leaves` (`fuse.rs:72`) is `out.contains(o)`, O(k) per leaf; +`leaf_count` (`:87`) allocates a fresh `Vec` and re-walks; `:173` calls it on **both** children +(`leaf_count(l) >= leaf_count(r)`) at every level. Mechanism sits in `lance-graph-mask-risc/src/fuse.rs` — `Lowering::lower`'s reduction calls `leaf_count` on BOTH children at every level, `leaf_count` calls `distinct_leaves`, and that uses `Vec::contains`, O(k) per leaf; quack's `assign_slots` left-folds an n-ary junction into a depth-n binary From 985d36aa6aa1b614286b3b89b95a0e879f6471be Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 18:17:02 +0000 Subject: [PATCH 17/25] issues: the workspace-build blocker has two remedies, and I had named one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The aws-smithy-json / aws-smithy-types incompatibility that reds linux-build, test, member-tests and test-with-coverage on main and every branch is not a resolver problem — no version selection fixes it. My stand-down comment on #1235 said the remedy "is a version pin on aws-smithy-types", which named one option as if it were the only one. The chain is feature-reachable. aws-config is optional in lance-io behind its "aws" feature; lance takes lance-io with default-features = false and re-exposes aws = ["lance-io/aws"] in its own default. Tested with cargo tree, not inferred: lance with its default minus "aws" makes aws-smithy-json report "did not match any packages" — gone from the graph, every other lance default feature retained. So the choice is pin a fifth coordinate outside the whitelist, or lose S3 object-store support. Both are operator decisions; the second leaves the pin rule untouched, which is why it belongs on the table. The S3 cost is stated precisely rather than waved at: no hard-coded AWS calls here, but lance-graph-catalog passes aws_access_key_id/secret/region through storage_options and DirNamespace parses s3:// URIs, so the capability is exposed to callers even though CI never exercises it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/ISSUES.md | 51 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index c9f15a889..a4fdaa465 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,3 +1,54 @@ +## ISS-AWS-SMITHY-BREAKS-THE-WORKSPACE-BUILD-AND-THERE-ARE-TWO-REMEDIES (2026-09-15) — OPEN, repo-wide, operator decision + +**`cargo build --workspace` fails on `main` and on every branch**, in a third-party crate, and +it is NOT a resolver problem — no version selection fixes it: + +``` +error[E0308] aws-smithy-json-0.63.0/src/codec/deserializer.rs:707 + expected `DocumentObject`, found `HashMap` +error[E0004] aws-smithy-json-0.63.0/src/serialize.rs:36 non-exhaustive patterns +``` + +`aws-smithy-types` 1.7.0 changed `Document::Object` to take a `DocumentObject` and marked +`Document` `#[non_exhaustive]`; `aws-smithy-json` 0.63.0 has not caught up. `aws-smithy-json` +declares `aws-smithy-types ^1.6.1`, so it ALWAYS resolves the breaking 1.7.0, and the newest +`aws-config` 1.12.0 requires `aws-smithy-json ^0.63.0` and cannot reach the fixed 0.64.0. This +repo tracks no `Cargo.lock`, so every CI run resolves fresh and picks the incompatible pair. +Jobs affected: `linux-build`, `test`, `member-tests`, `test-with-coverage` — all four die at the +same `cargo build --workspace` step, before any test body runs. + +**There are TWO remedies and both are operator decisions. Neither is a pin-vs-nothing choice.** + +| remedy | cost | touches the pin whitelist? | +|---|---|---| +| exact-pin `aws-smithy-types` | a FIFTH pinned coordinate | **yes** — `CLAUDE.md`: nothing outside lance/lancedb/arrow/datafusion is pinned at all | +| drop lance's `aws` feature | S3 object-store support | **no** | + +**The second one is the newly-established fact, and it is TESTED rather than inferred.** +`aws-config` is an OPTIONAL dependency of `lance-io` behind its `aws` feature; `lance` takes +`lance-io` with `default-features = false` and re-exposes `aws = ["lance-io/aws"]` inside its own +default. So the chain is feature-reachable from our side: + +```toml +lance = { version = "=11.0.0", default-features = false, features = [ + "azure", "gcp", "oss", "huggingface", "tencent", "tos", "goosefs", "geo", +] } +``` + +`cargo tree -i aws-smithy-json` then reports **"did not match any packages"** — gone from the +graph entirely, with every other lance default feature retained. (`default-features = false` +alone works too, but drops eight features instead of one.) + +**How real the S3 cost is, stated precisely.** No hard-coded AWS calls in this tree, but +`lance-graph-catalog`'s `storage_options` surface documents and passes through +`aws_access_key_id` / `aws_secret_access_key` / `aws_region`, and `DirNamespace` parses `s3://` +URIs. The capability is exposed to callers even though nothing here exercises it in CI. Whether +a deployment depends on it cannot be determined from the tree. + +**Correction recorded on purpose:** the stand-down comment on PR #1235 said the remedy "is a +version pin", naming one option as if it were the only one. That was incomplete, and the missing +option is the one that leaves the pin rule alone. Corrected on the PR. + ## ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE (2026-09-15) — OPEN, and it is the price of the P1 fix **`Query::and_by_skip`'s ordering lever buys exactly zero on any conjunction that carries a From b7e8251f8709effd541adfd7d30c4b1a74776198 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 18:19:02 +0000 Subject: [PATCH 18/25] board: tag the fenced blocks I added (MD040) CodeRabbit flagged one untagged fence in the new EPIPHANIES entry. Checking rather than fixing just the one found four: the three ISSUES blocks landed after its review commit, so it could not have seen them, and the next markdownlint run would have flagged them anyway. All four are console/pseudocode output, so all four take ```text. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/EPIPHANIES.md | 2 +- .claude/board/ISSUES.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index ed46b749d..cfd65586c 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -182,7 +182,7 @@ that is right and is the whole point (the accumulator is strictly narrower than it already absorbed). For a **nested** one it is wrong, because the accumulator belongs to the OUTER conjunction and has never seen the inner plane: -``` +```text P1 AND (Plane(focus) AND P2) ``` diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index a4fdaa465..6ecff9a4c 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -3,7 +3,7 @@ **`cargo build --workspace` fails on `main` and on every branch**, in a third-party crate, and it is NOT a resolver problem — no version selection fixes it: -``` +```text error[E0308] aws-smithy-json-0.63.0/src/codec/deserializer.rs:707 expected `DocumentObject`, found `HashMap` error[E0004] aws-smithy-json-0.63.0/src/serialize.rs:36 non-exhaustive patterns @@ -59,7 +59,7 @@ Created deliberately by the codex-P1 fix (`b7e6cef`, prefers the caller's plane over the accumulator, so every `Pred` inside a planed `AND` gates on the same FIXED plane regardless of position. Measured at HEAD, each entry `predicate <- its gate`: -``` +```text PLANE-FREE And[A,B,C] : [GtI32<-none, EqU32<-ACC(s0), LtI32<-ACC(s0)] And[C,B,A] : [LtI32<-none, EqU32<-ACC(s0), GtI32<-ACC(s0)] @@ -122,7 +122,7 @@ widening it on my own is the thing the repo's own push rules forbid. **2. A deep `Filter` ABORTS the process; `LowerError::TooManySlots` is unreachable on the in-place path.** Default 8 MiB stack, one process per depth: -``` +```text depth=15000 -> LOWER OK slots=2 ops=30001 depth=20000 -> fatal runtime error: stack overflow, aborting ``` From 5e954eba09e2fc8155b0af91c4593aa6d4322062 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 18:21:21 +0000 Subject: [PATCH 19/25] probe: document the two helpers whose behaviour is load-bearing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit's docstring-coverage check reads 75% against an 80% threshold. I measured what it is actually counting rather than padding to clear it: the public API is 100% documented — 23 of 24 non-test items in lib.rs, and the one gap is a `fmt` trait impl, which the trait documents. The whole shortfall is test-fixture helpers (`plane`, `new`, `n`, `i32_at`) and example-local helpers (`popcount`, `and_into`). Their names are their documentation; a `/// Returns the popcount` above `fn popcount` is noise, and writing fifteen of them to move a percentage is the box-ticking the falsifiability rule exists to discourage. Two are different, and they are documented here because a reader needs them, not because of the metric: - `lcg` is seeded from a constant on purpose. This probe's numbers are quoted in the translation matrix §8a and in `and_by_skip`'s doc table, so a non-reproducing run would silently invalidate a recorded measurement rather than fail loudly. - `permutations` is EXHAUSTIVE, not sampled, which is the whole reason the probe can report a true best and worst order instead of the best and worst it happened to try. Gates: fmt clean, clippy -D warnings clean, probe re-run and still reporting its four regimes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .../lance-graph-quack/examples/adaptive_order_probe.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/crates/lance-graph-quack/examples/adaptive_order_probe.rs b/crates/lance-graph-quack/examples/adaptive_order_probe.rs index 6467b626a..7579d73b4 100644 --- a/crates/lance-graph-quack/examples/adaptive_order_probe.rs +++ b/crates/lance-graph-quack/examples/adaptive_order_probe.rs @@ -55,6 +55,12 @@ struct Term { selectivity: f64, } +/// Knuth's LCG constants. Deterministic on purpose and seeded from a +/// constant: this probe's numbers are quoted in +/// `duckdb-to-v3-translation-matrix-v1.md` §8a and in `Query::and_by_skip`'s +/// doc table, so a run that did not reproduce them would silently invalidate +/// a recorded measurement rather than fail. The `>> 11` drops the low bits, +/// which are the weakest in an LCG. fn lcg(seed: &mut u64) -> u64 { *seed = seed .wrapping_mul(6364136223846793005) @@ -94,6 +100,10 @@ fn skipped_words(terms: &[&Term]) -> (usize, u32) { (skipped, popcount(&acc)) } +/// EXHAUSTIVE, not sampled — which is what lets the probe report a true +/// best and worst order rather than the best and worst it happened to try. +/// Factorial in the term count, so it is only viable because the fixtures +/// are deliberately small (4 terms = 24 orders). fn permutations<'a>(items: &[&'a Term]) -> Vec> { if items.len() <= 1 { return vec![items.to_vec()]; From 75c3f7b1aebc702768fc541ee1d35cfd8175ae61 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 19:09:19 +0000 Subject: [PATCH 20/25] =?UTF-8?q?deps:=20make=20the=20AWS=20SDK=20optional?= =?UTF-8?q?=20=E2=80=94=20the=20workspace=20build=20is=20green=20again?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `cargo check --workspace` now exits 0 in 2m56s. It had been red on main and on every branch since aws-smithy-types 1.7.0 published: aws-smithy-json 0.63.0 declares ^1.6.1 so it always resolves the breaking 1.7.0, and the newest aws-config requires ^0.63.0 and cannot reach the fixed 0.64.0. No version selection fixes it, and this repo tracks no lock, so every run re-resolves into the break. Made optional rather than pinned or deleted, per the operator: "make it optional so that later we fork 1.7 and fix it if we ever want it." lance takes default-features = false plus its own default list minus aws; lance-graph gains an opt-in aws-sdk = ["lance/aws"]. The capability stays addressable by name instead of vanishing. The cost I had recorded for this was wrong, and the operator caught it by asking whether this was the native AWS library rather than Tigris/Railway S3 slab hydration. It is the native SDK. lance-io's one aws feature bundles the SDK (aws-config) together with object_store/aws, the generic S3-COMPATIBLE backend — and only the first is dropped, because object_store with features = ["aws"] is declared directly by the workspace and by crates/lance-graph. Verified rather than reasoned: aws-smithy-json ABSENT aws-config ABSENT object_store feature "aws" ENABLED --features lance-graph/aws-sdk aws-config RETURNS That last line is the anti-vacuity check: a feature that cannot turn the thing back on would be decoration. lance-graph-hydrate is untouched — its slab hydration drives object_store with aws_endpoint + aws_virtual_hosted_style_request = false, an S3-compatible endpoint, and never names the SDK. Nothing in this workspace references aws_config, aws_sdk_*, or aws_credential_types at all. What is actually lost is AWS-native credential machinery: IMDS, SSO, STS assume-role. A past session had already built the insurance that makes this safe. crates/lance-graph/Cargo.toml:149 declares object_store/aws directly and says slimming lance's defaults "would silently remove S3 from THIS crate's own S3 callers ... it makes the capability this crate USES a thing this crate ASKS FOR." That is this move, anticipated. Checking upstream produced a root cause rather than a patch: lance-format/ lance-graph is green and has nothing to port — no aws-smithy pin or workaround anywhere in its tree. It is green because it tracks a Cargo.lock pinning the old compatible pair, which is the mechanism removed here in ISS-STALE-AUTHORITY-LOCKS-RESIDUE. The blocker is a symptom of that ruling. Scope: a repo-wide dependency change inside a feature PR. It is here because it unblocks this PR's CI; it splits out cleanly if a reviewer prefers. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/ISSUES.md | 40 +++++++++++++++++++++++++++++++ .claude/board/LATEST_STATE.md | 44 ++++++++++++++++++++++++++++++++++ Cargo.toml | 45 ++++++++++++++++++++++++++++++++++- crates/lance-graph/Cargo.toml | 13 ++++++++++ 4 files changed, 141 insertions(+), 1 deletion(-) diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 6ecff9a4c..57663be60 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,3 +1,43 @@ +## ISS-AWS-SMITHY-BREAKS-THE-WORKSPACE-BUILD-AND-THERE-ARE-TWO-REMEDIES (2026-09-15) — ⊘ RESOLVED SAME DAY, and the recorded cost was WRONG + +**Resolved by the operator's own framing — "make it optional so that later we fork 1.7 and fix +it if we ever want it"** — which is better than either remedy below, because it keeps the +capability addressable instead of deleting it. `lance` now takes `default-features = false` plus +its own default list MINUS `aws`, and `lance-graph` gains an opt-in `aws-sdk = ["lance/aws"]`. +**Measured after: `cargo check --workspace` EXIT=0 in 2m56s** — the job that had been red on +`main` and every branch. + +**⊘ The cost recorded below ("drop lance's `aws` feature → costs S3 object-store support") was +WRONG, and the operator caught it** by asking whether this was the native AWS library rather +than Tigris/Railway S3 slab hydration. It is the native SDK. `lance-io`'s `aws` feature bundles +TWO unrelated things — the AWS SDK (`aws-config`, broken) and `object_store/aws`, the generic +S3-COMPATIBLE backend — and only the first is dropped, because `object_store = { features = +["aws"] }` is declared DIRECTLY by the workspace and by `crates/lance-graph`. Verified: +`aws-smithy-json` ABSENT, `aws-config` ABSENT, `object_store` feature `aws` still ENABLED, and +`--features lance-graph/aws-sdk` brings `aws-config` back (so the switch is real, not +decoration). `lance-graph-hydrate`'s slab hydration is untouched — it drives `object_store` +with `aws_endpoint` + `aws_virtual_hosted_style_request = false`, an S3-compatible endpoint, +and never names the SDK. **Nothing in this workspace references `aws_config` / `aws_sdk_*` / +`aws_credential_types` at all.** What is actually lost: AWS-*native* credential machinery only +(IMDS, SSO, STS assume-role). + +**A past session had already built the insurance that makes this safe.** +`crates/lance-graph/Cargo.toml:149` declares `object_store/aws` directly and says why: +*"slimming them to `default-features = false` is a plausible future move — and it would silently +remove S3 from THIS crate's own S3 callers … it makes the capability this crate USES a thing +this crate ASKS FOR."* That is exactly this move, anticipated. + +**Upstream has no fix, and checking told us the root cause.** This repo is a fork of +`lance-format/lance-graph`, which is fully green — because it **tracks a `Cargo.lock`** pinning +`aws-smithy-json 0.61.5` / `aws-smithy-types 1.3.2`, and sits on `lance 1.0.1` / +`object_store 0.12.4`, ten majors behind. It carries no aws-smithy pin, patch or workaround; +grep finds the string nowhere in its tree. So there was nothing to port — its immunity is the +tracked lock we deliberately removed (`ISS-STALE-AUTHORITY-LOCKS-RESIDUE`, 2026-09-04), which +is why our CI re-resolves into newly-published breakage on every run. That is a THIRD remedy +(restore a tracked lock) and the only one that reverses a prior ruling; not taken. + +The original entry follows unchanged. + ## ISS-AWS-SMITHY-BREAKS-THE-WORKSPACE-BUILD-AND-THERE-ARE-TWO-REMEDIES (2026-09-15) — OPEN, repo-wide, operator decision **`cargo build --workspace` fails on `main` and on every branch**, in a third-party crate, and diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index f04dc921b..69f69b99a 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -25,6 +25,50 @@ result `n ∈ [8, 47]` is **predicted, not certified**), width on `ISS-BOUNDED-K-NEVER-FAILS-ON-ANY-GRAPH-TESTED`. - Arc entry: `PR_ARC_INVENTORY.md` under PR #1234. +## 2026-09-15 — the workspace build is GREEN again: the AWS SDK is now optional, and the cost I recorded for that was wrong + +`cargo check --workspace` **EXIT=0 in 2m56s** — the job that had been red on `main` and on every +branch since `aws-smithy-types 1.7.0` published. Not a pin, not a deletion: the operator's +framing, verbatim, was *"make it optional so that later we fork 1.7 and fix it if we ever want +it."* So `lance` takes `default-features = false` + its own default list minus `aws`, and +`lance-graph` gains `aws-sdk = ["lance/aws"]` — one documented switch instead of a silent +removal. + +**Two corrections came out of this, and both were prompted by the operator asking a question +rather than accepting my summary.** + +**1. "S3 support" was the wrong unit.** I had recorded the cost as "drops S3 object-store +support". The operator asked whether I meant the native AWS library, *"not to be confused with +Tigris RAILWAY S3 slab hydration"* — and that distinction is load-bearing, because `lance-io`'s +one `aws` feature bundles the AWS **SDK** (`aws-config`, the broken half) together with +`object_store/aws`, the generic **S3-compatible** backend. Only the first is dropped: the +workspace and `crates/lance-graph` declare `object_store = { features = ["aws"] }` directly, so +unification keeps it on. Measured — `aws-smithy-json` ABSENT, `aws-config` ABSENT, +`object_store` feature `aws` ENABLED, and `--features lance-graph/aws-sdk` brings `aws-config` +back, which is the anti-vacuity check that the switch is real. `lance-graph-hydrate`'s slab +hydration never touches the SDK; it drives `object_store` with `aws_endpoint` + +`aws_virtual_hosted_style_request = false`. Real cost: AWS-native credentials only — IMDS, SSO, +STS assume-role. + +**A past session had already paid for this.** `crates/lance-graph/Cargo.toml:149` declares +`object_store/aws` directly and explains that slimming lance's defaults *"would silently remove +S3 from THIS crate's own S3 callers … it makes the capability this crate USES a thing this crate +ASKS FOR."* That defensive declaration is the entire reason today's change is safe. Worth +noticing as a pattern: the comment cost one paragraph then and saved a capability now. + +**2. "Check upstream" produced a root cause, not a patch.** This repo is a fork of +`lance-format/lance-graph`. Upstream is fully green and has NOTHING to port — no aws-smithy pin, +patch or workaround anywhere in its tree. It is green because it **tracks a `Cargo.lock`** +(`aws-smithy-json 0.61.5` / `aws-smithy-types 1.3.2`) and sits on `lance 1.0.1` / +`object_store 0.12.4`, ten majors back. So its immunity is precisely the mechanism we removed in +`ISS-STALE-AUTHORITY-LOCKS-RESIDUE` (2026-09-04): with no tracked lock, every CI run re-resolves +and can pick up a crate published minutes earlier. **The blocker was a symptom of the +no-lock ruling**, which is a third remedy nobody has taken and the only one that reverses a +prior decision. + +Scope note, stated rather than buried: this is a repo-wide dependency change riding inside a +feature PR (#1235, quack). It is here because it is what unblocks that PR's CI and the operator +asked for it in that context; it splits out cleanly if a reviewer would rather see it alone. ## 2026-09-15 — PR #1233 merged (`030ad80`): eleven measurement arcs are on `main` — docs and probes only, NO contract inventory delta diff --git a/Cargo.toml b/Cargo.toml index 3eeb154ed..6152562cf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -260,7 +260,50 @@ resolver = "2" # deltalake was the only thing that pulled 53, and it is gone (0 hits in the # lock). [workspace.dependencies] -lance = "=11.0.0" +# `default-features = false` + lance's own default list MINUS `aws`. +# +# WHY: `lance/aws` -> `lance-io/aws` -> `dep:aws-config` -> `aws-smithy-json +# 0.63.0`, which does not compile against `aws-smithy-types 1.7.0` +# (`Document::Object` took a `DocumentObject`, `Document` went +# `#[non_exhaustive]`). 0.63.0 declares `aws-smithy-types ^1.6.1`, so it ALWAYS +# resolves the breaking 1.7.0, and the newest `aws-config` requires `^0.63.0` +# and cannot reach the fixed 0.64.0. No version selection fixes it; this repo +# tracks no lock, so every CI run re-resolves into the break. It is upstream's +# bug, reproducing on our `main` and on every branch. +# +# WHAT THIS DOES *NOT* COST -- measured, not assumed. `lance-io`'s `aws` +# feature bundles TWO unrelated things: the AWS *SDK* (`aws-config`, the broken +# half) and `object_store/aws`, the generic S3-COMPATIBLE backend. Only the +# first is dropped here: `object_store/aws` is declared DIRECTLY by this +# workspace and by `crates/lance-graph`, so cargo unification keeps it on. +# Verified with the change applied: `cargo tree -i aws-config` -> "did not +# match any packages", while `cargo tree -e features -i object_store` still +# lists feature "aws". So S3-compatible object storage is UNAFFECTED -- +# including `lance-graph-hydrate`'s slab hydration, which drives `object_store` +# with `aws_endpoint` + `aws_virtual_hosted_style_request = false` (an +# S3-compatible endpoint, not an AWS-native one) and never touches the SDK. +# Nothing in this workspace names `aws_config` / `aws_sdk_*` / +# `aws_credential_types` at all -- the SDK was pure transitive weight. +# +# WHAT IT DOES COST: AWS-*native* credential machinery only -- IMDS, SSO, +# STS assume-role -- which a deployment carrying explicit key/secret/endpoint +# does not use. +# +# TO RESTORE: `--features lance-graph/aws-sdk` (see that crate's [features]). +# If upstream is still broken when someone wants it, the escape hatch is a +# fork of `aws-smithy-json` fixed against `aws-smithy-types 1.7` -- the two +# call sites need `.into()` and one wildcard arm; upstream's own compiler +# hints spell both out. +lance = { version = "=11.0.0", default-features = false, features = [ + "azure", + "gcp", + "oss", + "huggingface", + "tencent", + "tos", + "goosefs", + "geo", +] } lance-linalg = "=11.0.0" lance-index = "=11.0.0" lancedb = { version = "=0.38.0", default-features = false } diff --git a/crates/lance-graph/Cargo.toml b/crates/lance-graph/Cargo.toml index 030e642a8..b3201d398 100644 --- a/crates/lance-graph/Cargo.toml +++ b/crates/lance-graph/Cargo.toml @@ -102,6 +102,19 @@ unity-catalog = ["lance-graph-catalog/unity-catalog"] # lancedb: high-level table/connection API (Python/JS SDK equivalent in Rust). # Enables LanceDB-compatible connection strings, table management, and the # vector search API surface that third-party tooling (DuckDB ext, Polars) expects. +# The AWS-NATIVE SDK path (IMDS / SSO / STS assume-role), OFF by default. +# +# Not a capability judgement -- it is off because `aws-smithy-json 0.63.0` does +# not compile against `aws-smithy-types 1.7.0` (full reasoning at the `lance` +# entry in the workspace `Cargo.toml`). S3-COMPATIBLE storage is NOT behind +# this flag: `object_store/aws` is declared directly below and stays on, so +# Tigris/Railway-style endpoints and `lance-graph-hydrate` are unaffected. +# +# Turning this on today re-introduces the broken build. It exists so the +# capability stays addressable by name instead of being silently deleted: +# when upstream fixes the pair -- or when someone forks `aws-smithy-json` +# against 1.7 -- this is the one switch to flip. +aws-sdk = ["lance/aws"] lancedb-sdk = ["dep:lancedb"] # ndarray-hpc: use AdaWorldAPI/ndarray for Fingerprint<256>, CAM-PQ, CLAM, BLAS, ZeckF64. # Disable for minimal builds (CI, wasm, embedded) — falls back to standalone ndarray_bridge.rs. From c2b4bc75ecc1a6906a1a8f3bafee3ace40e19dc7 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 19:25:11 +0000 Subject: [PATCH 21/25] ci: stop passing --all-features to the publish verification build `rust-publish.yml` passed `args: "--all-features"` to `katyo/publish-crates@v2`, which runs a verification build before publishing. That build fails, for two independent reasons, and neither could show up on a branch: aws-sdk ours, added in 947753d, opt-in by design because `aws-smithy-json 0.63.0` does not compile against `aws-smithy-types 1.7.0`. The flag exists so the capability stays addressable instead of being deleted -- but `--all-features` does not respect that intent, it turns on every declared feature. lancedb-sdk NOT ours, and older. `lancedb 0.38.0` declares `default = []`, gates `Error::Http` behind `#[cfg(feature = "remote")]` (src/error.rs:111), and leaves `pub mod job;` ungated (src/lib.rs:188) while job.rs uses `Error::Http` unconditionally at :56 and :66. The crate cannot compile without `remote`. The pin `lancedb = { version = "=0.38.0", default-features = false }` is on main at Cargo.toml:265 and 947753d never touched it, so `--all-features` has been broken since the lancedb 0.38 bump. Measured on this tree: cargo check --workspace EXIT 0 cargo check --workspace --all-targets EXIT 0 cargo check -p lance-graph --all-features EXIT 101 Replaced with an explicit list of the seven features that do build, verified by parsing both files and diffing the sets: `declared - passed == {aws-sdk, lancedb-sdk}` and `passed - declared == {}`. The drift that buys is named in ISS-PUBLISH-FEATURE-LIST-CAN-DRIFT rather than hidden; the upstream bug is ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE, with `features = ["remote"]` recorded as the candidate fix and deliberately not applied here (it adds reqwest to the graph, which wants its own measured PR). The board entry generalises both this and the reason the failure went unseen: a workflow that only runs on release is a deferred assertion, not a gate. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .github/workflows/rust-publish.yml | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/.github/workflows/rust-publish.yml b/.github/workflows/rust-publish.yml index 157a61570..f1b82d608 100644 --- a/.github/workflows/rust-publish.yml +++ b/.github/workflows/rust-publish.yml @@ -52,6 +52,30 @@ jobs: - uses: katyo/publish-crates@v2 with: registry-token: ${{ secrets.CARGO_REGISTRY_TOKEN }} - args: "--all-features" + # Every `lance-graph` feature EXCEPT the TWO that do not build. `--all-features` + # turns both on and fails the verification build this step runs before + # publishing. Measured on this tree: `cargo check -p lance-graph --all-features` + # exits 101; this list exits 0. + # + # aws-sdk — ours, opt-in by design. `aws-smithy-json 0.63.0` does not + # compile against `aws-smithy-types 1.7.0` (full reasoning at + # the `lance` entry in the workspace `Cargo.toml`). The feature + # exists so the capability stays addressable by name. + # lancedb-sdk — NOT ours, and it predates the `aws-sdk` flag: `lancedb 0.38.0` + # declares `default = []`, gates `Error::Http` behind + # `#[cfg(feature = "remote")]` (`src/error.rs:111`), but leaves + # `pub mod job;` ungated (`src/lib.rs:188`) while `job.rs` uses + # `Error::Http` unconditionally. The crate therefore cannot + # compile at all without `remote`. So `--all-features` has been + # broken here since the lancedb 0.38 bump — invisibly, because + # this is the only workflow that passes it and it never runs on + # push (`release: released` / `workflow_dispatch` only). + # See ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE. + # + # The cost of spelling the list out, stated rather than hidden: a feature added + # to `crates/lance-graph/Cargo.toml` later is NOT covered here, where + # `--all-features` would have picked it up for free. Add it in the same PR. + # See ISS-PUBLISH-FEATURE-LIST-CAN-DRIFT. + args: "--features unity-catalog,ndarray-hpc,with-cam-pq,planner,bgz17-codec,tensor-codec,lite-unified" path: crates/lance-graph dry-run: ${{ github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && github.event.inputs.mode == 'dry_run') }} From 99479a5829079b88d015a5723a3a0455ee7c8ff9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 19:27:12 +0000 Subject: [PATCH 22/25] board: the CI verdict you can see is not a verdict about the code you have EPIPHANIES -- E-A-CHECK-THAT-CANNOT-RUN-IS-INDISTINGUISHABLE-FROM-A-CHECK-THAT-PASSES-1. Two mechanisms, one shape, both live on this PR at the same moment: 1. The PR was conflicted (`mergeable_state: dirty`), so GitHub had no `refs/pull/1235/merge` to run the `pull_request` workflows against and produced ZERO runs for head 947753d. The page kept showing 36646a2's red beside five greens from non-`pull_request` workflows -- and not one of the seven described the head. 2. `rust-publish.yml` fires only on `release: released` / `workflow_dispatch`, so what it checks rots between releases with nothing to report it. That is how `--all-features` came to be broken since the lancedb 0.38 bump without a single red branch. ISSUES -- two new entries: ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE: lancedb 0.38.0 declares `default = []`, gates `Error::Http` behind `#[cfg(feature = "remote")]` (src/error.rs:111), and leaves `pub mod job;` ungated (src/lib.rs:188) while job.rs uses it at :56 and :66 -- so the crate cannot compile without `remote`. Blast radius measured: TWO unbuildable `lancedb-sdk` features (lance-graph, surreal_container), both off by default, which is why `cargo check --workspace` has always been EXIT 0. `features = ["remote"]` is recorded as the candidate fix and deliberately not applied -- it adds reqwest to the graph and wants its own measured PR. ISS-PUBLISH-FEATURE-LIST-CAN-DRIFT: the explicit list that replaced `--all-features` is verified complete once, by hand, not by a gate. The ~20-line CI check that would close it is named. Rebased onto d805c25 (main moved 2 commits). One conflict, LATEST_STATE.md, prepend-vs-prepend as before. Verified lossless three ways: union-of-headings 315/315 with 0 missing; every one of main's 28 added lines present (the 27-vs-28 line delta is a blank separator that coincided with the existing one, not a loss); and the append-only gate against the new merge-base reports 3724 -> 4076. The rebase changed no code -- `git diff cc7cac3 HEAD` is two board files. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/EPIPHANIES.md | 79 +++++++++++++++++++++++++++++++++++++ .claude/board/ISSUES.md | 73 ++++++++++++++++++++++++++++++++++ 2 files changed, 152 insertions(+) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index cfd65586c..5bfa98954 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,82 @@ +## 2026-09-15 (3) — E-A-CHECK-THAT-CANNOT-RUN-IS-INDISTINGUISHABLE-FROM-A-CHECK-THAT-PASSES-1 — "lots of CI errors" was stale red plus silence, and chasing it found `--all-features` broken since the lancedb 0.38 bump + +**Status:** FINDING. Two independent mechanisms, one shape. Both measured on PR #1235. +**Confidence:** HIGH — every claim below is an API read, a `cargo` exit code, or a line +number in a vendored crate. All re-runnable. + +### The shape + +A verdict you can see is not a verdict about the code you have. Two ways that breaks, and +both were live in this repo at the same moment: + +1. **The workflow could not run** — the PR was conflicted, so GitHub had no merge ref to + run `pull_request` workflows against. The last runnable SHA's red stayed on the page. +2. **The workflow does not run on push** — `rust-publish.yml` fires only on + `release: released` / `workflow_dispatch`, so what it checks rots between releases with + nothing to report it. + +Case 1 shows you a stale answer. Case 2 shows you no answer and you read it as "fine". +Neither is distinguishable from green by looking. + +### Mechanism 1 — a conflicted PR produces NO run, and the UI does not say so + +| what | measured | +|---|---| +| PR #1235 head | `947753d` (the aws-smithy fix) | +| workflow runs for `947753d` | **0** — none, in any state | +| newest runs on the branch | `36646a2`, the commit BEFORE the fix | +| PR `mergeable` / `mergeable_state` | `false` / **`dirty`** | + +`build.yml`, `rust-test.yml` and `style.yml` trigger on `pull_request`, which GitHub runs +against `refs/pull//merge` — a ref it can only synthesise when the PR merges cleanly. +A conflicted PR therefore produces no run at all. Five non-`pull_request` workflows +(`Supersession index`, `Append-only gate`, …) DID run and DID pass, so the page showed +five greens beside two stale reds and **not one of the seven described the head**. + +**The check, before believing any red on a PR:** compare the failing run's `head_sha` +against the PR's `head.sha`, and read `mergeable_state`. `dirty` means CI is silent, not +failing, and the remedy is to merge or rebase the base in — not to debug the code. + +### Mechanism 2 — `--all-features` was already broken, and only a release would have said so + +Chasing mechanism 1 turned up a second one by accident. `.github/workflows/rust-publish.yml` +passes `args: "--all-features"` to `katyo/publish-crates@v2`, which runs a verification +build before publishing. Measured on this tree: + +| invocation | exit | +|---|---| +| `cargo check --workspace` | 0 | +| `cargo check --workspace --all-targets` | 0 | +| `cargo check -p lance-graph --all-features` | **101** | + +TWO independent causes, and the interesting one is not ours: + +- **`aws-sdk`** — ours, added this session, opt-in by design because + `aws-smithy-json 0.63.0` does not build against `aws-smithy-types 1.7.0`. +- **`lancedb-sdk`** — **not ours, and older.** `lancedb 0.38.0` declares `default = []`, + gates `Error::Http` behind `#[cfg(feature = "remote")]` (`src/error.rs:111`), but leaves + `pub mod job;` ungated (`src/lib.rs:188`) while `job.rs` uses `Error::Http` + unconditionally at `:56` and `:66`. **That crate cannot compile without `remote`.** The + workspace pin `lancedb = { version = "=0.38.0", default-features = false }` sits on + `main` at line 265 and was untouched by `947753d`. + +So `--all-features` has been failing since the lancedb 0.38 bump (#1190), and **no branch +could ever have gone red for it**, because the only call site is a workflow that never +runs on push. The `aws-sdk` flag did not create this; it added a second reason to the +same silent failure. + +### What generalizes + +**Adding a feature you know to be broken is not a local act** — every `--all-features` +call site becomes a caller of it. Grep before landing one. Here there were two: +`rust-publish.yml` (fixed) and `style.yml:95`, which is scoped to `lance-graph-quack` — +one path dep, zero declared features — so it cannot reach `lance` or `lancedb`. + +**And the stronger one:** a workflow that only runs on release is not a gate, it is a +deferred assertion. Its failure is dated to whenever someone next cuts a release and will +be attributed to whatever PR happens to be adjacent. If a check matters, it has to run on +a cadence where its failure is still traceable to a cause. + ## 2026-09-15 — E-I-GRAFTED-HELIX-ONTO-HEXAGON-AND-THEN-DEPRECATED-THE-OPERATORS-TENANTS-ON-MY-OWN-AUTHORITY-1 — there is no residue in Hexagon; the guard that refused the graft was recorded as a missing feature; and then I called two of the operator's shipped tenants dead **Status:** CORRECTION. Operator-caught, same day, hours after diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 57663be60..621b1476d 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,3 +1,76 @@ +## ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE (2026-09-15) — OPEN, upstream bug, our `lancedb-sdk` feature does not build + +`lancedb 0.38.0` does not compile with its own default feature set. Measured, reading the +vendored crate: + +| site | fact | +|---|---| +| `Cargo.toml` | `default = []` — so `default-features = false` on our side is a **no-op** | +| `src/error.rs:111` | `Error::Http` is `#[cfg(feature = "remote")]` | +| `src/lib.rs:188` | `pub mod job;` — **not** gated | +| `src/job.rs:56,66` | uses `Error::Http` unconditionally | + +Result: `error[E0599]: no variant named 'Http' found for enum 'error::Error'`, twice, +inside lancedb itself. Reproduce with +`cargo check -p lance-graph --features lancedb-sdk`. + +**Not ours and not new.** `lancedb = { version = "=0.38.0", default-features = false }` is +on `main` (Cargo.toml:265), landed with the lance-11 / lancedb-0.38 bump (#1190). Nothing +caught it because `lancedb-sdk` is optional, off by default, enabled by no workspace +member, and reachable in CI only through `rust-publish.yml`'s `--all-features` — a +workflow that never runs on push. See +`E-A-CHECK-THAT-CANNOT-RUN-IS-INDISTINGUISHABLE-FROM-A-CHECK-THAT-PASSES-1`. + +**Candidate fix, NOT applied here:** add `features = ["remote"]` to the workspace `lancedb` +entry. `remote = ["dep:reqwest", "dep:http", "dep:urlencoding", "lance-namespace-impls/rest", +"lance-namespace-impls/rest-adapter"]` — no aws, so it does not reintroduce the +`aws-smithy` breakage. Deliberately deferred: it adds `reqwest` to anyone enabling +`lancedb-sdk`, which is a dependency-graph decision that wants its own measured PR rather +than a tail-end change in a PR about something else. The alternative — report it upstream +and pin the fixed patch — is also open. + +**Blast radius, measured** — every crate that can reach the crate, and whether anything +turns it on: + +| crate | declaration | default-on? | +|---|---|---| +| `lance-graph` | `lancedb-sdk = ["dep:lancedb"]` | no (`default` at :87 omits it) | +| `surreal_container` | `lancedb-sdk = ["dep:lancedb"]` | no (`default = []`) | +| `holograph` | a feature literally NAMED `lancedb` — but it maps to `["dep:lance"]`, i.e. it does **not** pull the lancedb crate | n/a | + +So **two** unbuildable `lancedb-sdk` features exist, not one, and a fix must clear both. +Nothing enables either, which is why `cargo check --workspace` is EXIT 0 and always has +been. The `holograph` entry is a naming trap worth knowing about separately: a feature +called `lancedb` that has nothing to do with lancedb. + +Until then `lancedb-sdk` is a declared-but-unbuildable feature, the same status +`aws-sdk` carries, and both are excluded from the publish verification list. + +## ISS-PUBLISH-FEATURE-LIST-CAN-DRIFT (2026-09-15) — OPEN, low severity + +`.github/workflows/rust-publish.yml` no longer passes `--all-features`; it passes an +explicit list of every `lance-graph` feature except the two that do not build — +`aws-sdk` (ISS-AWS-SMITHY-…) and `lancedb-sdk` (ISS-LANCEDB-038-NEEDS-REMOTE-TO-COMPILE). +That was forced: the publish step runs a verification build, and `--all-features` enables +both. Measured — `cargo check -p lance-graph --all-features` exits **101**; the explicit +list exits **0**. + +**The residue:** a feature added to `crates/lance-graph/Cargo.toml` later is silently NOT +covered by the publish verification, where `--all-features` would have picked it up for +free. The list was verified complete at landing by parsing both files and diffing the sets +(`declared - passed == {aws-sdk, lancedb-sdk}`, `passed - declared == {}`) — but that was a +one-off, not a gate. + +**The real fix, ~20 lines, not done:** a CI check that parses `[features]` out of +`crates/lance-graph/Cargo.toml`, parses the `args:` line out of `rust-publish.yml`, and +fails when the difference is anything other than the two known-broken names. Same shape as +the existing `append_only_gate.py` / `supersession_index.py` gates, and it would also make +the two exclusions expire loudly instead of silently outliving their cause. + +Severity is low: the publish workflow runs only on `release: released` / +`workflow_dispatch`, and a drift there costs one failed release job, not a bad artifact — +the verification build failing is precisely what *stops* the publish. + ## ISS-AWS-SMITHY-BREAKS-THE-WORKSPACE-BUILD-AND-THERE-ARE-TWO-REMEDIES (2026-09-15) — ⊘ RESOLVED SAME DAY, and the recorded cost was WRONG **Resolved by the operator's own framing — "make it optional so that later we fork 1.7 and fix From 188d6b34cb1307c80a0a513290ca294e93d8f446 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 19:31:28 +0000 Subject: [PATCH 23/25] board: address four citations by section, not by line number The `Citations have not decayed` gate went red on head 99479a5 with one new decay, and it was self-inflicted: the `aws-sdk` feature block added in 75c3f7b pushed `crates/lance-graph/Cargo.toml`'s dev-dependency `object_store` comment from line 149 to 164, so every citation pointing at ":149 declares object_store/aws" stopped resolving. The gate's own advice is explicit and worth obeying rather than working around: THE FIX IS NOT TO CORRECT THE LINE NUMBER. A number is a coordinate in a moving frame; a heading or a D-id is an address. So all four sites now address the target by section and key: ISSUES.md the dev-dependency `object_store` entry (was Cargo.toml:149) ISSUES.md the `lancedb` key in [workspace.dependencies] (was Cargo.toml:265) ISSUES.md "its `default` list omits it" (was ":87 omits it") EPIPHANIES.md the `lancedb` key in [workspace.dependencies] (was "line 265") LATEST_STATE.md the dev-dependency `object_store` entry (was Cargo.toml:149) Running the gate LOCALLY is what made this complete. CI reported one site (ISSUES.md:98); `citation_decay.py --since origin/main` reported a second the CI output had not surfaced (LATEST_STATE.md:53) -- the same sentence, duplicated into two board files in an earlier commit. Fixing only what CI named would have shipped red again. Deliberately kept: the `src/error.rs:111` / `src/lib.rs:188` references into lancedb 0.38.0. That is a registry crate at an exact pin, so those line numbers are immutable by construction -- the failure mode this commit fixes cannot reach them. After: `citation_decay.py --since origin/main` reports 0 new decay(s), 145 pre-existing (backlog, untouched), EXIT 0. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/EPIPHANIES.md | 5 +++-- .claude/board/ISSUES.md | 8 +++++--- .claude/board/LATEST_STATE.md | 5 +++-- 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 5bfa98954..df90e82d6 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -57,8 +57,9 @@ TWO independent causes, and the interesting one is not ours: gates `Error::Http` behind `#[cfg(feature = "remote")]` (`src/error.rs:111`), but leaves `pub mod job;` ungated (`src/lib.rs:188`) while `job.rs` uses `Error::Http` unconditionally at `:56` and `:66`. **That crate cannot compile without `remote`.** The - workspace pin `lancedb = { version = "=0.38.0", default-features = false }` sits on - `main` at line 265 and was untouched by `947753d`. + workspace pin `lancedb = { version = "=0.38.0", default-features = false }` — the + `lancedb` key in the root `Cargo.toml`'s `[workspace.dependencies]` — is on `main` + unchanged and was untouched by `947753d`. So `--all-features` has been failing since the lancedb 0.38 bump (#1190), and **no branch could ever have gone red for it**, because the only call site is a workflow that never diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 621b1476d..8ac9f1df9 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -15,7 +15,8 @@ inside lancedb itself. Reproduce with `cargo check -p lance-graph --features lancedb-sdk`. **Not ours and not new.** `lancedb = { version = "=0.38.0", default-features = false }` is -on `main` (Cargo.toml:265), landed with the lance-11 / lancedb-0.38 bump (#1190). Nothing +on `main` (the `lancedb` key in the root `Cargo.toml`'s `[workspace.dependencies]`), +landed with the lance-11 / lancedb-0.38 bump (#1190). Nothing caught it because `lancedb-sdk` is optional, off by default, enabled by no workspace member, and reachable in CI only through `rust-publish.yml`'s `--all-features` — a workflow that never runs on push. See @@ -34,7 +35,7 @@ turns it on: | crate | declaration | default-on? | |---|---|---| -| `lance-graph` | `lancedb-sdk = ["dep:lancedb"]` | no (`default` at :87 omits it) | +| `lance-graph` | `lancedb-sdk = ["dep:lancedb"]` | no — its `default` list omits it | | `surreal_container` | `lancedb-sdk = ["dep:lancedb"]` | no (`default = []`) | | `holograph` | a feature literally NAMED `lancedb` — but it maps to `["dep:lance"]`, i.e. it does **not** pull the lancedb crate | n/a | @@ -95,7 +96,8 @@ and never names the SDK. **Nothing in this workspace references `aws_config` / ` (IMDS, SSO, STS assume-role). **A past session had already built the insurance that makes this safe.** -`crates/lance-graph/Cargo.toml:149` declares `object_store/aws` directly and says why: +`crates/lance-graph/Cargo.toml`'s `[dev-dependencies]` `object_store` entry declares +`object_store/aws` directly and says why: *"slimming them to `default-features = false` is a plausible future move — and it would silently remove S3 from THIS crate's own S3 callers … it makes the capability this crate USES a thing this crate ASKS FOR."* That is exactly this move, anticipated. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 69f69b99a..508d9169c 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -50,8 +50,9 @@ hydration never touches the SDK; it drives `object_store` with `aws_endpoint` + `aws_virtual_hosted_style_request = false`. Real cost: AWS-native credentials only — IMDS, SSO, STS assume-role. -**A past session had already paid for this.** `crates/lance-graph/Cargo.toml:149` declares -`object_store/aws` directly and explains that slimming lance's defaults *"would silently remove +**A past session had already paid for this.** The `object_store` entry in +`crates/lance-graph/Cargo.toml`'s `[dev-dependencies]` declares `object_store/aws` +directly and explains that slimming lance's defaults *"would silently remove S3 from THIS crate's own S3 callers … it makes the capability this crate USES a thing this crate ASKS FOR."* That defensive declaration is the entire reason today's change is safe. Worth noticing as a pattern: the comment cost one paragraph then and saved a capability now. From e43d12fcdf85ff2b903a67442667a359cc7612e1 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 19:47:53 +0000 Subject: [PATCH 24/25] review: six verified findings from the CodeRabbit pass on 947753d/99479a5 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each was checked against the code before being accepted; two of the eleven reported are deferred and one is unverified (see below). [1] duckdb-to-v3 matrix:548,577 -- "up to 75 percentage points, 14.2x" is the SELECTIVE regime. The §8a table's own rows show clustered spreading 0.00% -> 99.90%, i.e. 99.90 points, which is the real maximum. Both summary lines now say so and attribute 75.00/14.2x to selective. [2] duckdb-to-v3 matrix:692 -- still asserted hoist_gate_subset is "a correctness requirement rather than a preference". lib.rs:386-394 already carries the ⊘ retraction: after the emit_gated fix the plane gates every child wherever it sits, so the rotation buys slot economy only. The plan was contradicting the code it specifies; corrected with the same ⊘ note. [3] and_by_skip is an inherent method of `Filter` (lib.rs:395, inside `impl Filter` at :225), not of `Query`. lib.rs:699 and ISSUES.md both named the wrong owning type -- while lib.rs:112/114 already said `Filter::`, so the file disagreed with itself and the intra-doc link did not resolve. [4] LowerError is public and had Display but not Error, so a caller could not `?` it into Box and no wrapper could surface it via source(). Added. The reviewer cited a sibling precedent; mask-risc does NOT have one (its own FuseError/ExecError lack it too), but the workspace does -- eight `impl std::error::Error for X {}` sites across cognitive-shader-driver, elixir-template, lance-graph-callcenter and others. Matched those. [5] EPIPHANIES cited `EPIPHANIES:19221` by LINE. Verified decayed: line 19221 today is style-table content, nothing to do with cam96. Replaced with the entry name `E-CAM96-DISTRIBUTION-MEASURED-1`, which is the convention this same file already states at line 120. [9] My own entry from 99479a5 described rust-publish.yml as PRESENTLY passing --all-features, when c2b4bc7 in this same PR had already replaced it -- the entry documented a state the PR removed. Rewritten as historical with the fix commit named. Its "no branch could ever have gone red" was also too strong: workflow_dispatch can be aimed at any branch, so the failure was reachable on demand, just never by the push/PR cadence that makes a failure traceable. Narrowed to "no push-triggered run", with the ⊘ note. DEFERRED, both real: [10] gate_walk/emit_gated recurse with no depth bound, so a deep enough public `Filter` exhausts the stack before dst.checked_add(1) can return TooManySlots -- an abort where the API contract promises an error. A depth budget is the fix; it wants its own falsifier (a filter past the cap must return Err, not die) rather than a tail-end patch. [11] Excluding aws-sdk and lancedb-sdk from the publish verification means the release job can now SUCCEED and ship a manifest declaring two features that do not compile for anyone who enables them -- where before it would have failed loudly. lance-graph is published (0.5.4, 89,276 downloads), so this is a real consumer-facing trap, not hypothetical. lancedb-sdk is fixable (workspace lancedb needs features = ["remote"]); aws-sdk is not, and removing it from the manifest contradicts the explicit instruction to keep the capability addressable. That is an owner decision, not mine. NOT VERIFIED: the outside-diff finding that the cam96 passage must not rank HelixResidue. It is pre-existing content and the claim is about what a past measurement showed; asserting either way without reading that measurement would be the exact error [9] just corrected. Gates: quack 14/14, clippy --all-targets --all-features -D warnings clean, fmt clean, citation-decay 0 new, append-only 9 files none shrank. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/EPIPHANIES.md | 21 ++++++++++++------- .claude/board/ISSUES.md | 2 +- .../duckdb-to-v3-translation-matrix-v1.md | 11 ++++++---- crates/lance-graph-quack/src/lib.rs | 9 +++++++- 4 files changed, 29 insertions(+), 14 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index df90e82d6..06b166489 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -39,9 +39,10 @@ failing, and the remedy is to merge or rebase the base in — not to debug the c ### Mechanism 2 — `--all-features` was already broken, and only a release would have said so -Chasing mechanism 1 turned up a second one by accident. `.github/workflows/rust-publish.yml` -passes `args: "--all-features"` to `katyo/publish-crates@v2`, which runs a verification -build before publishing. Measured on this tree: +Chasing mechanism 1 turned up a second one. `.github/workflows/rust-publish.yml` +PASSED `args: "--all-features"` to `katyo/publish-crates@v2`, which runs a verification +build before publishing; `c2b4bc7` in this same PR replaced it with an explicit list. +Measured on the tree before that fix: | invocation | exit | |---|---| @@ -61,10 +62,14 @@ TWO independent causes, and the interesting one is not ours: `lancedb` key in the root `Cargo.toml`'s `[workspace.dependencies]` — is on `main` unchanged and was untouched by `947753d`. -So `--all-features` has been failing since the lancedb 0.38 bump (#1190), and **no branch -could ever have gone red for it**, because the only call site is a workflow that never -runs on push. The `aws-sdk` flag did not create this; it added a second reason to the -same silent failure. +So `--all-features` has been failing since the lancedb 0.38 bump (#1190), and **no +push-triggered run could ever have gone red for it**, because the only call site is a +workflow that fires on `release: released` / `workflow_dispatch` only. ⊘ An earlier +draft of this line said *no branch* could have gone red — too strong: +`workflow_dispatch` can be aimed at any branch, so the failure was reachable on +demand, just never by the ordinary push/PR cadence that makes a failure traceable to +a cause. The `aws-sdk` flag did not create this; it added a second reason to the same +silent failure. ### What generalizes @@ -399,7 +404,7 @@ already running. > Hexagon was tested exhaustively and never involved one. The real learning > surface is seam 6 of this session's own inventory, already hexagon-shaped: > `FrozenStyle`/`LearnedStyle`/`ExploreStyle`, `U8×12` = **6 × 2 × palette256** each — -> the shape the operator ruled and this board MEASURED (`EPIPHANIES:19221`, ρ_all 0.966, +> the shape the operator ruled and this board MEASURED (`E-CAM96-DISTRIBUTION-MEASURED-1`, ρ_all 0.966, > near-orth 170×) as better than the 48-bit class `HelixResidue` belongs to — with the > shipped held-out promote gate already on it. Full storno: > `E-I-GRAFTED-HELIX-ONTO-HEXAGON-AND-THEN-DEPRECATED-THE-OPERATORS-TENANTS-ON-MY-OWN-AUTHORITY-1`. diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 8ac9f1df9..dc6a214e8 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -166,7 +166,7 @@ option is the one that leaves the pin rule alone. Corrected on the PR. ## ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE (2026-09-15) — OPEN, and it is the price of the P1 fix -**`Query::and_by_skip`'s ordering lever buys exactly zero on any conjunction that carries a +**`Filter::and_by_skip`'s ordering lever buys exactly zero on any conjunction that carries a resident plane — which is this crate's own headline query shape.** Created deliberately by the codex-P1 fix (`b7e6cef`, diff --git a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md index 7e1732899..9c912529b 100644 --- a/.claude/plans/duckdb-to-v3-translation-matrix-v1.md +++ b/.claude/plans/duckdb-to-v3-translation-matrix-v1.md @@ -545,7 +545,8 @@ gate Phase 2. > **⊘ A1 UPDATE 2026-09-14 — the falsifier ran (§8a) and moved the row to > ADAPT, which strengthens the shape above rather than dulling it.** Term order -> does change the skip fraction (up to 75 percentage points, 14.2×), so the +> does change the skip fraction (up to 99.90 percentage points in the clustered +> regime; 75.00 points and 14.2× are the SELECTIVE regime's spread), so the > mechanism is not deleted — but what transferred is the INTENT (order the > conjuncts) and what did not is the CARRIER (an adjacent-transposition > hill-climb over measured runtimes). That is the same intent-transfers / @@ -574,7 +575,8 @@ before any design: `AdaptiveFilter` is ELIMINATE and the only DuckDB mechanism V3 lacks turns out not to be needed. That is a cheap answer with a large consequence, and it is answered by measurement (1) plus one sweep.~~ **ANSWERED 2026-09-14 — see - §8a.** Order moves it (up to 75 percentage points, 14.2×), so A1 is not + §8a.** Order moves it (up to 99.90 percentage points, clustered — 75.00 + points and 14.2× are the selective regime), so A1 is not ELIMINATE; selectivity alone does **not** predict skip — `selective` and `clustered` have near-identical survivor counts (36 and 31) and differ by 94.24 points of as-written skip — and the hill-climb is not ported. @@ -689,8 +691,9 @@ No measurement, no decay, no intervals, no `observe=10 / execute=20 / warmup=5`. The crate builds programs and never evaluates one, so it cannot measure anything; putting the score at the boundary is the whole adaptation. -**One override, because it is a correctness requirement rather than a -preference:** when the `AND` also carries a resident plane the gate walk DROPS +**One override — and since the `emit_gated` fix it buys SLOT ECONOMY, not +correctness (⊘ corrected 2026-09-15; it WAS a correctness requirement before +that fix — see the ⊘ note on `hoist_gate_subset` in `lib.rs`):** when the `AND` also carries a resident plane the gate walk DROPS as implied, a child whose result is a subset of that plane is rotated to the front regardless of score (`hoist_gate_subset`). The ordering applies among the children that rotation leaves alone. diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index 8bafffbc3..f725a9aee 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -506,6 +506,13 @@ impl core::fmt::Display for LowerError { } } +/// `LowerError` is public, so it must be usable as an error: without this a +/// caller cannot `?` it into `Box` and no wrapper can surface it +/// through `source()`. Empty body — the variants carry no nested cause — which +/// is the shape eight sibling crates already use (`cognitive-shader-driver`, +/// `elixir-template`, `lance-graph-callcenter`, …). +impl std::error::Error for LowerError {} + /// What a query asks for. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Agg { @@ -696,7 +703,7 @@ enum Node { /// Since `b7e6cef` `emit_gated` does not narrow onto the accumulator at all /// while a plane gate is live — the plane always wins — so the rotation is an /// optimisation, not a precondition. See the disable run recorded on -/// [`Query::and_by_skip`]. +/// [`Filter::and_by_skip`]. /// /// Shared by BOTH `AND` arms of [`gate_walk`] — the already-gated one and the /// one that establishes a gate — because a first version rotated only in the From d77cd4ea8d0b8eba419a943e1c9ec2d2d4fff549 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 20:05:39 +0000 Subject: [PATCH 25/25] quack: say in the crate doc that fusing forfeits the skip MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two sessions independently derived this from `assign_slots` because the doc did not say it, which is the argument for writing it down rather than leaving it in a board entry. `lower` chains: the first predicate is ungated, each later one gates on the running accumulator, so a conjunct narrows what its successors read. `lower_fused` gates each predicate on the caller's resident plane ONLY (`under.map(Operand::Plane)`) and never on an accumulator, because the Boolean combination is deferred to the fuser -- and `MaskOp::Ternlog` has no `under` field at all, where `MaskOp::Pred` does. So a fused program has no progressive narrowing and term order cannot move its skipped-word count. The saving is physical, not bookkeeping: ndarray's `pack_under` (simd_masking_ops.rs:1541-1546) is `if gate == 0 { continue }` BEFORE the 64 values are loaded. That `continue` is the whole of what §8a measures. Consequence worth the doc line: ordering is worth up to 99.90 percentage points of skipped words on a clustered conjunction under `lower`, and exactly zero under `lower_fused` on the same query. `Filter::and_by_skip`'s lever is alive in one configuration -- gated lowering, plane-free conjunction, contiguous survivors. `adaptive_order_probe.rs` has no fused arm, so its table cannot see this and does not claim to. The board entry additionally records two things the matrix did not: - A1 fails in TWO places. The SEED (GetInitialOrder, static selectivity) is killed by §8a independently of the executor: selective (36 survivors) and clustered (31) are indistinguishable by density and 19.24 points apart best-vs-best, so a selectivity seed cannot separate the two regimes where the lever exists. The LOOP (swap/measure-runtime/revert/halve) would find contiguity -- and is the half quack structurally cannot run. The half that would work cannot run; the half that can run measures the wrong thing. - A ⊘ on my own under-qualified citation of D-GTM-0m: I quoted "22.4-22.8 µs, survivor-independent" without the limits the matrix states twice (one fixture, 62 % permeable, one tile size, 50 ms floor, no perf; and the u8 column widened 4x with its numbers reported as upper bounds). The offered rescue -- survivor-independence is structural since `gt_i32_to_mask` takes no gate -- is right about that function but must not be read as a general claim about predicate generation, or it contradicts §8a: `gt_i32_to_mask_under` exists at :1614. Accurate form: an UNGATED sweep is structurally survivor-independent, a GATED one is not. Gates: quack 14/14, clippy --all-targets --all-features -D warnings clean, fmt clean, citation-decay 0 new, append-only 9 files none shrank, supersession index current. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/board/EPIPHANIES.md | 88 +++++++++++++++++++++++++++++ crates/lance-graph-quack/src/lib.rs | 20 +++++++ 2 files changed, 108 insertions(+) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 06b166489..da91d3c62 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,91 @@ +## 2026-09-15 (4) — E-FUSING-FORFEITS-THE-SKIP-AND-ADAPTIVEFILTER-FAILS-IN-TWO-PLACES-NOT-ONE-1 — the fused lowering is order-independent BY CONSTRUCTION, two readers derived it from source because the crate doc does not say so, and DuckDB's A1 turns out to have a dead seed as well as an unrunnable loop + +**Status:** FINDING. Convergent — derived independently in two sessions from the same +line, neither having read the other, before comparing. +**Confidence:** HIGH — every claim is a source read or a row of §8a, all re-runnable. + +### The finding + +`lower` and `lower_fused` differ in a way neither name nor the crate doc suggested: +**only `lower` can skip.** + +| | gate on a predicate | order changes skipped words? | +|---|---|---| +| `lower` (in place) | first ungated, later ones on the running ACCUMULATOR | **yes** — up to 99.90 pts (§8a) | +| `lower_fused` | `under.map(\|m\| Operand::Plane(m.0))` — caller's plane ONLY | **no, ever** | + +`assign_slots` gives every predicate its own slot and never chains, because the Boolean +combination is deferred to the fuser — and `MaskOp::Ternlog { imm, a, b, c, dst }` has +**no `under` field at all**, where `MaskOp::Pred { pred, under, dst }` does. A ternlog +combines already-materialised masks; by the time it runs, every sweep is paid for. + +The saving the gated form buys is physical, not bookkeeping. `ndarray`'s `pack_under` +(`simd_masking_ops.rs:1541-1546`) is `if gate == 0 { out_words[w] = 0; continue; }` — +the 64 values are never loaded. That `continue` is the entire mechanism §8a measures. + +The crate doc framed the choice as *"a consumer picks by whether it is scratch-bound or +pass-bound"* and said nothing about the skip. Both readers fell into it and both had to +go to source. Fixed in the same commit. + +### Why it matters beyond the doc + +It is the whole of what survived DuckDB's `AdaptiveFilter`. §8a measures ordering worth +up to 99.90 percentage points of skipped words on a clustered conjunction — **under +`lower` only**. On the same query under `lower_fused` it is exactly zero. So +`Filter::and_by_skip`'s lever is alive in one configuration: gated lowering × plane-free +conjunction × contiguous survivors. Under a plane it is inert as well +(`ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE`). `adaptive_order_probe.rs` has no fused +arm, so its table cannot see this and does not claim to. + +### A1 fails in TWO places, and only one of them was recorded + +`AdaptiveFilter` is a seed plus a loop, and they die of different causes: + +- **the seed** — `GetInitialOrder`, from the optimizer's static SELECTIVITY heuristic. + §8a kills it independently of anything about V3's executor: selective (36 survivors, + 0.055 %) and clustered (31, 0.047 %) are indistinguishable by density and **19.24 + points apart best-vs-best**. A selectivity-seeded reorderer seeds on a statistic that + provably does not separate the two regimes where the lever exists. +- **the loop** — swap, measure RUNTIME, keep or revert, halve likeliness. This one would + find contiguity, because contiguity shows up in runtime whether or not you can name + it. It is the half quack structurally cannot run: `Program.ops` is a fixed `Vec` with + no measurement. + +**The half that would work cannot run; the half that can run measures the wrong thing.** +That is stronger than the matrix's "not DuckDB's algorithm" — it says why no amount of +porting reaches it. + +### The V3-native seed, and its one blocker + +The clustered regime IS an address prefix. `Filter::prefix_u32` builds +`care = u32::MAX << (32 - b)` — a contiguous high run — so a `Cmp::MatchU32/MatchU64` +whose care mask has that shape is prefix-shaped **at lowering time**: free, structural, +no measurement loop. That would feed an ordering score from structure instead of asking +the caller. + +The blocker is exact and is one bit: `prefix_u32`'s own doc says contiguous *"on an +address-ordered lane"* — a property of the DATA, not the query — and `Col(pub u16)` is +deliberately not a name ("name resolution is the catalogue's job"). quack carries no +catalogue by design, so the seed needs a schema fact the crate does not hold. Real +blocker, not a detail. + +### ⊘ A citation of mine that was under-qualified + +I cited D-GTM-0m as "22.4–22.8 µs, survivor-independent" without its stated limits. The +matrix carries them twice and I should have carried them too: R5 — *"measured on ONE +fixture (65 536 rows = 256×256 axial hex, 62 % permeable, one tile size, timing floor +50 ms, no `perf`)"*; E4 — it *"widened a u8 permeability column 4× to use +`gt_i32_to_mask`, and reported its `n_gen` and coal numbers as upper bounds because of +it."* + +The rescue offered for it — that survivor-independence is structural because +`gt_i32_to_mask` takes no gate parameter — is right about that function and must not be +read as a general claim about predicate generation, or it contradicts §8a. The accurate +form: **an UNGATED sweep is structurally survivor-independent; a GATED one structurally +is not**, `gt_i32_to_mask_under` exists (`simd_masking_ops.rs:1614`), and `pack_under`'s +`continue` is where the difference physically lives. The µs figure is one geometry; the +asymmetry is not. + ## 2026-09-15 (3) — E-A-CHECK-THAT-CANNOT-RUN-IS-INDISTINGUISHABLE-FROM-A-CHECK-THAT-PASSES-1 — "lots of CI errors" was stale red plus silence, and chasing it found `--all-features` broken since the lancedb 0.38 bump **Status:** FINDING. Two independent mechanisms, one shape. Both measured on PR #1235. diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index f725a9aee..829dc6779 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -49,6 +49,26 @@ //! differential suite runs both against the same per-row oracle; a consumer //! picks by whether it is scratch-bound or pass-bound. //! +//! **They also differ in a way neither name suggests: only [`lower`] can +//! skip.** [`lower`] chains — the first predicate is ungated and each later +//! one gates on the running accumulator, so a conjunct narrows the input its +//! successors read. [`lower_fused`] gives every predicate its own slot and +//! gates it on the caller's resident plane ONLY (`under.map(Operand::Plane)`), +//! never on an accumulator, because the Boolean combination is deferred to the +//! ternlog — and [`MaskOp::Ternlog`] has no `under` field at all. So the fused +//! program has no progressive narrowing, and **term order cannot change its +//! skipped-word count.** The saving the gated form buys is physical, not +//! bookkeeping: `pack_under` does `if gate == 0 { continue }` before loading +//! the 64 values, so a dead word's column read never happens. +//! +//! This matters because it is the whole of what survives from DuckDB's +//! `AdaptiveFilter` (matrix row A1 / §8a). Ordering is worth up to 99.90 +//! percentage points of skipped words on a clustered conjunction under +//! [`lower`] — and exactly zero under [`lower_fused`], on the same query. +//! [`Filter::and_by_skip`]'s lever is therefore alive in one configuration: +//! gated lowering, plane-free conjunction, contiguous survivors. Under a +//! plane it is inert too (`ISS-QUACK-AND-BY-SKIP-IS-INERT-UNDER-A-PLANE`). +//! //! # Provenance — this is harvest-driven, not remembered //! //! Every operator below answers to a row of