diff --git a/.claude/agents/masking-ops-cartographer.md b/.claude/agents/masking-ops-cartographer.md new file mode 100644 index 00000000..d4396cfc --- /dev/null +++ b/.claude/agents/masking-ops-cartographer.md @@ -0,0 +1,99 @@ +--- +name: masking-ops-cartographer +description: Holds the map of the masking-op surface — which of the seven named T1 gaps are SHIPPED, which are deliberately unbuilt, which are blocked on a measurement, and which are only outlook. Fires BEFORE proposing any new `*_to_mask` / `mask_*` / `masked_*` primitive; BEFORE a consumer repo hand-rolls a masking capability; BEFORE citing a masking benchmark number; and BEFORE recording a gap as "closed" on the strength of code existing. Read-only cartography — it tells you where you are, it does not build. +tools: Read, Glob, Grep, Bash +--- + +# Masking-ops cartographer + +**THE RULE: existing code is not a moved measurement, and an unbuilt gap is +not automatically a gap.** Three of the seven named gaps are deliberately +unbuilt; one is shipped with its own falsifier fired against it. Check the map +before proposing anything. + +Canonical map: `.claude/knowledge/masking-ops-state.md`. Gap definitions and +their pre-registered falsifiers: +`lance-graph/.claude/plans/duckdb-to-v3-translation-matrix-v1.md` §3. + +## The four questions, in order + +**1. Does it already exist?** `ndarray::simd` is the only legitimate home +(thinking → lance-graph behind Panama; **SIMD → ndarray**; storage → +lance-graph behind Valhalla). Check the facade re-export list in `src/simd.rs` +first, then `src/simd_masking_ops.rs`. **Check `src/simd_nightly/` too** — that +arm carries 18 compare-to-mask pairs across every width and is AHEAD of the +stable arms, so a "missing" primitive may already have its contract written +there. A census shaped around `simd_.rs` files skips the nightly +directory entirely; that exact mistake graded the nightly arm as having none. + +**2. Is it deliberately absent?** Three are, and each has a stated condition +for changing: + +| unbuilt | condition | +|---|---| +| `u16` compares, `i64` ordered compares, `_under` siblings at u8/u64 | **a caller**. A speculative family is surface with no falsifier attached. | +| **G7** lane-vs-lane compare | the narrowing to unary-with-constant is what makes the `ConstantVector` ELIMINATE hold. Needs a predicate census showing a real column-vs-column need. | +| **G5** masked compaction | *"Do not build it before that count exists"* — egress points per query in the intended consumers. It is also the most expensive gap to build (AVX2/NEON/WASM/scalar all need generated bodies). | + +Answering "yes, deliberately" is a **complete** answer. Do not build it. + +**3. Is the gap open, or only its MEASUREMENT?** G1 and G2 both ship code. +G1's falsifier is answered (5.84-6.48× on the coal re-chain — a WIDTH effect, +nearly tier-independent). **G2's is unrun and blocked** on a Win32 PE binary +the container does not have. Recording G2 as "closed" because the functions +exist is the error this card exists to stop. + +And **G1's answer does not transfer to G2**: part of G1's win is the packing +vanishing, which is true of u8 alone (64 lanes, 64-bit word, one chunk = one +word). At u64 eight groups share a word and each needs a shift. + +**4. If you are citing a number, is the code under test still there?** +The G1 ratio was first published at 6.75× and is actually 5.84× — the timed +closures had no `black_box`, and the protection was asymmetric (the i32 arms +happened to be read later; the u8 arms were not). A benchmark's correctness is +not only about what it MEASURES but about whether the measured code **survived +the optimizer**. Asymmetric protection is the dangerous case: unguard both and +the ratio is absurd and you notice; unguard one and you get a plausible wrong +answer. + +## Verdicts + +- **SHIPPED** — name the commit and, if a number is attached, the gate and + tier it was measured on (v3 and v4 differ, and a bare `cargo` here is **v3**). +- **DELIBERATELY-ABSENT** — name the condition from the table above. Not a gap. +- **CODE-LANDED-MEASUREMENT-OPEN** — the functions exist, the falsifier does + not. Say which probe is owed and what blocks it. +- **GENUINE-GAP** — absent, wanted, with a consumer. Then, and only then, the + missing-capability STOP rule applies: it lands HERE, substrate-first, never + hand-rolled in the consumer. + +## The consumer question has ONE answer, and it is not Java + +Operator ruling, 2026-09-16: *"Java doesnt use masking ops. `Mask.minus()`, +`RowStore.hop()`. Lance-graph does. Java just sees boring `sql()` handed to +duckdb (Example)."* + +So when a design asks *"which consumer calls this op?"*, **"the Java surface" +is never a valid answer** — it is a finding. The consumers of this file's ops +are lance-graph, the ABI kernels, and quack's lowering. Java sits one tier +above all of them and is never told any of this exists; its relation to +`sql()` is exactly a consumer crate's relation to `ndarray::simd` — *"java +doesnt know why there is `sql()` polyfill, we just make sure there is."* + +Two consequences for this card's verdicts: + +- A **GENUINE-GAP** whose justification is "a Java caller needs it" is + mis-scoped. Re-ask it as: which lance-graph or ABI path needs it in order + to answer an ordinary `sql()`? If none does, the gap is imaginary. +- A proposal to expose an op — by any name — on a public Java signature is + **DELIBERATELY-ABSENT by ruling**, not an open opportunity. The reason is + the endgame: low-code *"Bring your own software"* against Palantir Foundry, + where every unit of novel API is lock-in-by-learning-curve, which is the + one thing BYOS promises not to require. Route it to `java-surface-warden`. + +## What this card does not do + +It does not build, and it does not adjudicate the cost model. G4 is the warning: +`mask_shift_morton` shipped against a fitted model with 2.8% max residual, and +the falsifier still fired — the model was right about the COST of the term and +wrong about the REMEDY. A fitted model is not a licence. diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 71aa84ba..e9625644 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -1,3 +1,736 @@ +## 2026-09-16 (12) — ⊘ the G1 ratio was INFLATED by dead-store elimination; corrected 6.75× → 5.84×, and the conclusion survives + +codex P2 on PR #309, and it was load-bearing. Entry (11)'s numbers are +superseded by the ones here. + +**The defect.** `timed()` repeats a closure with nothing reading the result, and +`hex_tenant_mq_probe` had **zero `black_box` anywhere**. A write-only closure is +then dead code an optimizing build may delete. The exposure was not symmetric — +it ran in exactly the direction that flattered the new arm: + +- the widened-**i32** arms were accidentally protected by a LATER read: the + spread loop consumes `elig`, and the parity `assert_eq!` at `:764` consumes + `m_new`; +- the native-**u8** arms had **no reader at all** — `elig_u8` and `m_new_u8` are + read only by the asserts that run *before* their timed calls. + +So the very ratio the probe exists to measure could have been the optimizer +deleting one side of it. + +**Corrected, release, both tiers, all six timed sites `black_box`ed on BOTH +inputs and outputs:** + +| tier | M1b: 6 masks | coal: one re-chain | +|---|---|---| +| v4 / AVX-512 | 8.06× → **6.91×** | 6.75× → **5.84×** | +| v3 / AVX2 | 7.77× → **6.38×** | 6.53× → **6.48×** | + +Absolutes behind the v4 coal row are the tell: the u8 arm got **slower** +(1006 → 1082 ns) and the i32 arm got **faster** (6789 → 6319 ns) — the +signature of dead-store elimination on the u8 side specifically, not of noise. +Maintained steps at x=4 move with it: i32 1.02 → **0.74**, u8 0.15 → **0.13**. + +**v3's coal ratio barely moved (6.53 → 6.48) while v4's moved 13%**, which is +itself consistent with the mechanism rather than with noise: AVX-512 codegen had +more room to eliminate the dead stores. A noise explanation would not pick the +tier with more optimization headroom. + +**The conclusion survives; the number does not.** The widening was still the +cost, at ~5.8-6.5× rather than ~6.5-6.8×, and still nearly tier-independent, +so still a WIDTH effect rather than an ISA one. + +### The lesson, which is sharper than the fix + +I had the measurement discipline right in every other respect on this probe — +bit-identity asserted before timing, both arms in one process, the widened +column's materialization deliberately left outside the timed region — and +still shipped a number that an optimizer could have manufactured. **A +benchmark's correctness is not only about what it MEASURES; it is about +whether the code under test is still THERE.** The asymmetry is what made it +dangerous: had neither arm been protected, both would have been deleted and +the ratio would have looked absurd. One arm accidentally protected by an +unrelated later read is the case that produces a plausible wrong answer. + +`timed()` now carries the rule in its own doc comment with this incident as +provenance, so the next closure written against it cannot omit the guard by +accident — the same "put the rule where the next person will be standing" +move as the v4 `-D warnings` disable trap in `CLAUDE.md`. + +## 2026-09-16 (11) — the G1 falsifier is ANSWERED, and the widening WAS the cost: 6.75× on the re-chain + +> **⊘ SUPERSEDED ON ITS NUMBERS by entry (12) above** — every ratio below was +> inflated by dead-store elimination in the un-`black_box`ed timed closures. +> The coal ratio is **5.84× (v4) / 6.48× (v3)**, not 6.75×/6.53×. The +> conclusion (the widening was the cost; a width effect, not an ISA one) is +> unchanged; the method notes below stand and are worth reading — they are +> what made the correction cheap. + +The plan pre-registered this and carried it open through BOTH N2 and N3: +*"build `gt_u8_to_mask`, re-run both probes; if neither the 8.9 µs re-chain nor +the #308 crossover moves, the widening was not the cost and G1 drops in +priority."* Answered on the half that could be run. Commit `92b4fcd`. + +| tier | M1b: 6 masks (widened i32 → native u8) | ratio | coal: one re-chain | ratio | +|---|---|---|---|---| +| **v4 / AVX-512** | 40810 ns → **5064 ns** | **8.06×** | 6789 ns → **1006 ns** | **6.75×** | +| v3 / AVX2 | 43798 ns → 5638 ns | 7.77× | 7189 ns → 1101 ns | 6.53× | + +In the probe's own cost-model units (v4, x=4) a maneuver drops from **1.02 +maintained steps to 0.15** — a re-chain used to cost a whole maintained step +and now costs about a seventh of one. That changes the Mississippi Queen coal +budget qualitatively, not marginally: re-chaining was priced as roughly "skip a +step", and it is now close to free relative to a step. + +### The method is what makes the number worth anything + +- **Bit-identity asserted BEFORE either arm is timed.** Two `assert_eq!`s; the + probe aborts on mismatch. A timing comparison between operations that do not + produce the same answer measures nothing — the same rule the parity-oracle + work states as *"an oracle must reproduce the operation under test, not + something in its neighbourhood that also rotates."* +- **One process, one dataset, both arms.** The native arm was **added beside** + the widened one, never swapped in. That turned out to matter: **neither tier + reproduces the 8.9 µs this probe's re-chain is quoted at** (v4 6789, v3 + 7189), so that historical absolute came from a build this session does not + reproduce. A before/after across runs would have been contaminated by that + drift; a ratio measured side by side is not. +- **The widened column's materialization sits OUTSIDE the timed region** (built + once, up front), so 6.75× is steady-state sweep cost only — the reading that + FAVOURS the widened arm. The real end-to-end gap is larger. + +### The ratio is nearly tier-independent, and that is the interesting part + +6.75× (v4) vs 6.53× (v3) — essentially the same. **So this is a WIDTH effect, +not an ISA effect**, and it would not have shown up by picking a better +instruction. Arithmetic that bounds it: for N elements the i32 path issues +N/16 compares reading 4N bytes; the u8 path issues N/64 compares reading N +bytes — 4× fewer instructions AND 4× less memory. Observed 6.5-8× exceeds +either factor alone. + +The plausible remainder is **the packing**, which is exactly the property N3 +had to discover the hard way: at u8 one 64-lane chunk IS one whole 64-bit mask +word, so `out_words[g] = bits` with no shift, while the i32 path must +shift-and-OR each 16-bit group into its word (`out_words[g / 4] |= bits << +((g % 4) * 16)`). Labelled **CONJECTURE** — it is consistent with the numbers +and with the architecture, but it is an attribution, not a separate +measurement. Isolating it would need a third arm that packs u8 results the i32 +way. + +### Half the falsifier is BLOCKED, not skipped — and the distinction matters + +`r2il_column_scan_probe` is the other pre-registered half, and it is the more +interesting one: its own header names BOTH gaps this wave closed — *"1. No +`u8` comparator ... a consumer must keep a widened copy"* and *"2. No `u64` +RANGE comparator ... the ordered family stops at `i32`"*, i.e. N2 and N3 +respectively. It even records that **100 % of the real Ram-space offsets exceed +2³²**, and that it worked around G2 by splitting the offset into hi32/lo32 +columns, *"valid only because the chosen window lies inside one `hi32` +bucket"*. + +It cannot run here. It needs a column dump from `r2sleigh-lift`'s +`win32_census`, which needs a Win32 PE binary; the generator exists in the +sibling repo (`99d2553`) but **no PE binary exists in this container**. +Synthesizing a dump would yield a number shaped like the falsifier's answer +without being it, and that probe's own docs insist on *"a real lift rather than +a synthetic stream"*. So: + +- **G1: measured, and the answer is that closing it was worth it** (on the hex + tenant; the r2il crossover half is still owed). +- **G2: code landed on six realizations, measurement still OPEN.** Nothing yet + shows the u64 range family pays for itself, because the one consumer that + would show it is the blocked probe. Recorded as open rather than inferred + from G1's result — the widths are different and G1's win came partly from a + packing property that **does not hold at u64**. + +Reproduce: `CARGO_PROFILE_DEV_DEBUG=0 env -u RUSTFLAGS cargo --config +.cargo/config-v4.toml run --release --example hex_tenant_mq_probe`, and the +same without `--config` for the v3 row. + +## 2026-09-16 (10) — N3 / T1 gap G2 LANDED on all six realizations, and a cross-target "it compiles" needed a ROUTING PROOF to mean anything + +`{eq,ne,gt,ge,lt,le}_u64_to_mask` + the `simd.rs` facade + the lane primitive +`U64x8::{cmpeq_mask, cmpgt_mask} -> u8` on every arm that lacked it — commit +`e05afbd`. `i64` deliberately not built (no consumer), same rule that skipped +`u16` in N2. + +**The width where the packing is NOT free, which is the fact N2 could not +teach.** `U8x64` is 64 lanes against a 64-bit word, so one chunk was one whole +word with no shift at all. `U64x8` is 8 lanes: eight groups share a word and +each must land at its own byte, `out_words[g / 8] |= (bits as u64) << ((g % 8) +* 8)`. Nothing upstream of this family exercises that shift — every u8 test +would pass with it wrong — so it got a test whose whole job is the shift. + +**Per-arm, because only two of six have the instruction.** avx512 native +(`_mm512_cmp{eq,gt}_epu64_mask`; `__mmask8` IS `u8`, read out of the +toolchain's own stdarch rather than assumed). NEON native (`cmhi`/`cmeq` — and +the plan's earlier instinct to write this as a scalar fold would have been +wrong: a gap in this repo's WRAPPER is not a gap in the ISA). WASM has **no** +unsigned ordered 64-bit compare at all, so it flips the sign bit of both +operands into `i64x2_gt` — the same trick `U8x32::cmpgt_mask` already carries +at byte width, cited rather than re-derived. scalar and avx2 are both flat +`[u64; 8]` polyfills (the avx2 file says so itself), where Rust's `>` on `u64` +already IS the unsigned compare. + +### ⊘ The census defect this wave found in the plan's OWN table + +§18 graded the nightly arm "compare-to-mask: **none**". False, and not +marginally: `src/simd_nightly/` has carried **18 such pairs across every +width** — `U64x8::{cmpeq_mask, cmpgt_mask} -> u8` included — since before this +wave started. + +**Why the census missed it is the exact mirror of the trap §18 itself +documents.** §18 warns that a FILE-WIDE grep makes an absent surface look +present (`cmpeq_mask` hits in `simd_avx2.rs` come from the adjacent `U8x64` +block). This is the inverse: the nightly arm is a **directory**, not a +`simd_.rs` file, so a census shaped around the single-file arms skips it +and a PRESENT surface looks absent. Both failures are the same error — +letting the search SHAPE stand in for the thing searched. + +It reframes the wave, and downward in difficulty: **N2 and N3 were not adding +a capability, they were bringing the stable arms up to a contract the +validation arm already stated.** The nightly bodies are therefore the contract +reference (return width, bit order, `simd_gt` on a `u64` element type being +already unsigned), and `masking-parity.sh nightly` becomes a genuine +cross-realization differential rather than a same-author self-check. That gate +was not in the plan and is stronger than what was. + +### ★ ROUTING PROOFS — the new method, and it is the durable part of this entry + +A cross-target `cargo check` that passes is evidence about **nothing** unless +you know WHICH arm it compiled. `simd_neon` and `simd_wasm` are declared in +`lib.rs` behind `#[cfg(feature = "std")]` alone — no target cfg — so their +mere presence in a build proves nothing; the arch gate is inside each file. +A wasm check without `+simd128` compiles the scalar arm and reports success +while never touching `simd_wasm.rs`. + +So each arm was proved by RENAMING its `cmpgt_mask` and confirming that +exactly the target routing to it fails — and the compiler names the arm: + +| target | error names | +|---|---| +| `wasm32 +simd128` | `wasm32_simd::U64x8` | +| `wasm32` bare | `U64x8` (the scalar arm) | +| `x86_64` v3 | `simd_avx2::U64x8` | +| `aarch64` | `simd_neon::U64x8` | + +**The anchor assertion earned its place on the neon one.** `simd_neon.rs` has +TWO `pub fn cmpgt_mask(self, other: Self) -> u8` — the `U64x2` leaf and the +`U64x8` composition — so the script refused the non-unique anchor instead of +silently renaming whichever came first. Re-aimed at the composition by +anchoring on its `self.0[p]` loop body. + +### Three semantic disables, red-then-green + +| test | disable | observed | +|---|---|---| +| `gt_lt_u64_to_mask_are_unsigned_not_signed` | `epu64` → `epi64` | `0x8000000000000000 > 0x7fffffffffffffff must be true unsigned` | +| `u64_packing_places_each_group_in_its_own_byte` | `(g % 8) * 8` → `* 4` | `group 1 must occupy byte 1 alone (got 0x...0ff0)` | +| `u64_family_matches_scalar_reference_randomized` | `lt` loses the operand swap | `ge n=8 t=0 i=0` | + +Restore was by `cp` from a backup, **not** `git checkout` — which sidesteps the +workspace's known trap (checkout reverts to the last COMMIT, so a disable run +over uncommitted work deletes it) without needing to commit first, and let the +disables run while the last worker was still writing a different file. + +### Gate, all six realizations + +`avx512` (v4): 2370 lib tests, clippy `-D warnings`, 8 doctests, parity +`avx512f=true`. `avx2` (v3): 2319. `neon`: `cargo check --target aarch64` +(check only — no qemu here). `wasm+simd128` and `scalar` (wasm32 without +simd128): built and RUN under node, both PASS. `nightly`: PASS, +`nightly-simd=true`. The 2370-vs-2319 gap is arch-gated tests, not a +regression. + +### Two corrections to the plan's own honest-scope claims, both made before code + +§18b closed with *"no aarch64 or wasm32 target is installed in this +container"*. **Three are installed.** Both cross arms compile-check clean on +the unmodified tree, and wasm can additionally be RUN under node. The +paragraph would have sent a worker into those arms believing the gate did not +exist. Residual limit stated precisely instead of dropped: NEON can be checked +but not run. + +### Operator rule landed the same session: `CARGO_PROFILE_DEV_DEBUG=0` + +*"use debug 0."* Measured here on the identical tree and the identical run: +`target/debug` **1.9 GB → 291 MB**, 6.5×, for a suite that passed 2319 tests +either way. **Debug info is the disk hog, not the code** — and this container's +writable allowance is a fixed per-session budget that presents as `No space +left on device` mid-LINK, not as a full disk, so the symptom points at the +wrong thing. Recorded in `CLAUDE.md` beside the build-config block. Note a +profile change invalidates the whole cache, so `target/debug` gets deleted +before the switch rather than a second copy growing beside it. + +### Still open + +G2's own measurement half, exactly as G1's: the code exists, and existing code +is not a moved measurement. PR #308's `find_ram_in_range` re-expressed its +query by splitting offsets into hi32/lo32 buckets to dodge the missing +primitive; until that probe is re-run against the real `u64` family, the claim +that G2 was worth closing is unmeasured. Same for N2's `hex_tenant_mq_probe` +re-chain. Two probes, both still unrun, both pre-registered. + +## 2026-09-16 (9) — N2 / T1 gap G1 LANDED: at `u8` the packing is FREE, and a disable run has a second failure door + +Six functions in `simd_masking_ops.rs` (`{eq,ne,gt,ge,lt,le}_u8_to_mask`) plus +the `simd.rs` facade re-export that makes them reachable — commit `a8e7d7d`. +Gap G1 of the DuckDB→V3 translation matrix closed for the `u8` width; `u16` is +deliberately NOT built, because no consumer compares `u16` lanes and a +speculative family is a maintenance surface with no falsifier attached to it. + +**The one genuinely new fact, and it does not generalize.** `U8x64` is 64 lanes +and a mask word is 64 bits, so `cmpeq_mask` / `cmpgt_mask` over one +`as_chunks::<64>()` group yields *exactly one whole* `out_words[g]` — no shift, +no accumulate, no partial-word carry. That is why these bodies are visibly +shorter than their `i32` siblings, where `I32x16` needs four chunks and three +shifts per word. It is worth stating precisely because the temptation in N3 is +to assume the same shape: it is false there. `U64x2::cmpgt_mask -> u8` composes +×4 per the operator's `64x2 × 4` rule and then has to be PACKED, +`out_words[g / 8] |= (bits as u64) << ((g % 8) * 8)`. The u8 family is the only +width in the matrix where the lane count and the word width coincide. + +Three shapes, deliberately not four — `eq`/`gt` direct; `lt` as an OPERAND SWAP +(`t.cmpgt_mask(v)`, never `x > t - 1`, which underflows at `t == 0`, the same +reasoning the `i32` family already carries for `i32::MIN`); `ge`/`le`/`ne` as +the complement of their strict sibling with `clear_mask_tail` re-run, which +fixes both the partial last word and every surplus word. The tail is +`pad_tail` into one zero-padded register through the same packed compare, then +`& word_range_mask(0, tail.len())` — load-bearing, because on the `lt` arm the +zero padding compares TRUE against any `threshold > 0` and would report phantom +set bits. + +**Disable table — all three red-then-green, none assumed:** + +| test | disable | observed | +|---|---|---| +| `gt_lt_u8_to_mask_are_unsigned_not_signed` | `gt_u8_to_mask` compares signed | `0x80 > 0x7F must be true unsigned (signed: -128 > 127 is false)` | +| `complement_forms_clear_every_trailing_bit` | `ge_u8_to_mask` fixes only the last live word | `ge: bit 128 (n=65) must be clear` | +| `u8_family_matches_scalar_reference_randomized` | `lt_u8_to_mask` loses the operand swap | `lt n=1 t=0` | + +The unsigned falsifier is the one that matters at this width: `u8` values +`0x80..=0xFF` are NEGATIVE under a signed compare, so an implementation that +reached for a signed lane op passes every small-value test and inverts on +exactly half the domain. The surplus-word test pre-sets **two** words past +`ceil(n/64)` to `u64::MAX` at `n = 65` and `n = 100` (both non-multiples of 64), +so neither a writer that skips the surplus words nor one that fixes only the +last live word can pass by accident. + +**⊘ THE NEW TRAP, and I walked into it on the first disable.** The v4 config +carries `-D warnings`. The signed disable removed the last use of +`threshold_v`; `unused variable` became a hard error; the test binary was never +built; the run emitted no `test result:` line at all. Piped through a grep for +the failing assertion, that is **byte-identical to a guard that is not +load-bearing** — no output either way. I read it as "the falsifier is inert" +for one turn before re-running unfiltered and seeing `error: unused variable: +threshold_v`. + +This is the workspace's existing trap (*a disable that does not APPLY is +indistinguishable from a guard that does not bind*) with a **second door**: a +disable that applies, asserts its anchors, and still never runs. The anchor +assertion — which I did have, and which correctly reported `3 anchor(s) +asserted unique` — does not protect against it, because the edit genuinely +landed. **Read the exit status and the `test result:` line, never only a grep +of the assertions**; and prefix rather than delete when a disable orphans a +binding. Recorded in `CLAUDE.md` beside the v4 invocation, which is where a +session looks before running anything under that config. + +**Chunking is vindicated as the dispatch fix, on evidence.** Worker B vanished +after 3–5 h with nothing written; this identical class of work, handed out as +one small scoped chunk (A1: one file, six functions, an explicit "do not run +cargo", `tee -a` durability), came back complete with a correct hand-back — +including the one thing it was right NOT to do itself: it FLAGGED the +cross-file `simd.rs` re-export dependency rather than smuggling a second file +into a one-file task. Without that flag the six `///` examples (`use +ndarray::simd::eq_u8_to_mask;`) would not have compiled, and the failure would +have surfaced only in a doctest run nobody had scheduled. + +**v4 gate (`--config .cargo/config-v4.toml`, `env -u RUSTFLAGS`):** clippy +clean at `-D warnings`; **2366** lib tests pass, 0 failed; the **6** new +doctests pass; facade parity `avx512f=true`, 11 check groups bit-identical. + +Loose ends: the six `_under` siblings (care-masked forms) are not built — the +`i32`/`u32` families have them and the shape is mechanical, but no caller needs +a care-masked `u8` compare yet. `u16` is not built, same reason. The G1 +*measurement* (re-run `hex_tenant_mq_probe`'s 8.9 µs re-chain and PR #308's +`r2il_column_scan_probe` crossover) has NOT been done: if neither moves, the +4× widening was not the cost the matrix assumed and G1 drops in priority +regardless of the code now existing. + +## 2026-09-16 (8) — LLVM counter-test: the unfused bit-exactness contract SURVIVES codegen, and BF16 tile GEMM is fully fused — both now measured, not read + +Operator: *"ggf mit LLVM Gegentesten."* Correct reflex — entry (7) asserted +things about generated code having only read source. Done, with the repo's own +witness technique (`scripts/codegen-witness.sh`'s method: optimized build, +`--emit=asm`, per-symbol instruction counts), under **v4**: + +``` +env -u RUSTFLAGS cargo --config .cargo/config-v4.toml rustc --release --lib --features std -- --emit=asm +``` + +Counts are per mangled symbol body in the emitted `.s`: + +| symbol | `vfmadd*` | `vmul{pd,ps,sd,ss}` | `vadd*` | `%zmm` refs | +|---|---|---|---|---| +| `simd_ops::gemm_f64_tiled` — the UNFUSED contract | **0** | 22 | 15 | 57 | +| `simd_ops::gemm_f64_tiled_fma` — the fused tier | **15** | 7 | 0 | 33 | +| `simd_ops::bf16_tile_gemm_16x16` | **17** | **0** | 19 | 193 | + +### What this settles + +**The bit-exactness contract holds at the instruction level.** +`gemm_f64_tiled`'s doc (`simd.rs:632`) claims bit-identity across backends via +UNFUSED mul+add in ascending-k order. A contraction by LLVM would break that +silently, and nothing in the tree checked it. Measured: **zero `vfmadd` in that +symbol** under v4. Rust does not enable FP contraction by default, so this is +the expected outcome — but expected is not verified, and the whole point of the +contract is that it must not depend on a default nobody re-checks. + +**BF16 tile GEMM is fully fused, on zmm.** `vmul = 0` alongside 17 `vfmadd` +means every multiply in that symbol is fused; there is no unfused product +anywhere in the accumulate. With 193 `%zmm` references the accumulator is in +512-bit registers, which is the register-residency half of entry (7)'s claim. +The 19 `vadd` are the horizontal `reduce_sum` and the `c[i*16+j] +=`, not the +inner product. + +**The fused tier is fused as documented**: 15 `vfmadd`, 0 `vadd`; its 7 `vmul` +are the α/β scaling, outside the accumulate. + +### The gap this exposes, named + +`add_mul_f32` / `add_mul_f64` produce **no standalone symbol** — they are +`#[inline]` and were inlined away, so the counter-test cannot witness them from +the library asm. That is exactly why `examples/ternlog_codegen_probe.rs` exists +in the shape it does: `#[inline(never)]` bodies, `black_box` inputs so nothing +constant-folds, and a self-check that a packed-but-wrong body fails before its +assembly is trusted. **An `add_mul` arm in that probe is the clean follow-up**; +until it exists, the FMA claim for those two functions is verified by reading +only, and is labelled so. + +### Method note worth keeping + +`--config .cargo/config-v4.toml` is load-bearing here for the reason +`codegen-witness.sh` already documents: cargo JOINS matching `target.*.rustflags` +entries and the LAST `-Ctarget-cpu` wins, so the env-var form passes v4 and THEN +`.cargo/config.toml`'s v3, producing a v3 build that reports itself as v4. This +run used `--config`, which is the same cfg key at higher precedence. + +> **⊘ CORRECTED 2026-09-16 (codex/coderabbit, PR #309) — the mechanism above is +> stated BACKWARDS, and the two halves belong to different cargo rules.** +> Cargo joins matching `target..rustflags` across **config sources** +> (`--config` + `.cargo/config.toml`), last `-Ctarget-cpu` winning — that half +> is right, and it is why `--config` is load-bearing. But the `RUSTFLAGS` +> **environment variable is not a config source**: it is mutually exclusive +> with them and REPLACES the lot, so it can never "pass v4 and THEN v3." +> +> The real trap runs the other way, and `CLAUDE.md` states it correctly: an +> **already-set** `RUSTFLAGS` (from anything in the environment) silently +> discards `config-v4.toml`'s `-Ctarget-cpu=x86-64-v4` entirely, so the arm +> measures whatever that env var says while the command line claims v4. That +> is why every v4 invocation is written `env -u RUSTFLAGS cargo --config …`, +> and why the parity program prints `avx512f=true|false` rather than trusting +> the flag it was launched with. + +## 2026-09-16 (7) — `array_windows` + `add_mul` closes TWO different mantissa losses, and "bit exact" splits into two contracts the tree already distinguishes + +Operator: *"Und hilft dadurch immer bit exakt zu sein ohne mantissa Verluste — +besonders bei BF16X16 tile Gemm und F32 x add_mul."* Verified at both named +sites, and the verification turned up a distinction worth keeping. + +### The two losses the pair closes + +**(1) Product rounding.** The exact product of two f32s needs ~48 mantissa bits; +f32 holds 24. An unfused `a*b + c` rounds the product to 24 bits BEFORE the add, +then rounds again. `mul_add` carries the full-width product into the addend and +rounds ONCE. `add_mul_f32` (`simd_ops.rs:158`) is exactly this — +`va.mul_add(vb, vacc)` on `F32x16`, and its scalar tail (`:169`, +`acc[i] = a[i].mul_add(b[i], acc[i])`) uses the SAME fused op, so the tail is not +a second rounding regime and the function does not change behaviour with length. + +**(2) Accumulator narrowing — the bigger one, and it is what BF16 is about.** +BF16 carries 8 mantissa bits against f32's 24. `bf16_tile_gemm_16x16` +(`simd_ops.rs`) takes `a_bf16: &[u16]`, `b_bf16: &[u16]` and `c: &mut [f32]`, +decodes BF16 → f32 **once, before the loops** (`bf16_to_f32_batch`), then +accumulates entirely in an `F32x16` REGISTER via `mul_add`, writing to `c` only +at the closing `reduce_sum()`. **BF16 is the input format and never the +accumulate format.** Round-tripping the accumulator through BF16 storage would +truncate 16 mantissa bits per step — a loss of a different order than (1). + +**Where the const window comes in:** `array_windows::<{PREFERRED_*_LANES}>` +makes N a compile-time constant so the chain unrolls into registers and the +accumulator is never spilled. A spill to a narrower slot re-opens (2), which is +why the const size is not an ergonomic detail. + +### ⊘ But "bit exact" is TWO contracts here, and the tree chose differently for each + +`simd.rs:632-641` states it plainly for the f64 GEMM: + +> *"Crate-native tiled f64 GEMM … with a bit-exactness contract: **unfused +> mul+add in ascending-k order** per element → bit-identical on every backend +> (AVX-512/AVX2/NEON/WASM/scalar) and, at α=1 β=0, bit-identical to the naive +> triple-loop reference."* — `gemm_f64_tiled_fma` is *"the fast fused tier (same +> tiling/order, one rounding per step) … **not the backend engine, because its +> scalar polyfill can lower to libm `fma()` on baseline builds**."* + +So: + +| sense of "bit exact" | how it is achieved | which kernel | +|---|---|---| +| **fewest roundings / no mantissa loss** — the operator's sense | FUSE: one rounding per step | `add_mul_f32/f64`, `bf16_tile_gemm_16x16`, `gemm_f64_tiled_fma` | +| **reproducible across backends and against the naive reference** | DO NOT fuse, and pin the k-order | `gemm_f64_tiled` — the ground-truth GEMM for probes/certification | + +They pull opposite ways for one reason: whether `mul_add` becomes a hardware FMA +or an unfused pair is a TARGET property (the scalar polyfill may reach libm +`fma()`), so fusing buys accuracy at the cost of cross-backend identity. The +accuracy sense is right for compute; the reproducibility sense is what a parity +oracle needs. A session citing "bit exact" about a float kernel must say which. + +### One adjacent finding, named and NOT fixed + +`bf16_tile_gemm_16x16` allocates inside its `(i, j)` double loop — a +`vec![0.0f32; k]` per column gather, so 256 allocations per call, plus the two +decode buffers. `.claude/rules/data-flow.md` §1 says "never allocate inside a hot +loop — slice into pre-allocated storage". Recorded here rather than drive-by +fixed; it is a real defect in a kernel nobody asked me to touch today. + +## 2026-09-16 (6) — ⊘ `array_windows` exists for `add_mul`, not for "overlap"; I had the right conclusion from the wrong reason + +Operator: *"array_windows ist immer für add_mul um rounding roundtrips zu +vermeiden."* Verified against the facade, which says it itself +(`src/simd.rs:625-630`): + +> *"`array_chunks` (non-overlapping) and `array_windows` (overlapping) are the +> stable-Rust foundation primitives for SIMD-staged kernels — **together with +> `add_mul_f32` / `add_mul_f64` below**, they reach within a few % of a +> Cranelift-JIT'd inner loop on the BLAS-graph GEMM path and are the reason the +> JIT-native option was deemed unnecessary."* + +And `src/simd.rs:142-150` exists solely to serve it: `PREFERRED_F64_LANES` is +declared per arm (8 avx512 / 4 avx2 / 2 neon / 2 wasm / 4 scalar) *"for +`array_windows`"*, with the documented call shape +`data.array_windows::<{PREFERRED_F64_LANES}>()` → `F64x8::from_array(*window)`. + +**Why the const size is the whole point.** `array_windows` yields `&[T; N]` with +`N` a compile-time constant, so the FMA chain unrolls into registers and the +accumulator never spills. A dynamic-length `slice::windows()` cannot, and every +spill/reload is an extra rounding — the "rounding roundtrip". One fused +`mul_add` per step, one rounding; the alternative rounds at the multiply, again +at the reload, again at the add. + +> **⊘ CORRECTED 2026-09-16 (coderabbit, PR #309) — "every spill/reload is an +> extra rounding" is FALSE and mis-attributes a real effect.** An `f32` spilled +> to memory and reloaded as `f32` is bit-exact; the store/load is lossless. +> Rounding enters only on a NARROWING store (`f32 → bf16`/`f16`), or where a +> wider accumulator is forced back to the narrow type. +> +> The rounding difference being described is real but its cause is **FMA, not +> the spill**: `mul_add` computes `a*b + c` with a single rounding at the end, +> where separate multiply-then-add rounds the product first and the sum second. +> That is one extra rounding per step, and it is present whether or not +> anything spills. Read the paragraph above with "spill/reload" replaced by +> "the un-fused multiply's intermediate rounding"; the conclusion (prefer one +> fused `mul_add` per step) is unchanged and the mechanism is now the right one. + +**My error, recorded because I said it out loud.** I had written that +`array_windows` "fits nowhere in this wave" because *a compare-to-mask has no +overlap*. The conclusion is right and the reason is wrong: overlap is not what +the primitive is for. A compare-to-mask does not ACCUMULATE at all, which is why +it never reaches for `array_windows` — and a stencil that accumulates over a +fixed window reaches for it even though its windows could be described some +other way. + +Helper selection, corrected, for this wave and after: + +| helper | for | in this wave | +|---|---|---| +| `slice::as_chunks::()` | non-overlapping lanes **plus the tail**, which a predicate builder must mask | the predicate bodies | +| `array_chunks::` | same walk, **tail discarded** (`simd_ops.rs:423` is literally `as_chunks::().0.iter()`) | only where the length is a guaranteed multiple; `array_chunks_checked` surfaces a violation as `Err` instead of truncating | +| `array_windows::` + `add_mul_*` | the FMA staging pair, `N` const so the accumulator stays in registers | nowhere — masking accumulates nothing | +| `X::from_slice` / `copy_to_slice` | the typed-wrapper ↔ slice bridge | throughout | + +## 2026-09-16 (5) — why worker-side cargo is prohibited ABSOLUTELY: residue is the cost, BACKEND POLLUTION is the correctness failure + +Operator, verbatim: *"Worker are prohibited from running cargo"*, and the reason: +*"Würdest du mit jedem worker kompilieren hättest du target residue backend +pollution."* The second half was not written down anywhere and is the stronger +argument, so it now sits in `.claude/rules/agent-cargo-hygiene.md` beside the +residue one. + +The shared `target/` holds ONE realization at a time. A target-cpu change +invalidates the cache, so a worker's plain `cargo test` — which takes +`.cargo/config.toml`, i.e. **v3/AVX2** — run after the orchestrator built +`--config .cargo/config-v4.toml` **replaces** the v4 artifacts rather than +adding to them. The next probe reports whichever tier compiled last and says +nothing about which. + +**This is plan §17's defect one scale down, and the severity differs.** On the +orchestrator's own runs a tier is merely UNLABELLED — a re-run under a named +config repairs it, which is exactly what happened today (273.9×–300.9× v3 → +110.2×–123.2× v4). With N workers compiling on their own schedule it becomes +UNATTRIBUTABLE: the interleaving is gone and no number can be traced to a +backend afterwards. + +One shared `target/`, one realization, one compiler — the orchestrator. Workers +edit. Both worker briefs dispatched today carry the prohibition verbatim as +rule 1, with no carve-out for `test` or `clippy`. + +Separately, operator correction to my own framing: **clippy is not compiling.** +I had read the rule file's "tests yes, compile no" as contradicting the absolute +prohibition on the grounds that clippy compiles. It does not — it type-checks and +lints. The rule file was left unedited on that point; only the missing second +reason was appended. Practical consequence for the orchestrator's gates: a green +`cargo clippy` proves types and lints, NOT that a runnable artifact builds. The +evidence for today's landings was the `cargo run --release` probe and parity +runs, not the lint. + +## 2026-09-16 (4) — the AVX2 tier's `U8x64` byte compares were scalar loops beside a vectorized `U8x32` that already solved them + +Found while scoping T1 gap G1 (the `u8` compare-to-mask family): on the v3/AVX2 +arm `U8x64::{cmpeq_mask, cmpgt_mask}` built the `u64` one bit at a time, under a +comment calling itself a "scalar fallback", while `U8x32` in the SAME file +already carried `_mm256_cmpeq_epi8` + `_mm256_movemask_epi8` and an UNSIGNED +`cmpgt_mask` with the sign-bias XOR that AVX2 needs (it has only the signed +`_mm256_cmpgt_epi8`). Both now compose two halves: `(lo) | (hi << 32)`. + +**Scope, and it decides where this is exercised:** this body compiles ONLY on +the v3 arm. Under x86-64-v4 `U8x64` is the native `__m512i` from +`simd_avx512.rs` and none of it is reached. Per the operator ruling recorded in +entry (3), v3 is not where this workspace MEASURES — it is where the +realization matrix requires every backend to stay bit-exact, which is the whole +reason to fix it rather than leave it. + +Disable run (after the commit `3a5da8c`, so the restore could not eat it): +swapping the `lo`/`hi` halves fails 2 of 3 tests — the scalar-oracle test and +the lane-position test at indices 0/31/32/63. The sign-boundary test correctly +stays GREEN under that disable, because it tests signedness, not composition; +each of the three pins its own property and none of them is redundant. + +Gates on the v3 arm: clippy `-D warnings` clean, fmt clean, 70 tests across the +two touched modules, masking parity 11 groups bit-identical. + +## 2026-09-16 (3) — ⊘ every reveal-ratio number in this arc was v3/AVX2, including today's; the AVX-512 ratio is 110–123×, and CLAUDE.md was why + +Operator ruling: *"It's unacceptable to use avx2 / of course you need to run +ndarray with x86 v4 avx512f."* Correct, and the reason I was not is a +documentation defect I have now fixed rather than worked around. + +`CLAUDE.md:84` claimed `.cargo/config.toml` sets `target-cpu=x86-64-v4` +("AVX-512 mandatory"). It sets **v3**, and always has (`.cargo/config.toml:83`), +deliberately — v3 is the portable CI/distribution floor. So a plain `cargo run` +measures AVX2. Every number in §14, in the (5) STORNO, and in my own §16.2 +entry earlier today is a v3 number that does not say so. CLAUDE.md now carries +the v4 invocation and the `env -u RUSTFLAGS` trap explicitly. + +Same probe, same binary, same host (Xeon with the full AVX-512 set), verified by +the parity program's own `avx512f=` line: + +| | v3 / AVX2 | **v4 / AVX-512** | +|---|---|---| +| TCAM sweep | 14 126–15 141 ns | **5 298–5 484 ns** | +| range write | 50–52 ns | 43–49 ns | +| reveal ratio | 273.9×–300.9× | **110.2×–123.2×** | +| ternlogq | 149.0 ns (0.146/word) | **111.6 ns (0.109/word)** | +| coal | 5 630 ns = 0.79 steps | **4 666 ns = 0.62 steps** | + +**The generalizable finding: a ratio between two arms that vectorize +differently belongs to the pair AND the target-cpu.** The TCAM sweep gains 2.7× +from AVX-512; the range write is memory-bound and gains ~nothing. The ratio +therefore falls by ~2.5× *because the wider ISA helps the arm being beaten*. +§14's 291 ns/pass (0.285 ns/word) is 2.0× my v3 and 2.6× my v4 per word, so it +belongs to neither run and its config was never recorded — it cannot be placed +on this table at all. + +What survives on the correct tier: the matrix's G6 re-scope trigger is "below +~10×", and 110× clears it by an order of magnitude, so the claim stands while +the headline number is more than halved. The SHAPE is config-independent and was +never the headline — the TCAM arm is flat in node size on both tiers, the range +arm tracks the node. + +Standing rule: every timing published for this plan names its target-cpu. +`mask_set_range`'s 5/5 tests and the 11 parity groups are green under v4 as +well as v3. + +## 2026-09-16 (2) — N1 / T1 gap G6 LANDED: `mask_set_range`, and the primitive changed the measured SHAPE, not only the code + +`pub fn mask_set_range(out_words: &mut [u64], lo: usize, hi: usize)` — full +overwrite (bits `[lo, hi)` set, everything else zeroed), address-blind, no +per-bit loop, single-word case its own branch. Plan §16.6 N1; matrix §3 G6. +Consumer wired: `hex_tenant_mq_probe`'s `range_reveal` keeps its `(lo, hi)` +trie arithmetic and calls the primitive for the write. + +**The measurement is a MIXED result and is recorded as one.** The probe's range +arm: + +| level | rows/node | hand-rolled ns | with the primitive | +|---|---|---|---| +| 0 | 65 536 | 89 | 52 | +| 1 | 4 096 | 54 | 50 | +| 2 | 256 | 44 | 51 | +| 3 | 16 | 43 | 50 | +| 4 | 1 | 43 | 52 | + +Faster at large ranges, **marginally slower at small ones**, flat overall. +Mechanism, and it is the reason to keep the primitive rather than the ratio: +the hand-rolled version zeroed the WHOLE buffer and then overwrote the +interior, paying 1024 redundant word writes at level 0 and none at level 4; the +primitive splits the zero-fill around the range, so it writes each word once. +Reveal-vs-TCAM band: **161.8×–343.5× → 273.9×–300.9×** — a higher floor and a +flatter curve. Both bands are one host, one fixture, one seed; §16.2's +portability caveat applies unchanged. + +`n` read 6 556 ns with residual 2.2 % here against 7 678 / 15.9 % on the +immediately preceding run of the SAME binary. That spread is cross-run noise on +this host, not an effect of the change — recorded so neither number is cited as +a before/after. + +**Disable runs, both red-then-green, both after the commit (`33716b9`) so the +restore could not eat the work:** + +| disable | observed | +|---|---| +| drop the three zero-fills and make the single-word write an `\|=` (i.e. OR, not overwrite) | 3 of 5 unit tests FAIL, including `..._overwrites_a_dirty_destination_rather_than_oring`; the two `should_panic` guards correctly stay green | +| make the single-word branch unreachable so it falls into the two-edge path | 2 of 5 unit tests FAIL **and the parity program ABORTS** — so the new `0xBxx` group is non-vacuous, not decoration | + +Gates: clippy `-D warnings` clean, `cargo fmt` clean, 5/5 unit tests, masking +parity **11** groups bit-identical (was 10), probe gate (range mask == TCAM +mask, popcount == node size) green at every level. + +Not built, named: no OR-ing variant (no consumer, and a zero-caller function is +dead code here). Still open from §16.6: whether a nibble-aligned +`reveal(prefix, level)` beats the general `[lo, hi)` — the general form shipped +first as a decision, not a finding. + +## 2026-09-16 (1) — census of this plan against the board and git: four governance defects, a fourth hex-tenant run, and the wave is the DuckDB matrix's own T1 gap list + +Three read-only censuses (ndarray code, ndarray plan+board, lance-graph +consumers) plus my own verification of every decisive claim. Full text: +`.claude/plans/gemm-ternlog-mask-consolidation-v1.md` §16. + +- **§14's numbers are stale against entry (5)'s STORNO, and the STORNO's own + commit `e5a87e6` edited this plan without touching them** (verified by diff: + one line changed, §15's tail law). §16.1 tabulates every correction. +- **A fourth hex-tenant run** (§16.2) puts the reveal ratio at 161.8×–343.5×, + below the stated ~200× floor, so across four runs the honest range is + ~160–490×. Its `ternlogq` is 152 ns/pass against the board's 280–300 — a + different HOST, not a better estimate. Consequence worth keeping: **coal + denominated in "maintained steps" is not a portable unit** (µs fell, steps + rose, because the step got cheaper). The invariant across all four runs is the + SHAPE: the TCAM arm is flat in node size, the range arm tracks it. +- **`D-GTM-0m` names two unrelated probes** nine days apart (`f1f4023` + behavioural soak, `d9459f0` hex tenant). Not renumbered — that would break the + commit messages carrying the results — but no new work may use the id. +- **`D-GTM-0n` is measured, committed, and in NO governance doc.** Its + mask-vs-sparse-survivor crossover (0.1–0.8 % active) is directly relevant to + §12.5 pt 2's "no mask-beats-sparse claim until a sparse arm exists", and that + question could not be asked while the probe was invisible. +- **The wave is not new.** `duckdb-to-v3-translation-matrix-v1.md` §3 already + lists G6 (`mask_set_range`), G1 (u8/u16 compare-to-mask) and G2 (ordered + u64/i64) as verified-absent with pre-registered falsifiers. N1 = G6 (two + consumers working around it, payoff measured). N2 = G1 (two fixtures measure + the 4× widening cost; the matrix's falsifier stands unchanged). N3 = G2 — + **PR #308 answers the matrix's own open question**: the ordered-u64 predicate + count is ≥ 1, and 100 % of the real offsets exceed 2³², so narrowing is + unsound. G2 is a gap. + ## 2026-09-14 (5) — STORNO on entries (3) and (2), from the #307 council (measurement-skeptic, kernel-membrane, overclaim): same numbers, tighter words The numbers stand; the wording ran past them. Corrections, each to the sentence it replaces: diff --git a/.claude/knowledge/masking-ops-state.md b/.claude/knowledge/masking-ops-state.md new file mode 100644 index 00000000..bca884e0 --- /dev/null +++ b/.claude/knowledge/masking-ops-state.md @@ -0,0 +1,239 @@ +# Masking ops — what is DONE, what is PENDING, and what is only POTENTIAL + +> READ BY: any agent about to add, extend, or cite a `*_to_mask` / `mask_*` / +> `masked_*` primitive; any session picking up the DuckDB→V3 translation +> matrix; any consumer (lance-graph, lance-graph-java) that thinks it needs a +> new masking capability. **Read this before proposing a primitive** — three of +> the seven named gaps are deliberately unbuilt, and one is already shipped +> with its own falsifier fired. +> +> Status: **FINDING for the DONE rows** (each measured, dated, gated). +> **CONJECTURE for OUTLOOK.** POTENTIAL is explicitly not a plan. + +## Where this sits in the stack + +Operator ruling, 2026-09-16 — the allocation the whole arc serves: + +| what | lives in | membrane | +|---|---|---| +| **thinking** | lance-graph | **Panama** (computation never lives in Java) | +| **SIMD** | **ndarray** ← this repo | the `ndarray::simd` facade | +| **storage** | lance-graph | **Valhalla** (storage never lives in Java) | + +So: **every masking op belongs here, and only here.** A consumer that +hand-rolls one has found a gap in this file, not a licence. That is the +`simd-savant` / missing-capability STOP rule, and it is why gaps are tracked +as gaps rather than worked around downstream. + +The consumer-visible point of it (operator, same day): *"java is the low-code +intake glove around the lance-graph spine — lance-graph-java just happens to +offer the menu to the table in a pleasing way, using masking ops, offering 5 +star for the price of a blink."* The masking ops ARE what makes the menu cheap. + +> **⊘ CORRECTED, same day (operator).** *"Java doesnt use masking ops. +> `Mask.minus()`, `RowStore.hop()`. Lance-graph does. Java just sees boring +> `sql()` handed to duckdb (Example)."* The masking ops make the menu cheap — +> but they are the KITCHEN's, not the menu's, and the diner is never handed a +> mask verb. Read "using masking ops" as *served by* masking ops. The Java +> surface is the boring call a developer already knows; `Mask`/`RowStore` are +> the substrate's algebra standing on the wrong side of the wall. Full ruling +> + the BYOS endgame it serves: `lance-graph-java/CLAUDE.md` § "THE JAVA +> SURFACE IS `sql()`, NOT THE MASK ALGEBRA". + +## The gap list is not ours — it is the DuckDB→V3 matrix's own §3 + +`lance-graph/.claude/plans/duckdb-to-v3-translation-matrix-v1.md` §3 enumerates +G1-G7, each **verified absent** at the time with a pre-registered falsifier. +This file tracks their state. + +## DONE — shipped, gated, and (where claimed) measured + +| gap | what | state | +|---|---|---| +| **G6** | `mask_set_range(dst, lo, hi)` | **SHIPPED** (`33716b9`, fills vectorized `347875e`). Two consumers were working around its absence. `word_range_mask` built from two bits-below-N masks so `hi == 64` never computes `1u64 << 64`. | +| **G1** | `{eq,ne,gt,ge,lt,le}_u8_to_mask` | **SHIPPED** (`a8e7d7d`). The width where **packing is free**: 64 lanes, 64-bit word, one chunk is one whole word, no shift. **Falsifier ANSWERED — see below.** | +| **G2** | `{eq,ne,gt,ge,lt,le}_u64_to_mask` + `U64x8::{cmpeq_mask,cmpgt_mask}` on all six realizations | **SHIPPED** (`e05afbd`). Packing is NOT free here — eight groups share a word. Only avx512 (`epu64`) and NEON (`cmhi`) have the instruction; wasm has **no** unsigned ordered 64-bit compare and uses the sign-bias trick; scalar/avx2 are flat polyfills. **Measurement OPEN.** | +| **G4** | `mask_shift_morton` | **SHIPPED** (`255c36d`) — **and its own falsifier FIRED.** See OUTLOOK. | + +Also shipped alongside, not a numbered gap: `simd_avx2`'s `U8x64::{cmpeq_mask, +cmpgt_mask}` were 64-iteration **scalar loops** sitting beside an already- +vectorized `U8x32` that solved them (`3a5da8c`). + +### G1's falsifier — ANSWERED, and then corrected + +Pre-registered: *"build `gt_u8_to_mask`, re-run the probes; if neither moves, +the widening was not the cost and G1 drops in priority."* It moved. + +| tier | M1b: 6 masks | coal: one re-chain | +|---|---|---| +| v4 / AVX-512 | 41268 → **5973 ns** (**6.91×**) | 6319 → **1082 ns** (**5.84×**) | +| v3 / AVX2 | 38840 → 6084 ns (**6.38×**) | 6060 → 935 ns (**6.48×**) | + +Nearly tier-independent ⇒ a **WIDTH** effect, not an ISA one: 4× fewer +instructions AND 4× less memory, plus (conjecture) the vanished packing. + +⊘ **The first published figures (8.06× / 6.75×) were INFLATED by dead-store +elimination** — the timed closures carried no `black_box`, and the exposure was +asymmetric: the i32 arms were accidentally protected by a later read, the u8 +arms had none. Corrected in `daed0fd`/`95ac06f`. **The conclusion survived, the +number did not** — which is the reason to state both. + +## PENDING — the measurement half, and it is not optional + +**Existing code is not a moved measurement.** Two pre-registered falsifiers are +still unrun, and both run through the same probe: + +- **G1's second half** — PR #308's `r2il_column_scan_probe` crossover. +- **G2's only half** — its `find_ram_in_range` re-expressed against the real + `u64` family instead of the hi32/lo32 bucket split it used to dodge the + missing primitive. + +**Blocked, not skipped.** That probe needs a column dump from `r2sleigh-lift`'s +`win32_census`, which needs a **Win32 PE binary**; none exists in this +container. A synthetic dump would produce a number shaped like the answer +without being it, against that probe's own *"a real lift rather than a +synthetic stream"*. + +**G1's answer does not transfer to G2.** Part of G1's win is the packing +vanishing — a property of u8 alone. At u64 eight groups share a word and each +needs a shift, so the u64 family starts from a structurally weaker position. + +Also pending, and each is a deliberate non-build rather than an oversight: + +| not built | why | what would change it | +|---|---|---| +| `_under` (care-masked) siblings for u8/u64 | the i32/u32 families have them; the shape is mechanical; **no caller needs one** | a caller | +| `u16` compare-to-mask (half of G1) | no consumer compares u16 lanes | a consumer | +| `i64` ordered compare (half of G2) | the named consumer is an unsigned address window | a signed consumer | + +A speculative family is surface with no falsifier attached to it. That is the +rule these three rows apply. + +## OUTLOOK — named, grounded, not yet built + +**G3 — `masked_argmin_i32` / `masked_argmax_i32`.** `masked_min/max` return the +**value**; *"the row where x is minimal"* is unanswerable without a second full +pass, and is ambiguous on ties. Pre-registered measurement: two-pass +find-value-then-find-row vs a fused single-pass argmin at selectivity +{0.01, 0.5}. **Under 2× the gap is real but low priority.** Either way the +tie-break rule (first index wins) must be pinned in the API, because a fold has +no natural one. + +**G4 — shipped, and the model was half wrong.** `mask_shift_morton` landed and +its falsifier fired: over the full field the word op recovers **−14%** (14.5 vs +17.0 µs), not the modelled ~98%, and it **loses** to the NNUE delta arm +(9.3 µs). The **−66%** came from restricting the op to the trie node's own word +span — a mechanism the gap row never named. Verdict `[H]`: *the fitted cost +model was right about the COST of `n` and wrong about the REMEDY.* Worth +re-reading before trusting any other fitted model on this path. + +**G5 — masked compaction (`vpcompress`-shaped), and the standing order is DO +NOT BUILD IT YET.** Every ELIMINATE verdict in the matrix rests on "the mask +travels instead of the data" — true *inside* a plan, false *at the membrane*. +Egress needs the index list exactly once. It is also **the most expensive gap +on the list**: AVX-512 has `vpcompressd`; AVX2/NEON/WASM/scalar need generated +bodies. The gate is a count, not an opinion — egress points per query in the +intended consumers. One per query ⇒ scalar compaction is fine. The matrix says +it in capitals: *"Do not build it before that count exists."* + +**G7 — lane-vs-lane compare is DELIBERATELY ABSENT.** Every predicate here +takes a **scalar** threshold. DuckDB's surface is inherently binary; V3's is +unary-with-constant, and that narrowing is exactly the register in which the +`ConstantVector` ELIMINATE holds. If all intended predicates compare against a +constant, record G7 as *deliberately absent* **in the IR's docs**, so a future +session does not "fix" it. + +**The nightly arm is AHEAD of the stable arms, and it is the contract +reference.** `src/simd_nightly/` carries **18 compare-to-mask pairs across +every width**; the stable arms have a subset. So N2/N3 were not adding a +capability — they were bringing the stable arms up to a contract the validation +arm already stated. Consequence: `scripts/masking-parity.sh nightly` is a real +cross-realization differential, not a same-author self-check, and the nightly +bodies are what a new width should be matched against. + +## THE MENU — who serves these ops to whom (operator framing, 2026-09-16) + +The masking ops are not a library looking for a caller; they are the kitchen +behind a menu, and the courses are at different stages: + +| course | surface | state | +|---|---|---| +| **starter** | **SQL, via `lance-graph-quack`** — DuckDB→V3, zero-copy masked ops | **SHIPPED.** *"duckdb > quack is a nice proof-of-concept surface to offer SQL zero-copy masked ops, handed in the menu as a starter."* Plane leaf + survivor-skip gate, IN, projection, fused lowering, two-phase GROUP BY, each with a per-row oracle; and `plan_lower` is pinned EQUAL to quack's lowering by a differential, so there is one lowering rather than two that agree by luck. | +| **main** | **the Java glove** — a boring `sql()`, handed down zero-copy | **SHIPPED** (ABI minor 11) as the ABI floor; the SURFACE is being made boring. Reads as ordinary Java, costs a blink, because the work and the data are both elsewhere. ⊘ this row read `view.where(..).hop(..).count()` until the operator's 2026-09-16 ruling — that spelling is the mask algebra leaking onto the Java side, not the menu. | +| **another course** | **Gremlin / TinkerPop** | POTENTIAL, see below. | + +Why the starter matters beyond being a demo: **SQL is the surface where "zero +copy" is checkable by a stranger.** A `SELECT … WHERE` either returns the right +rows or it does not, and the per-row oracle says which — so the proof-of-concept +is also the cheapest available falsifier for the whole mask-travels-instead-of- +the-data claim. That is a better reason to keep it than novelty. + +**And the starter is also the MAIN's shape, which reframes this whole table.** +The operator's 2026-09-16 ruling makes `sql()` the Java surface too — so the +"starter" is not a lesser course that the Java glove improves on; it is the +same boring-front pattern, first. The relation to this file's own subject is +the exact one a consumer crate has to `ndarray::simd`: + +| tier | the caller writes | the caller does not know | +|---|---|---| +| consumer crate → ndarray | `U8x64::cmpeq_mask(..)` | which of six backends ran | +| Java → lance-graph | `sql("select …")` | that any of THIS FILE exists | + +*"java doesnt know why there is `sql()` polyfill, we just make sure there is."* +Which is this file's own STOP rule, one tier up: a missing `sql()` capability +is a gap to close in lance-graph/the ABI, never a licence for the Java side to +grow a mask verb — exactly as a missing primitive here is never a licence for a +consumer to write intrinsics. **The endgame that makes it non-negotiable:** +low-code *"Bring your own software"* against Palantir Foundry — novel API is +lock-in-by-learning-curve, which is the thing BYOS promises not to require, so +on the Java side novelty loses even when it is fast. + +## POTENTIAL — strategic, explicitly not a plan + +These are the operator's outlook statements. **None has a falsifier, none is +scheduled, and none should be started without one.** Recorded so the shape of +the opportunity is not lost. + +- **TinkerPop on lance-graph, replacing OGIT / JanusGraph / Cassandra** + (operator, 2026-09-16). This is the masking surface's natural upper bound: a + Gremlin `GraphTraversal` is `where`/`hop` composition, which is already + `Mask × ClassView → Mask`. The honest question to answer FIRST is which + Gremlin steps lower onto masks and which genuinely need G5 (compaction at an + egress membrane) or G7 (lane-vs-lane) — i.e. this outlook is partly a + **consumer census that would settle two open gaps**, which makes it cheaper + to scope than it looks. +- **r2sleigh as the ghidra backbone for the extended menu** (operator, same + day), **and the mechanism is a SHAPE match, not just a data source.** + Ghidra's p-code addresses **varnodes, which carry types**. r2sleigh's r2il + drops the typing and lands on a shape *similar to the V3 ABI* — which is the + interesting part for this file, because the V3 register is + `classid + 12 content-blind bytes` whose reading the ClassView picks per + read. An untyped IL whose operands are addresses-plus-bytes is already in the + register masks operate on; a typed varnode IR is not, and every type it + carries is one more thing to strip before a mask can see it. + + Concretely for the gaps here: predicates over an untyped r2il operand are + `lane + scalar` by construction — exactly G7's deliberate unary-with-constant + narrowing — whereas a typed varnode surface invites the lane-vs-lane form. + So this outlook, if pursued, is evidence FOR the narrowing rather than + against it. + + It is also the immediate practical blocker: r2sleigh is the source of the one + real-data probe this arc depends on (`win32_census` → + `r2il_column_scan_probe`), which is what PENDING is waiting on. **So the + outlook and the blocked measurement are the same dependency seen from two + ends: a PE binary in reach unblocks the measurement half of two gaps.** +- **Bitpacked mask representation.** Not raised as a gap; noted because + `strip_borders`-style binary morphology in sibling repos measured + representation as the first-order lever and SIMD as second-order. If a + masking consumer ever profiles hot, check representation before intrinsics. + +## The one procedural rule this arc earned + +**A benchmark's correctness is not only about what it MEASURES — it is about +whether the code under test is still THERE.** `timed()` in +`examples/hex_tenant_mq_probe.rs` now carries the `black_box` requirement in +its own doc comment, with the incident as provenance. Asymmetric protection is +the dangerous case: if neither arm is guarded both get deleted and the ratio +looks absurd; one arm accidentally guarded by an unrelated later read produces +a **plausible wrong answer**. diff --git a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md index e46911b8..90241bcb 100644 --- a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md +++ b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md @@ -994,3 +994,501 @@ the assert makes that explicit. loop; the spread gate (Morton == axial BFS) must stay green, and `n` is re-measured. The claim this rung makes is only that `n` drops; the number is the finding, not a prediction. + +--- + +## §16 — v1.5 GOVERNANCE CORRECTIONS + the G1/G2/G6 wave (2026-09-16) + +Append-only. Four defects found by a census of this plan against the board and +git history, each verified before being written here; then the wave they gate. + +### 16.1 ⊘ §14's numbers are STALE — the board corrected them and the plan was not updated + +`blackboard.md` 2026-09-14 (5) is a STORNO on entries (3) and (2) from the #307 +council. Its corrections apply to §14's bullets, which still state the +pre-STORNO figures as settled fact. **The commit that added the STORNO +(`e5a87e6`) DID edit this plan — and changed exactly one line, §15's tail law.** +So this is not a timing oversight: the file was open and the numbers were left. + +| §14 says | the board's correction | +|---|---| +| reveal 49–99 ns vs 22 µs — **228–462×** | ~200–490× across three runs; the range arm's floor is the 8 KiB output clear | +| ternlogq **291 ns/pass**, residual 2.8 % | 280–300 ns across runs; the "1.7 % at x=1" is fit-derived | +| coal = **0.48 maintained steps** | 8.8–13.7 µs = **0.47–0.76 steps**, single window each; cross-run spread on coal is **56 %** | +| (§15 opening) "`n` … 17.0 → 5.7 µs (−66 %)" | a STEP ratio, not an op ratio, on one fixture; the reset copy is **786 KiB**, not the 8 KiB the probe comment claimed | +| "the ratio survives the degree-1 ablation" | **a −33 % gain survives** (4 487 vs 6 704) — the ratio HALVES | +| "68/68 gates" | 68/68 probe ROWS, one fixture, one seed | + +Read §14 through this table. The numbers are not withdrawn; their CONFIDENCE is. + +### 16.2 A FOURTH run, measured today — it widens the spread on both axes + +`cargo run --release --example hex_tenant_mq_probe --features std`, this host: + +| level | rows/node | range ns | TCAM ns | ratio | +|---|---|---|---|---| +| 0 | 65 536 | 89 | 14 377 | 161.8× | +| 1 | 4 096 | 54 | 14 394 | 266.8× | +| 2 | 256 | 44 | 14 374 | 323.0× | +| 3 | 16 | 43 | 14 363 | 334.6× | +| 4 | 1 | 43 | 14 610 | 343.5× | + +`ternlogq = 152.4 ns/pass (0.149 ns/word)`, `n = 7 678 ns`, max rel residual +15.9 %; `coal = 6 938 ns = 45.5 ternlogq passes = 0.84 maintained steps at x=4`; +M2 linear, no cliff. Gates green. + +Two things this run does, and one it does not: +- It puts the reveal ratio's low end at **161.8×**, BELOW the STORNO's stated + ~200× floor. Across four runs the honest range is **~160–490×**. +- Its `ternlogq` is **152 ns/pass (0.149 ns/word)** against the board's 280–300 + (0.285 ns/word) — a 2× divergence, i.e. a DIFFERENT HOST, not a tighter + estimate. Consequently coal reads **0.84** maintained steps here against + 0.47–0.76 there: the µs went DOWN and the step count went UP, because the step + it is denominated in got cheaper. **Coal in "maintained steps" is not a + portable unit.** Report it with its host or report the µs. +- It does NOT resolve the fixture question. Still one geometry, one density. + +The shape everyone agrees on is the one worth keeping: the TCAM arm is FLAT +(~14.4 µs, it sweeps the column whatever the node size) while the range arm +tracks the node (89 ns at the root → 43 ns at one row). Cost follows the +selection, not the corpus. That is the stepless property, and it is invariant +across all four runs. + +### 16.3 ⊘ `D-GTM-0m` names TWO unrelated probes + +- `f1f4023` (2026-09-05) — `examples/behavioral_soak_probe.rs`, a cross-ISA + codebook soak. Real measured numbers in its commit message (coverage 50.4 % / + 56.8 % against a marginal-preserving null of 17.1 % / 37.5 %; real codebook 681 + tokens where shuffled needs 4 526, a 6.6× compression). +- `d9459f0` (2026-09-14) — `examples/hex_tenant_mq_probe.rs`, the hex tenant. + This is the one §14 and the board discuss. + +Nine days apart, same id. The first is invisible to any reader of the plan or +the board. **Not renumbered here** — a retroactive renumber would break the +commit messages that carry the results. Recorded so a future reader knows which +`0m` a citation means, and no new work may use the id. + +### 16.4 ⊘ `D-GTM-0n` is a real measured probe that appears in NO governance doc + +`741e34b` (2026-09-05, 2.5 minutes BEFORE the 0m-soak commit) — +`examples/ternlog_amortization_probe.rs`, with numbers: T3/T1 ~0.44–0.70 while +L1-resident, back to ~1.0 by K=32; L1 ~139 GB/s, L2 90–93, L3 29–30; the +mask-vs-sparse-survivor cross between 0.1 % and 0.8 % active. §12.7's own "what +W0 leaves open" list says "0c/0d/0e remain unrun" and never mentions 0n exists. +The omission is not staleness: the commit that updated §12's text +(`3e598a1`) also EDITED that probe file. + +**That crossover number is load-bearing and orphaned.** §12.5 pt 2 says no +"mask beats sparse GEMM" claim is available until a sparse arm exists — and 0n +measured a mask-vs-sparse-survivor crossover at 0.1–0.8 % active. Whether that +answers §12.5 pt 2 or is a different comparison is an open question, and it +could not be asked while the probe was invisible. + +### 16.5 ⊘ The Status header caps at v1.4 (2026-09-05); the body runs to §15 (2026-09-14) + +Three sections past the last version line, one of them (§14) carrying numbers +the board later corrected. This section is v1.5; the header stack is left +untouched per append-only, and this line is the pointer. + +### 16.6 THE WAVE — it is the DuckDB matrix's own T1 gap list, not a new idea + +`lance-graph/.claude/plans/duckdb-to-v3-translation-matrix-v1.md` §3 already +enumerates these, each **verified absent** with a pre-registered falsifier. The +wave closes them in this order: + +**N1 — G6 `mask_set_range(out, lo, hi)`.** Two independent consumers work +around its absence today: `lance-graph-quack`'s `Filter::prefix_u32` spells the +PREDICATE and cites the missing-capability STOP rule, and this repo's own +`hex_tenant_mq_probe.rs` hand-rolls the write inside an example. DuckDB carries +the same op on the same representation (`TemplatedValidityMask::SetRangeInvalid`). +Payoff already measured (§16.2). The matrix's own open question, carried here +unanswered: **does a general `[lo, hi)` beat a nibble-aligned +`reveal(prefix, level)`?** The latter is alignment-guaranteed and may be +strictly simpler. Shipping the general form first is a decision, not a finding — +if the aligned form measures better, the general one becomes its caller. + +**N2 — G1 `u8`/`u16` compare-to-mask.** The matrix's grounds: the V3 12-byte +register is carved into BYTES, so byte width is the substrate's NATIVE width and +i32 is the foreign one. Two independent fixtures now measure the widening cost — +the hex tenant widens a u8 permeability column 4× and reports `n_gen` and coal as +UPPER bounds because of it, and PR #308's r2il probe measures the mask arms at +**12 B/op against the scalar arm's 9 B/op**, with the mask arms LOSING from +~200 K ops. Falsifier, pre-registered by the matrix and unchanged: build +`gt_u8_to_mask`, re-run both probes; **if neither the 8.9 µs re-chain nor the +#308 crossover moves, the widening was not the cost and G1 drops in priority.** + +> **STATUS 2026-09-16 — the CODE half landed (`a8e7d7d`), the MEASUREMENT half +> has not.** `{eq,ne,gt,ge,lt,le}_u8_to_mask` ship with the `simd.rs` facade +> re-export, v4-gated (clippy `-D warnings` clean, 2366 lib tests, 6 doctests, +> parity `avx512f=true`), three disable runs red-then-green. `u16` deliberately +> NOT built — no consumer compares `u16` lanes, and a speculative family is +> surface with no falsifier attached. **⊘ THE FALSIFIER IS ANSWERED ON THE HALF THAT +> COULD RUN, AND THE WIDENING *WAS* THE COST** (`92b4fcd`). +> `hex_tenant_mq_probe` gained a NATIVE u8 arm beside its widened i32 one — +> same process, same data, bit-identity asserted before either is timed: +> +> ⊘ **The first published table was INFLATED by dead-store elimination** — the +> timed closures carried no `black_box`, and the exposure was asymmetric: the +> i32 arms were accidentally protected by a later read, the u8 arms had no +> reader at all (codex P2 on #309). Corrected, all six timed sites guarded on +> both inputs and outputs: +> +> | tier | M1b: 6 masks | coal: one re-chain | +> |---|---|---| +> | v4 / AVX-512 | 41268 → **5973 ns** (**6.91×**, was 8.06×) | 6319 → **1082 ns** (**5.84×**, was 6.75×) | +> | v3 / AVX2 | 38840 → 6084 ns (**6.38×**, was 7.77×) | 6060 → 935 ns (**6.48×**, was 6.53×) | +> +> The tell is in the absolutes: the u8 arm got SLOWER (1006 → 1082 ns) and the +> i32 arm FASTER (6789 → 6319 ns). And v3's coal ratio barely moved while v4's +> moved 13% — consistent with the mechanism (AVX-512 codegen had more room to +> delete the dead stores), not with noise. +> +> In the probe's own units a maneuver drops from **0.74 maintained steps to +> 0.13** at x=4. The ratio is nearly tier-INdependent, so this is a WIDTH +> effect, not an ISA one: 4× fewer instructions AND 4× less memory, plus +> (conjecture) the vanished packing arithmetic. Note **neither tier reproduces +> the 8.9 µs quoted above** (v4 6789, v3 7189) — that absolute came from a +> build this session does not reproduce, which is exactly why the native arm +> was ADDED rather than swapped in: a ratio survives baseline drift, a +> cross-run before/after would not have. +> +> **The #308 half is BLOCKED, not skipped.** `r2il_column_scan_probe` needs a +> column dump from `r2sleigh-lift`'s `win32_census`, which needs a Win32 PE +> binary; the generator exists in the sibling repo but no PE binary exists in +> this container. A synthetic dump would produce a number shaped like the +> answer without being it, against that probe's own *"a real lift rather than a +> synthetic stream"*. So the crossover verdict is still owed. Implementation note carried forward to N3: at `u8` the packing is +> FREE — `U8x64` is 64 lanes and a mask word is 64 bits, so one chunk is one +> whole word with no shift. That coincidence is unique to this width and is +> **false for `U64x2 × 4`**, which must pack +> `out_words[g / 8] |= (bits as u64) << ((g % 8) * 8)`. + +> **STATUS 2026-09-16 — CODE LANDED on all six realizations (`e05afbd`); +> MEASUREMENT still OPEN.** `{eq,ne,gt,ge,lt,le}_u64_to_mask` + the `simd.rs` +> facade + `U64x8::{cmpeq_mask, cmpgt_mask} -> u8` per arm. `i64` NOT built — +> the named consumer is unsigned, and a speculative family is surface with no +> falsifier. Gate: avx512 (v4) 2370 tests + clippy `-D warnings` + 8 doctests +> + parity `avx512f=true`; avx2 (v3) 2319; neon `cargo check --target aarch64` +> (no qemu, check only); wasm+simd128 and scalar both RUN under node; nightly +> PASS. Three semantic disables red-then-green, plus **four routing proofs** — +> renaming an arm's `cmpgt_mask` must fail exactly the target that routes to +> it, and the compiler names the arm (`wasm32_simd::U64x8` / `U64x8` / +> `simd_avx2::U64x8` / `simd_neon::U64x8`). That step is what turns a green +> cross-target check into evidence about WHICH arm compiled; without it, a +> wasm check lacking `+simd128` silently gates the scalar arm instead. +> +> **The falsifier this section pre-registered is still unrun.** PR #308's +> `find_ram_in_range` dodged the missing primitive by splitting offsets into +> hi32/lo32 buckets — valid for its fixture, not in general. Re-express it +> against the real `u64` family and re-measure. Until then "G2 was worth +> closing" is asserted, not shown; existing code is not a moved measurement. +> +> **And G1's measured win does NOT transfer here.** G1 came in at 6.5-8×, but +> part of that is the packing vanishing — at u8 one chunk IS one whole mask +> word. **That property is false at u64**, where eight groups share a word and +> each needs a shift. So the u64 family starts from a structurally weaker +> position than the u8 family did, and assuming it inherits G1's ratio is +> exactly the inference the tier-independence result warns against. + +**N3 — G2 ordered `u64`/`i64` compare.** The matrix left this conditional: +*"count how many intended predicates over a U64 lane are ordered rather than +equality. If the answer is zero, G2 is not a gap."* **PR #308 answers it: the +count is ≥ 1.** Its `find_ram_in_range` needs `lo <= offset < hi` over u64, and +**100 % of the real Ram-space offsets exceed 2³²**, so narrowing to the existing +i32 family is unsound. The probe re-expressed the query by splitting the offset +into hi32/lo32 and asserts the window lies in one hi32 bucket — valid for that +fixture, not in general. G2 is a gap. + +**Layering fence, load-bearing for N1.** `mask_set_range` is ADDRESS-BLIND: it +sets a bit range. A prefix maps to a contiguous ROW range only when row order IS +address order — the V3 SoA invariant. Deciding whether that holds is the +CALLER's job. ndarray must not grow a notion of "is this column sorted." + +**Not in this wave, and why.** The weighted arm / `pack_a_masked_f32` (D-GTM-5, +corrected three times, still unbuilt) stays after N1–N3: D-GTM-0j killed the +density framing and left a TYPE boundary — masks win where the relation is +Boolean, GEMM is required only where it carries VALUES — so the bridge's real +job is the weighted case, and §12.5 pt 2 still forbids any comparative claim +until a sparse arm exists. §16.4 may already have part of that arm. + +--- + +## §17 — ⊘ THE REVEAL RATIO WAS MEASURED ON THE WRONG TIER (2026-09-16, operator ruling: "it's unacceptable to use avx2") + +Every reveal-vs-TCAM number in this document — §14's 228–462×, the board +STORNO's ~200–490×, and §16.2's own 161.8×–343.5× — was produced by a **plain +`cargo run`**, which this repo's `.cargo/config.toml:83` pins to +**`x86-64-v3` (AVX2)**. None of them is an AVX-512 number and none said so. + +Root cause, and it is a documentation defect, not only my error: `CLAUDE.md:84` +claimed *"`.cargo/config.toml` — `target-cpu=x86-64-v4` (AVX-512 mandatory)"*. +The file has always set v3. CLAUDE.md is the first thing a session reads, so the +error propagates to every session that trusts it. Corrected in place 2026-09-16 +with the v4 invocation and the `env -u RUSTFLAGS` trap spelled out. + +### The same probe, same binary, same host, two configs + +Host: Xeon @ 2.10 GHz with the full AVX-512 set (`f bw cd dq vl` plus +`vnni bf16 fp16 vbmi vbmi2 vpopcntdq ifma bitalg`). Parity confirms the arm: +`avx512f=false` under the default, `avx512f=true` under `--config .cargo/config-v4.toml`. + +| quantity | v3 / AVX2 (the default) | **v4 / AVX-512** | +|---|---|---| +| TCAM sweep | 14 126 – 15 141 ns | **5 298 – 5 484 ns** | +| range write | 50 – 52 ns | 43 – 49 ns | +| **reveal ratio** | 273.9× – 300.9× | **110.2× – 123.2×** | +| ternlogq | 149.0 ns/pass (0.146 ns/word) | **111.6 ns/pass (0.109 ns/word)** | +| coal (one re-chain) | 5 630 ns = 0.79 steps @ x=4 | **4 666 ns = 0.62 steps @ x=4** | + +### The finding, and it generalizes past this probe + +**A ratio between two arms that vectorize DIFFERENTLY is not a property of the +algorithm pair. It is a property of the pair AND the target-cpu — and nothing in +this record was labelled with one.** The TCAM sweep is a compare over a value +column and gains **2.7×** from AVX-512; the range write is a handful of stores, +already memory-bound, and gains almost nothing. So the ratio falls by ~2.5× +*precisely because the wider ISA helps the arm being beaten*. Publishing such a +ratio without its build config is publishing half a measurement. + +Consequence for every "N×" in this plan: none of them is portable, and the +older ones are not even attributable. §14's ternlogq of 291 ns (0.285 ns/word) +is 2.0× my v3 and 2.6× my v4 per word, so it is neither of these runs and its +config is unrecorded — it cannot be placed on this table at all. + +### What SURVIVES, stated exactly + +The DuckDB matrix's re-scope condition for G6 is: *"If the range/sweep ratio +collapses below ~10× the 'top-down is a range, not a compare' claim needs +re-scoping to the specific geometry."* **110× clears that by an order of +magnitude.** So the CLAIM survives on the correct tier; the HEADLINE NUMBER does +not, and is more than halved. + +The shape survives too, and it is the part that was never the headline: the TCAM +arm is FLAT in node size on both tiers (it sweeps the whole column whatever the +node), while the range arm tracks the node. Cost follows the selection, not the +corpus. That is what makes the prefix stepless, and it is config-independent. + +### Standing rule from here + +Every timing published for this plan names its target-cpu, and any probe that +prints timings prints the realization it was built for (the parity program +already does: `arch=… avx2=… avx512f=…`). A number without that label is not a +measurement, it is an anecdote. + +--- + +## §18 — N3 / T1 gap G2 surface census (2026-09-16, prepared while N2's first chunk ran) + +Measured by brace-scoped extraction of `impl U64x8` blocks across all five +realizations, NOT by a file-wide grep. **That distinction is load-bearing:** a +file-wide grep for `cmpeq_mask` hits in `simd_avx2.rs` and `simd_scalar.rs` +come from the `U8x64` impl blocks, which sit adjacent in the same files and DO +carry compares. Read file-wide, the surface looks present. It is not. + +### What `U64x8` actually has + +| arm | representation | `impl U64x8` methods | compare-to-mask | +|---|---|---|---| +| avx512 | `__m512i` (`simd_avx512.rs:1736`) | 12 | **none** | +| avx2 | **scalar polyfill from the `avx2_int_type!` macro** — the file says so at `simd_avx2.rs:2295` | 6 (`rotate_left`, `rotate_right`, `popcnt`, `xor_popcount`, `andnot`, `ternlog`) | **none** | +| scalar | array-backed | same 6 | **none** | +| neon | `[U64x2; 4]` (`simd_neon.rs:2674`) | 13 | **none** | +| wasm | `[U64x2; 4]` (`simd_wasm.rs:1777`) | 13 | **none** | +| nightly | `u64x8` portable-simd | — | **none** | + +> **⊘ THE NIGHTLY ROW IS WRONG, and the correction reframes the whole wave +> (2026-09-16, found by the N3 worker assigned to that arm, verified here by +> brace-scoped extraction).** `src/simd_nightly/u_word_types.rs` has carried +> `U64x8::{cmpeq_mask, cmpgt_mask} -> u8` since before this wave, at lines +> 125/131 — `self.0.simd_eq(other.0).to_bitmask() as u8` and the `simd_gt` +> twin — under its own `// ── Compare -> bitmask ──` section header. It is not +> an isolated pair either: the nightly arm ships **18 such pairs across every +> width in the directory** (`u_word_types.rs` ×7 incl. `U64x4`/`U32x8`/ +> `U32x16`/`U16x32`/`U16x16`, `i_word_types.rs` ×6, `u8_types.rs` ×2, +> `i8_types.rs` ×2, `w1a_types.rs` ×1). +> +> **Why the census missed it, and the lesson is the mirror of §18's own.** +> §18 warns that a FILE-WIDE grep makes an absent surface look present +> (`cmpeq_mask` hits in `simd_avx2.rs` come from the adjacent `U8x64` block). +> This is the inverse: the nightly arm is a **directory** +> (`src/simd_nightly/*.rs`), not a `simd_.rs` file, so a census shaped +> around the single-file arms skips it entirely and a PRESENT surface looks +> absent. Both failures come from letting the search SHAPE stand in for the +> thing searched. +> +> **Three consequences, all load-bearing for N3:** +> +> 1. **This wave is not adding a capability — it is bringing the stable arms up +> to a contract the validation arm already states.** That is a materially +> different claim and a much easier one to get right. +> 2. **The nightly bodies are the CONTRACT REFERENCE.** Return width `u8` for +> 8 lanes, lane `i` → bit `i`, `simd_gt` on a `u64` element type (already +> unsigned, no bias). Every stable arm must match that, and its terse +> one-line doc style is the house convention for this family. +> 3. **An unplanned gate exists:** `scripts/masking-parity.sh nightly` builds +> the facade against this arm, so once the stable arms land, stable-vs- +> nightly is a genuine cross-realization differential rather than a +> same-author self-check. That gate was not in the plan and is stronger than +> what was. +> +> Coverage gap noted by the same worker, not fixed here: the nightly tests have +> `u64x8_cmpeq_mask_all` but **no `u64x8_cmpgt_mask` test**, where the sibling +> `U64x4` has both. Filed rather than folded in — it is that arm's own +> regression surface, not N3's. + +`U64x2`, the neon/wasm building block, has 10 methods on neon and **no compare +among them**, so composing four of those is not available either. + +### Consequence — N3 is NOT N2, and the difference decides the brief + +N2 (G1, the `u8` family) was facade-only: `U8x64::{cmpeq_mask, cmpgt_mask}` +already existed on every arm, so only the T1 wrapper was missing. **N3 has no +backend primitive on any arm.** Both layers are new. + +Scope, stated honestly rather than aspirationally: + +- **avx512 — native and cheap.** `_mm512_cmpeq_epu64_mask` / + `_mm512_cmpgt_epu64_mask` return `__mmask8` directly; 8 lanes is exactly one + byte of bitmask. This is the tier the workspace measures on (operator ruling, + §17), so it is the tier that decides whether G2 pays. +- **avx2 — a per-lane loop is the house pattern here, and that is not the same + concession as the `U8x64` defect.** On that arm `U64x8` is a *scalar polyfill + produced by a macro*: it holds no `__m256i` to exploit. The `U8x64` fix + (`3a5da8c`) was possible only because a vectorized `U8x32` sat beside it in + the same file; there is no `U64x4`-with-compares to compose here. Vectorizing + it would mean changing what the macro generates — out of N3's scope, and + named as such rather than silently skipped. +- **neon / wasm / scalar** — scalar per-lane, bit-exact via the parity program. + +### The pre-registered question the matrix asked, and #308's answer + +The matrix left G2 conditional: *"count how many intended predicates over a U64 +lane are ordered rather than equality. If the answer is zero, G2 is not a gap, +it is a correctly-scoped surface."* **PR #308 answers it: the count is ≥ 1.** +Its `find_ram_in_range` needs `lo <= offset < hi` over `u64`, and **100 % of the +real Ram-space offsets exceed 2³²**, so narrowing into the existing `i32` +ordered family is unsound. The probe worked around it by splitting the offset +into `hi32`/`lo32` and asserting the window lies inside one `hi32` bucket — +valid for that fixture, not in general. + +### Chunk plan (per `.claude/rules/agent-output-durability.md`) + +1. `U64x8::{cmpeq_mask, cmpgt_mask} -> u8` on avx512 + scalar, with a scalar + oracle test. One file each. +2. The same on neon / wasm / nightly. +3. Facade `lt/le/gt/ge_u64_to_mask`, packing `out_words[g / 8] |= (bits as u64) + << ((g % 8) * 8)` — 8 lanes per chunk, so eight chunks per word, unlike N2's + one-chunk-one-word and unlike the u32 family's four-groups-per-word. +4. `_under` siblings, then the facade re-export and a parity group. + +### 18a. ⊘ Correction to §18's scope — the composition is `64x2 × 4`, not "scalar per-lane" (operator, 2026-09-16) + +Operator: *"64x8 wird bei wasm und Skalar immer durch 64x2 \* 4 erledigt."* +§18 above wrote "scalar per-lane" for neon/wasm/scalar, which is looser than the +rule and would have produced four bespoke 8-lane bodies. Read as a build rule, +not a description, it says: **one narrow primitive per arm at `U64x2`, composed +×4 into the 8-lane result.** + +Measured state, so the rule is applied against facts rather than assumption: + +| arm | declaration | shape | +|---|---|---| +| neon | `struct U64x8(pub [U64x2; 4])` (`simd_neon.rs:2674`) | **`64x2 × 4`** | +| wasm | `struct U64x8(pub [U64x2; 4])` (`simd_wasm.rs:1777`) | **`64x2 × 4`** | +| scalar | `impl_int_type!(U64x8, u64, 8, 0u64)` (`simd_scalar.rs:527`) | flat `[u64; 8]` | +| avx2 | `avx2_int_type!` polyfill (`simd_avx2.rs:2295` names it) | flat | +| avx512 | `__m512i` (`simd_avx512.rs:1736`) | native | + +**One divergence, recorded rather than smoothed over:** on the scalar arm the +macro generates a FLAT eight-lane array today, not `[U64x2; 4]`. The rule is +still honoured without touching the macro-generated layout — the COMPOSITION is +four 2-lane groups even where the STORAGE is flat, so every non-avx512 body has +the identical shape and only the element access differs. Changing what +`impl_int_type!` emits is a separate decision with a much wider blast radius and +is not smuggled into N3. + +Consequence for the chunk plan in §18: chunk 1 and 2 merge into "add +`U64x2::{cmpeq_mask, cmpgt_mask} -> u8` (2 bits) per arm, then compose ×4". +`U64x2` has **10 methods on neon and no compare among them, and none at all on +the other arms** (measured), so that narrow primitive is new everywhere too — +but it is written once per arm instead of once per arm per width, and the ×4 +fold is shared. + +### 18b. N3's per-arm compare availability — verified, not assumed (2026-09-16) + +Operator: *"GitHub LLVM hat alle CPU SIMD ops dokumentiert"* — so look them up +instead of guessing. Done. Two rows compiled on this toolchain, two read from +stdarch's own source/spec; provenance is per row because it differs. + +| arm | unsigned 64-bit ORDERED compare | equality | how verified | +|---|---|---|---| +| **AVX-512** | `_mm512_cmpgt_epu64_mask` → `__mmask8` | `_mm512_cmpeq_epu64_mask` | **compiled** here under `avx512f` | +| **AVX2** | **ABSENT** — only the SIGNED `_mm256_cmpgt_epi64` | `_mm256_cmpeq_epi64` (bit equality is sign-agnostic) | **compile error** here: *"cannot find function `_mm256_cmpgt_epu64`"* | +| **NEON** | `vcgtq_u64` → **`cmhi`**, `vcgeq_u64` → **`cmhs`** | `vceqq_u64` → `cmeq` | stdarch `aarch64.spec.yml`, the generated-intrinsics spec | +| **WASM** | **ABSENT** — only `i64x2_{lt,le,gt,ge}`, signed; the file says so in its own doc (*"as if they were two vectors of 2 sixty-four-bit SIGNED integers"*) | `u64x2_eq` / `u64x2_ne` exist, as aliases of the i64x2 forms | stdarch `wasm32/simd128.rs` | +| scalar | plain `>` on `u64` | `==` | n/a | + +### What this decides, and it corrects an assumption I would otherwise have made + +**NEON gets REAL vector instructions, not a scalar fold.** `cmhi` / `cmhs` +support the 2D (64-bit) element form, so the `[U64x2; 4]` fan-out (§18a) lowers +to four genuine compares per arm. Had I not checked I would have written the +neon arm as a per-lane loop on the grounds that `U64x2` "has no compare today" — +which is true of this repo's wrapper and false of the ISA underneath it. + +**AVX2 and WASM need the SAME sign-bias trick, and it is already in the tree.** +Neither has an unsigned ordered 64-bit compare, so both flip the sign bit +(`^ 0x8000_0000_0000_0000`) and use the signed instruction — ordering is +preserved under that XOR. This is not an invention: `U8x32::cmpgt_mask` +(`simd_avx2.rs:2746-2754`) already documents and implements exactly this at byte +width, *"AVX2 only has signed `_mm256_cmpgt_epi8`, so we XOR both operands with +`0x80` to convert unsigned ↔ signed (preserves ordering for unsigned compare)."* +The N3 worker cites that precedent rather than deriving it. + +**Equality needs no bias anywhere.** Bit equality is sign-agnostic, which is why +wasm ships `u64x2_eq` as a plain alias. Only the ordered ops are affected. + +### Honest scope of this table + +The AVX-512 and AVX2 rows are compile-verified on the toolchain we actually +build with — the strongest form available here. The NEON and WASM rows are read +from stdarch's own generating spec and source, **not** compiled, because no +aarch64 or wasm32 target is installed in this container (`rustup target list +--installed` returns x86_64 only) and `rust-src` is absent so the vendored +stdarch cannot be grepped locally either. A cross-target `cargo check` is the +stronger gate and is the follow-up if either row is ever load-bearing for a +shipped decision rather than for planning. + +> **⊘ CORRECTION, same day, before any N3 code was written: the premise of the +> paragraph above is FALSE, and the stronger gate was available the whole time.** +> `rustup target list --installed` returns **three** targets here, not one: +> `aarch64-unknown-linux-gnu`, `wasm32-unknown-unknown`, `x86_64-unknown-linux-gnu`. +> Both cross arms compile-check clean on the unmodified tree: +> +> ```sh +> env -u RUSTFLAGS cargo check -p ndarray --lib --no-default-features --features std \ +> --target aarch64-unknown-linux-gnu # exit 0 +> RUSTFLAGS="-C target-feature=+simd128" cargo check -p ndarray --lib \ +> --no-default-features --features std --target wasm32-unknown-unknown # exit 0 +> ``` +> +> Two details make these REAL gates rather than vacuous ones, and both had to be +> checked rather than assumed. `simd_neon` and `simd_wasm` are declared in +> `lib.rs` behind **`#[cfg(feature = "std")]` alone, with no target cfg** +> (`lib.rs:416-418`, `:438-440`), so their mere presence in an x86 build proves +> nothing — the arch gate is INSIDE each file (`simd_neon.rs` per-item +> `#[cfg(target_arch = "aarch64")]`; `simd_wasm.rs:65` one +> `#[cfg(all(target_arch = "wasm32", target_feature = "simd128"))] pub mod +> wasm32_simd`). And `--features std` is load-bearing: `simd_masking_ops.rs` is +> std-gated, so a bare `--no-default-features` check compiles neither the facade +> nor its callers. A wasm check WITHOUT `+simd128` compiles the scalar arm and +> reports success while never touching `simd_wasm.rs` at all — the exact false +> gate this plan keeps finding elsewhere. +> +> Residual limit, stated precisely: NEON can be **checked** but not **run** (no +> `qemu-aarch64` in this container); WASM can be **both** (`node` is present, and +> `scripts/masking-parity.sh wasm` drives it). So the honest scope is *"NEON +> compile-verified, WASM compile- and run-verified"*, not *"both read from a +> spec"*. The stdarch readings in the table stay useful — they say which +> INSTRUCTION each intrinsic lowers to, which a `cargo check` cannot tell you — +> but they are no longer the only evidence for those two rows. diff --git a/.claude/rules/agent-cargo-hygiene.md b/.claude/rules/agent-cargo-hygiene.md index 35c1bc4f..341b63c8 100644 --- a/.claude/rules/agent-cargo-hygiene.md +++ b/.claude/rules/agent-cargo-hygiene.md @@ -14,9 +14,25 @@ build residue and twelve cold compiles competing for the same cores. - **Sonnet fleet agents: do NOT each run a full compile.** They edit code and reason; they must not spawn isolated worktrees or trigger their own cold `cargo build`/`check`/`test` that each grow a separate 7 GB `target/`. - - "tests yes, compile no": a targeted `cargo test`/`clippy` against the - **shared** workspace `target/` is fine; a bare compile-only - (`cargo check`/`build`) is wasted residue — clippy already compiles. + - ⊘ **SUPERSEDED — this file used to carve out "tests yes, compile no": a + targeted `cargo test`/`clippy` against the shared `target/` is fine; + clippy already compiles.** Both halves are now wrong, and the file + contradicted its own BACKEND POLLUTION section below (which says the + prohibition is absolute) for as long as the carve-out stood. Corrected + 2026-09-16 after coderabbit flagged the inconsistency on PR #309: + + 1. **Operator ruling: workers do not run cargo. At all.** Not `build`, + not `check`, not `test`, not `clippy`. The pollution argument below is + what makes this absolute rather than a budget — a worker's plain + `cargo test` takes `.cargo/config.toml` (v3) and REPLACES whatever + realization the orchestrator last built, so the next probe reports a + tier nobody can reconstruct. + 2. **"clippy already compiles" is false** (operator correction to my own + framing): clippy type-checks and lints, it does not produce a runnable + artifact. Practical consequence for the ORCHESTRATOR's own gates, which + is where cargo is still allowed: a green `cargo clippy` proves types and + lints, never that the thing builds and runs. Evidence for a landing is a + `cargo run`/`cargo test` result, not a lint. - **Verification is centralised.** The orchestrator (Opus) runs `cargo fmt` + `cargo clippy` + `cargo test` **once**, in the single shared `target/`, after the fleet's edits land. One build, not twelve. @@ -31,3 +47,27 @@ build residue and twelve cold compiles competing for the same cores. `cargo clippy -p `, `cargo test -p `) — keeping the tree `cargo clippy -- -D warnings`-clean (see `CLAUDE.md` Hard Rules) and `cargo fmt`-clean on the pinned toolchain, with no residue blowup. + + +## The second reason, and it is the stronger one: BACKEND POLLUTION + +(Operator, 2026-09-16: *"Würdest du mit jedem worker kompilieren hättest du +target residue backend pollution."* Residue is the cost. Pollution is the +correctness failure, and it was not written down here before.) + +The one shared `target/` holds **one realization at a time**. A target-cpu +change invalidates the cache, so a worker running a plain `cargo test` — which +takes `.cargo/config.toml`, i.e. **v3/AVX2** — after the orchestrator built +`--config .cargo/config-v4.toml` does not merely ADD residue. It **replaces** +the v4 artifacts with v3 ones. The next probe then reports whichever tier +compiled last, and nothing in its output says which. + +That is exactly the defect recorded as plan §17 (`gemm-ternlog-mask-consolidation-v1.md`), +one scale down: a timing without its target-cpu is an anecdote. The difference +is severity. On the orchestrator's own runs the tier is merely UNLABELLED, and a +re-run under a named config fixes it. With N workers compiling on their own +schedule it becomes UNATTRIBUTABLE — no one can reconstruct which backend a +number came from, because the interleaving is gone. + +**So: one shared `target/`, one realization, one compiler — the orchestrator.** +Workers edit. This is why the prohibition is absolute rather than a budget. diff --git a/.claude/rules/agent-output-durability.md b/.claude/rules/agent-output-durability.md new file mode 100644 index 00000000..c09629ab --- /dev/null +++ b/.claude/rules/agent-output-durability.md @@ -0,0 +1,49 @@ +# Agent Output Durability — `tee -a`, and why a brief's SIZE is a reliability parameter + +## The incident, 2026-09-16 + +Two Sonnet workers were dispatched in the same message, on disjoint files. + +| worker | brief | outcome | +|---|---|---| +| A | ONE file, ONE goal (vectorize two methods) | returned with a full report and a landed edit | +| B | THREE parts, FOUR files, twelve functions + a parity group + a correction | **vanished after 3-5 hours having written NOTHING** | + +`ListAgents` showed no running agent. There was no partial file, no scratch +output, no trace — the work was not slow, it was lost, and nothing about it was +recoverable. + +## Rule 1 — every worker tees its progress to its own tag-file + +A worker that writes only at the end has a single point of failure: a timeout, a +tool error, or an encoding problem takes the whole run with it. Build output +incrementally and let each chunk land on disk as it is produced: + +```sh +: > "$TAG" # init once +cat <<'CHUNK' | tee -a "$TAG" > /dev/null +... +CHUNK +``` + +Each chunk is visible in the session log AND on disk, so partial progress +survives a timeout and the orchestrator can salvage it. The sibling `MedCare-rs` +repo already carries this as its own discipline for any file over ~200 lines, +for exactly the same reason ("partial progress survives a timeout; the final +file is assembled atomically before commit"). + +**One writer per file.** A worker tees to ITS OWN tag-file, never to a shared +log — a shared append-log is a lost-write race, which is the same defect the +substrate removed at runtime and must not be re-created one layer up +(lance-graph `E-AGENT-LOG-SHARED-SINK-ANTIPATTERN-1`). + +## Rule 2 — brief size is a reliability parameter, not just a cost + +The A/B contrast above is one observation, not a law, but it points one way and +it is cheap to obey: **one chunk = one file, one deliverable.** Split a wave into +sequential chunks and dispatch them one at a time. A brief that names three parts +and four files is not "efficient", it is a bet that nothing goes wrong for hours. + +Corollary for the orchestrator: prefer a chunk small enough that you could do it +yourself if the dispatch is lost. Then a lost dispatch costs one chunk, not an +afternoon. diff --git a/CLAUDE.md b/CLAUDE.md index f2aad0e9..51c0dd52 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,8 +28,31 @@ This project uses specialized agents in `.claude/agents/`. Follow these rules: - OpenBLAS and MKL are **mutually exclusive** feature gates. Never both. - Zero-cost abstractions: generics monomorphize, no `Box` in hot paths. - Every `unsafe` block needs a `// SAFETY:` comment. +- **`&&`-chain a commit to the edit that produces it — never sequence it + after.** An anchor assertion in an edit script protects the FILE; it does + not protect the RECORD. Measured here 2026-09-16: an edit script's assertion + fired correctly (a mid-line anchor that did not match), the script aborted, + no bad edit landed — and the `git commit` that followed it ran anyway, + shipping a message claiming two files while `git show --stat` showed one. + For one commit a plan was documented as updated while it was not. The + narrative remedy (`git show` the diff before claiming it) is real but + optional; `python3 edit.py && git add … && git commit …` is mechanical and + cannot be forgotten. - All public APIs need `///` doc comments with examples. - `cargo clippy -- -D warnings` must pass. +- **Every compile runs with `CARGO_PROFILE_DEV_DEBUG=0`** (operator, 2026-09-16: + *"use debug 0"*). Not a preference — **debug info is the disk hog, not the + code**, and this container's writable allowance is a fixed per-session budget + that presents as `No space left on device` mid-link, not as a full disk. + Measured here the same day on the identical tree and the identical test run: + `target/debug` is **1.9 GB with debug info and 291 MB without** — a 6.5× + cut, for a run that passed 2319 tests either way. Export it (plus + `CARGO_PROFILE_TEST_DEBUG=0` and `CARGO_INCREMENTAL=0`) as ENV, never as a + profile edit in `Cargo.toml`: the rule governs the agent's compiles, not the + profile a human commits. `--release` is NOT a substitute — it is slower to + build and the test runs need the dev path. And because a profile change + invalidates the whole cache, **delete `target/debug` before switching rather + than growing a second copy beside it.** - **All new public `pub fn` in `src/simd_*.rs` follows the W1a consumer contract** at `.claude/knowledge/vertical-simd-consumer-contract.md` — struct methods on typed wrappers, closure-parameterized batch primitives, all three backends (AVX*/NEON/scalar) implemented, parity test mandatory, saturating/overflow semantics documented. The Ada stack (lance-graph + downstream) enforces "all SIMD from `ndarray::simd`" via its `simd-savant` agent; missing primitives in ndarray force consumer-side raw-intrinsic violations, so additions here are gating the consumer-side sweep. **VPABSB does NOT saturate `i8::MIN`** — see § "VPABSB correction" in the contract doc before implementing `saturating_abs` or any abs primitive. ## Compaction Preservation @@ -81,9 +104,80 @@ src/ ### New Modules - `src/hpc/styles/` — 34 cognitive primitives (rte, htd, smad, tcp, irs, mcp, tca, cdt, mct, lsi, pso, cdi, cws, are, tcf, ssr, etd, amp, zcf, hpm, cur, mpc, ssam, idr, spp, icr, sdd, dtmf, hkf). Each is `fn(Base17, NarsTruth) → result`. 49 tests. - `src/hpc/causal_diff.rs` — CausalEdge64 (u64 packed), scaffold_to_palette3d_layers(), quality scoring (GOOD/BAD/UNCERTAIN), NARS self-reinforcement LoRA, PAL8 serialization (4101 bytes). -- `.cargo/config.toml` — `target-cpu=x86-64-v4` (AVX-512 mandatory). +- **Build config — AVX-512 is NOT the default, and believing it is corrupts + measurements.** `.cargo/config.toml` sets **`x86-64-v3` (AVX2)**, deliberately: + it is the portable CI/distribution baseline, and its own comment explains why + (without a v3 floor the AVX2 intrinsics in `simd_avx2.rs` SIGILL). A plain + `cargo build`/`run`/`test` therefore measures **v3**. + **For AVX-512 you must ask for it, every time:** + + ```sh + env -u RUSTFLAGS cargo --config .cargo/config-v4.toml run --release --example + ``` + + `env -u RUSTFLAGS` is load-bearing: a RUSTFLAGS env var REPLACES every + cargo-config rustflags entry, so it silently drops `-Ctarget-cpu=x86-64-v4` + and the arm measures v3 while claiming v4 (the trap `scripts/masking-parity.sh` + documents). Verify the arm you got — the parity program prints + `avx512f=true|false`, and any probe that reports timings should too. + `.cargo/config-avx512.toml` is the stricter Sapphire Rapids EXECUTION config + (VNNI/BF16/FP16/AMX) and SIGILLs on earlier AVX-512 silicon; `config-native.toml` + resolves the host CPUID. + (This line previously claimed `config.toml` was v4 "AVX-512 mandatory" — it + never was, and that error made a whole measurement arc read v3 as v4. Corrected + 2026-09-16 against `.cargo/config.toml:83`.) + + **The v4 config also carries `-D warnings`, which makes a DISABLE RUN fail + in a way that reads as success.** A disable typically removes a use of + something; the variable it fed then goes unused; `-D warnings` promotes that + to a hard error; the test binary is never built, so the run emits no `test + result:` line at all. Piped through a `grep` for the failing assertion, "did + not compile" and "the guard was not load-bearing" look identical — the + workspace's known trap (*a disable that does not APPLY is indistinguishable + from a guard that does not bind*) with a second door. Measured 2026-09-16 on + the `gt_u8_to_mask` signed-compare disable: it silently produced `error: + unused variable: threshold_v` and I nearly recorded the falsifier as inert. + **Always read the disable run's exit status and the `test result:` line + itself, never only a grep of its assertions** — and prefix, don't delete, + when a disable orphans a binding. - `src/simd.rs` — compile-time AVX-512 dispatch via `cfg(target_feature = "avx512f")`. +### The parity arms are ALL reachable here — never report one as blocked without apt + +`scripts/masking-parity.sh` takes `native | nightly | wasm | wasm-scalar | +neon-qemu`, and the cross arms are an `apt-get` away, not an environment +limit. Measured 2026-09-16: `neon-qemu` failed with a bare +`No such file or directory (os error 2)` — which reads as "this target is not +available here" and is in fact a **missing linker**, then a missing +interpreter, in two separate steps: + +```sh +sudo apt-get update +sudo apt-get install -y gcc-aarch64-linux-gnu qemu-user-static +``` + +`qemu-user` alone is NOT enough: the script invokes `qemu-aarch64-static`, +and the dynamic `qemu-aarch64` from `qemu-user` leaves a second, differently +worded failure (`command not found`) that looks like a fresh problem rather +than the same one. Install `qemu-user-static`. + +**And `native` is the AVX2 arm, not the AVX-512 one** — it takes +`.cargo/config.toml` (v3), so a green `native` leaves every `_mm512_*` body +unwitnessed. The AVX-512 arm is the binary run under the v4 config directly: + +```sh +cd crates/simd-masking-parity +env -u RUSTFLAGS cargo --config ../../.cargo/config-v4.toml run --release +``` + +Read the program's own header line to confirm which arm you actually got +(`avx512f=true`, `neon=true`, …) — that line exists precisely because the +config can silently not apply. Five of the six realizations are reachable +without nightly (AVX2, AVX-512, NEON, wasm-simd128, wasm-scalar); only +`nightly-simd` needs a toolchain this repo does not pin. The same lesson the +sibling `lance-graph-java` records for the JDK: **a stale index or a missing +helper binary reporting absence is not evidence of absence.** + ### Key Data - 5 Qwen3.5 models indexed: 685 MB bgz7 from 201 GB BF16 safetensors - GitHub Release `v0.1.0-bgz-data` on AdaWorldAPI/lance-graph: 41 bgz7 files diff --git a/crates/simd-masking-parity/src/lib.rs b/crates/simd-masking-parity/src/lib.rs index 6f023440..6a849988 100644 --- a/crates/simd-masking-parity/src/lib.rs +++ b/crates/simd-masking-parity/src/lib.rs @@ -18,22 +18,28 @@ //! predicate→mask, `0x6xx` mask algebra, `0x7xx` care-match, `0x8xx` masked //! reductions and blend, `0x9xx` `mask_shift_morton` (the Morton hex //! neighbour shift, D-GTM-1m), `0xAxx` the gated `*_to_mask_under` -//! predicates (mask-risc `Pred { under }`, D-MRX-0). `main.rs` (native / qemu) and `selfcheck()` -//! (the wasm cdylib export, driven by `run.mjs`) both call [`run`]. +//! predicates (mask-risc `Pred { under }`, D-MRX-0), `0xBxx` `mask_set_range` +//! (the range WRITE, N1), `0xCxx` the unsigned `u8` / `u64` compare→mask +//! family (`eq`/`ne`/`gt`/`ge`/`lt`/`le`, N2/N3 — built earlier but never +//! exercised by this program until now). `main.rs` (native / qemu) and +//! `selfcheck()` (the wasm cdylib export, driven by `run.mjs`) both call +//! [`run`]. use ndarray::simd::{ blend_i32, eq_i32_to_mask, eq_i32_to_mask_under, eq_u32_strided_to_mask, eq_u32_to_mask, eq_u32_to_mask_under, - ge_i32_to_mask, ge_i32_to_mask_under, gt_i32_to_mask, gt_i32_to_mask_under, le_i32_to_mask, le_i32_to_mask_under, - lt_i32_to_mask, lt_i32_to_mask_under, mask_all, mask_and, mask_and_assign, mask_andnot, mask_andnot_assign, - mask_any, mask_not, mask_not_assign, mask_or, mask_or_assign, mask_shift_morton, mask_ternlog, mask_ternlog_assign, - mask_xor, mask_xor_assign, masked_max_i32, masked_min_i32, masked_strided_group_sum, masked_sum_i32, - ne_i32_to_mask, ne_i32_to_mask_under, ne_u32_to_mask, ne_u32_to_mask_under, ternary_match_strided_to_mask, - ternary_match_u32_to_mask, ternary_match_u32_to_mask_under, ternary_match_u64_to_mask, - ternary_match_u64_to_mask_under, ternlog, I32x16, MortonDir, U32x16, U64x8, + eq_u64_to_mask, eq_u8_to_mask, ge_i32_to_mask, ge_i32_to_mask_under, ge_u64_to_mask, ge_u8_to_mask, gt_i32_to_mask, + gt_i32_to_mask_under, gt_u64_to_mask, gt_u8_to_mask, le_i32_to_mask, le_i32_to_mask_under, le_u64_to_mask, + le_u8_to_mask, lt_i32_to_mask, lt_i32_to_mask_under, lt_u64_to_mask, lt_u8_to_mask, mask_all, mask_and, + mask_and_assign, mask_andnot, mask_andnot_assign, mask_any, mask_not, mask_not_assign, mask_or, mask_or_assign, + mask_set_range, mask_shift_morton, mask_ternlog, mask_ternlog_assign, mask_xor, mask_xor_assign, masked_max_i32, + masked_min_i32, masked_strided_group_sum, masked_sum_i32, ne_i32_to_mask, ne_i32_to_mask_under, ne_u32_to_mask, + ne_u32_to_mask_under, ne_u64_to_mask, ne_u8_to_mask, ternary_match_strided_to_mask, ternary_match_u32_to_mask, + ternary_match_u32_to_mask_under, ternary_match_u64_to_mask, ternary_match_u64_to_mask_under, ternlog, I32x16, + MortonDir, U32x16, U64x8, }; /// Number of check groups [`run`] executes (for the log line only). -pub const CHECKS: usize = 10; +pub const CHECKS: usize = 12; /// The wasm export: identical to [`run`], `extern "C"` so `run.mjs` can call it. #[no_mangle] @@ -46,7 +52,7 @@ pub fn run() -> u32 { let groups: [fn() -> Result<(), u32>; CHECKS] = [ check_ternlog_all_tables, check_u64x8_algebra, check_i32x16_compare, check_predicates_to_mask, check_mask_algebra, check_care_match, check_masked_reductions, check_blend, check_morton_shift, - check_predicates_under, + check_predicates_under, check_set_range, check_unsigned_compare_to_mask, ]; for g in groups { if let Err(code) = g() { @@ -943,3 +949,173 @@ fn check_morton_shift() -> Result<(), u32> { } Ok(()) } + +// ── 0xBxx: mask_set_range — the range WRITE, no per-bit loop (N1) ─────────── +// +// The reference below is a from-scratch bit-serial writer over `[lo, hi)` +// membership — it does not call `mask_set_range` and does not reuse +// `reference_mask`'s predicate-over-index shape, because a range write's law +// is membership in an interval, not a predicate over `values[i]`. + +fn set_range_reference(n_words: usize, lo: usize, hi: usize) -> Vec { + let mut w = vec![0u64; n_words]; + for i in lo..hi { + w[i / 64] |= 1u64 << (i % 64); + } + w +} + +fn check_set_range() -> Result<(), u32> { + let mut rng = SplitMix64(0xB000_0000_0006); + for &n_words in &[1usize, 2, 3, 4, 8] { + let capacity = n_words * 64; + // The boundary shapes the primitive's own doc names, plus randomized + // fuzz within capacity. `lo == hi` (both 0 and both `capacity`) is + // the legal empty range; the rest straddle every word-alignment case. + let mut cases: Vec<(usize, usize)> = + vec![(0, 0), (capacity, capacity), (0, capacity), (0, 1), (capacity - 1, capacity), (5, 6)]; + if n_words >= 2 { + cases.push((60, 70)); // adjacent words, neither aligned + cases.push((64, 128)); // both endpoints word-aligned + cases.push((0, 64)); // lo aligned, leaves surplus words unset + } + for _ in 0..12 { + let lo = (rng.next() as usize) % (capacity + 1); + let span = (rng.next() as usize) % (capacity + 1 - lo); + cases.push((lo, lo + span)); + } + for (lo, hi) in cases { + let mut got = vec![u64::MAX; n_words]; // pre-dirtied: an OR-er fails immediately + mask_set_range(&mut got, lo, hi); + let want = set_range_reference(n_words, lo, hi); + if got != want { + return Err(0xB00); + } + } + // Anti-vacuity: a genuinely non-trivial range sets EXACTLY `hi - lo` + // bits, not "some" bits — catches an always-set-everything or + // always-set-nothing implementation that could otherwise still pass + // every case above. + if capacity >= 8 { + let (lo, hi) = (3, capacity - 2); + let mut got = vec![0u64; n_words]; + mask_set_range(&mut got, lo, hi); + let popcount: u32 = got.iter().map(|w| w.count_ones()).sum(); + if popcount as usize != hi - lo { + return Err(0xB01); + } + } + } + Ok(()) +} + +// ── 0xCxx: unsigned u8 / u64 compare → mask — N2/N3's untested twelve ────── +// +// `eq_i32_to_mask` et al. (0x5xx) are SIGNED comparisons; `u8` and `u64` have +// no sign, and `simd_masking_ops.rs`'s own doc comments name the risk this +// group exists to catch: an arm that reaches its ordering by flipping the +// top bit and running the SIGNED compare instruction the hardware actually +// offers (the trick some realizations use for `gt`/`ge`/`lt`/`le` at these +// widths) gets every element with that bit set BACKWARDS if it forgets the +// flip. So the value sets below sit ON that bit — `0x80` for `u8`, `1 << 63` +// for `u64` — rather than merely including some large number somewhere in +// the fixture. +// +// The two widths also differ in PACKING, not just in domain. `u8`'s 64-lane +// `U8x64` register is exactly one output word wide, so `LENS` (which +// straddles the 64-ROW boundary) already straddles `u8`'s only packing seam +// too — reused directly below, no new lengths needed. `u64`'s `U64x8` +// register is 8 lanes, and EIGHT of those pack into one output word +// (`out_words[g / 8] |= (bits as u64) << ((g % 8) * 8)`) — a seam `LENS` +// alone never isolates, since none of its six values sit at a bare multiple +// of 8 outside the already-covered 0/64/128-adjacent cases. `U64_CMP_LENS` +// adds 7 / 8 / 9: a register short of full, exactly one register with +// nothing left over, and one register plus a one-element tail that must +// land at BYTE 1 of the SAME word rather than spilling into a new one — the +// placement a naive `g * 8` (forgetting the `% 8`) would get wrong starting +// at exactly this length. + +/// Row lengths for the `u64` half of [`check_unsigned_compare_to_mask`]: +/// [`LENS`] plus 7 / 8 / 9, which straddle the 8-lane `U64x8` register +/// boundary that `LENS` alone never isolates (see the banner above). +const U64_CMP_LENS: [usize; 9] = [0, 1, 7, 8, 9, 63, 64, 65, 130]; + +/// `u8` operands spanning the full unsigned range, with a fixed prefix +/// sitting exactly on `0x7F` / `0x80` — the byte pair a SIGNED `i8` compare +/// would misorder relative to unsigned `u8` ordering — before random fill. +fn u8_values(n: usize, rng: &mut SplitMix64) -> Vec { + let fixed = [0u8, 0xFF, 0x7F, 0x80, 0x81, 0xFE, 1, 7, 7]; + (0..n) + .map(|i| if i < fixed.len() { fixed[i] } else { rng.next() as u8 }) + .collect() +} + +/// `u64` operands including the sign-bit boundary a SIGNED `i64` compare +/// would misorder: `1 << 63` itself, one below it (the largest value still +/// "positive" under a signed read), one above it, and `u64::MAX`. +fn u64_values(n: usize, rng: &mut SplitMix64) -> Vec { + let fixed = [0u64, u64::MAX, 1u64 << 63, 0x7FFF_FFFF_FFFF_FFFF, (1u64 << 63) + 1, 0x7FFF_FFFF_FFFF_FFFE, 1, 7, 7]; + (0..n) + .map(|i| if i < fixed.len() { fixed[i] } else { rng.next() }) + .collect() +} + +fn check_unsigned_compare_to_mask() -> Result<(), u32> { + let mut rng = SplitMix64(0xC000_0000_0007); + + // u8: `LENS` already straddles its only packing seam (see the banner + // above), so it is reused directly rather than duplicated. + let u8_thresholds = [0u8, 1, 0x7F, 0x80, 0x81, 0xFE, 0xFF, 7]; + for &n in &LENS { + let out_len = words_for(n) + 1; + let vals = u8_values(n, &mut rng); + let mut out = vec![u64::MAX; out_len]; + for (k, &t) in u8_thresholds.iter().enumerate() { + let k = k as u32; + macro_rules! pred8 { + ($f:ident, $op:tt, $code:expr) => {{ + out.iter_mut().for_each(|w| *w = u64::MAX); + $f(&vals, t, &mut out); + if out != reference_mask(n, out_len, |i| vals[i] $op t) { + return Err($code | k); + } + }}; + } + pred8!(eq_u8_to_mask, ==, 0xC00); + pred8!(ne_u8_to_mask, !=, 0xC10); + pred8!(lt_u8_to_mask, <, 0xC20); + pred8!(le_u8_to_mask, <=, 0xC30); + pred8!(gt_u8_to_mask, >, 0xC40); + pred8!(ge_u8_to_mask, >=, 0xC50); + } + } + + // u64: `LENS` plus the register-boundary lengths 7 / 8 / 9 (see the + // banner above). + let u64_thresholds = + [0u64, 1, 0x7FFF_FFFF_FFFF_FFFF, 1u64 << 63, (1u64 << 63) + 1, 0x7FFF_FFFF_FFFF_FFFE, u64::MAX, 7]; + for &n in &U64_CMP_LENS { + let out_len = words_for(n) + 1; + let vals = u64_values(n, &mut rng); + let mut out = vec![u64::MAX; out_len]; + for (k, &t) in u64_thresholds.iter().enumerate() { + let k = k as u32; + macro_rules! pred64 { + ($f:ident, $op:tt, $code:expr) => {{ + out.iter_mut().for_each(|w| *w = u64::MAX); + $f(&vals, t, &mut out); + if out != reference_mask(n, out_len, |i| vals[i] $op t) { + return Err($code | k); + } + }}; + } + pred64!(eq_u64_to_mask, ==, 0xC60); + pred64!(ne_u64_to_mask, !=, 0xC70); + pred64!(lt_u64_to_mask, <, 0xC80); + pred64!(le_u64_to_mask, <=, 0xC90); + pred64!(gt_u64_to_mask, >, 0xCA0); + pred64!(ge_u64_to_mask, >=, 0xCB0); + } + } + Ok(()) +} diff --git a/examples/hex_tenant_mq_probe.rs b/examples/hex_tenant_mq_probe.rs index fdba8527..f95b83e1 100644 --- a/examples/hex_tenant_mq_probe.rs +++ b/examples/hex_tenant_mq_probe.rs @@ -48,13 +48,14 @@ //! cargo run --release --example hex_tenant_mq_probe --features std use std::alloc::{GlobalAlloc, Layout, System}; +use std::hint::black_box; use std::sync::atomic::{AtomicUsize, Ordering}; use std::time::Instant; use ndarray::simd::ternlog::{AND3, OR2_AND}; use ndarray::simd::{ - gt_i32_to_mask, mask_and, mask_shift_morton, mask_ternlog_assign, popcount_batch_u64, ternary_match_u32_to_mask, - MortonDir, + gt_i32_to_mask, gt_u8_to_mask, mask_and, mask_set_range, mask_shift_morton, mask_ternlog_assign, + popcount_batch_u64, ternary_match_u32_to_mask, MortonDir, }; // ── counting allocator: the 0k instrument ──────────────────────────────────── @@ -181,27 +182,16 @@ fn set(words: &mut [u64], i: usize) { /// A trie node at nibble level `level` (0..=4) with prefix `p`: rows /// `[p << shift, (p+1) << shift)`. The fixed spatial distribution makes the -/// reveal a range write, not a compare. +/// reveal a range write, not a compare — `ndarray::simd::mask_set_range` is +/// exactly that primitive (N1, +/// `.claude/plans/gemm-ternlog-mask-consolidation-v1.md` §16.6), so the +/// range write itself lives there now; this function only computes the +/// `(lo, hi)` node bounds. fn range_reveal(level: u32, p: u32, out: &mut [u64]) -> (usize, usize) { let shift = 16 - 4 * level; let lo = (p as usize) << shift; let hi = ((p + 1) as usize) << shift; - for w in out.iter_mut() { - *w = 0; - } - // inclusive last word; the run may start and end mid-word - let (w0, w1) = (lo >> 6, (hi - 1) >> 6); - let head = u64::MAX << (lo & 63); - let tail = u64::MAX >> (63 - ((hi - 1) & 63)); - if w0 == w1 { - out[w0] = head & tail; - } else { - out[w0] = head; - for w in out[w0 + 1..w1].iter_mut() { - *w = u64::MAX; - } - out[w1] = tail; - } + mask_set_range(out, lo, hi); (lo, hi) } @@ -451,8 +441,18 @@ fn masks_equal_axial(mask: &[u64], reference: &[bool]) -> bool { }) } +/// Run to a 50 ms floor and report ns per call. +/// +/// **Every closure passed here must `black_box` its output.** `timed` repeats +/// the call with nothing reading the result, so a write-only closure is dead +/// code an optimizing build may eliminate or hoist — and the damage is not +/// symmetric. Reported on PR #309: the widened-i32 arms happened to be +/// protected by a LATER read (the spread loop consumes `elig`; the parity +/// `assert_eq!` consumes `m_new`) while the native-u8 arms had no reader at +/// all, so the very ratio the probe exists to measure could have been the +/// optimizer deleting one side. Protect BOTH sides or the comparison is not +/// one. fn timed(mut f: F) -> f64 { - // run to a 50 ms floor; report ns per call let mut reps = 1usize; loop { let t = Instant::now(); @@ -483,9 +483,23 @@ fn main() { }) .collect(); let thr: u8 = 96; // ~62% of rails permeable - // Column views for the SIMD compare. The T1 compare is i32-wide today; a - // u8 column compare is a T1 addition (stated, not hidden). Widening costs - // 4× the bandwidth, so `n_gen` below is an UPPER bound on the reveal term. + + // Column views for the SIMD compare, BOTH widths, so the widening cost is + // measured in one process instead of argued across two runs. + // + // This probe's original note read: *"The T1 compare is i32-wide today; a + // u8 column compare is a T1 addition (stated, not hidden). Widening costs + // 4× the bandwidth, so `n_gen` below is an UPPER bound on the reveal + // term."* That T1 addition landed (N2/G1, `gt_u8_to_mask`), so the upper + // bound can be replaced by a measurement — which is the falsifier the plan + // pre-registered for G1: build the u8 comparator, re-run this probe, and + // **if the re-chain does not move, the widening was never the cost.** + // + // The rail byte is the NATIVE column: `r[2 * d]` is already `u8`, and the + // `as i32` below is the whole widening. + let perm_cols_u8: Vec> = (0..DIRS) + .map(|d| rails.iter().map(|r| r[2 * d]).collect()) + .collect(); let perm_cols: Vec> = (0..DIRS) .map(|d| rails.iter().map(|r| r[2 * d] as i32).collect()) .collect(); @@ -495,9 +509,32 @@ fn main() { let mut elig: [Vec; DIRS] = std::array::from_fn(|_| vec![0u64; WORDS]); let t_gen = timed(|| { for d in 0..DIRS { - gt_i32_to_mask(&perm_cols[d], thr as i32, &mut elig[d]); + gt_i32_to_mask(black_box(&perm_cols[d]), black_box(thr as i32), &mut elig[d]); + black_box(&elig[d]); } }); + + // The same six masks off the NATIVE u8 columns. Gate first, time second: + // a timing comparison between two operations that do not produce the same + // answer measures nothing, so bit-identity is asserted before any number + // is printed. + let mut elig_u8: [Vec; DIRS] = std::array::from_fn(|_| vec![0u64; WORDS]); + for d in 0..DIRS { + gt_u8_to_mask(&perm_cols_u8[d], thr, &mut elig_u8[d]); + assert_eq!(elig_u8[d], elig[d], "u8 and widened-i32 eligibility masks differ on rail {d}"); + } + let t_gen_u8 = timed(|| { + for d in 0..DIRS { + gt_u8_to_mask(black_box(&perm_cols_u8[d]), black_box(thr), &mut elig_u8[d]); + black_box(&elig_u8[d]); + } + }); + println!( + "M1b generation NATIVE u8: 6 masks = {t_gen_u8:.0} ns ({:.0} ns/mask, {:.2} ns/row) — {:.2}× the widened i32 arm", + t_gen_u8 / DIRS as f64, + t_gen_u8 / (DIRS * N) as f64, + t_gen / t_gen_u8 + ); println!( "M1b generation: 6 eligibility masks from 6 columns = {:.0} ns ({:.0} ns/mask, {:.2} ns/row)", t_gen, @@ -521,10 +558,12 @@ fn main() { assert!(ok, "reveal gate FAILED at level {level}"); let p = n_nodes / 2; let tr = timed(|| { - range_reveal(level, p, &mut m_range); + range_reveal(black_box(level), black_box(p), &mut m_range); + black_box(&m_range); }); let tt = timed(|| { - tcam_reveal(&addr, level, p, &mut m_tcam); + tcam_reveal(black_box(&addr), black_box(level), black_box(p), &mut m_tcam); + black_box(&m_tcam); }); println!("{level:>5} {:>9} {tr:>8.0} {tt:>8.0} {:>6.1}× ok", 1usize << (16 - 4 * level), tt / tr); } @@ -729,12 +768,28 @@ fn main() { // ── M3: coal — one maneuver = re-chain a resident mask from its column ── let mut m_new = vec![0u64; WORDS]; - let c = timed(|| gt_i32_to_mask(&perm_cols[0], thr as i32, &mut m_new)); + let c = timed(|| { + gt_i32_to_mask(black_box(&perm_cols[0]), black_box(thr as i32), &mut m_new); + black_box(&m_new); + }); println!( "[coal] one re-chain (gt_i32 sweep over one column) = {c:.0} ns = {:.1} ternlogq passes = {:.2} maintained steps at x=4", c / t_tern, c / (4.0 * t_tern + n_term) ); + let mut m_new_u8 = vec![0u64; WORDS]; + gt_u8_to_mask(&perm_cols_u8[0], thr, &mut m_new_u8); + assert_eq!(m_new_u8, m_new, "u8 and widened-i32 re-chain masks differ"); + let c_u8 = timed(|| { + gt_u8_to_mask(black_box(&perm_cols_u8[0]), black_box(thr), &mut m_new_u8); + black_box(&m_new_u8); + }); + println!( + "[coal] one re-chain NATIVE u8 (gt_u8 sweep) = {c_u8:.0} ns = {:.1} ternlogq passes = {:.2} maintained steps at x=4 → {:.2}× vs widened", + c_u8 / t_tern, + c_u8 / (4.0 * t_tern + n_term), + c / c_u8 + ); println!( "[M2] speed change x→x±1 costs one ternlogq pass ({t_tern:.0} ns); x→x±k costs k passes — linear, no cliff" ); diff --git a/src/simd.rs b/src/simd.rs index b975c3d1..33b1d953 100644 --- a/src/simd.rs +++ b/src/simd.rs @@ -777,20 +777,36 @@ pub use crate::simd_amx::amx_tile_available; pub use crate::simd_masking_ops::{ // 2026-09-13: the closed comparison family + complement/xor/any/all + care-masked // register match + masked min/max + blend (lance-graph-duckmask, lgj-abi D-MRL-1a). + // 2026-09-16: the u8 compare-to-mask family — one 64-lane `U8x64` chunk is + // exactly one whole 64-bit mask word, so these pack without a shift (N2/G1 + // of the DuckDB->V3 translation matrix). + // 2026-09-16: the ordered u64 family (N3/G2). The width where the packing + // is NOT free — `U64x8` is 8 lanes, so eight groups share a word and each + // lands at its own byte. blend_i32, eq_i32_to_mask, eq_i32_to_mask_under, eq_u32_strided_to_mask, eq_u32_to_mask, eq_u32_to_mask_under, + eq_u64_to_mask, + eq_u8_to_mask, ge_i32_to_mask, ge_i32_to_mask_under, + ge_u64_to_mask, + ge_u8_to_mask, gt_i32_to_mask, gt_i32_to_mask_under, + gt_u64_to_mask, + gt_u8_to_mask, le_i32_to_mask, le_i32_to_mask_under, + le_u64_to_mask, + le_u8_to_mask, lt_i32_to_mask, lt_i32_to_mask_under, + lt_u64_to_mask, + lt_u8_to_mask, mask_all, mask_and, mask_and_assign, @@ -801,6 +817,7 @@ pub use crate::simd_masking_ops::{ mask_not_assign, mask_or, mask_or_assign, + mask_set_range, mask_shift_morton, mask_ternlog, mask_ternlog_assign, @@ -814,6 +831,8 @@ pub use crate::simd_masking_ops::{ ne_i32_to_mask_under, ne_u32_to_mask, ne_u32_to_mask_under, + ne_u64_to_mask, + ne_u8_to_mask, ternary_match_strided_to_mask, ternary_match_u32_to_mask, ternary_match_u32_to_mask_under, diff --git a/src/simd_avx2.rs b/src/simd_avx2.rs index e4a41a29..71f46ea7 100644 --- a/src/simd_avx2.rs +++ b/src/simd_avx2.rs @@ -1371,15 +1371,17 @@ impl I16x32 { // These match the AVX-512 U8x64 methods in simd_avx512.rs. impl U8x64 { /// Byte-wise equality mask: bit i set if self[i] == other[i]. + /// + /// Composed from two native AVX2 `U8x32::cmpeq_mask` calls (lanes + /// 0..32 in the low half, 32..64 in the high half) instead of a + /// 64-iteration scalar loop — AVX2's natural byte width is 32, not 64 + /// (see the `U8x32` module doc above), and `U8x32` already carries the + /// `_mm256_cmpeq_epi8` + `_mm256_movemask_epi8` realization. #[inline(always)] pub fn cmpeq_mask(self, other: Self) -> u64 { - let mut mask = 0u64; - for i in 0..64 { - if self.0[i] == other.0[i] { - mask |= 1u64 << i; - } - } - mask + let lo = U8x32::from_slice(&self.0[..32]).cmpeq_mask(U8x32::from_slice(&other.0[..32])); + let hi = U8x32::from_slice(&self.0[32..]).cmpeq_mask(U8x32::from_slice(&other.0[32..])); + (lo as u64) | ((hi as u64) << 32) } /// Shift right each 16-bit lane by imm bits (operates on pairs of u8 as u16). @@ -1416,15 +1418,18 @@ impl U8x64 { } Self(out) } + /// Byte-wise UNSIGNED greater-than mask: bit i set if self[i] > other[i]. + /// + /// Composed from two native AVX2 `U8x32::cmpgt_mask` calls, which + /// already carry the sign-bias XOR trick AVX2 needs to get an unsigned + /// compare out of the signed-only `_mm256_cmpgt_epi8` (see that + /// method's doc comment). Do not replace this with a signed compare — + /// unsigned ordering is the entire reason the bias exists. #[inline(always)] pub fn cmpgt_mask(self, other: Self) -> u64 { - let mut m: u64 = 0; - for i in 0..64 { - if self.0[i] > other.0[i] { - m |= 1 << i; - } - } - m + let lo = U8x32::from_slice(&self.0[..32]).cmpgt_mask(U8x32::from_slice(&other.0[..32])); + let hi = U8x32::from_slice(&self.0[32..]).cmpgt_mask(U8x32::from_slice(&other.0[32..])); + (lo as u64) | ((hi as u64) << 32) } #[inline(always)] pub fn mask_blend(mask: u64, a: Self, b: Self) -> Self { @@ -2326,6 +2331,81 @@ impl U64x8 { } sum } + + /// Lane-wise equality comparison. Returns an 8-bit mask: bit `i` is set + /// iff `self[i] == other[i]`. Same 8-lanes-in-8-bits contract as the + /// AVX-512 `U64x8::cmpeq_mask` (`simd_avx512.rs`) — one bit per lane; + /// bits at or above 8 simply do not exist in the `u8` return type. + /// + /// **This is the scalar polyfill, not a missed vectorization** — as the + /// file's own note just above this `impl` block says: "The + /// `avx2_int_type!` macro generated `U64x8` as a scalar polyfill in + /// this file." AVX2 has no packed unsigned 64-bit ordered compare worth + /// wiring at this width, unlike the `rotate_left`/`rotate_right`/`Shl`/ + /// `Shr` impls further up this file, which DO reach for real `__m256i` + /// halves via `avx2_halves()` — because AVX2 has packed 64-bit + /// *shifts*, just not a packed 64-bit *unsigned compare*. + /// + /// Composed as four 2-lane groups (`p in 0..4`, lanes `2p`/`2p+1`) + /// rather than a flat 8-iteration loop — deliberate: storage flat, + /// composition 2×4. Every non-avx512 backend realizes `U64x8` as four + /// `U64x2` pairs (`simd_neon.rs`/`simd_wasm.rs`: + /// `pub struct U64x8(pub [U64x2; 4])`); walking the same four pairs + /// here keeps this arm's shape identical to theirs even though this + /// file's own storage is a flat `[u64; 8]`. Do not "simplify" this into + /// a single `for i in 0..8` loop. + /// + /// # Example + /// ```rust,ignore + /// let a = U64x8::from_array([1, 2, 3, 4, 5, 6, 7, 8]); + /// let b = U64x8::from_array([1, 0, 3, 0, 5, 0, 7, 0]); + /// assert_eq!(a.cmpeq_mask(b), 0b0101_0101); + /// ``` + #[inline(always)] + pub fn cmpeq_mask(self, other: Self) -> u8 { + let mut mask: u8 = 0; + for p in 0..4 { + if self.0[2 * p] == other.0[2 * p] { + mask |= 1 << (2 * p); + } + if self.0[2 * p + 1] == other.0[2 * p + 1] { + mask |= 1 << (2 * p + 1); + } + } + mask + } + + /// Lane-wise **unsigned** greater-than comparison. Returns an 8-bit + /// mask: bit `i` is set iff `self[i] > other[i]`. Symmetric to + /// `cmpeq_mask` above — same 8-bits-only footprint, same 2×4 grouping, + /// same "bits at or above 8 do not exist" contract. + /// + /// Plain `>` on Rust's `u64` is *already* the unsigned ordering. Unlike + /// this file's own `U8x32::cmpgt_mask` (signed-only `_mm256_cmpgt_epi8`, + /// biased into an unsigned answer via a sign-flip XOR) there is no + /// sign-bias trick to apply here — this arm is the scalar polyfill (see + /// `cmpeq_mask` above), so there is no signed intrinsic to bias in the + /// first place. + /// + /// # Example + /// ```rust,ignore + /// let a = U64x8::from_array([1, 2, 3, 4, 5, 6, 7, 8]); + /// let b = U64x8::from_array([0, 2, 0, 4, 0, 6, 0, 8]); + /// assert_eq!(a.cmpgt_mask(b), 0b0101_0101); + /// ``` + #[inline(always)] + pub fn cmpgt_mask(self, other: Self) -> u8 { + let mut mask: u8 = 0; + for p in 0..4 { + if self.0[2 * p] > other.0[2 * p] { + mask |= 1 << (2 * p); + } + if self.0[2 * p + 1] > other.0[2 * p + 1] { + mask |= 1 << (2 * p + 1); + } + } + mask + } } // Extra methods for U16x32 (widen/narrow, shift, multiply) — AVX2 scalar fallback. @@ -3196,6 +3276,130 @@ mod tests { assert!(fired > 256 * 8 / 2, "shifts did nothing on {fired} lanes"); } + /// `U8x64::cmpeq_mask`/`cmpgt_mask` are composed from two `U8x32` + /// halves; this is the scalar oracle that composition must match + /// bit-for-bit, over many random 64-byte pairs. Written here rather + /// than imported so the reference is never the function under test. + #[test] + fn u8x64_cmpeq_and_cmpgt_mask_match_scalar_oracle() { + // Same xorshift64 construction as `u64x8_variable_shifts_match_scalar_per_lane` + // above, seeded differently, narrowed to a byte per draw. + let mut seed = 0xD1B5_4A32_D192_ED03u64; + let mut next_u8 = || { + seed ^= seed << 13; + seed ^= seed >> 7; + seed ^= seed << 17; + (seed >> 56) as u8 + }; + + let mut saw_mixed_eq = false; + let mut saw_mixed_gt = false; + + for _ in 0..512 { + let a: [u8; 64] = core::array::from_fn(|_| next_u8()); + let b: [u8; 64] = core::array::from_fn(|_| next_u8()); + + let mut eq_expected = 0u64; + let mut gt_expected = 0u64; + for i in 0..64 { + if a[i] == b[i] { + eq_expected |= 1u64 << i; + } + if a[i] > b[i] { + gt_expected |= 1u64 << i; + } + } + + let eq_actual = U8x64::from_array(a).cmpeq_mask(U8x64::from_array(b)); + let gt_actual = U8x64::from_array(a).cmpgt_mask(U8x64::from_array(b)); + assert_eq!(eq_actual, eq_expected, "cmpeq_mask mismatch for a={a:?} b={b:?}"); + assert_eq!(gt_actual, gt_expected, "cmpgt_mask mismatch for a={a:?} b={b:?}"); + + if eq_actual != 0 && eq_actual != u64::MAX { + saw_mixed_eq = true; + } + if gt_actual != 0 && gt_actual != u64::MAX { + saw_mixed_gt = true; + } + } + + // Anti-vacuity: an always-zero or always-ones implementation must + // not be able to pass just because it happens to agree with the + // oracle on the all-zero/all-ones edge case alone. + assert!(saw_mixed_eq, "512 random draws never produced a mixed cmpeq_mask result"); + assert!(saw_mixed_gt, "512 random draws never produced a mixed cmpgt_mask result"); + } + + /// AVX2 only has a SIGNED byte compare; `U8x32::cmpgt_mask` (which + /// `U8x64::cmpgt_mask` composes) must bias both operands by 0x80 to + /// recover UNSIGNED ordering. Exercise the sign boundary in EACH + /// 32-lane half, since a wrong bias would only show up as a flipped + /// bit, never as a panic or a type error. + #[test] + fn u8x64_cmpgt_mask_respects_unsigned_ordering_at_the_sign_boundary() { + let mut a = [0u8; 64]; + let mut b = [0u8; 64]; + + for base in [0usize, 32] { + a[base] = 0x7F; + b[base] = 0x80; // 0x7F > 0x80 is FALSE unsigned; TRUE if signed + a[base + 1] = 0x80; + b[base + 1] = 0x7F; // 0x80 > 0x7F is TRUE unsigned; FALSE if signed + a[base + 2] = 0xFF; + b[base + 2] = 0x00; // TRUE either way — sanity anchor + a[base + 3] = 0x00; + b[base + 3] = 0xFF; // FALSE either way — sanity anchor + a[base + 4] = 0x80; + b[base + 4] = 0x80; // equal — never > regardless of signedness + a[base + 5] = 0xFF; + b[base + 5] = 0xFF; // equal + } + + let m = U8x64::from_array(a).cmpgt_mask(U8x64::from_array(b)); + + for base in [0usize, 32] { + assert!(m & (1u64 << (base + 1)) != 0, "0x80 > 0x7F must be TRUE at lane {}", base + 1); + assert!(m & (1u64 << base) == 0, "0x7F > 0x80 must be FALSE at lane {base}"); + } + + // Full-lane cross-check: Rust's native `u8: PartialOrd` is already + // an unsigned compare, so it doubles as the oracle for every lane, + // not just the six named above. + let mut expected = 0u64; + for i in 0..64 { + if a[i] > b[i] { + expected |= 1u64 << i; + } + } + assert_eq!(m, expected); + } + + /// A single differing lane at each half's edges (0, 31, 32, 63) must + /// set EXACTLY that one bit. 31/32 straddle the lo/hi `U8x32` split; + /// a wrong half chosen, or a `<< 32` applied to the wrong operand, + /// shows up here as a bit landing at the wrong position rather than + /// as a compile error. + #[test] + fn u8x64_cmpeq_and_cmpgt_mask_isolate_lane_position() { + for lane in [0usize, 31, 32, 63] { + // cmpeq_mask: every lane differs except `lane`, which matches. + let mut a = [1u8; 64]; + let mut b = [2u8; 64]; + a[lane] = 9; + b[lane] = 9; + let m = U8x64::from_array(a).cmpeq_mask(U8x64::from_array(b)); + assert_eq!(m, 1u64 << lane, "cmpeq_mask lane {lane}: expected exactly bit {lane} set"); + + // cmpgt_mask: every lane is equal except `lane`, where a > b. + let mut a2 = [7u8; 64]; + let mut b2 = [7u8; 64]; + a2[lane] = 200; + b2[lane] = 50; + let m2 = U8x64::from_array(a2).cmpgt_mask(U8x64::from_array(b2)); + assert_eq!(m2, 1u64 << lane, "cmpgt_mask lane {lane}: expected exactly bit {lane} set"); + } + } + #[test] fn test_dot_f32() { let a: Vec = (0..100).map(|i| i as f32).collect(); diff --git a/src/simd_avx512.rs b/src/simd_avx512.rs index 83cdcdb0..986017b9 100644 --- a/src/simd_avx512.rs +++ b/src/simd_avx512.rs @@ -1817,6 +1817,39 @@ impl U64x8 { pub fn reduce_sum(self) -> u64 { unsafe { _mm512_reduce_add_epi64(self.0) as u64 } } + + /// Lane-wise equality comparison. Returns an 8-bit mask: bit `i` is set + /// iff `self[i] == other[i]`. + /// + /// The mask is `u8`, not `u64`, because `U64x8` has exactly 8 lanes — one + /// bit per lane (contrast `U8x64::cmpeq_mask` above, whose 64 lanes need + /// a `u64` mask). Equality has no sign, so — unlike `cmpgt_mask` below — + /// there is no unsigned/signed instruction choice to make here. + #[inline(always)] + pub fn cmpeq_mask(self, other: Self) -> u8 { + // SAFETY: `Self` is a native `__m512i`; this arm is compiled only + // under the avx512f dispatch, the same guarantee every other method + // on this type relies on. `_mm512_cmpeq_epu64_mask` needs only + // avx512f (not avx512bw/vl) for its 512-bit form. + unsafe { _mm512_cmpeq_epu64_mask(self.0, other.0) } + } + + /// Lane-wise **unsigned** greater-than comparison. Returns an 8-bit mask: + /// bit `i` is set iff `self[i] > other[i]`, treating each full 64-bit + /// lane as unsigned. Symmetric to `cmpeq_mask` above. + /// + /// This calls `_mm512_cmpgt_epu64_mask` (`epu64` — unsigned), never the + /// signed `_mm512_cmpgt_epi64_mask` (`epi64`). A `u64` with the top bit + /// set (`>= 1 << 63`) is *negative* under a signed 64-bit compare, so an + /// `epi64` instruction would invert the answer for exactly that half of + /// the `u64` domain. `U64x8` stores unsigned lanes by construction, so + /// the unsigned comparison is the only one that matches the type. + #[inline(always)] + pub fn cmpgt_mask(self, other: Self) -> u8 { + // SAFETY: as `cmpeq_mask` above — avx512f-only dispatch, stable + // intrinsic. Unsigned compare via `_epu64`, never `_epi64`. + unsafe { _mm512_cmpgt_epu64_mask(self.0, other.0) } + } } impl_bin_op!(U64x8, Add, add, _mm512_add_epi64); diff --git a/src/simd_masking_ops.rs b/src/simd_masking_ops.rs index 5a29202a..6004e275 100644 --- a/src/simd_masking_ops.rs +++ b/src/simd_masking_ops.rs @@ -135,6 +135,19 @@ fn tail_lane_bits(n: usize) -> u16 { ((1u32 << n) - 1) as u16 } +/// The low `n` bits set (`n < 8`): the lane-validity mask for a padded +/// 8-lane predicate tail, so a padding lane can never contribute a match. +/// +/// Sibling of [`tail_lane_bits`], which serves the 16-lane registers. Kept +/// separate rather than made generic because the return width IS the +/// register's lane count, and the caller ORs the result into a byte-shifted +/// slot whose width has to match. +#[inline(always)] +fn tail_lane_bits_8(n: usize) -> u8 { + debug_assert!(n < 8, "a tail is shorter than one register"); + ((1u16 << n) - 1) as u8 +} + /// Packs `values[i] == needle` into `out_words`, one bit per element, /// LSB-first within each `u64` word (bit `k` of word `w` corresponds to /// element `w * 64 + k`). @@ -1056,6 +1069,528 @@ pub fn ne_u32_to_mask(values: &[u32], needle: u32, out_words: &mut [u64]) { clear_mask_tail(out_words, values.len()); } +// ──────────────────────────────────────────────────────────────────────── +// The u8 comparison family (`*_u8_to_mask`) — the byte-width sibling of the +// u32/i32 families above. [`crate::simd::U8x64`] is 64 lanes wide, so unlike +// those families (whose 16-lane groups pack four per `u64` word via +// `out_words[g / 4] |= (bits as u64) << ((g % 4) * 16)`), one chunk here +// fills exactly ONE whole output word: `out_words[g] = bits`, a plain +// assignment, no shift-and-OR. +// +// Comparison is UNSIGNED — `u8`'s only ordering — via `U8x64::cmpeq_mask` / +// `U8x64::cmpgt_mask`, exact with no saturation or wrapping. Derived exactly +// as the i32 family is: `gt`/`lt` are DIRECT (`cmpgt_mask` with the operands +// in the order the comparison needs); `ge`/`le`/`ne` are the complement of +// `lt`/`gt`/`eq` with [`clear_mask_tail`] re-clearing the tail — the same +// shape as [`ge_i32_to_mask`] / [`le_i32_to_mask`] / [`ne_u32_to_mask`]. +// +// The DIRECT forms' tail is a genuinely different shape from the u32/i32 +// family's, though: a 64-lane chunk's tail can hold up to 63 elements (not +// up to 15), so it needs a wider "valid lane" mask than [`tail_lane_bits`] +// provides (16-lane only, `debug_assert!(n < 16)`). [`word_range_mask`] — +// already built for [`mask_set_range`] to return the low-N-bits `u64` +// without ever evaluating `1u64 << 64` — is reused here as +// `word_range_mask(0, tail.len())`, rather than adding a second such helper. +// ──────────────────────────────────────────────────────────────────────── + +/// Packs `values[i] == needle` into `out_words`, one bit per element, +/// LSB-first within each `u64` word (bit `k` of word `w` corresponds to +/// element `w * 64 + k`). +/// +/// `out_words` is **fully overwritten**, not OR-ed into. Trailing bits in the +/// final word beyond `values.len()`, and any surplus words past +/// `ceil(len / 64)`, are written as `0`. +/// +/// Equality is exact bitwise comparison over the full `u8` range — `0` and +/// `u8::MAX` are ordinary needles, and there is no saturation, wrapping, or +/// signedness question to resolve. An empty `values` writes only zeros. +/// +/// Runs 64 lanes at a time through [`crate::simd::U8x64::cmpeq_mask`] — one +/// whole output word per chunk, since `U8x64` is exactly 64 lanes wide +/// (unlike the u32/i32 families' 16-lane groups, four of which pack into one +/// word). The final partial group is zero-padded into one register and run +/// through the same packed compare, with the padding lanes' bits masked off +/// by [`word_range_mask`] — no scalar tail, so the tail cannot disagree with +/// the body. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::eq_u8_to_mask; +/// +/// let values = [7u8, 1, 7, 2]; +/// let mut words = [0u64; 1]; +/// eq_u8_to_mask(&values, 7, &mut words); +/// // elements 0 and 2 match → bits 0 and 2 → 0b0101 +/// assert_eq!(words[0], 0b0101); +/// ``` +#[inline] +pub fn eq_u8_to_mask(values: &[u8], needle: u8, out_words: &mut [u64]) { + let n = values.len(); + let words = mask_words_for(n); + assert!(out_words.len() >= words, "eq_u8_to_mask: out_words.len()={} < required {}", out_words.len(), words); + + for w in out_words.iter_mut() { + *w = 0; + } + + let needle_v = crate::simd::U8x64::splat(needle); + let (chunks, tail) = values.as_chunks::<64>(); + for (g, chunk) in chunks.iter().enumerate() { + out_words[g] = crate::simd::U8x64::from_array(*chunk).cmpeq_mask(needle_v); + } + if !tail.is_empty() { + let g = chunks.len(); + out_words[g] = + crate::simd::U8x64::from_array(pad_tail(tail)).cmpeq_mask(needle_v) & word_range_mask(0, tail.len()); + } +} + +/// Packs `values[i] > threshold` (**unsigned** comparison — the only +/// ordering `u8` has) into `out_words`, one bit per element, LSB-first +/// within each `u64` word (bit `k` of word `w` corresponds to element +/// `w * 64 + k`). +/// +/// `out_words` is **fully overwritten**, not OR-ed into. Trailing bits in the +/// final word beyond `values.len()`, and any surplus words past +/// `ceil(len / 64)`, are written as `0`. +/// +/// Comparison is unsigned and strict (`>`, never `>=`), exact with no +/// saturation or wrapping: +/// * `threshold == 0` sets every lane except those equal to `0`. +/// * `threshold == u8::MAX` sets nothing — no `u8` exceeds it. +/// +/// An empty `values` writes only zeros. +/// +/// Runs 64 lanes at a time through [`crate::simd::U8x64::cmpgt_mask`] — one +/// whole output word per chunk. The final partial group is zero-padded into +/// one register and run through the same packed compare, with the padding +/// lanes' bits masked off by [`word_range_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::gt_u8_to_mask; +/// +/// let values = [5u8, 0, 200, 255]; +/// let mut words = [0u64; 1]; +/// gt_u8_to_mask(&values, 4, &mut words); +/// // elements 0, 2 and 3 exceed 4 → bits 0, 2, 3 → 0b1101 +/// assert_eq!(words[0], 0b1101); +/// ``` +#[inline] +pub fn gt_u8_to_mask(values: &[u8], threshold: u8, out_words: &mut [u64]) { + let n = values.len(); + let words = mask_words_for(n); + assert!(out_words.len() >= words, "gt_u8_to_mask: out_words.len()={} < required {}", out_words.len(), words); + + for w in out_words.iter_mut() { + *w = 0; + } + + let threshold_v = crate::simd::U8x64::splat(threshold); + let (chunks, tail) = values.as_chunks::<64>(); + for (g, chunk) in chunks.iter().enumerate() { + out_words[g] = crate::simd::U8x64::from_array(*chunk).cmpgt_mask(threshold_v); + } + if !tail.is_empty() { + let g = chunks.len(); + out_words[g] = + crate::simd::U8x64::from_array(pad_tail(tail)).cmpgt_mask(threshold_v) & word_range_mask(0, tail.len()); + } +} + +/// Packs `values[i] < threshold` (unsigned): computed directly as +/// `threshold > values[i]` through [`crate::simd::U8x64::cmpgt_mask`] with +/// the operands swapped — the same shape [`lt_i32_to_mask`] uses (there +/// `x > t - 1` would underflow at `t == i32::MIN`; `u8` has no such boundary, +/// but the swapped-operand derivation is kept identical across the two +/// families rather than special-cased per element type). Full overwrite, +/// trailing bits zero. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::lt_u8_to_mask; +/// +/// let values = [5u8, 0, 200, 255]; +/// let mut words = [0u64; 1]; +/// lt_u8_to_mask(&values, 4, &mut words); +/// // only element 1 (0) is less than 4 → bit 1 → 0b0010 +/// assert_eq!(words[0], 0b0010); +/// ``` +#[inline] +pub fn lt_u8_to_mask(values: &[u8], threshold: u8, out_words: &mut [u64]) { + let n = values.len(); + let words = mask_words_for(n); + assert!(out_words.len() >= words, "lt_u8_to_mask: out_words.len()={} < required {}", out_words.len(), words); + + for w in out_words.iter_mut() { + *w = 0; + } + + let t = crate::simd::U8x64::splat(threshold); + let (chunks, tail) = values.as_chunks::<64>(); + for (g, chunk) in chunks.iter().enumerate() { + out_words[g] = t.cmpgt_mask(crate::simd::U8x64::from_array(*chunk)); + } + if !tail.is_empty() { + let g = chunks.len(); + out_words[g] = t.cmpgt_mask(crate::simd::U8x64::from_array(pad_tail(tail))) & word_range_mask(0, tail.len()); + } +} + +/// Packs `values[i] >= threshold` (unsigned): the complement of +/// [`lt_u8_to_mask`] with the tail re-cleared. Full overwrite. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::ge_u8_to_mask; +/// +/// let values = [5u8, 0, 200, 255]; +/// let mut words = [0u64; 1]; +/// ge_u8_to_mask(&values, 4, &mut words); +/// // 5, 200 and 255 are >= 4 → bits 0, 2, 3 → 0b1101 +/// assert_eq!(words[0], 0b1101); +/// ``` +#[inline] +pub fn ge_u8_to_mask(values: &[u8], threshold: u8, out_words: &mut [u64]) { + lt_u8_to_mask(values, threshold, out_words); + for w in out_words.iter_mut() { + *w = !*w; + } + clear_mask_tail(out_words, values.len()); +} + +/// Packs `values[i] <= threshold` (unsigned): the complement of +/// [`gt_u8_to_mask`] with the tail re-cleared. Full overwrite. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::le_u8_to_mask; +/// +/// let values = [5u8, 0, 200, 255]; +/// let mut words = [0u64; 1]; +/// le_u8_to_mask(&values, 4, &mut words); +/// // only element 1 (0) is <= 4 → bit 1 → 0b0010 +/// assert_eq!(words[0], 0b0010); +/// ``` +#[inline] +pub fn le_u8_to_mask(values: &[u8], threshold: u8, out_words: &mut [u64]) { + gt_u8_to_mask(values, threshold, out_words); + for w in out_words.iter_mut() { + *w = !*w; + } + clear_mask_tail(out_words, values.len()); +} + +/// Packs `values[i] != needle`: the complement of [`eq_u8_to_mask`] with the +/// tail re-cleared. Full overwrite. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::ne_u8_to_mask; +/// +/// let values = [7u8, 1, 7, 2]; +/// let mut words = [0u64; 1]; +/// ne_u8_to_mask(&values, 7, &mut words); +/// // elements 1 and 3 differ from 7 → bits 1 and 3 → 0b1010 +/// assert_eq!(words[0], 0b1010); +/// ``` +#[inline] +pub fn ne_u8_to_mask(values: &[u8], needle: u8, out_words: &mut [u64]) { + eq_u8_to_mask(values, needle, out_words); + for w in out_words.iter_mut() { + *w = !*w; + } + clear_mask_tail(out_words, values.len()); +} + +// ──────────────────────────────────────────────────────────────────────── +// Ordered 64-bit predicates — T1 gap G2 of the DuckDB→V3 translation matrix. +// +// The width where the packing is NOT free. `U8x64` is 64 lanes against a +// 64-bit word, so one chunk is one whole word with no shift; `U64x8` is 8 +// lanes, so eight chunks share a word and each must be placed at its own +// byte, `out_words[g / 8] |= (bits as u64) << ((g % 8) * 8)`. That is the +// same packing `eq_u32_to_mask` does at four-chunks-per-word, one step +// further. +// +// Why `u64` and not `i64`: the named consumer is an ordered window over +// address-space offsets (`lo <= x < hi`), which are unsigned and routinely +// exceed `2^32`, so narrowing to the existing `i32` family is unsound. No +// caller compares `i64` lanes, so that family is deliberately not built — +// a speculative family is surface with no falsifier attached to it. +// +// The ORDERED ops need an unsigned compare, and only two of the six +// realizations have one natively (AVX-512 `_mm512_cmpgt_epu64_mask`, NEON +// `cmhi`). WASM and the byte-width AVX2 path reach it by flipping the sign +// bit of both operands — an order-preserving bijection — and the flat +// polyfill arms get it for free because Rust's `>` on `u64` already is the +// unsigned compare. All of that lives in the arm files; this layer never +// sees it, which is the point of the facade. +// ──────────────────────────────────────────────────────────────────────── + +/// Packs `values[i] == needle` into `out_words`, one bit per element, +/// LSB-first within each `u64` word (bit `k` of word `w` is element +/// `w * 64 + k`). +/// +/// `out_words` is **fully overwritten**, not OR-ed into. Trailing bits in +/// the final word beyond `values.len()`, and any surplus words past +/// `ceil(len / 64)`, are written as `0`. +/// +/// Runs 8 lanes at a time through [`crate::simd::U64x8::cmpeq_mask`], whose +/// 8-bit result is placed at byte `g % 8` of word `g / 8`. The final partial +/// group is zero-padded into one register and run through the same packed +/// compare, with the padding lanes' bits masked off — load-bearing, because +/// a `needle` of `0` matches the padding. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::eq_u64_to_mask; +/// +/// let values = [7u64, 1, 7, 1 << 63]; +/// let mut words = [0u64; 1]; +/// eq_u64_to_mask(&values, 7, &mut words); +/// // elements 0 and 2 equal 7 → bits 0 and 2 → 0b0101 +/// assert_eq!(words[0], 0b0101); +/// ``` +#[inline] +pub fn eq_u64_to_mask(values: &[u64], needle: u64, out_words: &mut [u64]) { + let n = values.len(); + let words = mask_words_for(n); + assert!(out_words.len() >= words, "eq_u64_to_mask: out_words.len()={} < required {}", out_words.len(), words); + + for w in out_words.iter_mut() { + *w = 0; + } + + let needle_v = crate::simd::U64x8::splat(needle); + let (chunks, tail) = values.as_chunks::<8>(); + for (g, chunk) in chunks.iter().enumerate() { + let bits = crate::simd::U64x8::from_array(*chunk).cmpeq_mask(needle_v); + out_words[g / 8] |= (bits as u64) << ((g % 8) * 8); + } + if !tail.is_empty() { + let g = chunks.len(); + let bits = crate::simd::U64x8::from_array(pad_tail(tail)).cmpeq_mask(needle_v) & tail_lane_bits_8(tail.len()); + out_words[g / 8] |= (bits as u64) << ((g % 8) * 8); + } +} + +/// Packs `values[i] > threshold` (**unsigned** — the only ordering `u64` +/// has) into `out_words`. Full overwrite, trailing bits zero. +/// +/// Unsigned is not a detail at this width: a `u64` at or above `1 << 63` +/// reads as NEGATIVE under a signed compare, so a signed instruction +/// inverts the answer on exactly half the domain — and address-space +/// offsets, the reason this family exists, live in that half. +/// +/// * `threshold == 0` sets every lane except those equal to `0`. +/// * `threshold == u64::MAX` sets nothing — no `u64` exceeds it. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::gt_u64_to_mask; +/// +/// let values = [5u64, 0, 1 << 63, u64::MAX]; +/// let mut words = [0u64; 1]; +/// gt_u64_to_mask(&values, 4, &mut words); +/// // 5, 2^63 and u64::MAX all exceed 4 → bits 0, 2, 3 → 0b1101 +/// assert_eq!(words[0], 0b1101); +/// ``` +#[inline] +pub fn gt_u64_to_mask(values: &[u64], threshold: u64, out_words: &mut [u64]) { + let n = values.len(); + let words = mask_words_for(n); + assert!(out_words.len() >= words, "gt_u64_to_mask: out_words.len()={} < required {}", out_words.len(), words); + + for w in out_words.iter_mut() { + *w = 0; + } + + let threshold_v = crate::simd::U64x8::splat(threshold); + let (chunks, tail) = values.as_chunks::<8>(); + for (g, chunk) in chunks.iter().enumerate() { + let bits = crate::simd::U64x8::from_array(*chunk).cmpgt_mask(threshold_v); + out_words[g / 8] |= (bits as u64) << ((g % 8) * 8); + } + if !tail.is_empty() { + let g = chunks.len(); + let bits = + crate::simd::U64x8::from_array(pad_tail(tail)).cmpgt_mask(threshold_v) & tail_lane_bits_8(tail.len()); + out_words[g / 8] |= (bits as u64) << ((g % 8) * 8); + } +} + +/// Packs `values[i] < threshold` (unsigned): computed directly as +/// `threshold > values[i]` through [`crate::simd::U64x8::cmpgt_mask`] with +/// the operands swapped. +/// +/// An **operand swap, never `x > t - 1`** — that spelling underflows at +/// `t == 0` and would report every lane as less-than nothing. Same +/// reasoning the `i32` family carries for `i32::MIN`. +/// +/// The padding lanes of a partial group are `0`, and `threshold > 0` holds +/// for every `threshold` above zero, so the tail mask here is what keeps a +/// short input from reporting phantom set bits. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::lt_u64_to_mask; +/// +/// let values = [5u64, 0, 1 << 63, u64::MAX]; +/// let mut words = [0u64; 1]; +/// lt_u64_to_mask(&values, 6, &mut words); +/// // only 5 and 0 are below 6 → bits 0, 1 → 0b0011 +/// assert_eq!(words[0], 0b0011); +/// ``` +#[inline] +pub fn lt_u64_to_mask(values: &[u64], threshold: u64, out_words: &mut [u64]) { + let n = values.len(); + let words = mask_words_for(n); + assert!(out_words.len() >= words, "lt_u64_to_mask: out_words.len()={} < required {}", out_words.len(), words); + + for w in out_words.iter_mut() { + *w = 0; + } + + let t = crate::simd::U64x8::splat(threshold); + let (chunks, tail) = values.as_chunks::<8>(); + for (g, chunk) in chunks.iter().enumerate() { + let bits = t.cmpgt_mask(crate::simd::U64x8::from_array(*chunk)); + out_words[g / 8] |= (bits as u64) << ((g % 8) * 8); + } + if !tail.is_empty() { + let g = chunks.len(); + let bits = t.cmpgt_mask(crate::simd::U64x8::from_array(pad_tail(tail))) & tail_lane_bits_8(tail.len()); + out_words[g / 8] |= (bits as u64) << ((g % 8) * 8); + } +} + +/// Packs `values[i] >= threshold` (unsigned): the complement of +/// [`lt_u64_to_mask`] with the tail re-cleared. Full overwrite. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::ge_u64_to_mask; +/// +/// let values = [5u64, 0, 1 << 63, u64::MAX]; +/// let mut words = [0u64; 1]; +/// ge_u64_to_mask(&values, 5, &mut words); +/// // 5, 2^63 and u64::MAX are >= 5 → bits 0, 2, 3 → 0b1101 +/// assert_eq!(words[0], 0b1101); +/// ``` +#[inline] +pub fn ge_u64_to_mask(values: &[u64], threshold: u64, out_words: &mut [u64]) { + lt_u64_to_mask(values, threshold, out_words); + for w in out_words.iter_mut() { + *w = !*w; + } + clear_mask_tail(out_words, values.len()); +} + +/// Packs `values[i] <= threshold` (unsigned): the complement of +/// [`gt_u64_to_mask`] with the tail re-cleared. Full overwrite. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::le_u64_to_mask; +/// +/// let values = [5u64, 0, 1 << 63, u64::MAX]; +/// let mut words = [0u64; 1]; +/// le_u64_to_mask(&values, 5, &mut words); +/// // only 5 and 0 are <= 5 → bits 0, 1 → 0b0011 +/// assert_eq!(words[0], 0b0011); +/// ``` +#[inline] +pub fn le_u64_to_mask(values: &[u64], threshold: u64, out_words: &mut [u64]) { + gt_u64_to_mask(values, threshold, out_words); + for w in out_words.iter_mut() { + *w = !*w; + } + clear_mask_tail(out_words, values.len()); +} + +/// Packs `values[i] != needle`: the complement of [`eq_u64_to_mask`] with +/// the tail re-cleared. Full overwrite. +/// +/// # Panics +/// +/// Panics if `out_words.len() < values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::ne_u64_to_mask; +/// +/// let values = [7u64, 1, 7, 2]; +/// let mut words = [0u64; 1]; +/// ne_u64_to_mask(&values, 7, &mut words); +/// // elements 1 and 3 differ from 7 → bits 1 and 3 → 0b1010 +/// assert_eq!(words[0], 0b1010); +/// ``` +#[inline] +pub fn ne_u64_to_mask(values: &[u64], needle: u64, out_words: &mut [u64]) { + eq_u64_to_mask(values, needle, out_words); + for w in out_words.iter_mut() { + *w = !*w; + } + clear_mask_tail(out_words, values.len()); +} + /// `dst = !src` over `n_rows` elements — the tail-aware complement. Bits at /// or past `n_rows` are written `0`, so a conforming input yields a /// conforming output (plain `!` on the words would set every tail bit). @@ -1121,6 +1656,129 @@ pub fn mask_not_assign(dst: &mut [u64], n_rows: usize) { clear_mask_tail(dst, n_rows); } +/// Sets bits `[lo, hi)` of `out_words` to `1` and every other bit — inside +/// the range's own edge words and every word outside it — to `0`. The +/// **range WRITE** this file's care-masked matches are the READ half of: +/// [`ternary_match_u32_to_mask`] answers "which rows equal this pattern", +/// this answers "make exactly these rows true, and nothing else." +/// +/// # Why this exists +/// +/// `lance-graph-quack`'s `Filter::prefix_u32` spells a full ternary-match +/// PREDICATE to express what is really a contiguous row-range write, and +/// says so in its own doc comment: the range write is missing from +/// `ndarray::simd`, and per the missing-capability STOP rule a consumer does +/// not hand-roll it one layer up. This repo's own `examples/hex_tenant_mq_probe.rs` +/// hand-rolled the identical shape as a private `range_reveal` and measured +/// it at 161.8×–343.5× faster than the equivalent TCAM compare sweep, across +/// trie levels from a 65,536-row reveal down to a 1-row one +/// (`.claude/plans/gemm-ternlog-mask-consolidation-v1.md` §16.2). DuckDB +/// carries the identical operation over the identical representation +/// (`TemplatedValidityMask::SetRangeInvalid`), so the shape is not invented +/// here. +/// +/// This primitive is deliberately **address-blind**: it sets a bit range and +/// nothing else. Whether a column prefix legally maps to a contiguous row +/// range — or whether the backing column is sorted at all — is the +/// **caller's** decision, kept out of this layer on purpose. +/// +/// `lo == hi` is a legal empty range and writes an all-zero mask (not a +/// no-op: every bit is still overwritten). +/// +/// # No per-bit loop +/// +/// The write is at most three passes over `out_words`: every word strictly +/// before `lo`'s word is zeroed, every word strictly after `hi`'s last live +/// word is zeroed, whole interior words are written `u64::MAX`, and the (at +/// most two) edge words each get one computed mask ([`word_range_mask`]). +/// When `lo` and `hi` fall in the SAME word, that word is written exactly +/// once through the single-word branch — never as two overlapping edge +/// writes. +/// +/// # Panics +/// +/// Panics if `lo > hi`, or if `hi > out_words.len() * 64`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::mask_set_range; +/// +/// let mut words = [0u64; 2]; +/// mask_set_range(&mut words, 60, 70); +/// // bits 60..64 of word 0, bits 0..6 of word 1 — an adjacent-word range. +/// assert_eq!(words[0], 0xF000_0000_0000_0000); +/// assert_eq!(words[1], 0b0011_1111); +/// +/// // lo == hi is a legal empty range: an all-zero overwrite, not a no-op. +/// words[0] = u64::MAX; +/// mask_set_range(&mut words, 3, 3); +/// assert_eq!(words, [0, 0]); +/// ``` +#[inline] +pub fn mask_set_range(out_words: &mut [u64], lo: usize, hi: usize) { + assert!(lo <= hi, "mask_set_range: lo={lo} > hi={hi}"); + let words = mask_words_for(hi); + assert!(out_words.len() >= words, "mask_set_range: out_words.len()={} < required {}", out_words.len(), words); + + if lo == hi { + fill_words(out_words, 0); + return; + } + + // `lo < hi` from here on, so `hi >= 1` and `hi - 1` cannot underflow. + let lo_word = lo / 64; + let hi_word = (hi - 1) / 64; + + fill_words(&mut out_words[..lo_word], 0); + fill_words(&mut out_words[hi_word + 1..], 0); + + if lo_word == hi_word { + // The single-word case: exactly one write, never two overlapping + // edge writes. + out_words[lo_word] = word_range_mask(lo % 64, hi - lo_word * 64); + } else { + out_words[lo_word] = word_range_mask(lo % 64, 64); + fill_words(&mut out_words[lo_word + 1..hi_word], u64::MAX); + out_words[hi_word] = word_range_mask(0, hi - hi_word * 64); + } +} + +/// Writes `value` into every word of `dst`, walking `as_chunks_mut::()` +/// and storing one `U64x8` per chunk — the same lane walk every contiguous +/// word op in this file uses (see the NORMATIVE note above and `mask_and`'s +/// body), rather than a scalar `iter_mut` loop. +/// +/// The tail is written scalar-wise on purpose: a padded-tail `from_array` would +/// have to read `dst`'s surplus lanes back before storing them, and there is +/// nothing to read here — every lane of a constant fill has the same value, so +/// the tail is a straight copy of `value` into the remaining words. +#[inline] +fn fill_words(dst: &mut [u64], value: u64) { + const L: usize = crate::simd::U64x8::LANES; + let splat = crate::simd::U64x8::splat(value).to_array(); + let (chunks, tail) = dst.as_chunks_mut::(); + for c in chunks.iter_mut() { + *c = splat; + } + for w in tail.iter_mut() { + *w = value; + } +} + +/// One word's worth of the half-open range `[lo_bit, hi_bit)` set, +/// `0 <= lo_bit <= hi_bit <= 64` — the edge-word primitive [`mask_set_range`] +/// composes. Built from two "bits below N" masks rather than one shifted +/// range so that `hi_bit == 64` never computes `1u64 << 64` (a shift amount +/// equal to the type's own bit width, which panics under overflow checks). +#[inline(always)] +fn word_range_mask(lo_bit: usize, hi_bit: usize) -> u64 { + debug_assert!(lo_bit <= hi_bit && hi_bit <= 64, "word_range_mask: lo_bit={lo_bit} hi_bit={hi_bit}"); + let below_hi = if hi_bit == 64 { u64::MAX } else { (1u64 << hi_bit) - 1 }; + let below_lo = if lo_bit == 0 { 0 } else { (1u64 << lo_bit) - 1 }; + below_hi & !below_lo +} + /// `dst = a ^ b`, elementwise over `u64` mask words — symmetric difference. /// XOR preserves the trailing-zero guarantee iff both inputs conform /// (`0 ^ 0 = 0`). Its own primitive, with its own realization on every @@ -3270,6 +3928,336 @@ mod tests { } } + // ── u8 comparison family (`*_u8_to_mask`) ─────────────────────────── + + /// Adversarial u8 corpus: boundary values plus randomness — the + /// byte-width sibling of [`i32_corpus`] for the family below. + fn u8_corpus(n: usize, seed: u64) -> Vec { + let mut s = seed; + let edge = [0u8, 1, 127, 128, 254, 255]; + (0..n) + .map(|i| { + if i % 5 == 0 { + edge[(splitmix(&mut s) % 6) as usize] + } else { + splitmix(&mut s) as u8 + } + }) + .collect() + } + + const U8_LENS: [usize; 6] = [1, 63, 64, 65, 127, 200]; + + /// Randomized agreement for all six `*_u8_to_mask` functions against a + /// scalar per-element oracle written here (never against the shipped + /// path's own tail handling, which would be tautological). + /// + /// Anti-vacuity: at least one (length, threshold, comparison) case must + /// select neither nothing nor everything — otherwise a function that + /// always answered all-zero or all-one for this corpus could pass by + /// accident. + #[test] + fn u8_family_matches_scalar_reference_randomized() { + let mut saw_a_partial_match = false; + for &n in &U8_LENS { + let v = u8_corpus(n, 0xFEED_BEEF); + let words = n.div_ceil(64); + for &t in &[0u8, 1, 42, 127, 128, 254, 255] { + // Dirty, over-long destination on every call: each function + // must overwrite it (never OR into it) and must clear the + // one surplus word past `words`. + let mut out = vec![u64::MAX; words + 1]; + + eq_u8_to_mask(&v, t, &mut out); + let eq = scalar_pred_mask(n, |i| v[i] == t); + assert_eq!(&out[..words], &eq[..], "eq n={n} t={t}"); + assert_eq!(out[words], 0, "eq surplus word n={n} t={t}"); + + ne_u8_to_mask(&v, t, &mut out); + let ne = scalar_pred_mask(n, |i| v[i] != t); + assert_eq!(&out[..words], &ne[..], "ne n={n} t={t}"); + assert_eq!(out[words], 0, "ne surplus word n={n} t={t}"); + + gt_u8_to_mask(&v, t, &mut out); + let gt = scalar_pred_mask(n, |i| v[i] > t); + assert_eq!(&out[..words], >[..], "gt n={n} t={t}"); + assert_eq!(out[words], 0, "gt surplus word n={n} t={t}"); + + lt_u8_to_mask(&v, t, &mut out); + let lt = scalar_pred_mask(n, |i| v[i] < t); + assert_eq!(&out[..words], <[..], "lt n={n} t={t}"); + assert_eq!(out[words], 0, "lt surplus word n={n} t={t}"); + + ge_u8_to_mask(&v, t, &mut out); + let ge = scalar_pred_mask(n, |i| v[i] >= t); + assert_eq!(&out[..words], &ge[..], "ge n={n} t={t}"); + assert_eq!(out[words], 0, "ge surplus word n={n} t={t}"); + + le_u8_to_mask(&v, t, &mut out); + let le = scalar_pred_mask(n, |i| v[i] <= t); + assert_eq!(&out[..words], &le[..], "le n={n} t={t}"); + assert_eq!(out[words], 0, "le surplus word n={n} t={t}"); + + for reference in [&eq, &ne, >, <, &ge, &le] { + let pc = crate::bitwise::popcount_batch_u64(reference); + if pc != 0 && pc != n as u64 { + saw_a_partial_match = true; + } + } + } + } + assert!(saw_a_partial_match, "anti-vacuity: every (n, t, op) case selected either nothing or everything"); + } + + /// `gt`/`lt` must compare `u8` as UNSIGNED, not as if it were a two's + /// complement `i8`. A signed compare gets both these boundary pairs + /// backwards: 0x80 is 128 unsigned but -128 signed, against 0x7F (127 + /// either way). + #[test] + fn gt_lt_u8_to_mask_are_unsigned_not_signed() { + let mut out = [0u64; 1]; + + gt_u8_to_mask(&[0x80u8], 0x7F, &mut out); + assert_eq!(out[0], 0b1, "0x80 > 0x7F must be true unsigned (signed: -128 > 127 is false)"); + + gt_u8_to_mask(&[0x7Fu8], 0x80, &mut out); + assert_eq!(out[0], 0, "0x7F > 0x80 must be false unsigned (signed: 127 > -128 is true)"); + + lt_u8_to_mask(&[0x7Fu8], 0x80, &mut out); + assert_eq!(out[0], 0b1, "0x7F < 0x80 must be true unsigned (signed: 127 < -128 is false)"); + + lt_u8_to_mask(&[0x80u8], 0x7F, &mut out); + assert_eq!(out[0], 0, "0x80 < 0x7F must be false unsigned (signed: -128 < 127 is true)"); + } + + /// For the complement forms (`ne`/`ge`/`le`), every bit at or past + /// `values.len()` — the tail of the last live word AND every surplus + /// word — must be clear. This is the case a naive `!mask` gets wrong: it + /// sets every one of those bits, since bitwise NOT has no notion of + /// "past the end". + #[test] + fn complement_forms_clear_every_trailing_bit() { + fn assert_tail_clear(out: &[u64], n: usize, label: &str) { + for i in n..out.len() * 64 { + assert_eq!((out[i / 64] >> (i % 64)) & 1, 0, "{label}: bit {i} (n={n}) must be clear"); + } + } + + for &n in &[65usize, 100] { + let v = u8_corpus(n, 0x1234_5678); + let words = n.div_ceil(64); + // Two surplus words, every bit pre-set, so a writer that merely + // narrows without re-clearing (or that fixes only the last live + // word) cannot pass by accident. + let mut out = vec![u64::MAX; words + 2]; + + ne_u8_to_mask(&v, 7, &mut out); + assert_tail_clear(&out, n, "ne"); + + ge_u8_to_mask(&v, 7, &mut out); + assert_tail_clear(&out, n, "ge"); + + le_u8_to_mask(&v, 7, &mut out); + assert_tail_clear(&out, n, "le"); + } + } + + /// A `u64` corpus that deliberately over-samples the half of the domain a + /// SIGNED compare gets wrong — values at or above `1 << 63` — plus the + /// boundary pair either side of it. + fn u64_corpus(n: usize, seed: u64) -> Vec { + let mut s = seed; + let edge = [ + 0u64, + 1, + 0x7FFF_FFFF_FFFF_FFFF, // i64::MAX — the last value a signed compare reads as positive + 0x8000_0000_0000_0000, // the first it reads as negative + 0x8000_0000_0000_0001, + u64::MAX, + 1 << 32, // above the i32 family's reach: why G2 exists at all + u64::from(u32::MAX) + 1, + ]; + (0..n) + .map(|i| { + if i % 3 == 0 { + edge[(splitmix(&mut s) % edge.len() as u64) as usize] + } else { + // Full 64-bit spread, half of it with the top bit set. + splitmix(&mut s) + } + }) + .collect() + } + + /// Lengths that straddle BOTH boundaries this width has: the 8-lane + /// register and the 64-bit word (8 registers to a word). 7/8/9 exercise a + /// partial first group; 63/64/65 a partial last group in a full word; 200 + /// several whole words plus a ragged tail. + const U64_LENS: [usize; 8] = [1, 7, 8, 9, 63, 64, 65, 200]; + + /// Randomized agreement for all six `*_u64_to_mask` functions against a + /// scalar per-element oracle written here — never against the shipped + /// path's own tail or packing logic, which would be tautological. + /// + /// Anti-vacuity: at least one (length, threshold, comparison) case must + /// select neither nothing nor everything, otherwise a function that always + /// answered all-zero or all-one for this corpus would pass by accident. + #[test] + fn u64_family_matches_scalar_reference_randomized() { + let mut saw_a_partial_match = false; + for &n in &U64_LENS { + let v = u64_corpus(n, 0x0BAD_C0DE_F00D); + let words = n.div_ceil(64); + for &t in &[0u64, 1, 0x7FFF_FFFF_FFFF_FFFF, 0x8000_0000_0000_0000, 1 << 32, u64::MAX] { + // Dirty, over-long destination on every call: each function + // must fully overwrite it, never OR into it, and must clear + // the surplus word past `words`. + let mut out = vec![u64::MAX; words + 1]; + + let check = |out: &[u64], pred: &dyn Fn(u64) -> bool, label: &str| { + for (i, &x) in v.iter().enumerate() { + let got = (out[i / 64] >> (i % 64)) & 1 == 1; + assert_eq!(got, pred(x), "{label} n={n} t={t} i={i} x={x:#018x}"); + } + for i in n..out.len() * 64 { + assert_eq!((out[i / 64] >> (i % 64)) & 1, 0, "{label} tail bit {i} n={n} t={t}"); + } + }; + + eq_u64_to_mask(&v, t, &mut out); + check(&out, &|x| x == t, "eq"); + ne_u64_to_mask(&v, t, &mut out); + check(&out, &|x| x != t, "ne"); + gt_u64_to_mask(&v, t, &mut out); + check(&out, &|x| x > t, "gt"); + ge_u64_to_mask(&v, t, &mut out); + check(&out, &|x| x >= t, "ge"); + lt_u64_to_mask(&v, t, &mut out); + check(&out, &|x| x < t, "lt"); + le_u64_to_mask(&v, t, &mut out); + check(&out, &|x| x <= t, "le"); + + let hits = v.iter().filter(|&&x| x > t).count(); + if hits > 0 && hits < n { + saw_a_partial_match = true; + } + } + } + assert!( + saw_a_partial_match, + "corpus never produced a partial `gt` selection — the agreement above is vacuous" + ); + } + + /// THE falsifier for this width, and the reason G2 could not be served by + /// narrowing to the existing `i32` family. + /// + /// A `u64` at or above `1 << 63` reads as NEGATIVE under a signed 64-bit + /// compare, so an implementation that reached for a signed lane op — which + /// is the ONLY ordered 64-bit compare wasm has, and the only one AVX2 has + /// at this width — passes every small-value test and inverts the answer on + /// exactly half the domain. Every arm that lacks an unsigned instruction + /// has to flip the sign bit of both operands; this is what proves it did. + #[test] + fn gt_lt_u64_to_mask_are_unsigned_not_signed() { + const HI: u64 = 0x8000_0000_0000_0000; // signed: i64::MIN + const LO: u64 = 0x7FFF_FFFF_FFFF_FFFF; // signed: i64::MAX + let v = [LO, HI, u64::MAX, 0]; + let mut out = [0u64; 1]; + + gt_u64_to_mask(&v, LO, &mut out); + // Unsigned: HI and u64::MAX exceed LO → bits 1, 2. + // Signed: HI is i64::MIN and would NOT exceed i64::MAX → bit 1 lost. + assert_eq!( + out[0], 0b0110, + "{HI:#018x} > {LO:#018x} must be true unsigned (signed: i64::MIN > i64::MAX is false)" + ); + + lt_u64_to_mask(&v, HI, &mut out); + // Unsigned: only LO and 0 are below HI → bits 0, 3. + assert_eq!( + out[0], 0b1001, + "{LO:#018x} < {HI:#018x} must be true unsigned (signed: i64::MAX < i64::MIN is false)" + ); + + // And the complement pair must agree with their strict siblings. + ge_u64_to_mask(&v, HI, &mut out); + assert_eq!(out[0], 0b0110, "ge at the sign boundary"); + le_u64_to_mask(&v, LO, &mut out); + assert_eq!(out[0], 0b1001, "le at the sign boundary"); + } + + /// The packing falsifier, specific to this width. + /// + /// `U64x8` yields 8 bits per chunk and eight chunks share one 64-bit word, + /// so group `g` must land at byte `g % 8` of word `g / 8`. The `u8` family + /// has no packing at all (64 lanes == one whole word), so nothing upstream + /// of here exercises the shift — a wrong `(g % 8) * 8` would survive every + /// other test in this file whose length happens to be a multiple of 64. + /// + /// Built by selecting exactly ONE group at a time and asserting the mask is + /// that group's byte and nothing else. + #[test] + fn u64_packing_places_each_group_in_its_own_byte() { + const NEEDLE: u64 = 0xDEAD_BEEF_CAFE_F00D; + for g in 0..8usize { + // 64 values = 8 groups = exactly one word. Only group `g` matches. + let mut v = vec![0u64; 64]; + for lane in 0..8 { + v[g * 8 + lane] = NEEDLE; + } + let mut out = [0u64; 1]; + eq_u64_to_mask(&v, NEEDLE, &mut out); + assert_eq!(out[0], 0xFFu64 << (g * 8), "group {g} must occupy byte {g} alone (got {:#018x})", out[0]); + } + + // Two words: group 8 is byte 0 of word 1, not byte 0 of word 0. + let mut v = vec![0u64; 128]; + for lane in 0..8 { + v[64 + lane] = NEEDLE; + } + let mut out = [0u64; 2]; + eq_u64_to_mask(&v, NEEDLE, &mut out); + assert_eq!(out[0], 0, "word 0 must stay clear when only group 8 matches"); + assert_eq!(out[1], 0xFF, "group 8 is byte 0 of word 1"); + } + + /// The consumer-shaped test: the half-open address window `lo <= x < hi` + /// that motivated G2 in the first place. + /// + /// Every offset here exceeds `2^32`, which is the point — narrowing them + /// into the existing `i32` family is not a precision trade, it is unsound. + /// The window is composed from two of this family's masks through + /// [`mask_and`], which is how a caller actually expresses a range. + #[test] + fn an_address_window_over_offsets_above_2_32_needs_the_u64_family() { + let offsets: Vec = vec![ + 0x0000_0001_0000_0000, // 2^32, below the window + 0x0000_0002_0000_0000, // in + 0x0000_0002_8000_0000, // in + 0x0000_0003_0000_0000, // == hi, EXCLUDED (half-open) + 0x0000_0004_0000_0000, // above + 0xFFFF_FFFF_FFFF_FFFF, // above, and negative under a signed compare + ]; + let lo: u64 = 0x0000_0002_0000_0000; + let hi: u64 = 0x0000_0003_0000_0000; + + let mut ge_lo = [0u64; 1]; + let mut lt_hi = [0u64; 1]; + let mut window = [0u64; 1]; + ge_u64_to_mask(&offsets, lo, &mut ge_lo); + lt_u64_to_mask(&offsets, hi, &mut lt_hi); + mask_and(&ge_lo, <_hi, &mut window); + + assert_eq!(window[0], 0b000110, "half-open window [lo, hi) selects exactly elements 1 and 2"); + + // Anti-vacuity on both halves: neither input mask may already be the + // answer, or the `mask_and` is decorative and the test would pass with + // one of the two predicates broken. + assert_ne!(ge_lo[0], window[0], "ge_lo alone must not equal the window"); + assert_ne!(lt_hi[0], window[0], "lt_hi alone must not equal the window"); + } + #[test] fn mask_not_clears_the_tail_and_round_trips() { for &n in &LENS { @@ -4030,4 +5018,121 @@ mod tests { let mut out = vec![0u64; 2]; gt_i32_to_mask_under(&values, 0, &gate, &mut out); } + + // ── mask_set_range: the range WRITE (N1, + // `.claude/plans/gemm-ternlog-mask-consolidation-v1.md` §16.6) ──────── + + /// Independent bit-serial reference: bit `b` set iff `lo <= b < hi`. + /// Never calls `mask_set_range` or `word_range_mask`. + fn reference_set_range(words: usize, lo: usize, hi: usize) -> Vec { + let mut v = vec![0u64; words]; + for b in lo..hi { + v[b / 64] |= 1u64 << (b % 64); + } + v + } + + #[test] + fn mask_set_range_matches_scalar_reference_randomized() { + let mut seed = 0xB000_0000_5E7A_11u64; + for &words in &[1usize, 2, 3, 4, 7, 8, 16] { + let capacity = words * 64; + for _ in 0..40 { + let lo = (splitmix64(&mut seed) as usize) % (capacity + 1); + let span = (splitmix64(&mut seed) as usize) % (capacity + 1 - lo); + let hi = lo + span; + let mut got = vec![u64::MAX; words]; // pre-dirtied: an OR-er fails immediately + mask_set_range(&mut got, lo, hi); + assert_eq!(got, reference_set_range(words, lo, hi), "words={words} lo={lo} hi={hi}"); + } + } + // Anti-vacuity: a genuinely non-trivial range sets EXACTLY `hi - lo` + // bits, not "some" bits — catches an always-set-everything or + // always-set-nothing implementation that could otherwise still pass + // the boundary and randomized cases above by accident of which + // random `(lo, hi)` pairs happened to be drawn. + let mut got = vec![0u64; 4]; + mask_set_range(&mut got, 7, 250); + let popcount: u32 = got.iter().map(|w| w.count_ones()).sum(); + assert_eq!(popcount, 250 - 7, "popcount must equal hi - lo exactly"); + } + + /// Every boundary shape named in the primitive's own doc comment, each + /// checked explicitly rather than only through the randomized fuzz above. + #[test] + fn mask_set_range_boundary_shapes() { + // lo == hi: the empty range. All-zero output — NOT a no-op, the + // destination is still fully overwritten (pre-dirtied to catch a + // "lo == hi means skip the call" shortcut). + let mut got = vec![u64::MAX; 2]; + mask_set_range(&mut got, 0, 0); + assert_eq!(got, vec![0, 0], "lo == hi == 0 must zero everything"); + let mut got = vec![u64::MAX; 2]; + mask_set_range(&mut got, 128, 128); + assert_eq!(got, vec![0, 0], "lo == hi == capacity must zero everything"); + + // lo == 0: the range starts at the very first bit. + let mut got = vec![u64::MAX; 2]; + mask_set_range(&mut got, 0, 70); + assert_eq!(got, reference_set_range(2, 0, 70), "lo == 0"); + + // hi == words * 64: the range runs all the way to the last bit. + let mut got = vec![u64::MAX; 2]; + mask_set_range(&mut got, 40, 128); + assert_eq!(got, reference_set_range(2, 40, 128), "hi == capacity"); + + // Both endpoints inside ONE word: the single-word branch, one write. + let mut got = vec![u64::MAX; 2]; + mask_set_range(&mut got, 2, 5); + assert_eq!(got[0], 0b0001_1100, "both endpoints in one word: bits 2,3,4"); + assert_eq!(got[1], 0, "the untouched second word must be cleared too"); + + // Endpoints in ADJACENT words, neither word-aligned. + let mut got = vec![u64::MAX; 2]; + mask_set_range(&mut got, 60, 70); + assert_eq!(got[0], 0xF000_0000_0000_0000, "adjacent words: low word's tail (bits 60..64)"); + assert_eq!(got[1], 0b0011_1111, "adjacent words: high word's head (bits 0..6)"); + + // Both endpoints word-ALIGNED (`lo % 64 == 0 && hi % 64 == 0`): the + // range is exactly one interior word, reached via the single-word + // branch (lo_word == hi_word here, not the interior-word loop). + let mut got = vec![u64::MAX; 3]; + mask_set_range(&mut got, 64, 128); + assert_eq!(got, vec![0, u64::MAX, 0], "word-aligned range is exactly one full word"); + + // A range of exactly 1 bit. + let mut got = vec![u64::MAX; 1]; + mask_set_range(&mut got, 5, 6); + assert_eq!(got[0], 1u64 << 5, "a single-bit range sets exactly that bit"); + } + + /// The falsifier for "full overwrite, not OR": every word starts + /// all-ones, and afterwards a bit survives iff it is inside `[lo, hi)` — + /// checked bit-by-bit across the whole buffer, including the surplus + /// bits of the partially-live last word, which an OR-based "fix" would + /// leave set. + #[test] + fn mask_set_range_overwrites_a_dirty_destination_rather_than_oring() { + let mut got = vec![u64::MAX; 3]; + mask_set_range(&mut got, 70, 130); + for b in 0..192 { + let want = (70..130).contains(&b); + let bit = (got[b / 64] >> (b % 64)) & 1 == 1; + assert_eq!(bit, want, "bit {b}"); + } + } + + #[test] + #[should_panic(expected = "mask_set_range: lo=")] + fn mask_set_range_rejects_lo_greater_than_hi() { + let mut out = vec![0u64; 1]; + mask_set_range(&mut out, 5, 3); + } + + #[test] + #[should_panic(expected = "mask_set_range: out_words.len()=")] + fn mask_set_range_rejects_hi_past_capacity() { + let mut out = vec![0u64; 1]; // capacity 64 + mask_set_range(&mut out, 0, 65); + } } diff --git a/src/simd_neon.rs b/src/simd_neon.rs index 173f975e..487b8688 100644 --- a/src/simd_neon.rs +++ b/src/simd_neon.rs @@ -2057,6 +2057,57 @@ impl U64x2 { let b = other.to_array(); Self::from_array([a[0].max(b[0]), a[1].max(b[1])]) } + + /// Lane-wise equality as a packed bitmask; bit `i` set iff + /// `self.lane(i) == other.lane(i)`. Only bits `0..2` are meaningful — + /// the upper 6 bits of the returned `u8` are always zero, which + /// `U64x8::cmpeq_mask` below relies on when it OR-shifts four of these + /// 2-lane results into one 8-bit result (a nonzero bit above position 1 + /// here would corrupt a neighbouring group). + /// + /// `vceqq_u64` (`CMEQ`) produces an all-ones (`u64::MAX`) or all-zero + /// lane per comparison; each lane is extracted with `vgetq_lane_u64` + /// and folded to its own bit. This file's other movemask-style + /// reduction (`quad_mask4`, used by `I32x16::{cmpge_zero_mask, + /// gt_bitmask}` further down) packs a `uint32x4_t` compare via a + /// weighted AND + `vaddvq_u32`, but its weights `[1, 2, 4, 8]` are + /// baked in for four 32-bit lanes -> four bits; reusing it here would + /// need reinterpreting this `uint64x2_t` as `uint32x4_t` (each 64-bit + /// lane becomes two identical 32-bit halves — the same reinterpret + /// `U64x8`'s `PartialEq::eq` impl already relies on) plus a + /// *different*, zero-padded weight set so the duplicate halves aren't + /// double-counted. That is a second, less obviously correct convention + /// just to save two scalar lane reads, so direct extraction is used + /// instead — simpler and unambiguously correct for only two lanes. + #[inline(always)] + pub fn cmpeq_mask(self, other: Self) -> u8 { + // SAFETY: NEON baseline; pure register compare plus two in-register + // lane reads (`vgetq_lane_u64` reads a lane already held in a NEON + // register — no memory access). + unsafe { + let cmp = vceqq_u64(self.0, other.0); + (vgetq_lane_u64(cmp, 0) & 1) as u8 | (((vgetq_lane_u64(cmp, 1) & 1) as u8) << 1) + } + } + + /// Lane-wise **unsigned** greater-than as a packed bitmask; bit `i` set + /// iff `self.lane(i) > other.lane(i)` under unsigned ordering. Same bit + /// convention and upper-bits-zero guarantee as `cmpeq_mask` above. + /// + /// `vcgtq_u64` maps to `CMHI` ("compare higher", the unsigned form) on + /// aarch64 — a genuine hardware unsigned 64-bit compare, so unlike a + /// backend with only a signed 64-bit compare (which must XOR the sign + /// bit of both operands first to fake an unsigned ordering), no bias + /// correction is needed on this arm. + #[inline(always)] + pub fn cmpgt_mask(self, other: Self) -> u8 { + // SAFETY: NEON baseline; pure register compare plus two in-register + // lane reads. + unsafe { + let cmp = vcgtq_u64(self.0, other.0); + (vgetq_lane_u64(cmp, 0) & 1) as u8 | (((vgetq_lane_u64(cmp, 1) & 1) as u8) << 1) + } + } } #[cfg(target_arch = "aarch64")] @@ -2840,6 +2891,40 @@ impl U64x8 { }) })) } + + /// Lane-wise equality as a packed 8-bit bitmask, bit `i` set iff + /// `self.lane(i) == other.lane(i)` (LSB-first — lane `0` is bit `0` — + /// same convention as `U32x16::eq_bitmask` / `I32x16::gt_bitmask` + /// elsewhere in this file, and matching the scalar backend's + /// `U64x8::cmpeq_mask` oracle bit-for-bit). + /// + /// Composed from four `U64x2::cmpeq_mask` calls, one per fanned-out + /// register: group `p` (lanes `2p` and `2p+1`) occupies bits `2p..2p+2` + /// of the result. `U64x8` is four independent `uint64x2_t` registers + /// with no single 8-lane NEON compare to reduce in one step, so this + /// composes four already-correct 2-lane masks rather than reinventing + /// the reduction at 8-lane width; `U64x2::cmpeq_mask`'s own doc comment + /// is what guarantees its upper 6 bits are zero, which is what makes + /// this OR-shift safe against corrupting a neighbouring group. + #[inline(always)] + pub fn cmpeq_mask(self, other: Self) -> u8 { + let mut m = 0u8; + for p in 0..4 { + m |= self.0[p].cmpeq_mask(other.0[p]) << (2 * p); + } + m + } + + /// Lane-wise **unsigned** greater-than as a packed 8-bit bitmask. Same + /// group-of-2 composition and bit convention as `cmpeq_mask` above. + #[inline(always)] + pub fn cmpgt_mask(self, other: Self) -> u8 { + let mut m = 0u8; + for p in 0..4 { + m |= self.0[p].cmpgt_mask(other.0[p]) << (2 * p); + } + m + } } #[cfg(target_arch = "aarch64")] diff --git a/src/simd_scalar.rs b/src/simd_scalar.rs index 3f1a6b35..6ed3efd6 100644 --- a/src/simd_scalar.rs +++ b/src/simd_scalar.rs @@ -1940,6 +1940,71 @@ impl U64x8 { } sum } + + /// Lane-wise equality comparison. Returns an 8-bit mask: bit `i` is set + /// iff `self[i] == other[i]`. `U64x8` has exactly 8 lanes, so the mask + /// is `u8` — one bit per lane, and nothing above bit 7: bits at or above + /// 8 simply do not exist in the return type (contrast + /// `U8x64::cmpeq_mask` above, whose 64 lanes need a `u64` mask). + /// + /// Composed as four 2-lane groups (`p in 0..4`, lanes `2p`/`2p+1`) + /// rather than a flat 8-iteration loop — deliberate: storage flat, + /// composition 2×4. Every non-avx512 backend realizes `U64x8` as four + /// `U64x2` pairs (`simd_neon.rs`/`simd_wasm.rs`: + /// `pub struct U64x8(pub [U64x2; 4])`); walking the same four pairs here + /// keeps this arm's shape identical to theirs even though this file's + /// own storage is a flat `[u64; 8]` (see `impl_int_type!` above). Do not + /// "simplify" this into a single `for i in 0..8` loop. + /// + /// # Example + /// ```rust,ignore + /// let a = U64x8::from_array([1, 2, 3, 4, 5, 6, 7, 8]); + /// let b = U64x8::from_array([1, 0, 3, 0, 5, 0, 7, 0]); + /// assert_eq!(a.cmpeq_mask(b), 0b0101_0101); + /// ``` + #[inline(always)] + pub fn cmpeq_mask(self, other: Self) -> u8 { + let mut mask: u8 = 0; + for p in 0..4 { + if self.0[2 * p] == other.0[2 * p] { + mask |= 1 << (2 * p); + } + if self.0[2 * p + 1] == other.0[2 * p + 1] { + mask |= 1 << (2 * p + 1); + } + } + mask + } + + /// Lane-wise **unsigned** greater-than comparison. Returns an 8-bit + /// mask: bit `i` is set iff `self[i] > other[i]`. Symmetric to + /// `cmpeq_mask` above — same 8-bits-only footprint, same 2×4 grouping, + /// same "bits at or above 8 do not exist" contract. + /// + /// Plain `>` on Rust's `u64` is *already* the unsigned ordering, so + /// unlike wasm's `i64x2_gt` (signed-only) or an AVX2 byte-width compare + /// (`_mm256_cmpgt_epi8`, also signed-only, needing a sign-bias XOR to + /// become unsigned), no sign-bias trick is needed at this lane width. + /// + /// # Example + /// ```rust,ignore + /// let a = U64x8::from_array([1, 2, 3, 4, 5, 6, 7, 8]); + /// let b = U64x8::from_array([0, 2, 0, 4, 0, 6, 0, 8]); + /// assert_eq!(a.cmpgt_mask(b), 0b0101_0101); + /// ``` + #[inline(always)] + pub fn cmpgt_mask(self, other: Self) -> u8 { + let mut mask: u8 = 0; + for p in 0..4 { + if self.0[2 * p] > other.0[2 * p] { + mask |= 1 << (2 * p); + } + if self.0[2 * p + 1] > other.0[2 * p + 1] { + mask |= 1 << (2 * p + 1); + } + } + mask + } } impl U64x4 { diff --git a/src/simd_wasm.rs b/src/simd_wasm.rs index 99cda4b3..dff29d18 100644 --- a/src/simd_wasm.rs +++ b/src/simd_wasm.rs @@ -1922,6 +1922,55 @@ pub mod wasm32_simd { }) })) } + + /// Per-lane equality as a packed 8-bit bitmask: bit `i` set iff + /// `self[i] == other[i]`. Bits at or above 8 are always zero — the + /// `u8` return type has room for exactly 8, and there are only 8 + /// lanes. + /// + /// `u64x2_eq` needs no sign bias: bit equality is sign-agnostic, and + /// wasm SIMD128 reflects that directly — it ships `u64x2_eq` as a + /// plain `pub use i64x2_eq as u64x2_eq` alias, because there is no + /// separate unsigned/signed equality instruction to begin with. + /// Each compare yields an all-ones/all-zeros lane; `i64x2_bitmask` + /// extracts the 2 per-quad bits directly — the same reduction + /// `F64x8::cmp_mask` above already uses to fold 4× `v128` compare + /// results into one packed mask. + #[inline(always)] + pub fn cmpeq_mask(self, other: Self) -> u8 { + let mut bits: u8 = 0; + for p in 0..4 { + bits |= (i64x2_bitmask(u64x2_eq(self.0[p].0, other.0[p].0)) as u8) << (2 * p); + } + bits + } + + /// Per-lane **unsigned** greater-than as a packed 8-bit bitmask: + /// bit `i` set iff `self[i] > other[i]`. Bits at or above 8 are + /// always zero. + /// + /// wasm SIMD128 has no unsigned ordered 64-bit compare — only the + /// signed `i64x2_lt`/`i64x2_gt`/`i64x2_le`/`i64x2_ge` family, which + /// stdarch documents as treating the operands "as if they were two + /// vectors of 2 sixty-four-bit signed integers". So both operands + /// are XORed with the sign bit (`0x8000_0000_0000_0000`) before the + /// signed compare — an order-preserving bijection from unsigned to + /// signed ordering. Same trick, same reasoning, as + /// `U8x32::cmpgt_mask` in `simd_avx2.rs`, which documents and uses + /// it at byte width: "AVX2 only has signed `_mm256_cmpgt_epi8`, so + /// we XOR both operands with `0x80` to convert unsigned ↔ signed + /// (preserves ordering for unsigned compare)." + #[inline(always)] + pub fn cmpgt_mask(self, other: Self) -> u8 { + let bias = u64x2_splat(0x8000_0000_0000_0000); + let mut bits: u8 = 0; + for p in 0..4 { + let a_s = v128_xor(self.0[p].0, bias); + let b_s = v128_xor(other.0[p].0, bias); + bits |= (i64x2_bitmask(i64x2_gt(a_s, b_s)) as u8) << (2 * p); + } + bits + } } impl Add for U64x8 {