From d9459f0ab5e5649b7a60b3b6b56495d822dab5dd Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Mon, 14 Sep 2026 13:52:03 +0000 Subject: [PATCH 1/9] Add the hex-tenant probe (D-GTM-0m): top-down + spread on one Morton-keyed SoA W0 probe over 65,536 rows = a 256x256 axial hex field keyed by Morton(q, r), payload = the V3 12-byte register read 6x(u8:u8) with rail d = direction d. Three gates green (range reveal == TCAM reveal; Morton spread == independent axial BFS incl. plasticity bytes; 0 heap B/step). Measured: node reveal is a range write at 49-99 ns vs 22 us for the ternary_match sweep (228-462x); step = x*ternlogq + n with ternlogq = 291 ns/pass, n = 17.3 us (chain = 1.7 % of the step); coal (one re-chain) = 0.48 maintained steps; the NNUE delta- frontier arm halves n at identical closure. Names the missing T1 word: a mask-level hex neighbour shift on the Morton lattice. Blackboard entry + plan section 14. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/blackboard.md | 53 ++ .../gemm-ternlog-mask-consolidation-v1.md | 25 + Cargo.toml | 4 + examples/hex_tenant_mq_probe.rs | 578 ++++++++++++++++++ 4 files changed, 660 insertions(+) create mode 100644 examples/hex_tenant_mq_probe.rs diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 45013786..044e1396 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -1,3 +1,56 @@ +## 2026-09-14 (2) — D-GTM-0m: the hex TENANT — top-down traversal AND spread on ONE Morton-keyed SoA; `step = x·ternlogq + n` measured, and the chain is 1.7 % of it + +Probe: `examples/hex_tenant_mq_probe.rs` (`--release`, committed; output banked +in the session scratchpad). Operator statement it builds (2026-09-14): *"static +traversal top down AND plasticity (spread) in the same substrate — SoA gets a +hex tenant with 6×2×8 bit and the field is a trie (fixed spatial distribution)."* +This is `gemm-ternlog-mask-consolidation-v1.md` §9 M1/M1b/M2/M3 and §11.10 +(`substrate == mask geometry == projection surface`) executed on the merged #306 +facade, not argued. Substrate: 65,536 rows = 256×256 axial hex cells, **row = +Morton(q, r)**, payload = the V3 12-byte register read `6×(u8:u8)` with rail `d` += hex direction `d`, `u8:u8 = (permeability, strength)`. §9 R1 ("not obviously +the same six") is resolved by construction: adjacency and carving ARE the same +six once the rail index is the direction. Three gates, all green (32/32 cells): +range reveal == TCAM reveal at every level/prefix; Morton-arm spread == an +independent row-major axial BFS at every step (plasticity bytes compared too); +hot-path heap = **0 B/step** everywhere (counting allocator). + +**White — top-down is a RANGE, not a compare.** A trie node at nibble level L is +`2^(16−4L)` CONTIGUOUS rows, so revealing it is a range write: **49–99 ns** vs +**22.4–22.8 µs** for the general `ternary_match_u32_to_mask` sweep over the +address column — **228–462×**. That is the fixed-spatial-distribution dividend +stated as a number: the TCAM op stays for addresses that are NOT laid out (the +D-GTM-0l linker case); a minted, Morton-keyed tenant never pays it. + +**Grey — the cost model, fitted (identity gates, survivors held at 1175):** +`step = x·ternlogq + n` with **ternlogq = 291 ns/pass** (8 KiB masks, 0.285 +ns/word) and **n = 17.3 µs**, max residual 2.8 % over x ∈ {0,1,2,4,8,16,32}. +So at x = 1 the chain is **1.7 %** of the step; even x = 32 only doubles it. +`n` is the ONE non-mask op on the path — the per-active-bit hex shift (dilated- +integer add per direction). **That is the missing substrate word**: a mask-level +neighbour shift on the Morton lattice (`mask_shift_hex(state, d, dst)` — within +a nibble a 4×4 block shift, carries across blocks), which would fold `n` into a +handful of word passes. Filed, not built. First rung without it: the **NNUE +reading** — spread from the DELTA frontier (`scratch & !state`), never from the +accumulated state — gives the identical closure (gate green) at **8.8 µs** +(−48 %), 8.4 µs with the real gates. The E-Q8 control is in the table (degree-1 +arm: 7.0 µs, 301 survivors — a different closure, so a cost floor, not a cost +equivalence). + +**Coal.** One re-chain (regenerate a resident mask from its column, +`gt_i32_to_mask` over 256 KiB) = **8.9 µs = 30.6 ternlogq passes = 0.48 +maintained steps** at x = 4. M2 is linear: x → x±1 is one pass (291 ns), no +cliff. M1b generation (6 eligibility masks) = 75 µs once per mask generation. +Plasticity: 740 rows' strength bytes bumped IN the register (1,263 firings over +24 steps), on survivors only — the first version fired on every re-reached cell +the gates then removed, which is the "fire before you know it survived" bug the +reverse walk fixes. + +**Stated limits.** The T1 compare is i32-wide, so the u8 permeability column is +widened 4× for `gt_i32_to_mask` — `n_gen` and coal are UPPER bounds; a u8/u16 +compare-to-mask is a T1 addition. One fixture density (62 % permeable), one +tile size, timing floor 50 ms, no `perf`. No production caller; this is W0. + ## 2026-09-14 — AVX2 arm of the mask family MEASURED, not rewritten: 6 of 10 shapes were already packed, 4 earned intrinsic realizations **The pre-compaction plan was wrong, and the instrument said so before code diff --git a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md index f23c72a6..6f34abef 100644 --- a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md +++ b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md @@ -886,3 +886,28 @@ rather than from a linker. If enrichment rises and k=2/k=3 compression goes with the mechanism survives on the substrate it was actually proposed for. If the compression column looks like the table above, the tract framing is dead generally and white matter needs to be a structure after all. + +## §14 — D-GTM-0m MEASURED: the hex TENANT — one Morton-keyed SoA, top-down AND spread (2026-09-14) + +Probe `examples/hex_tenant_mq_probe.rs`; full numbers in `.claude/blackboard.md` +(2026-09-14 (2)). What it settles, against this plan's own rows: + +- **§9 R1 is closed [G] by construction, not by argument.** With the SoA row = + Morton(q, r) and rail `d` = hex direction `d`, adjacency and carving are the + same six. Grey = the rail bytes (permeability, strength — plastic, one owner); + white = the address prefix (a trie node = a contiguous row range). +- **M1 (reveal ahead) has a measured dividend for FIXED distributions:** a node + reveal is a range write, 49–99 ns, against 22 µs for the TCAM sweep — 228–462×. + `ternary_match_*` remains the op for addresses that were not minted in order. +- **M2/M3 are priced:** `step = x·ternlogq + n`, ternlogq = 291 ns/pass on 8 KiB + masks, n = 17.3 µs, residual 2.8 %. Chain depth is ~free (1.7 % at x = 1); the + reveal/spread term is the whole cost; coal (one re-chain) = 0.48 maintained + steps. The formula holds only when `n` is held fixed — with real gates the + survivor population moves with x and a naive fit reads 13–32 % residual. That + is a measurement-apparatus fact, recorded so the next ladder does not repeat it. +- **The missing primitive is named:** a mask-level hex neighbour shift on the + Morton lattice. Until it exists, spread from the delta frontier (the NNUE + reading) halves `n` with the identical closure. +- **0k holds on this shape too:** 0 heap bytes/step on every arm. +- Still open: the u8/u16 compare-to-mask (T1), the shift primitive (T1), a + second density and tile size, and `perf` counters for the residency claim. diff --git a/Cargo.toml b/Cargo.toml index 2b9c5fba..aae4ed0b 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -59,6 +59,10 @@ required-features = ["std"] name = "ternlog_amortization_probe" required-features = ["std"] +[[example]] +name = "hex_tenant_mq_probe" +required-features = ["std"] + [[example]] name = "ternlog_codegen_probe" required-features = ["std"] diff --git a/examples/hex_tenant_mq_probe.rs b/examples/hex_tenant_mq_probe.rs new file mode 100644 index 00000000..896e7af3 --- /dev/null +++ b/examples/hex_tenant_mq_probe.rs @@ -0,0 +1,578 @@ +//! W0 probe D-GTM-0m — the hex TENANT: static top-down traversal AND +//! plasticity (spread) in ONE SoA substrate, and the Mississippi Queen cost +//! model `step = x·ternlogq + n`, coal = the re-chain budget. +//! +//! Plan: `.claude/plans/gemm-ternlog-mask-consolidation-v1.md` §9 (M1/M1b/M2/M3) +//! and §11.10 (`substrate == mask geometry == projection surface`). Operator +//! statement (2026-09-14): *"static traversal top down AND plasticity (spread) +//! in the same substrate — SoA gets a hex tenant with 6×2×8 bit and the field +//! is a trie (fixed spatial distribution)."* +//! +//! ## The substrate +//! +//! 65,536 rows = a 256×256 axial hex field `(q, r)`. **Row index = Morton(q, r)** +//! (8+8 bits interleaved), so the field IS the nibble trie: a trie node at +//! level `L` (4·L address bits) is a CONTIGUOUS run of `2^(16-4L)` rows. The +//! tenant payload is the V3 12-byte register read as `6×(u8:u8)`: rail `d` = +//! hex direction `d`, `u8:u8` = `(permeability, strength)`. Adjacency and +//! carving are the SAME six here — that is the claim §9 R1 left open and this +//! probe builds rather than argues. +//! +//! ## The two readings, one register +//! +//! * **white — top-down (static, replayable):** revealing a trie node is a +//! RANGE mask (bits `[lo, hi)`), never a compare sweep. The general (non- +//! fixed-distribution) path is `ternary_match_u32_to_mask` over an address +//! column — a TCAM sweep of the whole column. Both are timed; both must +//! yield the same mask. +//! * **grey — spread (plastic, winner-update):** `state' = (state ∪ ⋃_d +//! shift_d(state ∧ elig_d)) ∧ tile`, chained through `x` resident gate masks +//! with `mask_ternlog_assign::`. Each cell newly reached via rail `d` +//! fires that rail's `strength` byte (saturating) in place — Hebbian, one +//! owner, no second structure. Degree-1 control per the E-Q8 rule: the +//! same run with ONE direction; any advantage that survives it is not hex. +//! +//! ## The cost model under test +//! +//! Maintained speed `x` (chain depth) costs `x · t_ternlogq + n` per step, +//! where `n` is the reveal/spread term. A maneuver (re-chaining a resident +//! mask from its column) costs coal `c`. The probe fits `t_ternlogq` and `n` +//! from a depth ladder and reports `c` in units of maintained steps. +//! +//! ## Gates (a mismatch aborts the run) +//! +//! * range reveal == TCAM reveal, every level, every prefix tried; +//! * Morton-arm spread == an independent row-major axial BFS, every step; +//! * hot-path heap bytes == 0 (counting allocator). +//! +//! cargo run --release --example hex_tenant_mq_probe --features std + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use ndarray::simd::ternlog::{AND3, OR2_AND}; +use ndarray::simd::{gt_i32_to_mask, mask_ternlog_assign, popcount_batch_u64, ternary_match_u32_to_mask}; + +// ── counting allocator: the 0k instrument ──────────────────────────────────── +static ALLOCED: AtomicUsize = AtomicUsize::new(0); +static COUNTING: AtomicUsize = AtomicUsize::new(0); + +struct Counting; +// SAFETY: a pure pass-through allocator. Every call forwards the caller's own +// `Layout` (and, for `dealloc`, the pointer that `alloc` returned for that +// layout) to `System` unchanged, so `System` upholds the `GlobalAlloc` +// contract on our behalf; the only added work is a relaxed atomic counter +// that never touches the allocation. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, l: Layout) -> *mut u8 { + if COUNTING.load(Ordering::Relaxed) == 1 { + ALLOCED.fetch_add(l.size(), Ordering::Relaxed); + } + // SAFETY: `l` is the layout the caller passed, forwarded verbatim. + unsafe { System.alloc(l) } + } + unsafe fn dealloc(&self, p: *mut u8, l: Layout) { + // SAFETY: `p` was returned by `System.alloc(l)` above for this same `l`. + unsafe { System.dealloc(p, l) } + } +} +#[global_allocator] +static A: Counting = Counting; + +fn count_on() { + ALLOCED.store(0, Ordering::Relaxed); + COUNTING.store(1, Ordering::Relaxed); +} +fn count_off() -> usize { + COUNTING.store(0, Ordering::Relaxed); + ALLOCED.load(Ordering::Relaxed) +} + +fn splitmix(s: &mut u64) -> u64 { + *s = s.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = *s; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) +} + +// ── the field: 256×256 axial, Morton-keyed ─────────────────────────────────── +const SIDE: usize = 256; +const N: usize = SIDE * SIDE; // 65,536 rows +const WORDS: usize = N / 64; // 1,024 mask words = 8 KiB +const DIRS: usize = 6; +const X_BITS: u32 = 0x5555; // q lives in the even bit positions +const Y_BITS: u32 = 0xAAAA; // r lives in the odd bit positions + +/// Axial hex neighbours: (dq, dr) for rails 0..6. +const HEX: [(i32, i32); DIRS] = [(1, 0), (-1, 0), (0, 1), (0, -1), (1, -1), (-1, 1)]; + +fn dilate8(v: u32) -> u32 { + let mut x = v & 0xFF; + x = (x | (x << 4)) & 0x0F0F; + x = (x | (x << 2)) & 0x3333; + (x | (x << 1)) & 0x5555 +} +fn morton(q: u32, r: u32) -> u32 { + dilate8(q) | (dilate8(r) << 1) +} +fn compact8(v: u32) -> u32 { + let mut x = v & 0x5555; + x = (x | (x >> 1)) & 0x3333; + x = (x | (x >> 2)) & 0x0F0F; + (x | (x >> 4)) & 0x00FF +} +fn axial(m: u32) -> (u32, u32) { + (compact8(m), compact8(m >> 1)) +} + +/// Neighbour of Morton cell `m` along rail `d`, by dilated-integer arithmetic — +/// no table, no deinterleave on the hot path. `None` at the field boundary. +#[inline] +fn neighbour(m: u32, d: usize) -> Option { + let (dq, dr) = HEX[d]; + let mut x = m & X_BITS; + let mut y = m & Y_BITS; + match dq { + 1 => { + if x == X_BITS { + return None; + } + x = ((x | Y_BITS) + 1) & X_BITS; + } + -1 => { + if x == 0 { + return None; + } + x = (x - 1) & X_BITS; + } + _ => {} + } + match dr { + 1 => { + if y == Y_BITS { + return None; + } + y = ((y | X_BITS) + 1) & Y_BITS; + } + -1 => { + if y == 0 { + return None; + } + y = (y - 1) & Y_BITS; + } + _ => {} + } + Some(x | y) +} + +#[inline] +fn bit(words: &[u64], i: usize) -> bool { + (words[i >> 6] >> (i & 63)) & 1 == 1 +} +#[inline] +fn set(words: &mut [u64], i: usize) { + words[i >> 6] |= 1u64 << (i & 63); +} + +/// A trie node at nibble level `level` (0..=4) with prefix `p`: rows +/// `[p << shift, (p+1) << shift)`. The fixed spatial distribution makes the +/// reveal a range write, not a compare. +fn range_reveal(level: u32, p: u32, out: &mut [u64]) -> (usize, usize) { + let shift = 16 - 4 * level; + let lo = (p as usize) << shift; + let hi = ((p + 1) as usize) << shift; + for w in out.iter_mut() { + *w = 0; + } + // inclusive last word; the run may start and end mid-word + let (w0, w1) = (lo >> 6, (hi - 1) >> 6); + let head = u64::MAX << (lo & 63); + let tail = u64::MAX >> (63 - ((hi - 1) & 63)); + if w0 == w1 { + out[w0] = head & tail; + } else { + out[w0] = head; + for w in out[w0 + 1..w1].iter_mut() { + *w = u64::MAX; + } + out[w1] = tail; + } + (lo, hi) +} + +fn tcam_reveal(addr: &[u32], level: u32, p: u32, out: &mut [u64]) { + let shift = 16 - 4 * level; + let care = if shift == 32 { 0 } else { !((1u32 << shift) - 1) }; + ternary_match_u32_to_mask(addr, p << shift, care, out); +} + +/// One spread step on the Morton arm. Returns the number of rail firings. +/// `scratch` receives the shifted frontier; the chain then narrows in place. +/// The resident (white, replayable) inputs of one spread step: laid once per +/// mask generation, borrowed by every step — the M1b tile. +struct Resident<'a> { + elig: &'a [Vec; DIRS], + tile: &'a [u64], + gates: &'a [Vec], + x: usize, + dirs: usize, + from_delta: bool, +} + +#[inline(never)] +fn spread_step( + state: &mut [u64], delta: &mut [u64], scratch: &mut [u64], res: &Resident<'_>, rails: &mut [[u8; 12]], +) -> usize { + let Resident { + elig, + tile, + gates, + x, + dirs, + from_delta, + } = *res; + for w in scratch.iter_mut() { + *w = 0; + } + // the ONLY non-mask op on the path: the hex shift, per active bit. This is + // the `n` term — and the primitive the substrate does not have yet. + // NNUE reading: spread from the DELTA frontier (cells reached last step), + // never from the whole accumulated state — same closure, `n ∝ |delta|`. + let source: &[u64] = if from_delta { delta } else { state }; + for (wi, &w) in source.iter().enumerate() { + let mut bits = w; + while bits != 0 { + let i = (wi << 6) | bits.trailing_zeros() as usize; + bits &= bits - 1; + for (d, e) in elig.iter().enumerate().take(dirs) { + if bit(e, i) { + if let Some(j) = neighbour(i as u32, d) { + set(scratch, j as usize); + } + } + } + } + } + // scratch = (scratch | state) & tile, then the chain narrows scratch in + // place: x resident gate masks, one AND3 pass each. `state` stays the + // previous frontier until the survivors are known. + mask_ternlog_assign::(scratch, state, tile); + for k in 0..x { + let g1 = &gates[k % gates.len()]; + let g2 = &gates[(k + 1) % gates.len()]; + mask_ternlog_assign::(scratch, g1, g2); + } + // Hebbian, on SURVIVORS only: a rail fires when the cell it points at + // actually became active after the chain — reverse walk from each newly + // reached cell to the eligible neighbour that could have carried it. + let mut fired = 0usize; + for wi in 0..scratch.len() { + let new_bits_word = scratch[wi] & !state[wi]; + delta[wi] = new_bits_word; + let mut new_bits = new_bits_word; + while new_bits != 0 { + let j = (wi << 6) | new_bits.trailing_zeros() as usize; + new_bits &= new_bits - 1; + for d in 0..dirs { + // the rail that reaches j from i along d is d; i is j's neighbour along the reverse rail + let rev = d ^ 1; // HEX pairs (0,1) (2,3) (4,5) are opposites + if let Some(i) = neighbour(j as u32, rev) { + let i = i as usize; + if bit(state, i) && bit(&elig[d], i) { + rails[i][2 * d + 1] = rails[i][2 * d + 1].saturating_add(1); + fired += 1; + } + } + } + } + } + state.copy_from_slice(scratch); + fired +} + +/// Independent reference: row-major axial BFS with the same permeability, +/// tile and gate semantics, no Morton, no masks. +fn reference_step( + state: &mut [bool], perm: &[[u8; 12]], thr: u8, tile: &[bool], gates: &[Vec], x: usize, dirs: usize, +) { + let mut next = state.to_vec(); + for q in 0..SIDE as i32 { + for r in 0..SIDE as i32 { + let i = (q as usize) * SIDE + r as usize; + if !state[i] { + continue; + } + let m = morton(q as u32, r as u32) as usize; + for (d, &(dq, dr)) in HEX.iter().enumerate().take(dirs) { + if perm[m][2 * d] <= thr { + continue; + } + let (nq, nr) = (q + dq, r + dr); + if nq < 0 || nr < 0 || nq >= SIDE as i32 || nr >= SIDE as i32 { + continue; + } + next[(nq as usize) * SIDE + nr as usize] = true; + } + } + } + for i in 0..N { + let mut v = next[i] && tile[i]; + for k in 0..x { + v = v && gates[k % gates.len()][i] && gates[(k + 1) % gates.len()][i]; + } + state[i] = v; + } +} + +fn masks_equal_axial(mask: &[u64], reference: &[bool]) -> bool { + (0..N).all(|i| { + let (q, r) = axial(i as u32); + bit(mask, i) == reference[(q as usize) * SIDE + r as usize] + }) +} + +fn timed(mut f: F) -> f64 { + // run to a 50 ms floor; report ns per call + let mut reps = 1usize; + loop { + let t = Instant::now(); + for _ in 0..reps { + f(); + } + let el = t.elapsed(); + if el.as_millis() >= 50 { + return el.as_nanos() as f64 / reps as f64; + } + reps *= 2; + } +} + +fn main() { + let mut seed = 0x5EED_0000_0000_0001u64; + println!("hex tenant probe — N={N} rows (256×256 axial, Morton-keyed), {WORDS} words/mask"); + + // ── the tenant: 12-byte register per row, 6×(perm:strength) ──────────── + let rails: Vec<[u8; 12]> = (0..N) + .map(|_| { + let mut r = [0u8; 12]; + for d in 0..DIRS { + r[2 * d] = (splitmix(&mut seed) & 0xFF) as u8; // permeability + r[2 * d + 1] = 0; // strength (plastic) + } + r + }) + .collect(); + let thr: u8 = 96; // ~62% of rails permeable + // Column views for the SIMD compare. The T1 compare is i32-wide today; a + // u8 column compare is a T1 addition (stated, not hidden). Widening costs + // 4× the bandwidth, so `n_gen` below is an UPPER bound on the reveal term. + let perm_cols: Vec> = (0..DIRS) + .map(|d| rails.iter().map(|r| r[2 * d] as i32).collect()) + .collect(); + let addr: Vec = (0..N as u32).collect(); + + // ── M1b: eligibility masks laid ONCE per mask generation ───────────────── + let mut elig: [Vec; DIRS] = std::array::from_fn(|_| vec![0u64; WORDS]); + let t_gen = timed(|| { + for d in 0..DIRS { + gt_i32_to_mask(&perm_cols[d], thr as i32, &mut elig[d]); + } + }); + println!( + "M1b generation: 6 eligibility masks from 6 columns = {:.0} ns ({:.0} ns/mask, {:.2} ns/row)", + t_gen, + t_gen / DIRS as f64, + t_gen / (DIRS * N) as f64 + ); + + // ── white: top-down reveal, range vs TCAM, every level ─────────────────── + println!("\n[white] top-down reveal — fixed-distribution RANGE vs TCAM compare sweep"); + println!("level rows/node range ns tcam ns ratio gate"); + let mut m_range = vec![0u64; WORDS]; + let mut m_tcam = vec![0u64; WORDS]; + for level in 0..=4u32 { + let n_nodes = 1u32 << (4 * level); + let mut ok = true; + for p in [0u32, n_nodes / 2, n_nodes - 1] { + range_reveal(level, p, &mut m_range); + tcam_reveal(&addr, level, p, &mut m_tcam); + ok &= m_range == m_tcam && popcount_batch_u64(&m_range) as usize == (1usize << (16 - 4 * level)); + } + assert!(ok, "reveal gate FAILED at level {level}"); + let p = n_nodes / 2; + let tr = timed(|| { + range_reveal(level, p, &mut m_range); + }); + let tt = timed(|| { + tcam_reveal(&addr, level, p, &mut m_tcam); + }); + println!("{level:>5} {:>9} {tr:>8.0} {tt:>8.0} {:>6.1}× ok", 1usize << (16 - 4 * level), tt / tr); + } + + // ── grey: spread inside a level-1 tile from a level-2 seed ─────────────── + // tile = level-1 node p=5 (4096 cells); seed = its first level-2 child. + let mut tile = vec![0u64; WORDS]; + let (tlo, thi) = range_reveal(1, 5, &mut tile); + let mut tile_ref = vec![false; N]; + for i in tlo..thi { + let (q, r) = axial(i as u32); + tile_ref[(q as usize) * SIDE + r as usize] = true; + } + // resident gate masks: 4 random 90%-dense masks (the "rung/tenant" gates) + let n_gates = 4; + let gates: Vec> = (0..n_gates) + .map(|_| { + (0..WORDS) + .map(|_| { + let mut w = u64::MAX; + for _ in 0..6 { + w &= !(1u64 << (splitmix(&mut seed) & 63)); + } + w + }) + .collect() + }) + .collect(); + let gates_ref: Vec> = gates + .iter() + .map(|g| { + let mut v = vec![false; N]; + for i in 0..N { + let (q, r) = axial(i as u32); + v[(q as usize) * SIDE + r as usize] = bit(g, i); + } + v + }) + .collect(); + + let ones: Vec> = vec![vec![u64::MAX; WORDS]; n_gates]; + let ones_ref: Vec> = vec![vec![true; N]; n_gates]; + + let steps = 24usize; + println!("\n[grey] spread — Morton arm vs row-major axial BFS reference, {steps} steps, tile=4096 cells"); + println!("The LADDER arm uses identity gates so the survivor population is held fixed across x"); + println!("(otherwise the gates shrink the frontier and n moves with x — the fit measures nothing)."); + println!("The GATED arm uses the 4 real 90%-dense gates: what a rung/tenant chain does to the spread."); + println!("The NNUE arm spreads from the DELTA frontier only (nnue+g = with the real gates)."); + println!("arm dirs x ns/step fired/step heap B/step gate"); + let mut fits: Vec<(usize, f64)> = Vec::new(); + for (arm, gates, gates_ref, from_delta) in [ + ("ladder", &ones, &ones_ref, false), + ("gated", &gates, &gates_ref, false), + ("nnue", &ones, &ones_ref, true), + ("nnue+g", &gates, &gates_ref, true), + ] { + for dirs in [6usize, 1] { + for x in [0usize, 1, 2, 4, 8, 16, 32] { + if arm != "ladder" && !matches!(x, 0 | 4 | 32) { + continue; + } + // fresh state + reference, same seed node every time + let mut seed_state = vec![0u64; WORDS]; + range_reveal(2, 5 * 16, &mut seed_state); // first level-2 child of tile 5 + let mut reference = vec![false; N]; + for i in 0..N { + if bit(&seed_state, i) { + let (q, r) = axial(i as u32); + reference[(q as usize) * SIDE + r as usize] = true; + } + } + let res = Resident { + elig: &elig, + tile: &tile, + gates, + x, + dirs, + from_delta, + }; + let mut state = vec![0u64; WORDS]; + let mut delta = vec![0u64; WORDS]; + let mut scratch = vec![0u64; WORDS]; + let mut rails_run = rails.clone(); + let mut fired_total = 0usize; + let mut ok = true; + // repeat the whole 24-step run to a 50 ms floor; the reset is an + // 8 KiB copy and is inside the timed region (stated). + let mut reps = 1usize; + let (el, heap) = loop { + count_on(); + let t = Instant::now(); + for _ in 0..reps { + state.copy_from_slice(&seed_state); + delta.copy_from_slice(&seed_state); + rails_run.copy_from_slice(&rails); + fired_total = 0; + for _ in 0..steps { + fired_total += spread_step(&mut state, &mut delta, &mut scratch, &res, &mut rails_run); + } + } + let e = t.elapsed(); + let heap = count_off(); + if e.as_millis() >= 50 { + break (e.as_nanos() as f64 / (reps * steps) as f64, heap / reps); + } + reps *= 2; + }; + // the correctness gate runs OUTSIDE the timed/counted region + let mut state_chk = seed_state.clone(); + let mut delta_chk = seed_state.clone(); + let mut scratch_chk = vec![0u64; WORDS]; + let mut rails_chk = rails.clone(); + for _ in 0..steps { + spread_step(&mut state_chk, &mut delta_chk, &mut scratch_chk, &res, &mut rails_chk); + reference_step(&mut reference, &rails, thr, &tile_ref, gates_ref, x, dirs); + ok &= masks_equal_axial(&state_chk, &reference); + } + assert!(ok, "spread gate FAILED arm={arm} dirs={dirs} x={x}"); + assert_eq!(state, state_chk, "timed run diverged from checked run"); + assert_eq!(rails_run, rails_chk, "plasticity diverged between runs"); + if arm == "ladder" && dirs == 6 { + fits.push((x, el)); + } + println!( + "{arm:<7} {dirs:>4} {x:>2} {el:>8.0} {:>10.1} {heap:>11} ok (survivors {})", + fired_total as f64 / steps as f64, + popcount_batch_u64(&state) + ); + if arm == "ladder" && dirs == 6 && x == 0 { + // plasticity landed in the register, not beside it + let bumped = rails_run + .iter() + .zip(rails.iter()) + .filter(|(a, b)| a != b) + .count(); + println!(" plasticity: {bumped} rows' strength bytes changed in place, {fired_total} firings"); + } + } + } + } + + // ── the fit: step = x·t + n over the degree-6 ladder ───────────────────── + let k = fits.len() as f64; + let sx: f64 = fits.iter().map(|(x, _)| *x as f64).sum(); + let sy: f64 = fits.iter().map(|(_, y)| *y).sum(); + let sxx: f64 = fits.iter().map(|(x, _)| (*x as f64).powi(2)).sum(); + let sxy: f64 = fits.iter().map(|(x, y)| *x as f64 * y).sum(); + let t_tern = (k * sxy - sx * sy) / (k * sxx - sx * sx); + let n_term = (sy - t_tern * sx) / k; + let resid = fits + .iter() + .map(|(x, y)| (y - (t_tern * *x as f64 + n_term)).abs() / y) + .fold(0.0f64, f64::max); + println!("\n[fit] step = x·ternlogq + n ⇒ ternlogq = {t_tern:.1} ns/pass ({:.3} ns/word), n = {n_term:.0} ns, max rel residual {:.1}%", + t_tern / WORDS as f64, resid * 100.0); + + // ── M3: coal — one maneuver = re-chain a resident mask from its column ── + let mut m_new = vec![0u64; WORDS]; + let c = timed(|| gt_i32_to_mask(&perm_cols[0], thr as i32, &mut m_new)); + println!( + "[coal] one re-chain (gt_i32 sweep over one column) = {c:.0} ns = {:.1} ternlogq passes = {:.2} maintained steps at x=4", + c / t_tern, + c / (4.0 * t_tern + n_term) + ); + println!( + "[M2] speed change x→x±1 costs one ternlogq pass ({t_tern:.0} ns); x→x±k costs k passes — linear, no cliff" + ); +} From df7fe9c85565da4f5d81847a948fbdf57dd9330d Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Mon, 14 Sep 2026 14:00:09 +0000 Subject: [PATCH 2/9] Spec the Morton hex neighbour shift (D-GTM-1m, plan section 15) The word-level op that removes the n term section 14 measured: a fixed bit permutation inside each 8x8 block word plus one carry byte into one neighbour word, four axis directions, hex diagonals composed. Six falsifiers pre-registered. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .../gemm-ternlog-mask-consolidation-v1.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md index 6f34abef..76794cf4 100644 --- a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md +++ b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md @@ -911,3 +911,86 @@ Probe `examples/hex_tenant_mq_probe.rs`; full numbers in `.claude/blackboard.md` - **0k holds on this shape too:** 0 heap bytes/step on every arm. - Still open: the u8/u16 compare-to-mask (T1), the shift primitive (T1), a second density and tile size, and `perf` counters for the residency claim. + +## §15 — SPEC: `mask_shift_morton` — the hex neighbour shift as a word-level op (D-GTM-1m, 2026-09-14) + +§14 measured that `n` — the whole cost of a spread step — is the per-active-bit +neighbour shift. This section specifies the T1 word that removes it. It is a +spec for a Sonnet implementer; every claim below is either a bit-arithmetic +fact (checkable by the test in F1) or marked as the implementer's finding. + +**Address model.** Rows are Morton-keyed 2-D: `q` occupies the EVEN address +bits, `r` the ODD bits (`dilate8(q) | dilate8(r) << 1`; `hex_tenant_mq_probe.rs` +`morton`). A mask word `w` covers rows `[64w, 64w+64)`: the low 6 address bits = +3 q-bits + 3 r-bits = an **8×8 axial block**; the word index `w` is itself the +Morton key of the block coordinates `(Q, R) = (q >> 3, r >> 3)`. Within a word, +bit index `b ∈ 0..64` has `ql = bits {0,2,4} of b`, `rl = bits {1,3,5} of b`. + +**Four axis shifts, each = a fixed bit permutation inside the word + ONE carry +byte into ONE neighbour word.** For `+q` (increment the dilated `ql`): + +| cells (by `b`) | move | why | +|---|---|---| +| `b & 1 == 0` (ql bit0 = 0) | `b + 1` | `..0 → ..1` | +| `b & 5 == 1` (ql = ?01) | `b + 3` | `..01 → ..10`: −1 + 4 | +| `b & 21 == 5` (ql = 011) | `b + 11` | `011 → 100`: −1 − 4 + 16 | +| `b & 21 == 21` (ql = 111) | carry to word `inc_q(w)`, local `b − 21` | ql wraps to 000 | + +So per word: `dst[w] |= ((src[w] & MA) << 1) | ((src[w] & MB) << 3) | ((src[w] & MC) << 11)` +and `dst[inc_q(w)] |= (src[w] & MD) >> 21` when `Q < Q_max`. `MA..MD` are +`const` u64s **computed by a `const fn` over `b in 0..64`** from the three +predicates above — never typed as literals — and F1 asserts them against the +per-bit definition. `−q` is the mirror (`b & 1 == 1 → b − 1`; `b & 5 == 4 → b − 3`; +`b & 21 == 16 → b − 11`; `b & 21 == 0` carries to `dec_q(w)` at `b + 21`, when +`Q > 0`). `±r` are the same tables on the odd bit set (shifts 2 / 6 / 22, carry +42). `inc_q(w)` / `dec_q(w)` on the WORD index are the same dilated-integer +inc/dec the probe uses on cells (`neighbour`), applied to `w`'s own even/odd +bits — the number of address bits above the word is `log2(n_words)` and must be +even (a square field); a non-square field is REFUSED (assert), not approximated. + +**The six hex directions** are `+q`, `−q`, `+r`, `−r`, `+q−r`, `−q+r` +(`HEX` in the probe). The two diagonals are composed: `shift(+q)` into scratch, +then `shift(−r)` of scratch into `dst`. Two passes; do not derive a fused table +for the diagonals in this rung (it exists, it is not needed to remove `n`). + +**Facade signature (T1, `simd_masking_ops.rs`):** + +```rust +/// dst |= src shifted one cell along `dir` on the Morton-keyed 2-D lattice. +/// `src.len() == dst.len() == n_words`, `n_words` a power of FOUR (square field). +/// Cells on the far edge produce nothing (no wrap). `dst` is OR-accumulated, +/// not overwritten — callers clear it or chain six directions into one plane. +pub fn mask_shift_morton(src: &[u64], dir: MortonDir, dst: &mut [u64]) +pub enum MortonDir { PosQ, NegQ, PosR, NegR } // hex diagonals = two calls +``` + +Body shape: pass 1 over `as_chunks::()` computing the interior +permutation with `U64x8` and/shl/or lane ops (the same facade-over-lane-types +shape `mask_xor` uses — no per-ISA code in the facade; if a backend lacks +`Shl`/`Shl` for `U64x8`, the implementer STOPS and reports which, +never hand-rolls an intrinsic); pass 2 a scalar loop over words applying the +carry byte to `inc/dec(w)`. Tail law: the field is square so there is no tail; +the assert makes that explicit. + +**Falsifiers (all required, all must be able to fail):** + +- **F1** — the four `const` masks equal a runtime fold over `b in 0..64` of the + stated predicates (a literal typo fails here). +- **F2** — equivalence: on random masks over a 256×256 field (1024 words) and a + 64×64 field (64 words), `mask_shift_morton` == the per-bit oracle + (`neighbour()` from the probe, transcribed into the test) for all four dirs; + the oracle is written in the TEST, not imported from the op. +- **F3** — can-fire: one set bit at a random interior cell lands on exactly ONE + bit, at the oracle's address; can-stay-silent: a bit on the far edge of `dir` + produces an all-zero `dst`. +- **F4** — composition: for interior cells, `shift(+q)` then `shift(−q)` == identity, + and the hex diagonal via two calls == the oracle's diagonal neighbour. +- **F5** — parity: added to `crates/simd-masking-parity` so every flavour + (avx512 / avx2 / neon / wasm / scalar / nightly) is bit-exact on the same + seeds — the shape of the existing arms; the arm count printed by the run + must rise by the number of arms added. +- **F6** — the probe: `hex_tenant_mq_probe.rs` gains a `shift` arm using six + `mask_shift_morton` calls (four axis + two composed) in place of the per-bit + loop; the spread gate (Morton == axial BFS) must stay green, and `n` is + re-measured. The claim this rung makes is only that `n` drops; the number is + the finding, not a prediction. From d11dd0f4bd6c37c654fc433be34f1818e2d8ed00 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Mon, 14 Sep 2026 14:22:05 +0000 Subject: [PATCH 3/9] Fill U64x8 Shl/Shr on the AVX2 and nightly-simd backends The mask_shift_morton implementer stopped at the pre-registered gate: four of six backends carried Shl/Shr for U64x8 and two (AVX2 = the default v3 target, nightly) did not. Filled backend-locally: AVX2 via the two 256-bit halves with VPSLLVQ/VPSRLVQ, nightly via core::simd's own operators. Per-lane falsifier against scalar << / >> over 256 rounds x 8 lanes, non-uniform counts, anti-vacuity on moved bits. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- src/simd_avx2.rs | 67 +++++++++++++++++++++++++++++++- src/simd_nightly/u_word_types.rs | 21 ++++++++++ 2 files changed, 87 insertions(+), 1 deletion(-) diff --git a/src/simd_avx2.rs b/src/simd_avx2.rs index f0cf5d36..1ffc8749 100644 --- a/src/simd_avx2.rs +++ b/src/simd_avx2.rs @@ -501,7 +501,7 @@ pub fn dgemm_blocked( use core::fmt; use core::ops::{ Add, AddAssign, BitAnd, BitAndAssign, BitOr, BitOrAssign, BitXor, BitXorAssign, Div, DivAssign, Mul, MulAssign, - Neg, Not, Sub, SubAssign, + Neg, Not, Shl, Shr, Sub, SubAssign, }; /// 16×f32 via 2× AVX2 F32x8 (__m256). Same API as simd_avx512::F32x16. @@ -1671,6 +1671,38 @@ impl U64x8 { } } +/// Lane-wise variable shifts for the mask family's word ops (the Morton hex +/// neighbour shift composes `and`/`shl`/`or` over `U64x8`). Same signature as +/// the AVX-512 / NEON / WASM / scalar backends: the count is a per-lane +/// vector, so a uniform shift is `x << U64x8::splat(n)`. Counts of 64 or more +/// are not a portable contract — this backend zeroes the lane (`VPSLLVQ` / +/// `VPSRLVQ` semantics) while the scalar backend's `<<` would overflow — so +/// callers keep every count below 64, as the mask ops do. +impl Shl for U64x8 { + type Output = Self; + #[inline(always)] + fn shl(self, rhs: Self) -> Self { + let (lo, hi) = self.avx2_halves(); + let (clo, chi) = rhs.avx2_halves(); + // SAFETY: same obligation as `avx2_halves` — this is the x86-64-v3 + // arm, AVX2 is present on any host that runs it; `_mm256_sllv_epi64` + // is an AVX2 instruction operating on the register values only. + unsafe { Self::from_avx2_halves(_mm256_sllv_epi64(lo, clo), _mm256_sllv_epi64(hi, chi)) } + } +} + +impl Shr for U64x8 { + type Output = Self; + #[inline(always)] + fn shr(self, rhs: Self) -> Self { + let (lo, hi) = self.avx2_halves(); + let (clo, chi) = rhs.avx2_halves(); + // SAFETY: as for `shl` — AVX2 present by the arm's contract; + // `_mm256_srlv_epi64` touches registers only. + unsafe { Self::from_avx2_halves(_mm256_srlv_epi64(lo, clo), _mm256_srlv_epi64(hi, chi)) } + } +} + /// BLAKE3 `hash_many` shuffle surface — the transpose network's four unpacks /// and two half-concatenations, at DEGREE 16. /// @@ -3117,6 +3149,39 @@ pub type i64x4 = I64x4; mod tests { use super::*; + /// Fails if either variable shift disagrees with the scalar `<<` / `>>` + /// on any lane for any count in 0..64 (a swapped intrinsic, a halves + /// mix-up, or a count applied to the wrong lane all show here). + #[test] + fn u64x8_variable_shifts_match_scalar_per_lane() { + let mut seed = 0x9E37_79B9_7F4A_7C15u64; + let mut next = || { + seed ^= seed << 13; + seed ^= seed >> 7; + seed ^= seed << 17; + seed + }; + let mut fired = 0usize; + for round in 0..256 { + let vals: [u64; 8] = core::array::from_fn(|_| next()); + // per-lane counts, deliberately NOT uniform, all below 64 + let cnts: [u64; 8] = core::array::from_fn(|i| (round as u64 * 7 + i as u64 * 11) % 64); + let v = U64x8::from_array(vals); + let c = U64x8::from_array(cnts); + let l = (v << c).to_array(); + let r = (v >> c).to_array(); + for i in 0..8 { + assert_eq!(l[i], vals[i] << cnts[i], "shl lane {i} count {}", cnts[i]); + assert_eq!(r[i], vals[i] >> cnts[i], "shr lane {i} count {}", cnts[i]); + if cnts[i] != 0 && (l[i] != vals[i] || r[i] != vals[i]) { + fired += 1; + } + } + } + // anti-vacuity: the shifts actually moved bits on most lanes + assert!(fired > 256 * 8 / 2, "shifts did nothing on {fired} lanes"); + } + #[test] fn test_dot_f32() { let a: Vec = (0..100).map(|i| i as f32).collect(); diff --git a/src/simd_nightly/u_word_types.rs b/src/simd_nightly/u_word_types.rs index 71e9bd33..29d3741a 100644 --- a/src/simd_nightly/u_word_types.rs +++ b/src/simd_nightly/u_word_types.rs @@ -165,6 +165,27 @@ impl U64x8 { } } +/// Lane-wise variable shifts — `core::simd`'s own `<<` / `>>`, so this +/// backend carries the same `Shl` / `Shr` surface as every other +/// arm (the mask family's Morton hex shift composes `and`/`shl`/`or`). Counts +/// stay below 64 by the callers' contract; what `core::simd` does at 64+ is +/// its own business, not a portable promise. +impl core::ops::Shl for U64x8 { + type Output = Self; + #[inline(always)] + fn shl(self, rhs: Self) -> Self { + Self(self.0 << rhs.0) + } +} + +impl core::ops::Shr for U64x8 { + type Output = Self; + #[inline(always)] + fn shr(self, rhs: Self) -> Self { + Self(self.0 >> rhs.0) + } +} + impl Default for U64x8 { #[inline(always)] fn default() -> Self { From 255c36db45ad2ef1d0b5f759a6787446c13c1778 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Mon, 14 Sep 2026 15:00:26 +0000 Subject: [PATCH 4/9] Add mask_shift_morton (D-GTM-1m): the hex neighbour shift as a word-level op Per plan section 15: four axis directions as a fixed bit permutation inside each 8x8-block word (const tables from a const fn over the predicates, F1) plus one carry byte into the dilated-inc/dec neighbour word; hex diagonals composed from two calls. Facade body over U64x8 lane ops only. F2 per-bit oracle equivalence on 256x256 and 64x64 fields, F3 can-fire/silent, F4 composition/identity, F5 parity arm (9 check groups, native/nightly/wasm/ wasm-scalar bit-identical; neon-qemu not runnable here), F6 shift arms in the hex-tenant probe with the spread gate green. Measured, not smoothed: at this fixture the op is only ~5 % faster than the per-bit loop (16.5 vs 17.5 us at dirs=6, x=0) and slower once real gates thin the frontier, because it walks the full 1024-word field per direction while the per-bit loop scales with the active count. The follow-up measurement (shift restricted to the trie node's own word range) is the next commit. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- crates/simd-masking-parity/src/lib.rs | 113 +++++- examples/hex_tenant_mq_probe.rs | 134 ++++++- src/simd.rs | 2 + src/simd_masking_ops.rs | 556 ++++++++++++++++++++++++++ 4 files changed, 789 insertions(+), 16 deletions(-) diff --git a/crates/simd-masking-parity/src/lib.rs b/crates/simd-masking-parity/src/lib.rs index da1c24aa..27ad6ccc 100644 --- a/crates/simd-masking-parity/src/lib.rs +++ b/crates/simd-masking-parity/src/lib.rs @@ -16,20 +16,21 @@ //! op and shape: `0x1IM` / `0x2IM` = ternlog table `IM` on `U32x16` / //! `U64x8`, `0x3xx` `U64x8` algebra, `0x4xx` `I32x16` compares, `0x5xx` //! predicate→mask, `0x6xx` mask algebra, `0x7xx` care-match, `0x8xx` masked -//! reductions and blend. `main.rs` (native / qemu) and `selfcheck()` (the -//! wasm cdylib export, driven by `run.mjs`) both call [`run`]. +//! reductions and blend, `0x9xx` `mask_shift_morton` (the Morton hex +//! neighbour shift, D-GTM-1m). `main.rs` (native / qemu) and `selfcheck()` +//! (the wasm cdylib export, driven by `run.mjs`) both call [`run`]. use ndarray::simd::{ blend_i32, eq_i32_to_mask, eq_u32_strided_to_mask, eq_u32_to_mask, ge_i32_to_mask, gt_i32_to_mask, le_i32_to_mask, lt_i32_to_mask, mask_all, mask_and, mask_and_assign, mask_andnot, mask_andnot_assign, mask_any, mask_not, - mask_not_assign, mask_or, mask_or_assign, mask_ternlog, mask_ternlog_assign, mask_xor, mask_xor_assign, - masked_max_i32, masked_min_i32, masked_strided_group_sum, masked_sum_i32, ne_i32_to_mask, ne_u32_to_mask, - ternary_match_strided_to_mask, ternary_match_u32_to_mask, ternary_match_u64_to_mask, ternlog, I32x16, U32x16, - U64x8, + mask_not_assign, mask_or, mask_or_assign, mask_shift_morton, mask_ternlog, mask_ternlog_assign, mask_xor, + mask_xor_assign, masked_max_i32, masked_min_i32, masked_strided_group_sum, masked_sum_i32, ne_i32_to_mask, + ne_u32_to_mask, ternary_match_strided_to_mask, ternary_match_u32_to_mask, ternary_match_u64_to_mask, ternlog, + I32x16, MortonDir, U32x16, U64x8, }; /// Number of check groups [`run`] executes (for the log line only). -pub const CHECKS: usize = 8; +pub const CHECKS: usize = 9; /// The wasm export: identical to [`run`], `extern "C"` so `run.mjs` can call it. #[no_mangle] @@ -41,7 +42,7 @@ pub extern "C" fn selfcheck() -> u32 { pub fn run() -> u32 { let groups: [fn() -> Result<(), u32>; CHECKS] = [ check_ternlog_all_tables, check_u64x8_algebra, check_i32x16_compare, check_predicates_to_mask, - check_mask_algebra, check_care_match, check_masked_reductions, check_blend, + check_mask_algebra, check_care_match, check_masked_reductions, check_blend, check_morton_shift, ]; for g in groups { if let Err(code) = g() { @@ -757,3 +758,99 @@ fn check_blend() -> Result<(), u32> { } Ok(()) } + +// ── 0x9xx: mask_shift_morton — the Morton hex neighbour shift (D-GTM-1m) ──── +// +// The reference below is an INDEPENDENT transcription of +// `hex_tenant_mq_probe.rs::neighbour`'s dilated-integer add/sub (never a +// call into `mask_shift_morton`), generalized from the probe's fixed +// 256×256/16-bit field to whatever field size this check picks. + +/// Even/odd address-bit split for a field of `n_cells` cells. +fn morton_axis_bits(n_cells: usize) -> (u32, u32) { + let total_bits = n_cells.trailing_zeros(); + let mut x = 0u32; + let mut b = 0u32; + while b < total_bits { + x |= 1u32 << b; + b += 2; + } + (x, x << 1) +} + +/// One dilated-integer neighbour step; `None` at the field edge (no wrap). +fn morton_neighbour(m: u32, dq: i32, dr: i32, x_bits: u32, y_bits: u32) -> Option { + let mut x = m & x_bits; + let mut y = m & y_bits; + match dq { + 1 => { + if x == x_bits { + return None; + } + x = (x | y_bits).wrapping_add(1) & x_bits; + } + -1 => { + if x == 0 { + return None; + } + x = x.wrapping_sub(1) & x_bits; + } + _ => {} + } + match dr { + 1 => { + if y == y_bits { + return None; + } + y = (y | x_bits).wrapping_add(1) & y_bits; + } + -1 => { + if y == 0 { + return None; + } + y = y.wrapping_sub(1) & y_bits; + } + _ => {} + } + Some(x | y) +} + +fn morton_reference_shift(src: &[u64], dq: i32, dr: i32, n_cells: usize, x_bits: u32, y_bits: u32) -> Vec { + let mut dst = vec![0u64; src.len()]; + for i in 0..n_cells { + if (src[i >> 6] >> (i & 63)) & 1 == 0 { + continue; + } + if let Some(j) = morton_neighbour(i as u32, dq, dr, x_bits, y_bits) { + let j = j as usize; + dst[j >> 6] |= 1u64 << (j & 63); + } + } + dst +} + +fn check_morton_shift() -> Result<(), u32> { + let mut rng = SplitMix64(0x9000_0000_0001); + let dirs = [ + (MortonDir::PosQ, 1i32, 0i32), + (MortonDir::NegQ, -1, 0), + (MortonDir::PosR, 0, 1), + (MortonDir::NegR, 0, -1), + ]; + for &n_words in &[1usize, 4, 64, 1024] { + let n_cells = n_words * 64; + let (x_bits, y_bits) = morton_axis_bits(n_cells); + for _ in 0..4 { + let src: Vec = (0..n_words).map(|_| rng.next()).collect(); + for (k, &(dir, dq, dr)) in dirs.iter().enumerate() { + let mut got = vec![0u64; n_words]; + mask_shift_morton(&src, dir, &mut got); + let want = morton_reference_shift(&src, dq, dr, n_cells, x_bits, y_bits); + if got != want { + return Err(0x900 | k as u32); + } + } + } + } + Ok(()) +} diff --git a/examples/hex_tenant_mq_probe.rs b/examples/hex_tenant_mq_probe.rs index 896e7af3..dab4a634 100644 --- a/examples/hex_tenant_mq_probe.rs +++ b/examples/hex_tenant_mq_probe.rs @@ -52,7 +52,10 @@ use std::sync::atomic::{AtomicUsize, Ordering}; use std::time::Instant; use ndarray::simd::ternlog::{AND3, OR2_AND}; -use ndarray::simd::{gt_i32_to_mask, mask_ternlog_assign, popcount_batch_u64, ternary_match_u32_to_mask}; +use ndarray::simd::{ + gt_i32_to_mask, mask_and, mask_shift_morton, mask_ternlog_assign, popcount_batch_u64, ternary_match_u32_to_mask, + MortonDir, +}; // ── counting allocator: the 0k instrument ──────────────────────────────────── static ALLOCED: AtomicUsize = AtomicUsize::new(0); @@ -292,6 +295,95 @@ fn spread_step( fired } +/// The same step as [`spread_step`], but with the per-active-bit hex +/// neighbour loop — the `n` term §14/§15 name as the whole cost of a spread +/// step — replaced by [`mask_shift_morton`] over whole words. For each axis +/// rail `d`, the eligibility mask is applied to the SOURCE first (`elig[d]` +/// bit `i` means "cell `i` may leave along `d`", so `source & elig[d]` +/// selects the cells allowed to leave, THEN the shift moves them), and the +/// shifted result is OR-accumulated into `scratch`. The two diagonal rails +/// (4 = `+q-r`, 5 = `-q+r`) are two composed calls through `diag_scratch` +/// (§15: shift the masked source along the FIRST axis into `diag_scratch`, +/// then shift `diag_scratch` — never the masked source again — along the +/// SECOND axis into `scratch`). Everything from the tile/gate chain onward, +/// including the Hebbian reverse walk, is copied verbatim from +/// [`spread_step`] — this function changes only how `scratch` is built. +#[inline(never)] +fn spread_step_shift( + state: &mut [u64], delta: &mut [u64], scratch: &mut [u64], res: &Resident<'_>, rails: &mut [[u8; 12]], + masked: &mut [u64], diag_scratch: &mut [u64], +) -> usize { + let Resident { + elig, + tile, + gates, + x, + dirs, + from_delta, + } = *res; + for w in scratch.iter_mut() { + *w = 0; + } + let source: &[u64] = if from_delta { delta } else { state }; + + const AXES: [MortonDir; 4] = [MortonDir::PosQ, MortonDir::NegQ, MortonDir::PosR, MortonDir::NegR]; + for (d, &dir) in AXES.iter().enumerate().take(dirs.min(4)) { + mask_and(source, &elig[d], masked); + mask_shift_morton(masked, dir, scratch); + } + if dirs > 4 { + // rail 4 = +q-r + mask_and(source, &elig[4], masked); + for w in diag_scratch.iter_mut() { + *w = 0; + } + mask_shift_morton(masked, MortonDir::PosQ, diag_scratch); + mask_shift_morton(diag_scratch, MortonDir::NegR, scratch); + } + if dirs > 5 { + // rail 5 = -q+r (the mirror composition) + mask_and(source, &elig[5], masked); + for w in diag_scratch.iter_mut() { + *w = 0; + } + mask_shift_morton(masked, MortonDir::NegQ, diag_scratch); + mask_shift_morton(diag_scratch, MortonDir::PosR, scratch); + } + + // scratch = (scratch | state) & tile, then the chain narrows scratch in + // place: x resident gate masks, one AND3 pass each — verbatim from + // `spread_step`. + mask_ternlog_assign::(scratch, state, tile); + for k in 0..x { + let g1 = &gates[k % gates.len()]; + let g2 = &gates[(k + 1) % gates.len()]; + mask_ternlog_assign::(scratch, g1, g2); + } + // Hebbian reverse walk — stays as is, verbatim from `spread_step`. + let mut fired = 0usize; + for wi in 0..scratch.len() { + let new_bits_word = scratch[wi] & !state[wi]; + delta[wi] = new_bits_word; + let mut new_bits = new_bits_word; + while new_bits != 0 { + let j = (wi << 6) | new_bits.trailing_zeros() as usize; + new_bits &= new_bits - 1; + for d in 0..dirs { + let rev = d ^ 1; + if let Some(i) = neighbour(j as u32, rev) { + let i = i as usize; + if bit(state, i) && bit(&elig[d], i) { + rails[i][2 * d + 1] = rails[i][2 * d + 1].saturating_add(1); + fired += 1; + } + } + } + } + } + state.copy_from_slice(scratch); + fired +} + /// Independent reference: row-major axial BFS with the same permeability, /// tile and gate semantics, no Morton, no masks. fn reference_step( @@ -456,13 +548,17 @@ fn main() { println!("(otherwise the gates shrink the frontier and n moves with x — the fit measures nothing)."); println!("The GATED arm uses the 4 real 90%-dense gates: what a rung/tenant chain does to the spread."); println!("The NNUE arm spreads from the DELTA frontier only (nnue+g = with the real gates)."); + println!("The SHIFT arm replaces the per-bit hex loop with mask_shift_morton over whole words —"); + println!("the D-GTM-1m primitive; same identity/real-gate split as ladder/gated (shift+g)."); println!("arm dirs x ns/step fired/step heap B/step gate"); let mut fits: Vec<(usize, f64)> = Vec::new(); - for (arm, gates, gates_ref, from_delta) in [ - ("ladder", &ones, &ones_ref, false), - ("gated", &gates, &gates_ref, false), - ("nnue", &ones, &ones_ref, true), - ("nnue+g", &gates, &gates_ref, true), + for (arm, gates, gates_ref, from_delta, use_shift) in [ + ("ladder", &ones, &ones_ref, false, false), + ("gated", &gates, &gates_ref, false, false), + ("nnue", &ones, &ones_ref, true, false), + ("nnue+g", &gates, &gates_ref, true, false), + ("shift", &ones, &ones_ref, false, true), + ("shift+g", &gates, &gates_ref, false, true), ] { for dirs in [6usize, 1] { for x in [0usize, 1, 2, 4, 8, 16, 32] { @@ -490,6 +586,12 @@ fn main() { let mut state = vec![0u64; WORDS]; let mut delta = vec![0u64; WORDS]; let mut scratch = vec![0u64; WORDS]; + // only the `shift`/`shift+g` arms touch these; allocated + // unconditionally (outside the timed region, negligible) so + // the dispatch below stays a plain branch, not a second copy + // of the buffer setup per arm. + let mut masked = vec![0u64; WORDS]; + let mut diag_scratch = vec![0u64; WORDS]; let mut rails_run = rails.clone(); let mut fired_total = 0usize; let mut ok = true; @@ -505,7 +607,14 @@ fn main() { rails_run.copy_from_slice(&rails); fired_total = 0; for _ in 0..steps { - fired_total += spread_step(&mut state, &mut delta, &mut scratch, &res, &mut rails_run); + fired_total += if use_shift { + spread_step_shift( + &mut state, &mut delta, &mut scratch, &res, &mut rails_run, &mut masked, + &mut diag_scratch, + ) + } else { + spread_step(&mut state, &mut delta, &mut scratch, &res, &mut rails_run) + }; } } let e = t.elapsed(); @@ -519,9 +628,18 @@ fn main() { let mut state_chk = seed_state.clone(); let mut delta_chk = seed_state.clone(); let mut scratch_chk = vec![0u64; WORDS]; + let mut masked_chk = vec![0u64; WORDS]; + let mut diag_scratch_chk = vec![0u64; WORDS]; let mut rails_chk = rails.clone(); for _ in 0..steps { - spread_step(&mut state_chk, &mut delta_chk, &mut scratch_chk, &res, &mut rails_chk); + if use_shift { + spread_step_shift( + &mut state_chk, &mut delta_chk, &mut scratch_chk, &res, &mut rails_chk, &mut masked_chk, + &mut diag_scratch_chk, + ); + } else { + spread_step(&mut state_chk, &mut delta_chk, &mut scratch_chk, &res, &mut rails_chk); + } reference_step(&mut reference, &rails, thr, &tile_ref, gates_ref, x, dirs); ok &= masks_equal_axial(&state_chk, &reference); } diff --git a/src/simd.rs b/src/simd.rs index ec0099c9..1d563e7c 100644 --- a/src/simd.rs +++ b/src/simd.rs @@ -795,6 +795,7 @@ pub use crate::simd_masking_ops::{ mask_not_assign, mask_or, mask_or_assign, + mask_shift_morton, mask_ternlog, mask_ternlog_assign, mask_xor, @@ -808,6 +809,7 @@ pub use crate::simd_masking_ops::{ ternary_match_strided_to_mask, ternary_match_u32_to_mask, ternary_match_u64_to_mask, + MortonDir, }; // The popcount that closes the loop on the masks above: `mask_count` in ABI // terms. Already public at `ndarray::bitwise::popcount_batch_u64`; re-exported diff --git a/src/simd_masking_ops.rs b/src/simd_masking_ops.rs index 6e1183f3..0d3fa207 100644 --- a/src/simd_masking_ops.rs +++ b/src/simd_masking_ops.rs @@ -1595,6 +1595,269 @@ pub fn blend_i32(mask_words: &[u64], a: &[i32], b: &[i32], dst: &mut [i32]) { } } +// ──────────────────────────────────────────────────────────────────────── +// Morton hex neighbour shift — the D-GTM-1m word-level op (§15) +// ──────────────────────────────────────────────────────────────────────── +// +// Rows are Morton-keyed 2-D (`q` on the even address bits, `r` on the odd +// ones — `hex_tenant_mq_probe.rs::morton`). A mask word covers 64 rows, i.e. +// an 8×8 axial block; within a word, bit index `b` in `0..64` has its local +// `q` sub-coordinate at bits `{0,2,4}` and its local `r` sub-coordinate at +// bits `{1,3,5}`. `mask_shift_morton` moves every set bit one cell along one +// of the four axis directions, entirely as word-level ops: three fixed bit +// permutations cover the cells whose local sub-coordinate does not overflow +// the word, and one carry moves the cells that do into the adjacent word. +// +// `D-GTM-0m` (`.claude/blackboard.md`) measured that this per-active-bit +// shift is the WHOLE cost of a spread step once the mask chain itself is +// ~free; this op removes it by making the shift itself a handful of word +// passes instead of a loop over set bits. + +/// One axis direction, as a runtime value (never a const generic — a +/// consumer composing all four, or the two diagonal directions, picks the +/// direction at the call site, not at compile time). +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub enum MortonDir { + /// Increment the local `q` sub-coordinate (`HEX[0] = (+1, 0)`). + PosQ, + /// Decrement the local `q` sub-coordinate (`HEX[1] = (-1, 0)`). + NegQ, + /// Increment the local `r` sub-coordinate (`HEX[2] = (0, +1)`). + PosR, + /// Decrement the local `r` sub-coordinate (`HEX[3] = (0, -1)`). + NegR, +} + +/// Build a 64-bit mask of every `b` in `0..64` for which +/// `(b & test_mask) == test_pattern` — the case-selector predicates §15 +/// states literally as `b & M == P`. `const fn` so every `AxisTable` below is +/// computed once, at compile time, never as a runtime literal. +const fn morton_case_mask(test_mask: u32, test_pattern: u32) -> u64 { + let mut m = 0u64; + let mut b = 0u32; + while b < 64 { + if (b & test_mask) == test_pattern { + m |= 1u64 << b; + } + b += 1; + } + m +} + +/// One direction's word-level shift recipe: the three interior (non- +/// wrapping) case masks with their signed bit-shift amount (`> 0` is `<<`, +/// `< 0` is `>>`), and the fourth (wrapping) case that carries into the +/// neighbouring word. +struct AxisTable { + /// `(case mask, signed shift)` for the three cells whose local + /// sub-coordinate does not overflow the word. + interior: [(u64, i32); 3], + /// The cells whose local sub-coordinate wraps and must carry out. + carry_mask: u64, + /// Magnitude of the carry's local re-basing shift. + carry_shift: u32, + /// `true` for `NegQ`/`NegR`: the carry moves toward the DECREASING + /// word (`dec_word`) and re-bases with `<<`. `false` for `PosQ`/`PosR`: + /// toward the INCREASING word (`inc_word`), re-based with `>>`. + carry_decrements: bool, +} + +// The four tables, one per `MortonDir`, exactly as specified: `+r`/`-r` are +// `+q`/`-q`'s tables with every bit position and shift amount doubled (the +// local `r` sub-coordinate sits one bit to the left of `q`'s at every tier). + +const POS_Q: AxisTable = AxisTable { + interior: [(morton_case_mask(1, 0), 1), (morton_case_mask(5, 1), 3), (morton_case_mask(21, 5), 11)], + carry_mask: morton_case_mask(21, 21), + carry_shift: 21, + carry_decrements: false, +}; + +const NEG_Q: AxisTable = AxisTable { + interior: [(morton_case_mask(1, 1), -1), (morton_case_mask(5, 4), -3), (morton_case_mask(21, 16), -11)], + carry_mask: morton_case_mask(21, 0), + carry_shift: 21, + carry_decrements: true, +}; + +const POS_R: AxisTable = AxisTable { + interior: [(morton_case_mask(2, 0), 2), (morton_case_mask(10, 2), 6), (morton_case_mask(42, 10), 22)], + carry_mask: morton_case_mask(42, 42), + carry_shift: 42, + carry_decrements: false, +}; + +const NEG_R: AxisTable = AxisTable { + interior: [(morton_case_mask(2, 2), -2), (morton_case_mask(10, 8), -6), (morton_case_mask(42, 32), -22)], + carry_mask: morton_case_mask(42, 0), + carry_shift: 42, + carry_decrements: true, +}; + +/// `(self & mask) << shift` if `shift >= 0`, else `(self & mask) >> -shift` — +/// the one interior case, as a whole-register op. `Shl`/`Shr` on +/// `U64x8` take a per-lane vector count; a uniform shift is `splat(n)`, the +/// same shape every backend already carries (`d11dd0f` filled the two that +/// did not: AVX2 and nightly-simd). +#[inline(always)] +fn morton_case_shift(v: crate::simd::U64x8, mask: u64, shift: i32) -> crate::simd::U64x8 { + let masked = v & crate::simd::U64x8::splat(mask); + if shift >= 0 { + masked << crate::simd::U64x8::splat(shift as u64) + } else { + masked >> crate::simd::U64x8::splat((-shift) as u64) + } +} + +/// The word-space `(x_bits, y_bits)` pair for a field of `n_words` words: +/// the even/odd address-bit masks one tier UP from the cell-level `X_BITS`/ +/// `Y_BITS` in `hex_tenant_mq_probe.rs` — the word index is itself the +/// Morton key of the block coordinates, so the same dilated-integer +/// even/odd split applies, just over `log2(n_words)` bits instead of 16. +#[inline(always)] +fn word_axis_bits(n_words: usize) -> (u64, u64) { + let total_bits = n_words.trailing_zeros(); + let mut x = 0u64; + let mut b = 0u32; + while b < total_bits { + x |= 1u64 << b; + b += 2; + } + (x, x << 1) +} + +/// Dilated-integer increment of the `axis_bits` sub-coordinate of word `w`, +/// `other_bits`-coordinate held fixed. `None` at the far edge (`axis_bits` +/// already all-ones) — the same no-wrap contract as the cell-level +/// `neighbour()` in the probe, one tier up. +#[inline(always)] +fn inc_word(w: usize, axis_bits: u64, other_bits: u64) -> Option { + let w = w as u64; + let x = w & axis_bits; + if x == axis_bits { + return None; + } + let xn = (x | other_bits).wrapping_add(1) & axis_bits; + Some((xn | (w & other_bits)) as usize) +} + +/// Dilated-integer decrement, the mirror of [`inc_word`]. `None` at `x == 0`. +#[inline(always)] +fn dec_word(w: usize, axis_bits: u64, other_bits: u64) -> Option { + let w = w as u64; + let x = w & axis_bits; + if x == 0 { + return None; + } + let xn = x.wrapping_sub(1) & axis_bits; + Some((xn | (w & other_bits)) as usize) +} + +/// `dst |= src` shifted one cell along `dir` on the Morton-keyed 2-D +/// lattice (D-GTM-1m, `.claude/plans/gemm-ternlog-mask-consolidation-v1.md` +/// §15) — the mask-level replacement for the per-active-bit hex neighbour +/// shift `D-GTM-0m` measured as the whole cost of a spread step. +/// +/// `src.len()` and `dst.len()` must be equal and a power of **four**: the +/// word index doubles as the Morton key of the field's block coordinates, so +/// `log2(n_words)` (the address bits above one word's own 6) must split +/// evenly between the `q` and `r` halves — a non-square field is refused, +/// never approximated. Cells on the far edge of `dir` produce nothing (no +/// wrap-around). `dst` is **OR-accumulated, not overwritten** — clear it +/// first, or chain several directions (e.g. all six hex neighbours) into one +/// plane by calling this repeatedly with the same `dst`. A hex diagonal +/// (`+q-r` or `-q+r`) is two calls: shift into a scratch buffer along the +/// first axis, then shift that scratch (not `src`) along the second axis +/// into `dst`. +/// +/// # Panics +/// +/// Panics if `src.len() != dst.len()`, or if that length is not `4^k` for +/// some `k >= 0`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{mask_shift_morton, MortonDir}; +/// +/// // a 64-word field (an 8×8 grid of 8×8 blocks = 64×64 cells). +/// // cell 0 (q=0, r=0, bit 0 of word 0) shifted +q lands at cell (1, 0), +/// // which is bit 1 of the same word (case A: b&1==0 -> b+1). +/// let mut src = vec![0u64; 64]; +/// src[0] = 1; +/// let mut dst = vec![0u64; 64]; +/// mask_shift_morton(&src, MortonDir::PosQ, &mut dst); +/// assert_eq!(dst[0], 0b10); +/// ``` +#[inline] +pub fn mask_shift_morton(src: &[u64], dir: MortonDir, dst: &mut [u64]) { + assert_eq!(src.len(), dst.len(), "mask_shift_morton: src/dst length mismatch"); + let n_words = src.len(); + assert!( + n_words.is_power_of_two() && n_words.trailing_zeros().is_multiple_of(2), + "mask_shift_morton: n_words={n_words} is not a square Morton field (need n_words = 4^k, \ + so the word index carries an even q/r bit split)" + ); + + let table = match dir { + MortonDir::PosQ => &POS_Q, + MortonDir::NegQ => &NEG_Q, + MortonDir::PosR => &POS_R, + MortonDir::NegR => &NEG_R, + }; + + // Pass 1 — the interior permutation, vectorized: and/shl-or-shr/or over + // whole `U64x8` registers, OR-accumulated into `dst`. Same chunk/tail + // shape as `mask_xor_assign` (both `src` and `dst` tails are read, since + // this is an accumulate, not an overwrite). + const L: usize = crate::simd::U64x8::LANES; + let (cs, ts) = src.as_chunks::(); + let (cd, td) = dst.as_chunks_mut::(); + for (s, d) in cs.iter().zip(cd.iter_mut()) { + let vs = crate::simd::U64x8::from_array(*s); + let mut acc = crate::simd::U64x8::from_array(*d); + for &(mask, shift) in &table.interior { + acc |= morton_case_shift(vs, mask, shift); + } + *d = acc.to_array(); + } + if !ts.is_empty() { + let vs = crate::simd::U64x8::from_array(pad_tail(ts)); + let mut acc = crate::simd::U64x8::from_array(pad_tail(td)); + for &(mask, shift) in &table.interior { + acc |= morton_case_shift(vs, mask, shift); + } + td.copy_from_slice(&acc.to_array()[..td.len()]); + } + + // Pass 2 — the one-word carry, scalar (there is exactly one neighbour + // word per source word, so there is nothing here for a vector lane to + // parallelize over). + let (x_bits, y_bits) = word_axis_bits(n_words); + let (axis_bits, other_bits) = match dir { + MortonDir::PosQ | MortonDir::NegQ => (x_bits, y_bits), + MortonDir::PosR | MortonDir::NegR => (y_bits, x_bits), + }; + for (w, &s) in src.iter().enumerate() { + let carry_bits = s & table.carry_mask; + if carry_bits == 0 { + continue; + } + let neighbour = if table.carry_decrements { + dec_word(w, axis_bits, other_bits) + } else { + inc_word(w, axis_bits, other_bits) + }; + if let Some(nb) = neighbour { + dst[nb] |= if table.carry_decrements { + carry_bits << table.carry_shift + } else { + carry_bits >> table.carry_shift + }; + } + } +} + #[cfg(test)] mod tests { use super::*; @@ -2727,4 +2990,297 @@ mod tests { } } } + // ── D-GTM-1m: mask_shift_morton (§15) ─────────────────────────────────── + // + // The oracle below (`oracle_neighbour`) is a fresh transcription of + // `hex_tenant_mq_probe.rs::neighbour`'s dilated-integer add/sub — never a + // call into `mask_shift_morton` or its `AxisTable`s — generalized from + // the probe's fixed 16-bit field to whatever `n_cells` the test picks. + + /// Even/odd address-bit split for a field of `n_cells` cells (must be a + /// power of four): the cell-level analogue of `word_axis_bits`, kept + /// independent here rather than calling it. + fn oracle_axis_bits(n_cells: usize) -> (u32, u32) { + let total_bits = n_cells.trailing_zeros(); + let mut x = 0u32; + let mut b = 0u32; + while b < total_bits { + x |= 1u32 << b; + b += 2; + } + (x, x << 1) + } + + /// One dilated-integer neighbour step, `(dq, dr)` each in `{-1, 0, 1}` — + /// zero on an axis leaves it untouched, non-zero on both is a diagonal. + /// `None` at the field edge (no wrap), exactly `neighbour()`'s contract. + fn oracle_neighbour(m: u32, dq: i32, dr: i32, x_bits: u32, y_bits: u32) -> Option { + let mut x = m & x_bits; + let mut y = m & y_bits; + match dq { + 1 => { + if x == x_bits { + return None; + } + x = (x | y_bits).wrapping_add(1) & x_bits; + } + -1 => { + if x == 0 { + return None; + } + x = x.wrapping_sub(1) & x_bits; + } + _ => {} + } + match dr { + 1 => { + if y == y_bits { + return None; + } + y = (y | x_bits).wrapping_add(1) & y_bits; + } + -1 => { + if y == 0 { + return None; + } + y = y.wrapping_sub(1) & y_bits; + } + _ => {} + } + Some(x | y) + } + + fn dir_to_dq_dr(dir: MortonDir) -> (i32, i32) { + match dir { + MortonDir::PosQ => (1, 0), + MortonDir::NegQ => (-1, 0), + MortonDir::PosR => (0, 1), + MortonDir::NegR => (0, -1), + } + } + + /// Apply one (possibly diagonal) oracle step to every set bit of `src`, + /// bit-by-bit — the reference `mask_shift_morton` must match. + fn oracle_shift_field(src: &[u64], dq: i32, dr: i32, n_cells: usize, x_bits: u32, y_bits: u32) -> Vec { + let mut dst = vec![0u64; src.len()]; + for i in 0..n_cells { + if (src[i >> 6] >> (i & 63)) & 1 == 0 { + continue; + } + if let Some(j) = oracle_neighbour(i as u32, dq, dr, x_bits, y_bits) { + let j = j as usize; + dst[j >> 6] |= 1u64 << (j & 63); + } + } + dst + } + + /// F1 — the four `AxisTable`s' masks (3 interior + 1 carry, per + /// direction) equal an INDEPENDENT runtime fold of the exact §15 + /// predicates, computed here via explicit local-coordinate + /// classification (never `b & mask == pattern` arithmetic, which is how + /// the op itself is built) — a literal typo in either implementation + /// shows up as a mismatch. + #[test] + fn morton_axis_tables_match_a_per_bit_predicate_fold() { + // the local 3-bit sub-coordinate at `bits` (`{0,2,4}` for q, `{1,3,5}` + // for r), MSB-first from `bits[2]`. + fn local(b: u32, bits: [u32; 3]) -> u32 { + (((b >> bits[2]) & 1) << 2) | (((b >> bits[1]) & 1) << 1) | ((b >> bits[0]) & 1) + } + fn classify_pos(l: u32) -> usize { + match l { + 0 | 2 | 4 | 6 => 0, // ..0 -> ..1 (+1 / +2) + 1 | 5 => 1, // ..01 -> ..10 (+3 / +6) + 3 => 2, // 011 -> 100 (+11 / +22) + 7 => 3, // 111 wraps (carry) + _ => unreachable!("a 3-bit local value is in 0..=7"), + } + } + fn classify_neg(l: u32) -> usize { + match l { + 1 | 3 | 5 | 7 => 0, // ..1 -> ..0 (-1 / -2) + 2 | 6 => 1, // ..10 -> ..01 (-3 / -6) + 4 => 2, // 100 -> 011 (-11 / -22) + 0 => 3, // 000 wraps (carry) + _ => unreachable!("a 3-bit local value is in 0..=7"), + } + } + fn fold(bits: [u32; 3], classify: fn(u32) -> usize) -> [u64; 4] { + let mut out = [0u64; 4]; + for b in 0u32..64 { + out[classify(local(b, bits))] |= 1u64 << b; + } + out + } + + let q_bits = [0, 2, 4]; + let r_bits = [1, 3, 5]; + let pos_q = fold(q_bits, classify_pos); + let neg_q = fold(q_bits, classify_neg); + let pos_r = fold(r_bits, classify_pos); + let neg_r = fold(r_bits, classify_neg); + + assert_eq!(POS_Q.interior.map(|(m, _)| m), [pos_q[0], pos_q[1], pos_q[2]], "POS_Q interior masks"); + assert_eq!(POS_Q.carry_mask, pos_q[3], "POS_Q carry mask"); + assert_eq!(NEG_Q.interior.map(|(m, _)| m), [neg_q[0], neg_q[1], neg_q[2]], "NEG_Q interior masks"); + assert_eq!(NEG_Q.carry_mask, neg_q[3], "NEG_Q carry mask"); + assert_eq!(POS_R.interior.map(|(m, _)| m), [pos_r[0], pos_r[1], pos_r[2]], "POS_R interior masks"); + assert_eq!(POS_R.carry_mask, pos_r[3], "POS_R carry mask"); + assert_eq!(NEG_R.interior.map(|(m, _)| m), [neg_r[0], neg_r[1], neg_r[2]], "NEG_R interior masks"); + assert_eq!(NEG_R.carry_mask, neg_r[3], "NEG_R carry mask"); + + // anti-vacuity: each direction's four cases partition all 64 bits + // exactly once — a typo that dropped or doubled a bit shows here + // even if it happened to leave the individual masks pairwise unequal + // to something else that was also wrong. + for masks in [pos_q, neg_q, pos_r, neg_r] { + assert_eq!(masks[0] | masks[1] | masks[2] | masks[3], u64::MAX, "cases must cover every bit"); + let total: u32 = masks.iter().map(|m| m.count_ones()).sum(); + assert_eq!(total, 64, "cases must be pairwise disjoint"); + } + + // the shift amounts and carry directions, against §15's literal table. + assert_eq!(POS_Q.interior.map(|(_, s)| s), [1, 3, 11]); + assert_eq!((POS_Q.carry_shift, POS_Q.carry_decrements), (21, false)); + assert_eq!(NEG_Q.interior.map(|(_, s)| s), [-1, -3, -11]); + assert_eq!((NEG_Q.carry_shift, NEG_Q.carry_decrements), (21, true)); + assert_eq!(POS_R.interior.map(|(_, s)| s), [2, 6, 22]); + assert_eq!((POS_R.carry_shift, POS_R.carry_decrements), (42, false)); + assert_eq!(NEG_R.interior.map(|(_, s)| s), [-2, -6, -22]); + assert_eq!((NEG_R.carry_shift, NEG_R.carry_decrements), (42, true)); + } + + /// F2 — `mask_shift_morton` matches `oracle_shift_field` bit-for-bit, over + /// random masks, at two field sizes (64 words = 64×64 cells, 1024 words = + /// 256×256 cells) and all four axis directions. + #[test] + fn morton_shift_matches_the_per_bit_oracle_on_random_fields() { + let mut seed = 0x1357_2468_1357_2468u64; + for &n_words in &[64usize, 1024] { + let n_cells = n_words * 64; + let (x_bits, y_bits) = oracle_axis_bits(n_cells); + for _ in 0..8 { + let src: Vec = (0..n_words).map(|_| splitmix64(&mut seed)).collect(); + for dir in [MortonDir::PosQ, MortonDir::NegQ, MortonDir::PosR, MortonDir::NegR] { + let mut got = vec![0u64; n_words]; + mask_shift_morton(&src, dir, &mut got); + let (dq, dr) = dir_to_dq_dr(dir); + let want = oracle_shift_field(&src, dq, dr, n_cells, x_bits, y_bits); + assert_eq!(got, want, "n_words={n_words} dir={dir:?}"); + } + } + } + } + + /// F3 — can-fire: a lone interior bit moves to exactly one bit, at the + /// oracle's address, for every direction. Can-stay-silent: a NON-trivial + /// far-edge mask (every cell on `dir`'s edge, spread across many words — + /// not an empty mask) produces an all-zero `dst`. + #[test] + fn morton_shift_can_fire_and_can_stay_silent() { + let n_words = 64usize; + let n_cells = n_words * 64; + let (x_bits, y_bits) = oracle_axis_bits(n_cells); + + let mut seed = 0xABCD_EF01_2345_6789u64; + for dir in [MortonDir::PosQ, MortonDir::NegQ, MortonDir::PosR, MortonDir::NegR] { + let (dq, dr) = dir_to_dq_dr(dir); + for _ in 0..16 { + // an interior cell: strictly inside both axes, so it is never + // the far edge for any direction under test. + let i = loop { + let x = (splitmix64(&mut seed) as u32) & x_bits; + let y = (splitmix64(&mut seed) as u32) & y_bits; + if x != 0 && x != x_bits && y != 0 && y != y_bits { + break (x | y) as usize; + } + }; + let mut src = vec![0u64; n_words]; + src[i >> 6] |= 1u64 << (i & 63); + let mut dst = vec![0u64; n_words]; + mask_shift_morton(&src, dir, &mut dst); + let fired: Vec = (0..n_cells) + .filter(|&j| (dst[j >> 6] >> (j & 63)) & 1 == 1) + .collect(); + assert_eq!(fired.len(), 1, "exactly one bit must move, dir={dir:?} i={i}"); + let want = oracle_shift_field(&src, dq, dr, n_cells, x_bits, y_bits); + assert_eq!(dst, want, "the moved bit must land at the oracle's address, dir={dir:?}"); + } + } + + for dir in [MortonDir::PosQ, MortonDir::NegQ, MortonDir::PosR, MortonDir::NegR] { + let mut src = vec![0u64; n_words]; + let mut planted = 0usize; + for i in 0..n_cells { + let m = i as u32; + let (x, y) = (m & x_bits, m & y_bits); + let at_edge = match dir { + MortonDir::PosQ => x == x_bits, + MortonDir::NegQ => x == 0, + MortonDir::PosR => y == y_bits, + MortonDir::NegR => y == 0, + }; + if at_edge { + src[i >> 6] |= 1u64 << (i & 63); + planted += 1; + } + } + // non-trivial and spread: strictly more than "a handful", and + // touching more than one word (never a single-word fixture that + // could pass by accident of layout). + assert!(planted >= 16, "far-edge fixture must be non-trivial, dir={dir:?} planted={planted}"); + let touched_words = src.iter().filter(|&&w| w != 0).count(); + assert!(touched_words > 1, "far-edge fixture must spread across words, dir={dir:?}"); + + let mut dst = vec![0u64; n_words]; + mask_shift_morton(&src, dir, &mut dst); + assert!(dst.iter().all(|&w| w == 0), "far-edge bits must produce silence, dir={dir:?}"); + } + } + + /// F4 — composition: `+q` then `-q` is the identity on interior cells, + /// and a hex diagonal via two calls matches the oracle's one-step + /// diagonal, for both `+q-r` and its mirror `-q+r`. + #[test] + fn morton_shift_composes_to_identity_and_matches_the_hex_diagonal() { + let n_words = 64usize; + let n_cells = n_words * 64; + let (x_bits, y_bits) = oracle_axis_bits(n_cells); + let mut seed = 0x0F0F_1E1E_2D2D_3C3Cu64; + + // +q then -q == identity, restricted to cells that cannot fall off + // the +q edge (so the round trip is well posed for every one). + let raw: Vec = (0..n_words).map(|_| splitmix64(&mut seed)).collect(); + let mut interior = raw.clone(); + for i in 0..n_cells { + if (i as u32) & x_bits == x_bits { + interior[i >> 6] &= !(1u64 << (i & 63)); + } + } + assert!(interior.iter().any(|&w| w != 0), "the interior fixture must be non-empty"); + let mut forward = vec![0u64; n_words]; + mask_shift_morton(&interior, MortonDir::PosQ, &mut forward); + let mut back = vec![0u64; n_words]; + mask_shift_morton(&forward, MortonDir::NegQ, &mut back); + assert_eq!(back, interior, "+q then -q must be the identity on interior cells"); + + // the hex diagonal: two composed calls must match the oracle's own + // single-step diagonal (`dq` and `dr` both non-zero at once). + let diag_src: Vec = (0..n_words).map(|_| splitmix64(&mut seed)).collect(); + + let mut scratch = vec![0u64; n_words]; + mask_shift_morton(&diag_src, MortonDir::PosQ, &mut scratch); + let mut dst = vec![0u64; n_words]; + mask_shift_morton(&scratch, MortonDir::NegR, &mut dst); + let want = oracle_shift_field(&diag_src, 1, -1, n_cells, x_bits, y_bits); + assert_eq!(dst, want, "+q-r composed via two calls must match the oracle diagonal"); + + let mut scratch2 = vec![0u64; n_words]; + mask_shift_morton(&diag_src, MortonDir::NegQ, &mut scratch2); + let mut dst2 = vec![0u64; n_words]; + mask_shift_morton(&scratch2, MortonDir::PosR, &mut dst2); + let want2 = oracle_shift_field(&diag_src, -1, 1, n_cells, x_bits, y_bits); + assert_eq!(dst2, want2, "-q+r composed via two calls must match the oracle diagonal"); + } } From 855bc734a7902ea2d8b166e57deb8f317895a8c8 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Mon, 14 Sep 2026 15:05:50 +0000 Subject: [PATCH 5/9] =?UTF-8?q?Hex-tenant=20probe:=20node-span=20shift=20a?= =?UTF-8?q?rms=20=E2=80=94=20n=20=3D=2017.0=20->=205.7=20us=20(-66=20%)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runs mask_shift_morton over the trie node's own contiguous word span (a square Morton sub-field) instead of the full field; correctness unchanged (source subset of tile, so no carry enters the span and leaving carries are removed by the tile AND). Four new arms (node, node+g, node+nn, node+nn+g), 68/68 gate rows green, 0 heap B/step. Delta-frontier on top of the span buys nothing, as a span-bound op predicts; the degree-1 control keeps the ratio. Blackboard entry 2026-09-14 (3). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/blackboard.md | 38 ++++++++++++++++++++++ examples/hex_tenant_mq_probe.rs | 56 ++++++++++++++++++++++++--------- 2 files changed, 79 insertions(+), 15 deletions(-) diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 044e1396..3c6e5cef 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -1,3 +1,41 @@ +## 2026-09-14 (3) — D-GTM-1m MEASURED: `mask_shift_morton` lands; the win is in the NODE SPAN, not the op — n = 17.0 → 5.7 µs (−66 %) + +`mask_shift_morton` (255c36d) is bit-exact (F1–F4, parity 9/9 on native/ +nightly/wasm/wasm-scalar; neon-qemu absent here) — and over the FULL field it +barely moves `n`: 14.5–16.5 µs vs 17.0 µs for the per-bit loop. The Sonnet +implementer named the mechanism correctly: the op is FIELD-size-bound (1024 +words × ~8 passes per direction) while the per-bit loop is ACTIVE-bound, so at +1.8 % density they cost about the same and the op loses once gates thin the +frontier. + +**The fix is the fixed-spatial-distribution dividend a second time.** A trie +node is a contiguous word span AND a square Morton sub-field (the level-1 tile = +64 words = a 64×64 field, `log2(64)` even), so the shifts run over the node's +own span with no correctness change (the source ⊆ tile, so no carry enters the +span; a carry leaving it is what `& tile` removes anyway). Measured, dirs = 6, +x = 0, same gates green (68/68 rows, 0 heap B/step): + +| arm | ns/step | vs ladder | +|---|---|---| +| ladder (per-bit, full state) | 16,944 | — | +| nnue (per-bit, delta frontier) | 9,296 | −45 % | +| shift (word op, full field) | 14,517 | −14 % | +| **node (word op, tile span)** | **5,677** | **−66 %** | +| node+g / node+nn / node+nn+g | 5,722 / 6,044 / 5,774 | — | + +Delta-frontier on top of the node span buys nothing (5.7 → 6.0), as predicted: +a span-bound op does not care how many bits are set. Degree-1 control: node +4.5 µs vs ladder 6.7 µs — the ratio survives the E-Q8 ablation, so the gain is +the word op, not the six. The ternlog fit is unchanged (281 ns/pass, n = 17.0 µs +on the ladder arm by construction). Remaining 5.7 µs is the field-wide +housekeeping (scratch zeroing, `OR2_AND`, the Hebbian reverse walk over 1024 +words) — the next rung restricts THOSE to the node span too and re-measures; +not claimed here. + +Rule extracted: **a word-level op pays for the span it is given; give it the +node, never the field.** The same statement as "top-down is a range, not a +compare" (2026-09-14 (2)), now on the grey side. + ## 2026-09-14 (2) — D-GTM-0m: the hex TENANT — top-down traversal AND spread on ONE Morton-keyed SoA; `step = x·ternlogq + n` measured, and the chain is 1.7 % of it Probe: `examples/hex_tenant_mq_probe.rs` (`--release`, committed; output banked diff --git a/examples/hex_tenant_mq_probe.rs b/examples/hex_tenant_mq_probe.rs index dab4a634..0c44b721 100644 --- a/examples/hex_tenant_mq_probe.rs +++ b/examples/hex_tenant_mq_probe.rs @@ -222,6 +222,10 @@ struct Resident<'a> { x: usize, dirs: usize, from_delta: bool, + /// `Some((lo, hi))` = run the word-level shifts over ONLY this word range + /// (the trie node's own contiguous span — a square Morton sub-field), so + /// the op's cost is bound by the NODE, not the field. `None` = full field. + word_range: Option<(usize, usize)>, } #[inline(never)] @@ -235,6 +239,7 @@ fn spread_step( x, dirs, from_delta, + word_range: _, } = *res; for w in scratch.iter_mut() { *w = 0; @@ -320,34 +325,44 @@ fn spread_step_shift( x, dirs, from_delta, + word_range, } = *res; for w in scratch.iter_mut() { *w = 0; } let source: &[u64] = if from_delta { delta } else { state }; + // The word-level shifts run over the node's own span when one is given. + // Correctness is unchanged: the source is a subset of the tile, so no + // carry can arrive from outside the span, and a carry LEAVING the span + // would be removed by the `& tile` below anyway. + let (lo, hi) = word_range.unwrap_or((0, scratch.len())); + let src = &source[lo..hi]; + let masked = &mut masked[lo..hi]; + let diag_scratch = &mut diag_scratch[lo..hi]; + let out = &mut scratch[lo..hi]; const AXES: [MortonDir; 4] = [MortonDir::PosQ, MortonDir::NegQ, MortonDir::PosR, MortonDir::NegR]; for (d, &dir) in AXES.iter().enumerate().take(dirs.min(4)) { - mask_and(source, &elig[d], masked); - mask_shift_morton(masked, dir, scratch); + mask_and(src, &elig[d][lo..hi], masked); + mask_shift_morton(masked, dir, out); } if dirs > 4 { // rail 4 = +q-r - mask_and(source, &elig[4], masked); + mask_and(src, &elig[4][lo..hi], masked); for w in diag_scratch.iter_mut() { *w = 0; } mask_shift_morton(masked, MortonDir::PosQ, diag_scratch); - mask_shift_morton(diag_scratch, MortonDir::NegR, scratch); + mask_shift_morton(diag_scratch, MortonDir::NegR, out); } if dirs > 5 { // rail 5 = -q+r (the mirror composition) - mask_and(source, &elig[5], masked); + mask_and(src, &elig[5][lo..hi], masked); for w in diag_scratch.iter_mut() { *w = 0; } mask_shift_morton(masked, MortonDir::NegQ, diag_scratch); - mask_shift_morton(diag_scratch, MortonDir::PosR, scratch); + mask_shift_morton(diag_scratch, MortonDir::PosR, out); } // scratch = (scratch | state) & tile, then the chain narrows scratch in @@ -548,17 +563,27 @@ fn main() { println!("(otherwise the gates shrink the frontier and n moves with x — the fit measures nothing)."); println!("The GATED arm uses the 4 real 90%-dense gates: what a rung/tenant chain does to the spread."); println!("The NNUE arm spreads from the DELTA frontier only (nnue+g = with the real gates)."); + println!( + "The SHIFT arms use mask_shift_morton over the FULL field; the NODE arms over the tile's own 64-word span." + ); println!("The SHIFT arm replaces the per-bit hex loop with mask_shift_morton over whole words —"); println!("the D-GTM-1m primitive; same identity/real-gate split as ladder/gated (shift+g)."); - println!("arm dirs x ns/step fired/step heap B/step gate"); + println!("arm dirs x ns/step fired/step heap B/step gate"); let mut fits: Vec<(usize, f64)> = Vec::new(); - for (arm, gates, gates_ref, from_delta, use_shift) in [ - ("ladder", &ones, &ones_ref, false, false), - ("gated", &gates, &gates_ref, false, false), - ("nnue", &ones, &ones_ref, true, false), - ("nnue+g", &gates, &gates_ref, true, false), - ("shift", &ones, &ones_ref, false, true), - ("shift+g", &gates, &gates_ref, false, true), + // the tile is a level-1 node = one 64x64 Morton sub-field = 64 contiguous + // words; `node` arms run the word-level shifts over exactly that span + let tile_words = (tlo >> 6, thi >> 6); + for (arm, gates, gates_ref, from_delta, use_shift, word_range) in [ + ("ladder", &ones, &ones_ref, false, false, None), + ("gated", &gates, &gates_ref, false, false, None), + ("nnue", &ones, &ones_ref, true, false, None), + ("nnue+g", &gates, &gates_ref, true, false, None), + ("shift", &ones, &ones_ref, false, true, None), + ("shift+g", &gates, &gates_ref, false, true, None), + ("node", &ones, &ones_ref, false, true, Some(tile_words)), + ("node+g", &gates, &gates_ref, false, true, Some(tile_words)), + ("node+nn", &ones, &ones_ref, true, true, Some(tile_words)), + ("node+nn+g", &gates, &gates_ref, true, true, Some(tile_words)), ] { for dirs in [6usize, 1] { for x in [0usize, 1, 2, 4, 8, 16, 32] { @@ -582,6 +607,7 @@ fn main() { x, dirs, from_delta, + word_range, }; let mut state = vec![0u64; WORDS]; let mut delta = vec![0u64; WORDS]; @@ -650,7 +676,7 @@ fn main() { fits.push((x, el)); } println!( - "{arm:<7} {dirs:>4} {x:>2} {el:>8.0} {:>10.1} {heap:>11} ok (survivors {})", + "{arm:<10} {dirs:>4} {x:>2} {el:>8.0} {:>10.1} {heap:>11} ok (survivors {})", fired_total as f64 / steps as f64, popcount_batch_u64(&state) ); From 454fb4369c70b3e03de47888d0457c9613898f3e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 16:18:28 +0000 Subject: [PATCH 6/9] =?UTF-8?q?simd=5Fmasking=5Fops:=20*=5Fto=5Fmask=5Fund?= =?UTF-8?q?er=20=E2=80=94=20the=20ten=20gated=20predicates=20on=20one=20su?= =?UTF-8?q?rvivor-skip=20engine?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit lance-graph #1225's kernel-membrane review named the executor's missing T1 word: mask-risc's `Pred { under }` promises "evaluate only where the gate has a survivor", and neither `*_to_mask` + `mask_and` (the whole plane, twice) nor an executor that skips words itself (a compute path above the facade) is that word. This is it. `pack_under::` is the one private engine: `out[w] = under[w] & pred(values)[w]`, evaluated only where `under[w] != 0`. Word granularity is the finest a packed compare can skip at; cost is proportional to the live gate words, never to the rows. The predicate's own tail law makes a phantom gate bit past `n` vanish without a clean — the AND conforms regardless, so a gate never needs cleaning before use. Full overwrite, surplus words zero. Ten public members, one per `Pred` variant, each ONE `pack_under` call with a `group_bits` closure over the same `I32x16` / `U32x16` / `U64x8` lane op its ungated sibling uses: gt/lt/ge/le/eq/ne_i32, eq/ne_u32, ternary_match_u32/u64. Re-exported through `ndarray::simd`. Tests: the family equals `ungated & gate` word-for-word at ten lengths with phantom gate bits present; the skip is MEASURED (a counting closure sees 20 of 40 groups under an alternating gate, 40 under all-ones); phantom gate bits do not leak at n = 70; a short gate panics. Doctests 10/10. Contract doc carries the family's note. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .../vertical-simd-consumer-contract.md | 14 + src/simd.rs | 10 + src/simd_masking_ops.rs | 656 ++++++++++++++++++ 3 files changed, 680 insertions(+) diff --git a/.claude/knowledge/vertical-simd-consumer-contract.md b/.claude/knowledge/vertical-simd-consumer-contract.md index 307f682f..803376ae 100644 --- a/.claude/knowledge/vertical-simd-consumer-contract.md +++ b/.claude/knowledge/vertical-simd-consumer-contract.md @@ -513,3 +513,17 @@ simd_{avx512,avx2,neon,wasm,scalar}.rs peer backends, each owns realization because it is spelled as a loop; it is scalar when `--emit asm` on the shipped method says so — and "mostly packed" is a category the oracle must be able to report, because a peel is invisible to any parity test. +- **A gated predicate is a facade word, not an executor trick (2026-09-14, + `*_to_mask_under`).** mask-risc's `Pred { under }` promises "evaluate only + where the gate has a survivor"; the kernel-membrane review of lance-graph + #1225 ruled that T1 must carry that word before any executor composes it, + because an executor that spells it as `*_to_mask` + `mask_and` pays the + whole plane twice and an executor that skips words itself has grown a + compute path above the facade. The family (`gt/lt/ge/le/eq/ne_i32`, + `eq/ne_u32`, `ternary_match_u32/u64` — the ten `Pred` variants, one each) + shares ONE private engine (`pack_under`) whose skip is at word + granularity: `under[w] == 0` writes zero and evaluates nothing. Cost ∝ live + gate words; result ≡ `pred & under` bit-for-bit, and the predicate's own + tail law makes a phantom gate bit past `n` vanish without a clean. The + can-it-fire test counts the closure invocations (5 live words × 4 groups = + 20 of 40), so "skips" is measured, not asserted. diff --git a/src/simd.rs b/src/simd.rs index 1d563e7c..b975c3d1 100644 --- a/src/simd.rs +++ b/src/simd.rs @@ -779,12 +779,18 @@ pub use crate::simd_masking_ops::{ // register match + masked min/max + blend (lance-graph-duckmask, lgj-abi D-MRL-1a). blend_i32, eq_i32_to_mask, + eq_i32_to_mask_under, eq_u32_strided_to_mask, eq_u32_to_mask, + eq_u32_to_mask_under, ge_i32_to_mask, + ge_i32_to_mask_under, gt_i32_to_mask, + gt_i32_to_mask_under, le_i32_to_mask, + le_i32_to_mask_under, lt_i32_to_mask, + lt_i32_to_mask_under, mask_all, mask_and, mask_and_assign, @@ -805,10 +811,14 @@ pub use crate::simd_masking_ops::{ masked_strided_group_sum, masked_sum_i32, ne_i32_to_mask, + ne_i32_to_mask_under, ne_u32_to_mask, + ne_u32_to_mask_under, ternary_match_strided_to_mask, ternary_match_u32_to_mask, + ternary_match_u32_to_mask_under, ternary_match_u64_to_mask, + ternary_match_u64_to_mask_under, MortonDir, }; // The popcount that closes the loop on the masks above: `mask_count` in ABI diff --git a/src/simd_masking_ops.rs b/src/simd_masking_ops.rs index 0d3fa207..7dcc02d0 100644 --- a/src/simd_masking_ops.rs +++ b/src/simd_masking_ops.rs @@ -1481,6 +1481,447 @@ pub fn ternary_match_strided_to_mask( } } +// ── Gated predicates: `*_to_mask_under` (mask-risc `Pred { under }`) ──────── +// +// `out[w] = under[w] & pred(values)[w]`, with the predicate EVALUATED only +// on words where `under[w] != 0` — the survivor-word skip. Cost is +// proportional to the live words of `under`, never to the row count: a +// frontier that has narrowed to 3 % of the plane pays 3 % of the compares. +// The alternative — a full `*_to_mask` pass followed by `mask_and` — costs +// the whole plane twice and is exactly what a mask program must not do +// once its first predicate has already narrowed the population. +// +// Word granularity (64 rows) is the finest a packed compare can skip at: +// one word is four 16-lane groups (or eight 8-lane groups), and the skip +// decision is one `u64 != 0` test on a word the caller already holds. An +// executor is free to skip COARSER (mask-risc's IR speaks of 1024-row +// chunks); the result is identical, only the cost differs. +// +// Semantics are otherwise those of the ungated sibling, and the tail law is +// inherited from it: the predicate's bits past `n` are zero, so +// `pred & under` conforms EVEN IF `under` carries phantom tail bits — a +// gate never needs cleaning before it is applied. Full overwrite: surplus +// words of `out_words` are written zero, as in every `*_to_mask`. + +/// The shared survivor-skip loop under every `*_to_mask_under`. +/// +/// `group_bits(lanes, live)` packs one register's worth of predicate results +/// — `lanes` is the full `[T; L]` (zero-padded when `live < L`) and the +/// closure masks its answer down to the low `live` bits — into a `u64` +/// whose bits above `live` are zero. The loop places each group at bit +/// `g * L` of its word and ANDs the word with `under[w]`. +#[inline(always)] +fn pack_under( + name: &str, values: &[T], under: &[u64], out_words: &mut [u64], group_bits: impl Fn([T; L], usize) -> u64, +) { + const { assert!(64 % L == 0, "a mask word must hold whole lane groups") } + let n = values.len(); + let words = mask_words_for(n); + assert!(out_words.len() >= words, "{name}: out_words.len()={} < required {words}", out_words.len()); + assert!(under.len() >= words, "{name}: under.len()={} < required {words}", under.len()); + for w in out_words[words..].iter_mut() { + *w = 0; + } + for w in 0..words { + let gate = under[w]; + if gate == 0 { + out_words[w] = 0; + continue; + } + let mut bits = 0u64; + let mut start = w * 64; + let end = n.min(start + 64); + let mut shift = 0; + while start < end { + let live = (end - start).min(L); + let group = &values[start..start + live]; + let lanes: [T; L] = if live == L { + group.try_into().expect("a full group is exactly L lanes") + } else { + pad_tail(group) + }; + bits |= group_bits(lanes, live) << shift; + start += L; + shift += L; + } + out_words[w] = bits & gate; + } +} + +/// `live`-lane validity mask for a 16-lane group: all ones for a full group, +/// [`tail_lane_bits`] for a padded one. +#[inline(always)] +fn live16(live: usize) -> u16 { + if live == 16 { + u16::MAX + } else { + tail_lane_bits(live) + } +} + +/// Packs `values[i] > threshold` (signed) AND `under`: bit `i` is set iff +/// both hold, and words where `under` has no survivor are written zero +/// **without evaluating the compare** — the survivor-word skip whose cost +/// is proportional to `under`'s live words, not to `values.len()`. +/// +/// Same comparison, bit order, and full-overwrite contract as +/// [`gt_i32_to_mask`]; `under`'s bits past `values.len()` are harmless (the +/// predicate's own tail is zero, so the AND conforms regardless). +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{gt_i32_to_mask, gt_i32_to_mask_under}; +/// +/// let values = [5i32, -5, 9, 12]; +/// let mut full = [0u64; 1]; +/// gt_i32_to_mask(&values, 0, &mut full); +/// assert_eq!(full[0], 0b1101); +/// // gate out element 3: the answer is the ungated mask AND the gate +/// let mut gated = [0u64; 1]; +/// gt_i32_to_mask_under(&values, 0, &[0b0111], &mut gated); +/// assert_eq!(gated[0], 0b0101); +/// // an empty gate word is written zero and its compares are skipped +/// gt_i32_to_mask_under(&values, 0, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn gt_i32_to_mask_under(values: &[i32], threshold: i32, under: &[u64], out_words: &mut [u64]) { + let t = crate::simd::I32x16::splat(threshold); + pack_under::("gt_i32_to_mask_under", values, under, out_words, |lanes, live| { + (crate::simd::I32x16::from_array(lanes).gt_bitmask(t) & live16(live)) as u64 + }); +} + +/// Packs `values[i] < threshold` (signed) AND `under`, using the +/// survivor-word skip: words where `under` has no live bit are written zero +/// without evaluating the compare. Same contract as [`gt_i32_to_mask_under`]; +/// gated form of [`lt_i32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{lt_i32_to_mask, lt_i32_to_mask_under}; +/// +/// let values = [5i32, -5, -1, 10]; +/// let mut full = [0u64; 1]; +/// lt_i32_to_mask(&values, 0, &mut full); +/// assert_eq!(full[0], 0b0110); +/// // gate out element 2: only element 1 survives +/// let mut gated = [0u64; 1]; +/// lt_i32_to_mask_under(&values, 0, &[0b1011], &mut gated); +/// assert_eq!(gated[0], 0b0010); +/// // an empty gate word is written zero and its compares are skipped +/// lt_i32_to_mask_under(&values, 0, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn lt_i32_to_mask_under(values: &[i32], threshold: i32, under: &[u64], out_words: &mut [u64]) { + let t = crate::simd::I32x16::splat(threshold); + pack_under::("lt_i32_to_mask_under", values, under, out_words, |lanes, live| { + (t.gt_bitmask(crate::simd::I32x16::from_array(lanes)) & live16(live)) as u64 + }); +} + +/// Packs `values[i] >= threshold` (signed) AND `under`, complementing +/// [`lt_i32_to_mask_under`]'s per-group bits *inside* the group so the tail +/// law stays [`live16`]'s to enforce, never a whole-word complement. Same +/// contract as [`gt_i32_to_mask_under`]; gated form of [`ge_i32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{ge_i32_to_mask, ge_i32_to_mask_under}; +/// +/// let values = [5i32, -5, -1, 10]; +/// let mut full = [0u64; 1]; +/// ge_i32_to_mask(&values, 0, &mut full); +/// assert_eq!(full[0], 0b1001); +/// // gate out element 3: only element 0 survives +/// let mut gated = [0u64; 1]; +/// ge_i32_to_mask_under(&values, 0, &[0b0011], &mut gated); +/// assert_eq!(gated[0], 0b0001); +/// // an empty gate word is written zero and its compares are skipped +/// ge_i32_to_mask_under(&values, 0, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn ge_i32_to_mask_under(values: &[i32], threshold: i32, under: &[u64], out_words: &mut [u64]) { + let t = crate::simd::I32x16::splat(threshold); + pack_under::("ge_i32_to_mask_under", values, under, out_words, |lanes, live| { + (!t.gt_bitmask(crate::simd::I32x16::from_array(lanes)) & live16(live)) as u64 + }); +} + +/// Packs `values[i] <= threshold` (signed) AND `under`, complementing +/// [`gt_i32_to_mask_under`]'s per-group bits *inside* the group. Same +/// contract as [`gt_i32_to_mask_under`]; gated form of [`le_i32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{le_i32_to_mask, le_i32_to_mask_under}; +/// +/// let values = [5i32, -5, -1, 10]; +/// let mut full = [0u64; 1]; +/// le_i32_to_mask(&values, 0, &mut full); +/// assert_eq!(full[0], 0b0110); +/// // gate out element 2: only element 1 survives +/// let mut gated = [0u64; 1]; +/// le_i32_to_mask_under(&values, 0, &[0b1011], &mut gated); +/// assert_eq!(gated[0], 0b0010); +/// // an empty gate word is written zero and its compares are skipped +/// le_i32_to_mask_under(&values, 0, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn le_i32_to_mask_under(values: &[i32], threshold: i32, under: &[u64], out_words: &mut [u64]) { + let t = crate::simd::I32x16::splat(threshold); + pack_under::("le_i32_to_mask_under", values, under, out_words, |lanes, live| { + (!crate::simd::I32x16::from_array(lanes).gt_bitmask(t) & live16(live)) as u64 + }); +} + +/// Packs `values[i] != needle` (signed) AND `under`, in one pass as +/// `(needle > v) | (v > needle)` — no complement, so the tail is zero by +/// construction, same as the ungated sibling. Same contract as +/// [`gt_i32_to_mask_under`]; gated form of [`ne_i32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{ne_i32_to_mask, ne_i32_to_mask_under}; +/// +/// let values = [7i32, 1, 7, 2]; +/// let mut full = [0u64; 1]; +/// ne_i32_to_mask(&values, 7, &mut full); +/// assert_eq!(full[0], 0b1010); +/// // gate out element 3: only element 1 survives +/// let mut gated = [0u64; 1]; +/// ne_i32_to_mask_under(&values, 7, &[0b0011], &mut gated); +/// assert_eq!(gated[0], 0b0010); +/// // an empty gate word is written zero and its compares are skipped +/// ne_i32_to_mask_under(&values, 7, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn ne_i32_to_mask_under(values: &[i32], needle: i32, under: &[u64], out_words: &mut [u64]) { + let t = crate::simd::I32x16::splat(needle); + pack_under::("ne_i32_to_mask_under", values, under, out_words, |lanes, live| { + let v = crate::simd::I32x16::from_array(lanes); + ((t.gt_bitmask(v) | v.gt_bitmask(t)) & live16(live)) as u64 + }); +} + +/// Packs `values[i] == needle` (signed, exact) AND `under`, complementing +/// [`ne_i32_to_mask_under`]'s per-group bits *inside* the group. Same +/// contract as [`gt_i32_to_mask_under`]; gated form of [`eq_i32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{eq_i32_to_mask, eq_i32_to_mask_under}; +/// +/// let values = [7i32, 1, 7, 2]; +/// let mut full = [0u64; 1]; +/// eq_i32_to_mask(&values, 7, &mut full); +/// assert_eq!(full[0], 0b0101); +/// // gate out element 2: only element 0 survives +/// let mut gated = [0u64; 1]; +/// eq_i32_to_mask_under(&values, 7, &[0b0011], &mut gated); +/// assert_eq!(gated[0], 0b0001); +/// // an empty gate word is written zero and its compares are skipped +/// eq_i32_to_mask_under(&values, 7, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn eq_i32_to_mask_under(values: &[i32], needle: i32, under: &[u64], out_words: &mut [u64]) { + let t = crate::simd::I32x16::splat(needle); + pack_under::("eq_i32_to_mask_under", values, under, out_words, |lanes, live| { + let v = crate::simd::I32x16::from_array(lanes); + (!(t.gt_bitmask(v) | v.gt_bitmask(t)) & live16(live)) as u64 + }); +} + +/// Packs `values[i] == needle` (unsigned, exact) AND `under`. Same contract +/// as [`gt_i32_to_mask_under`]; gated form of [`eq_u32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{eq_u32_to_mask, eq_u32_to_mask_under}; +/// +/// let values = [7u32, 1, 7, 2]; +/// let mut full = [0u64; 1]; +/// eq_u32_to_mask(&values, 7, &mut full); +/// assert_eq!(full[0], 0b0101); +/// // gate out element 2: only element 0 survives +/// let mut gated = [0u64; 1]; +/// eq_u32_to_mask_under(&values, 7, &[0b0011], &mut gated); +/// assert_eq!(gated[0], 0b0001); +/// // an empty gate word is written zero and its compares are skipped +/// eq_u32_to_mask_under(&values, 7, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn eq_u32_to_mask_under(values: &[u32], needle: u32, under: &[u64], out_words: &mut [u64]) { + let needle_v = crate::simd::U32x16::splat(needle); + pack_under::("eq_u32_to_mask_under", values, under, out_words, |lanes, live| { + (crate::simd::U32x16::from_array(lanes).eq_bitmask(needle_v) & live16(live)) as u64 + }); +} + +/// Packs `values[i] != needle` (unsigned) AND `under`, complementing +/// [`eq_u32_to_mask_under`]'s per-group bits *inside* the group. Same +/// contract as [`gt_i32_to_mask_under`]; gated form of [`ne_u32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{ne_u32_to_mask, ne_u32_to_mask_under}; +/// +/// let values = [7u32, 1, 7, 2]; +/// let mut full = [0u64; 1]; +/// ne_u32_to_mask(&values, 7, &mut full); +/// assert_eq!(full[0], 0b1010); +/// // gate out element 3: only element 1 survives +/// let mut gated = [0u64; 1]; +/// ne_u32_to_mask_under(&values, 7, &[0b0011], &mut gated); +/// assert_eq!(gated[0], 0b0010); +/// // an empty gate word is written zero and its compares are skipped +/// ne_u32_to_mask_under(&values, 7, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn ne_u32_to_mask_under(values: &[u32], needle: u32, under: &[u64], out_words: &mut [u64]) { + let needle_v = crate::simd::U32x16::splat(needle); + pack_under::("ne_u32_to_mask_under", values, under, out_words, |lanes, live| { + (!crate::simd::U32x16::from_array(lanes).eq_bitmask(needle_v) & live16(live)) as u64 + }); +} + +/// Care-masked match of `values[i]` against `pattern` (bits selected by +/// `care`) AND `under`: element `i` survives iff `((values[i] ^ pattern) & +/// care) == 0` **and** its gate bit is set. Same contract as +/// [`gt_i32_to_mask_under`]; gated form of [`ternary_match_u32_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{ternary_match_u32_to_mask, ternary_match_u32_to_mask_under}; +/// +/// let values = [0b1010u32, 0b1110, 0b0010, 0b1011]; +/// let mut full = [0u64; 1]; +/// ternary_match_u32_to_mask(&values, 0b1010, 0b1011, &mut full); +/// assert_eq!(full[0], 0b0011); +/// // gate out element 1: only element 0 survives +/// let mut gated = [0u64; 1]; +/// ternary_match_u32_to_mask_under(&values, 0b1010, 0b1011, &[0b1101], &mut gated); +/// assert_eq!(gated[0], 0b0001); +/// // an empty gate word is written zero and its compares are skipped +/// ternary_match_u32_to_mask_under(&values, 0b1010, 0b1011, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn ternary_match_u32_to_mask_under(values: &[u32], pattern: u32, care: u32, under: &[u64], out_words: &mut [u64]) { + let p = crate::simd::U32x16::splat(pattern); + let c = crate::simd::U32x16::splat(care); + let zero = crate::simd::U32x16::splat(0); + pack_under::("ternary_match_u32_to_mask_under", values, under, out_words, |lanes, live| { + let bits = crate::simd::U32x16::from_array(lanes) + .ternlog::<{ crate::simd::ternlog::XOR_AND }>(p, c) + .eq_bitmask(zero); + (bits & live16(live)) as u64 + }); +} + +/// The 64-bit sibling of [`ternary_match_u32_to_mask_under`]: care-masked +/// match of `values[i]` against `pattern`/`care` AND `under`. Same contract +/// as [`gt_i32_to_mask_under`]; gated form of [`ternary_match_u64_to_mask`]. +/// +/// # Panics +/// +/// Panics if `out_words.len()` or `under.len()` is below +/// `values.len().div_ceil(64)`. +/// +/// # Examples +/// +/// ``` +/// use ndarray::simd::{ternary_match_u64_to_mask, ternary_match_u64_to_mask_under}; +/// +/// let values = [0b1010u64, 0b1110, 0b0010, 0b1011]; +/// let mut full = [0u64; 1]; +/// ternary_match_u64_to_mask(&values, 0b1010, 0b1011, &mut full); +/// assert_eq!(full[0], 0b0011); +/// // gate out element 1: only element 0 survives +/// let mut gated = [0u64; 1]; +/// ternary_match_u64_to_mask_under(&values, 0b1010, 0b1011, &[0b1101], &mut gated); +/// assert_eq!(gated[0], 0b0001); +/// // an empty gate word is written zero and its compares are skipped +/// ternary_match_u64_to_mask_under(&values, 0b1010, 0b1011, &[0], &mut gated); +/// assert_eq!(gated[0], 0); +/// ``` +#[inline] +pub fn ternary_match_u64_to_mask_under(values: &[u64], pattern: u64, care: u64, under: &[u64], out_words: &mut [u64]) { + let p = crate::simd::U64x8::splat(pattern); + let c = crate::simd::U64x8::splat(care); + pack_under::("ternary_match_u64_to_mask_under", values, under, out_words, |lanes, live| { + let r = crate::simd::U64x8::from_array(lanes) + .ternlog::<{ crate::simd::ternlog::XOR_AND }>(p, c) + .to_array(); + let mut bits = 0u64; + for (lane, &x) in r.iter().take(live).enumerate() { + bits |= ((x == 0) as u64) << lane; + } + bits + }); +} + /// Minimum of `values[i]` over set mask bits, `None` when no bit is set. /// Same bit order and "bits at or past `values.len()` are ignored" contract /// as [`masked_sum_i32`]; cost proportional to the popcount. @@ -3283,4 +3724,219 @@ mod tests { let want2 = oracle_shift_field(&diag_src, -1, 1, n_cells, x_bits, y_bits); assert_eq!(dst2, want2, "-q+r composed via two calls must match the oracle diagonal"); } + + // ── Gated predicates: `*_to_mask_under` (survivor-word skip) ──────────── + + /// Every `*_to_mask_under` member must equal `ungated_sibling(values) & + /// gate`, word for word, including when the gate itself carries dirty + /// bits past `values.len()` in its last live word — the gate's own + /// phantom bits must never leak into the answer, and the ungated + /// reference's own tail law is what proves it (its bits past `n` are + /// already zero, so `full & gate` conforms regardless of the gate). + #[test] + fn under_family_equals_ungated_and_gate() { + const UNDER_LENS: [usize; 10] = [0, 1, 15, 16, 17, 63, 64, 65, 130, 1000]; + let i32_thresholds = [i32::MIN, i32::MIN + 1, -1, 0, 7, i32::MAX - 1, i32::MAX]; + let u32_needles = [0u32, 1, 3, 6, 42, u32::MAX]; + let ternary_cases: [(u32, u32); 4] = [(0b0101, 0xF), (0b0101, 0b1010), (0, 0), (0xF, 0xF)]; + + for &n in &UNDER_LENS { + let words = mask_words_for(n); + let v_i32 = i32_corpus(n, 0x1357_9BDF_ACE1_1234u64.wrapping_add(n as u64)); + + let mut s_eq = 0xF00D_BEEF_1234_5678u64.wrapping_add(n as u64); + let v_u32_eq: Vec = (0..n).map(|_| (splitmix(&mut s_eq) % 7) as u32).collect(); + + let mut s_tern32 = 0x9E37_79B9_ABCD_EF01u64.wrapping_add(n as u64); + let v_u32_tern: Vec = (0..n) + .map(|_| (splitmix(&mut s_tern32) % 16) as u32) + .collect(); + + let mut s_tern64 = 0x1122_3344_5566_7788u64.wrapping_add(n as u64); + let v_u64_tern: Vec = (0..n).map(|_| splitmix(&mut s_tern64) % 16).collect(); + + // ~50% zero words, random bits elsewhere, with the last live + // word's phantom bits (past `n`) also forced dirty at random — + // the engine must clean those independently of the ungated + // reference, which already zeroes them by its own contract. + let mut s_gate = 0xDEAD_C0DE_F00D_BAADu64.wrapping_add(n as u64); + let mut gate = vec![0u64; words]; + for w in gate.iter_mut() { + *w = if splitmix(&mut s_gate).is_multiple_of(2) { + 0 + } else { + splitmix(&mut s_gate) + }; + } + if words > 0 && !n.is_multiple_of(64) { + let last = words - 1; + let live = n - last * 64; + let phantom = !((1u64 << live) - 1); + gate[last] |= phantom & splitmix(&mut s_gate); + } + + // gt / lt / ge / le / ne_i32 / eq_i32, over signed thresholds. + for &t in &i32_thresholds { + let mut full = vec![0u64; words]; + let mut gated = vec![u64::MAX; words + 1]; + + gt_i32_to_mask(&v_i32, t, &mut full); + gt_i32_to_mask_under(&v_i32, t, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "gt_i32 n={n} t={t} w={w}"); + } + assert_eq!(gated[words], 0, "gt_i32 n={n} t={t} surplus word"); + + lt_i32_to_mask(&v_i32, t, &mut full); + lt_i32_to_mask_under(&v_i32, t, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "lt_i32 n={n} t={t} w={w}"); + } + assert_eq!(gated[words], 0, "lt_i32 n={n} t={t} surplus word"); + + ge_i32_to_mask(&v_i32, t, &mut full); + ge_i32_to_mask_under(&v_i32, t, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "ge_i32 n={n} t={t} w={w}"); + } + assert_eq!(gated[words], 0, "ge_i32 n={n} t={t} surplus word"); + + le_i32_to_mask(&v_i32, t, &mut full); + le_i32_to_mask_under(&v_i32, t, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "le_i32 n={n} t={t} w={w}"); + } + assert_eq!(gated[words], 0, "le_i32 n={n} t={t} surplus word"); + + ne_i32_to_mask(&v_i32, t, &mut full); + ne_i32_to_mask_under(&v_i32, t, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "ne_i32 n={n} t={t} w={w}"); + } + assert_eq!(gated[words], 0, "ne_i32 n={n} t={t} surplus word"); + + eq_i32_to_mask(&v_i32, t, &mut full); + eq_i32_to_mask_under(&v_i32, t, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "eq_i32 n={n} t={t} w={w}"); + } + assert_eq!(gated[words], 0, "eq_i32 n={n} t={t} surplus word"); + } + + // eq_u32 / ne_u32, over unsigned needles. + for &needle in &u32_needles { + let mut full = vec![0u64; words]; + let mut gated = vec![u64::MAX; words + 1]; + + eq_u32_to_mask(&v_u32_eq, needle, &mut full); + eq_u32_to_mask_under(&v_u32_eq, needle, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "eq_u32 n={n} needle={needle} w={w}"); + } + assert_eq!(gated[words], 0, "eq_u32 n={n} needle={needle} surplus word"); + + ne_u32_to_mask(&v_u32_eq, needle, &mut full); + ne_u32_to_mask_under(&v_u32_eq, needle, &gate, &mut gated); + for w in 0..words { + assert_eq!(gated[w], full[w] & gate[w], "ne_u32 n={n} needle={needle} w={w}"); + } + assert_eq!(gated[words], 0, "ne_u32 n={n} needle={needle} surplus word"); + } + + // ternary_match_u32 / ternary_match_u64, over pattern/care cases. + for &(pattern, care) in &ternary_cases { + let mut full = vec![0u64; words]; + let mut gated = vec![u64::MAX; words + 1]; + + ternary_match_u32_to_mask(&v_u32_tern, pattern, care, &mut full); + ternary_match_u32_to_mask_under(&v_u32_tern, pattern, care, &gate, &mut gated); + for w in 0..words { + assert_eq!( + gated[w], + full[w] & gate[w], + "ternary_u32 n={n} pattern={pattern:#x} care={care:#x} w={w}" + ); + } + assert_eq!(gated[words], 0, "ternary_u32 n={n} pattern={pattern:#x} care={care:#x} surplus word"); + + let pattern64 = pattern as u64; + let care64 = care as u64; + ternary_match_u64_to_mask(&v_u64_tern, pattern64, care64, &mut full); + ternary_match_u64_to_mask_under(&v_u64_tern, pattern64, care64, &gate, &mut gated); + for w in 0..words { + assert_eq!( + gated[w], + full[w] & gate[w], + "ternary_u64 n={n} pattern={pattern64:#x} care={care64:#x} w={w}" + ); + } + assert_eq!(gated[words], 0, "ternary_u64 n={n} pattern={pattern64:#x} care={care64:#x} surplus word"); + } + } + } + + /// Can-it-fire test for the survivor-word skip: an empty gate word must + /// never reach the predicate. Gate alternates `u64::MAX` / `0` over 10 + /// words (5 live), so the predicate closure — which counts its own + /// invocations — must fire exactly `5 * 4` times (five live words, four + /// 16-lane groups per word); an all-ones gate must fire on every group. + #[test] + fn under_skips_compares_on_empty_gate_words() { + let n = 640; // exactly 10 words of 64 elements + let values = vec![0i32; n]; + let words = mask_words_for(n); + assert_eq!(words, 10); + + let counter = std::cell::Cell::new(0usize); + let gate: Vec = (0..words) + .map(|w| if w % 2 == 0 { u64::MAX } else { 0 }) + .collect(); + let mut out = vec![0u64; words]; + pack_under::("under_skips_compares_on_empty_gate_words", &values, &gate, &mut out, |_lanes, _live| { + counter.set(counter.get() + 1); + 0 + }); + assert_eq!(counter.get(), 5 * 4, "five live words x four 16-lane groups"); + + counter.set(0); + let all_ones = vec![u64::MAX; words]; + pack_under::( + "under_skips_compares_on_empty_gate_words", + &values, + &all_ones, + &mut out, + |_lanes, _live| { + counter.set(counter.get() + 1); + 0 + }, + ); + assert_eq!(counter.get(), 40, "an all-ones gate evaluates every group"); + } + + /// A dirty phantom bit in the gate's last live word (bits at or past + /// `n % 64`, which correspond to no real element) must never survive + /// into the answer — element count 70 leaves word 1 with only 6 live + /// bits, and the gate sets every bit of that word including the 58 + /// phantom ones. + #[test] + fn under_phantom_gate_bits_do_not_leak() { + let n = 70; + let values = vec![0i32; n]; // every value satisfies `> i32::MIN` + let gate = vec![u64::MAX, u64::MAX]; // word 1's bits 6..64 are phantom + let mut out = vec![0u64; 2]; + gt_i32_to_mask_under(&values, i32::MIN, &gate, &mut out); + assert_eq!(out[0], u64::MAX, "word 0 is fully live and fully satisfies `> i32::MIN`"); + assert_eq!(out[1], 0b11_1111, "only the 6 live bits of word 1 may be set"); + } + + /// `under.len()` one word short of `values.len().div_ceil(64)` must + /// panic, mirroring every other `*_to_mask` length guard in this module. + #[test] + #[should_panic(expected = "under.len()")] + fn under_rejects_short_gate() { + let values = vec![0i32; 65]; // needs 2 gate words + let gate = vec![0u64; 1]; // one short + let mut out = vec![0u64; 2]; + gt_i32_to_mask_under(&values, 0, &gate, &mut out); + } } From 53e4723051d732cc1b99d33145df679b581a617a Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 16:25:35 +0000 Subject: [PATCH 7/9] simd-masking-parity: 0xAxx gated-predicate group (10 groups); blackboard D-MRX-0 `check_predicates_under` diffs every `*_to_mask_under` against a scalar reference that reads the gate BIT per row, over a gate with every other word forced to zero (the skip path on every length >= 64) and random bits elsewhere including the last word's bits past `n`, one surplus out word pre-filled `u64::MAX`. Codes `0xA00..0xA9x`, one sub-range per member. Native AVX2, nightly `core::simd`, wasm simd128 and wasm scalar: 10/10 groups bit-identical. Clippy `-D warnings` and fmt clean on both crates. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/blackboard.md | 41 ++++++++++++ crates/simd-masking-parity/src/lib.rs | 93 ++++++++++++++++++++++++--- 2 files changed, 126 insertions(+), 8 deletions(-) diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 3c6e5cef..6ccc4217 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -1,3 +1,44 @@ +## 2026-09-14 (4) — D-MRX-0: `*_to_mask_under` — the gated predicate mask-risc's `Pred { under }` promised and T1 lacked + +**Why now.** lance-graph #1225's kernel-membrane review (PR2 council) ruled the +executor may not compose a gated predicate from `*_to_mask` + `mask_and` (the +whole plane, twice) nor skip words itself (a compute path above the facade): +*"PR3 lands `*_to_mask_under` first."* This is that word. + +**Shape.** ONE private engine, `pack_under::(name, values, under, out, +group_bits)`: `out[w] = under[w] & pred(values)[w]`, the predicate evaluated +only where `under[w] != 0`. Word granularity (64 rows) is the finest a packed +compare can skip at; an executor may skip coarser (mask-risc speaks of +1024-row chunks) with an identical result. Cost ∝ live gate words, never rows. +The predicate's own tail law makes a phantom gate bit past `n` vanish — the +AND conforms regardless, so a gate never needs cleaning before use. Ten public +members, one per `Pred` variant (`gt/lt/ge/le/eq/ne_i32`, `eq/ne_u32`, +`ternary_match_u32/u64`), each one `pack_under` call over the SAME lane op its +ungated sibling uses — no new backend semantics, so no backend file changed. + +**Falsifiers.** (1) family ≡ `ungated & gate` word-for-word at ten lengths with +phantom gate bits present; (2) the skip is MEASURED: a counting closure sees +20 of 40 groups under an alternating gate, 40 under all-ones — disable-run +red (`left: 40, right: 20`) with the `gate == 0` early-out removed, green +restored; (3) phantom bits do not leak at n = 70; (4) short gate panics; +doctests 10/10. Parity harness group 10 (`0xAxx`, `check_predicates_under`): +alternating-zero gate with random phantom bits, reference reads the gate BIT +per row. Native (AVX2), nightly (`core::simd`), wasm simd128 and wasm scalar: all +10/10 bit-identical (neon under qemu not runnable in this environment, as +before). + +**Allocation.** Engine + first member by the orchestrator; nine members + +tests by a Sonnet worker against a written spec (its Bash died on a full disk +mid-run — the tasks tmpfs and the checkout share one allowance — so it landed +Tasks A–C unverified and reported exactly that; the orchestrator gated +centrally after freeing 6.5 GB of stale scratch targets and landed the parity +arm itself). W1a deviation record as for `mask_andnot` / `mask_ternlog`: +free-fn family shape beside its siblings, not a struct method. + +**Loose ends.** The strided operand family (12-in-16-byte register compares +for `LaneRef`) and `u8/u16/u64` compare-to-mask (DuckDB matrix G1/G2) remain +T1 gaps; neither is needed for PR3's first executor. + ## 2026-09-14 (3) — D-GTM-1m MEASURED: `mask_shift_morton` lands; the win is in the NODE SPAN, not the op — n = 17.0 → 5.7 µs (−66 %) `mask_shift_morton` (255c36d) is bit-exact (F1–F4, parity 9/9 on native/ diff --git a/crates/simd-masking-parity/src/lib.rs b/crates/simd-masking-parity/src/lib.rs index 27ad6ccc..509f8de3 100644 --- a/crates/simd-masking-parity/src/lib.rs +++ b/crates/simd-masking-parity/src/lib.rs @@ -17,20 +17,23 @@ //! `U64x8`, `0x3xx` `U64x8` algebra, `0x4xx` `I32x16` compares, `0x5xx` //! predicate→mask, `0x6xx` mask algebra, `0x7xx` care-match, `0x8xx` masked //! reductions and blend, `0x9xx` `mask_shift_morton` (the Morton hex -//! neighbour shift, D-GTM-1m). `main.rs` (native / qemu) and `selfcheck()` +//! neighbour shift, D-GTM-1m), `0xAxx` the gated `*_to_mask_under` +//! predicates (mask-risc `Pred { under }`, D-MRX-0). `main.rs` (native / qemu) and `selfcheck()` //! (the wasm cdylib export, driven by `run.mjs`) both call [`run`]. use ndarray::simd::{ - blend_i32, eq_i32_to_mask, eq_u32_strided_to_mask, eq_u32_to_mask, ge_i32_to_mask, gt_i32_to_mask, le_i32_to_mask, - lt_i32_to_mask, mask_all, mask_and, mask_and_assign, mask_andnot, mask_andnot_assign, mask_any, mask_not, - mask_not_assign, mask_or, mask_or_assign, mask_shift_morton, mask_ternlog, mask_ternlog_assign, mask_xor, - mask_xor_assign, masked_max_i32, masked_min_i32, masked_strided_group_sum, masked_sum_i32, ne_i32_to_mask, - ne_u32_to_mask, ternary_match_strided_to_mask, ternary_match_u32_to_mask, ternary_match_u64_to_mask, ternlog, - I32x16, MortonDir, U32x16, U64x8, + blend_i32, eq_i32_to_mask, eq_i32_to_mask_under, eq_u32_strided_to_mask, eq_u32_to_mask, eq_u32_to_mask_under, + ge_i32_to_mask, ge_i32_to_mask_under, gt_i32_to_mask, gt_i32_to_mask_under, le_i32_to_mask, le_i32_to_mask_under, + lt_i32_to_mask, lt_i32_to_mask_under, mask_all, mask_and, mask_and_assign, mask_andnot, mask_andnot_assign, + mask_any, mask_not, mask_not_assign, mask_or, mask_or_assign, mask_shift_morton, mask_ternlog, mask_ternlog_assign, + mask_xor, mask_xor_assign, masked_max_i32, masked_min_i32, masked_strided_group_sum, masked_sum_i32, + ne_i32_to_mask, ne_i32_to_mask_under, ne_u32_to_mask, ne_u32_to_mask_under, ternary_match_strided_to_mask, + ternary_match_u32_to_mask, ternary_match_u32_to_mask_under, ternary_match_u64_to_mask, + ternary_match_u64_to_mask_under, ternlog, I32x16, MortonDir, U32x16, U64x8, }; /// Number of check groups [`run`] executes (for the log line only). -pub const CHECKS: usize = 9; +pub const CHECKS: usize = 10; /// The wasm export: identical to [`run`], `extern "C"` so `run.mjs` can call it. #[no_mangle] @@ -43,6 +46,7 @@ pub fn run() -> u32 { let groups: [fn() -> Result<(), u32>; CHECKS] = [ check_ternlog_all_tables, check_u64x8_algebra, check_i32x16_compare, check_predicates_to_mask, check_mask_algebra, check_care_match, check_masked_reductions, check_blend, check_morton_shift, + check_predicates_under, ]; for g in groups { if let Err(code) = g() { @@ -433,6 +437,79 @@ fn check_predicates_to_mask() -> Result<(), u32> { Ok(()) } +// ── 0xAxx: the gated predicates — `out = pred & under`, survivor-word skip ── + +/// Every `*_to_mask_under` against `reference_mask(n, out_len, |i| pred(i) && +/// gate_bit(i))`. The gate is built with every other word forced to ZERO (so +/// the skip path is exercised on every length ≥ 64) and random bits elsewhere +/// INCLUDING the last word's bits past `n` — the reference reads the gate BIT +/// for row `i`, so a phantom past `n` never enters it, and the primitive must +/// agree. One surplus out word pre-filled `u64::MAX` catches an OR-er. +fn check_predicates_under() -> Result<(), u32> { + let mut rng = SplitMix64(0x5EED_0000_00A0); + for &n in &LENS { + let out_len = words_for(n) + 1; + let vals = i32_values(n, &mut rng); + let uvals = u32_values(n, &mut rng); + let vals64: Vec = (0..n).map(|_| rng.next() % 16).collect(); + let gate: Vec = (0..words_for(n)) + .map(|w| if w % 2 == 1 { 0 } else { rng.next() | 1 }) + .collect(); + let gate_bit = |i: usize| (gate[i / 64] >> (i % 64)) & 1 == 1; + let mut out = vec![u64::MAX; out_len]; + let thresholds = [i32::MIN, -1, 0, 7, i32::MAX]; + for (k, &t) in thresholds.iter().enumerate() { + let k = k as u32; + macro_rules! pred { + ($f:ident, $op:tt, $code:expr) => {{ + out.iter_mut().for_each(|w| *w = u64::MAX); + $f(&vals, t, &gate, &mut out); + if out != reference_mask(n, out_len, |i| vals[i] $op t && gate_bit(i)) { + return Err($code | k); + } + }}; + } + pred!(gt_i32_to_mask_under, >, 0xA00); + pred!(lt_i32_to_mask_under, <, 0xA10); + pred!(ge_i32_to_mask_under, >=, 0xA20); + pred!(le_i32_to_mask_under, <=, 0xA30); + pred!(ne_i32_to_mask_under, !=, 0xA40); + pred!(eq_i32_to_mask_under, ==, 0xA50); + } + for (k, &needle) in [0u32, 7, u32::MAX, 0x8000_0000].iter().enumerate() { + let k = k as u32; + out.iter_mut().for_each(|w| *w = u64::MAX); + eq_u32_to_mask_under(&uvals, needle, &gate, &mut out); + if out != reference_mask(n, out_len, |i| uvals[i] == needle && gate_bit(i)) { + return Err(0xA60 | k); + } + out.iter_mut().for_each(|w| *w = u64::MAX); + ne_u32_to_mask_under(&uvals, needle, &gate, &mut out); + if out != reference_mask(n, out_len, |i| uvals[i] != needle && gate_bit(i)) { + return Err(0xA70 | k); + } + } + for (k, &(pattern, care)) in [(0b1010u32, 0b1011u32), (7, u32::MAX), (0, 0)] + .iter() + .enumerate() + { + let k = k as u32; + out.iter_mut().for_each(|w| *w = u64::MAX); + ternary_match_u32_to_mask_under(&uvals, pattern, care, &gate, &mut out); + if out != reference_mask(n, out_len, |i| (uvals[i] ^ pattern) & care == 0 && gate_bit(i)) { + return Err(0xA80 | k); + } + let (p64, c64) = (u64::from(pattern), u64::from(care)); + out.iter_mut().for_each(|w| *w = u64::MAX); + ternary_match_u64_to_mask_under(&vals64, p64, c64, &gate, &mut out); + if out != reference_mask(n, out_len, |i| (vals64[i] ^ p64) & c64 == 0 && gate_bit(i)) { + return Err(0xA90 | k); + } + } + } + Ok(()) +} + // ── 0x6xx: mask algebra, in-place forms, ternlog over slices, any / all ───── /// Word counts for the mask-algebra group. `LENS` are ROW counts, and at 130 From e5a87e65605d93511f04660fd2f7ea89d318f07b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 16:47:54 +0000 Subject: [PATCH 8/9] =?UTF-8?q?simd=5Fmasking=5Fops:=20council=20fixes=20f?= =?UTF-8?q?or=20#307=20=E2=80=94=20shift-count=20guards,=20sub-span=20law,?= =?UTF-8?q?=20accumulate=20falsifier?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the review findings on the gated-predicate + Morton-shift PR: - `U64x8::{Shl,Shr}` on all six backends carry a `debug_assert!` that every shift count is < 64, turning the doc contract into a checked one (counts >= 64 differ per backend — zero/masked/debug-panic — with no UB). - `mask_shift_morton`: documents "the slice IS the field" (a sub-span shift is not the restriction of the full-field shift; the caller aligns the span origin to its length); section and variant docs speak in lattice terms; F2 covers `n_words in {1, 4, 64, 1024}` so the padded-tail branch runs in-crate; new falsifier `morton_shift_or_accumulates_into_a_prefilled_dst` plus parity arm 0x904 pin the OR-accumulate contract. - Module header names the one accumulator exception to the full-overwrite guarantee; `pack_under` debug-asserts the closure contract (group bits above `live` are zero) at the one `|=`. - Cost prose corrected: compare cost tracks live words; the per-word gate test stays proportional to rows/64. The coarser-skip equivalence is stated as by-construction, untested. - Structural test pins ten `*_to_mask_under` definitions delegating to the single engine. - Parity harness: group-10 doc coverage stated exactly (65 and 130). - hex probe: Hebbian attribution reads the step's source frontier via a shared `hebbian_reverse_walk`; delta is published after the walk; the span-alignment assert at the shift call site; the timed reset is named in full (786 KiB, not 8 KiB). Measured inert on this fixture. - Spec §15 tail line corrected; knowledge doc and blackboard updated. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- .claude/blackboard.md | 43 ++++-- .../vertical-simd-consumer-contract.md | 28 +++- .../gemm-ternlog-mask-consolidation-v1.md | 2 +- crates/simd-masking-parity/src/lib.rs | 16 ++- examples/hex_tenant_mq_probe.rs | 91 +++++++------ src/simd_avx2.rs | 2 + src/simd_avx512.rs | 2 + src/simd_masking_ops.rs | 127 +++++++++++++++--- src/simd_neon.rs | 2 + src/simd_nightly/u_word_types.rs | 2 + src/simd_scalar.rs | 2 + src/simd_wasm.rs | 2 + 12 files changed, 244 insertions(+), 75 deletions(-) diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 6ccc4217..71aa84ba 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -1,3 +1,18 @@ +## 2026-09-14 (5) — STORNO on entries (3) and (2), from the #307 council (measurement-skeptic, kernel-membrane, overclaim): same numbers, tighter words + +The numbers stand; the wording ran past them. Corrections, each to the sentence it replaces: + +- (3) *"full-field −14 %"* → **−5 % to −14 % across two runs** (14 517 vs 16 944 and 16 507 vs 17 462): within this density's cross-run noise, i.e. no reliable full-field win over the per-bit loop. The table pinned the better run. +- (3) *"node-span −66 % (n 17.0 → 5.7 µs)"* → **on this fixture (frontier confined to one 64-word level-1 tile, 1/16 of the field, 29 % of the tile span), restricting the shifts to the tile span cuts the x=0 STEP from 16.9 to 5.7 µs (−66 %, one run).** It is a step ratio, not an op ratio: every arm still carries the full-field housekeeping (scratch zeroing, `OR2_AND`, the reverse walk over 1024 words) AND the probe's own per-rep reset — the `rails_run` copy is **786 KiB** (65 536 × 12 B), not the "8 KiB" the probe comment said (fixed), ~1 µs per step amortized inside every `n`. A frontier straddling k nodes pays Σ spans. +- (3) *"the ratio survives the degree-1 ablation"* → **a −33 % gain survives** (4 487 vs 6 704); the ratio halves. What the ablation shows is that the word op's advantage is not hex-specific; the other half of the degree-6 gain scales with the per-bit loop's direction count. +- (3) *"remaining 5.7 µs is the field-wide housekeeping"* → housekeeping **plus** the reset amortization; neither is separated. +- (3) *"delta-frontier on top of the node span buys nothing (5.7 → 6.0)"* → no gain; the +0.4 µs is inside single-window noise, not a finding. +- (2)/(3) *"coal = 8.9 µs = 0.48 maintained steps"* → **8.8–13.7 µs across three runs = 0.47–0.76 steps at x=4** (single window each). *"ternlogq 281 ns/pass"* → 280–300 across runs, max residual ≤ 2.8 %; the "1.7 % at x=1" is fit-derived. *"228–462×"* range-reveal ratio → ~200–490× across three runs at N = 65 536; the range arm's floor is the 8 KiB output clear. +- (3) *"68/68 gates"* → 68/68 probe ROWS pass the three spread gates on one fixture and one seed. *"1.8 % density"* → 1.8 % of the field, **29 % of the tile span** — the denominator that explains why the word op loses full-field and wins node-span. +- Method: one 50 ms window per cell, no repeats, no variance reported; cross-run spread is visible only because three runs were banked (ladder x=0 spread 3 %, `shift` 14 %, coal 56 %). Single-fixture observations, all of them. +- Kernel-membrane, on `mask_shift_morton`: the slice IS the field — a sub-span shift is NOT the restriction of the full-field shift (no carry crosses the span edge). The probe's node arm was correct only because a level-1 tile is Morton-aligned and the source is confined to it; that law is now on the op's doc and the probe asserts `lo % len == 0` at the call site. The op is a lattice axis shift (four Cartesian moves; hex diagonals are two-call compositions), renamed as such in its docs. +- Codex P2 on the probe: the Hebbian reverse walk credited any predecessor in the accumulated `state`; under the delta-frontier arms only `delta` cells propagated, so an older active neighbour of a newly reached cell could be credited for a firing it did not carry. Fixed (attribution against the step's SOURCE frontier, `delta` published after the walk) and **measured inert on this fixture — all 68 rows' fired/step and survivors identical before and after** — because eligibility is static here: an older predecessor's target was already settled when that predecessor was itself in the delta. Real for dynamic eligibility, unobservable in this probe. + ## 2026-09-14 (4) — D-MRX-0: `*_to_mask_under` — the gated predicate mask-risc's `Pred { under }` promised and T1 lacked **Why now.** lance-graph #1225's kernel-membrane review (PR2 council) ruled the @@ -7,11 +22,16 @@ whole plane, twice) nor skip words itself (a compute path above the facade): **Shape.** ONE private engine, `pack_under::(name, values, under, out, group_bits)`: `out[w] = under[w] & pred(values)[w]`, the predicate evaluated -only where `under[w] != 0`. Word granularity (64 rows) is the finest a packed -compare can skip at; an executor may skip coarser (mask-risc speaks of -1024-row chunks) with an identical result. Cost ∝ live gate words, never rows. -The predicate's own tail law makes a phantom gate bit past `n` vanish — the -AND conforms regardless, so a gate never needs cleaning before use. Ten public +only where `under[w] != 0`. The skip is word-granular (64 rows) — the unit +the caller holds; a per-group skip is reachable and deliberately not done. An +executor may skip coarser (mask-risc speaks of 1024-row chunks) with a result +identical by construction (a skipped chunk is an all-zero gate) — not +exercised by any test. COMPARE cost ∝ live gate WORDS (a sparse frontier +spread across every word pays every compare); the per-word gate test and +zero store stay ∝ rows/64; no timing of the skip exists, only the call count +below. The predicate's own tail law makes a phantom gate bit past `n` vanish — +the AND conforms regardless, so a gate never needs cleaning before use +(surplus `under` words are never read). Ten public members, one per `Pred` variant (`gt/lt/ge/le/eq/ne_i32`, `eq/ne_u32`, `ternary_match_u32/u64`), each one `pack_under` call over the SAME lane op its ungated sibling uses — no new backend semantics, so no backend file changed. @@ -20,15 +40,18 @@ ungated sibling uses — no new backend semantics, so no backend file changed. phantom gate bits present; (2) the skip is MEASURED: a counting closure sees 20 of 40 groups under an alternating gate, 40 under all-ones — disable-run red (`left: 40, right: 20`) with the `gate == 0` early-out removed, green -restored; (3) phantom bits do not leak at n = 70; (4) short gate panics; +restored (verified once, not re-runnable); (3) phantom bits do not leak at +n = 70; (4) short gate panics; (5) each public member is ONE `pack_under` +delegation, counted from the source, so the skip test on the private engine +covers all ten; doctests 10/10. Parity harness group 10 (`0xAxx`, `check_predicates_under`): alternating-zero gate with random phantom bits, reference reads the gate BIT per row. Native (AVX2), nightly (`core::simd`), wasm simd128 and wasm scalar: all -10/10 bit-identical (neon under qemu not runnable in this environment, as -before). +10/10 bit-identical here; neon has no qemu in this sandbox, and the +`neon-simd/parity-qemu` CI job on #307 is the witness for that arm (green). **Allocation.** Engine + first member by the orchestrator; nine members + -tests by a Sonnet worker against a written spec (its Bash died on a full disk +tests by a worker agent against a written spec (its Bash died on a full disk mid-run — the tasks tmpfs and the checkout share one allowance — so it landed Tasks A–C unverified and reported exactly that; the orchestrator gated centrally after freeing 6.5 GB of stale scratch targets and landed the parity @@ -43,7 +66,7 @@ T1 gaps; neither is needed for PR3's first executor. `mask_shift_morton` (255c36d) is bit-exact (F1–F4, parity 9/9 on native/ nightly/wasm/wasm-scalar; neon-qemu absent here) — and over the FULL field it -barely moves `n`: 14.5–16.5 µs vs 17.0 µs for the per-bit loop. The Sonnet +barely moves `n`: 14.5–16.5 µs vs 17.0 µs for the per-bit loop. The worker implementer named the mechanism correctly: the op is FIELD-size-bound (1024 words × ~8 passes per direction) while the per-bit loop is ACTIVE-bound, so at 1.8 % density they cost about the same and the op loses once gates thin the diff --git a/.claude/knowledge/vertical-simd-consumer-contract.md b/.claude/knowledge/vertical-simd-consumer-contract.md index 803376ae..e4e4c54c 100644 --- a/.claude/knowledge/vertical-simd-consumer-contract.md +++ b/.claude/knowledge/vertical-simd-consumer-contract.md @@ -522,8 +522,26 @@ simd_{avx512,avx2,neon,wasm,scalar}.rs peer backends, each owns realization compute path above the facade. The family (`gt/lt/ge/le/eq/ne_i32`, `eq/ne_u32`, `ternary_match_u32/u64` — the ten `Pred` variants, one each) shares ONE private engine (`pack_under`) whose skip is at word - granularity: `under[w] == 0` writes zero and evaluates nothing. Cost ∝ live - gate words; result ≡ `pred & under` bit-for-bit, and the predicate's own - tail law makes a phantom gate bit past `n` vanish without a clean. The - can-it-fire test counts the closure invocations (5 live words × 4 groups = - 20 of 40), so "skips" is measured, not asserted. + granularity: `under[w] == 0` writes zero and evaluates nothing. Compare + cost ∝ live gate WORDS (a sparse frontier spread across every word pays + every compare); the per-word gate test stays ∝ rows/64. Result ≡ + `pred & under` bit-for-bit, and the predicate's own tail law makes a + phantom gate bit past `n` vanish without a clean (surplus `under` words + are never read). The can-it-fire test counts the closure invocations + (5 live words × 4 groups = 20 of 40), so "skips" is a measured call + count — no timing of the skip exists. +- **`mask_shift_morton` is admitted on a narrower argument than the rest of + this module, and the argument is written down (2026-09-14, kernel-membrane + review of ndarray #307).** The admission test every word here passes: a + slice-level composition of lane ops with no ISA branch (`U64x8` shifts, + ANDs, ORs — zero `core::arch`, `#![forbid(unsafe_code)]` holds). What no + other word here does: it asserts a caller-held ADDRESS MODEL (word index = + Morton key of an 8×8-block grid; `q` on even bits, `r` on odd) and treats + the slice AS the field, so a sub-span shift is not the restriction of the + full-field shift. That is the cost of admitting it, and the fence is the + op's own doc paragraph ("The slice IS the field") plus the caller's + alignment assert. It stays here rather than in a consumer because the + alternative — the per-active-bit loop the hex probe measured at 17.0 µs a + step — is exactly the above-facade compute path T1 exists to prevent; and + it is named a lattice axis shift, not a hex op, because the four variants + are Cartesian and every hex diagonal is a two-call composition. diff --git a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md index 76794cf4..e46911b8 100644 --- a/.claude/plans/gemm-ternlog-mask-consolidation-v1.md +++ b/.claude/plans/gemm-ternlog-mask-consolidation-v1.md @@ -969,7 +969,7 @@ permutation with `U64x8` and/shl/or lane ops (the same facade-over-lane-types shape `mask_xor` uses — no per-ISA code in the facade; if a backend lacks `Shl`/`Shl` for `U64x8`, the implementer STOPS and reports which, never hand-rolls an intrinsic); pass 2 a scalar loop over words applying the -carry byte to `inc/dec(w)`. Tail law: the field is square so there is no tail; +carry byte to `inc/dec(w)`. Tail law: the field is square, but `4^0 = 1` and `4^1 = 4` words are not multiples of `U64x8::LANES`, so the implementation carries a padded-tail branch for those two sizes (corrected 2026-09-14 — an earlier draft of this line claimed there is no tail); the assert makes that explicit. **Falsifiers (all required, all must be able to fail):** diff --git a/crates/simd-masking-parity/src/lib.rs b/crates/simd-masking-parity/src/lib.rs index 509f8de3..6f023440 100644 --- a/crates/simd-masking-parity/src/lib.rs +++ b/crates/simd-masking-parity/src/lib.rs @@ -441,8 +441,9 @@ fn check_predicates_to_mask() -> Result<(), u32> { /// Every `*_to_mask_under` against `reference_mask(n, out_len, |i| pred(i) && /// gate_bit(i))`. The gate is built with every other word forced to ZERO (so -/// the skip path is exercised on every length ≥ 64) and random bits elsewhere -/// INCLUDING the last word's bits past `n` — the reference reads the gate BIT +/// the skip path is exercised on every length > 64 — 65 and 130 here) and +/// random bits elsewhere INCLUDING, at lengths whose last word index is even +/// (1, 63, 130), that word's bits past `n` — the reference reads the gate BIT /// for row `i`, so a phantom past `n` never enters it, and the primitive must /// agree. One surplus out word pre-filled `u64::MAX` catches an OR-er. fn check_predicates_under() -> Result<(), u32> { @@ -927,6 +928,17 @@ fn check_morton_shift() -> Result<(), u32> { return Err(0x900 | k as u32); } } + // 0x904: the OR-accumulate contract — a pre-filled `dst` keeps its + // prior bits (every other arm starts from zero and cannot see an + // overwrite). + let prior: Vec = (0..n_words).map(|_| rng.next()).collect(); + let mut got = prior.clone(); + mask_shift_morton(&src, MortonDir::PosQ, &mut got); + let want = morton_reference_shift(&src, 1, 0, n_cells, x_bits, y_bits); + let expect: Vec = prior.iter().zip(&want).map(|(p, w)| p | w).collect(); + if got != expect { + return Err(0x904); + } } } Ok(()) diff --git a/examples/hex_tenant_mq_probe.rs b/examples/hex_tenant_mq_probe.rs index 0c44b721..ff55d4c0 100644 --- a/examples/hex_tenant_mq_probe.rs +++ b/examples/hex_tenant_mq_probe.rs @@ -229,6 +229,38 @@ struct Resident<'a> { } #[inline(never)] +/// The Hebbian reverse walk shared by both spread arms: for every cell `j` +/// newly reached this step (`scratch & !state`), bump the strength byte of +/// each rail `d` whose predecessor `i` (j's neighbour along the opposite rail) +/// is BOTH in this step's `source` frontier AND eligible to leave along `d`. +/// Returns the number of firings. Reads `source` and `state`, writes only +/// `rails` — the caller publishes the new `delta` afterwards, because under +/// `from_delta` the source IS the old delta and must survive the walk. +fn hebbian_reverse_walk( + scratch: &[u64], state: &[u64], source: &[u64], elig: &[Vec], dirs: usize, rails: &mut [[u8; 12]], +) -> usize { + let mut fired = 0usize; + for wi in 0..scratch.len() { + let mut new_bits = scratch[wi] & !state[wi]; + while new_bits != 0 { + let j = (wi << 6) | new_bits.trailing_zeros() as usize; + new_bits &= new_bits - 1; + for d in 0..dirs { + // the rail that reaches j from i along d is d; i is j's neighbour along the reverse rail + let rev = d ^ 1; // HEX pairs (0,1) (2,3) (4,5) are opposites + if let Some(i) = neighbour(j as u32, rev) { + let i = i as usize; + if bit(source, i) && bit(&elig[d], i) { + rails[i][2 * d + 1] = rails[i][2 * d + 1].saturating_add(1); + fired += 1; + } + } + } + } + } + fired +} + fn spread_step( state: &mut [u64], delta: &mut [u64], scratch: &mut [u64], res: &Resident<'_>, rails: &mut [[u8; 12]], ) -> usize { @@ -275,26 +307,15 @@ fn spread_step( // Hebbian, on SURVIVORS only: a rail fires when the cell it points at // actually became active after the chain — reverse walk from each newly // reached cell to the eligible neighbour that could have carried it. - let mut fired = 0usize; + // The predecessor is tested against the SOURCE frontier of this step, + // not the accumulated state: under `from_delta` only delta cells + // propagated, so an older active neighbour of a newly reached cell did + // not carry it and must not be credited (codex P2 on #307). `delta` is + // therefore published only after attribution — it IS the source while + // the walk runs. + let fired = hebbian_reverse_walk(scratch, state, source, elig, dirs, rails); for wi in 0..scratch.len() { - let new_bits_word = scratch[wi] & !state[wi]; - delta[wi] = new_bits_word; - let mut new_bits = new_bits_word; - while new_bits != 0 { - let j = (wi << 6) | new_bits.trailing_zeros() as usize; - new_bits &= new_bits - 1; - for d in 0..dirs { - // the rail that reaches j from i along d is d; i is j's neighbour along the reverse rail - let rev = d ^ 1; // HEX pairs (0,1) (2,3) (4,5) are opposites - if let Some(i) = neighbour(j as u32, rev) { - let i = i as usize; - if bit(state, i) && bit(&elig[d], i) { - rails[i][2 * d + 1] = rails[i][2 * d + 1].saturating_add(1); - fired += 1; - } - } - } - } + delta[wi] = scratch[wi] & !state[wi]; } state.copy_from_slice(scratch); fired @@ -336,6 +357,11 @@ fn spread_step_shift( // carry can arrive from outside the span, and a carry LEAVING the span // would be removed by the `& tile` below anyway. let (lo, hi) = word_range.unwrap_or((0, scratch.len())); + // `mask_shift_morton` treats the slice AS the field (no carry across the + // span boundary), which is only the restriction of the full-field shift + // when the span is a Morton-aligned node: its origin is a multiple of + // its own length. Enforced where it is known, not assumed. + assert_eq!(lo % (hi - lo), 0, "word_range {lo}..{hi} is not a Morton-aligned node span"); let src = &source[lo..hi]; let masked = &mut masked[lo..hi]; let diag_scratch = &mut diag_scratch[lo..hi]; @@ -374,26 +400,11 @@ fn spread_step_shift( let g2 = &gates[(k + 1) % gates.len()]; mask_ternlog_assign::(scratch, g1, g2); } - // Hebbian reverse walk — stays as is, verbatim from `spread_step`. - let mut fired = 0usize; + // Hebbian reverse walk — the same attribution as `spread_step`, against + // the step's SOURCE frontier; `delta` is published after it. + let fired = hebbian_reverse_walk(scratch, state, source, elig, dirs, rails); for wi in 0..scratch.len() { - let new_bits_word = scratch[wi] & !state[wi]; - delta[wi] = new_bits_word; - let mut new_bits = new_bits_word; - while new_bits != 0 { - let j = (wi << 6) | new_bits.trailing_zeros() as usize; - new_bits &= new_bits - 1; - for d in 0..dirs { - let rev = d ^ 1; - if let Some(i) = neighbour(j as u32, rev) { - let i = i as usize; - if bit(state, i) && bit(&elig[d], i) { - rails[i][2 * d + 1] = rails[i][2 * d + 1].saturating_add(1); - fired += 1; - } - } - } - } + delta[wi] = scratch[wi] & !state[wi]; } state.copy_from_slice(scratch); fired @@ -622,7 +633,9 @@ fn main() { let mut fired_total = 0usize; let mut ok = true; // repeat the whole 24-step run to a 50 ms floor; the reset is an - // 8 KiB copy and is inside the timed region (stated). + // reset is two 8 KiB copies PLUS the 786 KiB `rails_run` copy + // (65 536 × 12 B), all inside the timed region (stated): it + // amortizes to ~1 µs per step and sits inside every arm's `n`. let mut reps = 1usize; let (el, heap) = loop { count_on(); diff --git a/src/simd_avx2.rs b/src/simd_avx2.rs index 1ffc8749..1d3d2f4a 100644 --- a/src/simd_avx2.rs +++ b/src/simd_avx2.rs @@ -1682,6 +1682,7 @@ impl Shl for U64x8 { type Output = Self; #[inline(always)] fn shl(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); let (lo, hi) = self.avx2_halves(); let (clo, chi) = rhs.avx2_halves(); // SAFETY: same obligation as `avx2_halves` — this is the x86-64-v3 @@ -1695,6 +1696,7 @@ impl Shr for U64x8 { type Output = Self; #[inline(always)] fn shr(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); let (lo, hi) = self.avx2_halves(); let (clo, chi) = rhs.avx2_halves(); // SAFETY: as for `shl` — AVX2 present by the arm's contract; diff --git a/src/simd_avx512.rs b/src/simd_avx512.rs index 9cc0ef69..83cdcdb0 100644 --- a/src/simd_avx512.rs +++ b/src/simd_avx512.rs @@ -1847,6 +1847,7 @@ impl Shr for U64x8 { type Output = Self; #[inline(always)] fn shr(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); Self(unsafe { _mm512_srlv_epi64(self.0, rhs.0) }) } } @@ -1855,6 +1856,7 @@ impl Shl for U64x8 { type Output = Self; #[inline(always)] fn shl(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); Self(unsafe { _mm512_sllv_epi64(self.0, rhs.0) }) } } diff --git a/src/simd_masking_ops.rs b/src/simd_masking_ops.rs index 7dcc02d0..5a29202a 100644 --- a/src/simd_masking_ops.rs +++ b/src/simd_masking_ops.rs @@ -67,6 +67,13 @@ // in-range elements, which makes the guarantee structural rather than a // tail-handling special case that could be forgotten. // +// ONE exception, named here so the guarantee stays checkable: the lattice +// shift `mask_shift_morton` is an ACCUMULATOR (`dst |= ...`) and is +// word-indexed, not element-indexed — it has no `values.len()` and no +// trailing-bit law, so the bit-order paragraph above does not apply to it. +// A caller that wants a fresh answer zeroes `dst` first. Every other writer +// in this file obeys the paragraph above. +// // ## Why free functions here, not methods on a wrapper // // The W1a consumer contract's "struct method, not free function" litmus @@ -1484,18 +1491,27 @@ pub fn ternary_match_strided_to_mask( // ── Gated predicates: `*_to_mask_under` (mask-risc `Pred { under }`) ──────── // // `out[w] = under[w] & pred(values)[w]`, with the predicate EVALUATED only -// on words where `under[w] != 0` — the survivor-word skip. Cost is -// proportional to the live words of `under`, never to the row count: a -// frontier that has narrowed to 3 % of the plane pays 3 % of the compares. -// The alternative — a full `*_to_mask` pass followed by `mask_and` — costs -// the whole plane twice and is exactly what a mask program must not do -// once its first predicate has already narrowed the population. +// on words where `under[w] != 0` — the survivor-word skip. The COMPARE cost +// is proportional to the live words of `under`; the per-word gate test and +// zero store remain proportional to `rows / 64`. The skip is word-granular, +// so what a frontier pays is its share of live WORDS, not of live rows: a +// frontier whose survivors occupy 3 % of the gate words pays 3 % of the +// compares, and a sparse frontier spread across every word pays all of +// them. (Measured: the counting closure in `under_skips_compares_on_empty_ +// gate_words` sees 20 of 40 groups under an alternating gate — a call +// count, not a timing.) The alternative — a full `*_to_mask` pass followed +// by `mask_and` — costs the whole plane twice and is exactly what a mask +// program must not do once its first predicate has already narrowed the +// population. // -// Word granularity (64 rows) is the finest a packed compare can skip at: -// one word is four 16-lane groups (or eight 8-lane groups), and the skip -// decision is one `u64 != 0` test on a word the caller already holds. An -// executor is free to skip COARSER (mask-risc's IR speaks of 1024-row -// chunks); the result is identical, only the cost differs. +// This engine skips at word granularity (64 rows), the unit the caller +// already holds: one word is four 16-lane groups (or eight 8-lane groups), +// and the skip decision is one `u64 != 0` test. A per-group skip (16 rows) +// would be reachable with one extra AND per group and is deliberately not +// done. An executor is free to skip COARSER (mask-risc's IR speaks of +// 1024-row chunks); the result is identical by construction — a skipped +// chunk is an all-zero gate, and `pred & 0 == 0` — though no test performs +// a coarser skip. Only the cost differs. // // Semantics are otherwise those of the ungated sibling, and the tail law is // inherited from it: the predicate's bits past `n` are zero, so @@ -1540,7 +1556,11 @@ fn pack_under( } else { pad_tail(group) }; - bits |= group_bits(lanes, live) << shift; + let g = group_bits(lanes, live); + // The closure contract (bits above `live` are zero) is what keeps + // one group from corrupting the next group's bit range. + debug_assert!(live == 64 || g >> live == 0, "{name}: group bits above live={live} must be zero"); + bits |= g << shift; start += L; shift += L; } @@ -2037,7 +2057,8 @@ pub fn blend_i32(mask_words: &[u64], a: &[i32], b: &[i32], dst: &mut [i32]) { } // ──────────────────────────────────────────────────────────────────────── -// Morton hex neighbour shift — the D-GTM-1m word-level op (§15) +// Morton lattice axis shift — the D-GTM-1m word-level op (§15); the hex +// probe composes its six rails from these four Cartesian axis moves // ──────────────────────────────────────────────────────────────────────── // // Rows are Morton-keyed 2-D (`q` on the even address bits, `r` on the odd @@ -2059,13 +2080,15 @@ pub fn blend_i32(mask_words: &[u64], a: &[i32], b: &[i32], dst: &mut [i32]) { /// direction at the call site, not at compile time). #[derive(Copy, Clone, Debug, PartialEq, Eq)] pub enum MortonDir { - /// Increment the local `q` sub-coordinate (`HEX[0] = (+1, 0)`). + /// Increment the even-bit (`q`) sub-coordinate — a lattice axis move; + /// the hex probe's rail 0 (`(+1, 0)`). The two hex diagonals are + /// compositions of two variants, never variants themselves. PosQ, - /// Decrement the local `q` sub-coordinate (`HEX[1] = (-1, 0)`). + /// Decrement the even-bit (`q`) sub-coordinate (the hex probe's rail 1). NegQ, - /// Increment the local `r` sub-coordinate (`HEX[2] = (0, +1)`). + /// Increment the odd-bit (`r`) sub-coordinate (the hex probe's rail 2). PosR, - /// Decrement the local `r` sub-coordinate (`HEX[3] = (0, -1)`). + /// Decrement the odd-bit (`r`) sub-coordinate (the hex probe's rail 3). NegR, } @@ -2211,6 +2234,18 @@ fn dec_word(w: usize, axis_bits: u64, other_bits: u64) -> Option { /// first axis, then shift that scratch (not `src`) along the second axis /// into `dst`. /// +/// **The slice IS the field.** A sub-span of a larger field is treated as +/// its own field: no carry enters or leaves the span, so the result is NOT +/// the restriction of the full-field shift — at every span boundary the +/// full field would carry into the adjacent block and the span version +/// emits nothing. This op receives only a slice and cannot check where it +/// sits in a parent field. A caller shifting a tile in place must +/// independently hold BOTH that (a) the span's origin is aligned to its own +/// length in the parent field's Morton order (a trie node: `lo % len == 0`) +/// and (b) the source is confined to the span, or accept that a boundary +/// carry is dropped. `examples/hex_tenant_mq_probe.rs` asserts (a) at its +/// call site and holds (b) by construction (`state ⊆ tile`). +/// /// # Panics /// /// Panics if `src.len() != dst.len()`, or if that length is not `4^k` for @@ -3598,7 +3633,9 @@ mod tests { #[test] fn morton_shift_matches_the_per_bit_oracle_on_random_fields() { let mut seed = 0x1357_2468_1357_2468u64; - for &n_words in &[64usize, 1024] { + // 1 and 4 words run pass 1's padded-tail branch (the whole field is + // shorter than one register); 64 and 1024 run the chunk body. + for &n_words in &[1usize, 4, 64, 1024] { let n_cells = n_words * 64; let (x_bits, y_bits) = oracle_axis_bits(n_cells); for _ in 0..8 { @@ -3614,6 +3651,38 @@ mod tests { } } + /// F6 — the documented OR-accumulate contract: `dst` is never cleared, + /// so a pre-filled `dst` keeps its prior bits and several directions + /// chain into one plane. FAILS IF pass 1 overwrites instead of + /// accumulating (`acc = splat(0)` instead of `from_array(*d)`) — every + /// other morton test starts from a zeroed `dst` and cannot see that + /// (falsifier audit on #307). + #[test] + fn morton_shift_or_accumulates_into_a_prefilled_dst() { + let mut seed = 0x0F1E_2D3C_4B5A_6978u64; + for &n_words in &[4usize, 64, 1024] { + let n_cells = n_words * 64; + let (x_bits, y_bits) = oracle_axis_bits(n_cells); + let src: Vec = (0..n_words).map(|_| splitmix64(&mut seed)).collect(); + let prior: Vec = (0..n_words).map(|_| splitmix64(&mut seed)).collect(); + // prior bits survive alongside the shifted field + let mut got = prior.clone(); + mask_shift_morton(&src, MortonDir::PosQ, &mut got); + let want_q = oracle_shift_field(&src, 1, 0, n_cells, x_bits, y_bits); + let expect: Vec = prior.iter().zip(&want_q).map(|(p, w)| p | w).collect(); + assert_eq!(got, expect, "n_words={n_words}: prior bits lost or shift missing"); + // anti-vacuity: the prior actually contributed bits the shift did not + assert!(prior.iter().zip(&want_q).any(|(p, w)| p & !w != 0), "prior added nothing — fixture too weak"); + // two directions chained into one plane + let mut chained = vec![0u64; n_words]; + mask_shift_morton(&src, MortonDir::PosQ, &mut chained); + mask_shift_morton(&src, MortonDir::NegR, &mut chained); + let want_r = oracle_shift_field(&src, 0, -1, n_cells, x_bits, y_bits); + let expect2: Vec = want_q.iter().zip(&want_r).map(|(a, b)| a | b).collect(); + assert_eq!(chained, expect2, "n_words={n_words}: chaining two directions lost one"); + } + } + /// F3 — can-fire: a lone interior bit moves to exactly one bit, at the /// oracle's address, for every direction. Can-stay-silent: a NON-trivial /// far-edge mask (every cell on `dir`'s edge, spread across many words — @@ -3913,6 +3982,28 @@ mod tests { assert_eq!(counter.get(), 40, "an all-ones gate evaluates every group"); } + /// Every public `*_to_mask_under` member is ONE `pack_under` delegation, + /// so the skip falsifier on the private engine covers all ten. Counted + /// from this file's own source: ten call sites before the test module. + /// FAILS IF a member hand-rolls its loop (baton audit P2 on #307). + #[test] + fn every_under_member_delegates_to_the_one_engine() { + let src = include_str!("simd_masking_ops.rs"); + let production = src + .split("#[cfg(test)]") + .next() + .expect("the file has a test module"); + // Count definitions, not mentions: the doc comments name the members + // many more times than the source defines them. + let members = production + .lines() + .filter(|l| l.trim_start().starts_with("pub fn ") && l.contains("_to_mask_under")) + .count(); + let delegations = production.matches("pack_under::<").count(); + assert_eq!(members, 10, "ten public `*_to_mask_under` members"); + assert_eq!(delegations, 10, "one pack_under call per member"); + } + /// A dirty phantom bit in the gate's last live word (bits at or past /// `n % 64`, which correspond to no real element) must never survive /// into the answer — element count 70 leaves word 1 with only 6 live diff --git a/src/simd_neon.rs b/src/simd_neon.rs index 43fe225d..173f975e 100644 --- a/src/simd_neon.rs +++ b/src/simd_neon.rs @@ -2939,6 +2939,7 @@ impl core::ops::Shl for U64x8 { type Output = Self; #[inline(always)] fn shl(self, r: Self) -> Self { + debug_assert!(r.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); // SAFETY: NEON baseline; pure register ops. Self(core::array::from_fn(|p| U64x2(unsafe { vshlq_u64(self.0[p].0, vreinterpretq_s64_u64(r.0[p].0)) }))) } @@ -2949,6 +2950,7 @@ impl core::ops::Shr for U64x8 { type Output = Self; #[inline(always)] fn shr(self, r: Self) -> Self { + debug_assert!(r.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); // SAFETY: NEON baseline; pure register ops. Self(core::array::from_fn(|p| { U64x2(unsafe { vshlq_u64(self.0[p].0, vnegq_s64(vreinterpretq_s64_u64(r.0[p].0))) }) diff --git a/src/simd_nightly/u_word_types.rs b/src/simd_nightly/u_word_types.rs index 29d3741a..5f8b3a7b 100644 --- a/src/simd_nightly/u_word_types.rs +++ b/src/simd_nightly/u_word_types.rs @@ -174,6 +174,7 @@ impl core::ops::Shl for U64x8 { type Output = Self; #[inline(always)] fn shl(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); Self(self.0 << rhs.0) } } @@ -182,6 +183,7 @@ impl core::ops::Shr for U64x8 { type Output = Self; #[inline(always)] fn shr(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); Self(self.0 >> rhs.0) } } diff --git a/src/simd_scalar.rs b/src/simd_scalar.rs index 49ffb450..3f1a6b35 100644 --- a/src/simd_scalar.rs +++ b/src/simd_scalar.rs @@ -1374,6 +1374,7 @@ impl Shr for U64x8 { type Output = Self; #[inline(always)] fn shr(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); let mut out = [0u64; 8]; for i in 0..8 { out[i] = self.0[i] >> rhs.0[i]; @@ -1385,6 +1386,7 @@ impl Shl for U64x8 { type Output = Self; #[inline(always)] fn shl(self, rhs: Self) -> Self { + debug_assert!(rhs.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); let mut out = [0u64; 8]; for i in 0..8 { out[i] = self.0[i] << rhs.0[i]; diff --git a/src/simd_wasm.rs b/src/simd_wasm.rs index d43cad5a..99cda4b3 100644 --- a/src/simd_wasm.rs +++ b/src/simd_wasm.rs @@ -2004,6 +2004,7 @@ pub mod wasm32_simd { type Output = Self; #[inline(always)] fn shl(self, r: Self) -> Self { + debug_assert!(r.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); let (a, n) = (self.to_array(), r.to_array()); Self::from_array(core::array::from_fn(|i| a[i].wrapping_shl(n[i] as u32))) } @@ -2013,6 +2014,7 @@ pub mod wasm32_simd { type Output = Self; #[inline(always)] fn shr(self, r: Self) -> Self { + debug_assert!(r.to_array().iter().all(|&n| n < 64), "U64x8 shift counts are a caller contract: < 64"); let (a, n) = (self.to_array(), r.to_array()); Self::from_array(core::array::from_fn(|i| a[i].wrapping_shr(n[i] as u32))) } From 3a124bbe522a80717ba5d70d7ec14c5e6c0c4d43 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 16:53:21 +0000 Subject: [PATCH 9/9] hex probe: heap gate asserts; U64x8 shift operators get doctests - `hex_tenant_mq_probe`: the header lists "hot-path heap bytes == 0" as a gate that aborts the run, but the loop only printed the count. It now asserts, so an allocation inside the timed region fails the probe instead of silently invalidating the ns/step column. Verified green on every arm in release. - `U64x8: Shl/Shr` on the AVX2 and nightly backends carry a per-lane shift doctest (the operators are public API; the hard rule wants an example, not only a unit test). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX --- examples/hex_tenant_mq_probe.rs | 6 ++++++ src/simd_avx2.rs | 12 ++++++++++++ src/simd_nightly/u_word_types.rs | 12 ++++++++++++ 3 files changed, 30 insertions(+) diff --git a/examples/hex_tenant_mq_probe.rs b/examples/hex_tenant_mq_probe.rs index ff55d4c0..fdba8527 100644 --- a/examples/hex_tenant_mq_probe.rs +++ b/examples/hex_tenant_mq_probe.rs @@ -658,6 +658,12 @@ fn main() { } let e = t.elapsed(); let heap = count_off(); + // The header names this as a gate, so it aborts like the + // other two rather than silently invalidating the ns/step. + assert_eq!( + heap, 0, + "heap gate FAILED arm={arm} dirs={dirs} x={x}: {heap} bytes allocated on the hot path" + ); if e.as_millis() >= 50 { break (e.as_nanos() as f64 / (reps * steps) as f64, heap / reps); } diff --git a/src/simd_avx2.rs b/src/simd_avx2.rs index 1d3d2f4a..e4a41a29 100644 --- a/src/simd_avx2.rs +++ b/src/simd_avx2.rs @@ -1678,6 +1678,16 @@ impl U64x8 { /// are not a portable contract — this backend zeroes the lane (`VPSLLVQ` / /// `VPSRLVQ` semantics) while the scalar backend's `<<` would overflow — so /// callers keep every count below 64, as the mask ops do. +/// +/// ``` +/// use ndarray::simd::U64x8; +/// let x = U64x8::splat(1); +/// let counts = U64x8::from_array([0, 1, 2, 3, 4, 5, 6, 63]); +/// let left = x << counts; +/// assert_eq!(left.to_array(), [1, 2, 4, 8, 16, 32, 64, 1 << 63]); +/// let right = left >> counts; +/// assert_eq!(right.to_array(), [1; 8]); +/// ``` impl Shl for U64x8 { type Output = Self; #[inline(always)] @@ -1692,6 +1702,8 @@ impl Shl for U64x8 { } } +/// Lane-wise variable right shift; see the `Shl` impl above for the +/// count contract and a worked example (the two are inverses below 64). impl Shr for U64x8 { type Output = Self; #[inline(always)] diff --git a/src/simd_nightly/u_word_types.rs b/src/simd_nightly/u_word_types.rs index 5f8b3a7b..9c8dd1c9 100644 --- a/src/simd_nightly/u_word_types.rs +++ b/src/simd_nightly/u_word_types.rs @@ -170,6 +170,16 @@ impl U64x8 { /// arm (the mask family's Morton hex shift composes `and`/`shl`/`or`). Counts /// stay below 64 by the callers' contract; what `core::simd` does at 64+ is /// its own business, not a portable promise. +/// +/// ``` +/// use ndarray::simd::U64x8; +/// let x = U64x8::splat(1); +/// let counts = U64x8::from_array([0, 1, 2, 3, 4, 5, 6, 63]); +/// let left = x << counts; +/// assert_eq!(left.to_array(), [1, 2, 4, 8, 16, 32, 64, 1 << 63]); +/// let right = left >> counts; +/// assert_eq!(right.to_array(), [1; 8]); +/// ``` impl core::ops::Shl for U64x8 { type Output = Self; #[inline(always)] @@ -179,6 +189,8 @@ impl core::ops::Shl for U64x8 { } } +/// Lane-wise variable right shift; see the `Shl` impl above for the +/// count contract and a worked example (the two are inverses below 64). impl core::ops::Shr for U64x8 { type Output = Self; #[inline(always)]