Merge pull request #317 from AdaWorldAPI/claude/great-pascal-k96kok

masking-ops: name G8 — the tree-depth column `lzcnt(bswap(x)) >> 2`; popcount is its tie-break

Author: AdaWorldAPI

.claude/blackboard.md

@@ -1,3 +1,21 @@
+## 2026-09-17 (19) — G8 named: a tree-depth column (`lzcnt(bswap(x)) >> 2`) is the missing primitive for basin-local ranking; popcount is only its tie-break
+
+Filed, not built. Full text in `masking-ops-state.md` § OUTLOOK G8 and the
+2026-09-17 EPIPHANIES entry. Two things for the next session:
+
+- **Loose end:** the gate is a named consumer call site that ranks by depth.
+  Candidates: lance-graph `FacetCascade` tail ranking (basin-local similarity)
+  and `NiblePath::common_prefix_depth` (the packed-`u64` carrier where this
+  `lzcnt` IS the fold — `ISS-NIBLEPATH-FOLD-IS-CARRIER-2-UNMASKED`). Either
+  one, once it exists as a call, licenses the build.
+- **Decision recorded:** do NOT add a `popcount` variant for this; the existing
+  `popcount_batch_u64` already serves the tie-break. The gap is `lzcnt` (+
+  `bswap`), and (8) on the lance-graph board had already named the `lzcnt`
+  half.
+
+Consistent with (18): stripping HEEL/HIP makes the operand start power-of-two,
+so the tail-padding question never arises for this op.
+
 ## 2026-09-16 (18) — the tail OPTIMISATION IS INERT ON EVERY POWER-OF-TWO POPULATION >= 512 ROWS, including the 4096-row tile
 
 Established before writing the rewrite (15)-(17) argued for, and it changes

.claude/board/EPIPHANIES.md

@@ -1,5 +1,38 @@
 # ndarray — Epiphanies (append-only)
 
+## 2026-09-17 — On a tree-path register the metric is lzcnt, popcount is the tie-break, and little-endian bytes need one bswap first
+**Status:** FINDING (the layout and the instruction facts) + NAMED GAP (G8, not built)
+**Scope:** @simd-savant @family-codec-smith domain:masking-ops domain:v3-facet
+**Cross-ref:** `.claude/knowledge/masking-ops-state.md` § OUTLOOK G8; lance-graph
+`E-THREE-CARRIERS-THREE-FOLDS-1`, `ISS-NIBLEPATH-FOLD-IS-CARRIER-2-UNMASKED`,
+LATEST_STATE 2026-09-15 (6)–(8); blackboard (18)
+
+Within a basin every V3 facet shares HEEL and HIP by construction, so the
+6-tier LCP saturates and the information is in the tail — bytes 8..16, tiers
+2–5, one aligned `u64` at a compile-time offset. Loading it is a PEEK (an
+address), not a mask (a reassembly), so register ops on it are legitimate
+under the carrier doctrine; and stripping the shared prefix is a load-offset
+choice that costs nothing. Two consequences the workspace had not written down:
+
+1. **Padding is useless once you can strip.** The tail descent / `[u64; 8]`
+   zero-padding story exists because a 128-bit facet operand is ragged in a
+   512-bit register. A 64-bit tail is not: eight tails tile a zmm exactly.
+   Blackboard (18) found the tail optimisation inert on power-of-two
+   populations *because there is no tail*; this is the same fact from the
+   operand side — choose the width that starts power-of-two.
+2. **The op on it is `lzcnt`, not `popcount`.** Each tier byte is a 4-level
+   4-ary centroid tree, so the tail is a path and its metric is depth of
+   divergence. Popcount is position-blind on a path (leaf flip == root flip).
+   Ranking = `lzcnt(bswap(a ^ b)) >> 2`; popcount belongs only to the
+   tie-break among equal-depth candidates (`x & below(depth)`) — coarse by
+   depth, fine by density.
+
+The `bswap` is load-bearing and is the trap: LE byte order puts the root tier
+at the LOW byte, but nibble order inside a byte is MSB-coarse, so neither
+`tzcnt` nor `lzcnt` alone reads the path in one direction. G8 records the
+falsifier that catches a `bswap`-less implementation. Gate before building is
+the G5 rule: one named consumer call site.
+
 ## 2026-07-29 — Which PACKAGE pulls a dep decides whether it can consume you
 **Status:** FINDING
 **Scope:** @simd-savant @truth-architect domain:build-graph

.claude/knowledge/masking-ops-state.md

@@ -200,6 +200,132 @@ unary-with-constant, and that narrowing is exactly the register in which the
 constant, record G7 as *deliberately absent* **in the IR's docs**, so a future
 session does not "fix" it.
 
+**G8 — `lzcnt_bswap_u64_to_u8` (tree-depth column) — NAMED 2026-09-17, not
+built.** The V3 facet's tail (bytes 8..16 = tiers 2–5, one aligned `u64` at a
+compile-time offset — a PEEK, not a gather) is a **tree path**. State the
+carving before the op, because "tier" and "byte" are not the same unit: the
+V3 facet is `classid(4) | payload(12)`, and `RailSpec::v3_facet`
+(`src/hpc/clam_v3.rs`) walks **six sequential `(u8:u8)` levels at key offsets
+`4..16`, stride 2** — so a tier is TWO bytes, one per axis, not one byte. The
+`u64` operand at `8..16` therefore covers tiers 2–5, i.e. **4 tiers × 2 bytes
+= 8 bytes = 16 nibbles**. Each *byte* within a tier
+is a **16-ary nibble hierarchy** — OGAR canon's *"1 hex digit = 1 nibble = 1
+level of the 16-ary tree (`FAN_OUT=16`)"* — high nibble coarse. The metric on
+a tree path is depth-of-divergence, which at that granularity on a `u64` is
+`lzcnt(a ^ b) >> 2` — position-aware.
+
+**Which factorization of 256 this op commits to, stated because OGAR carries
+two and they differ by 2×.** Beside the nibble reading above, the canon also
+says *"256 = 4⁴ — each codebook is built as a 4-level 4-ary centroid
+hierarchy"*. That one is the **Morton-interleaved** centroid space, where a
+nibble of `FacetTier::morton()` is a 2 bit × 2 bit quad-tree level across BOTH
+axes at once — a different tree, one level finer per step, over a different
+operand. G8 reads the **raw** tail `u64`, not the interleaved code, so it is
+the nibble tree and the shift is `>> 2` (depth 0..16 over 8 bytes). A 4-ary
+depth over the raw bytes would be `>> 1` (0..32). Both are monotone in
+leading-equal-bits, so **ranking is identical either way** — the granularity
+only bites where depth is consumed as a *value*, which is exactly the in-cell
+tie-break `x & below(depth)` proposed below. Do not cite 4⁴ as the
+justification for `>> 2`; it justifies `>> 1`. (Corrected 2026-09-18 after a
+sibling session measured the two against a fixture; the first draft of this
+entry asserted 4⁴ while shipping the nibble shift.) `popcount(a ^ b)` is **position-blind**
+on it (a leaf-nibble flip counts the same as a root-nibble flip; lance-graph
+LATEST_STATE 2026-09-15 (8): *"popcount also finds elephant : Wal"*), so the
+existing `popcount_batch_u64` / `xor_popcount` are the wrong primitive for
+ranking and the right one only for the in-cell tie-break (`x & below(depth)`).
+
+**The byte-order wrinkle the first implementation will get backwards:** the
+facet is little-endian (tier 2 at the LOW byte of the tail `u64`) but the
+hierarchy inside a byte is MSB-coarse. `tzcnt` gets byte order right and
+nibble order wrong; `lzcnt` the reverse. One `bswap` reconciles them:
+
+```text
+depth_nibbles = lzcnt(bswap(a ^ b)) >> 2      // 0..16, stepless, no branch
+```
+
+That is three scalar instructions (`xor`, `bswap`, `lzcnt`) or, as a column,
+`vpshufb` (byte reverse, **AVX-512BW** at zmm width) + `vplzcntq`
+(**AVX-512CD**) over 8 rows per zmm — BOTH feature bits, not CD alone; a
+target with CD but not BW needs a different byte reversal. Per 8-row group
+there is **no operand padding and no tail descent** — eight tails tile a zmm
+exactly, which is
+the whole reason the `u64` width is the right one here (and consistent with
+blackboard (18): the tail optimisation is inert on power-of-two populations
+because there is no tail; stripping HEEL/HIP makes the operand *start*
+power-of-two). **That is a claim about the OPERAND, not the population:** a
+row count not divisible by 8 still leaves a column remainder, exactly as the
+slice-level `U64x8` ops handle with `pad_tail`. No multiple-of-eight input
+contract is stated or intended, so an implementation keeps a remainder path
+and tests non-multiple-of-eight lengths. What is eliminated is the
+*within-operand* padding a narrower width would need, not the last partial
+group. Consumer: basin-local similarity in lance-graph (`FacetCascade`
+tail) and the CAKES nearest-ranking on `NiblePath` (lance-graph
+`ISS-NIBLEPATH-FOLD-IS-CARRIER-2-UNMASKED`, the packed-`u64` carrier where the
+same `lzcnt` IS the fold). Realizations: `vplzcntq` on avx512cd; avx2 has no
+vector lzcnt — polyfill via the float-exponent trick or a scalar peel (measure
+which); NEON has `clz` on 32-bit lanes (two halves + select); wasm/scalar flat;
+and the **nightly** `core::simd` arm (`src/simd_nightly/`) — six in total,
+which is the count the falsifier below means.
+Pre-registered falsifier: the column op must equal the scalar
+`(a ^ b).swap_bytes().leading_zeros() >> 2` on every row of a 64k fixture at
+all six realizations, AND a disable-run without the `bswap` must fail on a
+fixture whose divergence is in a low nibble of a high tier (that is the
+backwards-implementation trap, and the test must be able to see it).
+**Plus a third arm, because the first two cannot catch a wrong shift:** both
+compare the column op against `(a ^ b).swap_bytes().leading_zeros() >> 2`,
+which is the same formula in a different spelling, so they agree even if the
+shift is wrong. The third arm builds the fixture *independently* — plant a
+divergence at a KNOWN tier and nibble, assert the KNOWN depth — and is the
+only one that discriminates `>> 2` from `>> 1`. **Measured** (2026-09-18, all
+16 positions, against a scalar model): for tier `t` ∈ 2..5, byte-in-tier `b`
+(0 = `lo`, the LOWER address; 1 = `hi`) and nibble-in-byte `k` (0 = high/MSB,
+1 = low/LSB),
+
+```
+depth = 4*(t-2) + 2*b + k        // 0..15; 16 iff the tails are identical
+```
+
+— exact at every one of the 16 positions. (A first draft of this arm wrote
+`2*(t-2) + n`, which is right only for tier 2 and drifts by `2*(t-2)`
+thereafter; each tier is 2 bytes = **4** nibbles, not 2. It was caught by
+running the model, which is the point of the arm.)
+
+**Caveat the measurement surfaced, and the reason the consumer gate matters
+more than it looked.** `FacetTier` is `{ lo, hi }` with `lo` at the LOWER
+address, so after the `bswap` a single `lzcnt` over the raw tail walks
+`lo` BEFORE `hi` within every tier — i.e. it alternates between the two
+chains every two nibbles. But `hi_chain` and `lo_chain` are documented as two
+**orthogonal** hierarchies (`facet.rs`: *"the `hi` chain prefix-routes one
+hierarchy, the `lo` chain the orthogonal one"*), not coarse and fine of one.
+So raw-tail depth is a **mixed-axis** metric: still a valid monotone
+tie-breaker, NOT a depth in either hierarchy. A consumer that wants one
+axis needs a per-chain fold (the existing `hi_distance` / `lo_distance`) or a
+deinterleave before the `lzcnt`.
+
+**And `clam_v3.rs` already answers which carving the real bake wants — it is
+not the pair reading.** `RailSpec` carries TWO carvings, and the doc comment
+records a measurement against them: the interleaved `X:Y` pair reading
+(`v3_facet`, 6 levels, stride 2) *"fits only 44.25 % of paths"* on the
+medcare bake, while the contiguous per-axis slab (`RailSpec::slab`, 12 levels,
+stride 1) *"fits 99.62 % in twelve levels"*. That inverts the gate for G8: on
+the **slab** carving a contiguous `u64` IS one axis, the mixed-axis caveat
+above evaporates, and a plain `lzcnt(bswap(·))` is exactly right. On the
+**pair** carving it is a mixed-axis tiebreaker. So G8's consumer gate must
+name the CARVING as well as the call site, and the measured 99.62 %/44.25 %
+split says the slab is the likelier target. Do not build against the pair
+reading on the strength of it being the zero-fallback default.
+
+**Stacking, not widening, is how the levels grow.** The same module: *"a class
+that needs more than six levels does not widen a byte — it stacks a second
+register, e.g. into the edge lane, and chains it (`RailSpec::stacked`). Depth
+then runs 0..=12 over two registers, same hole rule, same arithmetic."* The
+edge lane at `16..32` is explicitly contemplated as that continuation register
+(*"`16..32` may be a continuation register (if the spec says so)"*). A G8
+column op over a stacked pair is therefore two operands chained, never one
+wider one — which is also why no `u128` variant of G8 is proposed. Gate
+before building: one named consumer call site that ranks by depth — the same
+count rule G5 carries.
+
 **The nightly arm is AHEAD of the stable arms, and it is the contract
 reference.** `src/simd_nightly/` carries **18 compare-to-mask pairs across
 every width**; the stable arms have a subset. So N2/N3 were not adding a