simd: masking layer (simd_masking_ops.rs), generated backend-local ternlog bodies, five-flavour realization

.cargo/config-v4.toml

@@ -0,0 +1,22 @@
+[build]
+# Plain AVX-512 baseline — `x86-64-v4` (F + BW + CD + DQ + VL), nothing above
+# it. Use with:
+#   cargo --config .cargo/config-v4.toml check --lib
+#   cargo --config .cargo/config-v4.toml test --lib --no-run
+#
+# This is the deterministic AVX-512 COMPILE contract the SIMD realization
+# matrix builds against: `simd.rs` keys its arm off `target_feature = "avx512f"`,
+# so this selects `simd_avx512` and emits `vpternlogq` etc. regardless of the
+# build host's silicon — code generation and execution are separate, and a
+# GitHub runner does not need AVX-512 to prove LLVM emits it. Do NOT run the
+# resulting binary on a host without AVX-512 (SIGILL); use the v3 or native
+# configs for execution, or qemu for semantic parity.
+#
+# `config-avx512.toml` is the stricter Sapphire Rapids EXECUTION config (VNNI,
+# BF16, FP16, AMX…); it SIGILLs on any earlier AVX-512 silicon, so it is never
+# the CI compile oracle.
+# `-Dwarnings` rides here because CI must `env -u RUSTFLAGS` to let this file
+# apply at all (a RUSTFLAGS env replaces every config rustflags entry), and
+# the repo rule is warnings-clean on every arm anyway.
+[target.'cfg(target_arch = "x86_64")']
+rustflags = ["-Ctarget-cpu=x86-64-v4", "-Dwarnings"]

.claude/AMX_GOTCHAS.md

@@ -267,6 +267,25 @@ in production under load, AVX-512 siblings unaffected.
 
 ---
 
+## Gotcha 15: the operand "mirror" was a misread of the byte table — use mnemonics
+
+`src/hpc/amx_ops.rs` (2026-09-14) assembles every AMX mnemonic on stable
+1.98.1 with `const` tile operands. Intel order is `tdpbusd tmmD, tmmS1, tmmS2`
+= `D += S1·S2`, S1 = ModRM.rm (plain M×K), S2 = VEX.vvvv (VNNI K×N). The
+validated `C4 E2 71 5E C2` is `tdpbusd tmm0, tmm2, tmm1`, i.e. the kernel's
+"A in tmm2, B in tmm1" placement is the plain SDM semantics, not a mirror.
+Gotcha 12's *placement* stays correct; its *explanation* is superseded.
+Aliased tile operands are now a compile error (`const` assert), so the SIGILL
+of Gotcha 11 cannot be written THROUGH `amx_ops` (the `.byte` path in
+`amx_matmul` is untouched and still lets a caller alias tiles). Encoding
+tests read each wrapper's bytes out of the test binary's ELF symtab and pin
+at least one op of every tier that has a register-only or masked encoding:
+the four GEMM-tier sequences to the EMR-validated table, everything else to
+LLVM 22.1.8's own emission (a drift guard). TF32 is hand-encoded raw bytes
+(nightly's LLVM 23 dropped the mnemonic) pinned to the bytes 22.1.8 once
+produced. The AVX512 row ops are pinned only on `avx512f` builds. NO tier
+beyond TILE/INT8/BF16 has executed on any host here.
+
 ## Hardware tiers
 
 ```

.claude/knowledge/agnostic-surface-cpu-matrix.md

@@ -85,15 +85,17 @@ two `__m256i` halves; "4×NEON" means four 128-bit NEON registers (e.g.
 | `I16x16` | `__m256i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | `__m256i`  | `__m256i`  | 2×`int16x8_t`  | ←  | ←  | `[i16;16]` |
 | `U16x32` | `__m512i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | 2×`__m256i`⏳| 2×`__m256i`⏳| 4×`uint16x8_t`  | ←  | ←  | `[u16;32]` |
 | `U16x16` | `__m256i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | `__m256i`  | `__m256i`  | 2×`uint16x8_t` | ←  | ←  | `[u16;16]` |
-| `I32x16` | `__m512i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | 2×`__m256i`| 2×`__m256i`| 4×`int32x4_t`  | ←  | ←  | `[i32;16]` |
+| `I32x16` | `__m512i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | 2×`__m256i`†| 2×`__m256i`†| 4×`int32x4_t`† | ←  | ←  | `[i32;16]` |
 | `I32x8`  | `__m256i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | `__m256i`  | `__m256i`  | 2×`int32x4_t`  | ←  | ←  | `[i32;8]`  |
 | `U32x16` | `__m512i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | 2×`__m256i`⏳| 2×`__m256i`⏳| 4×`uint32x4_t` | ←  | ←  | `[u32;16]` |
 | `U32x8`  | `__m256i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | `__m256i`⏳ | `__m256i`⏳ | 2×`uint32x4_t` | ←  | ←  | `[u32;8]`  |
 | `I64x8`  | `__m512i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | 2×`__m256i`| 2×`__m256i`| 4×`int64x2_t`  | ←  | ←  | `[i64;8]`  |
 | `I64x4`  | `__m256i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | `__m256i`  | `__m256i`  | 2×`int64x2_t`  | ←  | ←  | `[i64;4]`  |
-| `U64x8`  | `__m512i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | 2×`__m256i`| 2×`__m256i`| 4×`uint64x2_t` | ←  | ←  | `[u64;8]`  |
+| `U64x8`  | `__m512i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | 2×`__m256i`†| 2×`__m256i`†| 4×`uint64x2_t`† | ←  | ←  | `[u64;8]`  |
 | `U64x4`  | `__m256i`  | ←   | ←   | ←   | ←   | ←   | ←   | ←   | `__m256i`  | `__m256i`  | 2×`uint64x2_t` | ←  | ←  | `[u64;4]`  |
 
+† = native since 2026-09-13 (PR #306 five-flavour audit; the AVX2 column's "2×`__m256i`" is the storage SHAPE the codegen lowers to — the type stays the `#[repr(align(64))]` `[T; N]` array, measured packed for the bit-logic half and given two-half intrinsic bodies for rotate / reduce / compare-bitmask on 2026-09-14). Until then these two rows were WRONG: on aarch64 and wasm32 `simd.rs` re-exported the SCALAR `U64x8`/`I32x16`, and on the v3 arm they were `avx2_int_type!` array polyfills — the mask family (`simd_masking_ops`) rides exactly these two types, so it ran scalar on three of five flavours.
+
 ⏳ = TD-T22 polyfill audit — the 256-bit `U16x16/U16x32/U32x8/U32x16`
 inner ops may currently use scalar storage under `#[target_feature]` rather
 than real `__m256i` intrinsics. Needs verification (see § J integration plan).

.claude/knowledge/amx-enablement-and-kernel.md

@@ -169,15 +169,68 @@ which is bug #2. For the 16×16 int8/bf16 tile, all three tiles are 16 rows ×
 
 ---
 
+## 5b. The mnemonic surface — `src/hpc/amx_ops.rs` (2026-09-14, LLVM 22.1.8)
+
+The `.byte` tables above were forced by 1.94. Measured on **1.98.1 (LLVM
+22.1.8)**: the integrated assembler accepts EVERY AMX mnemonic inside `asm!`
+with no target feature, and `asm_const` makes the tile index a generic
+parameter (`tilezero tmm{t}`, `t = const T`). `amx_ops.rs` exposes the whole
+`X86InstrAMX.td` surface that way, and its tests read the emitted bytes back
+out of the text segment and pin them to this table — on any x86_64 host, no
+EMR needed. Tile-operand aliasing (Gotcha 11) is a `const` assert, so `#UD`
+is now a compile error.
+
+**The "mirror" (Gotcha 12, §4) is a reading of the byte table, not a hardware
+quirk.** `MRMSrcReg4VOp3` puts `dst` in ModRM.reg, **S1 in ModRM.rm, S2 in
+VEX.vvvv**, and Intel syntax names them in that order: `tdpbusd tmmD, tmmS1,
+tmmS2` = `D += S1(M×K, plain) · S2(K×N, VNNI)`, with the `U`/`S` letters
+naming S1 then S2. The table row `C4 E2 71 5E C2` (rm = tmm2, vvvv = tmm1)
+is the mnemonic `tdpbusd tmm0, tmm2, tmm1` — `amx_ops::tdpbusd::<0, 2, 1>` —
+which is exactly the kernel's placement (A u8 → tmm2, B VNNI i8 → tmm1). The
+row's comment "dst=tmm0,vvvv=tmm1,rm=tmm2" had been read as the operand
+list `(tmm0, tmm1, tmm2)`; the assembler reads `(tmm0, tmm1, tmm2)` as
+rm = tmm1, vvvv = tmm2 = `C4 E2 69 5E C1`. Pinned two-sided in
+`operand_order_is_intel_order_rm_then_vvvv`.
+
+Assembler-verified encodings, LLVM `Host.cpp` CPUID bits. The INT8/BF16 rows
+are listed for reference and DID execute on EMR in the `(0,2,1)` placement;
+the `(0,1,2)` placements shown here, and every row below them, have NOT
+executed in this workspace (no GNR/DMR host):
+
+```
+                                         CPUID          bytes (tmm0,tmm1,tmm2 / tmm3,[rdi+rsi])
+TDPBSSD/TDPBSUD/TDPBUSD/TDPBUUD  INT8    7.0:EDX[25]    C4 E2 {6B,6A,69,68} 5E C1
+TDPBF16PS                        BF16    7.0:EDX[22]    C4 E2 6A 5C C1
+TDPFP16PS                        FP16    7.1:EAX[21]    C4 E2 6B 5C C1
+TCMMIMFP16PS / TCMMRLFP16PS      COMPLEX 7.1:EDX[8]     C4 E2 {69,68} 6C C1
+TDPBF8PS/TDPBHF8PS/TDPHBF8PS/    FP8     1E.1:EAX[4]    C4 E5 {68,6B,6A,69} FD C1   (map5)
+  TDPHF8PS
+TMMULTF32PS                      TF32    1E.1:EAX[6]    C4 E2 69 48 C1   (dropped from LLVM main; 22.1.8 assembles the mnemonic, nightly LLVM 23 does not → emitted as raw bytes)
+TILELOADDRS / TILELOADDRST1      MOVRS   1E.1:EAX[8]    C4 E2 {7B,79} 4A 1C 37
+TCVTROWD2PS zmm0,tmm1,edi / ,3   AVX512  1E.1:EAX[7]    62 F2 46 48 4A C1 / 62 F3 7E 48 07 C1 03   (EVEX; needs avx512f cfg)
+TCVTROWPS2{PHH,PHL,BF16H,BF16L}  AVX512                 62 F2 {44,46,47,45} 48 6D C1
+TILEMOVROW zmm0,tmm1,edi / ,5    AVX512                 62 F2 45 48 4A C1 / 62 F3 7D 48 07 C1 05
+STTILECFG [rdi] / TILELOADDT1    TILE    7.0:EDX[24]    C4 E2 79 49 07 / C4 E2 79 4B 14 16
+```
+
+Detection: `amx_ops::amx_features()` (cached) returns the per-tier bits.
+The execute gate for tile STATE is `simd_amx::amx_tile_available()` (TILE +
+XCR0 + arch_prctl); `amx_available()` is that plus the INT8 bit and gates the
+INT8 ops only; every other tier gates on `amx_tile_available()` AND its
+`AmxFeatures` bit. `amx_report()` prints both gates and every tier bit.
+Gotcha 14 (VM tile-state corruption) applies to every tier.
+
 ## 6. Detection API (cached, CPU-aware)
 
 ```rust
-use ndarray::simd::{amx_available, cpu_model, amx_report, CpuModel};
+use ndarray::simd::{amx_available, amx_tile_available, amx_features, cpu_model, amx_report, CpuModel};
 
-amx_available()  // bool, cached once via LazyLock (the 4 gates of §1)
+amx_available()       // bool, cached once via LazyLock (the 4 gates of §1, INT8 bit last)
+amx_tile_available()  // the tier-agnostic tile gate (TILE + OSXSAVE + XCR0 + arch_prctl)
+amx_features()        // AmxFeatures — per-tier silicon bits (§5b)
 cpu_model()      // CpuModel::{SapphireRapids,EmeraldRapids,GraniteRapids,SierraForest,OtherX86,NonX86}
 cpu_model().has_amx()   // true for SPR/EMR/GNR; false for Sierra Forest (E-core)
-amx_report()     // e.g. "AMX [Emerald Rapids expects_amx=true]: TILE=true INT8=true BF16=true available=true"
+amx_report()     // e.g. "AMX [Emerald Rapids expects_amx=true]: TILE=true INT8=true BF16=true tile_available=true available=true | tiers: fp16=false complex=false fp8=false tf32=false avx512=false movrs=false"
 ```
 
 Why `LazyLock`: the four gates (CPUID, XGETBV, one `arch_prctl`) are all

.claude/knowledge/simd-codegen-oracle/README.md

@@ -82,6 +82,28 @@ Measured on x86_64 + `x86-64-v3`, rustc 1.95.0. Full narrative in
 | `gather_lookup_u8` | 0 | 0 | `movzbl` chain, no arithmetic |
 | `serial_dependent_chain` | 0 | 27 | loop-carried dependency |
 
+Group F — the mask family (PR #306), measured 2026-09-14 on rustc 1.98.1
+through the shipped library methods. Two runs: array polyfill first, then
+after the four non-packed shapes were given backend-local AVX2 intrinsic
+realizations. The bit-logic half needed nothing.
+
+| probe | polyfill run | after override | lowering |
+|---|---|---|---|
+| `ternlog_u64x8_maj3` / `_select` | 18 / 0 | unchanged | generated Shannon ladder → `vpand`/`vpandn`/`vpor`/`vpxor` |
+| `ternlog_u32x16_xor_and` | 8 / 0 | unchanged | same, 32-bit lanes |
+| `andnot_u64x8` | 6 / 0 | unchanged | `vpandn` |
+| `popcnt_u64x8` | 21 / 0 | unchanged | `vpshufb` nibble-LUT popcount, not 8× `popcntq` |
+| `xor_popcount_u64x8` | 25 / 0 | unchanged | `vpxor` + nibble popcount + add tree |
+| **`rotate_left_lib_u64x8`** | **0 / 8** `rolq` | 10 / 2 (count setup) | `vpsllq`+`vpsrlq`+`vpor` per half — the earned u64 override |
+| **`gt_bitmask_i32x16`** | **23 / 3** mixed (lanes 0, 13–15 peeled) | 9 / 0 | `vpcmpgtd` + movemask |
+| **`cmpge_zero_mask_i32x16`** | **17 / 11** mixed | 10 / 0 | complemented sign-bit movemask |
+| **`reduce_max_i32x16`** | **0 / 17** `cmpl` | 8 / 0 | `vpmaxsd` tree |
+
+The two "mixed" rows are the instructive ones: a shape can be *mostly*
+packed and still carry a scalar peel, and the method's doc comment had
+claimed a clean lowering it never had. Measure the shipped symbol, not the
+look-alike.
+
 **The headline:** LLVM vectorizes far more than intuition suggests —
 including cross-lane permutes, widening converts, and saturating
 arithmetic, all from plain scalar loops. It does **not** vectorize u64

.claude/knowledge/simd-codegen-oracle/baseline-x86_64-v3.toml

@@ -184,9 +184,91 @@ note = "MEASURED FULLY PACKED. Observed 81 packed / 0 scalar-lane-arith / 0 loop
 
 [probe.shiftor_rot_u64x8]
 expect = "unknown"
-note = "MEASURED SCALAR -- 0 packed / 8 scalar-lane-arith / 19 memory, i.e. indistinguishable from rot_u64x8. Writing the rotate as an explicit (x >> n) | (x << (64-n)) does NOT persuade LLVM to vectorize it, so the refusal is about the 64-bit OPERATION and not about the rotate idiom. Driver asserts equality with u64::rotate_right, so this is the same function in a different spelling."
+note = "MEASURED SCALAR on 2026-07 (0 packed / 8 scalar-lane-arith / 19 memory, indistinguishable from rot_u64x8): writing the rotate as an explicit (x >> n) | (x << (64-n)) did NOT persuade LLVM to vectorize it, so the refusal is about the 64-bit OPERATION and not about the rotate idiom. Driver asserts equality with u64::rotate_right, so this is the same function in a different spelling. OBSERVATION 2026-09-14, recorded rather than explained: in the run where U64x8::rotate_left/right became a vpsllq/vpsrlq/vpor intrinsic realization in simd_avx2.rs, THIS probe -- untouched -- read 10 packed / 2 scalar-lane-arith, byte-for-byte the library method's shape, while rot_u64x8 and shiftor_rot_const_u64x8 stayed 0 packed. The probe calls no library rotate; the mechanism (LLVM recognising the shift-or as the rotate operation now that a vector-typed lowering of that operation exists in the module and folding, or something else) is NOT established. Kept `unknown`; a probe whose reading moves without its source changing is a finding about the instrument, not the lane."
 
 [probe.shiftor_rot_const_u64x8]
 expect = "unknown"
 note = "MEASURED SCALAR -- 0 packed / 8 scalar-lane-arith / 8 memory, with BLAKE2b's COMPILE-TIME-CONSTANT 32/24/16/63. This is the sharper half: at u32 width a constant byte-granular amount folds to vpshufb and a constant bit-granular one folds to shift-or, both packed. At u64 width neither happens, for either kind of constant. A third independent confirmation that the u64 lane is the crate's one earned intrinsic override."
 
+
+# ============================================================================
+# Group F -- the MASK FAMILY (PR #306): the exact shapes `simd_masking_ops`
+# calls on the x86-64-v3 (AVX2) backend, each through the LIBRARY method so
+# the measurement is of shipped code. Rung 3 of the acceptance ladder, x86
+# arm. Measured 2026-09-14, rustc 1.98.1, in TWO runs: first on the pure
+# array polyfill, then after the four shapes that did not lower packed were
+# given backend-local intrinsic realizations in simd_avx2.rs.
+#
+# FIRST-RUN RESULT (array polyfill, no intrinsics): ternlog (both u64 tables
+# and the u32 XOR_AND), andnot, popcnt and xor_popcount were ALREADY fully
+# packed with 0 scalar lane arithmetic -- the oracle's standing finding
+# ("scalar source, packed codegen") holds for the whole bit-logic half of the
+# mask family, so those earned NO rewrite. Four shapes did not: rotate_left
+# (0 packed / 8 rolq -- the known u64 rotate refusal, now confirmed on the
+# shipped method), reduce_max (0 packed / 17 cmpl -- `iter().max()` is a
+# scalar chain), and the two compare-to-bitmask forms, which were MIXED:
+# LLVM vectorized lanes 1..=12 but peeled lanes 0 and 13..=15 into scalar
+# compares plus shll/orl bit assembly (cmpge_zero_mask 17 packed / 11 scalar;
+# gt_bitmask 23 / 3). Those four are the earned overrides; `expect` below is
+# the SECOND-run state and is what the gate now holds.
+# ============================================================================
+
+[probe.ternlog_u64x8_maj3]
+expect = "vectorized"
+min_packed = 10
+max_scalar_lane_arith = 0
+note = "ARRAY POLYFILL, unchanged. Observed 18 packed / 0 scalar-lane-arith / 0 loop-control on both runs: the generated Shannon ladder (t0 != t1, general arm) lowers to vpand/vpandn/vpor/vpxor over both ymm halves. No intrinsic override earned."
+
+[probe.ternlog_u64x8_select]
+expect = "vectorized"
+min_packed = 10
+max_scalar_lane_arith = 0
+note = "ARRAY POLYFILL, unchanged. 0xCA (c ? a : b) -- the (T0 & !c) | (T1 & c) arm. Observed 18 packed / 0 scalar-lane-arith, identical shape to MAJ3."
+
+[probe.ternlog_u32x16_xor_and]
+expect = "vectorized"
+min_packed = 4
+max_scalar_lane_arith = 0
+note = "ARRAY POLYFILL, unchanged. The exact immediate simd_masking_ops uses. Observed 8 packed / 0 scalar-lane-arith."
+
+[probe.andnot_u64x8]
+expect = "vectorized"
+min_packed = 3
+max_scalar_lane_arith = 0
+note = "ARRAY POLYFILL, unchanged. `self & !other` folds to vpandn per half. Observed 6 packed / 0 scalar-lane-arith."
+
+[probe.popcnt_u64x8]
+expect = "vectorized"
+min_packed = 12
+max_scalar_lane_arith = 0
+note = "ARRAY POLYFILL, unchanged -- and the surprise of the group: eight `count_ones()` in a lane loop did NOT become eight scalar popcntq; LLVM emitted the vpshufb nibble-LUT popcount over both ymm halves (21 packed / 0 scalar-lane-arith). No override earned."
+
+[probe.xor_popcount_u64x8]
+expect = "vectorized"
+min_packed = 15
+max_scalar_lane_arith = 0
+note = "ARRAY POLYFILL, unchanged. vpxor + the same vpshufb popcount + a horizontal add tree. Observed 25 packed / 0 scalar-lane-arith."
+
+[probe.rotate_left_lib_u64x8]
+expect = "vectorized"
+min_packed = 6
+max_scalar_lane_arith = 2
+note = "INTRINSIC REALIZATION EARNED. First run (array polyfill): 0 packed / 8 rolq %cl -- the shipped method scalarised exactly like the hand-written rot_u64x8 mirror. Second run (vpsllq + vpsrlq + vpor per 256-bit half, uniform xmm count): 10 packed / 2 scalar-lane-arith / 1 loop-control. The 2 scalar ops are `andl $63` and `subl` on the COUNT (n % 64, 64 - n), not on lane data; the branch is the n == 0 early return. The u32 lane got this lowering for free; the u64 lane had to be told."
+
+[probe.gt_bitmask_i32x16]
+expect = "vectorized"
+min_packed = 5
+max_scalar_lane_arith = 0
+note = "INTRINSIC REALIZATION EARNED. First run (index loop): 23 packed / 3 scalar-lane-arith / 5 loop-control -- MIXED: LLVM vectorized lanes 1..=12 (a ymm at offset 4 and an xmm at offset 36) and peeled lanes 0 and 13..=15 into scalar cmpl + shll/orl bit assembly. The doc comment on the method had claimed a clean packed lowering; the measurement did not bear it out. Second run (vpcmpgtd per half + vmovmskps): 9 packed / 0 scalar-lane-arith / 0 loop-control -- LLVM further folded the two movemasks into one vpackssdw/vpacksswb/vpmovmskb."
+
+[probe.cmpge_zero_mask_i32x16]
+expect = "vectorized"
+min_packed = 6
+max_scalar_lane_arith = 0
+note = "INTRINSIC REALIZATION EARNED. First run (index loop): 17 packed / 11 scalar-lane-arith / 4 loop-control -- the same lane-0 + lanes-13..=15 peel as gt_bitmask. Second run (complemented vmovmskps sign bits per half): 10 packed / 0 scalar-lane-arith / 0 loop-control (vpcmpgtd against all-ones + pack + vpmovmskb)."
+
+[probe.reduce_max_i32x16]
+expect = "vectorized"
+min_packed = 5
+max_scalar_lane_arith = 0
+note = "INTRINSIC REALIZATION EARNED. First run (`iter().max()`): 0 packed / 17 cmpl -- a scalar compare chain, nothing packed. Second run (vpmaxsd tree 16 -> 8 -> 4 -> 2 -> 1): 8 packed / 0 scalar-lane-arith. reduce_min is the vpminsd twin, not separately probed (same shape, same ladder)."