diff --git a/.claude/knowledge/masking-ops-state.md b/.claude/knowledge/masking-ops-state.md index bca884e0..7f7573c2 100644 --- a/.claude/knowledge/masking-ops-state.md +++ b/.claude/knowledge/masking-ops-state.md @@ -51,8 +51,8 @@ This file tracks their state. | gap | what | state | |---|---|---| | **G6** | `mask_set_range(dst, lo, hi)` | **SHIPPED** (`33716b9`, fills vectorized `347875e`). Two consumers were working around its absence. `word_range_mask` built from two bits-below-N masks so `hi == 64` never computes `1u64 << 64`. | -| **G1** | `{eq,ne,gt,ge,lt,le}_u8_to_mask` | **SHIPPED** (`a8e7d7d`). The width where **packing is free**: 64 lanes, 64-bit word, one chunk is one whole word, no shift. **Falsifier ANSWERED — see below.** | -| **G2** | `{eq,ne,gt,ge,lt,le}_u64_to_mask` + `U64x8::{cmpeq_mask,cmpgt_mask}` on all six realizations | **SHIPPED** (`e05afbd`). Packing is NOT free here — eight groups share a word. Only avx512 (`epu64`) and NEON (`cmhi`) have the instruction; wasm has **no** unsigned ordered 64-bit compare and uses the sign-bias trick; scalar/avx2 are flat polyfills. **Measurement OPEN.** | +| **G1** | `{eq,ne,gt,ge,lt,le}_u8_to_mask` | **SHIPPED** (`a8e7d7d`). The width where **packing is free**: 64 lanes, 64-bit word, one chunk is one whole word, no shift. **Both falsifiers ANSWERED** — see below, and § "The PENDING half is now RUN". | +| **G2** | `{eq,ne,gt,ge,lt,le}_u64_to_mask` + `U64x8::{cmpeq_mask,cmpgt_mask}` on all six realizations | **SHIPPED** (`e05afbd`). Packing is NOT free here — eight groups share a word. Only avx512 (`epu64`) and NEON (`cmhi`) have the instruction; wasm has **no** unsigned ordered 64-bit compare and uses the sign-bias trick; scalar/avx2 are flat polyfills. **MEASURED 2026-09-16 — and it is TIER-DEPENDENT.** See § "The PENDING half is now RUN". | | **G4** | `mask_shift_morton` | **SHIPPED** (`255c36d`) — **and its own falsifier FIRED.** See OUTLOOK. | Also shipped alongside, not a numbered gap: `simd_avx2`'s `U8x64::{cmpeq_mask, @@ -78,25 +78,82 @@ asymmetric: the i32 arms were accidentally protected by a later read, the u8 arms had none. Corrected in `daed0fd`/`95ac06f`. **The conclusion survived, the number did not** — which is the reason to state both. -## PENDING — the measurement half, and it is not optional - -**Existing code is not a moved measurement.** Two pre-registered falsifiers are -still unrun, and both run through the same probe: - -- **G1's second half** — PR #308's `r2il_column_scan_probe` crossover. -- **G2's only half** — its `find_ram_in_range` re-expressed against the real - `u64` family instead of the hi32/lo32 bucket split it used to dodge the - missing primitive. - -**Blocked, not skipped.** That probe needs a column dump from `r2sleigh-lift`'s -`win32_census`, which needs a **Win32 PE binary**; none exists in this -container. A synthetic dump would produce a number shaped like the answer -without being it, against that probe's own *"a real lift rather than a -synthetic stream"*. - -**G1's answer does not transfer to G2.** Part of G1's win is the packing -vanishing — a property of u8 alone. At u64 eight groups share a word and each -needs a shift, so the u64 family starts from a structurally weaker position. +## The PENDING half is now RUN (2026-09-16) — both falsifiers answered + +⊘ **The blocker recorded here was STALE, and that is the first finding.** This +section read *"that probe needs a column dump from `r2sleigh-lift`'s +`win32_census`, which needs a Win32 PE binary; none exists in this container."* +One does: `r2sleigh/probes/win32-census/legacy_app.exe`, 130 613 bytes, +committed since 2026-08-27. No synthetic dump was needed. The lesson is the +workspace's own: **a report of absence is not evidence of absence** — the same +shape as the `neon-qemu` "missing target" that was a missing linker. + +Run against the REAL lift — 12 408 p-code ops, `r2il_column_scan_probe`, both +tiers, `black_box` on every arm's inputs and outputs, 3 runs. + +### G1's second half — ANSWERED, decisively + +| tier | scalar | widened `eq_u32` | NATIVE `eq_u8` | +|---|---:|---:|---:| +| v4 / AVX-512 | 1× | 3.96× | **34.4×** | +| v3 / AVX2 | 1× | 1.84× | **26.1×** | + +The native primitive beats the widening workaround it replaced by **8.7× (v4) +/ 14× (v3)**. Do not compare the 26-34× to G1's own 6.91×: that pair is +u8-vectorized vs i32-vectorized; this one is scalar vs vectorized, where the +lane count (32-64 B/instruction vs a 1-byte loop) is the expected answer. +Bandwidth corroborates — 52 GB/s L1-resident vs 2 GB/s scalar. + +### G2 — ANSWERED, and the answer is TIER-DEPENDENT + +The same `find_ram_in_range` native arm (`eq_u8` + `ge_u64` + `lt_u64` + +`AND3`), 12 408 ops, against the scalar baseline: + +| tier | S ns/op | NATIVE ns/op | S/NATIVE | +|---|---:|---:|---:| +| v4 / AVX-512 | 0.73 | **0.48** | **1.51× WIN** | +| v3 / AVX2 | 0.72 | **1.31** | **0.55× LOSS** | + +**This file predicted the direction and the mechanism** — *"only avx512 +(`epu64`) and NEON (`cmhi`) have the instruction; scalar/avx2 are flat +polyfills"* — and the measurement puts a number on it: on v3 the native path +is a scalar loop wearing a vector signature, and it is **worse than the scalar +baseline**. Consequence for consumers: on an AVX2 baseline keep the hi32/lo32 +split; on v4 the native spelling is both faster and general (no bucket +assumption). + +### The u8 widening is a 4× tax; the u64 "widening" is a DISCOUNT + +PR #308 blamed its Q2 crossover on widened columns. Counted per arm, the +widened path is not merely no worse — it reads **less**: + +| arm | reads per logical row | total | +|---|---|---:| +| `NATIVE` | `space` 1 B + `offset` 8 B (`ge`) + `offset` 8 B (`lt`) | **17 B** | +| `AND` / `TERN` | `space32` 4 B + `hi32` 4 B + `lo32` 4 B + `lo32` 4 B | **16 B** | + +Offset-derived alone: **16 B native vs 12 B split**, a 25 % saving, because +each split predicate touches only the half it needs (`hi32` once, `lo32` twice) +while every native `u64` predicate pulls all eight bytes. The split therefore +halves the element width AND cuts traffic — which is why the widened arms beat +the native one on v3 and match it on v4. The `u8` case has the opposite sign: a +real 4× tax with no compensating structure. + +⊘ **Corrected 2026-09-16 (coderabbit, ndarray #310).** The first version of +this section claimed `hi32`+`lo32` came to "also 16 B/op" and concluded the Q2 +arms paid no tax at all. `hi32` is read ONCE, not twice — 12 B, not 16. The +direction is unchanged and rests on the measured timings and the native path's +generality; the equal-traffic mechanism offered for it was wrong. PR #308's +crossover number survives; both its stated mechanism and this file's first +correction of it do not. + +### What did NOT change + +Every arm still degrades to ≤ 1.0× above ~200 K ops, on both tiers. And the +consumer verdict is unmoved: a whole-census `find_ram_in_range` is ~9.0 µs +scalar vs ~5.9 µs native-on-v4 — **3 µs** on a binary whose SLEIGH lift costs +milliseconds. The primitive surface changed; *a word-level op pays for the span +it is given* did not. Also pending, and each is a deliberate non-build rather than an oversight: diff --git a/examples/r2il_column_scan_probe.rs b/examples/r2il_column_scan_probe.rs index f99e7889..0d4d2a8a 100644 --- a/examples/r2il_column_scan_probe.rs +++ b/examples/r2il_column_scan_probe.rs @@ -22,7 +22,12 @@ //! `space == Ram && lo <= offset < hi`. Four expressible predicates once the //! u64 offset is split (see below), so it is the conjunction question. //! -//! # Two primitive gaps this probe had to work around, and they are findings +//! # Two primitive gaps this probe reported — both now CLOSED by ndarray #309 +//! +//! Stated as the FIRST run found them, because the widened arms below still +//! exist and still need their rationale. What changed: `{eq,…}_u8_to_mask` and +//! `{ge,lt,…}_u64_to_mask` now ship, so the `NATIVE` arm spells the query +//! directly and the two workarounds are kept only as the comparison. //! //! 1. **No `u8` comparator.** `OpColumns::{tag,space}` are `Vec`; the //! facade's narrowest value type is `u32`. A consumer must keep a widened @@ -36,69 +41,120 @@ //! re-expressed EXACTLY here by splitting the offset into `hi32`/`lo32` //! columns — valid only because the chosen window lies inside one `hi32` //! bucket, which the probe asserts rather than assumes. A general range -//! needs the primitive. +//! needed the primitive — `ge_u64_to_mask` / `lt_u64_to_mask` are it, and +//! the `NATIVE` arm uses them with no split and no bucket assumption. //! //! # Arms (every arm must produce a bit-identical mask, or the run aborts) //! //! | arm | how the conjunction is formed | passes | //! |---|---|---| -//! | `S` | scalar over the NATIVE `u8`/`u64` columns (no widening at all) | 1 | -//! | `AND` | 4 `*_to_mask` + 3 `mask_and_assign` | 7 | -//! | `TERN` | 4 `*_to_mask` + `mask_ternlog::` + `mask_and_assign` | 6 | -//! | `UNDER` | the `_under` chain — each predicate narrows the live mask | 4 | +//! | `S` | scalar over the columns' own `u8`/`u64` types (no widening at all) | 1 | +//! | `AND` | 4 `*_to_mask` over WIDENED columns + 3 `mask_and_assign` | 7 | +//! | `TERN` | 4 `*_to_mask` over WIDENED columns + `mask_ternlog::` + `mask_and_assign` | 6 | +//! | `UNDER` | the `_under` chain over WIDENED columns — each predicate narrows the live mask | 4 | +//! | `NATIVE` | 3 `*_to_mask` over the columns' OWN types (`eq_u8` + `ge_u64` + `lt_u64`) + one `mask_ternlog::` | 4 | +//! +//! `NATIVE` is the arm ndarray #309 made possible: three predicates instead of +//! four (the `hi32 == 1` term disappears with the split), zero widened columns, +//! and no assumption that the window lies inside one `hi32` bucket. `S` is the +//! honest baseline precisely because it too needs no widened columns: charging +//! the widened arms for the layout they require is the comparison a consumer +//! actually faces. +//! +//! # Measured — and the two gaps this probe reported are now CLOSED +//! +//! The first run of this probe (2026-09-14) reported two missing primitives: +//! no `u8` comparator and no `u64` RANGE comparator. ndarray #309 shipped both +//! (`{eq,ne,gt,ge,lt,le}_{u8,u64}_to_mask`), so `find_ram_in_range` now has a +//! direct spelling — three predicates over the columns' own types, no widened +//! copies, no hi32/lo32 split, no assumption that the window sits inside one +//! `hi32` bucket. This is the re-run, and it answers the two falsifiers +//! `.claude/knowledge/masking-ops-state.md` had PENDING (G1's second half and +//! G2's only half). +//! +//! ## Two corrections to the first run, both of which changed a number +//! +//! 1. **It measured v3/AVX2 and the report implied AVX-512.** `.cargo/config.toml` +//! is `x86-64-v3`; the host having `avx512f` says nothing about what was +//! compiled. Every arm below is now run under BOTH configs and the program +//! prints its own realization line. +//! 2. **No `black_box`.** ndarray's own G1 figures were published 8.06×/6.75× +//! and corrected to 6.91×/5.84× for exactly this — asymmetric dead-store +//! elimination. Inputs AND outputs of every arm are now protected, all or +//! none. +//! +//! ## Q1 — `find_tag`, one predicate over a `u8` column (12 408 ops, real) +//! +//! | tier | scalar | widened `eq_u32` | NATIVE `eq_u8` | +//! |---|---:|---:|---:| +//! | v4 / AVX-512 | 1× | 3.96× | **34.4×** | +//! | v3 / AVX2 | 1× | 1.84× | **26.1×** | //! -//! `S` is the honest baseline precisely because it needs no widened columns: -//! charging the SIMD arms for the layout they require is the comparison a -//! consumer actually faces. +//! **G1 is decisively answered**: the native primitive is 8.7× (v4) / 14× (v3) +//! better than the widening workaround it replaced. The ~26-34× against scalar +//! is the expected lane count, not an anomaly — 32-64 bytes per instruction +//! against a 1-byte scalar loop; bandwidth confirms it (52 GB/s L1-resident vs +//! 2 GB/s scalar). Do NOT compare this to G1's own 6.91×: that figure is +//! u8-vectorized vs i32-vectorized, a different pair. //! -//! # Measured 2026-09-14 — Xeon @ 2.10 GHz, avx512f/bw/vl, release, 3 runs +//! ## Q2 — `find_ram_in_range`, the `u64` range query (ns/op) //! -//! Fixture: `probes/win32-census/legacy_app.exe` (PE32+ x86-64, `.text` 7 688 B), -//! **12 408 p-code ops**. Spans above that are the real stream TILED. Every arm -//! agreed bit-for-bit at every span; the numbers below are ns per op. +//! | span | S | AND | TERN | UNDER | NATIVE | S/NATIVE | +//! |---:|---:|---:|---:|---:|---:|---:| +//! | **12 408 v4** | 0.73 | 0.46 | 0.45 | 0.64 | **0.48** | **1.51×** | +//! | **12 408 v3** | 0.72 | 0.73 | 0.73 | 0.79 | **1.31** | **0.55×** | +//! | 3 176 448 T v4 | 1.24 | 1.73 | 1.68 | 1.58 | 1.87 | 0.66× | +//! | 3 176 448 T v3 | 1.44 | 2.27 | 2.35 | 1.77 | 2.59 | 0.55× | //! -//! | span | S | AND | TERN | UNDER | S/AND | S/TERN | S/UNDER | -//! |---:|---:|---:|---:|---:|---:|---:|---:| -//! | 1 024 | 0.78 | 0.51 | 0.50 | 0.60 | 1.54 | 1.56 | 1.30 | -//! | 4 096 | 0.76 | 0.54 | 0.53 | 0.57 | 1.42 | 1.45 | 1.33 | -//! | **12 408 (real)** | 0.70 | 0.52 | 0.53 | 0.49 | **1.35** | **1.32** | **1.42** | -//! | 49 632 T | 0.70 | 0.58 | 0.58 | 0.55 | 1.21 | 1.22 | 1.29 | -//! | 198 528 T | 0.87 | 0.89 | 0.88 | 0.67 | 0.98 | 0.99 | 1.31 | -//! | 794 112 T | 0.87 | 1.12 | 1.11 | 0.90 | **0.77** | **0.78** | 0.96 | -//! | 3 176 448 T | 0.91 | 1.22 | 1.13 | 0.95 | **0.75** | **0.80** | 0.95 | +//! **G2 is answered, and the answer is TIER-DEPENDENT — the headline finding.** +//! The same native arm is a **1.5× win on AVX-512 and a 0.55× LOSS on AVX2**. +//! That is not noise: only avx512 (`epu64`) and NEON (`cmhi`) have an unsigned +//! ordered 64-bit compare; avx2 and scalar are flat polyfills, so on v3 the +//! "native" path is a scalar loop wearing a vector signature. A consumer on a +//! v3 baseline should keep the hi32/lo32 split; on v4 the native spelling is +//! both faster and general. //! -//! Q1 (`find_tag`, one predicate): 2.45× / 2.07× / 1.90× at 256 / 1 K / 4 K, -//! **1.76× at the real 12 408**, 1.58× at 49 K, and **1.00× at 794 K** — the -//! widened column's extra memory traffic eats the whole win. +//! ## The u8 widening is a 4× tax; the u64 "widening" is a DISCOUNT //! -//! ## Four findings +//! The first run blamed Q2's crossover on widened columns. Counted per arm, +//! the widened path is not merely no worse — it reads **less**: //! -//! 1. **There is a crossover and it is low.** The mask arms win up to roughly -//! 50 K ops and LOSE from roughly 200 K. Nothing here is a memory-bandwidth -//! surprise: `S` reads 9 B/op (`u8` + `u64`), the mask arms read 12 B/op of -//! widened columns and write four mask buffers. The layout's own motivation -//! — fewer bytes touched — is partly spent paying for the primitives' value -//! types. -//! 2. **The ternlog fusion is not the lever for this query.** `TERN` and `AND` -//! are within noise at every span (1.32 vs 1.35 at the real size). The cost -//! is the four passes over the value columns, not the three mask combines -//! the fusion removes. Fusion pays where a conjunction is over masks a -//! caller ALREADY holds; here each predicate must first be computed. -//! 3. **`_under` is the arm that survives scale.** Narrowing the live mask in -//! place needs no separate combine and no extra buffers, so it is the best -//! arm at 12 408 (1.42×) and the only one still near parity at 3.2 M. -//! 4. **The ratio is favourable exactly where the absolute time is -//! irrelevant.** A whole-census `find_ram_in_range` is 8.9 µs scalar and -//! 6.4 µs vectorized: **2.5 µs saved** on a binary whose SLEIGH lift costs -//! milliseconds. Per the workspace's own rule — a word-level op pays for -//! the span it is given — this span is not worth paying for. +//! | arm | reads per logical row | total | +//! |---|---|---:| +//! | `NATIVE` | `space` 1 B + `offset` 8 B (`ge`) + `offset` 8 B (`lt`) | **17 B** | +//! | `AND` / `TERN` | `space32` 4 B + `hi32` 4 B + `lo32` 4 B + `lo32` 4 B | **16 B** | +//! +//! Offset-derived alone it is **16 B native vs 12 B split** — a 25 % saving, +//! because each split predicate touches only the half it needs (`hi32` once, +//! `lo32` twice) while every native `u64` predicate must pull all eight bytes. +//! So the split both halves the element width (what the vector units reward) +//! AND cuts traffic. The `u8` case is the opposite sign: widening `tag`/`space` +//! to `u32` is a real 4× tax with no compensating structure. +//! +//! ⊘ This corrects the first published version of this section, which claimed +//! `hi32`+`lo32` came to "also 16 B/op" and concluded the Q2 arms paid no tax +//! at all. `hi32` is read ONCE, not twice — 12 B, not 16. The **direction** +//! (widened beats native on v3, matches on v4) is unchanged and is what the +//! measured timings show; the equal-traffic mechanism offered for it was +//! wrong. The consumer guidance below follows the timings and the native +//! path's generality, never that retired claim. +//! +//! ## The crossover survives both corrections +//! +//! Every arm degrades to ≤ 1.0× above ~200 K ops on both tiers. And the +//! absolute stakes are unchanged: a whole-census `find_ram_in_range` is ~9.0 µs +//! scalar and ~5.9 µs native-on-v4 — **3 µs saved** on a binary whose SLEIGH +//! lift costs milliseconds. Per the workspace rule *a word-level op pays for +//! the span it is given*, this span still does not pay. What changed is the +//! primitive surface, not the verdict for this consumer. +use std::hint::black_box; use std::time::Instant; use ndarray::simd::ternlog::AND3; use ndarray::simd::{ - eq_i32_to_mask_under, eq_u32_to_mask, ge_i32_to_mask_under, lt_i32_to_mask, lt_i32_to_mask_under, mask_and_assign, - mask_ternlog, popcount_batch_u64, + eq_i32_to_mask_under, eq_u32_to_mask, eq_u8_to_mask, ge_i32_to_mask_under, ge_u64_to_mask, lt_i32_to_mask, + lt_i32_to_mask_under, lt_u64_to_mask, mask_and_assign, mask_ternlog, popcount_batch_u64, }; const RAM: u8 = 3; @@ -172,7 +228,17 @@ fn main() { let path = std::env::args().nth(1).expect("usage: "); let base = load(&path); let real_n = base.tag.len(); - println!("dump: {real_n} ops from {path}\n"); + println!("dump: {real_n} ops from {path}"); + // Which realization actually ran. `.cargo/config.toml` is v3/AVX2; AVX-512 + // needs `env -u RUSTFLAGS cargo --config .cargo/config-v4.toml`, and a + // RUSTFLAGS env var silently REPLACES the config's rustflags. A timing + // without its tier is an anecdote, so the arm prints its own. + println!( + "realization: avx512f={} avx2={} neon={}\n", + cfg!(target_feature = "avx512f"), + cfg!(target_feature = "avx2"), + cfg!(target_feature = "neon"), + ); // Gap 2's soundness precondition, asserted rather than assumed: the window // must lie inside one hi32 bucket for the split re-expression to be exact. @@ -183,8 +249,8 @@ fn main() { ); println!( - "{:<10} {:>9} {:>10} {:>10} {:>10} {:>10} {:>8} {:>8} {:>8}", - "span", "hits", "S ns/op", "AND", "TERN", "UNDER", "S/AND", "S/TERN", "S/UNDER" + "{:<10} {:>9} {:>10} {:>10} {:>10} {:>10} {:>10} {:>8} {:>8} {:>9}", + "span", "hits", "S ns/op", "AND", "TERN", "UNDER", "NATIVE", "S/AND", "S/TERN", "S/NATIVE" ); let mut spans: Vec<(usize, bool)> = vec![256, 1024, 4096] @@ -214,10 +280,10 @@ fn main() { let (mut m0, mut m1, mut m2, mut m3) = (vec![0u64; w], vec![0u64; w], vec![0u64; w], vec![0u64; w]); let build = |m0: &mut [u64], m1: &mut [u64], m2: &mut [u64], m3: &mut [u64]| { - eq_u32_to_mask(&space32, RAM as u32, m0); - ndarray::simd::eq_i32_to_mask(&hi32, want_hi, m1); - ndarray::simd::ge_i32_to_mask(&lo32, lo_lo, m2); - lt_i32_to_mask(&lo32, lo_hi, m3); + eq_u32_to_mask(black_box(&space32), black_box(RAM as u32), m0); + ndarray::simd::eq_i32_to_mask(black_box(&hi32), black_box(want_hi), m1); + ndarray::simd::ge_i32_to_mask(black_box(&lo32), black_box(lo_lo), m2); + lt_i32_to_mask(black_box(&lo32), black_box(lo_hi), m3); }; let and_arm = |a: &mut Vec, m0: &mut Vec, m1: &mut Vec, m2: &mut Vec, m3: &mut Vec| { @@ -233,8 +299,20 @@ fn main() { mask_ternlog::(m0, m1, m2, t); mask_and_assign(t, m3); }; + // The NATIVE arm — the two gaps this probe reported are now closed + // (`eq_u8_to_mask` / `ge_u64_to_mask` / `lt_u64_to_mask`, ndarray #309), + // so the query has a direct, general spelling: THREE predicates over the + // columns' own types, no widened copies, no hi32/lo32 split, and no + // assumption that the window lies inside one hi32 bucket. + let native_arm = |n0: &mut Vec, m0: &mut Vec, m1: &mut Vec, out: &mut Vec| { + eq_u8_to_mask(black_box(&c.space), black_box(RAM), n0); + ge_u64_to_mask(black_box(&c.offset), black_box(WIN_LO), m0); + lt_u64_to_mask(black_box(&c.offset), black_box(WIN_HI), m1); + mask_ternlog::(n0, m0, m1, out); + }; + let under_arm = |u: &mut Vec, m0: &mut Vec, m1: &mut Vec| { - eq_u32_to_mask(&space32, RAM as u32, u); + eq_u32_to_mask(black_box(&space32), black_box(RAM as u32), u); eq_i32_to_mask_under(&hi32, want_hi, u, m0); ge_i32_to_mask_under(&lo32, lo_lo, m0, m1); lt_i32_to_mask_under(&lo32, lo_hi, m1, u); @@ -245,6 +323,10 @@ fn main() { and_arm(&mut a, &mut m0, &mut m1, &mut m2, &mut m3); tern_arm(&mut t, &mut m0, &mut m1, &mut m2, &mut m3); under_arm(&mut u, &mut m0, &mut m1); + let mut nat = vec![0u64; w]; + let mut nscratch = vec![0u64; w]; + native_arm(&mut nscratch, &mut m0, &mut m1, &mut nat); + assert_eq!(s, nat, "NATIVE arm disagrees at n={n}"); assert_eq!(s, a, "AND arm disagrees at n={n}"); assert_eq!(s, t, "TERN arm disagrees at n={n}"); assert_eq!(s, u, "UNDER arm disagrees at n={n}"); @@ -263,22 +345,43 @@ fn main() { }}; } - let ns_s = bench!(q2_scalar(&c, &mut s)); - let ns_a = bench!(and_arm(&mut a, &mut m0, &mut m1, &mut m2, &mut m3)); - let ns_t = bench!(tern_arm(&mut t, &mut m0, &mut m1, &mut m2, &mut m3)); - let ns_u = bench!(under_arm(&mut u, &mut m0, &mut m1)); + // `black_box` on every arm's OUTPUT, not just some of them. The + // asymmetry is the trap: ndarray's own G1 figures were published 8.06x + // / 6.75x and corrected to 6.91x / 5.84x because one family's stores + // were read later and the other's were eliminated. Protect all or none. + let ns_s = bench!({ + q2_scalar(black_box(&c), &mut s); + black_box(&s); + }); + let ns_a = bench!({ + and_arm(&mut a, &mut m0, &mut m1, &mut m2, &mut m3); + black_box(&a); + }); + let ns_t = bench!({ + tern_arm(&mut t, &mut m0, &mut m1, &mut m2, &mut m3); + black_box(&t); + }); + let ns_u = bench!({ + under_arm(&mut u, &mut m0, &mut m1); + black_box(&u); + }); + let ns_n = bench!({ + native_arm(&mut nscratch, &mut m0, &mut m1, &mut nat); + black_box(&nat); + }); println!( - "{:<10} {:>9} {:>10.4} {:>10.4} {:>10.4} {:>10.4} {:>8.2} {:>8.2} {:>8.2}", + "{:<10} {:>9} {:>10.4} {:>10.4} {:>10.4} {:>10.4} {:>10.4} {:>8.2} {:>8.2} {:>9.2}", format!("{}{}", n, if tiled { "T" } else { "" }), hits, ns_s, ns_a, ns_t, ns_u, + ns_n, ns_s / ns_a, ns_s / ns_t, - ns_s / ns_u + ns_s / ns_n ); // Q1, the single-predicate scan, at the same span. @@ -288,15 +391,30 @@ fn main() { eq_u32_to_mask(&tag32, TAG_CALL as u32, &mut q1v); assert_eq!(q1s, q1v, "Q1 arms disagree at n={n}"); let q1_hits = popcount_batch_u64(&q1s); - let q1_ns_s = bench!(q1_scalar(&c, &mut q1s)); - let q1_ns_v = bench!(eq_u32_to_mask(&tag32, TAG_CALL as u32, &mut q1v)); + let mut q1n = vec![0u64; w]; + eq_u8_to_mask(&c.tag, TAG_CALL, &mut q1n); + assert_eq!(q1s, q1n, "Q1 native arm disagrees at n={n}"); + let q1_ns_s = bench!({ + q1_scalar(black_box(&c), &mut q1s); + black_box(&q1s); + }); + let q1_ns_v = bench!({ + eq_u32_to_mask(black_box(&tag32), black_box(TAG_CALL as u32), &mut q1v); + black_box(&q1v); + }); + let q1_ns_n = bench!({ + eq_u8_to_mask(black_box(&c.tag), black_box(TAG_CALL), &mut q1n); + black_box(&q1n); + }); println!( - "{:<10} {:>9} {:>10.4} {:>10.4} <- Q1 find_tag: scalar u8 vs eq_u32 over a widened column, ratio {:.2}", + "{:<10} {:>9} {:>10.4} {:>10.4} {:>10.4} <- Q1: scalar / widened eq_u32 ({:.2}x) / NATIVE eq_u8 ({:.2}x)", "", q1_hits, q1_ns_s, q1_ns_v, - q1_ns_s / q1_ns_v + q1_ns_n, + q1_ns_s / q1_ns_v, + q1_ns_s / q1_ns_n ); } }