RQ-64-HISTOGRAM (#1159): mask hex payloads in the census collapse, assert the sum invariant, wire the self-test — arm's non-rotated-immediate gap is rank 3 at 9, not rank 5 at 4

[avrabe]

RQ-64-HISTOGRAM (#1159): mask numeric payloads in the census collapse — and the arm ranking moved

Refs #1159, #242, #1156

What was wrong, precisely

normalize_reason() in scripts/repro/partial_census_1017.py already masked bare decimals (\b\d+\b -> N). A hex immediate survived because the x in 0x5dc glues the digits into one word, so there is no word boundary for the regex to find. That is exactly how immediate 0x5dc (1500) printed as immediate 0x5dc (N) — the decimal collapsed, the hex not — and one cause fragmented into one row per distinct immediate.

The mask therefore had to be extended in normalize_reason, not in collapse_instance_lists(): the latter is only reached by module-level reasons, and the encode_operand2 message is a per-function skip reason that goes through normalize_reason alone. Putting hex masking beside the existing decimal mask keeps one numeric mechanism on every path.

What changed (all measurement-side; ci-status: manual (measurement) preserved — no expected value about synth was added)

• normalize_reason: one tokenizing pass — hex -> 0xN, decimal -> N, and reference identifiers preserved (#1102, §4.5.5, VCR-MEM-002): they are format-string constants, cannot vary per instance, and the published tables were restoring them by hand. Digit-bearing op/register/function names (I64Store32, R11, func_25, RV32) were never word-bounded numbers and stay untouched. The line: mask VALUES, never IDENTIFIERS.
• module_reason(): the single strip-Error: /mask/list-collapse pipeline, now used by BOTH module-level rankings (primary-blocker histogram and the "module-level decline reasons (top 10)" list, which ranked RAW text until now).
• print_ranked(): prints a ranked histogram AND asserts the sum invariant on the PRINTED rows — exit 3 if they do not sum to the modules ranked; a top=N cut carries an explicit remainder row so the visible sum closes; a width cut under which two distinct causes would print identically prints them in full. Stated over printed rows deliberately: a Counter over one key per module sums by construction, so asserting that would be a check that cannot fail.
• --self-test: the two fragmenting shapes collapse to one key; idempotence; identifier survival; list collapse still composes; a negative control (2 rows for 3 modules -> exit 3; an empty key -> exit 3) and a top-cut positive control.

The self-test is WIRED — the file's ci-status moved from manual (measurement) to wired

The coordinator's review caught that the fix would otherwise ship a self-test CI never runs. The ledger had already written the rule: the claims.yaml note beside this file's manual slot says "if the census ever grows an expected value or a pass/fail verdict, it stops being a measurement and must be wired instead of counted here." --self-test is exactly that — an expected value and a verdict about the INSTRUMENT, hermetic, corpus-free, no synth binary.

• Header: # ci-status: wired — … + # ci-checks: stdout /^self-test OK: (\d+) assertions/ >= 14 — the floor binds the assertion COUNT, so a self-test whose body stopped running cannot pass green.
• CI: a step in the required claim-check job runs it through scripts/oracle_run.py, followed by the per-job evidence ledger (oracle_evidence.py --min-oracles 1).
• Ledger: manual ceiling 8 → 7 in claims.yaml (reason recorded beside the pin) and **7 manual** in ORACLE_WIRING.md; wiring gate now reads 181 wired / 7 manual / 0 unwired, stdout floor 468 → 482, emulations floor total 324845 unchanged (re-derived from the gate itself).
• The census MODES are unchanged: still a local measurement, still no verdict about synth. Only the instrument's self-check is a gate.

Red-first, shown both ways (mutants in the PR author's scratch, reproduced in the artifact's landed:):

mutation	direct --self-test	under oracle_run
hex alternative removed from NUMERIC_PAYLOAD_RE (pre-#1159 rule)	exit 1 — assertion 2 FAILED: hex shapes did not merge	exit 1
evidence line renamed (self-test ok:) — body still passes	exit 0 (!)	exit 1 — measured=0 vs floor 14, pattern did NOT match

And the discriminator is permanent: assertion 1 asserts that the OLD rule still keeps the two wild shapes apart, so if a future edit strips the hex payload from the embedded shapes, the test reports its own vacuity instead of passing on both masks.

Measurement — full 243-module corpus, both modes, three backends, BEFORE vs AFTER

Corpus verified against MANIFEST.sha256 (243/243). synth built at 8862ee2f (no Rust has changed since v0.63.0, so this is the compiler RQ-63-LADDER measured). Per-module verdicts and rungs are identical between the BEFORE and AFTER runs (0 diffs, 0 TIMEOUT anywhere at 120 s), so only the bucketing moved. Every printed histogram now ends with its sum invariant OK line.

Did any ranking change? Yes — arm, both strata. No — riscv and aarch64, both modes.

stratum	before	after
arm / core (110 non-accepted)	44 register exhaustion · 25 global initializers · 16 #929 AAPCS i64 pair · 16 active data · 4 encode_operand2 0x624 · 2 decode · 1 LdrSym · 1 encode_operand2 0x5dc · 1 no exports	44 · 25 · 16 active data · 15 #929 · 6 encode_operand2 0xN · 2 · 1 · 1
arm / components (106)	30 active data · 27 register exhaustion · 25 #929 · 7 start · 5 rotl · 4 call_indirect · 4 GI-FPU-002 · 1 · 1 · 1 · 1	30 · 26 · 25 · 7 · 5 · 4 · 3 encode_operand2 0xN (NEW ROW) · 2 GI-FPU-002 · 1 · 1 · 1 · 1

Why the merged row is 6 and not 5, and why components gained a row that did not exist: _modal() is a per-module plurality vote, and a fragmented cause loses it even when it is the module's largest. Per-module evidence (from the --json records):

• yolo_inference_release / yolo_inference_debug (components): encode_operand2 hit 84 functions across ~20 distinct immediates (largest fragment 6) and lost to GI-FPU-002 at 58. Merged, it wins 84 vs 58.
• sockets-tcp-connect (components): 10 functions across 8 immediates vs register exhaustion at 8.
• stat-dev-ino (core): 5 functions across 5 immediates vs [CRITICAL] thumb-2 silently miscompiles any call with an i64 parameter that is not last — args land in the wrong registers, exit 0, no warning #929 at 4.

So on arm the non-rotated-immediate class — which arm_encoder.rs:8744's own comment calls an INTERNAL selector bug (large constants must be materialized via MOVW/MOVT) — is 2x the size the v0.63 plan saw, and on components it was invisible.

The acceptance ladder — the table v0.63 published and v0.64 plans from. BEFORE reproduces RQ-63-LADDER's arm NEVER bucket exactly (141 of 243; rungs 27 / 72 / 3 / 0), zero rung diffs between runs:

rank	before (12 rows, Σ 141)	after (11 rows, Σ 141)
1	71 register exhaustion	70 register exhaustion
2	41 #929 AAPCS i64 pair	40 #929
3	7 start section	9 encode_operand2 non-rotated immediate (was rank 5)
4	5 rule_i32_rotl	7 start section
5	4 call_indirect · 4 encode_operand2 0x624	5 rule_i32_rotl
6	—	4 call_indirect
7	3 GI-FPU-002	2 decode
8	2 decode	1 · 1 · 1 · 1 GI-FPU-002
9	1 · 1 · 1 encode_operand2 0x5dc · 1	—

The non-rotated-immediate class is the third-largest real capability gap on arm, not the fifth, and the same four modules are re-attributed as in the plain census. Denominator stated: the arm ladder was measured TWICE per side — one full-corpus single pass (~110 min each) and four sorted corpus quarters (61/61/61/60, merged, 243 unique paths asserted) — and the two agree on every module's rung and primary blocker (0 of 243 differ, both sides); the full-pass reports carry the 12-row / 11-row NEVER tables above with their sum invariant OK lines.

riscv and aarch64: zero merges, zero rank moves, plain and ladder. Surveyed every field of every raw record: those backends' messages carry no hex payload at all, so the no-op there is by construction.

The second ranking the census prints was wrong in order. "module-level decline reasons (top 10)" ranked raw text: on aarch64 core the #1102 dangling-reloc refusal (64 of 86 modules, spread over 59 rows) printed BELOW active-data segments (17 modules), and the top-10 cut silently dropped 60/78, 35/103, 58/86 and 42/107 modules. Through the one pipeline it is now 5 / 4 / 6 / 5 rows that sum, with a remainder row when cut. This changes no published number (the primary histogram already re-attributes #952/#1102 to the skip reason behind them), but it was an untrustworthy order the instrument printed.

Spec census (scripts/spec_compile_census.py, 257 files, 3 backends, same binary): RESULT: PASS, PINS / FAMILY_OK_PINS / AT_LEAST_ONE_EXPORT unchanged (arm 22/62/130/17, riscv 12/62/140/17, aarch64 27/30/109/65; 84/74/57). Stated precisely: that script emits exact-pinned bucket counts and no histogram, so a reporting change in the corpus census cannot reach it — and it did not move.

The three questions #1159 actually asks, answered plainly

1. Did any cause change rank on arm once hex payloads stopped fragmenting it? Yes, but narrowly. In the ladder NEVER bucket — the table v0.63 was scoped from — one cause moved: encode_operand2 non-rotated immediate, rank 5 → 3 (4 → 9 modules), overtaking "start section" (7) and rule_i32_rotl (5); GI-FPU-002 fell 3 → 1 (rank 7 → 8); the two top gaps (register exhaustion 71 → 70, #929 41 → 40) kept ranks 1 and 2. In the plain census: core, #929 16 → 15 lost its tie for 3rd; components, encode_operand2 went from no row to rank 7 (3) above GI-FPU-002 (4 → 2). Everything else on arm, and everything on riscv and aarch64, is unchanged in rank. So: v0.63's top two were right; its 3rd–5th were misordered, and the misordered class is the one arm_encoder.rs itself labels an internal selector bug.

2. Did any cause below the top-N cut rise into view? No — in the ranked blocker histogram there was nothing below the cut to rise. The plain census prints every row; the ladder's top-12 cut was never exceeded (arm had exactly 12 rows before, riscv 11, aarch64 9). What moved was already IN view: a 4-row and a 1-row, plus four modules counted under other causes' rows because the fragments lost each module's plurality vote. The one place a cause did rise from below a cut is the census's secondary "module-level decline reasons (top 10)" list, where the #1102 refusal class was mostly singleton rows under the cut and is now rank 1 on every stratum — a policy-refusal listing, not the capability ranking.

3. How many modules was the cut dropping from the ranked view entirely? From the primary blocker histogram: zero (no cut applies, plain or ladder, on any backend). From the secondary module-level top-10 list: arm 56/110 core and 65/106 components; riscv 60/78 and 35/103; aarch64 58/86 and 42/107 — now zero, via the remainder row. Separately, the hand-transcribed v0.63 doc table listed 7 of the ladder's 12 rows (135 of 141 modules); one of the six omitted modules was an encode_operand2 fragment.

Reconciling 135 / 138 / 141 (coordinator review). A held pre-fix arm ladder listing shows nine rows summing to 138 under NEVER bucket (141). Replaying the PRE-FIX script's own report_ladder over the exact BEFORE NEVER set (each module's stored pre-fix blocker, no synth re-run) prints twelve rows summing to 141; the nine held rows are its first nine, same order and counts, and the three "missing" modules are rows 10–12: LdrSym … Thumb-N-only (1 — the pre-fix mask also ate the 2 in Thumb-2), encode_operand2 0x5dc (1), No exported functions (1). Both v0.63.0 and pre-fix main print most_common(12) and the Counter has 12 keys, so the shipped tool printed all twelve: 138 (9 of 12 rows) and the doc's 135 (7 of 12) are transcription cuts, not tooling losses. Therefore the new assertion does not go red on the pre-fix bucketing — correctly, since that report lost nothing (replay: 12 rows, sum 141, invariant OK; a cut forced to top=9 now prints 3 (+3 more rows below the top-9 cut) and still closes at 141). The invariant is exactly as claimed, not narrower: it covers the primary histogram, and 141 and the row population are one set. The hypothesised third mechanism — a fragment losing the plurality vote and falling out of attribution — does not exist: 0 of 141 NEVER modules have an empty primary blocker (_modal always returns a key when skip reasons exist; the module-level reason otherwise; the unparsed case is a named string). So the loss mechanisms are: the CUT hides rows (now closes via the remainder row), FRAGMENTATION misranks (now masked), and TRANSCRIPTION drops rows outside the tool — which the tool cannot fix, but its printed N rows sum to M = M modules ranked line now makes checkable.

Boring parts stated as such: riscv and aarch64 are identical before/after in both modes, and on arm the fix changed no total, no rung count and no top-two blocker. The material result is the single rank move of a cause that was counted at less than half its size.

Over-collapse audit (the "negative result" check)

Every MERGE the before/after comparison produced was inspected: all are the single encode_operand2 message. No two causes were merged. The only place the mask was NARROWED relative to the old rule is deliberate — reference identifiers are kept, because masking them can only lose identity and never un-fragment anything.

Gates (local, on the merged 8862ee2f tree)

• partial_census_1017.py --self-test: self-test OK: 14 assertions; via oracle_run: mode=stdout floor=14 measured=14, ledger below_floor=0
• claim_check.py claims.yaml: 59/59 (with the manual ceiling at 7)
• status_evidence_check.py: 0 failures (artifact implemented, landed:/verified-by: filled)
• oracle_wiring_check.py --exact-emulation-floor 324845 (re-derived from its own emulations floor total): exit 0 — 181 wired / 7 manual / 0 unwired / 0 undeclared
• rivet validate: 40 errors = main's baseline (delta 0); the artifact was validated with rivet's own loader
• cargo fmt --check: clean; cargo clippy --workspace --all-targets -- -D warnings: exit 0, 168 crates checked (no Rust touched)
• Both commits SSH-signed (%G? = G), rebased onto 7b8b7e36 (fix(rivet): RQ-64-FLOORPROSE was scoped from a wrong premise — corrected before any work was done #1178) cleanly

Queue note

#1179 (RQ-64-ARM64LINUX) is ahead of this PR and touches ci.yml and claims.yaml too (floor bump 324845 → 324847, two jobs). When it lands I will rebase, resolve hunk by hunk keeping BOTH sides, re-derive the emulation floor from oracle_wiring_check.py's own emulations floor total (never carried textually), and re-run claim_check / status_evidence_check on the merged tree.

Not done, on purpose

docs/status/ACCEPTANCE_LADDER.md keeps its v0.63 tables as measured by the v0.63 instrument; an addendum records the arm NEVER re-attribution under the fixed instrument so the number v0.64 plans from is the corrected one, without rewriting a shipped measurement.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!