model: the host union's ungated relu² form, for Bolt's routed experts (rebase of #355)

[midagedev]

Supersedes #355 (CLEAN and CI green at a2c9c1c6d, then CONFLICTING after main moved). This is the same patch rebased onto origin/main ce96affca: the code part of git diff origin/main HEAD equals #355's git diff db5c89e7c a2c9c1c6d line for line (1630 changed lines), and tools/gate-paths.tsv differs only by the gate-nemotron-h-moe-host token in the same 14 rows, re-filled with python3 tools/recipes.py check --fill. #355's review thread and Shingler's APPROVED are there; the review fixes are folded into this branch's commits.

What and why

The host union (model::moe) gets the ungated relu² form, so Bolt's routed experts (up [w,1024] → relu² → down [1024,w], no gate, inside the 1024 latent) can run on the host tier. It is an arm of the common union beside the gated SwiGLU form, not a Bolt copy: ExpertForm::Relu2 reads one stack where the gated form reads two (a UnionPass::Up first pass), and the combine is relu(up)². A NaN or an infinity in the up output is refused by name. The union scratch's expert width becomes a capacity, so one scratch serves a run's layers of differing widths (Bolt's ff runs 1536–2560 per layer). Prerequisite of the Bolt offload work (B3, #328); it changes no kernel.

Release notes: internal — host-tier prerequisite for offloading Bolt's routed experts; no user-visible change.

Signatures B3 calls

• HostLayer::experts_union_into<'x>(&self, src: R8Source<'_>, x: impl Into<Tensor2View<'x>>, lists: &[&[(u32, f32)]], out: &mut [f32], scratch: &mut UnionScratch) -> Result<(), ModelError> (unchanged shape; experts_step_union_into takes the same).
• A layer: arch::nemotron_h_moe::host::layer(src, &hp, l) -> Result<Option<HostLayer>, ModelError> — None for a layer that does not route (so a sparse Vec<Option<HostLayer>>), Err for a layer past the model. Or HostLayer::build_relu2(src, &Relu2Spec { up, down, n_expert, embd, ff }) (no gate field).
• Scratch: UnionScratch::new_routed(embd = 1024, ff = the widest layer's, cols, n_used = 9). A layer wider than the scratch, or another embd, is refused by name; a narrower layer gives the same bits as through a scratch made for it.
• An NVMe tier on a relu² layer is refused by name (TierError::Relu2Tiered) at the call (attach_tier itself does not refuse).

Code shape

• No second copy. The relu² combine, the first pass and the scratch capacity are in the common union (ops.rs, moe.rs). nemotron_h_moe/host.rs only wires the latent and each layer's shapes; the stack names come from the common arch::names::routed_stacks.
• Model code holds only architecture facts. host.rs: which layers route, each layer's own n_expert and ff, the latent, the relu² form. The common owner gained a form arm (ExpertForm), no per-model branch.
• Per-model copies. Added: none under arch/; arch/nemotron_h_moe/host.rs is the one new file and holds only the facts above. Removed from this PR's own earlier shape: two Bolt-only name functions in arch/nemotron_h_moe/names.rs, moe.rs's test relu2_file, and the tests' own copies of seeded, diff_cells, panic_text and the two layer-file writers (now tests/common/columns.rs, tests/common/panics.rs and one StackFile::write in tests/union.rs).
• Dead code is gone. The WIP's gate-name field and its form-vs-file refusal are removed: nothing in the tree could reach them.
• No silent failure. A layer past the model, a scratch narrower than the layer, another embd, a relu² layer with a tier, and a non-finite up value are each a named error; each has a gate clause and a mutant.
• Comments state what is true now. tools/check-comments.sh: ok.
• No relaxed gate. The union gate's existing refusal "scratch of other widths" still passes with the same text.
• cargo fmt clean; cargo clippy -p bloomery-model --all-targets: one warning, ops.rs collapsible_if (existing, from 9e894eb). Workspace just lint not run by me.

Duplication found or removed: see "Per-model copies" above. Still open and not touched here (other families, for a follow-up): the (gate, up, down) name triple in deepseek41/host.rs, glm5next/host.rs, mimo2/host.rs, qwen35moe/host.rs can take routed_stacks; three synthetic stack byte generators (tests/common/r8layer.rs, moe.rs tests stack_bytes, tests/union.rs q8_stack).

Gates (box, tools/gate-batch.sh --round-ledger, real tier, head 38d65a48c)

gate	result	box job
gate-union (8 tests: V4.1 ×3, V2-Lite, Q6_K/Q8_0/r8 synthetic, relu² new)	PASS (8 passed)	s2569
gate-nemotron-h-moe-host (new, Bolt file)	PASS (1 passed)	s2566
gate-ops (lib 385 + ops 8; group _into gate gains relu² and non-finite clauses)	PASS	s2563
gate-moe / gate-alloc / gate-mt / gate-r8 / gate-ds41-host	PASS	s2562 / s2575 / s2576 / s2578 / s2572
gate-qwen4exp-meta / gate-nemotron-h-moe-meta	PASS	s2565 / s2571

DONE total=10 red=0 tier=real skipped=0 wall=326s laneA=326s laneB=325s (tools/gate-batch.sh --list h1-gates.list --round-ledger, queue lanes any). Every log shows Compiling bloomery-model. The list is the owning gates of the host paths this change touches.

Narrowing (just narrow origin/main, at 948cac02e)

Run on #355's tree (948cac02e), whose patch this PR carries unchanged; not repeated on 38d65a48c (the ten owning gates above are).

96240135..948cac02: ptx-scan --no-jit of gate_glm5next_e2e, gate_kquant, gate_swap is identical on both sides (crates/model holds no device code), so 159 selected recipes narrow to 122. The narrowed list ran in the fixture tier:

Fixture tier, head 948cac02e over 962401358 (tools/gate-batch.sh --list narrowed.txt --round-ledger --tier fixture): 83 recipes ran and 48 are deferred to the real tier (29 real-only, 1 of a family with no fixture, 18 real-file; deferred.list).

• First pass: DONE total=131 red=28 tier=fixture wall=2665s (55 green). 21 of the 28 reds were my own doing: I exported BLOOMERY_Q_CARD=any into the batch, which box.sh refuses (rc 64) beside the recipes' own BLOOMERY_CARD pick. The 28 re-ran without it: DONE total=28 red=7 tier=fixture wall=2003s, so 76 of 83 are green.
• The 7 reds are not this change:
  • Files absent on the box: gate-gpu-nemotron-e2e and gate-gpu-nemotron-serve (the nemotron_h family has no fixture file, rc 66), gate-gemma-embedding-meta, gate-gemma-embedding2-meta, gate-gpu-gemma-embedding2-e2e (/models/embeddinggemma-2/embeddinggemma-2-Q8_0.gguf is missing).
  • gate-gpu-qwen4exp-e2e (job s2182) and gate-gpu-qwen4exp-mtp (s2197) fail one card-plan clause each on the 3090 (and on the A6000 in the first pass): card leg plan: 4 of 4 layers hold routed experts on the card, a layer's count Some(255)..Some(256) (want 256 on every layer: half the experts, the header's budget) FAIL. The same line, with Compiling bloomery-model in the log, fails on origin/main e195c9e6a (jobs s2230, s2235, BLOOMERY_GATE_CARD=3090, fixture tier). The plan code is not touched here.
• The two commits after the fixture run (here 0b5f40a5c, fc7733e7f) change only crates/model host-tier source and tests (no device code). The ten owning gates above re-ran on them; the fixture batch was not re-run.

FAIL-first (one mutant each, red, then reverted; every log shows Compiling bloomery-model)

mutant	caught by
relu not squared / combine reads the wrong slot / first pass writes the wrong slot / leg reads two up blocks	union relu² clause
capacity check removed (both halves) / column-room half removed	union relu² refusal clauses
caller-side and claim-side combine copy the up (write_par, col_into)	union relu² / group gate (ne1 = 10 through matmul_q_group_cols_into)
warm_spans warms three stacks / tier refusal removed (one-column, union)	moe unit test
non-MoE layer is Err (the WIP's behaviour) / embd is the model's, not the latent	gate-nemotron-h-moe-host
routed_stacks up and down swapped in host::layer	gate-nemotron-h-moe-host (rc 101)
non-finite refusal removed (union / group gate) / refusal catches NaN only	union relu² overflow clause / group gate ±inf clauses

The col_into mutant survived the union clause and the ne1 = 1, 6 clauses (only the bench-only matmul_q_group_cols_into with an input past DEFER_MAX_COLS reaches it); the group gate now runs it at ten columns and that mutant is red.

GPU Evidence

• Box: 0, NVIDIA GeForce RTX 3090, 615.71.09 and 1, NVIDIA RTX A6000, 615.71.09 (nvidia-smi); AMD Ryzen Threadripper PRO 5975WX 32-Cores, Linux 6.8.0-142-generic. These gates run the host tier on the CPU (AVX2 qdot kernels); no device kernel changed, and no card kernel ran in them.
• Commits: base ce96affca (origin/main), head 38d65a48c. The narrow and fixture runs were made on model: the host union's ungated relu² form, for Bolt's routed experts #355's tree (base 962401358, head 948cac02e), whose patch this PR carries.
• Commands: just gate-union, just gate-nemotron-h-moe-host, just gate-ops, … through tools/gate-batch.sh --list h1-gates.list --round-ledger (jobs in the table); the FAIL-first runs through tools/box.sh with the gates' own commands (jobs in the table below).
• Output (raw lines, head 38d65a48c):

PASSED: union relu² — IQ2_XXS up and down, ungated: the union call at k = 1, 8, 9, 64 on two layers of different widths through one scratch made for the wider equal to experts_into and to the transcription on qdot's calls, bit for bit under both deferral arms and in their dispatches; the widths' refusals by name, and an overflowing up row refused by name
PASSED: nemotron_h_moe host — 40 MoE layers build as relu², the rest as none; layers 3 and 21 (the narrowest and widest ff) through one scratch for ff 2560: union equal to experts_into at k = 1, 9 under both deferral arms
test result: ok. 8 passed; 0 failed (gate-union)
test result: ok. 385 passed; 0 failed (gate-ops lib)
test result: ok. 8 passed; 0 failed (gate-ops ops)

Finiteness check removed (FAIL-first of the refusal clauses at tests/ops.rs:949 and tests/union.rs:1869). Baselines first (tools/box.sh, the gates' own commands, unmutated head): union job s2580 rc 0, ops job s2581 rc 0. Then one mutant of relu2_into (crates/model/src/ops.rs) at a time, applied, run, restored (tree clean after each); every log shows Compiling bloomery-model:

mutant	gate	job	result
invalid |= !u.is_finite(); → invalid |= false; (the test removed)	ops	s2587	rc 101, tests/ops.rs:988: "relu² with NaN in an up block (ne1=1, deferred, allocating) must panic naming the non-finite value, got None"
if invalid { → if false { (the refusal removed)	ops	s2584	rc 101, same line, same message (got None)
u.is_finite() → u.is_nan() (NaN only)	ops	s2588	rc 101, tests/ops.rs:988: "... with +inf in an up block ... got Some("qdot: non-finite activation value inf at offset 31 of a 32-value block")"
the test removed	union	s2585	rc 101, tests/union.rs:1890: "experts_into: an overflowing up row must panic naming the non-finite value, got Some("qdot: non-finite activation value inf at offset 34 of a 256-value block")"
the refusal removed	union	s2582	rc 101, same line, same message (offset 216)

Two things the runs show. A NaN up value is turned into zero by f32::max(NaN, 0.0) and nothing downstream sees it: without the check the ops gate gets None (no panic at all). An infinity is still refused downstream by qdot's activation quantizer, under qdot's name; the union clause pins the relu² name, which is the only thing that fails there, so the union gate alone does not prove the NaN case. The ops gate does, and both entries call the one relu2_into.

Timing

No timed run was made. Derived, not measured: the form choice is one predictable branch per expert combine, and the relu² finiteness check is a branch-free OR over the up row, about 0.05–0.1 µs per 2560-wide combine against a derived ~6.6 µs per host expert (under 2 %). A same-lease just ab-decode A/B belongs to perf.

---

AI use: this change and its gates were written and reviewed by a Claude agent session (Claude Sonnet 5.5, the h1 worker of the Bolt offload team); the lead reviews and merges. This text was written by that bot.

🤖 Generated with Claude Code

[github-actions]

Verdict: MUST_FIX: the relu² path breaks the scratch output contract and adds two duplicated owners.

Must fix

1. Preserve the up outputs exposed by HostScratch

crates/model/src/moe.rs:1052 writes relu² ups into contiguous blocks, but HostScratch::up at line 937 reads gate_up[2*i+1]. With fresh scratch, a one-expert HostLayer::experts_into call with a nonzero up output writes block 0; scratch.up(0) then returns untouched block 1, containing zeros. Larger lists return another expert's output or stale values. This violates the documented scratch-output contract (kind A).
Restore that layout in the common owner. After let dt = dt?; in serve_resolved, swap buffer headers:

    if let ExpertForm::Relu2 = run.form {
        for i in (0..n).rev() {
            gate_up.swap(i, 2 * i + 1);
        }
    }

Add an up-output assertion to the existing relu² gate. Its final-output comparisons pass on this bug.

2. Reuse the existing tier-refusal owner

crates/model/src/moe.rs:1746 repeats the relu²/NVMe rejection already owned by refuse_relu2_tier at line 2889. This is the same form policy and error, with two maintenance sites (kind B, R14).
Consolidate it while all entry points are in this diff. In union_on, replace its first statement with:

        self.read_beside(src)?;
        self.refuse_relu2_tier()?;

Delete the initial relu² guard and its two-line comment from serve_tiered. The other tiered entries already call the helper; the free union entry is always gated.

3. Reuse the existing synthetic stack-name constants

crates/model/tests/union.rs:1649 adds RELU2_UP and RELU2_DOWN with the exact values of Q8_UP and Q8_DOWN at line 1251. Both pairs name the synthetic up/down stacks (kind B, R14).
Remove these extra owners before more gate clauses use them. Delete the two new constants. Replace the two tuples in Relu2Layer::write with:

                (Q8_UP, ty, [embd, ff, n_expert], &up),
                (Q8_DOWN, ty, [ff, embd, n_expert], &down),

Replace the name fields in both relu² gate Relu2Spec initializers with:

            up: Q8_UP,
            down: Q8_DOWN,

Checked

Duplicates: the policy and stack-name copies above are confirmed. Dryer reported no pairs.
Complexity: diagnostics report new build_form at 22; complexity alone is not a finding.
Silent failure: relu2_into explicitly panics on non-finite up values.
Tests: the relu² union gate compares final outputs without reading HostScratch::up, so it misses the accessor defect.
Gates: the PR body supplies FAIL-first reports; these were not rerun.
Escape catalog: no C1–C4 patterns matched.

Notes

Static review only. Hardware gates were not rerun.

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Since the last review:

• 1. Fixed: crates/model/src/moe.rs:1084 restores each expert's up slot after the down dispatch.
• 2. Fixed: crates/model/src/moe.rs:3051 reuses the tier-refusal owner; the duplicate guard is removed.
• 3. Fixed: crates/model/tests/union.rs:1664 reuses the existing stack-name constants.

Verdict: CLEAN: all three earlier findings are fixed, and the new commits introduce no must-fix defect.

Non-blocking

⚠ Current revision gate evidence: crates/model/tests/union.rs:1793 adds the accessor comparison. The supplied gate output predates this fix. Provide just gate-union and just gate-moe output for this head, including a FAIL-first run with the slot swap removed, to verify the new regression check.

Checked

Duplicates: 2 new test helpers checked; the existing transcription is shared. Dryer reported 0 pairs.
Complexity: crapper flags build_form (22, unchanged in this increment) and serve_resolved (9 to 15, form branches and loops). The added layout loop has one job.
Silent failure: 1 moved error path traced. Tier rejection propagates before arena access; no new default suppresses an error.
Tests: static mutation analysis links moe.rs:1084 to union.rs:1793; removing the swap makes the accessor comparison fail. Removing the step refusal at moe.rs:3119 makes the named-error assertion at moe.rs:3539 fail. These mutations were not run against the Rust tests.
State and bounds: static inspection confirms reverse swaps stay within scratch capacity and preserve list order, including empty lists. Gated calls overwrite their interleaved slots.
Performance: the host decode leg adds one buffer-header swap per routed expert, with no activation copy, allocation, launch or synchronization.
Gates: no bands, thresholds or existing clauses were relaxed or removed.
Escape catalog: no C1–C4 patterns matched.

Notes

Validation is static. The supplied owning-gate output targets the earlier head 38d65a4.

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Since the last review:

• Slot restoration: fixed; crates/model/src/moe.rs:1084 keeps each expert's up output in its accessor slot.
• Duplicate tier guard: fixed; crates/model/src/moe.rs:3051 reuses the existing refusal.
• Duplicate stack names: fixed; crates/model/tests/union.rs:1664 uses the existing constants.

Verdict: CLEAN: the merge preserves the earlier fixes and introduces no new must-fix defect.

Checked

Scope: the increment merges main; its 24 source and documentation paths are identical in base and head. The merged recipes preserve the host gate and add its dependency mapping.
Duplicates: 0 new PR-only definitions; all 36 added definitions in the increment already exist in base. Dryer reported 0 pairs.
Complexity: crapper flags build_form (22, complex loader) and serve_resolved (9 to 15, form branches and slot loops); neither function changes in this increment.
Silent failure: 3 refusal paths followed. Non-finite up values panic by name; scratch-capacity and tier errors propagate.
Tests: static mutation analysis shows removing moe.rs:1084's swap fails union.rs:1793's accessor comparison. The index model passes for 0 through 9 experts. Rust gates were not run.
State and bounds: empty lists, full routed lists, narrower layers sharing capacity, and tier entry guards checked. Unclamped gated legs still reach the original SwiGLU implementation.
Gates: no PR-specific band, threshold or clause was relaxed or removed.
Escape catalog: no C1–C4 matches in PR-only changes.

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}