feat(h3): extend block validation to full-model and generation qualification

[fusheng-ji]

Current qualification status — 2026-10-13

Full H3 candidate acceptance remains incomplete. Independent reference job 1106656 completed from clean source 2681cadb: FP32/BF16 runs across 50 distinct transformer blocks, both refiners and boundaries, with 638 parameter-gradient artifacts per precision. Reference generation and repeats passed for seeds 42/43/44 at 124 frames, 1344×768, 24 fps and 32 kHz audio. All 49 predicted-velocity steps matched their repeated provider controls; these are provider results, not candidate acceptance.

Candidate source 98b07a3f commits the unchanged 764 tensor bounds and a 1,635-file SHA-256 reference inventory before execution. The runner builds an independent H3 CUDA extension, verifies native identity and all consumed reference/model hashes, then gates block-0 golden tests, complete forward/backward, numerical comparison, preflight and three-seed candidate generation. Actual per-layer candidate-node calls are mandatory; fallback or missing nodes fail. First drift is ordered by real forward/backward execution. Row/interim/reference labels and the existing quality thresholds are retained.

• Failed: candidate job 1107493, clean source 98b07a3f, one B200, exited 1 after 15m34s. Native build and all reference/checkpoint hash checks passed. Block-0 golden regression reported 8 passed, 1 failed, 0 skipped: attn.norm_q.weight candidate gradient relative error 0.03398705939217804 exceeds the unchanged bound 0.03044327425907444 (provider error 0.016962182839382958). Full-model candidate and generation stages correctly did not execute. The original logs/JUnit and binary are retained.
• Diagnostic completed; localization continues: job 1107566 completed in 25 seconds, reproducing the block q-norm gradient failure. With identical inputs/upstream gradients, isolated Q/K norm candidate and provider errors were equal (Q 0.0016656962, K 0.0029305728). This does not prove the full-chain error is fixed. New one-B200 job 1107836, clean source 9317a5e1, is queued for 34 single-node substitution controls (one provider node in the candidate graph, and the reverse). These intentional hybrids are diagnostic only and cannot establish candidate acceptance. The native binary is pinned to the failed run; thresholds remain unchanged.
• CPU validation: changed-file pre-commit passed. At latest repair commit 9317a5e1, lint, unit tests and docs all actually ran and passed. GPU qualification remains incomplete.
• History retained: reference job 1105544 failed generation because torchvision/processor was missing after completing its reference backward stages. The processor repair and matching torchvision overlay enabled successful retry 1106656. Earlier local expanded CPU results (149 passed, 358 hardware skips, three subprocess-startup timeouts) remain historical, not an all-pass claim.

GPU jobs use partition overflow, name test, 512 GB host memory and a 24-hour limit. Candidate numerical/quality results and raw candidate evidence remain outstanding. Historical block-0 results below do not qualify the complete model. See examples/models/minimax_h3/README.md for the committed candidate gates.

Historical implementation and evidence

Draft. Opened before the claim on #420; it will be marked ready once the row is assigned. Stack: #483 → … → #494 → this PR, so it merges last. This PR's own diff contains the code and evidence commits plus the evidence-directory migration.

Evidence directory migration

Commit 03cd824b moves the one-block report.json and figure.png to reports/experiments/h3-one-block-b200/, updates documentation and the plotting example, and records both moves in the layout audit map.

Validation: both moved files retain identical SHA-256 hashes; the report retains its original 41e4293 measurement revision and environment. JSON parsing, migration mappings, documentation link targets, Python syntax, Ruff lint/format checks, and git diff --check pass. The existing plotting CLI successfully renders a temporary copy of the report without overwriting committed evidence. Old paths remain only in the migration map.

This follow-up changes static evidence locations and references only. No GPU numerical tests or benchmarks were rerun; the historical two-GPU results and four-/eight-GPU coverage limitations below remain unchanged.

Fast-math guard (2026-10-13)

From the #483 review. KERNEL_ALIGN_USE_FAST_MATH=1 adds --use_fast_math to the whole rl_engine._C build. That replaces expf/sinf/cosf/rsqrtf and FP32 division with approximations and enables FTZ, which breaks the H3 kernels' bitwise promises. Before the fix, a fast-math build of the top of the stack failed 33 H3 tests: the timestep_sinusoid_h3 (#483) and h3_rmsnorm (#487) bitwise tests, and the chain/block tests built on them. det_linear, adaln_row_gather and gate_residual still passed.

nvcc 13 defines no macro for fast math, so the guard lives in the build:

• build_tools/extensions.py lists every csrc/cuda/h3 source in H3_BITWISE_SOURCES and leaves them all out of fast-math builds;
• the bindings are gated on KERNEL_ALIGN_WITH_H3_KERNELS, so dispatch falls back to the PyTorch references;
• tests that assert CUDA dispatch skip through a shared require_h3_kernels fixture, and a build test asserts that no csrc/cuda/h3 source survives a fast-math build.

On this branch: merge 4ba3a8fb and 9cb10d03 (block dispatch-check skip). Tested on 2× B200 with pinned weights (tests/models/minimax_h3, tests/build, tests/runtime/test_dispatch.py):

Build	H3 kernels in _C	passed / skipped / failed
default	yes	545 / 7 / 0
KERNEL_ALIGN_USE_FAST_MATH=1	no (reference fallback)	196 / 356 / 0

Status

• Fixed: the flaky test_h3_adaln_modulation.py::TestFusedModulation::test_backward_keeps_table_gradient_fp32 in [WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/9) #486 (46347118), merged here. The test's golden rounded an FP64 SiLU where the kernel rounds the FP32 SiLU, and temb was unseeded; the kernel itself was deterministic. Full suite at 27bb0295 on 2× B200: 550 passed, 7 skipped in the default build, and 201 passed, 356 skipped in the fast-math build, 0 failed.

No known failures at 41e4293 (code) / caaf754 (evidence). 11 of the 17 block nodes run on interim operators or the provider replay until their rows land (see Bindings). The block-level promises below hold with those bindings.

What

RFC #420 row ws1_one_h3_block: one real H3 block, forward and backward, with every intermediate and a first-drift report.

• The block. Block 0 of MiniMax-H3@42ed227 runs as a 17-node graph, from the AdaLN projection to the second gated residual. It runs on packed FL2VA layouts whose position_ids and tags are bitwise equal to the pinned diffusers pipeline's build_packed_sequence; the hashes are pinned in a CPU test.
• Three executions per node. Each node runs as:
  • the diffusers replay (op for op, no diffusers import);
  • the RL-Kernel binding, both chained and isolated (fed diffusers' inputs);
  • an FP32 golden.
• Ownership. Each node names the RFC row that owns its arithmetic, and whether it is bound to that row's operator, an interim deterministic operator, or the provider replay.
• Forward report, per node:
  • repeat and batch-row bitwise checks;
  • chained and isolated agreement with diffusers (including the first differing packed row);
  • candidate and diffusers error against the golden;
  • the node's sha256;
  • first_{,isolated_,repeat_,batch_}drift.
• Backward report:
  • gradients of the block input, temb and all 12 block parameters (repeat, error vs golden);
  • the gradient reaching every node (repeat, batch row), with first_grad_{repeat,batch}_drift.
• Weights. Block-0 attention/FFN weights are pinned in a new manifest section (block_tensors). They are extracted by prepare_h3_weights.py --block from the same shard, so the existing conditioning file is unchanged.

Bindings

Node	Owning row	Bound to
adaln_projection	adaln_projection_3mod	row (#485), via the registry
norm1, norm2	h3_rmsnorm	row (#487), via the registry
residual_attn, residual_mlp	adaln_gate_residual	row (#488), via the registry
q/k/v_proj, o_proj, ffn_gate_up, ffn_down	h3_qkv_gemm, h3_attention_o_gemm, h3_ffn_gate_up_gemm, h3_ffn_down_gemm	interim: Triton FP32-accumulate GEMM, one BF16 rounding
q_norm, k_norm	h3_qk_rmsnorm_d128	interim: #487's plain H3RMSNormCudaOp on the 128-wide heads (bitwise equal to nn.RMSNorm)
rope_q, rope_k	h3_mm_rope_3axis_partial	reference: the provider replay, since no MM-RoPE kernel exists yet
attention	h3_full_attention	interim: DeterministicAttentionOp(causal=False)
swiglu	h3_swiglu	interim: SwiGLUCudaOp

When an owning row lands, swapping its node's candidate callable is the whole integration, and the report entries become that operator's block-level evidence.

Prior art & reuse decision

This is a validation closeout, not a new kernel; no operator is written here.

• diffusers. MiniMaxH3TransformerBlock is the executable reference. It is replayed op for op as the provider (h3_provider), which pins its dtypes and op order without importing diffusers.
• [ws1]: WS1 Full Qwen3-8B Dense Train-Inference Closeout #315 Qwen3-8B chain gate (rl_engine/validation/models/chain_gate.py). Its method is reused: per-node digests and first drift in graph order. Its graph is Qwen3's (causal, no AdaLN, Qwen shapes), so the H3 block is built on the h3_chain stage pattern of [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/9) #483.
• Interim operators are reused from RL-Kernel. The one decision taken here is the GEMM entry point. TritonDetGemmOp.__call__ reduces K as a tree with BF16 nodes; its FP32-accumulate kernel is equally deterministic and row-invariant, matches cuBLAS's accuracy, and is 9–17× faster (B200, error vs FP64):

Shape	Tree path	FP32 K loop → BF16	cuBLAS	Tree time	FP32 K loop time
264 × 5376 → 7168	7.0e-3	2.4e-3	2.4e-3	10.4 ms	1.2 ms
925 × 14336 → 5376	7.5e-3	2.6e-3	2.6e-3	47 ms	2.7 ms

Direct bitwise comparison of reuse candidates (2026-10-13). The H3 block is composed of the WS1 rows below it, and their reuse candidates are compared byte for byte with tools/validation/models/h3_bitwise_reuse.py (committed in #483, merged down the stack). It runs every candidate that h3_prior_art.candidates() defines on identical inputs and compares every output and leaf gradient with diffusers' path and with the rl-kernel op. Each row's report, written from a clean tree on its own branch: #483 timestep_sinusoid; #484 timestep_mlp; #485 adaln_projection; #486 adaln_row_gather; #487 norm_modulate; #488 gate_residual; #489 final_adaln_out.

Results

Generated from a clean tree at 41e4293 on an otherwise idle B200 (torch 2.13.0+cu130), with tools/validation/models/h3_block_replay.py --layouts tiny,small,medium: report.json.

S	First drift (chained / isolated)	Repeat or batch drift, any node or gradient	Output err: RL-Kernel / diffusers	Fwd ms	Fwd+bwd ms
264	adaln_projection / adaln_projection	none	5.21e-3 / 5.21e-3	7.4 vs 0.61	22.7 vs 2.8
925	adaln_projection / adaln_projection	none	7.87e-3 / 7.87e-3	30.1 vs 1.12	81.7 vs 5.0
3160	adaln_projection / adaln_projection	none	8.97e-3 / 8.97e-3	225 vs 3.8	577 vs 11.5

• Isolated bitwise promises. Every node that promises it is bitwise equal to diffusers on diffusers' inputs at every size: norm1/norm2, q_norm/k_norm, RoPE, and both gated residuals. The first drift is the first reduction whose tree differs from cuBLAS.
• Forward accuracy. Per-node error vs the FP32 golden is 0.93–1.20× diffusers'. The largest is the interim attention at S = 3160.
• Gradient accuracy:
  • The candidate is up to 7× more accurate than diffusers on adaln_proj.weight, adaln_proj.bias and temb. diffusers' index_select backward accumulates in BF16 atomics; the row operators use FP32 segment sums.
  • The candidate is within 1.16× on the other leaves, except attn.norm_q.weight at S = 3160 (9.5e-3 vs 5.5e-3). At S = 925 the same leaf is 2× more accurate than diffusers, so the 128-element gradient's error follows BF16 rounding, not one side.
• Control H-C3, reported, not promised. Permuting the packed rows with their positions and indices changes 79 / 450 / 2199 output rows (max 8). Attention sums keys in packed order; settling this belongs to h3_full_attention.
• Speed. The 12–60× forward cost is the interim GEMMs and attention (FP32 scores materialised), not the row operators.

Tests

RL_KERNEL_H3_WEIGHTS=<dir> CUDA_VISIBLE_DEVICES=0,1 python -m pytest tests/models/minimax_h3 -q

501 passed, 5 skipped on two B200s, including the 9 new tests in test_h3_one_block.py. The new tests check:

• CPU:
  • the layouts are pinned to diffusers;
  • the graph is well formed;
  • MM-RoPE rotates 96 of 128 channels.
• GPU forward (264, 925):
  • every node repeats and is batch-row bitwise;
  • promised nodes are isolated-bitwise;
  • node and output errors are at most 1.25 × diffusers' + 1e-3.
• GPU backward (264):
  • every leaf repeats;
  • dX is batch-row bitwise;
  • there is no gradient drift;
  • leaf errors are at most 1.5 × diffusers' + 5e-3.

Scope / Known limitations

• ROCm. CUDA only. The interim attention and Triton GEMM build for ROCm, but the block has not been run on AMD hardware.
• Interim bindings. 11 of 17 nodes are interim or reference. Their speed and rounding are not their owning rows' final ones, and RoPE carries no RL-Kernel evidence until h3_mm_rope_3axis_partial lands.
• Stand-in inputs. The block input is a seeded stand-in (the input projections are other rows), and temb is the provider's (the conditioning chain is [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/9) #483–[WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/9) #486).
• Block 0 only. The full 50-block transformer and the refiner are ws1_full_h3_transformer.

Documentation repair at 9317a5e1: replace 14 report links outside the MkDocs docs tree with immutable GitHub/raw URLs at the evidence-containing commit 68e472b7. The strict remote docs build passed. No evidence files or numerical results were changed.