Laguna: mixed-precision quant support (APEX IQ4_XS/Q6_K, official BF16) + metadata-driven rope

[jasontitus]

Laguna: mixed-precision quant support (APEX IQ4_XS/Q6_K, official BF16) + metadata-driven rope

Adds support for the mixed-precision Laguna S 2.1 exports that ds4 currently
rejects, closes the resulting prefill gap, and reports two artifact issues
found along the way.

Branch: laguna-apex-quants (four commits on top of laguna-s2.1).
All numbers measured on an M5 Max 128 GB, Metal backend, on this branch as
submitted, so they should reproduce directly.

What this enables

Export	Before	After
Myric/Laguna-S-2.1-APEX-GGUF i-quality (68.9 GiB, 5.03 bpw)	unsupported Laguna quantization layout marker q6_k	loads and runs
poolside official Q4_K_Mx40-BF16x8 (89.4 GiB)	routed bf16 rejected	loads and runs
Any 1M-rope Laguna export	expected context_length=262144 … got 1048576	loads

Commits

1. d3935ba — APEX quant support. New IQ4_XS type end-to-end (block struct,
   codebook, CPU reference dequant, Metal QMV pair-swiglu/down kernels,
   mul_mm_id instantiations), a Q6_K routed pair-swiglu kernel, Q6_K
   attention/embedding/output acceptance, a third Laguna weight layout beside
   legacy and signal-q8, and Laguna context_length/rope scale+attn factors
   adopted from GGUF metadata after a coherence check
   (orig_ctx × factor == context_length) instead of pinning one export.
2. 0f2d58d — first optimisations. Batched Q6_K dense matmuls routed
   through the tiled GEMM instead of a per-token QMV; vectorised IQ4_XS QMV
   inner loops.
3. 552bc4c — close the prefill gap. Vectorised the dequantize_q6_K /
   dequantize_iq4_xs tile-staging hooks (word loads, branchless extraction,
   identical per-element arithmetic).
4. 96a710c — bf16 routed experts for the official mixed-precision export.

Measured (vs the Q4_K_M export)

Perplexity: wikitext-style local corpus, 8160 scored tokens, ctx 8192.
Prefill: 15K-token prompt, 6 interleaved pairs with cooldowns.

	Q4_K_M	APEX i-quality
perplexity	14.098	13.265 (−5.9 %)
prefill	454 ±24 t/s	442 ±18 t/s (−2.6 % ±3.0 — parity)
decode	52–54 t/s	parity
size	63.6 GiB	68.9 GiB
max context	262 144	1 048 576

The prefill deficit was −12.4 % ±1.7 before commit 552bc4c; the tile-dequant
hooks were essentially the whole gap, since they stage every mul_mm
(attention) and mul_mm_id (routed-expert) tile.

Two findings worth flagging

1. block_q6_K is 210 bytes, so its payload is not 4-byte aligned.
Consecutive blocks alternate between 0- and 2-mod-4 addresses, making ushort
the widest always-safe load. An earlier uint version of the dequant hook
silently misdecoded the misaligned half of the blocks: perplexity moved only
+0.25 %, but greedy generation degenerated into loops. Teacher-forced scoring
barely notices; autoregressive decode compounds it. Every kernel change here is
gated on bit-exact perplexity (ppl=13.265259871 on this branch) rather than a tolerance, which
is what caught it. Alignment should be derived from the block stride, not the
field offset — block_iq4_xs (136 B) keeps 32-bit loads for that reason.

2. The three mixed-precision exports rank in a surprising order. Same
corpus, same settings:

export	size	perplexity
Q4_K_M	63.6 GiB	14.098
Myric/…-APEX-GGUF i-quality	68.9 GiB	13.265
poolside official Q4_K_Mx40-BF16x8	89.4 GiB	13.399

The community APEX build edges out the official mixed-precision build by
1.0 % while being 23 % smaller. The two recipes make opposite
bets: APEX raises precision on the edge layers (q5_k/q6_k on layers 1-9 and
38-47) and compresses the middle to iq4_xs, while the official build leaves
layers 1-39 at q4_k and raises only the last eight to bf16. On this corpus the
edge-weighted allocation wins, which is consistent with quantisation error
mattering more in early layers than late ones.

Sizing note for 128 GB machines: the official build plans 89.88 GiB at ctx
8192 and roughly 107 GiB at 256K, and Laguna has no --ssd-streaming
fallback (--ssd-streaming is not implemented for Laguna S 2.1 yet). It needs
a mostly idle machine — a first attempt with ~30 GiB held by other
applications thrashed and never became resident. Since it now ships under the
same laguna-s-2.1-Q4_K_M.gguf filename as the 63.6 GiB build it replaced,
users re-pulling "Q4_K_M" get a substantially larger model than before.

Notes for reviewers

• Metal-4 tensor matmuls remain suppressed in the Laguna graph prefill path
  (existing ds4_gpu_set_tensor_matmul_suppressed(true) call and its
  route-stability comment). The Q6_K MPP direct-RHS instantiations added here
  are consequently dormant on that path by design; they engage wherever MPP is
  otherwise permitted.
• The rope change validates rather than trusts: a GGUF whose
  context_length disagrees with original_context_length × factor is
  rejected with a diagnostic instead of loading.
• APEX perplexity is bit-identical before and after the bf16 commit, i.e. the
  new type does not disturb existing paths.
• Decode throughput across the three: official bf16 build 56.9 t/s, APEX
  51-57 t/s, Q4_K_M 52-54 t/s. bf16 needs no unpacking at all, so it is
  marginally the fastest per token despite being the largest to stream.

Related

• Update Laguna to corrected GGUF revision #613 (corrected GGUF revision) — while testing this branch, the stale
  706fa69 revision pinned in download_model.sh reproduced the looping /
  "never emits the tool call" behaviour that revision e2ccc05 fixes. Weights
  are identical between the two (416-byte file delta; perplexity agrees to nine
  significant digits on the same build); only the chat template differs, by 121 lines including
  enable_thinking default false→true. With the stale artifact a thinking
  agent run never produced output; with the corrected one it succeeded in 3 of
  4 runs. Worth merging.
• Fix Laguna KV ring commits for oversized prefills #614 (KV-ring commits for oversized prefills) — not exercised here, but
  sweep-style workloads prefill far past the 512-row window, so it likely
  affects the same users.