Add packed ternary GGUF export for CAT-Q

[genggng]

Released CAT-Q checkpoints only shipped the learnable parameters, and the one
export path available produced a fake-quantized Hugging Face model: the values
were ternary but every weight still occupied 16 bits. This adds a real export,
so a checkpoint can be turned into a deployable file where each quantized weight
takes 2 bits.

What is added

main.py --export_gguf_path writes the restored checkpoint straight to a single
GGUF file. Quantized projections are stored as Q2_0, the group-128 ternary
weight type of the Bonsai
llama.cpp fork (34 bytes per 128 weights, 2.125 bits per weight). Everything the
CAT-Q recipe leaves in floating point - token embeddings, LM head, norms and the
MoE router - is written exactly as a stock llama.cpp conversion would.

File	Role
quantize/ternary_export.py	Recovers the ternary codes and per-group scales from the quantizer
quantize/q2_0.py	The Q2_0 block layout and packing
quantize/gguf_export.py	Metadata, vocabulary and tensor writing
export_gguf.sh	Launcher matching export_model.sh
deployment/README.md	Export-and-serve walkthrough

Exporting stays a pure CAT-Q step: it reads the checkpoint and writes the GGUF
with no intermediate model and no inference runtime involved. Beyond the current
requirements it only needs the gguf package. llama.cpp is needed to run the
result, not to build it.

Dense (Qwen3, LLaMA) and MoE (Qwen3-MoE) checkpoints are both supported. For MoE
models the per-expert gate_proj/up_proj/down_proj weights are packed and
stacked into the ffn_*_exps tensors the runtime expects.

Correctness

ternary_export.extract_ternary does not re-estimate anything from the merged
float weights: it re-runs the quantizer and keeps the two factors it works with,
then asserts that scale * ternary reproduces quantizer(weight) exactly before
the tensor is handed on. A packed export is therefore numerically identical to
the fake-quantized model it replaces.

tests/test_ternary_export.py (220 lines) covers the round trip, the Q2_0 bit
layout against the reference dequantization, the LLaMA q/k row permutation and
the MoE expert stacking.

Large models

By default the exporter keeps the whole 16-bit model resident while packing,
which needs about the size of the original model plus the size of the GGUF.
Qwen3-235B-A22B does not fit that way on a 512 GiB host, so --gguf_low_memory
drops each weight as soon as it has been packed and spools the GGUF through a
temporary file.

Two things had to be fixed to make that actually work:

• Folding the merged floats back into the model faulted in a private copy of
  every memory-mapped page the loader handed out, even though the low-memory
  path never reads those values again. Those writes are now skipped.
• Freeing a decoder layer worth of tensors on each iteration grew the glibc
  dynamic mmap threshold, after which same-sized allocations came from the heap
  and their memory was never returned - about 10 GiB per layer late in a run.
  The threshold is now pinned.

Peak host memory for Qwen3-235B-A22B drops from over 500 GiB to roughly 62 GiB,
and the output is byte-for-byte identical to a default export (verified on
Qwen3-1.7B and Qwen3-30B-A3B). The flag cannot be combined with --tasks or
--export_model_path, since the 16-bit weights are gone by the time packing ends.

Released files

All eight checkpoints have been exported and uploaded next to their parameters
on IntelLabsChina/CAT-Q:

Model	GGUF
Llama2-7B	2.25 GB
Qwen3-1.7B	1.00 GB
Qwen3-4B	1.75 GB
Qwen3-8B	4.34 GB
Qwen3-14B	6.63 GB
Qwen3-32B	11.41 GB
Qwen3-30B-A3B	9.24 GB
Qwen3-235B-A22B	64.80 GB

Testing

• python -m compileall -q .
• python -m unittest discover -s tests -p "test_*.py" - 29 tests pass
• End-to-end export of all eight checkpoints, each loaded by the Bonsai runtime