Add packed ternary GGUF export for CAT-Q

README.md

@@ -17,12 +17,12 @@
 
 | Project | Venue | Public Release |
 | --- | --- | --- |
-| [CAT-Q](projects/cat-q) | ICML 2026 Oral | Model checkpoints, inference, and evaluation code |
+| [CAT-Q](projects/cat-q) | ICML 2026 Oral | Model checkpoints, inference, evaluation, and packed ternary deployment code |
 
 ## Latest News
 
 <!-- `[Stay tuned]` We are preparing to release the CAT-Q training code, etc. -->
 - `[04/08/2026]` 🔥 [The technical report of ScaleQ-1.58](https://arxiv.org/abs/2608.01078) "**Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization**" is now available on arXiv.
-- `[22/07/2026]` 🚀 [The CAT-Q model checkpoints](projects/cat-q) (including Qwen3-1.7B/4B/8B/14B/32B, Llama2-7B, Qwen3-30B-A3B and Qwen3-235B-A22B), inference, and evaluation code are now available.
+- `[22/07/2026]` 🚀 [The CAT-Q model checkpoints](projects/cat-q) (including Qwen3-1.7B/4B/8B/14B/32B, Llama2-7B, Qwen3-30B-A3B and Qwen3-235B-A22B), inference, evaluation, and **real ternary deployment** code are now available.
 - `[25/06/2026]` 🔥 [The CAT-Q paper](https://arxiv.org/abs/2606.26650) is now available on arXiv.
 - `[01/05/2026]` 🎉Our paper "**CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs**" is accepted to **ICML 2026 as an oral**. The project page for our sliding-layer reconstruction framework used in CAT-Q is available at [SliderQuant (ICLR 2026)](https://github.com/deep-optimization/SliderQuant).

projects/cat-q/README.md

@@ -15,10 +15,11 @@ This repository contains the official implementation of **CAT-Q (ICML 2026 Oral)
 
 ## Latest News
 
-- `[Stay tuned]` We are preparing to release the CAT-Q training code.
-- `[22/07/2026]` We release the CAT-Q model checkpoints, inference, and evaluation code.
-- `[25/06/2026]` The [CAT-Q paper](https://arxiv.org/abs/2606.26650) is available on arXiv.
-- `[01/05/2026]` 🎉🎉🎉**CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs** is accepted to ICML 2026 as an oral.
+- `[Stay tuned]` We are preparing to release the CAT-Q training code, etc. 
+- `[04/08/2026]` [The technical report](https://arxiv.org/abs/2608.01078) "**Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization**" is now available on arXiv.
+- `[22/07/2026]` 🎉🎉🎉The CAT-Q model checkpoints, **scaling from Qwen3-1.7B all the way to Qwen3-235B-A22B** (Qwen3-1.7B/4B/8B/14B/32B, Llama2-7B, Qwen3-30B-A3B and Qwen3-235B-A22B), inference, evaluation, and **real ternary deployment** code are now available.
+- `[25/06/2026]` [The CAT-Q paper](https://arxiv.org/abs/2606.26650) is now available on arXiv.
+- `[01/05/2026]` 🎉🎉🎉Our paper "**CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs**" is accepted to **ICML 2026 as an oral**. The project page for our sliding-layer reconstruction framework used in CAT-Q is available at [SliderQuant (ICLR 2026)](https://github.com/deep-optimization/SliderQuant).
 
 ## Overview
 
@@ -49,6 +50,7 @@ Using only 512 calibration samples, CAT-Q scales W1.58 quantization from 1.7B de
 - [Installation](#installation)
 - [Evaluation](#evaluation)
 - [Hugging Face Export](#hugging-face-export)
+- [Packed Ternary Deployment](#packed-ternary-deployment)
 - [Citation](#citation)
 - [Acknowledgement](#acknowledgement)
 - [License](#license)
@@ -87,6 +89,9 @@ The following W1.58A16 checkpoints are available on Hugging Face:
 | Qwen3-235B-A22B | MoE | [qwen3-moe-235B-A22B](https://huggingface.co/IntelLabsChina/CAT-Q/tree/main/qwen3-moe-235B-A22B) |
 
 All checkpoints are hosted under [IntelLabsChina/CAT-Q](https://huggingface.co/IntelLabsChina/CAT-Q).
+Every folder holds the learnable CAT-Q parameters (`parameters.pth`) with the config that
+produced them, plus a ready-to-run packed ternary `*-catq-q2_0.gguf`; see
+[deployment/README.md](deployment/README.md) for how to serve it.
 
 > **Note:** The code was refactored for open-source release, so checkpoint accuracy may differ slightly from the paper results (typically within ±0.2 percentage points).
 
@@ -137,6 +142,16 @@ Export the restored model as a fake-quantized Hugging Face model:
 
 The exported model retains the original Hugging Face architecture and stores merged fake-quantized floating-point weights; it is not a packed ternary checkpoint.
 
+## Packed Ternary Deployment
+
+Export the restored model as a packed ternary GGUF, where each quantized weight occupies 2 bits next to one fp16 scale per group of 128:
+
+```bash
+./export_gguf.sh
+```
+
+Conversion is self-contained: it reads the checkpoint and writes the GGUF, with no intermediate model and no inference runtime involved. The result runs on the ternary kernels of the [Bonsai](https://github.com/PrismML-Eng/Bonsai-demo) runtime, with real weight compression rather than fake quantization. See [`deployment/README.md`](deployment/README.md) for the full export-and-serve walkthrough.
+
 ## Citation
 
 If CAT-Q is useful in your research, please cite:
@@ -152,7 +167,7 @@ If CAT-Q is useful in your research, please cite:
 
 ## Acknowledgement
 
-CAT-Q is implemented based on [SliderQuant](https://github.com/deep-optimization/SliderQuant).
+CAT-Q is implemented based on [SliderQuant](https://github.com/deep-optimization/SliderQuant). Packed ternary deployment builds on the group-128 ternary kernels of [Bonsai](https://github.com/PrismML-Eng/Bonsai-demo) and its [llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp).
 
 ## License
 

projects/cat-q/deployment/README.md

@@ -0,0 +1,157 @@
+# Deploying CAT-Q Models with Packed Ternary Weights
+
+The [Hugging Face export](../README.md#hugging-face-export) writes a *fake-quantized*
+model: the weights are ternary, but each one still occupies 16 bits and runs through
+ordinary floating-point matrix multiplication.
+
+This document covers the other export path, which stores the quantized projections as
+**group-128 packed ternary weights** and runs them through the ternary kernels of
+[llama.cpp](https://github.com/PrismML-Eng/llama.cpp) as used by the
+[Bonsai demo](https://github.com/PrismML-Eng/Bonsai-demo).
+
+CAT-Q quantizes a weight group as
+
+```
+W_g = s_g * T_g,    T_g in {-1, 0, +1},    |g| = 128
+```
+
+which is exactly the `Q2_0` block type of that runtime (`block_q2_0`: one fp16 scale
+plus 128 two-bit codes, 34 bytes, 2.125 bits per weight). The exporter takes the codes
+and scales straight out of the CAT-Q quantizer and packs them, so the deployed weights
+are bit-for-bit the ones the fake-quantized model uses. Embeddings, norms, the LM head
+and MoE routers stay in floating point, as they are outside the quantized set.
+
+The implementation lives with the rest of the quantization code:
+`quantize/ternary_export.py` recovers the codes and scales, `quantize/q2_0.py` packs
+them into `Q2_0` blocks, and `quantize/gguf_export.py` writes the GGUF.
+
+## 1. Export the model
+
+Exporting is a pure CAT-Q step: it reads the checkpoint and writes the GGUF directly.
+No inference runtime is involved, and no intermediate model is produced. Beyond the
+project requirements it only needs the `gguf` package (`pip install gguf`, already in
+`pyproject.toml`).
+
+```bash
+cd BitTern/projects/cat-q
+
+# select the checkpoint, exactly as for evaluation
+#   result_dir=configs/qwen3-4b     in task_list.conf
+./export_gguf.sh
+```
+
+This writes `configs/<model>/export-gguf/<net>-catq-q2_0.gguf`.
+
+The launcher is a thin wrapper; the same thing can be run directly:
+
+```bash
+python main.py \
+  --config configs/qwen3-4b/config.yaml \
+  --checkpoint configs/qwen3-4b/parameters.pth \
+  --output_dir configs/qwen3-4b/export-gguf \
+  --export_gguf_path configs/qwen3-4b/export-gguf/Qwen3-4B-catq-q2_0.gguf
+```
+
+`--export_gguf_path` takes either a `.gguf` file or a directory, in which case the file
+is named `<net>-catq-q2_0.gguf`. `--gguf_float_type {f16,bf16,f32}` selects the dtype of
+the tensors CAT-Q leaves in floating point and defaults to `f16`; norms and MoE routers
+are always `F32`, as in a stock llama.cpp conversion.
+
+Dense (Qwen3, LLaMA) and MoE (Qwen3-MoE) checkpoints are both supported. For MoE models
+the per-expert `gate_proj`/`up_proj`/`down_proj` weights are packed and stacked into the
+`ffn_*_exps` tensors the runtime expects, while the router stays in `F32`.
+
+### Very large checkpoints
+
+By default the exporter keeps the whole 16-bit model resident while it packs, which needs
+roughly the size of the original model plus the size of the GGUF. Add `--gguf_low_memory`
+when that does not fit - Qwen3-235B-A22B, for instance, needs it on a 512 GiB host:
+
+```bash
+python main.py \
+  --config configs/qwen3-moe-235B-A22B/config.yaml \
+  --checkpoint configs/qwen3-moe-235B-A22B/parameters.pth \
+  --output_dir configs/qwen3-moe-235B-A22B/export-gguf \
+  --export_gguf_path configs/qwen3-moe-235B-A22B/export-gguf \
+  --gguf_low_memory
+```
+
+Each weight is then dropped as soon as it has been packed and the GGUF is assembled
+through a temporary file (set `TMPDIR` to a filesystem with room for the result), which
+holds host memory to about the size of the packed model. The file is byte-for-byte the
+same as without the flag. Because the 16-bit weights are gone by the time packing ends,
+the flag cannot be combined with `--tasks` or `--export_model_path`.
+
+## 2. Get a runtime with ternary kernels
+
+The GGUF needs a llama.cpp build with group-128 ternary kernels:
+
+```bash
+git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git
+export LLAMA_CPP_DIR=$PWD/llama.cpp
+```
+
+The easiest way to build it is with the Bonsai demo's own scripts, which also fetch the
+runtime for you if it is missing:
+
+```bash
+git clone https://github.com/PrismML-Eng/Bonsai-demo.git
+cd Bonsai-demo
+./scripts/build_cuda_linux.sh "$LLAMA_CPP_DIR"   # CUDA; build_cpu_linux.sh / build_mac.sh also exist
+```
+
+Binaries land in `Bonsai-demo/bin/<backend>/`. Prebuilt binaries and other backends
+(Metal, Vulkan, ROCm) are described in the Bonsai demo README.
+
+## 3. Run it
+
+The result is a standard GGUF file, so any tool from the runtime works with it:
+
+```bash
+BIN=/path/to/Bonsai-demo/bin/cuda
+export LD_LIBRARY_PATH="$BIN:$LD_LIBRARY_PATH"
+
+# one-off generation
+"$BIN/llama-cli" -m Qwen3-4B-catq-q2_0.gguf -ngl 99 -p "Explain ternary quantization."
+
+# OpenAI-compatible server + web UI on http://localhost:8080
+"$BIN/llama-server" -m Qwen3-4B-catq-q2_0.gguf -ngl 99 -c 8192 -fa on
+
+# throughput and memory
+"$BIN/llama-bench" -m Qwen3-4B-catq-q2_0.gguf -ngl 99
+```
+
+Notes for Qwen3 checkpoints, which are thinking models:
+
+- `--reasoning-format deepseek` keeps `<think>` blocks out of `message.content` and puts
+  them in `message.reasoning_content`.
+- `--chat-template-kwargs '{"enable_thinking": false}'` turns thinking off.
+- `-fit off` stops the server from growing the KV cache to fill the device memory, which
+  is worth setting when measuring the memory footprint.
+
+For model management, the web UI, tool calling, speculative decoding and non-Linux
+platforms, follow the [Bonsai demo](https://github.com/PrismML-Eng/Bonsai-demo)
+documentation; a CAT-Q GGUF can be used wherever it expects a Bonsai ternary model.
+
+## Reference numbers
+
+Qwen3-4B, single NVIDIA L40, context 2048, measured with `llama-bench`:
+
+| | packed ternary | fake-quantized `F16` | ratio |
+| --- | ---: | ---: | ---: |
+| file size | 1.63 GiB | 7.50 GiB | 4.60x |
+| device memory | 2093 MiB | 8745 MiB | 4.18x |
+| decode (tg128) | 285.1 t/s | 93.5 t/s | 3.05x |
+
+252 of the 398 tensors are packed ternary and hold 3.63 B of the weights at 2.125 bits
+each; the remaining floating-point tensors (mostly the token embedding) account for most
+of what is left, which is why the whole-file ratio is below the 7.53x of the quantized
+part alone. Task accuracy matches the fake-quantized model to within run-to-run noise.
+
+## Acknowledgement
+
+The packed ternary format and the kernels used here come from
+[Bonsai](https://github.com/PrismML-Eng/Bonsai-demo) by PrismML and its
+[llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) (branch `prism`), built on
+[llama.cpp](https://github.com/ggml-org/llama.cpp). We thank their authors for making
+efficient ternary inference available to the community.

projects/cat-q/export_gguf.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Export the selected CAT-Q checkpoint as a packed ternary GGUF model.
+#
+# The GGUF is written directly from the checkpoint; no llama.cpp checkout and no
+# intermediate model are involved.  See deployment/README.md for how to run the
+# result.
+set -euo pipefail
+
+source ./task_list.conf
+source ./scripts/gpu_lock.sh
+
+config_path="${result_dir}/config.yaml"
+checkpoint_path="${result_dir}/parameters.pth"
+export_path="${result_dir}/export-gguf"
+if [[ ! -f "${config_path}" ]]; then
+  echo "Config file not found: ${config_path}" >&2
+  exit 2
+fi
+if [[ ! -f "${checkpoint_path}" ]]; then
+  echo "Checkpoint not found: ${checkpoint_path}" >&2
+  exit 2
+fi
+
+mkdir -p "${export_path}"
+catq_acquire_gpus 1 "${THRESHOLD}" "${WAIT_MODE}" "${WAIT_INTERVAL}"
+trap catq_release_gpus EXIT
+trap 'exit 130' INT
+trap 'exit 143' TERM
+
+python main.py \
+  --config "${config_path}" \
+  --output_dir "${export_path}" \
+  --export_gguf_path "${export_path}" \
+  --gguf_float_type f16 \
+  --checkpoint "${checkpoint_path}"

projects/cat-q/main.py

@@ -7,7 +7,10 @@
 
 def build_parser():
     parser = argparse.ArgumentParser(
-        description="Load released CAT-Q parameters for evaluation or fake-quantized Hugging Face export."
+        description=(
+            "Load released CAT-Q parameters for evaluation, fake-quantized Hugging Face "
+            "export, or packed ternary GGUF export."
+        )
     )
     parser.add_argument(
         "--config",
@@ -29,6 +32,30 @@ def build_parser():
         default=None,
         help="Directory for the fake-quantized HF model",
     )
+    parser.add_argument(
+        "--export_gguf_path",
+        type=str,
+        default=None,
+        help=(
+            "Where to write the packed ternary GGUF: a .gguf file, or a directory "
+            "in which <net>-catq-q2_0.gguf is created"
+        ),
+    )
+    parser.add_argument(
+        "--gguf_float_type",
+        choices=["f16", "bf16", "f32"],
+        default="f16",
+        help="Dtype for the GGUF tensors CAT-Q keeps in floating point",
+    )
+    parser.add_argument(
+        "--gguf_low_memory",
+        action="store_true",
+        help=(
+            "Free each 16-bit weight as soon as it has been packed and spool the "
+            "GGUF through a temporary file; needed for models that do not fit in "
+            "host memory twice. Cannot be combined with --tasks or --export_model_path"
+        ),
+    )
     parser.add_argument("--net", type=str, default=None)
     parser.add_argument(
         "--quant_layer_list",
@@ -112,12 +139,36 @@ def parse_arguments(argv=None):
         parser.error("--model is required (directly or through --config)")
     if not args.checkpoint:
         parser.error("--checkpoint is required")
-    if not args.tasks and not args.export_model_path:
-        parser.error("select at least one action: --tasks or --export_model_path")
+    if not args.tasks and not args.export_model_path and not args.export_gguf_path:
+        parser.error("select at least one action: --tasks, --export_model_path, or --export_gguf_path")
+    if args.gguf_low_memory:
+        if not args.export_gguf_path:
+            parser.error("--gguf_low_memory requires --export_gguf_path")
+        if args.tasks or args.export_model_path:
+            parser.error(
+                "--gguf_low_memory discards the 16-bit weights while packing, so it "
+                "cannot be combined with --tasks or --export_model_path"
+            )
     args.ignored_config_keys = ignored_config_keys
     return args
 
 
+def _save_hf_model(lm, directory, logger):
+    directory.mkdir(parents=True, exist_ok=True)
+    lm.model.save_pretrained(directory)
+    lm.tokenizer.save_pretrained(directory)
+    logger.info("Saved fake-quantized Hugging Face model to %s", directory)
+
+
+def _gguf_outfile(args):
+    path = Path(args.export_gguf_path)
+    if path.suffix == ".gguf":
+        path.parent.mkdir(parents=True, exist_ok=True)
+        return path
+    path.mkdir(parents=True, exist_ok=True)
+    return path / f"{args.net}-catq-q2_0.gguf"
+
+
 def main(argv=None):
     args = parse_arguments(argv)
 
@@ -146,15 +197,31 @@ def main(argv=None):
         args.net = args.model.rstrip("/").split("/")[-1]
     args.quant_rate = 1.0
 
+    exporter = None
+    if args.export_gguf_path:
+        from quantize.gguf_export import TernaryGGUFExporter
+
+        exporter = TernaryGGUFExporter(
+            args.model,
+            _gguf_outfile(args),
+            float_type=args.gguf_float_type,
+            low_memory=args.gguf_low_memory,
+        )
+
     lm = LMClass(args)
-    merge_catq_checkpoint(lm, args, logger)
+    merge_catq_checkpoint(
+        lm,
+        args,
+        logger,
+        ternary_sink=exporter.capture if exporter else None,
+        release_packed_weights=args.gguf_low_memory,
+    )
 
     if args.export_model_path:
-        export_dir = Path(args.export_model_path)
-        export_dir.mkdir(parents=True, exist_ok=True)
-        lm.model.save_pretrained(export_dir)
-        lm.tokenizer.save_pretrained(export_dir)
-        logger.info("Saved fake-quantized Hugging Face model to %s", export_dir)
+        _save_hf_model(lm, Path(args.export_model_path), logger)
+
+    if exporter is not None:
+        logger.info("Saved packed ternary model to %s", exporter.write(lm.model))
 
     if args.tasks:
         evaluate(lm, args, logger)