feat(imf): Interscript Model Format v1 — spec, validator, packer

.github/workflows/test.yml

@@ -23,7 +23,7 @@ jobs:
       - name: Install (dev extras only — train/export need GPU + system deps)
         run: |
           python -m pip install --upgrade pip
-          pip install -e ".[dev]"
+          pip install -e ".[dev,export]"
       - name: Run pytest
         run: PYTHONPATH=src python -m pytest tests/ -v
       - name: Lint (ruff)
@@ -38,7 +38,7 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.11"
-      - run: pip install -e ".[dev]"
+      - run: pip install -e ".[dev,export]"
       - name: CLI smoke test
         run: |
           PYTHONPATH=src python -m src.cli list

.gitignore

@@ -17,6 +17,7 @@ data/augmented/
 
 # Model artifacts (Git-LFS tracked if committed at all)
 models/*.onnx
+models/**/*.zip
 models/*.safetensors
 models/*.ckpt
 *.tmp

LICENSE

@@ -0,0 +1,30 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Ribose Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice,
+   this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -98,6 +98,11 @@ export tests are gated behind `@pytest.mark.gpu` and require the
 
 ## Distribution
 
+Models ship as **IMF v1** zips (Interscript Model Format — spec in
+[`docs/imf-v1.md`](./docs/imf-v1.md)): byte-level tokenizer only, ONNX
+opset 14, sha256-verified graphs, metrics traceable to `RESULTS.md`
+anchors. Build/validate with `PYTHONPATH=src python -m imf pack|validate`.
+
 Models reach end users through three channels (full plan in
 [`TODO.distribution/`](./TODO.distribution/)):
 
@@ -117,5 +122,4 @@ completes.
 
 ## License
 
-MIT. Model weights are released under their own licenses (see
-`docs/model_card.md` per task).
+BSD-3-Clause, for code and model weights alike (see `LICENSE`).

docs/imf-v1.md

@@ -0,0 +1,132 @@
+# Interscript Model Format (IMF) v1
+
+IMF v1 is the versioned, portable artifact of **interscript-ml** — the
+phonological layer of Interscript. A model.zip is adoptable on its own
+terms, like ONNX itself: any runtime that can open a zip, sha256 a file,
+and run two ONNX sessions can serve the model. Adopting the artifact does
+not require adopting our training code.
+
+The format exists to make three guarantees:
+
+1. **One runtime everywhere.** v1 supports exactly one tokenizer: raw
+   UTF-8 bytes (pad=0, EOS=1, ByT5 convention). No vocab files, no
+   sentencepiece, no per-model tokenization code in Ruby/TS/Python.
+   Non-byte models enter via distillation (TODO.runtime-arch/07), never
+   via a second tokenizer system.
+2. **Old-runtimes load it.** Opset is pinned to 14 because the Ruby
+   `onnxruntime` gem bundles an old ORT that cannot load higher opsets
+   (verified the hard way — secryst PR #44). The validator enforces
+   graph opset == metadata opset <= 14.
+3. **Every number is traceable.** Metrics in metadata must cite a
+   `RESULTS.md` anchor; the parity block records ONNX-vs-reference
+   agreement. Numbers that cannot be traced do not ship.
+
+## Zip layout
+
+```
+model.zip
+├── metadata.yaml      # manifest (schema below) — never hand-write the
+│                      # sha256 block; `imf pack` computes it
+├── encoder.onnx       # required. inputs: [input_ids], dynamic batch/seq
+├── decoder.onnx       # required (fallback path). inputs: [input_ids,
+│                      #   encoder_hidden_states] -> [logits]
+├── decoder-kv.onnx    # optional (default artifact when present).
+│                      #   inputs add past_*; outputs add present_*
+└── README.md          # required. Usage in all three APIs
+```
+
+## metadata.yaml schema
+
+| Field | Type | Constraint |
+|---|---|---|
+| `format` | str | must be `imf-v1` |
+| `id` | str | `<name>-<major>.<minor>`, lowercase segments, e.g. `khm-latn-1.0` |
+| `task` | enum | `g2p` \| `diacritization` \| `translit` |
+| `source_script` | str | ISO 15924 script code (e.g. `Khmr`) |
+| `target` | str | target script or scheme (e.g. `Latn`) |
+| `tokenizer` | enum | `bytes` (the only v1 value) |
+| `opset` | int | 7..14; must equal the graphs' opset |
+| `decoder` | enum | `plain` \| `kv` (`kv` requires decoder-kv.onnx) |
+| `precision` | enum | `fp32` \| `fp16` \| `int8` |
+| `license` | str | non-empty (strict gate) |
+| `trained_from` | str | repo + run/checkpoint id |
+| `metrics` | list | `{name, value, protocol, source}`; `source` must be a `RESULTS.md#anchor` (strict gate) |
+| `parity` | map? | `{samples, cer_delta}`; strict gate: samples >= 500, cer_delta <= 0.2pp |
+| `sha256` | map | every `*.onnx` member -> hex digest; no dangling entries |
+
+The `id` does not encode precision: `khm-latn-1.0-fp16.zip` and
+`khm-latn-1.0-int8.zip` share id `khm-latn-1.0`; the model index
+(TODO.runtime-arch/08) resolves channel and precision.
+
+Metrics blocks are generated from `docs/RESULTS.md`, never hand-written
+(TODO.runtime-arch/10). Parity is produced by the WO03 gate.
+
+## Example
+
+```yaml
+format: imf-v1
+id: khm-latn-1.0
+task: translit
+source_script: Khmr
+target: Latn
+tokenizer: bytes
+opset: 14
+decoder: plain
+precision: fp16
+license: BSD-3-Clause
+trained_from: secryst train_khmer_byt5.py run-001 (secryst-checkpoints:/khmer_byt5/run-001/best)
+metrics:
+  - name: cer
+    value: 27.42
+    protocol: "greedy decode; 895 held-out pairs; split 16,120/895/895 seed 42"
+    source: secryst/docs/RESULTS.md#khmer-transliteration-2026-08-14
+  - name: em
+    value: 59.66
+    protocol: "greedy decode; 895 held-out pairs; split 16,120/895/895 seed 42"
+    source: secryst/docs/RESULTS.md#khmer-transliteration-2026-08-14
+parity:
+  samples: 500
+  cer_delta: 0.03
+sha256:
+  encoder.onnx: a4a4eb...
+  decoder.onnx: b54b5c...
+```
+
+## Validation
+
+Two levels (`src/imf/validator.py`):
+
+- **Base** — what every runtime does on load: zip integrity (CRC),
+  required members, metadata parses, every `.onnx` sha256-verified,
+  graph opset matches metadata and stays <= 14, decoder contract
+  (`input_ids` / `encoder_hidden_states` / `past_*`-`present_*` names).
+- **Strict** — the release gate: base + non-empty anchored metrics,
+  parity within thresholds, license present. No zip ships without it.
+
+CLI (also the CI entry point):
+
+```
+PYTHONPATH=src python -m imf validate models/khm-latn/khm-latn-1.0-fp16.zip
+PYTHONPATH=src python -m imf validate <zip> --strict     # release gate
+PYTHONPATH=src python -m imf info <zip>                  # print manifest
+PYTHONPATH=src python -m imf pack --source <dir-or-legacy-zip> \
+    --metadata <yaml> [--readme <file>] --out <zip>       # sha256 computed
+```
+
+Legacy notes:
+
+- Old secryst zips (`vocabs.yaml` + single `transformer.onnx`) and the
+  PR #44 byte-level zips (`metadata.yaml: name: byt5`) predate IMF.
+  They fail base validation with a pointer to re-export/upgrade.
+- The fp32 Khmer zip on `secryst-checkpoints:/khmer_byt5/` has a CRC
+  error in `encoder.onnx` (found 2026-08-16, exactly the corrupt-download
+  class of failure the sha256-on-load rule exists for). It must be
+  re-exported by the WO02 pipeline; the fp16 zip is intact and was
+  upgraded to IMF v1 via `imf pack`.
+
+## Versioning
+
+Format changes bump the `format` field (`imf-v2`, ...). Model versioning
+lives in `id` (`khm-latn-1.1`). Adding an optional member or metadata
+field is v1-compatible; anything a v1 runtime would misinterpret is a
+new format version.

docs/model_card.md

@@ -5,7 +5,7 @@
 - **Name:** `interscript/rababa_arabic`
 - **Type:** Character-level transformer (4 layers, 256 dim, 4 heads)
 - **Trained from:** distilled from Qwen3.5-4B-Instruct teacher
-- **License:** MIT (code), CC-BY-4.0 (model weights)
+- **License:** BSD-3-Clause (code and weights)
 
 ## Training data
 

models/khm-latn/khm-latn-1.0-fp16.metadata.yaml

@@ -0,0 +1,26 @@
+format: imf-v1
+id: khm-latn-1.0
+task: translit
+source_script: Khmr
+target: Latn
+tokenizer: bytes
+opset: 14
+decoder: plain
+precision: fp16
+license: BSD-3-Clause
+trained_from: >-
+  secryst train_khmer_byt5.py run-001; checkpoint
+  secryst-checkpoints:/khmer_byt5/run-001/best
+metrics:
+  - name: cer
+    value: 27.42
+    protocol: >-
+      greedy decode; 895 held-out pairs; split 16,120/895/895 seed 42;
+      ByT5-small early stop @ep15
+    source: secryst/docs/RESULTS.md#khmer-transliteration-2026-08-14
+  - name: em
+    value: 59.66
+    protocol: >-
+      greedy decode; 895 held-out pairs; split 16,120/895/895 seed 42;
+      ByT5-small early stop @ep15
+    source: secryst/docs/RESULTS.md#khmer-transliteration-2026-08-14

models/khm-latn/khm-latn-1.0.README.md

@@ -0,0 +1,42 @@
+# khm-latn-1.0 (fp16)
+
+Khmer → Latin transliteration. Byte-level seq2seq (ByT5-small):
+the tokenizer is raw UTF-8 bytes (pad=0, EOS=1) — no vocab files.
+IMF v1 artifact; format spec: interscript/ml-models docs/imf-v1.md.
+
+- precision: fp16 (mixed: LayerNorm parameters in fp32)
+- decoder: plain greedy (KV-cache variant ships with the WO02 export)
+- metrics: CER 27.42 / EM 59.66 on 895 held-out pairs —
+  secryst/docs/RESULTS.md#khmer-transliteration-2026-08-14
+- trained from: secryst train_khmer_byt5.py run-001
+  (secryst-checkpoints:/khmer_byt5/run-001/best)
+- license: BSD-3-Clause
+
+## Usage
+
+Ruby (secryst gem, the Ruby binding of interscript-ml):
+
+```ruby
+require "secryst"
+translator = Secryst::Translator.new(model: "khm-latn-1.0")
+translator.translate("ភាសា")
+```
+
+TypeScript (@interscript/ml):
+
+```ts
+import { loadModel } from "@interscript/ml";
+const model = await loadModel("khm-latn-1.0");
+await model.translate("ភាសា");
+```
+
+Python (interscript-ml):
+
+```python
+from interscript_ml import Model
+model = Model.load("khm-latn-1.0")
+model.translate("ភាសា")
+```
+
+All three runtimes verify the sha256 of every ONNX member in this zip
+against metadata.yaml before loading.

pyproject.toml

@@ -7,7 +7,7 @@ name = "interscript-ml-models"
 version = "0.1.0"
 description = "Unified training framework for Interscript ML models (rababa + secryst)"
 readme = "README.md"
-license = { text = "MIT" }
+license = { text = "BSD-3-Clause" }
 requires-python = ">=3.10"
 authors = [{ name = "Interscript Project" }]
 keywords = ["nlp", "transliteration", "diacritization", "onnx", "distillation"]
@@ -48,7 +48,7 @@ interscript-ml = "src.cli:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
-include = ["framework*", "tasks*"]
+include = ["framework*", "tasks*", "imf*"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

src/imf/__init__.py

@@ -0,0 +1,25 @@
+"""Interscript Model Format (IMF) v1 — spec, validator, packer."""
+
+from imf.pack import PackError, pack_zip
+from imf.schema import (
+    FORMAT,
+    MAX_OPSET,
+    MetadataError,
+    Metric,
+    ModelMetadata,
+    Parity,
+)
+from imf.validator import ValidationResult, validate_zip
+
+__all__ = [
+    "FORMAT",
+    "MAX_OPSET",
+    "MetadataError",
+    "Metric",
+    "ModelMetadata",
+    "PackError",
+    "Parity",
+    "ValidationResult",
+    "pack_zip",
+    "validate_zip",
+]

src/imf/__main__.py

@@ -0,0 +1,4 @@
+from imf.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())