Skip to content

feat(imf): Interscript Model Format v1 — spec, validator, packer - #2

Merged
ronaldtse merged 1 commit into
mainfrom
feat/imf-v1
Aug 16, 2026
Merged

feat(imf): Interscript Model Format v1 — spec, validator, packer#2
ronaldtse merged 1 commit into
mainfrom
feat/imf-v1

Conversation

@ronaldtse

Copy link
Copy Markdown

Summary

Work order TODO.runtime-arch/01 (rababa): the Interscript Model Format v1
the versioned, portable artifact of interscript-ml. A model.zip adoptable on
its own terms, like ONNX itself: zip + sha256 + two ONNX sessions, no training
code required.

Three guarantees baked into the format:

  1. One runtime everywhere — tokenizer is raw UTF-8 bytes only (pad=0,
    EOS=1). No vocab files, no sentencepiece. Non-byte models enter via
    distillation (WO07), never a second tokenizer system.
  2. Old runtimes load it — opset pinned to 14 (Ruby onnxruntime gem's
    bundled ORT is old; verified the hard way in secryst PR #44).
  3. Every number traceable — metrics cite a RESULTS.md anchor; the strict
    gate refuses zips without parity + anchored metrics.

What's here

  • docs/imf-v1.md — full schema, validation levels, CLI usage
  • src/imf/ — metadata schema, validator (base load-time checks + strict
    release gate), packer (computes sha256, self-validates), CLI:
    python -m imf validate|info|pack
  • tests/test_imf.py — 20 specs over real zips with real ONNX graphs
    (tiny graphs built via the onnx package; no doubles)
  • CI: pytest job installs export extras so graph contract checks run
  • models/khm-latn/khm-latn-1.0-fp16.zip upgraded from the secryst PR #44
    zip via imf pack — base-valid (strict correctly pending the WO03 parity
    gate). Pack sources (metadata.yaml + README.md) committed; the zip
    itself is gitignored (769 MB).
  • License normalized to BSD-3-Clause across the repo (code and weights);
    LICENSE file added (was MIT placeholders).

Found along the way

  • The fp32 khm zip on secryst-checkpoints:/khmer_byt5/khm-latn-byt5.zip
    has a CRC error in encoder.onnx on the volume itself — exactly the
    corrupt-download failure class the sha256-on-load rule exists for. It must
    be re-exported by the WO02 pipeline from /khmer_byt5/run-001/best; the
    fp16 zip is intact and validates.

Test plan

  • pytest tests/ — 57 passed (20 new IMF specs)
  • ruff check src tests clean
  • python -m imf validate on the repacked fp16 zip: OK (base)
  • python -m imf validate --strict correctly fails pending parity
  • python -m imf validate on the corrupt fp32 zip: caught (CRC mismatch)
  • mypy runs in CI (strict mode configured repo-wide)

IMF v1 is the versioned portable artifact of interscript-ml: byte-level
tokenizer only, ONNX opset 14 (Ruby onnxruntime gem compat), sha256 of
every graph verified on load, metrics traceable to RESULTS.md anchors.

- docs/imf-v1.md: full schema + validation-level documentation
- src/imf: metadata schema, zip validator (base + strict release gate),
  packer that computes sha256 and self-validates, CLI (python -m imf)
- tests: 20 specs over real zips with real ONNX graphs, no doubles
- CI: pytest job installs export extras so graph checks run
- khm-latn-1.0-fp16 upgraded from the PR #44 zip via 'imf pack'
  (base-valid; strict pending the WO03 parity gate)
- license: BSD-3-Clause for code and model weights (LICENSE added;
  replaces the placeholder MIT)

Found on the way: the fp32 khm zip on secryst-checkpoints:/khmer_byt5
has a CRC error in encoder.onnx — re-export needed (WO02); the fp16 zip
is intact (validated).
@ronaldtse
ronaldtse merged commit 3b4ac96 into main Aug 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant