feat(imf): Interscript Model Format v1 — spec, validator, packer - #2
Merged
Conversation
IMF v1 is the versioned portable artifact of interscript-ml: byte-level tokenizer only, ONNX opset 14 (Ruby onnxruntime gem compat), sha256 of every graph verified on load, metrics traceable to RESULTS.md anchors. - docs/imf-v1.md: full schema + validation-level documentation - src/imf: metadata schema, zip validator (base + strict release gate), packer that computes sha256 and self-validates, CLI (python -m imf) - tests: 20 specs over real zips with real ONNX graphs, no doubles - CI: pytest job installs export extras so graph checks run - khm-latn-1.0-fp16 upgraded from the PR #44 zip via 'imf pack' (base-valid; strict pending the WO03 parity gate) - license: BSD-3-Clause for code and model weights (LICENSE added; replaces the placeholder MIT) Found on the way: the fp32 khm zip on secryst-checkpoints:/khmer_byt5 has a CRC error in encoder.onnx — re-export needed (WO02); the fp16 zip is intact (validated).
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Work order TODO.runtime-arch/01 (rababa): the Interscript Model Format v1 —
the versioned, portable artifact of interscript-ml. A model.zip adoptable on
its own terms, like ONNX itself: zip + sha256 + two ONNX sessions, no training
code required.
Three guarantees baked into the format:
EOS=1). No vocab files, no sentencepiece. Non-byte models enter via
distillation (WO07), never a second tokenizer system.
bundled ORT is old; verified the hard way in secryst PR #44).
gate refuses zips without parity + anchored metrics.
What's here
docs/imf-v1.md— full schema, validation levels, CLI usagesrc/imf/— metadata schema, validator (base load-time checks + strictrelease gate), packer (computes sha256, self-validates), CLI:
python -m imf validate|info|packtests/test_imf.py— 20 specs over real zips with real ONNX graphs(tiny graphs built via the onnx package; no doubles)
exportextras so graph contract checks runmodels/khm-latn/khm-latn-1.0-fp16.zipupgraded from the secryst PR #44zip via
imf pack— base-valid (strict correctly pending the WO03 paritygate). Pack sources (
metadata.yaml+README.md) committed; the zipitself is gitignored (769 MB).
LICENSEfile added (was MIT placeholders).Found along the way
secryst-checkpoints:/khmer_byt5/khm-latn-byt5.ziphas a CRC error in
encoder.onnxon the volume itself — exactly thecorrupt-download failure class the sha256-on-load rule exists for. It must
be re-exported by the WO02 pipeline from
/khmer_byt5/run-001/best; thefp16 zip is intact and validates.
Test plan
pytest tests/— 57 passed (20 new IMF specs)ruff check src testscleanpython -m imf validateon the repacked fp16 zip: OK (base)python -m imf validate --strictcorrectly fails pending paritypython -m imf validateon the corrupt fp32 zip: caught (CRC mismatch)