0.3.1a2: move the en-zh translation demo out of the kernel

.github/workflows/ci.yml

@@ -4,7 +4,7 @@ on:
   push:
     branches: [main, dev]
   pull_request:
-    branches: [main]
+    branches: [main, dev]
 
 jobs:
   test:

CHANGELOG.md

@@ -2,6 +2,39 @@
 
 ## [Unreleased]
 
+## [0.3.1a2] — 2026-11-07 (beta)
+
+Kernel/extension split: the `chronos` kernel trains general models and holds no translation-specific code. The en↔zh demo moved to `demos/translation` (`chronos-translate`), which imports the kernel and is not part of the `Project_Chronos` wheel. The extension is not published on PyPI; install it from a checkout with `pip install -e demos/translation` (see `demos/translation/README.md`).
+
+### Moved (to `demos/translation/chronos_translate/`)
+- `chronos.data.corpus.*` → `chronos_translate.corpus.*` (WMT19/OPUS-100/FLORES+ download, cleaning, dedup, en–zh shards, doc-mix sets).
+- `chronos.bundle.*` → `chronos_translate.bundle.*`; `chronos bundle …` → `chronos-translate bundle …`. A bundle's `bin/chronos bundle …` still works: the launcher sends `bundle` to `chronos-translate`. `bin/setup.sh` installs both wheels from `wheels/`; `bundle build --wheel` can be repeated.
+- `chronos.mlx.training.doc_run` → `chronos_translate.doc_run` (`chronos-translate doc-run`).
+- `inspect_checkpoint.translation_probe` → `chronos_translate.probe` (`chronos-translate inspect`).
+- `chronos.model.factory.ENZH_DEMO_BASE` → `chronos_translate.model_presets.ENZH_50M_BASE`.
+- `chronos/mlx/training/recipes/enzh_50m_72h.json` → `demos/translation/recipes/`, with paths relative to the file.
+
+### Added
+- Monitor (`chronos monitor`): a "Lazy experts / 懒加载" tab shows every metric the lazy-expert probe records after each validation. It has four tables (efficiency with the full-memory reference row; paging and loads; hit rates; fidelity), per-variant history charts (hit rate, fidelity, speed, loads, peak RSS) and the full probe log with skip and failure reasons. New endpoint: `GET /api/lazy_probes?run=…&limit=…`.
+- Web UI monitor page: the probe card shows the same four grouped tables and four history charts in all four UI languages. Previously it showed a subset of the columns and no load, timing or reference metrics.
+- The run snapshot's `lazy_probe` entries carry every measured probe field plus `reference_measured` (the full-memory reference per backend).
+- Probes run from Web UI training jobs. For a CLI run (`chronos longrun`, `chronos-translate run`), start `chronos lazy-probe --watch <run_dir>` next to it, or the tab stays empty.
+
+### Changed
+- `LongRun` has task hooks: `task_eval(phase, n, final=)` (returns `{}` in the kernel), `eval_samples_for`, `generate`, `write_samples`, `pinned_modules`. The kernel `RunSpec` keeps `eval_samples` / `final_eval_samples` (default 0) and `export_prefix` (default `chronos`); `bleu_samples` and `probe_samples` moved to `TranslationRunSpec`. Run translation recipes with `chronos-translate run --spec …`; the kernel runner rejects unknown keys and names the task runner.
+- `chronos longrun` and `load_spec` resolve relative recipe paths against the recipe's folder (they resolved against the working directory before). `--max-steps` and `--run-dir` give a short smoke run.
+- Task-eval exceptions during SFT validation and the final evaluation are logged as `task_eval_error` instead of stopping the run (pretrain probes were already logged). The smoke path still raises.
+- Tokenizer special tokens: the kernel default is `<|endoftext|>`, `<|im_start|>`, `<|im_end|>`, `<|pad|>` and reserved slots up to 64. Task tokens come from `special_tokens_with(extra)` or `chronos tokenizer --extra-special …`. `special_tokens_with(["<|en|>", "<|zh|>", "<|translate|>"])` reproduces the 0.3.1a1 order, so existing tokenizers and checkpoints are unchanged.
+- The shard readers (`PackedTokenStream`, `SFTRecords`, `dtype_for`, `manifest_dtype`) live in `chronos.data.shards`. `median_record_tokens` and `cluster_locality_config` moved to `chronos.expert_clusters`.
+- Preset "Chronos-v3 50M (en→zh recipe)" is now "Chronos-v3 50M (16k vocab)" (the old name is an alias) and uses the default 20 tokens/param schedule instead of the production-shard measurement. Same shape: 49,691,717 total / 12,090,437 active parameters (measured, meta device).
+
+### Fixed
+- Web UI monitor page: a card with nothing to show no longer prints a stray "null".
+
+### Removed
+- `chronos corpus`, `chronos bundle` and the `translate` extra (`sacrebleu`, `sacremoses` are dependencies of `chronos-translate`).
+- Translation metric labels (`bleu_zh`, `flores_devtest_n`) from the Web UI i18n files; the monitor still shows any numeric metric a task writes.
+
 ## [0.3.1a1] — 2026-11-07 (beta)
 
 Web UI half of 0.3.1: the rewritten Web UI, file picking and upload, LAN access, and the lazy-expert probe.

MANIFEST.in

@@ -12,4 +12,4 @@ prune results
 prune reports
 prune .archive
 prune Translation?Training?Data
-prune chronos/mlx/training/recipes
+prune demos

README.md

@@ -107,14 +107,15 @@ At this size a full-depth step costs about as much as a batched verify pass. Wit
 | Command | What it does |
 |---|---|
 | `chronos tokenizer` | Self-built byte-fallback BPE with script-aware pre-tokenization and language-balanced merges; writes an HF-compatible `tokenizer.json` + chat template |
-| `chronos corpus download / build / shards` | Pinned downloads, cleaning, global dedup, eval-set leak removal, uint16 shards |
-| `chronos longrun --spec recipe.json` (alias `run`) | Time-budgeted MLX run: warmup + cosine per phase wall-clock budget, NaN rollback, bit-exact resume, validation and translation probes |
+| `chronos longrun --spec recipe.json` (alias `run`) | Time-budgeted MLX run: warmup + cosine per phase wall-clock budget, NaN rollback, bit-exact resume, validation CE; tasks add their own metrics by subclassing `LongRun` (`task_eval`) |
 | `chronos serve` | Ollama/OpenAI protocol server on the native runtime (see below) |
 | `chronos monitor` | Read-only training monitor page + JSON API (see below) |
 
 `chronos.model.factory.fit_config` searches for a configuration that matches total and active parameter targets.
 
-**Status of the 50M en→zh demo (in progress).**
+**Kernel and extensions.** The `chronos` package trains general models and carries no task-specific code. Task tooling lives in extensions that import the kernel. The en↔zh translation demo is one: `demos/translation` (`chronos-translate`: corpus pipeline, BLEU/chrF runs, model bundles; see its README). It is not part of the `Project_Chronos` wheel and is not published on PyPI; install it from a checkout with `pip install -e demos/translation`.
+
+**Status of the 50M en→zh demo (in progress; `demos/translation`).**
 - Model: 49.69M total / 12.09M active parameters, 36 experts, MLA latent 128, SWA window 128, 16k self-built vocabulary.
 - Run: a 72-hour MLX bf16 run (pretrain, then SFT) started on 2026-11-05.
 - Interim measurement: valid CE 2.477 at pretrain step 13,996 of about 24.7k planned (it was 3.012 at step 1,656). That checkpoint is packaged as `chronos-enzh-50m-pretrain-s13996` (see Model bundle).
@@ -450,7 +451,7 @@ pip install "Project_Chronos[mlx]"
 
 ```bash
 pip install "Project_Chronos[serve]"      # chronos serve / chronos monitor (fastapi, uvicorn)
-pip install "Project_Chronos[translate]"  # corpus detokenization + BLEU/chrF probes (sacremoses, sacrebleu)
+pip install -e demos/translation          # en↔zh demo extension, from a checkout only (not on PyPI): chronos-translate
 ```
 
 **vLLM serving (optional, Linux + CUDA only):**
@@ -491,7 +492,7 @@ The Config tab ships five v3 presets. Each one is generated by the planner and s
 | Preset | Total / active params (measured) | Shape | Memory level | Est. training peak |
 |---|---|---|---|---|
 | Chronos-v3 Tiny (smoke / CI) | 1.51M / 1.29M | E4 top-2, vocab 6400, seq 128, depth 2 | none | 2.13 GB |
-| Chronos-v3 50M (en→zh recipe) | 49.69M / 12.09M | h384, L6, E36 top-2, vocab 16384, seq 512 | none | 16.4 GB (the live run measures 8.72 GB peak) |
+| Chronos-v3 50M (16k vocab) | 49.69M / 12.09M | h384, L6, E36 top-2, vocab 16384, seq 512 | none | 16.4 GB (the live run measures 8.72 GB peak) |
 | Chronos-v3 200M (32 GB) — default | 200.72M / 28.20M | h576, E32 top-2 | balanced | 10.14 GB |
 | Chronos-v3 500M (32 GB) | 499.53M / 61.59M | E48 top-3 | aggressive | 9.26 GB |
 | Chronos-v3 1B (64 GB) | 1002.17M / 99.74M | E64 top-4, seq 2048 | aggressive | 18.92 GB |
@@ -558,12 +559,13 @@ OLLAMA_HOST=127.0.0.1:11435 ollama run <model-name> "Hello"
 - Stock `ollama` / llama.cpp cannot execute Chronos GGUF files (MLA+SWA hybrid layers and the contractive recurrence have no stock equivalent); use this server instead.
 - Security: binds 127.0.0.1 without authentication by default. Other addresses require `--allow-remote` and `--api-key`; clients then send `Authorization: Bearer <key>`.
 
-### Model bundle (ready-to-run pretrained model)
+### Model bundle (translation demo extension)
 
-`chronos bundle build` packages one exported checkpoint of a `chronos longrun` run into a folder that works right after unpacking. Building only reads the run directory.
+Bundles belong to the translation extension (`demos/translation`). Build the two wheels first (`python -m build --outdir dist .` and `python -m build --outdir dist demos/translation`). `chronos-translate bundle build` packages one exported checkpoint of a run into a folder that works right after unpacking. Building only reads the run directory.
 
 ```bash
-chronos bundle build --run-dir runs/enzh_50m_v1 --which best --wheel dist/project_chronos-0.3.0-py3-none-any.whl --pack
+chronos-translate bundle build --run-dir runs/enzh_50m_v1 --which best --pack \
+    --wheel dist/project_chronos-0.3.1a2-py3-none-any.whl --wheel dist/chronos_translate-0.3.1a2-py3-none-any.whl
 ```
 
 | Bundle path | Content |
@@ -572,14 +574,14 @@ chronos bundle build --run-dir runs/enzh_50m_v1 --which best --wheel dist/projec
 | `eval/` | FLORES+ dev/devtest (CC BY-SA 4.0) for `bundle eval` / `probe` |
 | `data/sft_valid/` | the run's SFT validation shards |
 | `third_party/minimind/` | vendored minimind (Apache-2.0), so no git clone is needed |
-| `wheels/`, `bin/setup.sh`, `bin/chronos` | offline install into `.venv`, launcher that sets `CHRONOS_BUNDLE` / `CHRONOS_MINIMIND_DIR` |
+| `wheels/`, `bin/setup.sh`, `bin/chronos` | offline install of the kernel and extension wheels into `.venv`; launcher that sets `CHRONOS_BUNDLE` / `CHRONOS_MINIMIND_DIR` and sends `bundle …` to `chronos-translate` |
 | `recipes/source_run_spec.json`, `recommended.json` | the recipe that produced the weights; fine-tune tiers `smoke` / `quick` / `standard` / `full` |
 | `bundle.json` | provenance (step, tokens, valid CE), measured quality probe, SHA-256 of every file |
 
 On the recipient side:
 
 ```bash
-bin/setup.sh                                              # venv + wheel + `bundle check`
+bin/setup.sh                                              # venv + wheels + `bundle check`
 bin/chronos bundle translate "The weather is very nice today."
 bin/chronos bundle translate --backend mlx --speculative --file examples/translate_en.txt
 bin/chronos bundle eval --split devtest -n 200
@@ -588,7 +590,7 @@ bin/chronos bundle finetune --data work/data --run-dir work/ft --tier quick --mo
 bin/chronos bundle serve -- --port 11435
 ```
 
-`--speculative` needs `--repetition-penalty 1.0` (its default), because the penalty changes the target distribution at every position. `finetune` writes a `chronos longrun` spec with `init_checkpoint` set to the bundle weights and refuses a run directory that already holds a run. Bundles built from data with unknown licences (WMT19, OPUS-100) say in `NOTICE.md` that the weights are for research only.
+`--speculative` needs `--repetition-penalty 1.0` (its default), because the penalty changes the target distribution at every position. `finetune` writes a `chronos-translate run` spec with `init_checkpoint` set to the bundle weights and refuses a run directory that already holds a run. Bundles built from data with unknown licences (WMT19, OPUS-100) say in `NOTICE.md` that the weights are for research only.
 
 ### Deployment export
 
@@ -768,7 +770,7 @@ Project_Chronos/
 │   │   └── benchmark.py           # End-to-end benchmarking
 │   ├── data/
 │   │   ├── flexible_dataset.py    # Flexible JSONL dataset loader
-│   │   └── corpus/                # download / clean / dedup / leak filter / shards
+│   │   └── shards.py              # memory-mapped pretrain / SFT shard readers
 │   ├── tokenizer/                 # Chronos BPE builder (extract, normalize, train, evaluate)
 │   ├── monitor/                   # Read-only run monitor: snapshot, API server, static page
 │   ├── webui/                     # Web UI: FastAPI server, field schema, i18n/*.json, static/ SPA
@@ -778,7 +780,6 @@ Project_Chronos/
 │   │   ├── dispatcher.py          # Capability probing + priority logic
 │   │   └── ext/opencl.py          # Third-party OpenCL extension hook
 │   ├── _backend_legacy.py         # Backward-compatible older APIs
-│   ├── bundle/                    # chronos bundle: build/check/translate/finetune
 │   ├── serving/
 │   │   ├── ollama_server.py       # chronos serve: Ollama/OpenAI protocol layer
 │   │   ├── runtime.py             # Model discovery, lazy experts, speculative generation
@@ -815,6 +816,8 @@ Project_Chronos/
 │   └── fixtures/
 ├── docs/
 │   └── vllm_integration.md
+├── demos/
+│   └── translation/               # en↔zh extension (chronos-translate): corpus, runs, bundles; not in the wheel
 ├── pyproject.toml
 └── README.md / README_zh.md / THIRD_PARTY_NOTICES.md
 ```