diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5c6d2fd..2fd818d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -36,3 +36,8 @@ jobs: - name: Unit tests run: | python -m pytest tests/ -v --tb=short + + - name: Demo driver tests + run: | + # The translation demo's model driver; the kernel suite above stays task-neutral. + python -m pytest demos/translation/tests/test_model_enzh_50m_sft.py -v --tb=short diff --git a/CHANGELOG.md b/CHANGELOG.md index e953045..b76a55e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,44 @@ # Changelog +## [0.3.3a1] — 2026-11-12 (beta) + +This release closes three gaps in 0.3.3a0. Every loop knob now has a planner recommendation and an auto-tune switch. The auto-tune probe now scores what it was meant to score. The en-zh model folder has left the repository root. Kernel defaults are unchanged, so existing configs and checkpoints behave as in 0.3.3a0. + +### Added +- Planner: a looped plan (`recurrent_depth > 1`) now also sets six more loop knobs. Each comes with a `sources` entry that states the measurement behind it (15M parameters, one seed per arm). The Web UI shows these values as planner recommendations, and `ui/presets_v3.json` gains the four keys whose value differs from the base defaults. + - `recurrent_later_topk=0`: routing later iterations to 1 expert raised validation CE from 4.543 to 4.600 at the same speed (71.6k against 71.5k tokens/s). + - `recurrent_later_groups=0`: the full block has the lowest loss per step. Running 2 of 4 layers in later iterations gives about 1.3x more steps per second at depth 4. It costs up to 0.012 more training loss per step with dt fixed at 1.0 and 0.013-0.055 with the sigmoid step size. 1 is never recommended, because one layer looping alone is no longer a block loop. + - `lambda_iteration_cluster=0`: with cluster locality on, 99.5-100% of the clusters that later iterations use are already used at iteration 0. + - `lookahead_teacher_scope="first"`: the lookahead head already picks the right cluster for 0.97 of tokens at every iteration. + - `recurrent_dt_min=0.05` and `recurrent_dt_max=1.0`: the sigmoid arms used the ranges 0.05-1 (depth 4) and 0.1-1 (depth 8), and their learned dt stayed between 0.14 and 0.30, inside the range. +- Auto-tune can now search every loop knob. The list adds `recurrent_dt_param`, `recurrent_dt_min`, `recurrent_dt_max`, `recurrent_later_topk` (0 or 1), `recurrent_later_groups` (0 or 2), `lambda_iteration_cluster` (0, 0.1, 0.3 or 1.0) and `lookahead_teacher_scope`. They join `recurrent_order` and `recurrent_dt_init`. + - Searching any of them makes the probe a looped model with 4 block layers, one prelude layer and one coda layer. The 0.3.3a0 probe had 2 block layers, so `recurrent_later_groups=2` would have meant the full block. + - `recurrent_later_groups` is drawn only in block order, and 1 is not offered. + - If a drawn `recurrent_dt_init` falls outside the drawn `[dt_min, dt_max]`, the trial starts at the nearest bound. + - Each trial logs its median step time and stores it as the Optuna user attribute `step_time_s`. `recurrent_later_topk` and `recurrent_later_groups` save time, not loss, so a cross-entropy score alone favours "off". +- Help text for these knobs and for the auto-tune search list now gives the measured reason and the recommended value, in English, 简体中文, 繁體中文 and 日本語. + +### Fixed +- Auto-tune score. The probe trained on and scored `CE + lambda_balance * L_balance + lambda_temporal * L_temporal`, so a smaller weight lowered a trial's score by itself. The probe now trains on the trainers' objective (`chronos_loss_term`, every weighted term of the config) and scores cross-entropy only. +- Auto-tune `lambda_lookahead` had no effect. The probe never passed it to the model config or to the loss, and it never trained the lookahead head, so `lookahead_steps` changed the model without supervising it. With the 0.3.3a0 probe, `lambda_lookahead` 1e-3 and 1.0 gave identical per-step scores; with this release the scores differ after the first update. The lookahead, iteration-cluster and recurrent terms of the config now act in the probe as they do in training. +- Web UI schema payload. `Field.as_dict` dropped every value equal to `False`, and in Python `0 == False`. So `min=0` of 48 fields, and `recommended=0` of `max_steps` and `cluster_locality_clusters`, never reached the page. Those number inputs had no lower bound, and their tooltips showed the minimum as "−∞". The payload now drops only unset values. + +### Changed +- The en-zh model folder moved from `models/chronos-enzh-50m-sft` to `demos/translation/models/chronos-enzh-50m-sft`, and its driver test moved from `tests/` to `demos/translation/tests/`. CI runs that test in its own step after the kernel suite, and the repository root no longer holds a task-specific model. The driver finds its files relative to itself, so commands run inside the folder are unchanged, and the weights stay on the release `enzh-50m-sft-v1`. `MANIFEST.in` no longer prunes `models/`. +- The main READMEs now describe the speculative-decoding, native-kernel parity and MLX compile measurements as taken on the 50M demo checkpoint. They link to a new section of `demos/translation/README.md` and `README_zh.md` that states the exact conditions (FLORES+ dev en→zh prompts, en→zh shards); the numbers are unchanged. Each main README names the translation demo in one pointer paragraph only. `tests/test_ui_generic.py::test_task_models_and_tests_live_under_demos` checks this rule and the folder layout. +- The preset label "Chronos-v3 50M (16k vocab)" stays. It names the preset's explicit 16,384-token vocabulary and no task. + +### Verified +- End to end through the Web UI endpoint, with an in-process client and no open port: a 4-trial study of 10 probe steps each searched all nine loop knobs plus `lambda_lookahead`. It trained on 19,090 real dataset tokens with the bundled 32k tokenizer, completed every trial, logged each trial's cross-entropy and median step time, and returned a best-config patch with every searched knob. + +### Tests +- New `tests/test_loop_knobs_wiring.py` (10 items). It checks that every loop knob is searchable and reaches the probe model, and that `recurrent_later_groups` never offers 1 and is drawn only in block order. It checks that `lambda_lookahead` 1e-3 and 1.0 give the same score at step 0 and different scores after the first update. It also covers the best-config patch, the Web UI search list with help in four languages, planner recommendations with a source at depths 2, 4 and 8, a depth-1 plan with no loop knob, and a schema payload that keeps 0. +- New `tests/test_ui_generic.py::test_task_models_and_tests_live_under_demos`: `models/` and `tests/` hold no task-specific names, and each main README names the en-zh demo in one line, which points to `demos/translation`. +- `tests/test_autotune_loop.py`: `recurrent_later_groups` is now expected in the search list. +- `demos/translation/tests/test_model_enzh_50m_sft.py` (15 items, moved from `tests/`) passes from its new folder; CI runs it as the step "Demo driver tests". +- Full suite on macOS (Apple silicon, Python 3.11, MLX on CPU): 728 passed, 4 skipped. `ruff check .` passes on the whole tree. +- A local wheel and sdist build both carry the `general_32k` tokenizer and no file from `demos/`, no model folder and no weights. + ## [0.3.3a0] — 2026-11-12 (beta) ### Added diff --git a/MANIFEST.in b/MANIFEST.in index 26d442f..da4788b 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -13,4 +13,3 @@ prune reports prune .archive prune Translation?Training?Data prune demos -prune models diff --git a/README.md b/README.md index 4f4c207..82fc1f0 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ flowchart LR **Self-speculative decoding (no extra parameters).** The same model drafts γ tokens at a shallow depth and verifies them with one full-depth batched forward pass. Greedy output is token-for-token identical to full-depth greedy. Sampling uses rejection sampling, which preserves the full-depth distribution. After a rejection, the cache is truncated: MLA slices `c_kv`, and the sliding-window cache keeps `window + γ_max` entries. Torch (`chronos/runtime/speculative.py`) and MLX (`chronos/mlx/speculative.py`, `ChronosMLXInferenceEngine.generate(speculative=True)`) share the algorithm. Tests cover greedy identity, the rejection distribution and rollback against a fresh prefill on both backends. The Web UI Inference tab has a switch for it, with draft depth and γ. -For the 50M demo it is exact but not faster on any runtime we measured (pretrain step-13,996 checkpoint, greedy, batch 1, FLORES+ dev en→zh, draft depth 1, γ 4; reports in `reports/speculative/`): +For the 50M demo checkpoint it is exact but not faster on any runtime we measured (pretrain step-13,996 checkpoint of the demo in `demos/translation`, greedy, batch 1, the demo's dev prompts, draft depth 1, γ 4; exact conditions in [`demos/translation/README.md`](demos/translation/README.md); reports in `reports/speculative/`): | Runtime | Plain tok/s | Speculative tok/s | Ratio | Acceptance | |---|---|---|---|---| @@ -113,7 +113,7 @@ At this size a full-depth step costs about as much as a batched verify pass. Wit `chronos.model.factory.fit_config` searches for a configuration that matches total and active parameter targets. -**Kernel and extensions.** The `chronos` package trains general models and carries no task-specific code. Task tooling lives in extensions that import the kernel. The en↔zh translation demo is one: `demos/translation` (`chronos-translate`: corpus pipeline, BLEU/chrF runs, model bundles; see its README). It is not part of the `Project_Chronos` wheel and is not published on PyPI; install it from a checkout with `pip install -e demos/translation`. The 50M en→zh status, the released model `chronos-enzh-50m-sft` with its measurements, and the model bundle are documented in [`demos/translation/README.md`](demos/translation/README.md). +**Kernel and extensions.** The `chronos` package trains general models and carries no task-specific code. Task tooling lives in extensions that import the kernel. The en↔zh translation demo is one: `demos/translation` (`chronos-translate`: corpus pipeline, BLEU/chrF runs, model bundles; see its README). It is not part of the `Project_Chronos` wheel and is not published on PyPI; install it from a checkout with `pip install -e demos/translation`. The 50M en→zh status, the released model `chronos-enzh-50m-sft` (files in `demos/translation/models/`) with its measurements, and the model bundle are documented in [`demos/translation/README.md`](demos/translation/README.md). **Inference and memory options (all off by default).** Each option is a config field and a `chronos serve` / Inference-tab override. Tests compare each one with its reference path on Torch and MLX. @@ -153,11 +153,11 @@ Up to 0.3.1a3, Chronos took its config base, RoPE tables, gated FFN, dense atten **Old vs new, measured 2026-11-08** -Setup: Apple M5 Max, PyTorch 2.10 on CPU, FP32, 4 threads, median of 5–7 runs, each kernel in its own process, runs alternated twice. The 50M en→zh SFT run was training on the GPU at the same time, so absolute timings carry noise. Old = 0.3.1a3 + minimind `24896cd`. +Setup: Apple M5 Max, PyTorch 2.10 on CPU, FP32, 4 threads, median of 5–7 runs, each kernel in its own process, runs alternated twice. A 50M training run was using the GPU at the same time, so absolute timings carry noise. Old = 0.3.1a3 + minimind `24896cd`. | Check | 0.3.1a3 + minimind | 0.3.2 native | |---|---|---| -| Logits, `out/chronos_256_moe.pth` and the 50M en→zh checkpoint (full, padded and prefill+decode paths) | reference | bit-identical (max abs diff 0) | +| Logits, `out/chronos_256_moe.pth` and the 50M demo checkpoint (full, padded and prefill+decode paths) | reference | bit-identical (max abs diff 0) | | Random-init `state_dict` and config values (same seed) | reference | identical | | `import chronos` with no cache and no network | fails (`git clone` error) | works | | minimind on `sys.path`, top-level `model` module imported | yes | no | @@ -181,7 +181,7 @@ Up to 0.3.1a3 the MLX trainers ran every micro step eagerly and kept every activ - **Compiled micro step** (`chronos/mlx/training/compiled.py`, on by default). Loss and gradients run as one `mx.compile` graph per signature. The executed recurrent depth, the ACT switch, input shapes and plain config values are part of the signature, so a step never reuses a stale graph. Weights and the RNG state are compile inputs and outputs, so dropout and the cache-miss simulation consume the RNG stream exactly as before. Multimodal batches, lazy (paged) experts and any signature that fails to trace run eagerly, and the reason is logged once. `CHRONOS_MLX_COMPILE=auto|on|off` or the recipe's `extra.mlx_compile` switches it. - **Activation recomputation** (`chronos/mlx/checkpointing.py`, `activation_checkpointing=true`; the planner's memory-constrained tiers turn it on). Each block call becomes one recomputed region. A plain `mx.checkpoint` is wrong here in two ways, both measured on mlx 0.32.4. Dropout masks and the cache-miss draws would be drawn again during recomputation; they are recorded in the forward pass and replayed. A region that returns the same graph node twice keeps the gradient of only one copy. A block's KV cache and its shared view are such a pair whenever the SWA window covers the sequence (a full-length slice is a new Python object over the same node), and the gradients were 37–40% off. Region outputs are therefore packed into one array per dtype. -**Measured 2026-11-08.** Setup: Apple M5 Max, mlx 0.32.4, bf16, the 50M en→zh recipe's model config from random init, real en→zh shards, the recipe's step sizes (SFT 8 micro-batches per step, pretrain 16). Median of the second half of 16 (SFT) or 12 (pretrain) steps. The 72-hour run was training on the same GPU, so absolute timings carry noise. +**Measured 2026-11-08.** Setup: Apple M5 Max, mlx 0.32.4, bf16, the 50M demo recipe's model config (`demos/translation/recipes/`) from random init, the demo's real training shards, the recipe's step sizes (SFT 8 micro-batches per step, pretrain 16). Median of the second half of 16 (SFT) or 12 (pretrain) steps. The 72-hour run was training on the same GPU, so absolute timings carry noise. | Micro step | SFT s/step | SFT peak memory | Pretrain s/step | Pretrain peak memory | |---|---|---|---|---| @@ -519,7 +519,7 @@ pip install "Project_Chronos[mlx]" ```bash pip install "Project_Chronos[serve]" # chronos serve / chronos monitor (fastapi, uvicorn) -pip install -e demos/translation # en↔zh demo extension, from a checkout only (not on PyPI): chronos-translate +pip install -e demos/translation # translation demo extension (chronos-translate), from a checkout only, not on PyPI ``` **vLLM serving (optional, Linux + CUDA only):** @@ -858,7 +858,7 @@ Project_Chronos/ ├── docs/ │ └── vllm_integration.md ├── demos/ -│ └── translation/ # en↔zh extension (chronos-translate): corpus, runs, bundles; not in the wheel +│ └── translation/ # demo extension (chronos-translate): corpus, runs, bundles, the 50M demo model; not in the wheel ├── pyproject.toml └── README.md / README_zh.md / THIRD_PARTY_NOTICES.md ``` @@ -908,7 +908,7 @@ timeline : int8 KV, MLA absorbed decode, shared router scope (opt-in) : Auto planner, v3 presets, tokenizer and serve tabs Next - : 50M en-zh results and ablations + : Multi-seed and larger-size runs of the 0.3.3 loop settings : Speculative decoding: a draft path that pays off on larger models and lazy experts : Train IntentClassifier on large activation corpora : Benchmark 7B+ checkpoints diff --git a/README_zh.md b/README_zh.md index c7eea6d..dcb5f85 100644 --- a/README_zh.md +++ b/README_zh.md @@ -94,7 +94,7 @@ flowchart LR - Torch(`chronos/runtime/speculative.py`)与 MLX(`chronos/mlx/speculative.py`、`ChronosMLXInferenceEngine.generate(speculative=True)`)使用同一套算法。 - 两个后端的测试都覆盖 greedy 一致性、拒绝采样分布,以及回滚结果与从头 prefill 一致。 - Web UI 推理 Tab 提供自推测开关,可调起草深度和 γ。 -- 50M demo 在实测过的所有运行时上都结果一致,但都没有提速。实测条件:预训练 step 13,996 checkpoint,greedy,batch 1,FLORES+ dev 英→中,起草深度 1,γ 4。报告在 `reports/speculative/`。 +- 50M demo checkpoint 在实测过的所有运行时上都结果一致,但都没有提速。实测条件:`demos/translation` 中 demo 的预训练 step 13,996 checkpoint,greedy,batch 1,该 demo 的 dev 提示,起草深度 1,γ 4;完整条件见 [`demos/translation/README_zh.md`](demos/translation/README_zh.md)。报告在 `reports/speculative/`。 | 运行时 | 普通 tok/s | 自推测 tok/s | 比值 | 接受率 | |---|---|---|---|---| @@ -125,7 +125,7 @@ flowchart LR `chronos.model.factory.fit_config` 按总参数量和激活参数量目标搜索模型配置。 -**内核与扩展。** `chronos` 包用于训练通用模型,不含任何任务特化代码。任务工具放在调用内核的扩展里。英↔中翻译 demo 就是这样一个扩展:`demos/translation`(`chronos-translate`:语料流水线、BLEU/chrF 训练、模型包,详见其 README)。它不在 `Project_Chronos` wheel 里,也不发布到 PyPI,需从仓库检出后用 `pip install -e demos/translation` 安装。50M 英译中的进展、已发布模型 `chronos-enzh-50m-sft` 及其实测结果、模型包,均写在 [`demos/translation/README_zh.md`](demos/translation/README_zh.md)。 +**内核与扩展。** `chronos` 包用于训练通用模型,不含任何任务特化代码。任务工具放在调用内核的扩展里。英↔中翻译 demo 就是这样一个扩展:`demos/translation`(`chronos-translate`:语料流水线、BLEU/chrF 训练、模型包,详见其 README)。它不在 `Project_Chronos` wheel 里,也不发布到 PyPI,需从仓库检出后用 `pip install -e demos/translation` 安装。50M 英译中的进展、已发布模型 `chronos-enzh-50m-sft`(文件在 `demos/translation/models/`)及其实测结果、模型包,均写在 [`demos/translation/README_zh.md`](demos/translation/README_zh.md)。 **推理与内存选项(全部默认关闭)。** 每个选项既是 config 字段,也可作为 `chronos serve` 和 Inference Tab 的覆盖项;测试在 Torch 和 MLX 上把每个选项与参考路径做对比。 @@ -165,11 +165,11 @@ Web UI 新增 Tokenizer Tab(抽取 → 训练 → 评估)和 Serve Tab(启 **新旧对比(2026-11-08 实测)** -条件:Apple M5 Max,PyTorch 2.10 CPU,FP32,4 线程,取 5–7 次的中位数;每个内核在独立进程中运行,交替运行两轮。测量时 50M 英译中 SFT 正在 GPU 上训练,绝对耗时有噪声。旧版 = 0.3.1a3 + minimind `24896cd`。 +条件:Apple M5 Max,PyTorch 2.10 CPU,FP32,4 线程,取 5–7 次的中位数;每个内核在独立进程中运行,交替运行两轮。测量时一个 50M 训练任务正在使用 GPU,绝对耗时有噪声。旧版 = 0.3.1a3 + minimind `24896cd`。 | 检查项 | 0.3.1a3 + minimind | 0.3.2 原生 | |---|---|---| -| Logits:`out/chronos_256_moe.pth` 与 50M 英译中 checkpoint(完整、padding、prefill+decode 三条路径) | 参考 | 逐位一致(最大绝对差 0) | +| Logits:`out/chronos_256_moe.pth` 与 50M demo checkpoint(完整、padding、prefill+decode 三条路径) | 参考 | 逐位一致(最大绝对差 0) | | 随机初始化的 `state_dict` 和 config 取值(同一种子) | 参考 | 一致 | | 无缓存、无网络时 `import chronos` | 失败(`git clone` 报错) | 正常 | | minimind 在 `sys.path` 上,导入了顶层 `model` 模块 | 是 | 否 | @@ -193,7 +193,7 @@ Web UI 新增 Tokenizer Tab(抽取 → 训练 → 评估)和 Serve Tab(启 - **编译的 micro step**(`chronos/mlx/training/compiled.py`,默认开启)。损失和梯度按签名各编译成一张 `mx.compile` 图。实际执行的循环深度、ACT 开关、输入形状和普通 config 取值都属于签名,所以训练步不会复用过期的图。权重和 RNG 状态同时是编译的输入和输出,dropout 与缓存未命中模拟消耗 RNG 流的方式与之前完全相同。多模态 batch、懒加载(分页)专家,以及任何 trace 失败的签名都以 eager 方式运行,并记录一次原因。用 `CHRONOS_MLX_COMPILE=auto|on|off` 或 recipe 的 `extra.mlx_compile` 切换。 - **激活重计算**(`chronos/mlx/checkpointing.py`,`activation_checkpointing=true`;规划器的内存受限档位会自动开启)。每次 block 调用成为一个重计算区域。在这里直接用 `mx.checkpoint` 会出错,两处问题都在 mlx 0.32.4 上实测确认。第一,重计算时 dropout mask 和缓存未命中抽样会被重新抽取;现在前向时记录,重计算时回放。第二,区域两次返回同一个图节点时,只有一份能拿到梯度。只要 SWA 窗口覆盖整个序列,block 的 KV cache 和它的共享视图就是这样一对(全长切片是同一节点上的新 Python 对象),梯度偏差达 37–40%。因此区域输出按 dtype 打包成每种 dtype 一个数组。 -**2026-11-08 实测。** 条件:Apple M5 Max,mlx 0.32.4,bf16,50M 英译中 recipe 的模型配置,随机初始化,真实英译中分片,recipe 的步长(SFT 每步 8 个 micro-batch,预训练 16 个)。取 16 步(SFT)或 12 步(预训练)后半段的中位数。72 小时训练任务同时在同一块 GPU 上运行,绝对耗时有噪声。 +**2026-11-08 实测。** 条件:Apple M5 Max,mlx 0.32.4,bf16,50M demo recipe(`demos/translation/recipes/`)的模型配置,随机初始化,该 demo 的真实训练分片,recipe 的步长(SFT 每步 8 个 micro-batch,预训练 16 个)。取 16 步(SFT)或 12 步(预训练)后半段的中位数。72 小时训练任务同时在同一块 GPU 上运行,绝对耗时有噪声。 | Micro step | SFT s/步 | SFT 峰值内存 | 预训练 s/步 | 预训练峰值内存 | |---|---|---|---|---| @@ -521,7 +521,7 @@ pip install "Project_Chronos[mlx]" **可选依赖组**(版本已固定): ```bash pip install "Project_Chronos[serve]" # chronos serve / chronos monitor(fastapi、uvicorn) -pip install -e demos/translation # 英↔中 demo 扩展 chronos-translate,仅限仓库检出安装(不在 PyPI) +pip install -e demos/translation # 翻译 demo 扩展 chronos-translate,仅限仓库检出安装(不在 PyPI) ``` **vLLM 服务(可选,仅 Linux+CUDA):** @@ -867,7 +867,7 @@ Project_Chronos/ ├── docs/ │ └── vllm_integration.md ├── demos/ -│ └── translation/ # 英↔中扩展(chronos-translate):语料、训练、模型包;不进 wheel +│ └── translation/ # demo 扩展(chronos-translate):语料、训练、模型包、50M demo 模型;不进 wheel ├── pyproject.toml └── README.md / README_zh.md / THIRD_PARTY_NOTICES.md ``` @@ -917,7 +917,7 @@ timeline : int8 KV、MLA 吸收式解码、共享路由(显式开启) : 自动规划器、v3 预设、分词器与 serve Tab Next - : 50M 英译中结果与消融 + : 0.3.3 循环设置的多种子与更大规模实验 : 自推测解码:在更大模型与懒加载专家下真正提速的起草路径 : 大规模激活语料训练 IntentClassifier : 7B+ checkpoint 基准测试 diff --git a/chronos/__init__.py b/chronos/__init__.py index a858ebd..26dbd98 100644 --- a/chronos/__init__.py +++ b/chronos/__init__.py @@ -1,5 +1,5 @@ # Project Chronos: On-device low-latency lookahead dual-layer MoE inference -__version__ = "0.3.3a0" +__version__ = "0.3.3a1" # Register with HuggingFace AutoXXX factories so # AutoModelForCausalLM.from_pretrained("path/to/chronos") works out of the box. diff --git a/chronos/model/planner.py b/chronos/model/planner.py index 73e6535..1198e02 100644 --- a/chronos/model/planner.py +++ b/chronos/model/planner.py @@ -588,6 +588,20 @@ def save_interval(steps: float) -> int: sources["recurrent_order"] = "measured, 15M one seed: block beat layer at equal step with a healthy dt" sources["recurrent_dt_param"] = "measured, 15M: the clamp default collapsed onto its floor and stopped learning" sources["recurrent_dt_init"] = "estimated: horizon 2 / depth (15M, depth 4 and 8; other sizes not measured)" + # The other loop knobs keep the values the 15M runs favoured; each source says why. + loop_knobs.update(recurrent_later_topk=0, recurrent_later_groups=0, lambda_iteration_cluster=0.0, + lookahead_teacher_scope='first', recurrent_dt_min=0.05, recurrent_dt_max=1.0) + sources["recurrent_later_topk"] = ("measured, 15M one seed: 1 expert in later iterations raised valid CE " + "4.543 -> 4.600 at the same speed") + sources["recurrent_later_groups"] = ("measured, 15M one seed: the full block is best per step; 2 of 4 layers " + "in later iterations give about 1.3x steps per second at depth 4 for up to 0.055 more training loss " + "per step; never 1 (one layer looping alone)") + sources["lambda_iteration_cluster"] = ("measured, 15M: with cluster locality on, 99.5-100% of later-iteration " + "clusters are already used at iteration 0") + sources["lookahead_teacher_scope"] = ("measured, 15M: the lookahead head picks the right cluster for 0.97 of " + "tokens at every iteration") + sources["recurrent_dt_min"] = sources["recurrent_dt_max"] = ("measured, 15M: sigmoid arms with range 0.05-1 " + "(depth 4) and 0.1-1 (depth 8) learned dt 0.14-0.30, inside the range") model = dict( architecture_version=3, recurrent_attention_mode="shared_first", rope_layout="per_head", hidden_size=h, num_hidden_layers=L, num_attention_heads=arch.num_attention_heads, diff --git a/chronos/tuning/chronos_auto_tuner.py b/chronos/tuning/chronos_auto_tuner.py index 1bacf27..24c2c3a 100644 --- a/chronos/tuning/chronos_auto_tuner.py +++ b/chronos/tuning/chronos_auto_tuner.py @@ -11,6 +11,10 @@ from typing import List +# Knobs that only change a looped model: searching any of them makes the probe loop. +LOOP_KEYS = ("recurrent_order", "recurrent_dt_param", "recurrent_dt_min", "recurrent_dt_max", "recurrent_dt_init", + "recurrent_later_topk", "recurrent_later_groups", "lambda_iteration_cluster", "lookahead_teacher_scope") + from chronos.tuning._base_tuner import ( AutoTuner, SearchSpaceConfig, AutoTuneResult, _gc_collect, ) @@ -54,11 +58,10 @@ class ChronosSearchSpaceConfig(SearchSpaceConfig): lambda_lookahead_min: float = 1e-3 lambda_lookahead_max: float = 1.0 - # Loop knobs (0.3.3a0). They only matter for a looped model, so searching either one makes the probe model - # run `probe_recurrent_depth` iterations (the other probes stay at depth 1). The probe loss comes from a few - # steps and cannot resolve differences as small as dt 0.2-0.5 at depth 4: treat these two as a coarse screen. - # `recurrent_later_groups` is not searched: it trades quality for cost, so a loss-only search would always - # pick the full block. + # Loop knobs. They only matter for a looped model, so searching any of them makes the probe model run + # `probe_recurrent_depth` iterations over `probe_recurrent_layers` block layers (other probes stay at depth 1). + # The probe score comes from a few steps and cannot resolve differences as small as dt 0.2-0.5 at depth 4: + # treat the loop knobs as a coarse screen. tune_recurrent_order: bool = False recurrent_order_choices: List[str] = field(default_factory=lambda: ["layer", "block"]) @@ -66,7 +69,34 @@ class ChronosSearchSpaceConfig(SearchSpaceConfig): recurrent_dt_init_min: float = 0.1 recurrent_dt_init_max: float = 1.0 + tune_recurrent_dt_param: bool = False + recurrent_dt_param_choices: List[str] = field(default_factory=lambda: ["clamp", "sigmoid"]) + + tune_recurrent_dt_min: bool = False + recurrent_dt_min_min: float = 0.01 + recurrent_dt_min_max: float = 0.2 + + tune_recurrent_dt_max: bool = False + recurrent_dt_max_min: float = 0.5 + recurrent_dt_max_max: float = 1.0 + + # Cost knobs: they remove expert routes or layer calls from later iterations, so the cross-entropy score + # alone favours "off"; each trial also logs its median step time. later_groups never offers 1: a single + # layer looping alone is no longer a block loop. + tune_recurrent_later_topk: bool = False + recurrent_later_topk_choices: List[int] = field(default_factory=lambda: [0, 1]) + + tune_recurrent_later_groups: bool = False + recurrent_later_groups_choices: List[int] = field(default_factory=lambda: [0, 2]) + + tune_lambda_iteration_cluster: bool = False + lambda_iteration_cluster_choices: List[float] = field(default_factory=lambda: [0.0, 0.1, 0.3, 1.0]) + + tune_lookahead_teacher_scope: bool = False + lookahead_teacher_scope_choices: List[str] = field(default_factory=lambda: ["first", "iterations"]) + probe_recurrent_depth: int = 4 + probe_recurrent_layers: int = 4 class ChronosAutoTuner(AutoTuner): @@ -129,11 +159,35 @@ def _sample_params(self, trial, ss) -> dict: if ss.tune_recurrent_order: params["recurrent_order"] = trial.suggest_categorical( "recurrent_order", ss.recurrent_order_choices) + if ss.tune_recurrent_dt_param: + params["recurrent_dt_param"] = trial.suggest_categorical( + "recurrent_dt_param", ss.recurrent_dt_param_choices) + if ss.tune_recurrent_dt_min: + params["recurrent_dt_min"] = trial.suggest_float( + "recurrent_dt_min", ss.recurrent_dt_min_min, ss.recurrent_dt_min_max, log=True) + if ss.tune_recurrent_dt_max: + params["recurrent_dt_max"] = trial.suggest_float( + "recurrent_dt_max", ss.recurrent_dt_max_min, ss.recurrent_dt_max_max) if ss.tune_recurrent_dt_init: params["recurrent_dt_init"] = trial.suggest_float( "recurrent_dt_init", ss.recurrent_dt_init_min, ss.recurrent_dt_init_max, log=True) - if ss.tune_recurrent_order or ss.tune_recurrent_dt_init: + if ss.tune_recurrent_later_topk: + params["recurrent_later_topk"] = trial.suggest_categorical( + "recurrent_later_topk", ss.recurrent_later_topk_choices) + if ss.tune_recurrent_later_groups and params.get("recurrent_order", "block") == "block": + # Prefix blocks exist only in block order; a trial that drew layer order skips this knob. + params["recurrent_order"] = "block" + params["recurrent_later_groups"] = trial.suggest_categorical( + "recurrent_later_groups", ss.recurrent_later_groups_choices) + if ss.tune_lambda_iteration_cluster: + params["lambda_iteration_cluster"] = trial.suggest_categorical( + "lambda_iteration_cluster", ss.lambda_iteration_cluster_choices) + if ss.tune_lookahead_teacher_scope: + params["lookahead_teacher_scope"] = trial.suggest_categorical( + "lookahead_teacher_scope", ss.lookahead_teacher_scope_choices) + if any(key in params for key in LOOP_KEYS): params["recurrent_depth"] = max(2, int(ss.probe_recurrent_depth)) + params["num_hidden_layers"] = 2 + max(1, int(ss.probe_recurrent_layers)) else: params["lambda_balance"] = 5e-4 params["lambda_temporal"] = 1e-3 @@ -168,8 +222,7 @@ def _run_probe_trial( load_checkpoint_state_dict, load_state_dict_controlled, ) - from chronos.model.temporal_loss import total_loss - from chronos.model.moe_chronos import ChronosMOEFeedForward + from chronos.trainer.loss_mixin import chronos_loss_term _, device = resolve_training_device("auto") if device == "mlx": @@ -185,12 +238,14 @@ def _run_probe_trial( kv = int(params.get("kv_latent_dim", 64)) real = self._probe_tokens(dataset_path, kwargs.get("tokenizer_dir")) vocab_kw = {"vocab_size": real[1]} if real else {} - loop = {} + layers = int(params.get("num_hidden_layers", 4)) + # The searched objective weights reach the model config, so the probe trains on the objective they set. + loop = {key: params[key] for key in ("lambda_lookahead",) if key in params} if params.get("recurrent_depth"): depth = int(params["recurrent_depth"]) - loop = dict(recurrent_enabled=True, recurrent_depth=depth, min_recurrent_depth=max(1, depth // 2), + loop.update(recurrent_enabled=True, recurrent_depth=depth, min_recurrent_depth=max(1, depth // 2), max_inference_depth=depth, prelude_layers=1, coda_layers=1) - for key in ("recurrent_order", "recurrent_dt_init"): + for key in LOOP_KEYS: if key in params: loop[key] = params[key] @@ -212,7 +267,7 @@ def _run_probe_trial( ) except Exception: cfg = ChronosConfig( - hidden_size=hs, num_hidden_layers=4, num_experts=ne, + hidden_size=hs, num_hidden_layers=layers, num_experts=ne, num_experts_per_tok=2, num_shared_experts=ns, kv_latent_dim=kv, lookahead_steps=la, lambda_balance=lb, lambda_temporal=lt, @@ -221,7 +276,7 @@ def _run_probe_trial( else: cfg = ChronosConfig( hidden_size=hs, - num_hidden_layers=4, + num_hidden_layers=layers, num_experts=ne, num_experts_per_tok=2, num_shared_experts=ns, @@ -238,6 +293,9 @@ def _run_probe_trial( # A checkpoint's smaller vocabulary cannot embed this tokenizer's ids. self._put({"type": "log", "line": f"[probe] model vocab {cfg.vocab_size} < tokenizer vocab {real[1]}; trials use random token ids"}) real = None + if any(key in loop for key in ("recurrent_dt_init", "recurrent_dt_min", "recurrent_dt_max")): + # Independently drawn bounds can exclude the drawn init; start inside the range instead. + cfg.recurrent_dt_init = min(max(float(cfg.recurrent_dt_init), float(cfg.recurrent_dt_min)), float(cfg.recurrent_dt_max)) model = ChronosForCausalLM(cfg).to(device) if os.path.exists(model_id): try: @@ -251,35 +309,36 @@ def _run_probe_trial( seq_len = 64 batch = 2 model.train() - losses = [] + losses, step_times = [], [] for step in range(probe_steps): if real: from chronos.tuning.probe_data import probe_batch ids = torch.from_numpy(probe_batch(real[0], step, batch, seq_len)).to(device) else: ids = torch.randint(0, vocab_size, (batch, seq_len), device=device) - out, _ = model(ids, labels=ids, legacy_output=True) - moe_layers = [l.mlp for l in model.model.layers - if isinstance(l.mlp, ChronosMOEFeedForward)] - if moe_layers and moe_layers[0].last_router_probs is not None: - probs = torch.stack( - [l.last_router_probs for l in moe_layers], dim=2 - ).mean(dim=2) - loss = total_loss(out.loss, out.aux_loss, probs, lb, lt) - else: - loss = out.loss + out.aux_loss - + t0 = time.perf_counter() + out, lookahead_probs = model(ids, labels=ids, legacy_output=True) + # Train on the same objective the trainers use (every weighted term of the config), but score the + # trial by cross-entropy: a score that includes the weighted terms rewards small weights for their own sake. + loss = chronos_loss_term(model, out.loss, lookahead_probs, cfg, aux_loss=out.aux_loss, + input_ids=ids, labels=ids) loss.backward() optimizer.step() optimizer.zero_grad(set_to_none=True) - losses.append(loss.item()) - trial.report(loss.item(), step=step) + ce = float(out.loss.item()) + step_times.append(time.perf_counter() - t0) + losses.append(ce) + trial.report(ce, step=step) if trial.should_prune(): del model _gc_collect() raise optuna.exceptions.TrialPruned() final_loss = sum(losses[-max(1, probe_steps // 5):]) / max(1, probe_steps // 5) + step_s = sorted(step_times)[len(step_times) // 2] + if hasattr(trial, "set_user_attr"): + trial.set_user_attr("step_time_s", step_s) + self._put({"type": "log", "line": f" probe cross-entropy {final_loss:.4f}, median step {step_s * 1000:.0f} ms"}) del model _gc_collect() return final_loss @@ -292,7 +351,7 @@ def get_best_chronos_config_patch(self) -> dict: "lambda_balance", "lambda_temporal", "lambda_lookahead", "lookahead_steps", "hidden_size", "num_experts", "num_shared_experts", "kv_latent_dim", - "recurrent_order", "recurrent_dt_init", + *LOOP_KEYS, ): if key in p: patch[key] = p[key] diff --git a/chronos/webui/i18n/en.json b/chronos/webui/i18n/en.json index 76a9e8d..e204153 100644 --- a/chronos/webui/i18n/en.json +++ b/chronos/webui/i18n/en.json @@ -262,7 +262,7 @@ "field.lambda_balance.label": "Load balance", "field.lambda_cluster_locality.help": "Trains tokens in a segment to stay inside one expert cluster, so lazy inference can page whole clusters. 0 disables it.", "field.lambda_cluster_locality.label": "Cluster locality", - "field.lambda_iteration_cluster.help": "Weight on keeping every loop iteration of a token inside the same expert clusters (experts inside a cluster may still differ per iteration). 0 disables it.", + "field.lambda_iteration_cluster.help": "Weight on keeping every loop iteration of a token inside the same expert clusters (experts inside a cluster may still differ per iteration). 0 disables it. Recommended 0: with cluster locality on, 99.5-100% of the clusters that later iterations use are already used at iteration 0, so the term has almost nothing to act on.", "field.lambda_iteration_cluster.label": "Iteration cluster locality", "field.lambda_lookahead.help": "Trains the lookahead router to predict the experts future tokens will use.", "field.lambda_lookahead.label": "Lookahead loss", @@ -288,7 +288,7 @@ "field.lookahead_input_detach.label": "Detach lookahead input", "field.lookahead_steps.help": "How many future tokens the lookahead router predicts experts for, so they can be prefetched before they are needed.", "field.lookahead_steps.label": "Lookahead steps", - "field.lookahead_teacher_scope.help": "Which routing the LookaheadRouter learns to predict. first: loop iteration 0 only. iterations: the average over every iteration of the layer, so prefetch also covers the experts deeper iterations need.", + "field.lookahead_teacher_scope.help": "Which routing the LookaheadRouter learns to predict. first: loop iteration 0 only. iterations: the average over every iteration of the layer, so prefetch also covers the experts deeper iterations need. Recommended first: the lookahead head already picks the right cluster for 0.97 of tokens at every iteration.", "field.lookahead_teacher_scope.label": "Lookahead teacher", "field.max_gen_len.help": "Tokens generated per GRPO rollout.", "field.max_gen_len.label": "Rollout length", @@ -414,9 +414,9 @@ "field.recurrent_distill_coef.label": "Depth distillation", "field.recurrent_dt_init.help": "Starting value of the learned loop step size dt, the same for every recurrent group. One iteration is an Euler step, so depth x dt is the integration horizon T. The planner sets dt = min(0.5, 2 / depth), which holds T = 2: 0.5 at depth 4, 0.25 at depth 8. Measured at 15M (one seed per arm): the loss is flat for dt from 0.2 to 0.5 at depth 4; at depth 8, dt 0.25 beat 0.5 and a start of 0.0625 was far worse. T = 2 is a safe start, not a fitted optimum: a sigmoid run that started there settled at T of about 1.2. The kernel default stays 0.25.", "field.recurrent_dt_init.label": "Initial step size", - "field.recurrent_dt_max.help": "Upper bound of dt. 1.0 applies the whole block update to the residual stream in one iteration.", + "field.recurrent_dt_max.help": "Upper bound of dt. 1.0 applies the whole block update to the residual stream in one iteration. Recommended 1.0 as the bound; a dt fixed at 1.0 was worse than 0.5 at depths 4 and 8.", "field.recurrent_dt_max.label": "Largest step size", - "field.recurrent_dt_min.help": "Lower bound of dt. With the clamp parameterisation a dt that reaches it stops learning; at the default 0.05, dt collapsed onto this bound in our depth-4 and depth-8 runs.", + "field.recurrent_dt_min.help": "Lower bound of dt. With the clamp parameterisation a dt that reaches it stops learning; at the default 0.05, dt collapsed onto this bound in our depth-4 and depth-8 runs. Recommended 0.05 together with the sigmoid parameterisation, which keeps dt trainable inside [dt_min, dt_max]; learned dt stayed between 0.14 and 0.30 in the 15M runs.", "field.recurrent_dt_min.label": "Smallest step size", "field.recurrent_dt_param.help": "How the learned loop step size dt is stored. clamp: dt = clamp(w, dt_min, dt_max); the gradient is zero once w leaves the range, so a dt that falls below dt_min can never come back. sigmoid: dt = dt_min + (dt_max - dt_min) * sigmoid(w); the gradient never vanishes, so dt can still move. Both start at the configured initial dt. Measured in a 15M model (one seed per arm): the clamp default let dt collapse onto its floor of 0.05, which was worse at equal step than a healthy dt; a sigmoid dt kept moving and settled where the loss was flat. The planner recommends sigmoid.", "field.recurrent_dt_param.label": "Step-size parameterisation", @@ -424,9 +424,9 @@ "field.recurrent_enabled.label": "Recurrent depth", "field.recurrent_groups.help": "How many layers between prelude and coda form looping groups. Empty derives it from layers − prelude − coda.", "field.recurrent_groups.label": "Recurrent groups", - "field.recurrent_later_groups.help": "Block order only: loop iterations after the first run only the first N layers of the block (0 = all layers). Fewer layer calls make deeper iterations cheaper and touch fewer expert pages. Needs shared_first or first_only attention.", + "field.recurrent_later_groups.help": "Block order only: loop iterations after the first run only the first N layers of the block (0 = all layers). Fewer layer calls make deeper iterations cheaper and touch fewer expert pages. Needs shared_first or first_only attention. Recommended 0 (the full block). At 15M, 2 of 4 layers gave about 1.3x steps per second for up to 0.055 more training loss per step. Do not use 1: one layer looping alone is no longer a block loop.", "field.recurrent_later_groups.label": "Later-iteration layers", - "field.recurrent_later_topk.help": "Experts per token in loop iterations after the first (0 = same as the first iteration). The shared expert always runs. A smaller value makes deeper iterations cheaper and loads fewer expert pages.", + "field.recurrent_later_topk.help": "Experts per token in loop iterations after the first (0 = same as the first iteration). The shared expert always runs. A smaller value makes deeper iterations cheaper and loads fewer expert pages. Recommended 0: at 15M, 1 expert raised validation CE from 4.543 to 4.600 with no speed gain.", "field.recurrent_later_topk.label": "Later-iteration top-k", "field.recurrent_order.help": "Order of the loop. layer: each recurrent layer repeats depth times in a row. block: the whole group of recurrent layers repeats, so iteration 0 is a normal depth-1 pass and the loop refines it. Changes the computation: not interchangeable with a layer-order checkpoint. Measured at 15M (one seed per arm): with a trainable step size block had the lower loss at equal step; with step sizes fixed at 1.0 layer won late. The planner recommends block.", "field.recurrent_order.label": "Loop order", @@ -538,7 +538,7 @@ "field.tune.n_trials.label": "Trials", "field.tune.probe_steps.help": "Training steps per trial; poor trials are pruned early.", "field.tune.probe_steps.label": "Steps per trial", - "field.tune.search.help": "Parameters the search may change.", + "field.tune.search.help": "Parameters the search may change. The score is the probe's cross-entropy. Any loop knob makes the probe a looped model (the configured depth, 4 block layers); treat loop knobs as a coarse screen. recurrent_later_topk and recurrent_later_groups save time, not loss: compare each trial's step time in the log.", "field.tune.search.label": "Search over", "field.use_hybrid_attention.help": "Alternate MLA (compressed latent KV) and sliding-window layers. Keeps the KV cache small at long context.", "field.use_hybrid_attention.label": "Hybrid attention", diff --git a/chronos/webui/i18n/ja.json b/chronos/webui/i18n/ja.json index 6210b37..539da2f 100644 --- a/chronos/webui/i18n/ja.json +++ b/chronos/webui/i18n/ja.json @@ -262,7 +262,7 @@ "field.lambda_balance.label": "負荷分散", "field.lambda_cluster_locality.help": "区間内のトークンを 1 つのエキスパートクラスタ内にとどめるよう学習させ、遅延推論でクラスタ単位のページングを可能にします。0 で無効。", "field.lambda_cluster_locality.label": "クラスタ局所性", - "field.lambda_iteration_cluster.help": "同じトークンのすべてのループ反復を同じエキスパートクラスタ内に収める重み(クラスタ内のエキスパートは反復ごとに異なってよい)。0 で無効。", + "field.lambda_iteration_cluster.help": "同じトークンのすべてのループ反復を同じエキスパートクラスタ内に収める重み(クラスタ内のエキスパートは反復ごとに異なってよい)。0 で無効。推奨は 0:クラスタ局所性が有効なとき、後の反復が使うクラスタの 99.5-100% は反復 0 ですでに使われており、この項が働く余地はほとんどない。", "field.lambda_iteration_cluster.label": "反復クラスタ局所性", "field.lambda_lookahead.help": "将来のトークンが使うエキスパートを予測するよう先読みルーターを学習させます。", "field.lambda_lookahead.label": "先読み損失", @@ -288,7 +288,7 @@ "field.lookahead_input_detach.label": "ルックアヘッド入力の勾配遮断", "field.lookahead_steps.help": "先読みルーターが何トークン先までエキスパートを予測するか。必要になる前にプリフェッチできます。", "field.lookahead_steps.label": "先読みステップ", - "field.lookahead_teacher_scope.help": "LookaheadRouter が予測を学ぶルーティング。first:反復 0 のみ。iterations:その層の全反復の平均で、先読みがより深い反復に必要なエキスパートも対象にする。", + "field.lookahead_teacher_scope.help": "LookaheadRouter が予測を学ぶルーティング。first:反復 0 のみ。iterations:その層の全反復の平均で、先読みがより深い反復に必要なエキスパートも対象にする。推奨は first:先読みヘッドはどの反復でもトークンの 0.97 で正しいクラスタを選んでいる。", "field.lookahead_teacher_scope.label": "先読み教師", "field.max_gen_len.help": "GRPO のロールアウトごとに生成するトークン数。", "field.max_gen_len.label": "ロールアウト長", @@ -414,9 +414,9 @@ "field.recurrent_distill_coef.label": "深さ蒸留", "field.recurrent_dt_init.help": "学習するループのステップ幅 dt の開始値。すべてのループグループで共通。1 回の反復はオイラー法の 1 ステップなので、深さ × dt が積分区間 T になる。プランナーは dt = min(0.5, 2 / 深さ) とし、T = 2 に固定する:深さ 4 で 0.5、深さ 8 で 0.25。15M モデルの実測(各条件シード 1 つ):深さ 4 では dt が 0.2〜0.5 の間で損失は同じ。深さ 8 では dt 0.25 が 0.5 に勝ち、0.0625 からの開始は大きく悪化した。T = 2 は安全な出発点であり、当てはめた最適値ではない:そこから始めた sigmoid の実験は T が約 1.2 に落ち着いた。カーネルの既定値は 0.25 のまま。", "field.recurrent_dt_init.label": "初期ステップ幅", - "field.recurrent_dt_max.help": "dt の上限。1.0 は 1 回の反復でブロックの更新をそのまま残差ストリームに加える。", + "field.recurrent_dt_max.help": "dt の上限。1.0 は 1 回の反復でブロックの更新をそのまま残差ストリームに加える。上限としては 1.0 を推奨する。深さ 4 と 8 の両方で、dt を 1.0 に固定すると 0.5 より悪かった。", "field.recurrent_dt_max.label": "最大ステップ幅", - "field.recurrent_dt_min.help": "dt の下限。clamp では下限に達した dt は学習しなくなる。既定の 0.05 では、深さ 4 と深さ 8 の実測で dt がこの下限に落ち込んだ。", + "field.recurrent_dt_min.help": "dt の下限。clamp では下限に達した dt は学習しなくなる。既定の 0.05 では、深さ 4 と深さ 8 の実測で dt がこの下限に落ち込んだ。推奨は 0.05 で、sigmoid パラメータ化と組み合わせると dt は [dt_min, dt_max] の中で学習し続ける。15M の実測で学習された dt は 0.14 から 0.30 の間だった。", "field.recurrent_dt_min.label": "最小ステップ幅", "field.recurrent_dt_param.help": "学習するループのステップ幅 dt の保持方法。clamp:dt = clamp(w, dt_min, dt_max)。w が範囲外に出ると勾配がゼロになり、dt_min を下回った dt は戻れない。sigmoid:dt = dt_min + (dt_max - dt_min) * sigmoid(w)。勾配が消えないため dt は動き続けられる。どちらも設定した初期 dt から始まる。15M モデルの実測(各条件シード 1 つ):clamp の既定では dt が下限 0.05 に張り付き、同じステップ数で健全な dt より悪かった。sigmoid の dt は動き続け、損失が変わらない位置に落ち着いた。プランナーは sigmoid を推奨する。", "field.recurrent_dt_param.label": "ステップ幅のパラメータ化", @@ -424,9 +424,9 @@ "field.recurrent_enabled.label": "再帰的な深さ", "field.recurrent_groups.help": "プレリュードとコーダの間で反復するグループを構成する層の数。空欄なら 層数 − プレリュード − コーダ から求めます。", "field.recurrent_groups.label": "再帰グループ数", - "field.recurrent_later_groups.help": "ブロック順序のみ:最初の反復より後のループ反復では、ブロックの先頭 N 層だけを実行する(0 で全層)。レイヤー呼び出しが減り、深い反復が安くなり、触れるエキスパートページも減る。shared_first または first_only の注意モードが必要。", + "field.recurrent_later_groups.help": "ブロック順序のみ:最初の反復より後のループ反復では、ブロックの先頭 N 層だけを実行する(0 で全層)。レイヤー呼び出しが減り、深い反復が安くなり、触れるエキスパートページも減る。shared_first または first_only の注意モードが必要。推奨は 0(ブロック全体)。15M の実測では 4 層中 2 層にすると毎秒のステップ数が約 1.3 倍になり、ステップあたりの訓練損失は最大 0.055 高かった。1 は使わない:1 層だけの自己ループはもうブロックループではない。", "field.recurrent_later_groups.label": "後続反復のレイヤー数", - "field.recurrent_later_topk.help": "最初の反復より後のループ反復でトークンごとに使うエキスパート数(0 で最初の反復と同じ)。共有エキスパートは常に動作する。小さくすると深い反復が安くなり、読み込むエキスパートページも減る。", + "field.recurrent_later_topk.help": "最初の反復より後のループ反復でトークンごとに使うエキスパート数(0 で最初の反復と同じ)。共有エキスパートは常に動作する。小さくすると深い反復が安くなり、読み込むエキスパートページも減る。推奨は 0:15M の実測では 1 エキスパートにすると検証 CE が 4.543 から 4.600 に上がり、速度は変わらなかった。", "field.recurrent_later_topk.label": "後続反復の top-k", "field.recurrent_order.help": "ループの実行順序。layer:各再帰層を depth 回連続で反復。block:再帰層のグループ全体を反復し、反復 0 は通常の深さ 1 の順伝播になり、以降がそれを精緻化する。計算が異なるため、layer 順序のチェックポイントとは互換性がない。15M モデルの実測(各条件シード 1 つ):ステップ幅を学習できるとき、同じステップ数で block の損失が低かった。ステップ幅を 1.0 に固定すると後半は layer が良かった。プランナーは block を推奨する。", "field.recurrent_order.label": "ループ順序", @@ -538,7 +538,7 @@ "field.tune.n_trials.label": "試行回数", "field.tune.probe_steps.help": "試行ごとの学習ステップ数。成績の悪い試行は早めに打ち切られます。", "field.tune.probe_steps.label": "試行あたりのステップ数", - "field.tune.search.help": "探索で変更してよいパラメータ。", + "field.tune.search.help": "探索で変更してよいパラメータ。評価値はプローブモデルの交差エントロピー。ループ系のパラメータを選ぶとプローブはループモデルになる(設定した深さ、ブロック 4 層)。ループ系の結果は粗い選別として扱う。recurrent_later_topk と recurrent_later_groups が減らすのは損失ではなく時間なので、ログにある各試行のステップ時間と比べる。", "field.tune.search.label": "探索対象", "field.use_hybrid_attention.help": "MLA(圧縮潜在 KV)層とスライディングウィンドウ層を交互に配置します。長いコンテキストでも KV キャッシュを小さく保てます。", "field.use_hybrid_attention.label": "ハイブリッドアテンション", diff --git a/chronos/webui/i18n/zh-Hans.json b/chronos/webui/i18n/zh-Hans.json index 6e70a60..554cebd 100644 --- a/chronos/webui/i18n/zh-Hans.json +++ b/chronos/webui/i18n/zh-Hans.json @@ -262,7 +262,7 @@ "field.lambda_balance.label": "负载均衡", "field.lambda_cluster_locality.help": "训练同一片段内的 token 留在同一个专家簇里,懒加载推理就能整簇分页。0 表示关闭。", "field.lambda_cluster_locality.label": "簇局部性", - "field.lambda_iteration_cluster.help": "让同一 token 的每个循环迭代都落在相同的专家簇内的权重(簇内专家仍可逐迭代不同)。0 表示关闭。", + "field.lambda_iteration_cluster.help": "让同一 token 的每个循环迭代都落在相同的专家簇内的权重(簇内专家仍可逐迭代不同)。0 表示关闭。推荐 0:开启簇局部性时,后续迭代用到的簇有 99.5-100% 在第 0 轮已经用过,这一项几乎无事可做。", "field.lambda_iteration_cluster.label": "迭代簇局部性", "field.lambda_lookahead.help": "训练前瞻路由器预测未来 token 会用到的专家。", "field.lambda_lookahead.label": "前瞻损失", @@ -288,7 +288,7 @@ "field.lookahead_input_detach.label": "前瞻输入断梯度", "field.lookahead_steps.help": "前瞻路由为未来多少个 token 预测专家,以便在用到之前预取。", "field.lookahead_steps.label": "前瞻步数", - "field.lookahead_teacher_scope.help": "前瞻路由器学习预测哪一种路由。first:只取第 0 轮迭代。iterations:取该层所有迭代的平均,预取因此也覆盖更深迭代需要的专家。", + "field.lookahead_teacher_scope.help": "前瞻路由器学习预测哪一种路由。first:只取第 0 轮迭代。iterations:取该层所有迭代的平均,预取因此也覆盖更深迭代需要的专家。推荐 first:前瞻头在每一轮迭代上都已为 0.97 的 token 选对了簇。", "field.lookahead_teacher_scope.label": "前瞻教师", "field.max_gen_len.help": "每次 GRPO 采样生成的 token 数。", "field.max_gen_len.label": "采样长度", @@ -414,9 +414,9 @@ "field.recurrent_distill_coef.label": "深度蒸馏", "field.recurrent_dt_init.help": "学习到的循环步长 dt 的起始值,所有循环组相同。一次迭代是一个欧拉步,所以 深度 × dt 就是积分时程 T。规划器取 dt = min(0.5, 2 / 深度),即固定 T = 2:深度 4 为 0.5,深度 8 为 0.25。15M 模型实测(每组一个种子):深度 4 时 dt 在 0.2 到 0.5 之间损失持平;深度 8 时 dt 0.25 优于 0.5,而从 0.0625 起步则差得多。T = 2 是安全的起点,不是拟合出的最优:从这里起步的 sigmoid 实验最终停在 T 约 1.2。内核默认值仍为 0.25。", "field.recurrent_dt_init.label": "初始步长", - "field.recurrent_dt_max.help": "dt 的上界。1.0 表示一次迭代把整个块的更新量加到残差流上。", + "field.recurrent_dt_max.help": "dt 的上界。1.0 表示一次迭代把整个块的更新量加到残差流上。推荐以 1.0 作为上界;深度 4 和深度 8 下,dt 固定为 1.0 都比 0.5 差。", "field.recurrent_dt_max.label": "最大步长", - "field.recurrent_dt_min.help": "dt 的下界。使用 clamp 时,dt 到达下界就不再学习;默认 0.05 下,深度 4 和深度 8 的实测中 dt 都塌缩到了这个下界。", + "field.recurrent_dt_min.help": "dt 的下界。使用 clamp 时,dt 到达下界就不再学习;默认 0.05 下,深度 4 和深度 8 的实测中 dt 都塌缩到了这个下界。推荐 0.05,并配合 sigmoid 参数化,使 dt 在 [dt_min, dt_max] 内始终可学习;15M 实测中学到的 dt 在 0.14 到 0.30 之间。", "field.recurrent_dt_min.label": "最小步长", "field.recurrent_dt_param.help": "循环步长 dt 的存储方式。clamp:dt = clamp(w, dt_min, dt_max);w 越出范围后梯度为零,掉到 dt_min 以下的 dt 再也回不来。sigmoid:dt = dt_min + (dt_max - dt_min) * sigmoid(w);梯度不会消失,dt 仍可调整。两者都从设定的初始 dt 开始。15M 模型实测(每组一个种子):clamp 默认值会让 dt 塌缩到 0.05 的下界,同一步数下比健康的 dt 更差;sigmoid 的 dt 能继续移动,停在损失持平的位置。规划器推荐 sigmoid。", "field.recurrent_dt_param.label": "步长参数化", @@ -424,9 +424,9 @@ "field.recurrent_enabled.label": "循环深度", "field.recurrent_groups.help": "前置层与收尾层之间组成循环组的层数。留空则按 层数 − 前置 − 收尾 推导。", "field.recurrent_groups.label": "循环组数", - "field.recurrent_later_groups.help": "仅块顺序有效:第一轮之后的循环迭代只运行块内前 N 层(0 表示全部)。层调用更少,更深的迭代更便宜,也会访问更少的专家页。需要 shared_first 或 first_only 注意力模式。", + "field.recurrent_later_groups.help": "仅块顺序有效:第一轮之后的循环迭代只运行块内前 N 层(0 表示全部)。层调用更少,更深的迭代更便宜,也会访问更少的专家页。需要 shared_first 或 first_only 注意力模式。推荐 0(完整块)。15M 实测中 4 层只跑 2 层时,每秒步数约为 1.3 倍,每步训练损失最多高 0.055。不要设为 1:只剩一层自循环就不再是块循环。", "field.recurrent_later_groups.label": "后续迭代层数", - "field.recurrent_later_topk.help": "第一轮之后的循环迭代中每个 token 使用的专家数(0 表示与第一轮相同)。共享专家始终运行。较小的值让更深的迭代更便宜,也会加载更少的专家页。", + "field.recurrent_later_topk.help": "第一轮之后的循环迭代中每个 token 使用的专家数(0 表示与第一轮相同)。共享专家始终运行。较小的值让更深的迭代更便宜,也会加载更少的专家页。推荐 0:15M 实测中改为 1 个专家使验证 CE 从 4.543 升到 4.600,速度没有提升。", "field.recurrent_later_topk.label": "后续迭代 top-k", "field.recurrent_order.help": "循环的执行顺序。layer:每个循环层连续重复 depth 次。block:整组循环层一起重复,第 0 轮就是普通的单深度前向,后续轮次在其上精炼。计算方式不同,不能与 layer 顺序的检查点互换。15M 模型实测(每组一个种子):步长可训练时,同一步数下 block 的损失更低;步长固定为 1.0 时,layer 在后期更好。规划器推荐 block。", "field.recurrent_order.label": "循环顺序", @@ -538,7 +538,7 @@ "field.tune.n_trials.label": "试验次数", "field.tune.probe_steps.help": "每次试验的训练步数;表现差的试验会被提前剪枝。", "field.tune.probe_steps.label": "每次试验步数", - "field.tune.search.help": "搜索可以调整的参数。", + "field.tune.search.help": "搜索可以调整的参数。评分是探针模型的交叉熵。选中任一循环参数时,探针改为循环模型(使用配置的深度,块内 4 层);循环参数的结果只作粗筛。recurrent_later_topk 和 recurrent_later_groups 节省的是时间而不是损失,请对照日志里每次试验的步时。", "field.tune.search.label": "搜索参数", "field.use_hybrid_attention.help": "MLA(压缩潜在 KV)与滑动窗口层交替,长上下文时 KV 缓存仍然很小。", "field.use_hybrid_attention.label": "混合注意力", diff --git a/chronos/webui/i18n/zh-Hant.json b/chronos/webui/i18n/zh-Hant.json index 737f432..cc80fe3 100644 --- a/chronos/webui/i18n/zh-Hant.json +++ b/chronos/webui/i18n/zh-Hant.json @@ -262,7 +262,7 @@ "field.lambda_balance.label": "負載平衡", "field.lambda_cluster_locality.help": "訓練同一片段內的 token 留在同一個專家叢集,延遲載入推論就能整叢分頁。0 表示關閉。", "field.lambda_cluster_locality.label": "叢集區域性", - "field.lambda_iteration_cluster.help": "讓同一 token 的每個循環迭代都落在相同的專家簇內的權重(簇內專家仍可逐迭代不同)。0 表示關閉。", + "field.lambda_iteration_cluster.help": "讓同一 token 的每個循環迭代都落在相同的專家簇內的權重(簇內專家仍可逐迭代不同)。0 表示關閉。推薦 0:開啟簇局部性時,後續迭代用到的簇有 99.5-100% 在第 0 輪已經用過,這一項幾乎無事可做。", "field.lambda_iteration_cluster.label": "迭代簇局部性", "field.lambda_lookahead.help": "訓練前瞻路由器預測未來 token 會用到的專家。", "field.lambda_lookahead.label": "前瞻損失", @@ -288,7 +288,7 @@ "field.lookahead_input_detach.label": "前瞻輸入斷梯度", "field.lookahead_steps.help": "前瞻路由為未來多少個 token 預測專家,以便在用到之前預先擷取。", "field.lookahead_steps.label": "前瞻步數", - "field.lookahead_teacher_scope.help": "前瞻路由器學習預測哪一種路由。first:只取第 0 輪迭代。iterations:取該層所有迭代的平均,預取因此也涵蓋更深迭代需要的專家。", + "field.lookahead_teacher_scope.help": "前瞻路由器學習預測哪一種路由。first:只取第 0 輪迭代。iterations:取該層所有迭代的平均,預取因此也涵蓋更深迭代需要的專家。推薦 first:前瞻頭在每一輪迭代上都已為 0.97 的 token 選對了簇。", "field.lookahead_teacher_scope.label": "前瞻教師", "field.max_gen_len.help": "每次 GRPO 取樣生成的 token 數。", "field.max_gen_len.label": "取樣長度", @@ -414,9 +414,9 @@ "field.recurrent_distill_coef.label": "深度蒸餾", "field.recurrent_dt_init.help": "學習到的循環步長 dt 的起始值,所有循環組相同。一次迭代是一個歐拉步,所以 深度 × dt 就是積分時程 T。規劃器取 dt = min(0.5, 2 / 深度),即固定 T = 2:深度 4 為 0.5,深度 8 為 0.25。15M 模型實測(每組一個種子):深度 4 時 dt 在 0.2 到 0.5 之間損失持平;深度 8 時 dt 0.25 優於 0.5,而從 0.0625 起步則差得多。T = 2 是安全的起點,不是擬合出的最優:從這裡起步的 sigmoid 實驗最終停在 T 約 1.2。核心預設值仍為 0.25。", "field.recurrent_dt_init.label": "初始步長", - "field.recurrent_dt_max.help": "dt 的上界。1.0 表示一次迭代把整個區塊的更新量加到殘差流上。", + "field.recurrent_dt_max.help": "dt 的上界。1.0 表示一次迭代把整個區塊的更新量加到殘差流上。推薦以 1.0 作為上界;深度 4 和深度 8 下,dt 固定為 1.0 都比 0.5 差。", "field.recurrent_dt_max.label": "最大步長", - "field.recurrent_dt_min.help": "dt 的下界。使用 clamp 時,dt 到達下界就不再學習;預設 0.05 下,深度 4 和深度 8 的實測中 dt 都塌縮到了這個下界。", + "field.recurrent_dt_min.help": "dt 的下界。使用 clamp 時,dt 到達下界就不再學習;預設 0.05 下,深度 4 和深度 8 的實測中 dt 都塌縮到了這個下界。推薦 0.05,並搭配 sigmoid 參數化,使 dt 在 [dt_min, dt_max] 內始終可學習;15M 實測中學到的 dt 在 0.14 到 0.30 之間。", "field.recurrent_dt_min.label": "最小步長", "field.recurrent_dt_param.help": "循環步長 dt 的儲存方式。clamp:dt = clamp(w, dt_min, dt_max);w 超出範圍後梯度為零,掉到 dt_min 以下的 dt 再也回不來。sigmoid:dt = dt_min + (dt_max - dt_min) * sigmoid(w);梯度不會消失,dt 仍可調整。兩者都從設定的初始 dt 開始。15M 模型實測(每組一個種子):clamp 預設值會讓 dt 塌縮到 0.05 的下界,同一步數下比健康的 dt 更差;sigmoid 的 dt 能繼續移動,停在損失持平的位置。規劃器推薦 sigmoid。", "field.recurrent_dt_param.label": "步長參數化", @@ -424,9 +424,9 @@ "field.recurrent_enabled.label": "循環深度", "field.recurrent_groups.help": "前置層與收尾層之間組成循環組的層數。留空則按 層數 − 前置 − 收尾 推導。", "field.recurrent_groups.label": "循環組數", - "field.recurrent_later_groups.help": "僅區塊順序有效:第一輪之後的循環迭代只運行區塊內前 N 層(0 表示全部)。層呼叫更少,更深的迭代更便宜,也會存取更少的專家頁。需要 shared_first 或 first_only 注意力模式。", + "field.recurrent_later_groups.help": "僅區塊順序有效:第一輪之後的循環迭代只運行區塊內前 N 層(0 表示全部)。層呼叫更少,更深的迭代更便宜,也會存取更少的專家頁。需要 shared_first 或 first_only 注意力模式。推薦 0(完整區塊)。15M 實測中 4 層只跑 2 層時,每秒步數約為 1.3 倍,每步訓練損失最多高 0.055。不要設為 1:只剩一層自循環就不再是區塊循環。", "field.recurrent_later_groups.label": "後續迭代層數", - "field.recurrent_later_topk.help": "第一輪之後的循環迭代中每個 token 使用的專家數(0 表示與第一輪相同)。共享專家始終運行。較小的值讓更深的迭代更便宜,也會載入更少的專家頁。", + "field.recurrent_later_topk.help": "第一輪之後的循環迭代中每個 token 使用的專家數(0 表示與第一輪相同)。共享專家始終運行。較小的值讓更深的迭代更便宜,也會載入更少的專家頁。推薦 0:15M 實測中改為 1 個專家使驗證 CE 從 4.543 升到 4.600,速度沒有提升。", "field.recurrent_later_topk.label": "後續迭代 top-k", "field.recurrent_order.help": "循環的執行順序。layer:每個循環層連續重複 depth 次。block:整組循環層一起重複,第 0 輪就是普通的單深度前向,後續輪次在其上精煉。計算方式不同,不能與 layer 順序的檢查點互換。15M 模型實測(每組一個種子):步長可訓練時,同一步數下 block 的損失更低;步長固定為 1.0 時,layer 在後期更好。規劃器推薦 block。", "field.recurrent_order.label": "循環順序", @@ -538,7 +538,7 @@ "field.tune.n_trials.label": "試驗次數", "field.tune.probe_steps.help": "每次試驗的訓練步數;表現差的試驗會被提前剪枝。", "field.tune.probe_steps.label": "每次試驗步數", - "field.tune.search.help": "搜尋可以調整的參數。", + "field.tune.search.help": "搜尋可以調整的參數。評分是探針模型的交叉熵。選中任一循環參數時,探針改為循環模型(使用配置的深度,區塊內 4 層);循環參數的結果只作粗篩。recurrent_later_topk 和 recurrent_later_groups 節省的是時間而不是損失,請對照日誌裡每次試驗的步時。", "field.tune.search.label": "搜尋參數", "field.use_hybrid_attention.help": "MLA(壓縮潛在 KV)與滑動視窗層交替,長上下文時 KV 快取仍然很小。", "field.use_hybrid_attention.label": "混合注意力", diff --git a/chronos/webui/schema.py b/chronos/webui/schema.py index cb52088..1e3e7d5 100644 --- a/chronos/webui/schema.py +++ b/chronos/webui/schema.py @@ -37,7 +37,9 @@ class Field: nullable: bool = False # "auto" / empty allowed def as_dict(self) -> dict: - out = {k: v for k, v in asdict(self).items() if v not in (None, (), False) or k in ("default",)} + # Drop unset values only. A membership test such as `v not in (None, (), False)` also dropped 0 and 0.0 + # (0 == False), so min=0 and recommended=0 never reached the page. + out = {k: v for k, v in asdict(self).items() if k == "default" or not (v is None or v is False or v == ())} out["choices"] = list(self.choices) if self.choices else None out["scope"] = list(self.scope) return {k: v for k, v in out.items() if v is not None or k == "default"} @@ -88,14 +90,18 @@ def _f(key, group, type_, default=None, **kw) -> Field: choices=["shared_first", "per_iteration", "first_only"], recommended="shared_first", source="code"), _f("recurrent_order", "recurrent", "choice", "layer", choices=["layer", "block"], recommended="block", source="planner", advanced=True), - _f("recurrent_later_topk", "recurrent", "int", 0, min=0, max=16, step=1, advanced=True), - _f("recurrent_later_groups", "recurrent", "int", 0, min=0, max=64, step=1, advanced=True), + _f("recurrent_later_topk", "recurrent", "int", 0, min=0, max=16, step=1, recommended=0, + source="planner", advanced=True), + _f("recurrent_later_groups", "recurrent", "int", 0, min=0, max=64, step=1, recommended=0, + source="planner", advanced=True), _f("recurrent_dt_param", "recurrent", "choice", "clamp", choices=["clamp", "sigmoid"], recommended="sigmoid", source="planner", advanced=True), _f("recurrent_dt_init", "recurrent", "float", 0.25, min=0.01, max=1.0, step=0.01, recommended=0.5, source="planner", advanced=True), - _f("recurrent_dt_min", "recurrent", "float", 0.05, min=0.001, max=1.0, step=0.005, advanced=True), - _f("recurrent_dt_max", "recurrent", "float", 1.0, min=0.05, max=1.0, step=0.05, advanced=True), + _f("recurrent_dt_min", "recurrent", "float", 0.05, min=0.001, max=1.0, step=0.005, + recommended=0.05, source="planner", advanced=True), + _f("recurrent_dt_max", "recurrent", "float", 1.0, min=0.05, max=1.0, step=0.05, + recommended=1.0, source="planner", advanced=True), _f("adaptive_halting", "recurrent", "bool", False), _f("halting_threshold", "recurrent", "float", 0.99, min=0.5, max=1.0, step=0.001), _f("depth_schedule", "recurrent", "choice", "fixed_then_adaptive", @@ -148,9 +154,10 @@ def _f(key, group, type_, default=None, **kw) -> Field: recommended="data", source="rule", ref="window_median_record"), _f("cluster_locality_mode", "objectives", "choice", "window", choices=["window", "segment"], source="code", advanced=True), - _f("lambda_iteration_cluster", "objectives", "float", 0.0, min=0.0, max=10.0, step=0.01, advanced=True), + _f("lambda_iteration_cluster", "objectives", "float", 0.0, min=0.0, max=10.0, step=0.01, + recommended=0.0, source="planner", advanced=True), _f("lookahead_teacher_scope", "objectives", "choice", "first", choices=["first", "iterations"], - source="code", advanced=True), + recommended="first", source="planner", advanced=True), # ---- memory / lazy experts --------------------------------------------- _f("vram_budget_gb", "memory", "float", 4.0, min=0.01, max=256.0, step=0.01), _f("pinned_memory_max_fraction", "memory", "float", 0.25, min=0.01, max=0.95, step=0.01), @@ -338,7 +345,9 @@ def _f(key, group, type_, default=None, **kw) -> Field: _f("tune.probe_steps", "autotune", "int", 80, min=10, max=1000, step=10, scope=("autotune",)), _f("tune.search", "autotune", "multi", ["lambda_balance", "lambda_temporal", "lr"], choices=["lambda_balance", "lambda_temporal", "lambda_lookahead", "lookahead_steps", "lr", "hidden_size", - "num_experts", "num_shared_experts", "kv_latent_dim", "recurrent_order", "recurrent_dt_init"], + "num_experts", "num_shared_experts", "kv_latent_dim", "recurrent_order", "recurrent_dt_param", + "recurrent_dt_min", "recurrent_dt_max", "recurrent_dt_init", "recurrent_later_topk", + "recurrent_later_groups", "lambda_iteration_cluster", "lookahead_teacher_scope"], scope=("autotune",)), # ---- planner ----------------------------------------------------------- _f("plan.target_m", "planner", "float", 50.0, min=0.1, max=100000.0, step=0.1, scope=("planner",)), @@ -372,6 +381,8 @@ def _f(key, group, type_, default=None, **kw) -> Field: "pretrain", "sft", "dpo", "orpo", "grpo", "distill", "shared_first", "per_iteration", "first_only", "layer", "first", "iterations", "clamp", "sigmoid", "lambda_balance", "lambda_temporal", "lambda_lookahead", "lookahead_steps", "lr", "hidden_size", "num_experts", "num_shared_experts", "kv_latent_dim", "recurrent_order", "recurrent_dt_init", + "recurrent_dt_param", "recurrent_dt_min", "recurrent_dt_max", "recurrent_later_topk", "recurrent_later_groups", + "lambda_iteration_cluster", "lookahead_teacher_scope", "sync_on_demand", "on_demand", "fallback_diagnostic", }) diff --git a/demos/translation/README.md b/demos/translation/README.md index f0ff6c4..b8e630a 100644 --- a/demos/translation/README.md +++ b/demos/translation/README.md @@ -106,14 +106,14 @@ are for research use only. FLORES+ (CC BY-SA 4.0) is used for evaluation only. ## Released model: chronos-enzh-50m-sft -[`models/chronos-enzh-50m-sft/`](../../models/chronos-enzh-50m-sft/) packages a 49.7M-parameter Chronos v3 model (12.1M active per token) trained for 69 hours on one Apple M5 Max: configuration, tokenizer, trained prompt, decoding settings, a driver (`translate.py`) and the evaluation behind every number. The weights (99.5 MB, bfloat16 safetensors) are attached to the release [`enzh-50m-sft-v1`](https://github.com/FonaTech/Project_Chronos/releases/tag/enzh-50m-sft-v1). +[`models/chronos-enzh-50m-sft/`](models/chronos-enzh-50m-sft/) packages a 49.7M-parameter Chronos v3 model (12.1M active per token) trained for 69 hours on one Apple M5 Max: configuration, tokenizer, trained prompt, decoding settings, a driver (`translate.py`) and the evaluation behind every number. The weights (99.5 MB, bfloat16 safetensors) are attached to the release [`enzh-50m-sft-v1`](https://github.com/FonaTech/Project_Chronos/releases/tag/enzh-50m-sft-v1). - **Boundary:** English → Chinese only, one sentence (≤ 128 tokens) at a time. Chinese → English is disabled: English-only fine-tuning erased it (FLORES+ BLEU 0.00 after SFT). The driver refuses input outside the boundary. - **Measured** (MLX bf16, greedy): FLORES+ devtest BLEU 33.48 / chrF 29.52; WMT22 BLEU 32.60 / chrF 28.83; WMT24++ BLEU 26.89 / chrF 24.43 (split into sentences). - **Research use only.** The training corpora's licences are unknown; see the folder's `NOTICE`. ```bash -cd models/chronos-enzh-50m-sft +cd demos/translation/models/chronos-enzh-50m-sft pip install --pre "Project_Chronos[mlx]==0.3.2a0" # or "Project_Chronos==0.3.2a0" without MLX python translate.py --download && python translate.py "The meeting was moved to Friday." ``` @@ -149,3 +149,10 @@ bin/chronos bundle serve -- --port 11435 ``` `--speculative` needs `--repetition-penalty 1.0` (its default), because the penalty changes the target distribution at every position. `finetune` writes a `chronos-translate run` spec with `init_checkpoint` set to the bundle weights and refuses a run directory that already holds a run. Bundles built from data with unknown licences (WMT19, OPUS-100) say in `NOTICE.md` that the weights are for research only. + +## Kernel measurements taken on this demo + +Three measurements in the main README use this demo's model or data: +- Self-speculative decoding table: pretrain step-13,996 checkpoint, greedy, batch 1, FLORES+ dev en→zh prompts, draft depth 1, γ 4 (reports in `reports/speculative/`). +- 0.3.2 native-kernel parity: the logits of the 50M en→zh checkpoint are bit-identical between 0.3.1a3 + minimind and the native kernel; the timings ran while the 50M en→zh SFT run was training on the GPU. +- MLX compiled step and activation recomputation (2026-11-08): the 50M en→zh recipe's model config from random init, real en→zh shards. diff --git a/demos/translation/README_zh.md b/demos/translation/README_zh.md index 05caf76..8111b7a 100644 --- a/demos/translation/README_zh.md +++ b/demos/translation/README_zh.md @@ -100,14 +100,14 @@ FLORES+(CC BY-SA 4.0)只用于评测。 ## 已发布模型:chronos-enzh-50m-sft -[`models/chronos-enzh-50m-sft/`](../../models/chronos-enzh-50m-sft/) 打包了一个 4,969 万参数的 Chronos v3 模型(每个 token 激活 1,209 万),在一台 Apple M5 Max 上训练了 69 小时。目录中有模型配置、分词器、训练时的 prompt、解码参数、驱动脚本(`translate.py`),以及每个数字背后的评测。权重(99.5 MB,bfloat16 safetensors)作为附件放在 release [`enzh-50m-sft-v1`](https://github.com/FonaTech/Project_Chronos/releases/tag/enzh-50m-sft-v1) 中。 +[`models/chronos-enzh-50m-sft/`](models/chronos-enzh-50m-sft/) 打包了一个 4,969 万参数的 Chronos v3 模型(每个 token 激活 1,209 万),在一台 Apple M5 Max 上训练了 69 小时。目录中有模型配置、分词器、训练时的 prompt、解码参数、驱动脚本(`translate.py`),以及每个数字背后的评测。权重(99.5 MB,bfloat16 safetensors)作为附件放在 release [`enzh-50m-sft-v1`](https://github.com/FonaTech/Project_Chronos/releases/tag/enzh-50m-sft-v1) 中。 - **边界:** 只做英译中,每次一句(≤ 128 token)。中译英已屏蔽:只含英译中的微调抹掉了这一能力(SFT 后 FLORES+ BLEU 0.00)。驱动会拒绝边界以外的输入。 - **实测**(MLX bf16,贪心解码):FLORES+ devtest BLEU 33.48 / chrF 29.52;WMT22 BLEU 32.60 / chrF 28.83;WMT24++ BLEU 26.89 / chrF 24.43(分句)。 - **仅供研究使用。** 训练语料的许可未知,见目录中的 `NOTICE`。 ```bash -cd models/chronos-enzh-50m-sft +cd demos/translation/models/chronos-enzh-50m-sft pip install --pre "Project_Chronos[mlx]==0.3.2a0" # 不用 MLX 时装 "Project_Chronos==0.3.2a0" python translate.py --download && python translate.py "The meeting was moved to Friday." ``` @@ -143,3 +143,10 @@ bin/chronos bundle serve -- --port 11435 ``` `--speculative` 要求 `--repetition-penalty 1.0`(此时的默认值),因为惩罚会逐位置改变目标分布。`finetune` 写出一份 `init_checkpoint` 指向包内权重的 `chronos-translate run` spec;运行目录里已有训练时会拒绝写入。用许可未知的数据(WMT19、OPUS-100)训练的模型包,`NOTICE.md` 中写明权重仅供研究使用。 + +## 在本 demo 上做的内核实测 + +主 README 中有三项实测用到了本 demo 的模型或数据: +- 自推测解码表:预训练 step 13,996 checkpoint,greedy,batch 1,FLORES+ dev 英→中提示,起草深度 1,γ 4(报告在 `reports/speculative/`)。 +- 0.3.2 原生内核一致性:50M 英译中 checkpoint 的 logits 在 0.3.1a3 + minimind 与原生内核之间逐位一致;计时时 50M 英译中 SFT 正在 GPU 上训练。 +- MLX 编译步与激活重计算(2026-11-08):50M 英译中 recipe 的模型配置,随机初始化,真实英译中分片。 diff --git a/models/chronos-enzh-50m-sft/NOTICE b/demos/translation/models/chronos-enzh-50m-sft/NOTICE similarity index 100% rename from models/chronos-enzh-50m-sft/NOTICE rename to demos/translation/models/chronos-enzh-50m-sft/NOTICE diff --git a/models/chronos-enzh-50m-sft/README.md b/demos/translation/models/chronos-enzh-50m-sft/README.md similarity index 100% rename from models/chronos-enzh-50m-sft/README.md rename to demos/translation/models/chronos-enzh-50m-sft/README.md diff --git a/models/chronos-enzh-50m-sft/README_zh.md b/demos/translation/models/chronos-enzh-50m-sft/README_zh.md similarity index 100% rename from models/chronos-enzh-50m-sft/README_zh.md rename to demos/translation/models/chronos-enzh-50m-sft/README_zh.md diff --git a/models/chronos-enzh-50m-sft/config.json b/demos/translation/models/chronos-enzh-50m-sft/config.json similarity index 100% rename from models/chronos-enzh-50m-sft/config.json rename to demos/translation/models/chronos-enzh-50m-sft/config.json diff --git a/models/chronos-enzh-50m-sft/eval/boundary_long_input.py b/demos/translation/models/chronos-enzh-50m-sft/eval/boundary_long_input.py similarity index 100% rename from models/chronos-enzh-50m-sft/eval/boundary_long_input.py rename to demos/translation/models/chronos-enzh-50m-sft/eval/boundary_long_input.py diff --git a/models/chronos-enzh-50m-sft/eval/boundary_zh_en.py b/demos/translation/models/chronos-enzh-50m-sft/eval/boundary_zh_en.py similarity index 100% rename from models/chronos-enzh-50m-sft/eval/boundary_zh_en.py rename to demos/translation/models/chronos-enzh-50m-sft/eval/boundary_zh_en.py diff --git a/models/chronos-enzh-50m-sft/eval/evaluate.py b/demos/translation/models/chronos-enzh-50m-sft/eval/evaluate.py similarity index 100% rename from models/chronos-enzh-50m-sft/eval/evaluate.py rename to demos/translation/models/chronos-enzh-50m-sft/eval/evaluate.py diff --git a/models/chronos-enzh-50m-sft/eval/results/selftest.json b/demos/translation/models/chronos-enzh-50m-sft/eval/results/selftest.json similarity index 100% rename from models/chronos-enzh-50m-sft/eval/results/selftest.json rename to demos/translation/models/chronos-enzh-50m-sft/eval/results/selftest.json diff --git a/models/chronos-enzh-50m-sft/eval/results/summary.json b/demos/translation/models/chronos-enzh-50m-sft/eval/results/summary.json similarity index 100% rename from models/chronos-enzh-50m-sft/eval/results/summary.json rename to demos/translation/models/chronos-enzh-50m-sft/eval/results/summary.json diff --git a/models/chronos-enzh-50m-sft/eval/selftest.json b/demos/translation/models/chronos-enzh-50m-sft/eval/selftest.json similarity index 100% rename from models/chronos-enzh-50m-sft/eval/selftest.json rename to demos/translation/models/chronos-enzh-50m-sft/eval/selftest.json diff --git a/models/chronos-enzh-50m-sft/generation_config.json b/demos/translation/models/chronos-enzh-50m-sft/generation_config.json similarity index 100% rename from models/chronos-enzh-50m-sft/generation_config.json rename to demos/translation/models/chronos-enzh-50m-sft/generation_config.json diff --git a/models/chronos-enzh-50m-sft/prompts.json b/demos/translation/models/chronos-enzh-50m-sft/prompts.json similarity index 100% rename from models/chronos-enzh-50m-sft/prompts.json rename to demos/translation/models/chronos-enzh-50m-sft/prompts.json diff --git a/models/chronos-enzh-50m-sft/tokenizer/special_tokens_map.json b/demos/translation/models/chronos-enzh-50m-sft/tokenizer/special_tokens_map.json similarity index 100% rename from models/chronos-enzh-50m-sft/tokenizer/special_tokens_map.json rename to demos/translation/models/chronos-enzh-50m-sft/tokenizer/special_tokens_map.json diff --git a/models/chronos-enzh-50m-sft/tokenizer/tokenizer.json b/demos/translation/models/chronos-enzh-50m-sft/tokenizer/tokenizer.json similarity index 100% rename from models/chronos-enzh-50m-sft/tokenizer/tokenizer.json rename to demos/translation/models/chronos-enzh-50m-sft/tokenizer/tokenizer.json diff --git a/models/chronos-enzh-50m-sft/tokenizer/tokenizer_config.json b/demos/translation/models/chronos-enzh-50m-sft/tokenizer/tokenizer_config.json similarity index 100% rename from models/chronos-enzh-50m-sft/tokenizer/tokenizer_config.json rename to demos/translation/models/chronos-enzh-50m-sft/tokenizer/tokenizer_config.json diff --git a/models/chronos-enzh-50m-sft/tokenizer/tokenizer_report.json b/demos/translation/models/chronos-enzh-50m-sft/tokenizer/tokenizer_report.json similarity index 100% rename from models/chronos-enzh-50m-sft/tokenizer/tokenizer_report.json rename to demos/translation/models/chronos-enzh-50m-sft/tokenizer/tokenizer_report.json diff --git a/models/chronos-enzh-50m-sft/translate.py b/demos/translation/models/chronos-enzh-50m-sft/translate.py similarity index 100% rename from models/chronos-enzh-50m-sft/translate.py rename to demos/translation/models/chronos-enzh-50m-sft/translate.py diff --git a/tests/test_model_enzh_50m_sft.py b/demos/translation/tests/test_model_enzh_50m_sft.py similarity index 100% rename from tests/test_model_enzh_50m_sft.py rename to demos/translation/tests/test_model_enzh_50m_sft.py diff --git a/pyproject.toml b/pyproject.toml index fcd5e62..5ff8f44 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "Project_Chronos" -version = "0.3.3a0" +version = "0.3.3a1" description = "Storage-aware recurrent-depth MoE: lookahead expert prefetch, SSD-lazy experts, hybrid MLA/SWA attention, self-speculative decoding" readme = "README.md" license = "Apache-2.0" diff --git a/tests/test_autotune_loop.py b/tests/test_autotune_loop.py index ab210fc..6d70fda 100644 --- a/tests/test_autotune_loop.py +++ b/tests/test_autotune_loop.py @@ -128,7 +128,7 @@ def test_webui_exposes_the_loop_knobs_to_the_search_and_maps_them_onto_the_space from chronos.webui import schema choices = schema.FIELDS['tune.search'].choices assert 'recurrent_order' in choices and 'recurrent_dt_init' in choices - assert 'recurrent_later_groups' not in choices # trades quality for cost: a loss-only search would always pick the full block + assert 'recurrent_later_groups' in choices # 0.3.3a1: searched with choices 0 and 2; the trial log carries its step time space = ChronosSearchSpaceConfig(**{f'tune_{name}': name in ('recurrent_order',) for name in choices}) assert space.tune_recurrent_order and not space.tune_recurrent_dt_init base = os.path.join(os.path.dirname(schema.__file__), 'i18n') diff --git a/tests/test_loop_knobs_wiring.py b/tests/test_loop_knobs_wiring.py new file mode 100644 index 0000000..00596f9 --- /dev/null +++ b/tests/test_loop_knobs_wiring.py @@ -0,0 +1,153 @@ +"""0.3.3a1: every loop knob in the planner, the schema and the auto-tune probe; the probe scores cross-entropy.""" +import json +import os + +import numpy as np +import pytest + +optuna = pytest.importorskip('optuna') + +from chronos.model.planner import PlanRequest, plan_training # noqa: E402 +from chronos.tuning.chronos_auto_tuner import LOOP_KEYS, ChronosAutoTuner, ChronosSearchSpaceConfig # noqa: E402 +from chronos.webui import schema # noqa: E402 + +NEW_KNOBS = ('recurrent_dt_param', 'recurrent_dt_min', 'recurrent_dt_max', 'recurrent_later_topk', + 'recurrent_later_groups', 'lambda_iteration_cluster', 'lookahead_teacher_scope') +PICKS = dict(recurrent_order='block', recurrent_dt_param='sigmoid', recurrent_dt_min=0.15, recurrent_dt_max=0.9, + recurrent_dt_init=0.1, recurrent_later_topk=1, recurrent_later_groups=2, lambda_iteration_cluster=0.3, + lookahead_teacher_scope='iterations') +TINY = dict(learning_rate=1e-3, lambda_balance=5e-4, lambda_temporal=1e-3, lookahead_steps=2, hidden_size=64, + num_experts=4, num_shared_experts=1, kv_latent_dim=16) + + +class _Trial: + """Answers suggest_* with fixed picks (else the last choice / the lower bound) and records reports.""" + + def __init__(self, picks=None): + self.picks, self.reports, self.attrs = dict(picks or ()), [], dict() + + def suggest_categorical(self, name, choices): + return self.picks.get(name, choices[-1]) + + def suggest_float(self, name, lo, hi, log=False): + return self.picks.get(name, lo) + + def report(self, value, step): + self.reports.append(value) + + def should_prune(self): + return False + + def set_user_attr(self, key, value): + self.attrs[key] = value + + +def _space_with_every_loop_knob(): + return ChronosSearchSpaceConfig(tune_lr=False, tune_lambda_balance=False, tune_lambda_temporal=False, + **{'tune_' + key: True for key in LOOP_KEYS}) + + +def _spy_config(monkeypatch): + import chronos.model.model_chronos as model_chronos + seen = dict() + real_model = model_chronos.ChronosForCausalLM + + class Spy(real_model): + def __init__(self, cfg, *args, **kwargs): + seen['cfg'] = cfg + super().__init__(cfg, *args, **kwargs) + + monkeypatch.setattr(model_chronos, 'ChronosForCausalLM', Spy) + return seen + + +def _probe(tmp_path, params, steps, seed=0): + import torch + torch.manual_seed(seed) + trial = _Trial() + score = ChronosAutoTuner()._run_probe_trial(trial, str(tmp_path / 'no_checkpoint.pth'), str(tmp_path / 'none.jsonl'), + 0.95, steps, str(tmp_path), 0, params, tokenizer_dir=None) + return score, trial + + +def test_every_loop_knob_is_searchable_and_reaches_the_probe_model(tmp_path, monkeypatch): + seen = _spy_config(monkeypatch) + params = ChronosAutoTuner()._sample_params(_Trial(PICKS), _space_with_every_loop_knob()) + assert all(params[key] == value for key, value in PICKS.items()) + assert params['recurrent_depth'] == 4 and params['num_hidden_layers'] == 6 + score, trial = _probe(tmp_path, dict(params, **TINY), steps=2) + cfg = seen['cfg'] + for key in LOOP_KEYS: + if key != 'recurrent_dt_init': + assert getattr(cfg, key) == PICKS[key], key + assert cfg.recurrent_dt_init == pytest.approx(0.15) # a drawn init below the drawn dt_min starts at dt_min + assert cfg.recurrent_groups == 4 and cfg.prelude_layers == 1 and cfg.coda_layers == 1 + assert np.isfinite(score) and trial.attrs['step_time_s'] > 0 + + +def test_later_groups_never_offers_one_and_needs_block_order(): + space = _space_with_every_loop_knob() + assert 1 not in space.recurrent_later_groups_choices and 0 in space.recurrent_later_groups_choices + layer = ChronosAutoTuner()._sample_params(_Trial(dict(PICKS, recurrent_order='layer')), space) + assert layer['recurrent_order'] == 'layer' and 'recurrent_later_groups' not in layer + only = ChronosSearchSpaceConfig(tune_recurrent_later_groups=True) + drawn = ChronosAutoTuner()._sample_params(_Trial(), only) + assert drawn['recurrent_order'] == 'block' and drawn['recurrent_later_groups'] in (0, 2) + + +def test_probe_scores_cross_entropy_and_trains_on_the_weighted_objective(tmp_path): + low, trial_low = _probe(tmp_path, dict(TINY, lambda_lookahead=1e-3), steps=4) + high, trial_high = _probe(tmp_path, dict(TINY, lambda_lookahead=1.0), steps=4) + # Step 0 is scored before any update: equal scores mean the weighted terms are not in the score. + assert trial_low.reports[0] == pytest.approx(trial_high.reports[0], abs=1e-6) + # Later steps differ: lambda_lookahead now reaches training (0.3.3a0 ignored it). + assert any(abs(x - y) > 1e-6 for x, y in zip(trial_low.reports[1:], trial_high.reports[1:])) + + +def test_best_patch_carries_every_loop_knob(): + from types import SimpleNamespace + tuner = ChronosAutoTuner() + tuner.result = SimpleNamespace(best_params=dict(PICKS, lr=1e-3)) + patch = tuner.get_best_chronos_config_patch() + assert all(patch[key] == value for key, value in PICKS.items()) + + +def test_webui_search_choices_cover_every_loop_knob_with_help_in_four_languages(): + choices = set(schema.FIELDS['tune.search'].choices) + assert set(LOOP_KEYS) <= choices + base = os.path.join(os.path.dirname(schema.__file__), 'i18n') + for lang in ('en', 'zh-Hans', 'zh-Hant', 'ja'): + catalog = json.load(open(os.path.join(base, lang + '.json'), encoding='utf-8')) + for key in NEW_KNOBS: + assert catalog.get('field.' + key + '.label') and catalog.get('field.' + key + '.help'), (lang, key) + assert any(ch.isdigit() for ch in catalog['field.' + key + '.help']), (lang, key) # carries the measured reason + + +def _plan(depth): + return plan_training(PlanRequest(target_total=5e6, num_experts=4, active_experts=2, vocab_size=6400, seq_len=128, + recurrent_depth=depth, memory_saving='none', token_budget=2e6, ram_gb=16), verify=False) + + +@pytest.mark.parametrize('depth', [2, 4, 8]) +def test_planner_recommends_every_loop_knob_with_a_source(depth): + plan = _plan(depth) + config, sources = plan['config'], plan['report']['sources'] + for key in LOOP_KEYS: + assert key in config and sources.get(key), key + field = schema.FIELDS[key] + if key != 'recurrent_dt_init': # dt_init depends on depth; test_planner_loop.py covers it + assert field.source == 'planner' and field.recommended == config[key], key + assert config['recurrent_later_groups'] != 1 and 'never 1' in sources['recurrent_later_groups'] + + +def test_plan_without_a_loop_sets_no_loop_knob(): + config = _plan(1)['config'] + assert not any(key in config for key in LOOP_KEYS) + + +def test_schema_payload_keeps_zero_values_and_drops_unset_flags(): + topk = schema.FIELDS['recurrent_later_topk'].as_dict() + assert topk['min'] == 0 and topk['recommended'] == 0 and topk['default'] == 0 + assert schema.FIELDS['max_steps'].as_dict()['recommended'] == 0 + plain = schema.FIELDS['hidden_size'].as_dict() + assert 'advanced' not in plain and 'nullable' not in plain and 'recommended' not in plain diff --git a/tests/test_ui_generic.py b/tests/test_ui_generic.py index 5e3b744..93f9225 100644 --- a/tests/test_ui_generic.py +++ b/tests/test_ui_generic.py @@ -135,6 +135,23 @@ def test_kernel_has_no_translation_task_code(): assert not hits, "\n".join(hits) +def test_task_models_and_tests_live_under_demos(): + """A task's model folder and its tests sit inside its demo, and each main README points at the demo once.""" + import re + task = re.compile(r"enzh|en_zh|zh_en|translat", re.I) + misplaced = [] + for top in ("models", "tests"): + for folder, dirs, files in os.walk(os.path.join(_repo_root, top)): + dirs[:] = [d for d in dirs if d != "__pycache__"] + misplaced += [os.path.join(folder, n) for n in dirs + files if task.search(n)] + assert not misplaced, misplaced + mention = re.compile(r"en→zh|en↔zh|en-zh|英→中|英↔中|英译中|FLORES|BLEU") + for name in ("README.md", "README_zh.md"): + with open(os.path.join(_repo_root, name), encoding="utf-8") as fh: + lines = [line for line in fh if mention.search(line)] + assert len(lines) == 1 and "demos/translation" in lines[0], (name, [line[:100] for line in lines]) + + # ---- data inspector ------------------------------------------------------------- def test_record_schema_covers_common_dataset_shapes(): from ui.tabs.data_tab import record_schema diff --git a/ui/presets_v3.json b/ui/presets_v3.json index e0749a2..ebf929c 100644 --- a/ui/presets_v3.json +++ b/ui/presets_v3.json @@ -121,6 +121,10 @@ "recurrent_order": "block", "recurrent_dt_param": "sigmoid", "recurrent_dt_init": 0.5, + "recurrent_later_topk": 0, + "recurrent_later_groups": 0, + "lambda_iteration_cluster": 0.0, + "lookahead_teacher_scope": "first", "halting_threshold": 0.95, "depth_schedule_warmup_fraction": 0.3, "recurrent_latent_refinement": false, @@ -244,6 +248,10 @@ "recurrent_order": "block", "recurrent_dt_param": "sigmoid", "recurrent_dt_init": 0.5, + "recurrent_later_topk": 0, + "recurrent_later_groups": 0, + "lambda_iteration_cluster": 0.0, + "lookahead_teacher_scope": "first", "halting_threshold": 0.95, "depth_schedule_warmup_fraction": 0.3, "recurrent_latent_refinement": false, @@ -363,6 +371,10 @@ "recurrent_order": "block", "recurrent_dt_param": "sigmoid", "recurrent_dt_init": 0.5, + "recurrent_later_topk": 0, + "recurrent_later_groups": 0, + "lambda_iteration_cluster": 0.0, + "lookahead_teacher_scope": "first", "halting_threshold": 0.95, "depth_schedule_warmup_fraction": 0.3, "recurrent_latent_refinement": false, @@ -483,6 +495,10 @@ "recurrent_order": "block", "recurrent_dt_param": "sigmoid", "recurrent_dt_init": 0.5, + "recurrent_later_topk": 0, + "recurrent_later_groups": 0, + "lambda_iteration_cluster": 0.0, + "lookahead_teacher_scope": "first", "halting_threshold": 0.95, "depth_schedule_warmup_fraction": 0.3, "recurrent_latent_refinement": false, @@ -604,6 +620,10 @@ "recurrent_order": "block", "recurrent_dt_param": "sigmoid", "recurrent_dt_init": 0.5, + "recurrent_later_topk": 0, + "recurrent_later_groups": 0, + "lambda_iteration_cluster": 0.0, + "lookahead_teacher_scope": "first", "halting_threshold": 0.95, "depth_schedule_warmup_fraction": 0.3, "recurrent_latent_refinement": false,