Arabic + Hebrew SOTA stack, SadeedDiac-25 benchmark win, papers - #55
Open
ronaldtse wants to merge 28 commits into
Open
Arabic + Hebrew SOTA stack, SadeedDiac-25 benchmark win, papers#55ronaldtse wants to merge 28 commits into
ronaldtse wants to merge 28 commits into
Conversation
- pyproject.toml (PEP 621): name, version, license, requires-python - tool.ruff: conservative starter (E/F/W/I/UP); ignore E501/E402/E741 - tool.pytest config ready for future tests - 165 import-sort / unused-import fixes auto-applied - 116 remaining violations are research-code semantics (F811 dup defs in trainer.py, etc.) — manual review needed, not in this PR Refs: TODO.complete/08-ruff-rababa-python.md
- pyproject.toml (PEP 621): name, version, license, requires-python
- tool.ruff: conservative starter (E/F/W/I/UP); ignore E501/E402/E741
- tool.pytest config ready for future tests
- 245 fixes auto-applied (165 safe + 80 unsafe): PEP 585 annotations,
deprecated imports, unused vars, yield-in-for, isort
- ruff format applied to all 56 .py files
- CI: new lint job (non-blocking) runs ruff check + format check
- .gitignore: exclude python/{log_dir,data,models}/ training artifacts
36 violations remain (F821 false positives on tuple-unpack assigns,
F811 dup defs in trainer.py, E722 bare-except, E721 type-compare) —
manual review needed.
Refs: TODO.complete/08-ruff-rababa-python.md
ModernCharTransformer: RoPE + SDPA Flash + mHC residual + AttnRes + RMSNorm + SwiGLU. 113M params at 12L/768d. Selected via cfg.model.arch=modern. Multi-task seg head optional.
Optimizer: MuonAdamWHybrid (Muon Newton-Schulz for 2D weights, AdamW for 1D) + qk_clip_ weight-rescaling callback with anneal. Prevents attention-logit explosion during from-scratch pretrain.
Decoding: trie-constrained beam decoder with per-word exact search. scripts/build_lexicon.py builds the word-to-haraqat-sequences JSON.
Data: combined corpus (GPLv2 Tashkeela-full + Sadeed HF + QCRI EMNLP 2025) built by fetch_data on first call. Graceful fallback when HF_TOKEN unset.
Idempotency: training/resume.py auto-detects latest epoch checkpoint. train_all.py skips done stages via _status.json on checkpoints volume. scripts/status.py queries Modal volumes.
Configs: rababa_arabic_pro{,_pretrain}.yaml → arch=modern, max_len=512, optimizer=muon, with_seg_head=true, root=/datasets/arabic-combined.
References: arXiv:2606.19348 (DS V4), 2607.24653 (Kimi K3), 2512.24880 (mHC), 2507.20534 (MuonClip).
…wnload Modal secret 'huggingface' was registered with HF_TOKEN. fetch_data now reads it via env var to authenticate the Sadeed_Tashkeela download.
Pretrain failed because MuonAdamWHybrid is not a torch.optim.Optimizer. WarmupCosine is now a duck-typed scheduler; GradScaler is skipped for Muon (bf16 autocast is enough). Add run_sota_pipeline + sota_pipeline entrypoint: fetch -> pretrain -> train -> export ONNX/TFLite entirely on Modal via .remote() chaining. Idempotent stage skips + volume status. Survives --detach disconnect. Fix Tashkeela-full layout discovery (tashkeela_full_train/ subdirs).
Usage: python scripts/status.py [--watch|--json] [--task rababa_arabic_pro]. Reports Modal app state, stage status JSON, pipeline log, per-epoch checkpoints, and exported artifacts.
…len 512
ModernMultiHeadCharTransformer mirrors ModernCharTransformer's encoder
(RoPE + SDPA + mHC + AttnRes + RMSNorm + SwiGLU) but with a ModuleList
of per-category linear heads for niqqud/dagesh/sin.
Encoder weights are key-compatible with ModernCharTransformer — a single
pretrain checkpoint can fine-tune into either Arabic single-head or
Hebrew multi-head.
rababa_hebrew{,_pretrain}.yaml now use:
arch=modern_multi_head, max_len=512, optimizer=muon (MuonAdamWHybrid)
Smoke-tested: forward + backward + Muon step OK, 2.4M params at 384d/6L.
…e won't trip on stale format
- 12L/768d/113M was too big for single-A100 pretrain on 1.7M lines (0 checkpoints in 35min before death). 6L/512d/40M keeps the modern stack but is tractable. - sota_pipeline --force now only wipes arabic-combined for Arabic tasks. Hebrew force-wipe was destroying Arabic corpus when both pipelines ran in parallel.
…on't interfere Both Arabic and Hebrew pipelines write to the same /checkpoints/_status.json. Without task-keying, Hebrew's 'pretrain done' marker caused Arabic's pretrain to skip in non-force mode. Keys are now 'rababa_arabic_pro:pretrain' etc.
ByT5/seq2seq training paths alongside the char-level encoder: Muon optimizer variants (AdaMuon, NorMuon, HTMuon, Spectral Cap), ResFormer, MoE, ELECTRA pretraining, curriculum sampler, EMA, SAM, multi-seed and distillation harnesses, plus eval scripts for DictaBERT, Nakdimon and dNIKUD baselines. Arabic 0.99% DER, Hebrew 17.46% DER (beam 4).
RESULTS.md as ground truth for Arabic (0.99% DER) and Hebrew (17.46% DER, DictaBERT 35.63% on same test), three LaTeX papers (arabic, hebrew, umbrella), and the TODO.publish checklist.
ByT5-base run-002 (full 1.42M-line corpus, 2 epochs) to beat Claude's 1.39 DER on SadeedDiac-25. Checkpoints commit to the Modal volume at every save so preemption no longer discards hours of training; EVAL_DONE marker makes relaunch-after-completion a no-op.
Rejection-sampling fine-tuning on ByT5 r2 (TODO.research/12): sample K=4 per prompt, keep letter-exact-DER winners over greedy, SFT on winners. Selection on the frozen private dev split (1,372 lines, sha256-pinned, byte-identical to r2's held-out val); SadeedDiac-25 measured once at the end. Self-fires when r2's EVAL_DONE marker appears; per-iter volume commits + markers make preemptions resume cleanly.
Steers RAFT iterations: quantifies how much residual DER sits in the word-final iʿrāb zone and which haraqat get confused, from the eval CSV.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan