Add large resumable MuonClip nanoGPT experiment

baseline/experiments/nanogpt_muonclip_large_2026_08_30/README.md

@@ -0,0 +1,145 @@
+# Large MuonClip nanoGPT long run — 2026-08-30
+
+This is a single-seed, MuonClip-only scaling run intended for Charles's
+16-GiB Apple M2 Pro. It does **not** run AdamW or any other comparison arm.
+
+## Scale point
+
+| Quantity | Value |
+|---|---:|
+| Transformer blocks | 6 |
+| Attention heads per block | 8 |
+| Embedding width | 384 |
+| Context length | 512 |
+| Trainable parameters | 30,117,120 |
+| Unique training tokens | 512,000,000 |
+| Validation / test tokens | 4,000,000 / 4,000,000 |
+| Processed training tokens | 512,000,000 |
+| Tokens per parameter | 17.0 |
+| Effective optimizer batch | 8,192 tokens |
+| Optimizer updates | 62,500 |
+| Warmup updates | 1,000 |
+| Permanent WeightWatcher states | 26 |
+| WeightWatcher matrices per state | 36 |
+
+This is near a Chinchilla-style token budget in ratio, but it is not presented
+as a scaling-law measurement: there is only one model size, one optimizer, and
+one seed. The purpose is to obtain a serious long trajectory without repeatedly
+cycling over the same small corpus.
+
+The peak MuonClip learning rate remains the empirically exercised `2e-4`.
+Warmup is lengthened to 1,000 updates and cosine decay spans the full fresh-data
+pass, ending at `1e-5`. Weight decay is `0.1`, the RMS update scale is `0.2`,
+gradient clipping is `1.0`, and QK-Clip retains the tested threshold of `100`.
+
+Expected M2 Pro wall time is roughly **4–6 days**, based on the measured
+four-head run and the increase in block count, width, and context. The first
+few evaluation rows provide a machine-specific ETA; `status` reports it.
+
+## Exact protocol
+
+The frozen YAML is [`configs/muonclip_long_mps.yaml`](configs/muonclip_long_mps.yaml).
+FineWeb-Edu is streamed from pinned revision
+`593b3a867298afb8ce42625a270ef20ddcad28f9`. Train, validation, and test are
+document-disjoint. The test split remains untouched until the final and
+validation-selected checkpoint audit.
+
+## Mac setup and preflight
+
+Use the Conda Python that already passed the earlier campaign's dependency
+check:
+
+```bash
+cd /tmp/rg_optimizers
+
+git switch main
+git pull --ff-only origin main
+
+CONDA_PY="/Users/charleshmartin/opt/anaconda3/envs/ww_prod310/bin/python"
+
+"$CONDA_PY" -m pip install -e baseline/nanogpt_one_head
+
+cd baseline/experiments/nanogpt_muonclip_large_2026_08_30
+
+export RG_NANOGPT_LARGE_EXPERIMENT_ROOT="/Users/charleshmartin/rg_runs/nanogpt_muonclip_large_20260830"
+export PYTORCH_ENABLE_MPS_FALLBACK=1
+export PYTHONUNBUFFERED=1
+
+"$CONDA_PY" scripts/run_experiment.py doctor --device auto --smoke-step
+```
+
+The smoke step instantiates the full 30.1M-parameter model on MPS and performs
+one real MuonClip forward/backward/update before any 512M-token download.
+
+## Start the detached long run
+
+The following creates one detached `tmux` session. It prepares the larger
+dataset and then starts or resumes the single MuonClip seed. No AdamW command is
+present.
+
+```bash
+tmux new-session -d -s muonclip-large \
+  "/usr/bin/caffeinate -dimsu '$CONDA_PY' scripts/run_experiment.py prepare && /usr/bin/caffeinate -dimsu '$CONDA_PY' scripts/run_experiment.py run --device auto --mps-retries 20"
+```
+
+Detach/closing the Cloud or Terminal window does not stop a process inside
+`tmux`. To watch the live terminal and detach again, use:
+
+```bash
+tmux attach -t muonclip-large
+```
+
+Press `Control-b`, release both keys, then press `d`.
+
+## Check progress at any time
+
+Run these from the experiment directory with the same three exported variables
+shown above:
+
+```bash
+"$CONDA_PY" scripts/run_experiment.py status
+```
+
+For the last terminal output:
+
+```bash
+tmux capture-pane -p -t muonclip-large -S -40
+```
+
+For the durable combined training log:
+
+```bash
+tail -n 40 "$RG_NANOGPT_LARGE_EXPERIMENT_ROOT/logs/train.log"
+```
+
+## Generate a live report without stopping training
+
+```bash
+"$CONDA_PY" scripts/run_experiment.py report --open
+```
+
+The report is regenerated atomically at:
+
+```text
+$RG_NANOGPT_LARGE_EXPERIMENT_ROOT/live_report/report.html
+```
+
+It includes train/validation loss, perplexity, optimizer diagnostics,
+throughput, MPS memory, QK-Clip activity, and per-block curves for all six
+matrix types for raw alpha, clip_xmax alpha, ERG gap, random distance, and trap
+count. It uses only already-completed CSV rows and does not touch the model or
+checkpoint.
+
+## Resume after interruption or reboot
+
+Re-export the variables, return to this directory, and run the same command:
+
+```bash
+/usr/bin/caffeinate -dimsu "$CONDA_PY" scripts/run_experiment.py run \
+  --device auto \
+  --mps-retries 20
+```
+
+The verified `checkpoint_latest.pt` includes the model, both optimizer states,
+RNG state, training generator, and resume diagnostics. The launcher resumes by
+default. Do not pass `--overwrite` unless intentionally discarding the run.

baseline/experiments/nanogpt_muonclip_large_2026_08_30/configs/muonclip_long_mps.yaml

@@ -0,0 +1,141 @@
+protocol:
+  name: nanogpt_muonclip_large_2026_08_30_long_mps
+  version: 1
+  description: >
+    Single-seed, MuonClip-only long run on Apple MPS. The model has six
+    transformer blocks, eight attention heads, width 384, and context length
+    512 (30,117,120 trainable parameters). It sees one pass over 512 million
+    pinned FineWeb-Edu tokens: 62,500 optimizer updates and approximately
+    17.0 training tokens per parameter. Rolling finite full-state checkpoints
+    make the run resumable, while 26 permanent states retain per-block
+    WeightWatcher raw and clip_xmax diagnostics.
+
+dataset:
+  name: HuggingFaceFW/fineweb-edu
+  config: sample-10BT
+  split: train
+  revision: 593b3a867298afb8ce42625a270ef20ddcad28f9
+  tokenizer: gpt2
+  train_tokens: 512000000
+  val_tokens: 4000000
+  test_tokens: 4000000
+
+model:
+  vocab_size: 50257
+  block_size: 512
+  n_layer: 6
+  n_head: 8
+  n_embd: 384
+  dropout: 0.0
+  bias: false
+  tie_weights: true
+
+training:
+  seeds: [20260830]
+  batch_size: 1
+  grad_accum_steps: 16
+  target_epochs: 1.0
+  epoch_interval: 0.04
+  eval_interval_steps: 500
+  eval_batches: 64
+  checkpoint_interval_steps: 250
+  grad_clip: 1.0
+
+optimizer_profiles:
+  # Compatibility profiles required by the shared trainer. The launcher in
+  # this experiment hard-codes muon_clip and never executes these three arms.
+  sgd_momentum:
+    display_name: SGD + Nesterov momentum (inactive compatibility profile)
+    family: sgd
+    learning_rate: 0.05
+    min_learning_rate: 0.005
+    warmup_fraction: 0.10
+    lr_schedule_epochs: 1.0
+    schedule: warmup_cosine
+    momentum: 0.90
+    dampening: 0.0
+    nesterov: true
+    weight_decay: 0.01
+
+  adamw:
+    display_name: AdamW (inactive compatibility profile)
+    family: adamw
+    learning_rate: 0.0006
+    min_learning_rate: 0.00006
+    warmup_fraction: 0.01
+    lr_schedule_epochs: 1.0
+    schedule: warmup_cosine
+    beta1: 0.90
+    beta2: 0.95
+    epsilon: 1.0e-8
+    weight_decay: 0.10
+
+  muon:
+    display_name: Muon + auxiliary AdamW (inactive compatibility profile)
+    family: muon
+    matrix_learning_rate: 0.02
+    matrix_min_learning_rate: 0.002
+    aux_learning_rate: 0.0003
+    aux_min_learning_rate: 0.00003
+    warmup_fraction: 0.05
+    lr_schedule_epochs: 1.0
+    schedule: warmup_cosine
+    momentum: 0.95
+    nesterov: true
+    newton_schulz_steps: 5
+    muon_epsilon: 1.0e-7
+    matrix_weight_decay: 0.01
+    beta1: 0.90
+    beta2: 0.95
+    epsilon: 1.0e-8
+    aux_weight_decay: 0.01
+
+  muon_clip:
+    display_name: MuonClip + RMS-matched updates + auxiliary AdamW
+    family: muon_clip
+    learning_rate: 0.0002
+    min_learning_rate: 0.00001
+    warmup_fraction: 0.016
+    lr_schedule_epochs: 1.0
+    schedule: warmup_cosine
+    momentum: 0.95
+    nesterov: false
+    newton_schulz_steps: 5
+    muon_epsilon: 1.0e-7
+    weight_decay: 0.10
+    update_rms_scale: 0.20
+    qk_clip_threshold: 100.0
+    qk_clip_balance: 0.50
+    qk_diagnostics_interval: 250
+    beta1: 0.90
+    beta2: 0.95
+    epsilon: 1.0e-8
+
+evaluation:
+  train_probe_seed: 31001
+  validation_probe_seed: 32001
+  test_probe_seed: 33001
+  bleu_probe_seed: 34001
+  bleu_examples: 64
+  bleu_prompt_tokens: 128
+  bleu_continuation_tokens: 64
+  bleu_batch_size: 1
+
+weightwatcher:
+  enabled: true
+  ERG: true
+  randomize: true
+  strict: true
+  min_evals: 20
+  fix_fingers: clip_xmax
+  max_fingers: 10
+  require_raw_alpha: true
+
+runtime:
+  matmul_precision: high
+  allow_tf32: false
+  cudnn_benchmark: false
+  mps_fallback: true
+  deterministic_algorithms: false
+  deterministic_warn_only: true
+  empty_mps_cache_after_weightwatcher: true