Skip to content

Phase 9: modeling upgrades - #69

Merged
LikelyCause merged 36 commits into
mainfrom
phase-9
Nov 16, 2026
Merged

LikelyCause merged 36 commits into
mainfrom
phase-9

Conversation

@LikelyCause

Copy link
Copy Markdown
Owner

Phase 9 of the v2 roadmap, from the owner's review: modeling upgrades.

What it adds

  • Native categoricals for tree models. This is the default for every categorical in tree baselines; linear and neural models keep their encodings. They run end to end through export, training, scoring, the serving bundle, the registry equality check and the code script. Random forest refuses them with a clear message, and "Encode these fields" switches them to ordinal for one experiment.

  • One rule for missing values. Text such as "None", "NA" or "?" counts as missing only when the column never held it as a real value. This fixes a train/serve skew from before this phase, which made Diabetes readmission models impossible to register.

  • GPU/CPU switch on Train, batches and tuning. GPU runs reserve a GPU slot. A queued GPU job says it is waiting, and "Run on the CPU instead" is available to its starter, project owners and admins. Run facts show the hardware a run used.

  • Optuna tuning:

    • an editable search space, with "also try none" per setting;
    • pruning for LightGBM and XGBoost;
    • limits of 200 trials or 4 h on the GPU, and 24 h on the CPU;
    • one Stop path, which refits the best trial so far;
    • journal storage that resumes after a crash or a stale lock;
    • trials shown as rows, with a best-so-far chart and setting importance.
  • Metrics:

    • plain, micro, macro and weighted accuracy, precision, recall and F1;
    • false alarm rate;
    • a confusion matrix with Types I–III against a chosen reference class, plus Type IV ("fragile correct");
    • per-class counts.

    The UI and the backend share one fixture of confusion cases.

  • Field description and notes. These are edited on the field page by reviewers and above, and shown in the sidebar, tables and history. Baseline exchange format rainforest-baseline/2 carries them; import fills only empty ones, and the dry run previews the fill.

  • A new baseline version from a run's drops. It comes with a dry run and pending derivations. Whoever applied a drops-only derivation can finalize it, rechecked against their current role; anything else is owner-only. From there you can re-run the experiment as it was, customise it first, or re-run its study. A re-run draft rebases automatically when its version finalizes.

  • Migration 043, with RLS on every new table, and a route-matrix row for every new route.

How it was built

The spec and plan were reviewed by Fable. There are 20 tasks, each reviewed separately: security-heavy tasks by Opus, the rest by Sonnet. Fix rounds were re-reviewed. Two privilege bugs were found and fixed with tests: a self-finalize from an unlocked read, and a demoted creator still finalizing. So was a regression that silently reverted one of those fixes.

Tests

  • API: 2,731 passed on the Mac in Docker, run twice (xdist, then serial on a fresh database).
  • UI: vitest 683 passed, twice in a row. Playwright fixture and gallery: 58 passed on dev-01. No baseline snapshots changed, because the new stories are live stories.
  • Deploy smoke still to run on rainforest-dev: the smoke script with a 3-trial study, an XGBoost CUDA run on the 4080 (which also confirms the driver range), and a GPU study within its limits.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS

LikelyCause and others added 30 commits November 15, 2026 20:47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…instead

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…es and tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…son, zeros note in the read-only view, partial-save report

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
… code script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…rid from the legend's matrix

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…n clamped, gating and key tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…registers, review minors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…re record, run on the CPU instead

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
… worker rows only, CPU hardware test, 404 on a vanished job, batch note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…sortable trials

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
… per run, structured 409, expandable drop list, scoped invalidation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…se alarm rate, error types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…easons, V1 copy period, formula-guard test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
optuna==4.9.0 (journal storage, JournalFileOpenLock and fANOVA imports verified in
the testrunner). Setting.tune holds each model's default range; tuning/space.py
builds the default space (respecting `when`; a booster chain tunes its trees),
checks a user-sent space, fixes LightGBM's bagging and early-stopping trees, and
maps a space (mixed choices, Range.none) to Optuna suggestions and back.
POST /exports/{id}/batches kind "tune" checks the base run, metric, device
ceilings (TUNE_MAX_TRIALS, TUNE_MAX_TRIALS_GPU, TUNE_MAX_HOURS_GPU) and space,
then inserts the batch and its one run.tune job; batch-plan gains tune limits;
the feeder ignores tune batches. Pruning is off for CV, chains and models other
than LightGBM/XGBoost. run.tune is registered with its describe; its handler is
B8's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…PI names them by an unordered method set)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
… one stop path, refit as a run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…stranded study blocks nothing, stop checks in every trial, time budget counts interrupted trials

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…; drops-only self-finalize

bulk_apply extracted from the bulk decisions route; POST /runs/{id}/derive-baseline (immediate, pending, dry run);
derive.after_finalize hooked into finalize; a derived version's creator finalizes it while it is drops-only
(audited as baseline.self_finalize); team state gains `derived`; lineage gains baseline nodes and the run edges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…finalize_now re-checks the creator's role; review minors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
LikelyCause and others added 6 commits November 15, 2026 22:51
…e of re-run drafts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…r leaves a run or study stuck; study only from its refit; more tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Every parked B1-B10 finding and the suite triage: stale tests fixed (043 replay,
RF encodes its categoricals, golden pins phase-0 categorical decisions), killed
trials count their time (tuning_trials.last_seen), a cancel past the deadline
is a stop, chain steps count members, logistic default space by target kind,
target null rule for number labels, NUL refused in field texts, adjust_denied
on experiments, one kind of pending.id, _start_study holds nothing while it
starts the study, explicit /api/auth/check operation ids, openapi.json
regenerated, docs for every new route and field, a tuning study in smoke.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…r, server-side Run again, study failures

- I1: derive-baseline and rerun 404 a run whose experiment the caller can't see.
- I2: TuneDialog always sends order unless one-vs-rest.
- I3: POST /batches/{id}/again rebuilds the study from its stored spec (derive.study_request);
  TuneSpec/StudyDetail types; fixture uses the real spec shape.
- I4: failed studies show their error; then_run.error shown on the export row.
- Minors 1-7, 10, 11 and the no-op half of 8.
- CI: API tests timeout 20 -> 45 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
@LikelyCause
LikelyCause merged commit 17f7e94 into main Nov 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant