[Vedaant-J]

Series: kumo-engine MLOps series: every snapshot, training table, model and prediction run carries a self-contained _provenance.json; with kumo.mlops(...) stores each artifact under one artifact root, tracks fits and prediction runs in MLflow or Snowflake and registers models; volumes and stages can hold files; prediction runs are monitored for drift and realized performance.
Stack: #53 → #112 → #50 → #60 → #64; #114 on #112; #115 on #64
Merge order: one linear stack; merge bottom to top (#53 first); #114 sits on #112 and merges any time after it; #115 (draft) sits on #64 and merges after it
This PR: Makes TrainingResult.holdout_path the published model's holdout, or None, and makes the answer independent of whether explanations are on.

Summary

This PR makes TrainingResult.holdout_path the published model's holdout, or None, and makes the answer independent of whether explanations are on.

What the holdout is. During fit(), the trained model scores the test split, and the predictions are written to test_df.parquet in the trial's run directory. TrainingResult.holdout_path points at that file and TrainingResult.holdout_df() reads it. Teams use it to check calibration and score distributions of the model they are about to ship.

Which models exist. One fit can produce up to three models from the best trial:

Fit mode	Trained on	Evaluated on the test split
fit	train	yes
refit_trainval (training_job.refit_trainval)	train + val	yes
refit_full (training_job.refit_full)	train + val + test	no: it trained on the test split, so it has no holdout by construction

The bundle ships the model trained on the most data that was enabled: refit_full, else refit_trainval, else fit (kumo-core's _set_fit_mode_to_use, called when the bundle's checkpoint is chosen).

Problem. Whether fit() returned a holdout, and whose it was, depended on whether explanations were on:

• refit_full with explanations off: holdout_path was None.
• refit_full with explanations on: holdout_path pointed at the fit or refit_trainval model's predictions, with nothing saying it was not the shipped model.

Cause. fit() looked for test_df.parquet in best_trainer.log_dir, which follows whatever fit_mode the trainer object was last left in. Publishing switches it to refit_full, whose directory has no holdout. explain() switches the mode back as a side effect, so with explanations on the lookup landed in an earlier model's directory.

Run directory from a real CPU fit on a small synthetic graph (users, items, orders) with refit_trainval and refit_full both on; the best trial is version_1:

<scratch>/trainers/<trainer>/runs/<run>/
  published-model/                     # the bundle; holds the refit_full model
  models/<run>/automl/
    version_0/  version_2/  version_3/  # other trials (validation only)
    version_1/                         # best trial, fit mode 'fit'
      test_df.parquet  test_df_meta.json  test_result.csv  checkpoint/  ...
    version_1_refit_trainval/          # refit on train+val, evaluated on test
      test_df.parquet  test_df_meta.json  test_result.csv  checkpoint/  ...
    version_1_refit_full/              # refit on train+val+test, never evaluated
      checkpoint/  tensorboard/  train_progress.json  train_result.json

Change. The lookup asks kumo-core's Trainer.published_fit_mode() (the same refit-flag order the bundle is published in) and reads only that model's directory, instead of following the trainer's current fit_mode:

• refit_full published: holdout_path = None. No holdout of the shipped model exists, and another model's would be misleading.
• refit_trainval published: the version_N_refit_trainval/ holdout.
• fit published: the version_N/ holdout.

kumo-core gains two small additive pieces: Trainer.published_fit_mode() (the refit-flag order _set_fit_mode_to_use already applied, moved into a method so the holdout lookup and prediction share it) and an optional fit_mode argument on Trainer.get_log_dir (default: the trainer's own), so the trainer object is never mutated and no empty refit directories are created. The run directory layout and TrainingResult fields are unchanged.

From two real fits on the graph above (explanations off; the first picked trial 3 as best):

refit_trainval:              holdout_path   = <run>/models/<run>/automl/version_3_refit_trainval/test_df.parquet
                             # 50 rows, not truncated
refit_trainval + refit_full: holdout_path   = None

Every combination:

refit_trainval	refit_full	explain	before	after
no	no	off/on	version_N	version_N (fit)
yes	no	off/on	version_N_refit_trainval	unchanged (refit_trainval)
no	yes	off	none	none
no	yes	on	version_N	none
yes	yes	off	none	none
yes	yes	on	version_N_refit_trainval	none

Back-compat

This PR changes only which existing file is returned. It adds no fields, checks or errors.

Situation	Behaviour
No refit, or refit_trainval only	Same holdout_path as before, with explanations on or off.
refit_full + explanations off	holdout_path=None, as before.
refit_full + explanations on	Used to return an unlabelled holdout from an earlier model (fit or refit_trainval). Now returns None, because no holdout of the shipped model exists and substituting another model's would be misleading. holdout_df() raises the same FileNotFoundError it already raises whenever there is no holdout.
No holdout written (e.g. ranking tasks)	holdout_path=None, as before.
Trainer.load() result	holdout_path=None, as before.

Known limitations

• A refit_full model has no holdout. To inspect the fit or refit_trainval model's test predictions, read them from the run directory above; they are not returned as the shipped model's.

Testing Done

• test_holdout_df.py: real CPU fits for no refit (path under version_N/), refit_trainval with explanations on (path under version_N_refit_trainval/), and refit_trainval + refit_full (None); the default fit asserts the holdout is not inside the bundle. Verified with the kumo-core artifact, training and checkpoint suites when get_log_dir changed.
• test_training_explanations.py, test_semantic_carry_forward.py: fake trainers expose published_fit_mode / get_log_dir.
• On the current head (based on main): test_holdout_df.py + test_training_explanations.py + test_semantic_carry_forward.py: 35 passed (CPU).

Checklist

• No breaking changes.
• Unit tests added.

🤖 Generated with Claude Code

[Vedaant-J]

Change map

File	Function / scope	What changed	Why
packages/kumo-core/src/kumo_core/train/trainer.py	Trainer.get_log_dir, Trainer.published_fit_mode, Trainer._set_fit_mode_to_use	New optional fit_mode argument on get_log_dir that overrides self.fit_mode; the refit-flag order _set_fit_mode_to_use applied moves into published_fit_mode()	Resolve the published model's directory without mutating the trainer or creating empty refit directories; one definition of the published model for prediction and the holdout
packages/kumo-core/tests/test_artifacts.py	tests	fake SourceTrainer gains published_fit_mode	_set_fit_mode_to_use now calls it, so the fake must expose it
packages/kumo-engine/kumo/engine/trainer/trainer.py	TrainingResult, _published_holdout, Trainer.fit	Holdout read only from the directory of published_fit_mode() (refit_trainval or fit), None when refit_full is published; docstring says so	The holdout is the shipped model's or none, never another model's, and no longer depends on explain side effects
packages/kumo-engine/test/test_holdout_df.py	_fit, test_holdout_df_after_fit, test_holdout_is_the_published_models	_fit takes refit/explain flags; real fits for no refit, refit_trainval + explain, refit_trainval + refit_full	Cover each published mode through real fits
packages/kumo-engine/test/test_semantic_carry_forward.py	_best_trainer	The fake best trainer answers published_fit_mode() and get_log_dir(fit_mode)	Holdout lookup now asks the trainer for the published fit mode's log dir
packages/kumo-engine/test/test_training_explanations.py	test_fit_runs_shared_explanations_after_training	The fake best trainer answers published_fit_mode() and get_log_dir(fit_mode) instead of log_dir	Holdout lookup now asks the trainer for the published fit mode's log dir

Rule: return the published model's holdout or None; never substitute another model's. Stopping condition: each refit/explain combination returns the shipped model's holdout, or None for refit_full.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

• Last reviewed commit: ea7119d
• Latest commit, not reviewed: 86fa3c6

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review.

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 02f9dcd3-edf8-40f6-b115-ff21b676c041

📥 Commits

Reviewing files that changed from the base of the PR and between 0c37212 and ea7119d.

📒 Files selected for processing: 5

• packages/kumo-core/src/kumo_core/train/trainer.py
• packages/kumo-engine/kumo/engine/trainer/trainer.py
• packages/kumo-engine/test/test_holdout_df.py
• packages/kumo-engine/test/test_semantic_carry_forward.py
• packages/kumo-engine/test/test_training_explanations.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

Walkthrough

Trainer now resolves the highest completed fit mode through published_fit_mode. get_log_dir can use an explicit fit mode. Trainer.fit uses the published mode to locate holdout artifacts and returns no holdout path for full-data refits. Tests cover standard fitting, train-validation refitting, full-data refitting, explanation settings, path naming, and updated trainer interfaces.

Sequence Diagram(s)

sequenceDiagram
  participant Trainer.fit
  participant published_fit_mode
  participant _published_holdout
  participant Holdout artifacts
  Trainer.fit->>published_fit_mode: determine the completed fit mode
  published_fit_mode-->>Trainer.fit: return the published mode
  Trainer.fit->>_published_holdout: request holdout metadata
  _published_holdout->>Holdout artifacts: locate artifacts for the published mode
  Holdout artifacts-->>_published_holdout: return artifact metadata or no artifact
  _published_holdout-->>Trainer.fit: return holdout metadata

Loading

Priority: Low

Merge Risk: Minimal · previous review ea711

Holdout selection matches the published model in the inspected paths. No issue identified here prevents merging after normal checks.

Pre-merge checks: 2 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Change Map Design Review		The required PR conversation comment beginning with <!-- kumo-pr-change-map --> is not available. Repository search found only the review documentation reference, and the supplied PR description is …	Provide the complete PR conversation comment beginning with <!-- kumo-pr-change-map -->, including its linked problem, inspected behavior, proposed changes, verification, and stopping condition.

Passed checks: 2 passed

Check name	Status	Explanation
Pr Size		The pull request changes 5 files. This count does not exceed 40.
Public Resource Access		The pull-request diff adds or modifies no external resource reference. The changed lines contain code, documentation, imports, test paths, and local artifact names only; they add no URL, repository lo…

Full details: Public Resource Access

Explanation

The pull-request diff adds or modifies no external resource reference. The changed lines contain code, documentation, imports, test paths, and local artifact names only; they add no URL, repository location, or package source location that requires public-access verification.

Full details: Change Map Design Review

Explanation

The required PR conversation comment beginning with <!-- kumo-pr-change-map --> is not available. Repository search found only the review documentation reference, and the supplied PR description is explicitly truncated. The change-map design cannot be assessed for necessity, abstraction layer, stopping condition, and focused verification without that comment.

Resolution

Provide the complete PR conversation comment beginning with <!-- kumo-pr-change-map -->, including its linked problem, inspected behavior, proposed changes, verification, and stopping condition.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments: 1

packages/kumo-engine/test/test_holdout_df.py: 1

112-125: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover explanation processing with refit_trainval.

The production path runs best_trainer.explain(...) before _published_holdout(...). The matrix does not combine explain=True with refit_trainval=True, so a regression in mode restoration or holdout selection for this path can pass. Add both reachable refit_trainval explanation cases at this matrix site.

Suggested fix

 @pytest.mark.parametrize('refit_trainval,refit_full,explain,model', [
     (False, False, False, 'fit'),
     (True, True, False, 'refit_trainval'),
+    (True, False, True, 'refit_trainval'),
+    (True, True, True, 'refit_trainval'),
     (False, True, True, 'fit'),
 ])

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: b9ce2f06-0cec-4ff9-9845-51a2fb62bc8d

📥 Commits

Reviewing files that changed from the base of the PR and between 3f8cf56 and ab2e389.

📒 Files selected for processing: 1

• packages/kumo-engine/kumo/engine/trainer/trainer.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.