[Vedaant-J]

Series: kumo-engine MLOps series: every snapshot, training table, model and prediction run carries a self-contained _provenance.json; with kumo.mlops(...) stores each artifact under one artifact root, tracks fits and prediction runs in MLflow or Snowflake and registers models; volumes and stages can hold files; prediction runs are monitored for drift and realized performance.
Stack: Base (lineage, MLOps switch): #50 → #55 → #56
Merge order: one linear stack; merge bottom to top (#53 first)
This PR: Adds a stable bundle_id and a self-contained _provenance.json manifest to every model bundle.

Summary

This PR adds a stable bundle_id and a self-contained _provenance.json manifest to every model bundle. The id says which fitted model a bundle is. The manifest records what the model was trained from and how: the training table's manifest from #55 (which embeds the snapshot's from #50, with its source table versions), the warm-start model, which fit mode was shipped, timing and engine build. With MLOps on (#50), each bundle is also stored under the artifact root.

Problem. Teams running Kumo in production need to answer questions like these:

• Which model produced these scores?
• Two registry versions exist. Do they hold the same weights?
• Which training table, snapshot, source table versions or warm-start model did it come from, and which engine build trained it? Can we still tell once the training machine's scratch directory is gone?
• Which of the test metrics in metrics.json, if any, belong to the model that was shipped?

Today none of these can be answered from the bundle.

Cause. A bundle has no id. verify_model_bundle already computed a 16-hex content digest on load, but every caller discarded it. The bundle holds only what scoring needs:

published-model/
  model/
    checkpoint  config.json  encoders.json  target-encoder.json  resolved-model-plan.json
  manifests/
    graph.json  task.json  input.json
  metrics.json                 # selected trial's metrics (see below)
  prediction-schema.json
  training-spec.json           # query, generation plan, model plan, snapshot id
  publication-manifest.json    # relative_path, role, size, sha256 of each file above

None of these files record:

• the training table or warm-start model the fit used, or where a custom training table's rows came from;
• the source tables (connector, location, version) behind the snapshot. Record provenance for every artifact under one MLOps artifact root #50's snapshot manifest has them, but it lives next to the index in scratch, not in the bundle;
• the fit start and end time, or duration;
• the engine build and library versions;
• which model the bundle ships. One fit can train up to three models from the best trial: fit (trained on train), refit_trainval (retrained on train+val) and refit_full (retrained on train+val+test). The bundle ships the one trained on the most data that was enabled: refit_full, else refit_trainval, else fit. metrics.json has test_* keys, which are the fit model's, and test_refit_* keys, which are the refit_trainval model's (_refit_metrics in kumo-core's training orchestration). Nothing says which of them, if any, belongs to the shipped model. For refit_full none do: it trained on the test split, so it has no test metrics of its own.

Change. Fit now writes _provenance.json (the series' one manifest name) and lists it in the publication manifest with role provenance. Nothing else in the bundle changes:

published-model/
  ...same files as above...
  _provenance.json             # new
  publication-manifest.json    # now also lists _provenance.json

Example from a fit on a three-table synthetic graph (users 200 rows, items 20, orders 1,000; query PREDICT COUNT(orders.*, 0, 14, days) > 0 FOR EACH users.USER_ID), captured from this branch. Hashes are shortened, environments elided, and the embedded snapshot's source_tables shown once:

{
  "kind": "model", "schema_version": 1, "id": "55dded33…",
  "parents": {
    "training_table": {
      "kind": "training_table", "schema_version": 1, "id": "tt-6a9795e8…",
      "parents": {"snapshot": {
        "kind": "snapshot", "schema_version": 1, "id": "01d8ad46…@b0d7459155debdeb",
        "parents": {"source_tables": {"orders": {"connector": "LOCAL_FILE", "location": "<data>/orders.parquet", "version": "1792862945", "version_kind": "timestamp"}, "...": "…"}},
        "record_sources": true, "environment": {"...": "…"}, "created_at": "2026-10-24T17:38:34+00:00"}},
      "query": "PREDICT COUNT(orders.*, 0, 14, days) > 0 FOR EACH users.USER_ID",
      "generation_plan": {"split": "inferred", "timeframe_step": 14, "...": "…"},
      "task_type": "binary_classification", "split_seed": null,
      "environment": {"...": "…"}, "created_at": "2026-10-24T17:38:34+00:00"},
    "warm_start": null},
  "fit": {"started_at": "2026-10-24T17:38:34+00:00", "finished_at": "2026-10-24T17:38:50+00:00", "seconds": 15.291},
  "fit_mode": "refit_trainval",
  "environment": {"kumo_engine": "1.0.0.dev0", "build_commit": "13209487", "...": "…"},
  "created_at": "2026-10-24T17:38:50+00:00"
}

fit returned a TrainingResult with id = bundle_id = 55dded33… and manifest = the record above. With fit_mode: "refit_trainval", the test_refit_* keys in the unchanged metrics.json are the shipped model's.

What each field records:

• parents.training_table is the training table's Record training-table lineage across export and reload #55 manifest, embedded. It includes the generating snapshot's manifest with its source tables' connector, location and version, and, for a table loaded with from_source_table, the source table the rows came from. The bundle therefore stays self-contained once scratch is gone. Embedded copies leave out statistics and fields written later. If the table has no manifest, a stub {kind, id, reconstructed: true} takes its place.
• parents.warm_start, for fit(warm_start=...), is the parent model's manifest. Its own model parents are cut to stubs ({kind, id, reconstructed}), so a chain of daily warm-start retrains keeps each manifest the same size. A parent bundle without a manifest becomes a stub with its bundle id.
• fit gives the wall-clock start, end and duration of fit().
• fit_mode is the fit mode of the checkpoint the bundle ships: fit, refit_trainval or refit_full. It comes from kumo-core's Trainer.published_fit_mode(), the same function (and order) that picks the checkpoint predictions use by default. Read it with metrics.json: fit means the test_* keys describe the shipped model, refit_trainval means the test_refit_* keys do, and refit_full means none do.
• environment and created_at use Record provenance for every artifact under one MLOps artifact root #50's format.
• id is the bundle id (below). It is computed before this file is written, since the file is excluded from it.

Seeds, per-trial outcomes and a bundle-format version are not recorded: nothing reads them, or another bundle file already holds them.

Artifact root. With MLOps on (fit(mlops=...) or a kumo.mlops block, #50), the bundle is also stored under <root>/models/<bundle id>/ (upload_bundle, #112, writes _provenance.json last). A copy already there with every file at the right size is kept; a partial one is replaced; a failure warns. With MLOps off nothing is stored. export_config= publishes exactly as before, independently.

bundle_id. verify_model_bundle now returns the full 64-hex id instead of a discarded 16-hex digest, and a fresh fit computes the same id with a new kumo-core bundle_id_from_manifest from the digests the publication manifest already lists, instead of re-reading every file (about 0.76 s of SHA-256 per fit for a bundle with a 1 GiB checkpoint, now about 0). Loading and fetching still verify every file fully. The id is the SHA-256 of the sorted [relative_path, sha256] pairs of every verified file in the publication manifest, plus training-spec.json if an older manifest left it unlisted. The manifest file is excluded (both _provenance.json and the earlier name provenance.json). Its timings and environment fields change from run to run without changing which model the bundle is. It is still size- and digest-checked like every other listed file. Role names are no longer part of the payload.

• Copies, downloads and registry round trips keep the id, because the files and their digests are unchanged.
• A refit never shares an id with an earlier fit, even with identical weights, because listed files such as model/config.json carry run-specific paths. Equal ids therefore mean "the same fitted artifact", not "equal weights from two runs".

Reading:

• fit returns a new TrainingResult per call. It has id (the bundle id), bundle_id and manifest (a ModelManifest, exported as kumo.engine.ModelManifest, or None).
• Trainer.result is the latest fit's result, or the loaded bundle's. Trainer.bundle_id and Trainer.manifest read it. All are set at fit and at Trainer.load.
• The manifest is read only when the publication manifest lists it. A newer file round-trips byte for byte (test_manifest_round_trips_unchanged).
• A new kumo-core add_bundle_file(bundle_dir, role, relative_path) lists a file in the publication manifest. It replaces the engine's own training-spec amend.
• kumo-core Trainer.published_fit_mode() returns the fit mode predictions use by default (refit_full, else refit_trainval, else fit; None before fitting).

Back-compat

This PR only records and exposes information. It adds no new checks or errors on any existing path, and predictions are unchanged.

Situation	Behaviour
New bundle	_provenance.json written and listed. bundle_id set on the result and the trainer. metrics.json is unchanged.
Existing bundle built before this PR, loaded with Trainer.load	Not rewritten. bundle_id is computed at load from the files it already has. manifest returns None. Predictions are identical.
Fit on a training table without a manifest, or warm-started from a bundle without one	That parent is a stub with its id. The fit and bundle are otherwise unchanged.
Existing bundle read directly, without loading	Unchanged. It has no manifest file.
Bundle written by this PR, loaded by an older engine	_provenance.json is an extra listed role. The older loader verifies its digest and ignores it (checked against the v1.0.0 loader).
Manifest write fails	A warning is logged, and the bundle publishes without it. A failure to compute the id also warns, and leaves bundle_id as None.
Manifest present but not listed, or unreadable, or a newer schema_version	Not listed: ignored, and manifest returns None. Unreadable or newer: reads as None with a warning.

Testing Done

• kumo-core test_artifacts.py: the manifest file (new and old name) is excluded from the bundle id, and the id from the publication manifest equals the verified id.
• test_bundle_provenance.py (7, behaviour tests):
  • a fit's manifest embeds the training table, which embeds snapshot.manifest; id equals the bundle id on the result and trainer; Trainer.load gives the same result.id and manifest;
  • a newer file round-trips byte for byte;
  • a bundle with its manifest removed loads with the same id and identical predictions;
  • each fit returns a new result;
  • missing parents are stubs; a three-generation warm-start chain embeds the parent with its own parent cut to a stub;
  • a kumo.mlops settings object stores the bundle once and repairs a damaged copy, while a block that does not export models stores nothing;
  • a manifest write failure warns and the fit completes with a bundle id.
• test_warm_start.py asserts the embedded warm-start parent.
• Also run at this head: test_trainer_registry.py (27), test_semantic_carry_forward.py (20), test_holdout_df.py (11) and kumo-core test_artifacts.py (10).
• The example above was captured from a real run of this branch on synthetic data.

Checklist

• No breaking changes.
• Unit tests added.

🤖 Generated with Claude Code

[Vedaant-J]

Change map

File	Function / scope	What changed	Why
packages/kumo-core/src/kumo_core/artifacts.py	verify_model_bundle, bundle_id_from_manifest, _bundle_id, add_bundle_file, _OLD_PROVENANCE_FILE	returns the 64-hex bundle id over sorted [relative_path, sha256] pairs, leaving out the manifest file (_provenance.json, and the earlier provenance.json); add_bundle_file lists a file in the publication manifest	stable model id that ignores the run-specific manifest
packages/kumo-core/tests/test_artifacts.py	test_bundle_id_ignores_provenance_only	id formula; the manifest file (new and old name) is excluded	core contract
packages/kumo-engine/docs/mlops.md	docs	models section: manifest, chain depth, results, artifact root	user docs for this PR
packages/kumo-engine/kumo/engine/__init__.py	__all__	exports ModelManifest	public manifest type
packages/kumo-engine/kumo/engine/_manifest.py	ModelParents, FitTiming, ModelManifest	model manifest dataclasses; parents embed the training-table and warm-start/derived-from manifests	typed, self-contained manifest
packages/kumo-engine/kumo/engine/trainer/_provenance.py	write_provenance, embedded_model, parent_model, read_provenance	writes _provenance.json with the bundle id, embedded training table (or stub), depth-1 warm-start model, fit timing and mode	lineage without copies, bounded chain size
packages/kumo-engine/kumo/engine/trainer/_spec.py	write_training_spec, listed_file, read_bundle_artifact_paths, PROVENANCE_ROLE	spec listed through add_bundle_file; listed_file finds a listed file by role; read_bundle_artifact_paths returns (paths, bundle_id) and keeps the provenance role out of the scoring paths	dedup; id available at load
packages/kumo-engine/kumo/engine/trainer/trainer.py	TrainingResult.id/manifest, Trainer.result/manifest/bundle_id, fit(mlops=), _finish_fit, _store_model, load	id from the publication manifest; manifest accessors; a new result per fit; with MLOps on, the bundle stored under models/<id>/ (complete copy kept, partial replaced)	model id, lineage and the artifact root
packages/kumo-engine/test/test_bundle_provenance.py	tests	manifest contents and round trip, load back-compat, new result per fit, stubs, flat warm-start chains, root store and repair, write failure	behaviour coverage
packages/kumo-engine/test/test_semantic_carry_forward.py	_best_trainer	Fit tests share a fake best trainer that answers published_fit_mode()	Fit now asks the trainer for the published fit mode
packages/kumo-engine/test/test_trainer_registry.py	finish-fit tests	checks the returned result's id	result now carries the id
packages/kumo-engine/test/test_warm_start.py	warm-start test	embedded warm-start parent asserted	lineage

Requirement: a stable bundle id tied to the scoring files, and a record of the training table, snapshot, source tables, engine build, timing and shipped fit mode that survives without scratch. Smallest change: reuse the existing verifier's digest and add one listed file; metrics.json unchanged. Stopping condition: id and provenance on every new bundle; old bundles load unchanged.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Walkthrough

The change defines bundle identity as a full SHA-256 digest of sorted path and file-digest pairs, excluding provenance.json and _provenance.json. It adds helpers to register bundle files and derive identity from a manifest. Training records fit and parent details in provenance. Trainer and TrainingResult expose bundle IDs and manifests, and bundle loading restores the ID. Optional MLOps settings store bundles. Tests cover provenance recording, missing provenance, warm-start parents, and storage behavior.

Sequence Diagram(s)

sequenceDiagram
  participant TrainerFit
  participant FinishFit
  participant VerifyModelBundle
  participant WriteProvenance
  participant AddBundleFile
  TrainerFit->>FinishFit: Pass training table, fit time, warm-start ID, fit mode, and snapshot
  FinishFit->>VerifyModelBundle: Verify bundle and compute bundle ID
  VerifyModelBundle-->>FinishFit: Return bundle ID
  FinishFit->>WriteProvenance: Pass bundle ID and fit provenance
  WriteProvenance->>AddBundleFile: Register provenance file

Loading

Priority: Normal

Merge Risk: Low · up to 456b9

With MLOps enabled, a failed re-upload of an existing stored model can leave the artifact-root copy empty or partial, although the local bundle and the fit result are unaffected. The scenario is narrow and recoverable, so the change is mergeable. Staging the overwrite before merge would remove the risk.

Pre-merge checks: 2 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Change Map Design Review		The supplied context summarizes the change map but does not include the conversation comment beginning with <!-- kumo-pr-change-map -->, and the comment is not present in the repository checkout. Th…	Provide the full marked change-map comment, including its linked problem, inspected behavior, change table, verification, and stopping condition. Then assess the design against the reviewed diff.

Passed checks: 2 passed

Check name	Status	Explanation
Pr Size		The pull request changes 12 files. The limit is 40 files, so the PR passes this check.
Public Resource Access		The pull request adds or modifies no external resource reference. The reviewed diff contains no URL, repository link, package source URL, or Markdown external link. The paths and module names added in…

Full details: Public Resource Access

Explanation

The pull request adds or modifies no external resource reference. The reviewed diff contains no URL, repository link, package source URL, or Markdown external link. The paths and module names added in code are local bundle paths and internal package imports, not external resources.

Full details: Change Map Design Review

Explanation

The supplied context summarizes the change map but does not include the conversation comment beginning with <!-- kumo-pr-change-map -->, and the comment is not present in the repository checkout. This check requires review of that comment's proposed design, evidence, verification, and stopping condition.

Resolution

Provide the full marked change-map comment, including its linked problem, inspected behavior, change table, verification, and stopping condition. Then assess the design against the reviewed diff.

Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 1

packages/kumo-engine/docs/mlops.md: 1

48-48: Minor · Maintainability & Code Quality · Quick win

State that artifact-root storage can fail without failing the fit.

_store_model warns and lets fit return when an upload fails. This sentence can lead a user to treat a successful fit as proof that the bundle is durable under the artifact root. State that storage is attempted and that users must check its warning or confirm the stored copy when durability matters.

---

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/kumo-engine
• Branch: vedaant/mlops-training-table-manifest
• File: .coderabbit.yaml
• Commit: 732f4c3
• Layers: Repository YAML
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 03cd3b4e-0949-41c8-b95f-5f90b1388acd

Commits

Reviewing files that changed from the base of the PR and between 4274048 and 456b97d.

Files selected for processing: 9

• packages/kumo-core/src/kumo_core/artifacts.py
• packages/kumo-core/tests/test_artifacts.py
• packages/kumo-engine/docs/mlops.md
• packages/kumo-engine/kumo/engine/__init__.py
• packages/kumo-engine/kumo/engine/_manifest.py
• packages/kumo-engine/kumo/engine/trainer/_provenance.py
• packages/kumo-engine/kumo/engine/trainer/trainer.py
• packages/kumo-engine/test/test_bundle_provenance.py
• packages/kumo-engine/test/test_warm_start.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 0 remain after this review.

[coderabbitai]

Major · Data Integrity & Integration · Quick win

🔎 Supported by static analysis

Script executed:

sed -n '195,245p' packages/kumo-engine/kumo/engine/trainer/trainer.py

Repository: NVIDIA/kumo-engine

Length of output: 2123

---

Script executed:

# Locate upload_bundle and the bundle-id computation, then show their bodies.
rg -n -F -e 'def upload_bundle' -e 'def compute_bundle_id' -e '_provenance.json' -- packages/ || test "$?" -eq 1
for f in $(rg -l -F -e 'def upload_bundle' -- packages/ 2>/dev/null); do
  echo "== $f"
  ast-grep outline "$f" --view expanded
done

Repository: NVIDIA/kumo-engine

Length of output: 5838

---

Script executed:

sed -n '20,62p' packages/kumo-core/src/kumo_core/object_store_sink.py
echo "== artifacts.py bundle id / provenance exclusion"
rg -n -F -e 'PROVENANCE_FILE' -e 'def ' -- packages/kumo-core/src/kumo_core/artifacts.py || test "$?" -eq 1

Repository: NVIDIA/kumo-engine

Length of output: 3133

---

Stage the bundle overwrite; delete stale files only after a successful upload.

_store_model deletes the remote copy before it uploads. A remote copy with the same bundle ID but a different _provenance.json size fails the size comparison, so the code deletes it. Provenance is excluded from the bundle ID, so the model files in that copy are complete and loadable. If the upload then fails, the warning leaves models/<bundle id>/ empty or partial, and the loadable copy is gone.

Uploading over the existing files first keeps a loadable copy in place during the upload. The model files have the same digests, so overwriting them does not change what loads. _provenance.json is uploaded last. Remove remote files that are absent locally only after the upload succeeds.

🐛 Suggested fix

--- "a/packages/kumo-engine/kumo/engine/trainer/trainer.py"
+++ "b/packages/kumo-engine/kumo/engine/trainer/trainer.py"
@@ -226,11 +226,11 @@
                 FileSelector(root, recursive=True, allow_not_found=True))
             if info.type is FileType.File
         }
         if remote == local:
             return
-        if remote:
-            filesystem.delete_dir_contents(root, missing_dir_ok=True)
         upload_bundle(filesystem, root, str(result.bundle_dir))
+        for name in set(remote) - set(local):
+            filesystem.delete_file(f'{root}/{name}')
     except Exception as error:
         warnings.warn(
             f"Could not store model {result.bundle_id} under the artifact "

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	if remote:
	filesystem.delete_dir_contents(root, missing_dir_ok=True)
	upload_bundle(filesystem, root, str(result.bundle_dir))
	upload_bundle(filesystem, root, str(result.bundle_dir))
	for name in set(remote) - set(local):
	filesystem.delete_file(f'{root}/{name}')

[Vedaant-J]

Folded into #50 (artifacts & provenance), which now contains this PR's final code: the consolidated stack shows only the end state. Review continues there.