Record provenance for every artifact under one MLOps artifact root by Vedaant-J · Pull Request #50 · NVIDIA/kumo-engine

Vedaant-J · 2026-10-20T22:11:58Z

Series: kumo-engine MLOps series: every snapshot, training table, model and prediction run carries a self-contained _provenance.json; with kumo.mlops(...) stores each artifact under one artifact root, tracks fits and prediction runs in MLflow or Snowflake and registers models; volumes and stages can hold files; prediction runs are monitored for drift and realized performance.
Stack: #53 → #112 → #50 → #60 → #64; #114 on #112; #115 on #64
Merge order: one linear stack; merge bottom to top (#53 first); #114 sits on #112 and merges any time after it; #115 (draft) sits on #64 and merges after it
This PR: Adds artifacts and provenance: every graph snapshot, training table and model bundle carries a self-contained _provenance.json that leads back to the source table versions it came from, snapshots can be exported and loaded on another machine, and with kumo.mlops(...) stores each artifact under one artifact root.

Update (10-29, recheck): an artifact root outside its connector fails when built

Problem. kumo.MLOps((S3Connector(root_dir="s3://<bucket>/data"), "s3://<bucket>/artifacts")) was accepted. Every store then failed with Output path is outside the selected connector target, only as warnings under on_error="warn", so the fit returned with nothing stored and no registered model.
Change. kumo.MLOps checks a (connector, path) root against the connector's root_dir when it is built and raises: artifact_root 's3://bucket/kumo' is outside this S3Connector's root_dir s3://bucket/data (...), so nothing could be stored there. Give the root a connector that contains it: (S3Connector(root_dir='s3://bucket/kumo', identity=...), 's3://bucket/kumo').
Verified. test_mlops: 5 passed.

Update (10-29, usability pass 2): same-schema graph changes warn

Problem. Scoring a model with the same tables read from another location (a copy of users, items and orders under other names) changed the graph id but showed nothing: a graph that differs only in table connectors or locations was logged at info level, which default logging hides, while the docs promised a warning.
Change. predict and GraphSnapshot.load_from_connector(..., graph=) raise a UserWarning for it too: Snapshot <id> was taken of graph <key>, but this model was trained on a snapshot of graph <key>. The schema is the same; only table connectors or locations differ. Scores are unchanged.
Verified. test_graph_mismatch, test_snapshot_export: 42 passed.

Update (10-28): UX simplifications and a live-run fix

kumo.MLOps(store=...) replaces export= (the kinds stored under the root), so "export" again means only a copy elsewhere.
Read back through the settings: a kumo.MLOps stands in for the connector in load_from_connector, with paths relative to the artifact root: GraphSnapshot.load_from_connector(cfg, id=snapshot_id), TrainingTable.load_from_connector(cfg, path=f"training_tables/{id}", ...), Trainer.load_from_connector(cfg, path=f"models/{id}"). No root connector is rebuilt by hand.
TrainingResult.snapshot_id names the training snapshot, read from the bundle's training spec.
DatabricksConnector(identity=None) uses the Databricks SDK's ambient auth (notebook, job, profile); from_notebook stays.
Fixes from the live validation: a fit whose test split is empty now raises ValueError: The test split is empty: shorten the query window or add data. before training (it raised an opaque evaluation error after training); kumo.MLOps("@DB.SCHEMA.STAGE/...") as a string raises and asks for the (SnowflakeConnector, path) pair instead of becoming a local directory.
docs/mlops.md gains a quickstart, an id table and a Reference section.

Examples further down that show export= predate this update.

Summary

This PR adds artifacts and provenance: every graph snapshot, training table and model bundle carries a self-contained _provenance.json that leads back to the source table versions it came from, snapshots can be exported and loaded on another machine, and with kumo.mlops(...) stores each artifact under one artifact root. It also gives models a stable content id and makes Trainer.predict(snapshot) the one scoring call, so a job holding only a stored model and a stored snapshot can score without the training job's Graph code.

The PRs above build on it:

Record fits in experiment tracking and publish models to registries #60 adds tracking and registry: MLOps(tracking=..., registry=...) logs fits to MLflow or Snowflake and registers models.
Record prediction runs and monitor score drift #64 adds prediction and monitoring: prediction-run manifests stored under predictions/<id>/, mlops= on predict, and drift and realized-performance monitoring.

Problem. Teams running Kumo as scheduled jobs need to answer questions like these:

Scores shifted this week. Was it the data or the model?
Which tables, at which location and version, did the snapshot behind this model come from? Which engine build wrote it?
Which training table, query, generation plan and warm-start model produced this model? Can we still tell once the training machine's scratch directory is gone?
Two registry versions exist. Do they hold the same fitted model?
Which test metrics in metrics.json, if any, belong to the model that was shipped?
Our job runs on compute that starts with an empty disk. Why does every run rebuild the index from the sources? Can we build it once and train or score elsewhere?
A scoring job has the model bundle and the snapshot. How does it score without re-declaring every table, key and edge in Python?

Today none of these can be answered: artifacts carry no lineage, a model has no id, an index can only be used on the machine that built it, and scoring needs a Graph.

Cause. Each artifact holds only what the engine needs to run it.

A snapshot is identified only by graph.id@data_version[:16]. Both parts are SHA-256 hashes: they can confirm a guess but cannot be reversed into tables and versions. The index directory holds DiskGraph's files only:

<scratch>/snapshots/<graph_id>/<data_version>/
  _SUCCESS                           # ingest marker; records this absolute path
  _MANIFEST.json                     # DiskGraph file list with sizes
  diskgraph_config.json  ingest_stats.json  col_stats.json  edge_stats.json  index_metadata.json
  kumo_col_stats/  nodes/  train_tables/

Nothing records each source table's connector, location and version, the engine build or the build time. _SUCCESS records the absolute path the index was built at, so an index copied to another machine is treated as missing and rebuilt.

A generated training table's staging directory and its export:

<index>/train_tables/.<tt_id>.staging/
  tt_ingest_spec.json  manifest.json  stats.json  split_stats.json  train_table_col_stats.json
  tt.parquet/  train_seeds/  val_seeds/  test_seeds/

<export>/
  part-00000.parquet  part-00001.parquet  part-00002.parquet
  _KUMO_EXPORT  _SUCCESS

tt_ingest_spec.json holds a random tt_id. Nothing links it to the snapshot, the query or the plan, and a table loaded from rows (TrainingTable.from_source_table) keeps no record of where the rows came from.

A model bundle:

published-model/
  model/{checkpoint, config.json, encoders.json, target-encoder.json, resolved-model-plan.json}
  manifests/{graph.json, task.json, input.json}
  metrics.json  prediction-schema.json
  training-spec.json                 # query, generation plan, model plan, snapshot id
  publication-manifest.json          # relative_path, role, size, sha256 of each file

It has no id: verify_model_bundle computed a 16-hex digest on load and every caller discarded it. It records neither the training table, the warm-start model, the fit time nor the engine build. One fit can train up to three models from the best trial (fit, refit_trainval, refit_full), and the bundle ships the one trained on the most data, but nothing says which. metrics.json holds test_* keys for the fit model and test_refit_* keys for refit_trainval.

Scoring needed the graph. PredictiveQuery parsed against graph.native_graph and checked snapshots with snapshot.id.startswith(f'{self.graph.id}@'), and Trainer.predict took only a prediction table built from that query.

Change.

The MLOps switch

Off unless asked for:

import kumo.engine as kumo

cfg = kumo.MLOps(artifact_root="s3://<bucket>/kumo")   # or (connector, path) for a volume or stage
with kumo.mlops(cfg):                                   # or mlops=cfg on one call
    snapshot = graph.snapshot()
    table = pq.generate_training_table(snapshot)
    result = trainer.fit(table, snapshot=snapshot)

artifact_root is required; an empty one raises ValueError. It is either a URI (s3://, gs://, file:// or a local path, including /Volumes/... on a Databricks cluster), mapped to a connector by kumo.engine.connector.connector_for_root, or a (connector, path) pair whose sink holds files at path (FileStoreSink.holds_files, Let any sink hold files: FileStoreSink protocol and a chunked filesystem #112). A pair whose sink does not hold files raises ValueError.
export names the kinds stored under the root when they are produced: snapshot, training_table, model (kumo.engine._manifest.KINDS; Record prediction runs and monitor score drift #64 adds prediction). The default True means all and False none; an unknown kind raises.
Precedence: a call's mlops= replaces the enclosing block, and an inner block replaces an outer one. The block is a contextvar, so it covers its own thread or asyncio task. kumo.mlops() takes only a kumo.MLOps (TypeError otherwise).
Calls that take mlops=: Graph.snapshot, PredictiveQuery.generate_training_table, TrainingTable.from_source_table and Trainer.fit.
Plain exports never write to the root: GraphSnapshot.export(config), TrainingTable.export(config) and fit(export_config=...) behave the same inside or outside a block.

With MLOps on, a run on a three-table synthetic graph (users 200 rows, items 20, orders 1,000) writes this root. Everything under it is new:

<root>/
  snapshots/f896d098…/2377a8031c90cc98/         # Graph.snapshot()
    9c41e0d2…/                                  # one export generation
      index/                                    # the index files, same relative paths
        _MANIFEST.json  col_stats.json  diskgraph_config.json  edge_stats.json  index_metadata.json  ingest_stats.json
        kumo_col_stats/{col_stats_full,col_stats_min}/<table>.json
        nodes/...
      _provenance.json                          # the snapshot manifest
      _DONE                                     # written after its commit
    kumo_snapshot.json                          # export marker, written last; names the generation
  training_tables/tt-3dc0f233…/                 # generate_training_table / from_source_table
    part-00000.parquet  part-00001.parquet  part-00002.parquet
    _KUMO_EXPORT  _provenance.json  _SUCCESS
  models/7b604851…/                             # Trainer.fit
    model/  manifests/  metrics.json  prediction-schema.json  training-spec.json
    publication-manifest.json  _provenance.json

Each kind is written once per id:

Snapshots are skipped when kumo_snapshot.json there has this engine's format and the same snapshot id.
Training tables are skipped when _SUCCESS is there.
Models are skipped when a copy with every file at the right size is there; a partial copy is replaced.
A store failure warns and the call returns as usual.

Manifests

Every artifact carries _provenance.json (kumo_core.artifacts.PROVENANCE_FILE, #112). The leading underscore keeps Spark and Hive Parquet readers off it. A manifest is metadata only and self-contained:

a training table's embeds its snapshot's manifest;
a model's embeds its training table's (and so the snapshot's) and its warm-start model's.

Any one model manifest therefore leads to source table versions without the scratch directory that built them.

Shared envelope (kumo.engine._manifest.Manifest):

kind is snapshot, training_table or model; the prediction manifest comes with Record prediction runs and monitor score drift #64.
schema_version is 1.
id is the artifact's id.
created_at is a UTC timestamp.
environment records:
- the engine version and build_commit, stamped into the wheel by setup.py from KUMO_ENGINE_BUILD_COMMIT or git rev-parse, since dev builds share one version string;
- kumo_core, torch and python;
- DiskGraph's __version__ and BUILD_COMMIT, stamped by diskgraph-pybind/build.rs (DISKGRAPH_BUILD_COMMIT overrides it; "unknown" without git).
reconstructed: true marks a manifest rebuilt after the fact or a stub. A stub has only kind, id and reconstructed, and stands in for a parent with no manifest of its own. A malformed parent also reads as a stub, with a warning, rather than dropping the whole manifest.

Python reads every manifest as a frozen dataclass with to_dict()/from_dict():

kumo.engine.SnapshotManifest, TrainingTableManifest and ModelManifest are public.
The envelope, stub() and embedded() stay private in kumo.engine._manifest.
Keys a newer engine writes are kept in extra and written back, so a newer file round-trips byte for byte.
A file that is unreadable, or whose schema_version is missing, not an integer or newer than this engine, reads as None with a warning.

Snapshot manifest

Graph.snapshot() writes it next to the index. From the run above; the hash is shortened, the data directory is <data>, and the engine and kumo-core versions are shown as released-style values (a source checkout records its git revision and kumo_core: null):

{
  "kind": "snapshot",
  "schema_version": 1,
  "id": "f896d098…@2377a8031c90cc98",
  "parents": {
    "source_tables": {
      "items":  {"connector": "LOCAL_FILE", "location": "<data>/items.parquet",  "version": "1793025958", "version_kind": "timestamp"},
      "orders": {"connector": "LOCAL_FILE", "location": "<data>/orders.parquet", "version": "1793025958", "version_kind": "timestamp"},
      "users":  {"connector": "LOCAL_FILE", "location": "<data>/users.parquet",  "version": "1793025958", "version_kind": "timestamp"}
    }
  },
  "record_sources": true,
  "environment": {"kumo_engine": "1.0.0.dev0", "build_commit": "ad8ce653", "kumo_core": "0.1.0.dev1",
                  "diskgraph": "0.1.0", "diskgraph_build_commit": "639b1017c63c",
                  "torch": "2.12.0+cu130", "python": "3.14.7"},
  "created_at": "2026-10-26T14:45:59+00:00"
}

id is GraphSnapshot.id. The graph hash and the full data version are in it and in the index directory name, so they are not repeated.
parents.source_tables has one entry per table alias (SourceTableEntry):
- connector is the connector type: LOCAL_FILE, S3, GCS, SNOWFLAKE, DATABRICKS, BIGQUERY, ...
- location is where the data is. Credentials are never recorded.
  - A FileConnector records the resolved absolute path.
  - S3 and GCS record the full s3:// or gs:// URI, including bucket and root, percent-encoded as the store spells it.
  - Warehouses record the fully qualified table name. Snowflake prefixes it with the account (<account>/"DB"."SCHEMA"."TABLE").
  - Custom SQL records sql:<sha256 of the SQL>.
  - Other connector types record the table name.
  - Each backend provides table_location.
- version is the stamp the engine already uses to decide index reuse:
  - a file's modification time;
  - an object-store listing fingerprint;
  - Snowflake LAST_ALTERED;
  - BigQuery's modification time;
  - the Delta version and commit time.
- version_kind says what that stamp proves. The connector that produced it sets it (kumo-core SourceVersion, a str subclass):
  - exact: a real data version (Delta).
  - timestamp: a modification time (single files and objects, Snowflake and BigQuery tables).
  - listing: a sha256 of the paths, sizes and mtimes of a directory's parts.
  - hour_bucket: the hour the source was read in. Views use it, and so does Custom SQL, whose version folds in a 24-hour bucket.
  - unknown: anything else.
- Custom SQL only:
  - dependencies: the version of each table the query reads.
  - normalized_sql_fingerprint: custom_sql_<md5>, unchanged by whitespace, comment and keyword-case edits.
  - sql, dialect, namespace: the query text, its dialect, and the [database, schema], [catalog, schema] or [project, dataset] its unqualified names resolve in.
- If a location cannot be determined, connector and location are null and a warning is logged.
record_sources is the choice the snapshot was taken with. graph.snapshot(record_sources=False) records each location as the connector's table name only, plus location_fingerprint, a SHA-256 of the connector type and full location, so source_changes still notices a source moved elsewhere, and Custom SQL as its sql: hash and fingerprint only. That is for deployments that cannot carry source details in artifacts that travel.

GraphSnapshot.manifest returns the SnapshotManifest or None. GraphSnapshot.record_sources (bool | None, not part of equality) is the choice, or None for a hand-built or loaded snapshot.

Training-table manifest

Generation writes _provenance.json into the table's staging directory:

{
  "kind": "training_table", "schema_version": 1,
  "id": "tt-3dc0f233…",
  "parents": {"snapshot": {"kind": "snapshot", "id": "f896d098…@2377a8031c90cc98", "...": "the snapshot manifest above, embedded"}},
  "query": "PREDICT COUNT(orders.*, 0, 14, days) > 0 FOR EACH users.USER_ID",
  "task_type": "binary_classification",
  "split_seed": null,
  "generation_plan": {"split": "inferred", "timeframe_step": 14, "train_start_offset": "inferred", "train_end_offset": "inferred", "...": "…"},
  "environment": {"...": "as above"},
  "created_at": "2026-10-26T14:45:59+00:00"
}

id (TrainingTable.id) is tt- plus 32 hex of a SHA-256 over:
- the snapshot id, query and generation plan;
- the drawn split seed;
- for a table loaded from rows, the source's version;
- the engine and DiskGraph builds.
Generating the same table twice gives one id (tt-3dc0f233… both times in the run above). DiskGraph's own tt_id (TrainingTable.tt_id) stays random per generation (tt-0a56e445… and tt-d08e6761…), so two generations never share a staging directory. A loaded source that reports no version gets a random id. A hand-built TrainingTable has id=None.
parents.snapshot is the manifest of the snapshot the table was generated from, or validated against for loaded rows. It is a stub if that snapshot has none.
parents.source_table is present only for from_source_table, in the snapshot's SourceTableEntry shape. Its version is the one the staging read itself returned, so loading makes no extra call. Locations follow the snapshot's record_sources, then its manifest's; when neither is known, only connector and table name are recorded. An export loaded back records:
```
"source_table": {"connector": "LOCAL_FILE", "location": "<out>/tt", "version": "ad803af0…", "version_kind": "listing"}
```
query, task_type and generation_plan record what produced the rows. The plan is the effective one the table carries into fit.
split_seed is the seed of a RandomSplit/RandomLinkSplit (DiskGraph's default 0 if unset). It is null for temporal and manual splits.

TrainingTable.manifest returns it, or None for tables generated before this PR. Split row counts and whether the table was generated or loaded stay in DiskGraph's manifest.json.

TrainingTable.export to a Parquet directory on a file destination now writes the manifest as a sidecar. It lands after the parts and before _SUCCESS; the parts are byte-identical to before:

<export>/
  part-00000.parquet  part-00001.parquet  part-00002.parquet
  _KUMO_EXPORT
  _provenance.json                   # new: the training-table manifest
  _SUCCESS

The sidecar is for whoever receives the export; the engine never reads it. CSV, single-file and warehouse destinations get none. A failed sidecar read or write warns, and the export completes without it. kumo-core's export_selected_files and ObjectStoreSinkBackend.write_files gain sidecars: Mapping[str, bytes] = {} to carry it.

Model manifest and bundle id

fit writes _provenance.json into the bundle and lists it in publication-manifest.json with role provenance:

{"relative_path": "_provenance.json", "role": "provenance", "sha256": "312fca52…", "size_bytes": 3264}

The manifest, from the same run:

{
  "kind": "model", "schema_version": 1, "id": "7b604851…",
  "parents": {
    "training_table": {"kind": "training_table", "id": "tt-3dc0f233…", "...": "the training-table manifest above, embedded with its snapshot"},
    "warm_start": null
  },
  "fit": {"started_at": "2026-10-26T14:45:59+00:00", "finished_at": "2026-10-26T14:46:25+00:00", "seconds": 26.004},
  "fit_mode": "refit_trainval",
  "environment": {"...": "as above"},
  "created_at": "2026-10-26T14:46:25+00:00"
}

id is the bundle id, a 64-hex content hash:
- It is the SHA-256 of the sorted [relative_path, sha256] pairs of every file the publication manifest lists, plus training-spec.json if an older manifest left it unlisted.
- The manifest file (_provenance.json) is left out, since its timings change from run to run.
- Copies, downloads and registry round trips keep the id.
- A refit never shares an id with an earlier fit, because listed files such as model/config.json carry run-specific paths. Equal ids mean the same fitted artifact.
- verify_model_bundle now returns this id (it returned a 16-hex digest that was discarded).
- A fresh fit computes the same id from the digests already listed, with kumo-core's bundle_id_from_manifest, instead of re-reading the checkpoint.
parents.training_table is the training table's manifest, or a stub with its id.
parents.warm_start, for fit(warm_start=...), is the parent model's manifest. Its own model parents are cut to stubs, so a chain of daily warm-start retrains keeps each manifest the same size. A parent bundle without a manifest is a stub with its bundle id.
fit is the wall-clock start, end and duration of fit().
fit_mode is the mode of the shipped checkpoint, from kumo-core Trainer.published_fit_mode(): refit_full, else refit_trainval, else fit. Read it with metrics.json:
- fit: the test_* keys describe the shipped model.
- refit_trainval: the test_refit_* keys describe it.
- refit_full: no test keys do, since that model trained on the test split.

Reading:

fit returns a new TrainingResult per call with id (the bundle id), bundle_id and manifest.
Trainer.result is the latest fit's result, or the loaded bundle's.
Trainer.load and load_from_connector compute the id from the files they verify.
kumo-core gains add_bundle_file(bundle_dir, role, relative_path), which lists a written file in the publication manifest. The training spec uses it too.

Moving snapshots between machines

from kumo.engine import GraphSnapshot, S3Connector, SnapshotOutputConfig

location = graph.snapshot().export(SnapshotOutputConfig(S3Connector(), "s3://<bucket>/snapshots"))
# On another machine, with no index and no source access:
snapshot = GraphSnapshot.load_from_connector(S3Connector(), path=location)

Export. GraphSnapshot.export(config) writes <path>/<graph id>/<data_version[:16]>/, the two halves of snapshot.id, and returns that location:

It copies into index/ only the files listed in DiskGraph's _MANIFEST.json. Machine-local files (_SUCCESS, generated training tables) stay behind.
It carries the snapshot's _provenance.json byte for byte, so a newer-schema manifest is preserved. An index without one gets a reconstructed manifest with empty source_tables.
It writes kumo_snapshot.json last.

kumo_snapshot.json from the run above (hashes shortened, graph_config and the checksum map trimmed):

{
  "format": 1,
  "snapshot_id": "f896d098…@2377a8031c90cc98",
  "graph_key": "f896d098…",
  "data_version": "2377a8031c90cc98426d66ff…",
  "index_manifest": "c6c0fd31…",
  "graph_config": {"...": "…"},
  "environment": {"...": "as in the manifests"},
  "range_bytes": 67108864,
  "crc32": {"_MANIFEST.json": [38525221], "col_stats.json": [2598887258], "diskgraph_config.json": [3756067451], "...": "…"}
}

generation names the directory holding this export's files.
format is the export layout version. Load rejects any other value with a ValueError saying to load it with the exporting kumo-engine. Unknown extra fields are ignored.
snapshot_id, graph_key, data_version name the snapshot and its place in the local store. Load refuses ids that are not the 64-hex pairs this package produces.
index_manifest is the SHA-256 of the uploaded index/_MANIFEST.json. Load checks it before trusting that file's names and sizes.
graph_config makes the loaded GraphSnapshot complete without the source Graph.
environment is the engine that ran the export.
range_bytes, crc32 hold one CRC32 per 64 MiB range of every index file. DiskGraph's new snapshot_range_checksums(index_path, range_bytes) computes them in Rust with rayon while the files upload.

Reuse: exporting again first reads the destination. When kumo_snapshot.json names this snapshot, the local index holds the export's bytes (the .kumo_export identity, or one checksum pass when the index has none), one listing of the generation shows _DONE, index/_MANIFEST.json and every listed index file at its size, and its _provenance.json equals the local one byte for byte (so a re-export after record_sources=False publishes the redacted manifest), the upload is skipped and the location returned, with an info log. The same rule covers the other path exports: a training-table or prediction directory with _SUCCESS and the same _KUMO_EXPORT identity, and a model bundle path whose files, sizes and _provenance.json match (kumo_core.object_store_sink.holds_bundle, which the artifact-root model store now uses too).

Transfer and commit:

Files move in parallel through kumo-core's copies worker sizing and run_in_parallel. Retries build a fresh client so credentials renew.
Every upload passes its size to open_sized (Let any sink hold files: FileStoreSink protocol and a chunked filesystem #112), so a chunked store keeps files up to one part as plain objects.
Each export uploads into a new generation directory (a random 32-hex name), then writes that generation's index/_MANIFEST.json and _provenance.json, and commits by writing kumo_snapshot.json, which names the generation. On a local or volume path the marker is written inside the generation and renamed into place; on an object store it is one PUT. Either way a reader sees one whole marker. Until then the previous export stays loadable. Concurrent exports, even of different builds of one snapshot, never write the same file, so the last commit wins whole.
After committing it marks its generation _DONE and removes, best-effort, every other done generation that the re-read marker does not name, and generations with no file newer than a day (abandoned exports). An export still writing (a file within the last day) is never removed, and directories that are not generations are never touched.
A destination whose sink does not hold files raises NotImplementedError naming a file destination.

Load. GraphSnapshot.load_from_connector(connector, *, path, graph=None) downloads ranges in parallel into <target>.loading, after checking free space and pre-allocating the files. It then:

runs DiskGraph's verify_snapshot_integrity and the checksum pass;
checks that DiskGraph can open the index;
writes _provenance.json as downloaded;
renames into the local snapshot store, where Graph.snapshot() would have built it, with _SUCCESS rewritten for this machine.

On failure:

A failed load leaves nothing behind.
A missing marker says to re-run export; an unreadable one raises the corrupt-export ValueError (the artifact-root store re-exports instead).
A short file, checksum mismatch or malformed marker raises one ValueError: the export is corrupt; delete it and export again.

On reuse:

A later graph.snapshot() on that machine reuses the loaded index.
A second load reuses the local copy only if it holds this export's bytes: the load records the export's identity (a hash of its range checksums and index manifest) as .kumo_export next to the index, and an index without one (built on this machine) is checksummed once and compared. Another build of the same snapshot id is downloaded again rather than served from the cache.
The exporting machine records the export's identity next to its own index, so loading its own export there reuses the index without a checksum pass.
If a newer export of the same snapshot commits while a load downloads, the load's files disappear; it re-reads the marker and loads the newer generation.

Is it still current? snapshot.source_changes(graph) compares, per alias, the recorded version with what the source reports now, through the connector's metadata-only get_data_version. It never stages rows and never raises. From the run, after orders.parquet was rewritten:

{"users":  {"status": "unchanged", "recorded": "1793025958", "current": "1793025958"},
 "items":  {"status": "unchanged", "recorded": "1793025958", "current": "1793025958"},
 "orders": {"status": "changed",   "recorded": "1793025958", "current": "1793025992"}}

The status is unknown when there is nothing reliable to compare:

either version_kind is hour_bucket or unknown;
the table is Custom SQL;
the manifest is absent or reconstructed;
the table now reads from a different connector or location (with record_sources=False, compared through the recorded location_fingerprint);
the current version could not be read.

load_from_connector(..., graph=graph) runs the same check and issues one UserWarning if any table changed. It also warns when graph is a different graph. Nothing is refused.

Snapshot at snapshots/f896d098…/2377a8031c90cc98 predates source table(s) orders; see `snapshot.source_changes`.

Scoring from a snapshot

snapshot = GraphSnapshot.load_from_connector(store, path=location)
trainer = Trainer.load_from_connector(store, path="models/<bundle id>")
result = trainer.predict(snapshot, binary_classification_threshold=0.5)

Trainer.predict(snapshot, *, entities=None, prediction_time=None, output_config=None, num_classes_to_return=None, num_workers=1, binary_classification_threshold=None, prediction_table=None):

With only a snapshot, it binds the trained query (from the last fit, or the bundle's training-spec.json) to the snapshot and builds the prediction table with the trained generation plan. It then scores the query's population at prediction_time (the latest source time when omitted).
entities= scores a source table of ENTITY1 (and TIMESTAMP) rows instead; passing it with prediction_time raises ValueError.
Before any fit or load, it raises RuntimeError.
If the snapshot's graph key differs from the training snapshot's, it still scores. The model's training-spec.json records the training graph config (graph_config), so the check compares schemas with each table's connector and location left out: a schema difference warns and names it (for example Schema changes: events edges.); a graph that differs only in connectors or locations, such as the same schema in dev and prod, warns that the schema is the same (10-29; it was logged at info level). A model saved without a recorded graph keeps the plain warning with a hint. GraphSnapshot.load_from_connector(..., graph=) uses the same comparison. The check says the snapshot has the model's schema, not that its features match the model.
predict(prediction_table, snapshot=...) scores a prediction table generated beforehand, as before, positionally or with prediction_table= as a keyword. A prediction table already holds its entities and times, so passing it with entities= or prediction_time= raises ValueError; a snapshot or prediction table given twice raises TypeError.
TrainingResult.predict(...) scores with that result's own model: after a later fit on the same Trainer, an earlier result loads its own bundle on first use. A hand-built result raises RuntimeError.

In the run above, a fresh scratch directory loaded the stored snapshot and model and scored this way.

Underneath:

PredictiveQuery.graph widens to Graph | GraphSnapshot. The private PredictiveQuery._from_snapshot binds a snapshot, which parses against snapshot.graph_config through a cached private GraphSnapshot._native_graph.
The snapshot check compares graph keys (the private GraphSnapshot._graph_key) for both bindings. A snapshot of another graph raises ValueError.

Shortcut

One convenience over the calls above:

TrainingTable.load_from_connector(connector, *, path, pq, snapshot) reloads an export: it is from_source_table(connector[path], TrainingTableSpec(split_col='SPLIT'), pq=pq, snapshot=snapshot).

Boundaries

Public in kumo.engine:

MLOps, mlops
SnapshotManifest, TrainingTableManifest, ModelManifest
SnapshotOutputConfig
GraphSnapshot.export, load_from_connector, source_changes, manifest, record_sources
TrainingTable.id, tt_id, manifest, load_from_connector
TrainingResult.id, bundle_id, manifest, predict
Trainer.result (trainer.result.id, trainer.result.manifest)
connector.connector_for_root

Private:

_manifest: records, stub, embedded, reader and writer.
_mlops: resolve.
_provenance: environment and source entries.
_snapshot_io: export generations, load, export identity and store.
trainer/_provenance: the bundle manifest.
GraphSnapshot._graph_key, _data_version, _native_graph.

kumo-core:

io.filesystem.atomic_write_bytes and fsync_directory, now shared by the artifact store, the trainer spec and the manifest writers.
artifacts.bundle_id_from_manifest, add_bundle_file and the 64-hex verify_model_bundle.
sources.identity.SourceVersion and version_kind.
ObjectStoreSessions.table_location.
sources.object_store.copies, now public.
the sidecars export argument.
object_store_sink.holds_export and holds_bundle: whether a directory holds a complete export with a given identity, or a complete copy of a bundle.

Warehouse plugins: the Snowflake, Databricks and BigQuery plugins gain table_location and typed version stamps.

DiskGraph:

__version__ and BUILD_COMMIT.
snapshot_range_checksums, which shares _MANIFEST.json reading with verify_snapshot_integrity and adds the crc32fast crate.

The docs are packages/kumo-engine/docs/mlops.md, linked from the README.

Back-compat

This PR only records and exposes information and adds calls. Without a block or mlops=, every existing call behaves as before, and no new check raises on an existing path.

Situation	Behaviour
New snapshot, training table or bundle	`_provenance.json` written; the bundle also lists it. Ingest, rows, splits, weights and predictions are unchanged.
Bundle of a new fit with a holdout	Also carries `holdout.parquet` (the published model's holdout predictions, the file `holdout_df()` reads; capped at 1 GiB in memory when written, usually a few MB on disk: ENTITY, label and score columns per holdout row) and `holdout-meta.json` (its truncation flag), both listed and hash-checked. So new fits get a different bundle id than they would have before; existing bundles and their ids are unchanged. Registry uploads and the artifact-root copy grow by the file's size.
`holdout_df()` on a loaded or fetched model	Returns the bundled holdout (it raised `FileNotFoundError` before, which broke the Snowflake monitor baseline in a later job); a bundle without the copy still raises `No holdout dataframe was produced`.
Index built before this PR, reused by `snapshot()`	Not rebuilt. A manifest is backfilled with `"reconstructed": true`. Its `parents` are accurate, because reuse needs every current version to match the index's data version. `created_at` and `environment` are `null`.
Same, in a read-only location	One warning per index path; the snapshot is returned as before.
Index reused with a different `record_sources` choice	Not rebuilt. `source_tables` and `record_sources` are rewritten; `created_at`, `environment` and unknown `parents` fields are kept.
Training table generated before this PR	`manifest` is `None`, and `tt_id` still reads from `tt_ingest_spec.json`. Exporting it writes no sidecar.
Bundle published before this PR, loaded	Not rewritten. Its id is computed at load and `manifest` is `None`. Predictions are identical. As a warm-start parent it becomes a stub with its id.
Index, export or bundle written by this PR, read by an older engine	The extra file is ignored. An older bundle loader verifies the `provenance` entry's digest like any listed file and does not use it.
`GraphSnapshot(...)` or `TrainingResult(...)` built by hand	Unchanged: the new fields have defaults (`GraphSnapshot.record_sources` is also left out of equality).
`PredictiveQuery(graph, query)`	Unchanged, except the refusal message for a snapshot of another graph now reads "was not materialized from this query's graph (id ...) -- pass a snapshot of that graph."
`trainer.predict(prediction_table, snapshot=...)`	Unchanged; argument checks still run first.
Parquet directory export of a training table	Parts byte-identical; `_provenance.json` added before `_SUCCESS`. Edited or sidecar-less exports load as before.
`verify_model_bundle` return value	64 hex instead of a 16-hex digest. No caller used the old value.
Manifest, sidecar or root-store write fails	A warning is logged; the snapshot, table, fit or export completes as before. A failure to compute the bundle id warns and leaves it `None`.
Manifest unreadable or with an unsupported `schema_version`	Reads as `None` with a warning; a malformed embedded parent reads as a stub.
Snapshot export with an unknown `format`, or written by a DiskGraph this one cannot open	`ValueError` naming the format, or a `RuntimeError` carrying DiskGraph's error and the exporting kumo-engine version.
Path export to a destination that already holds a complete copy of the same artifact	The upload is skipped and logged at info; before, the snapshot was uploaded into a new generation and a directory was emptied and rewritten. An incomplete copy, or one with another identity, is exported as before.
Third-party `ObjectStoreSinkBackend` subclass overriding `write_files`	Directory exports of training tables to it now pass `sidecars=`; an override without that argument or `**kwargs` gets a `TypeError`. None exists in this repo.

Known limitations

View-backed sources only get an hour_bucket version. Lineage says "read during this hour", and source_changes reports unknown.
source_changes reports Custom SQL as unknown even though its dependency versions are recorded.
The graph hash in a snapshot id includes connector details such as warehouse and role, so the same graph in dev and prod gets different ids. Match across environments on parents.source_tables.
build_commit is null for a wheel built without KUMO_ENGINE_BUILD_COMMIT and outside git (for example from an sdist).
Snapshot export is atomic per location, not isolated. A load that overlaps a re-export of the same location can fail and should be retried, and every export uploads the whole index.
Checksums detect corruption, not tampering. Load only from paths whose writers you trust.
The graph check in predict and PredictiveQuery compares graph keys only. It does not check that a snapshot's features, types or encoders match the model.
Live S3 and GCS round trips are env-gated (test_snapshot_export_live.py) and were not run for this tree.

Testing Done

Run at this tree, one file at a time, all passing:

Test file	Passed
`test/connector/test_custom_source.py`	20
`test_bundle_provenance.py`	7
`test_custom_sql.py`	11
`test_filestore_sink.py`	5
`test_holdout_df.py`	12
`test_mlops.py`	4
`test_prediction_completion.py`	14
`test_predictive_query.py`	67
`test_query_from_snapshot.py`	7
`test_semantic_carry_forward.py`	20
`test_snapshot_export.py`	40
`test_snapshot_manifest.py`	22
`test_trainer_registry.py`	27
`test_training_explanations.py`	16
`test_training_table_export.py`	18
`test_warm_start.py`	6

What the new tests cover:

test_mlops.py: root forms (URI, (connector, path), a non-file location raises); export kinds; a call argument beats nested blocks; threads and asyncio tasks; stubs.
test_snapshot_manifest.py:
- a fresh manifest and the version change after appending rows;
- reuse and backfill of a pre-manifest index;
- write failure leaves the snapshot unchanged;
- unsupported files read as None;
- locations for S3 (with percent-encoding), Snowflake, Databricks, BigQuery and Custom SQL; an unrecordable location is null;
- record_sources=False;
- byte-for-byte round trip.
test_training_table_export.py:
- the manifest and sidecar, with byte-identical parts;
- one id for two generations;
- a reloaded export records its source as listing;
- custom tables follow the snapshot's record_sources;
- write and read failures warn;
- no sidecar for CSV or warehouse;
- a block stores each table once.
test_bundle_provenance.py:
- the self-contained manifest and id on result and load;
- round trip;
- a bundle without a manifest loads with the same id and identical predictions;
- a new result per fit;
- stubs and flat warm-start chains;
- a block stores each model once and repairs a damaged copy;
- write failure warns.
test_snapshot_export.py:
- round trip and reuse; a damaged in-place index;
- an interrupted re-export leaves the last export loadable; concurrent exports; interleaved exports of two builds with equal file sizes commit one whole build; a load overtaken by a newer commit loads it; a cached load is reused only for the same export (two builds of one snapshot id into one scratch dir); overlapping marker commits leave one whole marker; an unreadable marker is corrupt on load and re-exported by the store; loading on the exporting machine skips the checksum pass; retry with a fresh client;
- a failed load leaves nothing; the free-space check;
- hostile and malformed markers; unknown formats;
- the manifest carried or reconstructed;
- warehouse connectors refused; the DiskGraph-cannot-open message;
- S3 and GCS through fakes; a filesystem-only sink;
- source_changes cases, including a graph rebound to another directory with equal versions (unknown with and without record_sources), and the load warning;
- a block stores each snapshot once.
test_query_from_snapshot.py:
- predict(snapshot) with and without entities, and with a prediction table;
- the other-graph warning;
- a prediction table together with entities= or prediction_time= raises ValueError, and a snapshot or prediction table given twice raises TypeError;
- two fits on one Trainer: each result scores exactly as its own reloaded bundle, and the two differ;
- an exported snapshot, loaded into a fresh scratch dir with no Graph, scores through TrainingResult.predict exactly as the Graph-bound path did.

Package suites:

kumo-core changed tests: 225 passed, 1 skipped. They include the bundle id ignoring only the manifest, sidecars before _SUCCESS, version_kind on object-store versions, and kept SQL dependency versions.
kumo-snowflake, kumo-databricks and kumo-bigquery changed tests pass; they assert each stamp's version_kind.
isort and flake8 are clean.

The new DiskGraph tests (pybind test_exception_exports.py and the Rust range_checksums_cover_every_listed_file_and_the_manifest) were not part of this run.

The examples above were captured from a real run of this tree on synthetic data: snapshot, two generations, a one-epoch CPU fit, a training-table export and reload, a snapshot export, then a load and predict(snapshot) in a fresh scratch dir, and source_changes after rewriting orders.

Checklist

No breaking changes.
Docs updated (docs/mlops.md).
Unit tests added.

🤖 Generated with Claude Code

Vedaant-J · 2026-10-20T22:12:00Z

Change map

File	What changed	Why
`3rdparty/notices/inventory.json`	Adds `rust:crc32fast@1.5.0`; its notice now also covers `diskgraph-ingest`	`snapshot_range_checksums` uses the `crc32fast` crate
`THIRD-PARTY.txt`	`crc32fast` is also distributed in `diskgraph/bin/ingest`	Notice accuracy for the new dependency
`packages/diskgraph/Cargo.lock`	`crc32fast` added to `diskgraph-ingest`'s dependencies	Lockfile for the new dependency
`packages/diskgraph/diskgraph-ingest/Cargo.toml`	`crc32fast = "1.5"`	Range checksums for snapshot export and load
`packages/diskgraph/diskgraph-ingest/src/snapshot_mirror.rs`	`_MANIFEST.json` reading and validation split out of `verify_snapshot_integrity` into `read_manifest`; new `snapshot_range_checksums(output_dir, range_bytes)`, a CRC32 per range of every listed file and of the manifest, computed with rayon; Rust test	Export and load check content as well as size, and the data-proportional pass runs in Rust
`packages/diskgraph/diskgraph-pybind/build.rs`	Stamps `DISKGRAPH_BUILD_COMMIT` (environment override, else `git rev-parse --short=12 HEAD`, else `unknown`) and reruns when HEAD moves	Manifests record which DiskGraph build wrote an artifact
`packages/diskgraph/diskgraph-pybind/python/diskgraph/__init__.py`	Exports `__version__`, `BUILD_COMMIT` and `snapshot_range_checksums`	Python access to the new extension members
`packages/diskgraph/diskgraph-pybind/python/tests/test_exception_exports.py`	Build identity is exported; range checksums match `zlib.crc32` and a missing index raises `DiskgraphCorruptionError`	Contract of the new extension members
`packages/diskgraph/diskgraph-pybind/src/lib.rs`	`snapshot_range_checksums` pyfunction (GIL released); module constants `__version__` and `BUILD_COMMIT`	Bindings for the above
`packages/diskgraph/third_party/artifact-manifest.json`	`crc32fast` entry and the new `Cargo.lock` hash	License manifest stays in sync with the lockfile
`packages/kumo-bigquery/src/kumo_bigquery/identity.py`	Table stamps are `SourceVersion`s: `timestamp` for tables, `hour_bucket` for views, `unknown` when undated	Manifests record what each version proves
`packages/kumo-bigquery/src/kumo_bigquery/session.py`	`table_location`: fully qualified name in the connector's target	Snapshot lineage records where a BigQuery table lives
`packages/kumo-bigquery/tests/test_identity.py`	Asserts each stamp's `version_kind`	Covers the typed stamps
`packages/kumo-core/src/kumo_core/artifact_export.py`	`export_selected_files(..., sidecars={})` writes name-to-bytes files into a directory destination after the data and before `_SUCCESS`; a failed sidecar write warns	Training-table exports carry `_provenance.json` without changing the parts
`packages/kumo-core/src/kumo_core/artifacts.py`	Bundle id is the SHA-256 of sorted `[relative_path, sha256]` pairs excluding `_provenance.json`, returned in full by `verify_model_bundle`; new `bundle_id_from_manifest` and `add_bundle_file`; the artifact store's atomic write uses the shared helper	Stable model ids that ignore the run-specific manifest; one way to list a file in a bundle
`packages/kumo-core/src/kumo_core/io/filesystem.py`	New `atomic_write_bytes` and `fsync_directory`	One atomic metadata writer for the artifact store, trainer spec and manifests
`packages/kumo-core/src/kumo_core/object_store_sink.py`	`ObjectStoreSinkBackend.write_files(..., sidecars={})` passes sidecars on; `_ArrowExportFiles.write_bytes`; `holds_export` and `holds_bundle`; `write_files` and a bundle `publish_model` to a path skip the upload, with an info log, when the destination already holds a complete copy of the same export or bundle	Sidecar plumbing for object-store and file destinations
`packages/kumo-core/src/kumo_core/sources/identity.py`	`SourceVersion` (a `str` with `kind` and `dependencies`) and `version_kind()`; `hour_version` is `hour_bucket`; Custom SQL versions are `hour_bucket` and keep the dependency versions they fold	Connectors say what a version proves; Custom SQL lineage records its inputs
`packages/kumo-core/src/kumo_core/sources/object_store.py`	Single objects stamp `timestamp`, directories `listing`; `_copies` becomes public `copies` with an optional ceiling	Typed stamps; snapshot export and load reuse the copy-worker sizing
`packages/kumo-core/src/kumo_core/sources/object_store_backend.py`	`get_data_version` returns the reader's version as-is instead of `str(...)`	Keeps the `SourceVersion` kind
`packages/kumo-core/src/kumo_core/sources/object_store_sessions.py`	`table_location`: full `scheme://bucket/key`, percent-encoded, without credentials	Snapshot lineage records full object-store locations
`packages/kumo-core/src/kumo_core/train/_artifact_storage.py`	Uses the shared `atomic_write_bytes`/`fsync_directory`; creates the parent directory first	Removes a duplicate writer
`packages/kumo-core/tests/sources/parquet/test_object_store.py`	Asserts `listing` for directories and `timestamp` for single files	Covers the typed stamps
`packages/kumo-core/tests/sources/unit/test_sql_identity.py`	The Custom SQL version keeps the dependency versions it folds	Covers recorded dependencies
`packages/kumo-core/tests/test_artifact_export.py`	Sidecars land before `_SUCCESS`; a failing sidecar write warns and `_SUCCESS` is still written	Sidecar contract
`packages/kumo-core/tests/test_artifacts.py`	The manifest file is excluded from the bundle id, and the id from the publication manifest equals the verified one	Bundle id contract
`packages/kumo-databricks/src/kumo_databricks/identity.py`	Delta history stamps are `exact`; no history falls back to `hour_bucket`	Typed stamps
`packages/kumo-databricks/src/kumo_databricks/session.py`	`table_location`: fully qualified name	Snapshot lineage for Databricks tables
`packages/kumo-databricks/tests/test_identity.py`	Asserts `exact` and `hour_bucket`	Covers the typed stamps
`packages/kumo-engine/README.md`	Links `docs/mlops.md`	Discoverability
`packages/kumo-engine/docs/mlops.md`	New: turning MLOps on, lineage, training tables, models, moving snapshots, scoring without the graph; path exports reuse a complete identical copy at the destination (snapshot Reuse bullet, Off-by-default bullet); Models: the bundle carries the holdout, so `holdout_df()` works on a loaded or fetched model; 10-28: restructured: quickstart, id table, `store=`, reading artifacts back through the `kumo.MLOps`, Reference section	User docs for the series' base; One short main path; details move to Reference
`packages/kumo-engine/kumo/engine/__init__.py`	Exports `MLOps`, `mlops`, `SnapshotManifest`, `TrainingTableManifest`, `ModelManifest`, `SnapshotOutputConfig`	Public surface
`packages/kumo-engine/kumo/engine/_manifest.py`	New: `Record` (unknown keys kept in `extra`), `Environment`, `SourceTableEntry`, `KINDS`, the `Manifest` envelope, `stub`, `embedded`, `SnapshotManifest`, `TrainingTableManifest`, `ModelManifest` and their parents, and the atomic writer and tolerant reader	One manifest format for every artifact, with embedded parents and stubs
`packages/kumo-engine/kumo/engine/_mlops.py`	New: frozen `MLOps(artifact_root, *, export=True)` (options keyword-only) with root validation, `wants(kind)`, the contextvar `mlops()` block, and `resolve()`; 10-28: `export=` renamed `store=`; `on_root` and `snapshot_path` let an `MLOps` stand in for a connector; a string `@stage` root raises	Opt-in switch with call > block > off precedence; "export" means a copy elsewhere; users stop rebuilding root paths; a stage string was silently local
`packages/kumo-engine/kumo/engine/_provenance.py`	New: `environment()` (engine, build commit, kumo-core, DiskGraph, torch, Python) and `source_table()`, which builds a `SourceTableEntry` with location, version kind, Custom SQL fields and the `record_sources` rule (a redacted location keeps a `location_fingerprint`)	Shared by snapshot and training-table manifests
`packages/kumo-engine/kumo/engine/_snapshot_io.py`	New: snapshot export into a fresh generation directory committed by the `kumo_snapshot.json` marker (renamed into place on local paths), with the export identity recorded in the source index, with best-effort removal of superseded and abandoned generations; `export_identity`/`holds_export` (a cached load is reused only for the same export's bytes); ranged parallel load with checksum and DiskGraph checks, and `store()` under `<root>/snapshots/`; `export` returns the location without uploading when the destination's marker names this snapshot, its generation has `_DONE` and the local index holds its bytes; reuse also requires, from one listing, every index file at its size plus `_DONE`, and a byte-identical `_provenance.json`	Moving an index between machines; the snapshot kind of the artifact root
`packages/kumo-engine/kumo/engine/connector/__init__.py`	`connector_for_root(uri)` maps `s3://`, `gs://`, `file://` or a local path to a connector	URI form of `artifact_root`
`packages/kumo-engine/kumo/engine/connector/file.py`	`table_location`: resolved absolute path	Snapshot lineage for file sources
`packages/kumo-engine/kumo/engine/graph.py`	`Graph.snapshot(*, record_sources=True, mlops=None)` writes or backfills `_provenance.json` and stores to the root; `GraphSnapshot` gains `record_sources`, `manifest`, `export`, `load_from_connector`, `source_changes` and the private `_graph_key`, `_data_version`, `_native_graph`; the snapshot lock and local store become shared helpers; `load_from_connector` reuses a cached index only when it holds the export's bytes and follows a newer generation that commits mid-load; `source_changes` compares redacted sources by `location_fingerprint`; `schema_changes` and `report_other_graph` compare graph configs without connectors and locations: schema changes warn naming them, location-only changes log at info; changing `record_sources` on a reused index keeps unknown `parents` fields; 10-28: `GraphSnapshot.load_from_connector(cfg, id=...)`	Snapshot lineage, portability and staleness check; Load a stored snapshot by id from the root
`packages/kumo-engine/kumo/engine/pquery/_custom_source.py`	`stage_source_table` also returns the version the staging read returned	A loaded table records its source version without an extra call
`packages/kumo-engine/kumo/engine/pquery/prediction_table.py`	Unpacks the new `stage_source_table` return	Signature change
`packages/kumo-engine/kumo/engine/pquery/predictive_query.py`	`graph: Graph \| GraphSnapshot`; private `_from_snapshot`; parsing through `_native_graph`; the snapshot check compares graph keys; `generate_training_table(..., *, mlops=None)` finishes through `_finish`	Scoring from a snapshot without a `Graph`; training-table lineage
`packages/kumo-engine/kumo/engine/pquery/training_table.py`	`TrainingTable.id` (deterministic `tt-` id), `tt_id`, `manifest`; `from_source_table(..., mlops=None)` records `parents.source_table`; `_finish` writes the staging manifest and stores under `<root>/training_tables/<id>/` unless `_SUCCESS` is there; `export` adds the sidecar on Parquet directory file destinations; 10-28: `load_from_connector` reads paths under a `kumo.MLOps` root	Training-table lineage, identity and storage; Same read-back as every artifact
`packages/kumo-engine/kumo/engine/trainer/__init__.py`	Exports `SnapshotOutputConfig`	Public config
`packages/kumo-engine/kumo/engine/trainer/_provenance.py`	New: `write_provenance` (the model manifest, listed with role `provenance`), `embedded_model` (model parents cut to stubs), `parent_model`, `read_provenance`	Model lineage with flat warm-start chains
`packages/kumo-engine/kumo/engine/trainer/_spec.py`	Uses `atomic_write_bytes` and `add_bundle_file`; `PROVENANCE_ROLE`; `listed_file`; `read_bundle_artifact_paths` also returns the verified bundle id and skips the provenance role; `HOLDOUT_ROLE`, `HOLDOUT_META_ROLE`, kept out of the artifact paths	Shared writer; bundle id on load
`packages/kumo-engine/kumo/engine/trainer/output_config.py`	`SnapshotOutputConfig(connector, path)`	Snapshot export config in the style of the other output configs
`packages/kumo-engine/kumo/engine/trainer/trainer.py`	`TrainingResult` gains `bundle_id`, `id`, `manifest`, `predict`; `Trainer.result`; `fit(..., mlops=None)` computes the bundle id, writes the manifest (fit timing, fit mode, embedded parents) and stores to `<root>/models/<id>/`, keeping a complete copy; `predict(snapshot, *, entities=, prediction_time=, ...)` with the old `predict(prediction_table, snapshot=)` form kept; the other-graph warning; `load` sets the bundle id; `predict` refuses a prediction table together with `entities=` or `prediction_time=`, and an argument given twice; `TrainingResult.predict` scores with that result's own bundle; the other-graph warning names schema changes and is an info log for location-only changes; the artifact-root model store checks for a complete copy with `holds_bundle`; `fit` copies the holdout and its truncation flag into the bundle (`_bundle_holdout`) before the bundle id is computed; `Trainer.load`/`load_from_connector` read them back (`_listed_holdout`); 10-28: `TrainingResult.snapshot_id`; `Trainer.load_from_connector(cfg, path=...)`; a fit with an empty test split raises a `ValueError` before training	Model identity, lineage and storage; one scoring call; Training snapshot without five attribute hops; model by path from the root; live run crashed after training with an opaque error
`packages/kumo-engine/setup.py`	`build_py` stamps `BUILD_COMMIT` into `_version.py` (`KUMO_ENGINE_BUILD_COMMIT`, else `git rev-parse`)	Distinguishes dev builds that share a version string
`packages/kumo-engine/test/conftest.py`	Shared two-table `graph` fixture (`users`, `events`)	Used by the snapshot export, training-table and live tests
`packages/kumo-engine/test/connector/test_custom_source.py`	Unpacks the new `stage_source_table` return; prepared-source stubs pass `version=None`	Signature change
`packages/kumo-engine/test/test_bundle_provenance.py`	New, 7 tests: self-contained manifest and id on fit and load; byte-for-byte round trip; a bundle without a manifest loads with the same id and predictions; a new result per fit; stubs and flat warm-start chains; store-once and repair under a block; write failure warns; 10-28: `Trainer.load_from_connector(cfg, path=...)`, `snapshot_id`	Model manifest and root behaviour; Root read-back
`packages/kumo-engine/test/test_custom_sql.py`	Unpacks the new `stage_source_table` return	Signature change
`packages/kumo-engine/test/test_filestore_sink.py`	Publishing an identical bundle to the same path again reuses the copy; a changed `_provenance.json` uploads again	Path exports never duplicate a complete copy
`packages/kumo-engine/test/test_graph_mismatch.py`	New: a location-only graph change logs at info; a dropped edge warns naming it; an older model's warning carries a hint; `graph_config` round-trips through the training spec	Graph-mismatch warning
`packages/kumo-engine/test/test_holdout_df.py`	a loaded model and one fetched from a path registry return the same holdout and truncation; a bundle without the copy raises; 10-28: empty test split fails the fit with a clear message	covers the bundled holdout; Reproduces the live-run crash
`packages/kumo-engine/test/test_mlops.py`	New, 4 tests: root forms and validation, export kinds, call over nested blocks across threads and asyncio tasks, stubs; 10-28: `store=`; a stage string root raises	MLOps switch contract; Rename and stage-root guard
`packages/kumo-engine/test/test_predictive_query.py`	Monkeypatched `stage_source_table` returns `(rows, None)`	Signature change
`packages/kumo-engine/test/test_query_from_snapshot.py`	New, 7 tests: `predict(snapshot)` with `entities` and with a prediction table equals the Graph-bound scores, refuses `entities` with `prediction_time`, a prediction table with either, and an argument given twice; two fits each score with their own model; warns for another graph; an exported snapshot scores in a fresh scratch with no `Graph`	One scoring call
`packages/kumo-engine/test/test_semantic_carry_forward.py`	Reformats a test double; 10-28: fake tasks carry a test split	Formatting only
`packages/kumo-engine/test/test_snapshot_export.py`	New, 40 cases: overlapping marker commits, an unreadable marker, own-machine reuse without a checksum pass, round trip and reuse, damaged index, an interrupted re-export, concurrent exports, interleaved exports of two equal-size builds, a load overtaken by a newer commit, cache reuse only for the same export, a source rebound to another directory, retries, failed loads, the free-space check, hostile and unknown markers, manifest carried or reconstructed, unsupported connectors, S3/GCS fakes, filesystem-only sinks, `source_changes`, the load warning, store-once under a block; a moved graph with the same schema loads with an info log only; re-exporting an identical snapshot uploads nothing and logs it, and a generation without `_DONE` is exported again; a rebuilt copy drops the source's export identity; a re-export after redaction commits the redacted manifest; an export missing a remote index file is replaced and loads into empty scratch; 10-28: `load_from_connector(cfg, id=...)`	Snapshot export and load contract; Load by id from the root
`packages/kumo-engine/test/test_snapshot_export_live.py`	New, env-gated S3/GCS round trip (`KUMO_SNAPSHOT_LIVE_S3_ROOT`, `KUMO_SNAPSHOT_LIVE_GCS_ROOT`)	Manual live check against real object stores
`packages/kumo-engine/test/test_snapshot_manifest.py`	New: a fresh manifest, backfill of a pre-manifest index, write failure, unsupported files, locations per connector, Custom SQL with and without `record_sources`, `record_sources=False`, round trip; toggling `record_sources` keeps an unknown `parents` field	Snapshot manifest contract
`packages/kumo-engine/test/test_trainer_registry.py`	The atomic-write failure is injected in `kumo_core.io.filesystem`; `_finish_fit` returns a result whose bundle id equals the loaded bundle's	Follows the shared writer and the new result
`packages/kumo-engine/test/test_training_table_export.py`	New tests: manifest and sidecar with byte-identical parts and one id per table; byte-for-byte round trip; `record_sources` for custom tables; write and read failures; no sidecar for CSV or warehouse; store-once under a block; a re-export to the same directory reuses it without uploading; 10-28: `TrainingTable.load_from_connector(cfg, ...)`	Training-table manifest contract; Root read-back
`packages/kumo-engine/test/test_warm_start.py`	Asserts the embedded warm-start parent and its cut-off grandparent	Depth rule
`packages/kumo-snowflake/src/kumo_snowflake/identity.py`	`LAST_ALTERED` stamps are `timestamp`; a missing table falls back to `hour_bucket`	Typed stamps
`packages/kumo-snowflake/src/kumo_snowflake/session.py`	`table_location`: `account/"DB"."SCHEMA"."TABLE"`, without the account when it cannot be resolved	Snapshot lineage for Snowflake tables
`packages/kumo-snowflake/tests/test_identity.py`	Asserts `timestamp` and `hour_bucket`	Covers the typed stamps
`packages/kumo-engine/kumo/engine/connector/databricks.py`	10-28: `identity=None` defaults to the ambient `DatabricksIdentity()`	Notebook and profile auth without `from_notebook`
`packages/kumo-engine/test/test_databricks_connector.py`	10-28: a connector without an identity is ambient	Ambient default
`packages/kumo-engine/test/test_training_explanations.py`	10-28: the fake materialized task carries a test split	The empty-test-split check reads its stage counts
`packages/kumo-engine/kumo/engine/graph.py`	10-29: `report_other_graph` warns (`UserWarning`) for a graph that differs only in table connectors or locations, instead of logging at info; the module logger goes	Default logging hid it; the docs promised a warning
`packages/kumo-engine/test/test_graph_mismatch.py`	10-29: the location-only change warns that the schema is the same	Same
`packages/kumo-engine/test/test_snapshot_export.py`	10-29: loading with a moved graph warns (renamed `test_load_warns_about_a_graph_that_moved_with_the_same_schema`)	Same
`packages/kumo-engine/docs/mlops.md`	10-29: a graph that differs only in connectors or locations warns	Same
`packages/kumo-engine/kumo/engine/_mlops.py`	10-29: new `_require_within_root`: a `(connector, path)` root outside an object store connector's `root_dir` raises when `MLOps` is built	Every store failed later, as warnings only
`packages/kumo-engine/test/test_mlops.py`	10-29: new `test_a_connector_root_outside_its_root_dir_fails_when_built`	Covers the check

Requirement: each snapshot, training table and model must carry a self-contained record leading back to its source table versions and engine build. Models need a stable id, snapshots must move between machines, and a block or mlops= must store each artifact under one root. Existing behaviour: artifacts held only engine-internal files, a bundle's digest was discarded, an index was tied to the path it was built at, and scoring needed a Graph. Smallest change: one manifest format (_manifest) written beside each artifact, with parents embedded or stubbed; typed version stamps and table_location from the existing connectors; the bundle id from the publication manifest's existing digests; snapshot export and load over the #112 file stores, with checksums in DiskGraph; an opt-in MLOps setting read once per call; and predict(snapshot) reusing the trained query and plan. Verification: the test files listed in the PR body, all passing at this tree, plus the kumo-core and warehouse plugin suites and a real synthetic run (snapshot, training table, fit, export, then load and score in a fresh scratch dir). Stopping condition: every artifact carries a readable manifest that reaches the source versions; plain calls without MLOps behave as before; the root holds one copy per id; and a stored snapshot plus a stored model score with one call.

copy-pr-bot · 2026-10-20T22:12:03Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

coderabbitai · 2026-10-20T22:12:28Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Last reviewed commit: 75f9e80
Latest commit, not reviewed: 3a85f75

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: d6ee79fd-a832-4873-b67d-fdca81f9f3f8

📥 Commits

Reviewing files that changed from the base of the PR and between 045fff7 and 75f9e80.

📒 Files selected for processing: 4

packages/kumo-engine/kumo/engine/_manifest.py
packages/kumo-engine/kumo/engine/_provenance.py
packages/kumo-engine/kumo/engine/graph.py
packages/kumo-engine/test/test_snapshot_manifest.py

💤 Files with no reviewable changes: 1

packages/kumo-engine/kumo/engine/_manifest.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The diskgraph Python API now exposes supported on-disk format versions. Kumo Engine adds shared manifest and provenance utilities, records source and snapshot metadata for graph snapshots, and uses shared atomic writes for training outputs. Object-store sessions add a canonical table URI method.

Sequence Diagram(s)

sequenceDiagram
  participant GraphSnapshot
  participant source_table
  participant source_location
  participant ObjectStoreSessions
  participant new_manifest
  participant environment
  participant scheduler_context
  participant write_manifest
  participant atomic_write
  GraphSnapshot->>source_table: Build source-table metadata
  source_table->>source_location: Resolve source coordinate
  source_location->>ObjectStoreSessions: Resolve object-store table URI
  GraphSnapshot->>new_manifest: Build snapshot manifest
  new_manifest->>environment: Collect environment metadata
  new_manifest->>scheduler_context: Collect scheduler metadata
  GraphSnapshot->>write_manifest: Persist manifest
  write_manifest->>atomic_write: Write manifest atomically

Priority: Normal

Merge Risk: Moderate · previous review 75f9e

Some snapshots can identify the wrong source file or object, making their lineage unreliable. Correct those provenance paths before merging; also address the suppressed backfill warning.

Pre-merge checks: 2 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Change Map Design Review		The required conversation comment beginning with `<!-- kumo-pr-change-map -->` is not available in the supplied context or repository checkout. The available summary states the change map's broad impl…	Provide the complete PR conversation comment beginning with `<!-- kumo-pr-change-map -->`, including its Change map and linked rationale. Then assess each proposed change against the repository diff and existing behavior.

Passed checks: 2 passed

Check name	Status	Explanation
Pr Size		The pull request changes 11 files. This does not exceed the failure threshold of 40 files.
Public Resource Access		The pull request adds no concrete external URL, repository location, or package source location that is required to build, test, use, or contribute. The only URI-like additions are the schematic `sche…

Full details: Public Resource Access

Explanation

The pull request adds no concrete external URL, repository location, or package source location that is required to build, test, use, or contribute. The only URI-like additions are the schematic scheme://bucket/key docstring and the s3://bucket-a/... local test fixture, both excluded by the check. /opt/ml/config/processingjobconfig.json is a local runtime path, not an external resource reference.

Full details: Change Map Design Review

Explanation

The required conversation comment beginning with  is not available in the supplied context or repository checkout. The available summary states the change map's broad implementation, but it does not provide the required claims about the observed failure, requested release requirement, inspected existing behavior, smallest sufficient change, verification, and stopping condition. Those claims are necessary for this design review.

Resolution

Provide the complete PR conversation comment beginning with , including its Change map and linked rationale. Then assess each proposed change against the repository diff and existing behavior.

✨ Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: 27abe888-d440-4212-bd1a-b59a2746c5b0

📥 Commits

Reviewing files that changed from the base of the PR and between 09bc7e5 and eed1867.

📒 Files selected for processing: 4

packages/kumo-core/src/kumo_core/sources/object_store_sessions.py
packages/kumo-engine/kumo/engine/_provenance.py
packages/kumo-engine/kumo/engine/graph.py
packages/kumo-engine/test/test_snapshot_manifest.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

coderabbitai

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments: 1

packages/kumo-engine/kumo/engine/graph.py: 1
269-269: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Track backfill warnings only after a backfill failure.

If a fresh manifest write fails, Line 269 adds its path to _warned_backfill. If the index is later reused and its backfill also fails, the backfill failure produces no warning. Add the path to this set only when fresh is false.
Proposed change
         if fresh or path not in _warned_backfill:
-            _warned_backfill.add(path)
+            if not fresh:
+                _warned_backfill.add(path)
             warnings.warn(

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
Review profile: QUIET
Plan: Enterprise
Run ID: fec48a97-da42-43cd-9e74-6708ef97c388

📥 Commits

Reviewing files that changed from the base of the PR and between eed1867 and 045fff7.

📒 Files selected for processing: 4

packages/kumo-engine/kumo/engine/_manifest.py
packages/kumo-engine/kumo/engine/_provenance.py
packages/kumo-engine/kumo/engine/graph.py
packages/kumo-engine/test/test_snapshot_manifest.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

aaron-kumo

The separation between snapshots, model bundles, and explicit scoring inputs looks sound. My main concern is preserving enough information now for future restoration, freshness, and compatibility checks. We can defer those policies, but information omitted from today’s artifacts will be difficult to recover later.

P2 — Record the DiskGraph producer version/build.

environment() records engine, core, Torch, and Python versions, but omits the DiskGraph package that actually writes the index. Format constants identify the declared storage formats, but cannot distinguish implementations with different behavior or fixes under the same format version. Please record the installed DiskGraph version and a build identifier when available. This should be diagnostic provenance; package-version differences should not automatically imply incompatibility.

Discussion — Should the artifact retain the Custom SQL definition?

Could we retain literal SQL alongside its fingerprint, with an explicit option to omit/redact it for distribution? The product already persists literal SQL in table definitions and exposes it through SDK objects and generated scripts. Hash-only storage introduces a more restrictive policy here and loses the ability to inspect transformations, explain changes, or reconstruct the source definition.

SQL can contain sensitive literals or business logic, so an explicit retention policy makes sense. Is there a specific sharing or privacy requirement driving unconditional omission? If not, I’d favor retaining the SQL, dialect, resolution namespace, and referenced tables. The exact-text fingerprint and normalized-query fingerprint should also have distinct purposes; the existing Custom SQL identity already uses normalization.

Discussion — Can source identity and version evidence be structured before publication?

Could we preserve a credential-free source descriptor and typed version evidence rather than only {connector, location, version}? For example, a Snowflake table coordinate needs account context, and an actual source version, a metadata timestamp, and an hour-bucket fallback have different guarantees.

For Custom SQL, the version calculation already gathers dependency versions and folds in a staleness bucket. Could those components be retained alongside the resulting hash? Otherwise future freshness checks cannot explain whether the query, a dependency, or only the bucket changed.

I’m not suggesting implementing every comparison now. A small versioned representation that retains the evidence—including observation time, fallback status, and observed versus reconstructed provenance—would leave those decisions open.

Discussion — Should publication offer a metadata-completeness requirement?

Best-effort manifest writing seems appropriate for preserving existing local snapshot() behavior. Should export/publication optionally require complete provenance, though? Otherwise a newly created snapshot whose manifest write failed can become a durable artifact without the evidence needed for future compatibility checks. Legacy artifacts could remain supported with an explicit incomplete-provenance state.

Vedaant-J · 2026-10-21T21:21:07Z

P2 — Record the DiskGraph producer version/build.
Done in 2552bc1: environment now records diskgraph (diskgraph.__version__) and diskgraph_build_commit (diskgraph.BUILD_COMMIT, stamped from git at build time by diskgraph-pybind/build.rs, "unknown" when built without git). It is diagnostic only; a different DiskGraph build never makes an index unusable.

Discussion — Should the artifact retain the Custom SQL definition?
We'll retain the Custom SQL text by default (with its dialect, next to the hash and dependency versions), the same as exact source locations (0d73b76). graph.snapshot(record_sources=False) omits both, recording only the connector and table name and a hash of the SQL, for deployments that can't carry source details in artifacts that travel (exports, bundles). The training-table manifest (#55) and bundle provenance (#56) follow the snapshot's choice; the choice now travels on GraphSnapshot.record_sources (72e2932), and a custom training table whose snapshot's choice is unknown records the table name only (#55, c5baa35).

Your two remaining points, both in 60e1ab6:

Resolution namespace: the Custom SQL source record now has namespace, the context unqualified table names in the query resolve in: [database, schema] (Snowflake), [catalog, schema] (Databricks) or [project, dataset] (BigQuery), read from the connector with the same attributes the Custom SQL identity uses (CUSTOM_SQL_NAMESPACE_ATTRS). It is source detail, so it is omitted with record_sources=False, like sql and dialect.
Normalized-SQL fingerprint: normalized_sql_fingerprint sits next to the exact-text hash (location = sql:<sha256>). It reuses the existing Custom SQL identity normalisation (custom_sql_synthetic_name, the name hosted and the SDK give the query's table), with no new normaliser, so whitespace, comment and keyword-case edits leave it unchanged while the exact-text hash changes; comparing the two tells a reformatting from a real query change. It is a hash, so it is kept with record_sources=False.

Discussion — Can source identity and version evidence be structured before publication?
Yes, in 8b16cd1. Each source-table entry now has version_kind (exact, timestamp, hour_bucket, unknown), set by the connector that produced the version, and source_changes (#54) uses it instead of guessing from hour-aligned values. Custom SQL keeps the dependency table versions it folds into its hash (dependencies), so a later check can tell a dependency change from a bucket rollover. Snowflake locations now include the account (<account>/"DB"."SCHEMA"."TABLE", no credentials).

Discussion — Should publication offer a metadata-completeness requirement?
Not as a requirement for now, so export never fails where snapshots work today. Instead, export (#54, 3b1d2bd) writes a manifest marked "reconstructed": true (empty source_tables, null build fields) when a snapshot has none, so missing provenance is stated explicitly in the artifact.

CodeRabbit: object-store locations keep their percent-encoding, a FileConnector table records its resolved file (orders.parquet), and a failed fresh manifest write no longer suppresses a later backfill warning (8b16cd1).

aaron-kumo

Approving with three P2 suggestions below covering provenance redaction, remote export completeness, and preservation of unknown metadata.

Validation: source review and isolated checks with dependency stubs; full integration suites were not rerun.

aaron-kumo · 2026-10-27T19:49:25Z

+        filesystem, _ = _location(connector, location)
+        return (manifest.get('snapshot_id') == snapshot_id
+                and filesystem.get_file_info(f'{root}/{_DONE}').type
+                is FileType.File and holds_export(index, manifest))


P2 — Include provenance changes in snapshot export reuse

_holds_copy() compares index identity but ignores _provenance.json. After exporting a snapshot with full provenance, calling graph.snapshot(record_sources=False) and exporting to the same destination reuses the previous generation. The operation succeeds, but the published artifact still contains the original SQL and source locations.

A customer could share that artifact believing those details were removed. SQL can contain sensitive literals or business logic, so redaction must affect publication even when the index data is unchanged.

I suggest comparing provenance bytes as well as index identity before reuse, and publishing a new generation when either differs. A regression could export with full provenance, re-export with record_sources=False, and verify the committed generation contains the redacted manifest.

Fixed in 8b6b2ca: reuse now also requires the destination generation's _provenance.json to equal the local one byte for byte, so a re-export after record_sources=False commits a new generation with the redacted manifest (test: test_reexport_after_redaction_publishes_the_redacted_manifest).

aaron-kumo · 2026-10-27T19:49:26Z

+        root, manifest = read(connector, location)
+        filesystem, _ = _location(connector, location)
+        return (manifest.get('snapshot_id') == snapshot_id
+                and filesystem.get_file_info(f'{root}/{_DONE}').type


P2 — Verify remote completeness before skipping snapshot export

A generation’s _DONE marker and matching local index identity do not establish that its remote files are still present. If an exported index file is deleted while the markers remain, _holds_copy() returns true and re-export skips the upload.

This prevents a healthy local snapshot from repairing an incomplete remote copy. The customer sees export succeed, may discard local scratch, and later discovers that a fresh scoring machine cannot restore the snapshot. Rebuilding may be expensive, and exact historical reproduction may be unavailable if the sources have changed.

I suggest verifying the remote inventory before treating the export as complete—at minimum, ensuring every listed index file exists with its expected size—and publishing a replacement generation when incomplete. A regression could delete a remote index file without changing the markers, re-export, and successfully load into empty scratch.

Fixed in 8b6b2ca: before skipping, one recursive listing of the generation must show _DONE, index/_MANIFEST.json and every index file the local _MANIFEST.json lists at its size; otherwise a replacement generation is published (test: test_reexport_replaces_an_export_missing_a_remote_file, which then loads into empty scratch).

aaron-kumo · 2026-10-27T19:49:26Z

+                return
+            manifest = replace(
+                existing, record_sources=record_sources,
+                parents=SnapshotParents(source_tables=source_tables))


P2 — Preserve unknown parent metadata when rewriting source records

Changing record_sources constructs a fresh SnapshotParents, discarding unknown fields retained in existing.parents.extra. This also happens when switching from redacted to full source recording, so the loss is not limited to removing sensitive source details.

In a mixed-version workflow, a newer engine may have written metadata that this reader does not understand. Dropping it during an otherwise routine rewrite undermines the preservation contract and could remove information needed by later lineage, compatibility, or migration checks.

I suggest updating the existing parent record with replace(existing.parents, source_tables=...), handling absent parents separately, so unrelated unknown fields survive. A regression could inject an unknown parent field, change record_sources=False to True, and verify the field remains.

Fixed in 8b6b2ca: a reused index now gets replace(existing.parents, source_tables=...), so unknown parents fields survive; a manifest without parents still gets a fresh SnapshotParents (test: test_toggling_record_sources_keeps_unknown_parent_fields).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kumo.MLOps stands in for a connector in load_from_connector: snapshots by id, training tables and models by their path under the artifact root. MLOps(store=...) replaces export=. TrainingResult.snapshot_id names the training snapshot. DatabricksConnector without an identity is ambient. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…its connector A query window that leaves no test rows raised an opaque evaluation error after training; fit now refuses before training and says to shorten the window or add data. kumo.MLOps('@DB.SCHEMA.STAGE/...') was read as a local directory; it now asks for the (SnowflakeConnector, path) pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The empty-test-split check reads the materialized task's stage counts; the fakes now carry one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Scoring tables read from another location (a dev graph scored on prod tables, or a renamed copy) changes the graph id, but predict and GraphSnapshot.load_from_connector only logged that at info level, so with default logging nobody saw it. It is now a UserWarning like the other graph-mismatch cases, saying the schema is the same. Verified: test_graph_mismatch, test_snapshot_export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MLOps((S3Connector(root_dir='s3://b/data'), 's3://b/artifacts')) was accepted, then every store failed with "Output path is outside the selected connector target". Under on_error='warn' those were warnings, so the fit returned with nothing stored and no model registered. kumo.MLOps now checks a (connector, path) root against the connector's root_dir when it is built and raises, naming the root_dir and the connector to pass instead. Verified: test_mlops. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 20, 2026 22:12 — with GitHub Actions Inactive

github-actions Bot assigned Vedaant-J Oct 20, 2026

This was referenced Oct 20, 2026

Export and load graph snapshots through any file destination #54

Closed

Record training-table lineage across export and reload #55

Closed

Vedaant-J added this pull request to stack #62 October 20, 2026 22:16

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 20, 2026 23:04 — with GitHub Actions Inactive

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 21, 2026 07:57 — with GitHub Actions Inactive

github-actions Bot added the ml-algorithms label Oct 21, 2026

coderabbitai Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread packages/kumo-core/src/kumo_core/sources/object_store_sessions.py Outdated

Comment thread packages/kumo-engine/kumo/engine/_provenance.py Outdated

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 21, 2026 08:47 — with GitHub Actions Inactive

coderabbitai Bot reviewed Oct 21, 2026

View reviewed changes

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 21, 2026 09:58 — with GitHub Actions Inactive

Vedaant-J mentioned this pull request Oct 21, 2026

Score a snapshot with Trainer.predict(snapshot) and TrainingResult.predict #51

Closed

3 tasks done

Vedaant-J force-pushed the vedaant/mlops-snapshot-manifest branch from 75f9e80 to a748d54 Compare October 21, 2026 18:55

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 21, 2026 18:55 — with GitHub Actions Inactive

aaron-kumo reviewed Oct 21, 2026

View reviewed changes

Vedaant-J mentioned this pull request Oct 21, 2026

Add kumo.job() for scheduled job scripts #70

Closed

3 tasks done

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 21, 2026 21:06 — with GitHub Actions Inactive

Vedaant-J force-pushed the vedaant/mlops-snapshot-manifest branch from c354971 to 71fcb43 Compare October 21, 2026 22:12

Vedaant-J had a problem deploying to gha-kumo-workflow-tests October 21, 2026 22:12 — with GitHub Actions Error

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 21, 2026 22:16 — with GitHub Actions Inactive

Vedaant-J mentioned this pull request Oct 21, 2026

Record fits in experiment tracking and publish models to registries #60

Open

3 tasks done

Vedaant-J mentioned this pull request Oct 24, 2026

Log prediction runs to MLflow and Snowflake through the tracking adapters #73

Closed

2 tasks done

Vedaant-J added this pull request to stack #113 October 24, 2026 19:06

Vedaant-J force-pushed the vedaant/mlops-snapshot-manifest branch from 27996c2 to f590d71 Compare October 26, 2026 18:23

Vedaant-J changed the title ~~Record a snapshot manifest next to every graph index~~ Record provenance for every artifact under one MLOps artifact root Oct 26, 2026

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 26, 2026 18:24 — with GitHub Actions Inactive

Vedaant-J removed this pull request from stack #113 October 26, 2026 18:25

This was referenced Oct 26, 2026

Record prediction runs and monitor score drift #64

Open

Hold MLOps artifacts on Unity Catalog volumes through the Files API #114

Open

Hold MLOps artifacts in Snowflake stages and score registry versions with run_batch #115

Draft

Vedaant-J added this pull request to stack #118 October 26, 2026 18:34

Vedaant-J force-pushed the vedaant/mlops-snapshot-manifest branch from f590d71 to 7349aad Compare October 26, 2026 18:50

Vedaant-J temporarily deployed to gha-kumo-workflow-tests October 26, 2026 18:50 — with GitHub Actions Inactive

Vedaant-J mentioned this pull request Oct 26, 2026

Remove hosted-SDK references from kumo-engine user-facing text #120

Closed

Vedaant-J force-pushed the vedaant/mlops-snapshot-manifest branch from 7349aad to 4fbe479 Compare October 26, 2026 21:39

Vedaant-J deployed to gha-kumo-workflow-tests October 26, 2026 21:39 — with GitHub Actions Active

Vedaant-J force-pushed the vedaant/mlops-snapshot-manifest branch from 4fbe479 to 59ea860 Compare October 27, 2026 07:21

Vedaant-J had a problem deploying to gha-kumo-workflow-tests October 27, 2026 07:22 — with GitHub Actions Error

Vedaant-J force-pushed the vedaant/mlops-snapshot-manifest branch from 59ea860 to f62c4ec Compare October 27, 2026 07:30

Vedaant-J mentioned this pull request Oct 27, 2026

Let any sink hold files: FileStoreSink protocol and a chunked filesystem #112

Open

1 of 2 tasks

Vedaant-J deployed to gha-kumo-workflow-tests October 27, 2026 07:31 — with GitHub Actions Active

aaron-kumo previously approved these changes Oct 27, 2026

View reviewed changes

Vedaant-J mentioned this pull request Oct 27, 2026

Return the published model's holdout, independent of explain #53

Open

2 tasks done

Record provenance for every artifact under one MLOps artifact root

29f4354

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aaron-kumo previously approved these changes Oct 28, 2026

View reviewed changes

Vedaant-J and others added 5 commits October 28, 2026 20:39

Give the fit test fakes a test split

ef3a72d

The empty-test-split check reads the materialized task's stage counts; the fakes now carry one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Conversation

Vedaant-J commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update (10-29, recheck): an artifact root outside its connector fails when built

Update (10-29, usability pass 2): same-schema graph changes warn

Update (10-28): UX simplifications and a live-run fix

Summary

The MLOps switch

Manifests

Snapshot manifest

Training-table manifest

Model manifest and bundle id

Moving snapshots between machines

Scoring from a snapshot

Shortcut

Boundaries

Back-compat

Known limitations

Testing Done

Checklist

Uh oh!

Vedaant-J commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change map

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

aaron-kumo left a comment

Choose a reason for hiding this comment

Uh oh!

Vedaant-J commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaron-kumo left a comment

Choose a reason for hiding this comment

Uh oh!

aaron-kumo Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

Vedaant-J Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

aaron-kumo Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

Vedaant-J Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

aaron-kumo Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

Vedaant-J Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Vedaant-J commented Oct 20, 2026 •

edited

Loading

Vedaant-J commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

Vedaant-J commented Oct 21, 2026 •

edited

Loading