[Vedaant-J]

Series: kumo-engine MLOps series: every snapshot, training table, model and prediction run carries a self-contained _provenance.json; with kumo.mlops(...) stores each artifact under one artifact root, tracks fits and prediction runs in MLflow or Snowflake and registers models; volumes and stages can hold files; prediction runs are monitored for drift and realized performance.
Stack: Snapshot portability: #54 → #51
Merge order: one linear stack; merge bottom to top (#53 first)
This PR: Adds snapshot export and load: GraphSnapshot.export(SnapshotOutputConfig(...)) copies a built graph index to any file destination (S3, GCS, a file path, or a volume or stage once those sinks hold files, #112), and GraphSnapshot.load_from_connector() copies it back onto any machine as a usable snapshot.

Summary

This PR adds snapshot export and load: GraphSnapshot.export(SnapshotOutputConfig(...)) copies a built graph index to any file destination (S3, GCS, a file path, or a volume or stage once those sinks hold files, #112), and GraphSnapshot.load_from_connector() copies it back onto any machine as a usable snapshot. The export carries #50's snapshot manifest with the index. With MLOps on (#50), each snapshot is stored under the artifact root when built.

Problem. Teams running Kumo as scheduled jobs need to answer questions like these:

• Our job runs on ephemeral compute (Databricks job clusters, SageMaker jobs, Snowflake ML Jobs) that starts with an empty disk. Why does every run re-read every source table and rebuild the index?
• Can we build the index once and train or score on a different machine?
• Can we reproduce last month's scores on the exact snapshot they were computed from, after the source tables have moved on?
• Is the snapshot we are about to score on still current with its source tables?

Today none of these are possible: an index can only be used on the machine that built it.

Cause. Graph.snapshot() writes the index under the local scratch directory and has no way to move it:

<scratch>/snapshots/<graph_id>/<data_version>/
  _SUCCESS                            # ingest marker; records this absolute index path
  _provenance.json              # from #50
  _MANIFEST.json                      # DiskGraph file list with sizes
  diskgraph_config.json  ingest_stats.json  col_stats.json  edge_stats.json  index_metadata.json
  kumo_col_stats/{col_stats_full,col_stats_min}/<table>.json
  nodes/<table>/{node_data.ctx, pk_data-00000.bin, pk_hash-00000.bin, pk_hash-00000.meta.json, shard-00000/...}

When the machine goes away, so does the index. Copying the directory by hand does not help either: _SUCCESS records the absolute path it was built at, so snapshot() treats a moved index as missing and rebuilds it from the sources.

Change. Two new calls on GraphSnapshot:

location = graph.snapshot().export(SnapshotOutputConfig(connector, 's3://<bucket>/snapshots'))
# On another machine, with no index and no source access needed:
snapshot = GraphSnapshot.load_from_connector(connector, path=location)

export writes to <path>/<graph_id>/<data_version[:16]>/, which can be derived from snapshot.id, and returns that location. Only files listed in DiskGraph's _MANIFEST.json are copied into index/, plus #50's _provenance.json (the series' one manifest name) next to it. Machine-local files (_SUCCESS, generated training tables) stay behind:

<path>/<graph_id>/<dv16>/
  index/                              # new: the index files, same relative paths
    _MANIFEST.json  diskgraph_config.json  ingest_stats.json  col_stats.json  edge_stats.json  index_metadata.json
    kumo_col_stats/...
    nodes/<table>/...
  _provenance.json              # new: #50's manifest, byte for byte
  kumo_snapshot.json                  # new: written last; its presence means the export is complete

The list of files to copy is index/_MANIFEST.json, DiskGraph's own file list; there is no second list.

load_from_connector puts the index in the local snapshot store, at the same location Graph.snapshot() would have built it:

<scratch>/snapshots/<graph_id>/<data_version>/
  _SUCCESS                            # rewritten for this machine's path
  _provenance.json              # #50 manifest, byte-identical to the export
  _MANIFEST.json  diskgraph_config.json  ...  nodes/...   # byte-identical to the export

So a later graph.snapshot() on the loading machine reuses the loaded index instead of re-ingesting, and a second load_from_connector reuses the local copy if it passes DiskGraph's verify_snapshot_integrity. If it fails that check, the copy is downloaded again.

kumo_snapshot.json from an export of a two-table synthetic graph (users 50 rows, events 500 rows) through a FileConnector (hashes shortened, graph_config and the checksum map trimmed):

{
  "format": 1,
  "snapshot_id": "def5ae30…@4699bb2f72c3a16f",
  "graph_key": "def5ae30…",
  "data_version": "4699bb2f…(full 64 hex)",
  "index_manifest": "724c0a28…",
  "graph_config": {"name": "def5ae30…", "tables": [{"table_name": "users", …}, {"table_name": "events", …}]},
  "environment": {"kumo_engine": "1.0.0.dev0", "build_commit": "447de176", "kumo_core": "0.1.0.dev1",
                  "diskgraph": "0.1.0", "diskgraph_build_commit": "023fa815cd62", "torch": "2.12.0+cu130", "python": "3.14.7"},
  "range_bytes": 67108864,
  "crc32": {
    "_MANIFEST.json": [371284983],
    "col_stats.json": [10832008],
    "diskgraph_config.json": [553928727],
    …
  }
}

What each field records:

• format is the export layout version (1). Load rejects any other value with a ValueError naming the format and saying to load it with the kumo-engine that exported it. Unknown extra fields are ignored, so a later engine can add optional fields within format 1.
• snapshot_id, graph_key, data_version name the snapshot and where it goes in the local store. Load refuses ids that are not the 64-hex pairs this package produces.
• index_manifest is the SHA-256 of the uploaded index/_MANIFEST.json. Load checks it before trusting that file's list of names and sizes. It is not a build identity and nothing is reused on it.
• graph_config is the portable graph config, so the loaded GraphSnapshot is complete without the source Graph.
• environment is the engine that ran the export.
• range_bytes and crc32 hold one CRC32 per 64 MiB range of every index file (including _MANIFEST.json). DiskGraph computes them (snapshot_range_checksums) from the local index while the files upload. Load downloads ranges in parallel, then recomputes and compares them.

_provenance.json travels as a file, byte for byte: export copies the local file, and load writes the downloaded bytes into the loaded index unchanged. Because the engine never parses it on the way through, a manifest with a newer schema_version than the exporting or loading engine understands is preserved rather than dropped. If the snapshot has no manifest (an index built before #50), export writes one with "reconstructed": true, empty source_tables and null created_at/environment, so the export says explicitly that its provenance is incomplete.

Transfer:

• Files move in parallel through kumo-core's parallel runner, with retries that renew credentials. Bytes move through the Arrow S3/GCS clients, not DiskGraph's own transfer, which cannot take the refreshing credentials of a connector.
• Export commit protocol: delete kumo_snapshot.json at the location (so it is not loadable), delete everything else there, delete kumo_snapshot.json again (another export may have committed during the wipe), upload every index file, then write index/_MANIFEST.json, _provenance.json and finally kumo_snapshot.json, which carries the checksums, in one write. After committing, export lists the index: if another export to the same location removed files in the meantime, it uploads them again and rewrites the three files, so a kumo_snapshot.json never stays over missing files; if its marker is gone, a later export has taken the location over and decides the outcome. Every export uploads every file; there is no resume and no "already exported" shortcut. An interrupted export is never loadable, and re-running it starts from scratch. Exporting over a completed export replaces it the same way.
• If another export keeps replacing the location after three repair rounds, export gives up with ConcurrentExportError (a RuntimeError), with the message: run GraphSnapshot.export again once the other export finishes.
• To keep several builds of one snapshot (for example after an engine upgrade, for rollback or exact replay), export them to different paths; which path holds which build is the caller's choice.
• Load stages into <target>.loading and checks there is free space for the full size, because any old copy stays until the rename, and pre-allocates the files. It then runs DiskGraph's verify_snapshot_integrity and the checksum pass, checks that DiskGraph can open the index, and only then renames into place. A failed load leaves nothing behind. A missing kumo_snapshot.json says to re-run export; a short file, a checksum mismatch or a malformed kumo_snapshot.json raises one ValueError: the export at <path> is corrupt (with the detail), delete it and run GraphSnapshot.export again.
• A destination its connector's sink does not hold as files (FileStoreSink.holds_files, Let any sink hold files: FileStoreSink protocol and a chunked filesystem #112: for example a warehouse table name) raises one NotImplementedError: <Connector> cannot hold a snapshot export: export needs a file destination (artifact root), such as an S3Connector, GCSConnector or a FileConnector on a volume.

Routes on Databricks and Snowflake compute, until their sinks hold files (#114 for Unity Catalog volumes, #115 for Snowflake stages):

• Databricks or Snowflake compute: a FileConnector on a mounted volume, e.g. FileConnector(root_dir="/Volumes/<catalog>/<schema>/<volume>") on a Unity Catalog Volume, or a stage volume mounted into a Snowflake service (not yet tested with multi-GB indexes).
• Otherwise: the S3 or GCS connector on the bucket behind the volume or external stage.

Artifact root. Graph.snapshot(mlops=...), or a kumo.mlops block (#50), also exports the snapshot under <root>/snapshots/. A complete export of the same format and id already there is reused rather than uploaded again; a failure warns. With MLOps off nothing is stored.

Is a snapshot still current? snapshot.source_changes(graph) compares, per table alias, the version the snapshot recorded (parents.source_tables.<alias>.version in the manifest) with the version the source reports now, provided the table still reads from the recorded connector and location. It calls the connector's get_data_version, which is the same stamp Graph.snapshot() records and is read from metadata only: a catalog query on a warehouse, a listing on an object store. It never stages or reads rows and never raises. Output from the loaded snapshot above, in the loading process, after events.parquet was rewritten with 520 rows:

{"users":  {"status": "unchanged", "recorded": "1792609776", "current": "1792609776"},
 "events": {"status": "changed",   "recorded": "1792609776", "current": "1792609788"}}

unknown means there is nothing reliable to compare:

• the recorded or current version_kind (Record provenance for every artifact under one MLOps artifact root #50) is hour_bucket or unknown (exact, timestamp and a directory's listing are compared). Views and sources without a readable version are stamped with the current hour, so they always get unknown; a real modification time that happens to fall on the hour is compared normally;
• the table is Custom SQL (its version folds in a 24-hour window);
• the alias is missing from the manifest, or the manifest is absent or reconstructed;
• the table now reads from a different connector type or location than the one recorded (compared with the snapshot's record_sources choice, so a location-free manifest still compares table names);
• the current version could not be read.

load_from_connector(..., graph=graph) makes the same metadata call for every table, so the loading machine needs source credentials. If any table is changed, it issues one UserWarning:

Snapshot at snapshots/219d0003…/874e6a5fb23f842d predates source table(s) events; see `snapshot.source_changes`.

Nothing is refused. The rule for scoring jobs, in the docs: either build a fresh snapshot, or load one whose source_changes are all unchanged. Loading an older snapshot on purpose, for example to reproduce past scores, is legitimate.

Supporting changes: export and load reach the destination through kumo.engine.connector.file_store (#112), and every upload passes its size to open_sized, so a chunked store keeps files up to one part as plain objects. kumo-core's object-store copies worker sizing becomes public, so export and load reuse it. DiskGraph gains snapshot_range_checksums(index_path, range_bytes), next to verify_snapshot_integrity and sharing its manifest reader; it checksums in Rust with rayon. GraphSnapshot gains the private _graph_key and _data_version, read from the snapshot id.

Back-compat

This PR only adds two calls and records what it copies. It adds no checks or errors on any existing path.

Situation	Behaviour
Graph.snapshot() without export or load	Unchanged, including reuse of an existing (or damaged) in-place index. The snapshot lock moves into a shared helper.
Index moved by hand to another scratch dir	Rebuilt as before. Only load_from_connector rewrites _SUCCESS for the new location.
Index built before #50, exported	The export carries a _provenance.json marked "reconstructed": true with empty source_tables; the loaded copy has that manifest.
Snapshot manifest with a newer schema_version, exported or re-exported	Copied byte for byte, both ways. This engine reads it as None (with #50's warning) but never drops it.
Location already holds an export (any build, any snapshot) or other data	Replaced: the marker and the old contents are deleted first, then everything is uploaded.
A load runs while the same location is being re-exported	The load can fail (missing marker, missing file or checksum mismatch); retry it after the export finishes.
Loaded with a graph= whose id differs from the exported graph	Warning. The snapshot still loads.
load_from_connector without graph	Unchanged; source_changes is not called.
load_from_connector with graph	One metadata call per source table; may issue one warning naming changed tables. The snapshot still loads.
Export written with an unknown format	Load raises ValueError naming the format. Unknown optional fields in format 1 are ignored. A malformed kumo_snapshot.json raises the corrupt-export ValueError (delete and re-export).
Export built by a DiskGraph whose layout this one cannot open	Load fails with the error DiskGraph raises on opening, plus the kumo-engine version that exported it. Versions are not compared up front.
Missing, truncated or corrupted file in the export	Load fails its size or checksum check, says to delete the export and run GraphSnapshot.export again, and leaves no partial copy.
Source entry recorded with a null connector or location (it could not be determined at build time)	source_changes reports that table as unknown.

Known limitations

• Export is atomic per location, not isolated: a load that overlaps a re-export of the same location can fail and should be retried, and two exports racing to one location leave whichever committed last. If the last export to start dies, the location has no kumo_snapshot.json and re-running export repairs it.
• Every export uploads the whole index, even when an identical export is already there.
• Checksums detect corruption, not tampering. Load only from paths whose writers you trust.
• source_changes reports Custom SQL tables as unknown even though their dependency versions are recorded (Record provenance for every artifact under one MLOps artifact root #50); comparing those is left for later.
• A file-connector table that is a folder with Parquet or CSV files under a _- or .-prefixed path (for example a Delta _delta_log checkpoint) always reports changed: kumo-core's metadata stamp lists those files, while the stamp taken during the read skips them.
• GCS was not run live (see below).

Live S3 run

Synthetic data, exported to a dev S3 bucket, loaded into an empty scratch dir in a new process. This run predates the later changes on this branch (DiskGraph checksums, the atomic export protocol and layout, _provenance.json carried as a file, source_changes on version_kind, and the shortened error messages). It was not re-run live after them.

Store	Result	Timings	Size
S3	PASS. All 33 index files are sha256-identical after export and load. graph.snapshot() on the loading side reused the loaded index (0 ingest calls). Predict on the loaded snapshot equals the original (200 rows). A second load reused the local copy (0 downloads). source_changes reported all unchanged, then orders as changed after its parquet was rewritten (and load_from_connector(graph=...) warned), then unchanged after the file was restored.	build 1.1 s, export 0.59 s, re-export 0.09 s, load 0.67 s, second load 0.09 s	33 files, 493 KB
GCS	Not run: the repo's GCS test credential is CI-only.

Performance

Checksumming moved from Python (64 threads, zlib) to DiskGraph (Rust, rayon, 1 MiB buffers). On a 13 GB, 350-file index:

Step	Before (Python)	After (DiskGraph)
Per-range CRC32 pass over the whole index	1.98 s	0.55 s
Local FileConnector export	about 1.3 s; user CPU 13.1 s	about 1.3 s; user CPU 9.3 s
Local load	about 12 s, disk-bound	unchanged within noise; peak RSS +65 MB

Testing Done

• test_snapshot_export.py (30 cases) covers:
  • export through a sink that only offers a chunked filesystem (no object store): the export is split into parts and the index loads byte for byte; a table name raises;
  • a kumo.mlops block stores a snapshot once (a second snapshot() does not export again) and a block that does not export snapshots stores nothing;
  • the round trip: relocated to a new scratch dir, byte-identical index files, _provenance.json byte-identical after export, load and re-export of the loaded snapshot, and reuse by a second load and by graph.snapshot() with no ingest;
  • a newer-schema snapshot manifest carried through unchanged, and a reconstructed manifest exported for a snapshot without one;
  • a damaged in-place index still reused by Graph.snapshot() but downloaded again by load;
  • an interrupted re-export over a completed one is not loadable, and re-running it starts over and removes stray files; re-export replaces corrupted files, other data and another snapshot's marker; two concurrent exports leave a loadable export; files wiped by a concurrent export that then dies, just before the commit, are uploaded again and the export loads; a failed upload retried with a fresh client; an export whose files another export keeps wiping raises ConcurrentExportError;
  • a failed load leaving nothing behind; the free-space check before any download;
  • hostile and malformed manifests refused without touching other paths; an unknown format rejected, unknown fields accepted, and the other-graph warning;
  • unsupported connectors; the message when the installed DiskGraph cannot open the index;
  • S3 and GCS through fakes;
  • source_changes (with and without record_sources): unchanged including a timestamp that falls on the hour, changed after a source parquet is rewritten, a Parquet directory (listing version) unchanged then changed after a part is added, unknown for hour_bucket versions, a version read error, and an absent or reconstructed manifest, with staging patched to fail throughout;
  • the stale-load warning fires only with graph= and names only the changed table.
• Run on the current head: test_snapshot_export.py, test_snapshot_manifest.py, test_snapshot_export_live.py, test_snapshot_cleanup.py (59 passed, 2 skipped live); kumo-s3 tests/ pass; DiskGraph test_exception_exports.py (3 passed); cargo clippy --workspace --all-targets -D warnings clean. cargo test -p diskgraph-ingest snapshot_mirror was run by the implementer on the commit that added snapshot_range_checksums and not re-run.
• CodeRabbit items: source_changes reports unknown when the connector type or location differs.
• The kumo_snapshot.json example was captured from a real export on the current head (engine and kumo-core versions shown as released-style values); the source_changes example is from the earlier run.
• Live S3/GCS round-trip tests are env-gated. S3 was run by hand (above, on the earlier layout); GCS was not.

Checklist

• No breaking changes.
• Docs updated (docs/mlops.md).
• Unit tests added.

🤖 Generated with Claude Code

[Vedaant-J]

Change map

File	Function / scope	What changed	Why
3rdparty/notices/inventory.json	crc32fast	crc32fast is now also shipped in diskgraph/bin/ingest	The ingest binary checksums exported snapshot files with crc32fast
THIRD-PARTY.txt	crc32fast 1.5.0	Lists diskgraph/bin/ingest as a place crc32fast is distributed	Regenerated notice for the new direct dependency
packages/diskgraph/Cargo.lock	dependency list	diskgraph-ingest depends on crc32fast	CRC32 for range checksums in Rust
packages/diskgraph/diskgraph-ingest/Cargo.toml	[dependencies]	adds crc32fast	CRC32 for range checksums in Rust
packages/diskgraph/diskgraph-ingest/src/snapshot_mirror.rs	read_manifest, verify_snapshot_integrity, new snapshot_range_checksums	manifest reading shared by verify_snapshot_integrity and the new snapshot_range_checksums(dir, range_bytes), which returns one CRC32 per range for every file in _MANIFEST.json plus the manifest itself (rayon, 1 MiB buffers); unit test over a synthetic index	export computed these in Python (1.98 s vs 0.55 s on a 13 GB, 350-file index) and kept a second file list
packages/diskgraph/diskgraph-pybind/python/diskgraph/__init__.py	exports	re-exports snapshot_range_checksums	callable from kumo-engine export and load
packages/diskgraph/diskgraph-pybind/python/tests/test_exception_exports.py	tests	snapshot_range_checksums matches zlib.crc32 per range and for the manifest; a missing directory raises DiskgraphCorruptionError	pybind contract
packages/diskgraph/diskgraph-pybind/src/lib.rs	snapshot_range_checksums, module init	binds snapshot_range_checksums(output_dir, range_bytes) returning {relative path: [crc32 per range]}, GIL released; errors map to DiskgraphCorruptionError	expose the Rust checksum pass to Python
packages/diskgraph/third_party/artifact-manifest.json	diskgraph-ingest	Adds crc32fast to the resolved ingest package graph	The license checks reproduce this manifest from Cargo.lock
packages/kumo-core/src/kumo_core/sources/object_store.py	copies	made public with a default ceiling	transfer worker sizing
packages/kumo-engine/docs/mlops.md	docs	"Moving snapshots between machines" section (moved from the README)	user docs for export, load and the artifact root
packages/kumo-engine/kumo/engine/__init__.py	__all__	exports SnapshotOutputConfig	public config for snapshot export
packages/kumo-engine/kumo/engine/_snapshot_io.py	new module: export, export_location, store, read, load_into, _location, _open_output, _snapshot_manifest, _transfer, _retried, _check_readable	the destination comes from file_store (#112), any file store; uploads pass their size to open_sized; store exports under the MLOps root unless a complete export is there; export deletes kumo_snapshot.json, everything else at the location and kumo_snapshot.json again, uploads every file listed in index/_MANIFEST.json, copies _provenance.json byte for byte (or writes a reconstructed one), and writes kumo_snapshot.json (metadata, range_bytes, DiskGraph CRC32 map) last in one write; then lists the index and re-uploads files a concurrent export removed (_delete_marker, _missing); no resume or reuse; read rejects unknown formats; load stages, verifies sizes, CRCs and DiskGraph integrity, restores the snapshot manifest unchanged and renames into place	portable snapshots with an atomic per-location commit
packages/kumo-engine/kumo/engine/graph.py	GraphSnapshot.export(SnapshotOutputConfig), load_from_connector, source_changes, _graph_key, _data_version, _source_change, _loaded, _store, _lock, Graph.snapshot	new API; Graph.snapshot(mlops=) stores the snapshot under the artifact root; private _graph_key/_data_version are read from the snapshot id; source_changes compares only exact/timestamp/listing versions on the same connector and location (using the snapshot's record_sources), else unknown; with graph=, load warns once naming changed tables	export/load and a metadata-only freshness report
packages/kumo-engine/kumo/engine/trainer/__init__.py	__all__	exports SnapshotOutputConfig	config lives with the other output configs
packages/kumo-engine/kumo/engine/trainer/output_config.py	SnapshotOutputConfig	new (connector, path) config	snapshot export in the style of the other exports
packages/kumo-engine/test/conftest.py	graph fixture	shared synthetic graph	tests
packages/kumo-engine/test/test_snapshot_export.py	new	export through a filesystem-only chunked sink, MLOps store once, export/load round trip, snapshot manifest carried byte for byte or reconstructed, interrupted re-export not loadable, re-export replaces, files wiped by a dying concurrent export repaired before commit, unknown format rejected, malformed manifests, free space, object-store fakes, source_changes cases (including a Parquet directory), stale-load warning	coverage
packages/kumo-engine/test/test_snapshot_export_live.py	new	env-gated S3/GCS round trip	live coverage (skipped unless configured)

Requirement: a snapshot built on one machine must be usable on another, and a scoring job must be able to tell whether it is current. Existing behaviour: the index is machine-local. Smallest change: export/load keyed by snapshot id through the existing object-store sinks, plus a metadata-only freshness report. Verification: tests above, a real FileConnector export/load in separate processes on the current head, and a live S3 run (before the DiskGraph checksum layout and later changes). Stopping condition: round trip works on S3/GCS/file; local reuse unchanged; nothing refused on existing paths.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

• Last reviewed commit: 70e3212
• Latest commit, not reviewed: 46465ff

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The changes add snapshot export through S3, GCS, and file connectors. Exports support retries, resumption, manifests, and CRC32 checksums. Loading validates manifests, downloads verified ranges, and replaces local snapshots atomically. Graph snapshot APIs reuse shared stores and locks and report source changes. S3 filesystem creation now supports parallel-upload options. Tests and documentation cover these behaviors.

Sequence Diagram(s)

sequenceDiagram
  participant GraphSnapshot
  participant snapshot_io
  participant ConnectorFilesystem
  participant LocalSnapshot
  GraphSnapshot->>snapshot_io: export(snapshot, connector, path)
  snapshot_io->>ConnectorFilesystem: upload files and commit manifest
  GraphSnapshot->>snapshot_io: read(connector, path)
  snapshot_io->>ConnectorFilesystem: read and validate manifest
  GraphSnapshot->>snapshot_io: load_into(connector, path, root, manifest, target)
  snapshot_io->>ConnectorFilesystem: download and verify ranges
  snapshot_io->>LocalSnapshot: replace validated snapshot

Loading

Priority: Low

Merge Risk: Low · up to 70e32

Production export and load behavior looks sound. The S3 round-trip test can fail on older supported pyarrow versions. Fix that assertion before merging so CI is reliable.

Pre-merge checks: 2 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Change Map Design Review		The required PR conversation comment is not available. The checkout contains only the change-map marker in review instructions and the template; searches of the repository and Git history found no aut…	Provide the complete PR conversation comment beginning with <!-- kumo-pr-change-map -->, including its Change map and the required design evidence. Then reassess each row against the authoritative PR diff and repository code.

Passed checks: 2 passed

Check name	Status	Explanation
Pr Size		The pull request changes 12 files. This is below the failure threshold of more than 40 files.
Public Resource Access		PASS. The authoritative diff adds no concrete external URL, repository location, or package source location that the changed code requires. The added s3://bucket/snapshots, s3://bucket/prefix/, an…

Full details: Public Resource Access

Explanation

PASS. The authoritative diff adds no concrete external URL, repository location, or package source location that the changed code requires. The added s3://bucket/snapshots, s3://bucket/prefix/, and /Volumes/<catalog>/<schema>/<volume> references are illustrative examples or test fixtures. The live test uses caller-provided environment variables rather than a committed resource. The existing GitHub implementation link is unchanged.

Full details: Change Map Design Review

Explanation

The required PR conversation comment is not available. The checkout contains only the change-map marker in review instructions and the template; searches of the repository and Git history found no authored comment beginning with <!-- kumo-pr-change-map -->. The supplied summary describes the comment but does not provide its Change map rows, rationale, existing behavior, focused verification, or stopping condition. Therefore, I cannot assess whether each proposed change is necessary, at the correct abstraction layer, or behavior-preserving.

Resolution

Provide the complete PR conversation comment beginning with <!-- kumo-pr-change-map -->, including its Change map and the required design evidence. Then reassess each row against the authoritative PR diff and repository code.

✨ Finishing Touches: 💡 1

🛠️ Fix failing CI checks: 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 7e54c006-738e-49c6-850a-5f0efb2258e0

📥 Commits

Reviewing files that changed from the base of the PR and between 3973269 and c1979f0.

📒 Files selected for processing: 11

• packages/kumo-core/src/kumo_core/object_store_sink.py
• packages/kumo-core/src/kumo_core/sources/object_store.py
• packages/kumo-engine/README.md
• packages/kumo-engine/kumo/engine/_snapshot_io.py
• packages/kumo-engine/kumo/engine/graph.py
• packages/kumo-engine/test/conftest.py
• packages/kumo-engine/test/test_snapshot_export.py
• packages/kumo-engine/test/test_snapshot_export_live.py
• packages/kumo-s3/src/kumo_s3/backend.py
• packages/kumo-s3/src/kumo_s3/sink.py
• packages/kumo-s3/tests/test_sink.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments: 1

packages/kumo-engine/kumo/engine/graph.py: 1

248-249: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Reject comparisons across different source connectors.

If two file connectors use different roots but the same table name and version stamp, same_source is true. source_changes() then reports unchanged for data from a different connector. Graph.id distinguishes those connectors, but this comparison does not. Return unknown when the supplied graph does not match the snapshot’s graph identity, or record and compare connector identity for each source.

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 02bb012f-3d39-4f33-bcd8-359e4cd17a04

📥 Commits

Reviewing files that changed from the base of the PR and between d151486 and 8adc47c.

📒 Files selected for processing: 4

• packages/kumo-engine/README.md
• packages/kumo-engine/kumo/engine/_snapshot_io.py
• packages/kumo-engine/kumo/engine/graph.py
• packages/kumo-engine/test/test_snapshot_export.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments: 1

packages/kumo-engine/test/test_snapshot_export.py: 1

519-522: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

The S3 option assertion fails when pyarrow does not support allow_delayed_open.

sink._parallel_upload_options() returns {'background_writes': False} when S3FileSystem rejects allow_delayed_open. In that case, option['allow_delayed_open'] raises KeyError. This makes test_object_store_round_trip[s3] fail on pyarrow versions that the packages declare as supported: kumo-s3 declares pyarrow>=14, and kumo-core declares pyarrow>=15,<25. The production code handles these versions correctly, but the test does not.

Compare each recorded option set with the probed result. Do not hard-code the delayed-open flag.

💚 Proposed fix

     if scheme == 's3':
-        assert seen and all(
-            option['allow_delayed_open'] and not option['background_writes']
-            for option in seen)
+        from kumo_s3 import sink
+        expected = sink._parallel_upload_options()
+        assert seen and all(option == expected for option in seen)

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 9749e921-9993-459f-8487-5daa7be1b286

📥 Commits

Reviewing files that changed from the base of the PR and between 1661366 and 70e3212.

📒 Files selected for processing: 8

• packages/kumo-core/src/kumo_core/object_store_sink.py
• packages/kumo-engine/README.md
• packages/kumo-engine/kumo/engine/_snapshot_io.py
• packages/kumo-engine/kumo/engine/graph.py
• packages/kumo-engine/test/test_snapshot_export.py
• packages/kumo-s3/src/kumo_s3/backend.py
• packages/kumo-s3/src/kumo_s3/sink.py
• packages/kumo-s3/tests/test_sink.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

[aaron-kumo]

P2 — Reject unsupported export formats instead of interpreting them as format 1.

read() currently warns on an unknown format and continues using format-1 semantics. A future format can change field meanings or required verification while retaining a superficially similar JSON structure.

Please reject unsupported format versions clearly, while continuing to accept unknown optional fields within supported formats. This does not require changing the current format or dropping support for existing snapshots/model bundles. Future engines can retain readers for older formats.

P2 — Do not use the inventory hash as an exact build fingerprint.

index_manifest hashes _MANIFEST.json, which contains filenames and sizes rather than file-content digests. Different materializations with identical filenames and sizes can therefore share this fingerprint.

This matters for interrupted-export resumption: existing files are skipped by size, while checksums are calculated from the current local files. Resuming against another materialization can commit mismatched files and checksums, leaving an export that cannot load.

Please make reuse depend on content identity or another reliable materialization identity, with appropriate verification of resumed files. Per-range CRCs can remain the transfer checks; the inventory hash should not be described or used as proof of exact build identity.

P2 — Preserve unsupported provenance when carrying an artifact forward.

A newer snapshot-manifest.json schema currently reads as None; exporting that snapshot then writes "snapshot": null. An older engine can consequently discard provenance it does not understand while successfully copying the index.

Please preserve that metadata opaquely during export/re-export, or explicitly refuse the operation if it cannot be preserved safely. Ignoring unsupported provenance for interpretation should not erase it from the next artifact.

Discussion — Should multiple materializations coexist for one logical snapshot?

Currently, exporting a rebuilt snapshot to an existing logical snapshot location keeps the previous export, even when its build differs. What should happen after an engine/storage-format upgrade when the same graph and source versions produce a new materialization?

Would it be better to distinguish logical snapshot identity from materialization identity and allow both builds to coexist? That would support upgrades, rollback, and exact replay without making users delete the previous export. I’m open to a smaller initial approach, but think we should define this behavior before making the logical-id destination layout a durable contract.

[Vedaant-J]

Folded into #50 (artifacts & provenance), which now contains this PR's final code: the consolidated stack shows only the end state. Review continues there.