DataLab-Capsule is a pure Python library (CPython 3.9+ and Pyodide). The core needs the standard library and jsonschema; functions marked h5py also need NumPy and h5py. Every error derives from datalab_capsule.integrity.ProvenanceError. Nothing in the library accesses the network or executes code named in a file.
from datalab_capsule.archive import read_capsule
from datalab_capsule.diff import diff_manifests
from datalab_capsule.manifest import inspect_manifest
from datalab_capsule.verify import verify_capsule
capsule = read_capsule("workspace.dlcapsule") # safe reader, validated manifest
summary = inspect_manifest(capsule.manifest) # activities, roles, states, workspace
result = verify_capsule("workspace.dlcapsule") # integrity checks, no computation
changes = diff_manifests(old.manifest, new.manifest)| Function | Purpose |
|---|---|
archive.read_capsule(source, policy=DEFAULT_SIZE_POLICY, resolve_workspace=None) |
Read a capsule (path or bytes) in memory and check it: archive safety, size limits, manifest validity and fingerprint, workspace SHA-256 and size. Returns Capsule(manifest, workspace, workspace_ref, attachments); workspace is None for a referenced workspace without resolver; attachments maps each declared attachment name to its checked bytes. resolve_workspace(iri) -> bytes is the only way a referenced workspace is obtained. Raises CapsuleError or ManifestError. |
archive.SizePolicy(max_archive_bytes, max_manifest_bytes, max_workspace_bytes, max_attachment_bytes) |
Size limits of the calling edition. |
manifest.validate_manifest(manifest) |
Offline validation (no JSON-LD processor): structure, terms defined by the embedded RO-Crate context or the profile, references, fingerprint, revision, embedded JSON. Raises ManifestError listing every problem. |
manifest.inspect_manifest(manifest) |
Summary: name, description, licence, date, profiles, fingerprint, parent revision, workspace (id, referenced, sha256, size), attachments (name, media type, size, sha256, description), activities (roles in order, analysis-result digests, parameters, origin, limits, legacy record, observed state), states (kind, object, fingerprint, locator, ROI, producer, consumers). |
manifest.manifest_fingerprint(manifest) |
sha256: digest of the RFC 8785 canonical JSON of the manifest without its own fingerprint. |
verify.verify_capsule(source, *, resolve_workspace=None, reports=(), reviews=(), policy=...) |
h5py for the stored states. Checks the capsule, then the workspace ledger against the manifest and every stored state (fingerprint and ROI), then the supplied reports and reviews. Returns valid, fingerprint, workspace (verified, integrity_only without h5py, not_available), states_located, states_altered, reports, reviews, problems. |
diff.diff_manifests(old, new) |
Semantic difference grouped as scientific, presentation, storage (workspace bytes, storage mode, locators), environment and attachments (content not interpreted), plus identical (same manifest fingerprint), revision_of_old and scientific_conclusion (differences, none, or none_detected when the workspace files or the attachment contents differ but the recorded activities and states do not). |
| Function | Purpose |
|---|---|
archive.create_from_hdf5(source, *, name=None, description=None, license_=None, date_published=None, revision_of=None, workspace_ref=None, attachments=()) |
h5py. Build a capsule from a workspace holding a provenance block. revision_of names the parent's manifest fingerprint; workspace_ref (absolute IRI) makes the capsule reference its workspace instead of including it; attachments lists light files as {name, data, media_type, description} (at most MAX_ATTACHMENTS, each at most MAX_ATTACHMENT_BYTES; names: letters, digits, ., _, -). Returns bytes. |
review.create_review(target, *, reviewer, statement, verdict=None, reviewer_id=None, reports=(), date=None) |
Create a review of one revision (target: manifest or fingerprint). verdict: endorsed, disputed, comment or None. Cited reports keep their activity and verdict. |
review.validate_review(review, manifest=None) |
Check the schema, the review's own fingerprint and, when given, that it targets manifest. Raises ReviewError. |
review.review_fingerprint(review) |
Fingerprint of a review without its own fingerprint. |
exchange holds what the capsule dialogs of DataLab Desktop and DataLab-Web (File > Workspace capsule… in both editions) need beyond reading and verifying, so that both give the same answers. Each edition passes its own SizePolicy. The workspace-capsule command line is an independent client of the same library. Nothing is computed.
| Function | Purpose |
|---|---|
exchange.inspect_capsule(data, *, workspace=None, reports=(), reviews=(), policy=...) |
Inspect a capsule with the supplied workspace (referenced), reports and reviews (JSON texts). Returns summary and manifest (None when unreadable), integrity (valid, fingerprint, workspace, states_located, states_altered, problems; the problems of supplied documents are not integrity problems) and documents (reports, reviews, unreadable, each with its status). |
exchange.open_capsule(data, workspace=None, policy=...) |
Return (workspace bytes, origin). Raises CapsuleError for a referenced workspace not supplied or not matching. |
exchange.capsule_origin(manifest) |
{fingerprint, name, attachments}: what an edition keeps of the capsule a workspace was opened from (the parent of a later revision). |
exchange.capsule_attachment(data, name, policy=...) |
Checked bytes of one attachment, for an explicit download (never opened). |
exchange.check_registry_revision(manifest, fingerprint, reviews=()) |
Check a manifest returned by a registry (valid, the requested fingerprint) and each review. Returns summary, manifest, reviews ({review, status}). |
exchange.diff_checked_manifests(old, new) |
Validate both manifests, then diff.diff_manifests. |
exchange.review_of(manifest, *, reviewer, statement, verdict=None, reports=()) |
Validate the manifest, then review.create_review. |
exchange.attachment_name(filename, taken=()) |
Attachment name allowed by the profile, unique among taken. |
| Function | Purpose |
|---|---|
ledger.Ledger |
Append-only ledger: observe(object_uuid, facts), record_activity(...), record_legacy_activity(...), states, activities, activity(id), latest_state(object_uuid), to_json(), from_json(text), validate(). |
calls.make_call(operation_id, contract_version, parameters, inputs) |
Build an operation call bound to states by role; is_opaque(call) tells whether it names no operation. |
environment.collect_environment(edition, edition_version, ...), environment_id(record) |
Environment record (no host or user name) and its content digest. |
integrity.state_facts(obj), signal_fingerprint_record(...), signal_rows(...), signal_fingerprint(...), signal_fingerprint_v2(...), image_fingerprint(...), roi_facts(roi), value_digest(value) |
State facts and fingerprints of signal-like and image-like objects (duck-typed), ROI and analysis-result digests. signal_fingerprint_record chooses the scheme from the content (datalab-signal-v1 when all rows share one dtype, datalab-signal-v2 otherwise, e.g. float32 or complex values); objects and their HDF5 rows give the same record. Every numeric dtype is covered, complex included. |
hdf5.save_ledger(h5file, ledger, capsule_origin=None), load_ledger(h5file), load_block(h5file), read_capsule_origin(h5file) |
h5py. Write the /DataLab_Provenance block after the panels, with the capsule the workspace was opened from when known (optional capsule_origin_json, checked by check_capsule_origin); read it back with the status of each state (altered, unavailable); load_block also returns the locators, reading the block once; read_capsule_origin returns the origin or None. |
hdf5.read_block, write_block, locate_current_states, locate_fingerprints, read_signal, read_image, read_roi, scan_object_index |
h5py. Lower-level block and object readers; links and virtual datasets are refused. |
legacy.import_legacy(ledger, objects, *, is_known, saved_version, current_version), legacy_summary(ledger) |
Import processing records of a workspace saved without provenance as legacy_metadata activities, and summarise them. |
replay.prepare_activity(ledger, activity_id, *, resolve_contract, find_object, observe_object, check_preconditions, decode_parameters, state_status=None, apply_context=None, find_state_object=None, restore_roi=None) |
Shared eligibility checks before a replay; returns a Plan or a Refusal with its eligibility code. An input whose data changed since the activity is refused (input_changed). An input whose data are unchanged but whose region of interest changed is replayed on a copy with the recorded region of interest, returned by restore_roi (Computation.with_roi, i.e. sigima.proc.contracts.with_roi); without it, or when the copy does not carry the recorded definition, the input is refused (input_changed). The edition injects every resolver. |
replay.compare_state(state, facts, file_status=None) |
The single definition of "the same state" (data fingerprint and region of interest), shared by replay preparation, inspection and verification (inputs and recorded results, through inspection.output_state_status: a result whose data or region of interest changed is altered and is not used as reference): available, changed, roi_changed (same data, other region of interest: restorable for a replay, reported as changed by inspection), altered or unsupported, with a reason. |
inspection.Inspector, INSPECTION_VERSION, MAX_DEPTH, MAX_INPUTS, MAX_ANALYSES, MAX_OBJECT_STATES |
Read-only provenance description of a result, used through the Recorder methods below. A bounded JSON projection: the anchored state (content: unchecked until the data are checked, then matches, differs, absent or unavailable; an absent object is reported absent, and an absent input or result missing, without checking, since nothing has to be read; file_status read when the workspace was opened), the producing activity (operation, typed parameters as encoded, ordered inputs with role, presence, producer, has_ancestors and cycle, outputs, context without metadata values, environment, limits, local support: function known/unknown/incompatible/opaque/legacy and verification qualification), the latest states of the object, its legacy records and the analyses recorded on it (artifact status, no_result). Ancestors are expanded by another call. |
compare.compare_rows(candidate, reference, rule=None, *, output="result", environment="unknown"), compare_exact(candidate, reference), compare_artifacts(kind, candidate, reference, rule=None, ...), artifact_content(kind, value), compare_environments(...), build_report(..., outputs=None), validate_report(report) |
Comparison under the rule declared by the qualification of the operation contract ({"rule": "exact"} by default, or per-output tolerances giving within_tolerance, applied exactly within the same environment when declared so); complex values are compared on both parts. Analysis results (stored forms of tables and geometries) are compared on their values under the rule and on their structure (kinds, column names, ROI indices) exactly. Environment comparison and verification reports; with outputs, a report compares every output by role (state, analysis result or no result) and its verdict is the most severe one. |
recorder.Recorder(host, computation), RecorderHost, Computation, computation_from_sigima(contracts), CandidateError |
Provenance recorder shared by the editions: begin, complete, complete_analysis, complete_without_result (an optional output without result, not an error), deferred/finalize_deferred (the activities recorded in a deferred context are listed in deferred.activity_ids), command_activity_ids(command_id), artifact_statuses, prepare_replay, verify(activity_id, execute_candidate) (every output compared; execute_candidate returns what the function returned and raises CandidateError on error or cancellation), reset, save, read ((ledger, state_status, capsule_origin)), load(block, replaced, saved_version=None, current_version=None, appended=()) (capsule_origin, the capsule the workspace was opened from, is saved with the workspace and restored when a file replaces the workspace, kept when a file is appended; an appended file never merges its ledger, and recorder.APPEND_NOTICE is added to notices when it held provenance; appended objects holding an identifier already known to the ledger get a new observed state marked appended_without_provenance (inspection.IDENTITY_LIMIT), so that nothing recorded for another object is attributed to them, and earlier states are not shown as theirs; consulting, checking and verifying apply the same rule, inspection.holder_of_state / holder_of_artifact: an input, a result or an analysis result of a state before the break is missing). Inspection: describe_object(object_uuid, check_data=False), describe_state(state_id, check_data=False, trail=()), describe_activity(activity_id, check_data=False) never prepare nor run a computation and never change the ledger; consulting computes no fingerprint, checking the data compares the object and the direct inputs and outputs of the described activity. producing_call(object_uuid) returns the call that produced the latest recorded state of an object (every input in order, parameters as encoded, the locally known contract or why it is unavailable), which Processing restores. resolve_function returns a contract known to the local library, qualified or not (ordinary processing); resolve_contract requires the qualification (verification). The edition provides its accesses (RecorderHost) and the computation library its functions (Computation); errors of the library are mapped to eligibility codes once, in computation_from_sigima, which receives the sigima.proc.contracts package. The global options and the digests of the input metadata read by a computation are recorded in the execution context; begin(..., executed=...) takes the metadata from the objects actually computed when they differ from the recorded inputs. |
| Name | Purpose |
|---|---|
integrity.canonical_json(value), canonical_json_bytes(value), json_digest(value), sha256_digest(data) |
RFC 8785 canonical JSON and SHA-256 digests (sha256:<hex>). |
integrity.load_json(text), check_json_depth(value), MAX_JSON_DEPTH |
Parse JSON from a file with a nesting limit (64). |
archive.MAX_ENTRIES, MAX_MANIFEST_BYTES, MAX_WORKSPACE_BYTES, MAX_ATTACHMENTS, MAX_ATTACHMENT_BYTES, MAX_ARCHIVE_BYTES, DEFAULT_SIZE_POLICY |
Default archive limits. |
hdf5.MAX_SIGNAL_BYTES, MAX_IMAGE_BYTES, MAX_JSON_BYTES |
Dataset limits in workspaces. |
workspace-capsule validate | inspect | diff | verify | attachment | workspace | create | review | registry list | registry publish | registry fetch (see the README). cli.main(argv) runs it and returns the exit code.
registry.Registry(root, *, max_manifest_bytes=8 MiB, max_attachment_bytes=16 MiB) keeps manifests, declared attachments and reviews in a directory (SQLite plus files); it needs only the standard library (sqlite3, so not Pyodide). registry_app.create_app(registry, *, token=None, origins=("*",)) returns the FastAPI application (registry extra); workspace-capsule-registry ROOT [--host] [--port] [--token-env VARIABLE] [--origin ORIGIN]... serves it.
Method and route (/api/v1) |
Registry method |
Result |
|---|---|---|
GET /status |
Registry name, version, whether publishing needs a token. | |
GET /revisions?limit= |
list_revisions(limit) |
Flat list, most recently published first: fingerprint, name, description, revision_of, dates, workspace_referenced, attachments (stored once uploaded), number of reviews. |
POST /revisions (manifest JSON) |
publish_manifest(data) |
Validates and stores it; 201 when created, 200 when already published. |
GET /revisions/{fingerprint} |
get_manifest(fingerprint) |
The manifest as published. |
PUT /revisions/{fingerprint}/attachments/{name} (raw bytes) |
put_attachment(fingerprint, name, data) |
Stores a declared attachment whose size and SHA-256 match; 204. |
GET /revisions/{fingerprint}/attachments/{name} |
get_attachment(fingerprint, name) |
The bytes, checked again, served as application/octet-stream with Content-Disposition: attachment and nosniff. |
GET /revisions/{fingerprint}/reviews |
list_reviews(fingerprint) |
Reviews of the revision, oldest first. |
POST /reviews (review JSON) |
publish_review(data) |
Validates it against its published target; 201 or 200. |
GET /reviews/{fingerprint} |
get_review(fingerprint) |
The review as published. |
Errors: 401 without the token for a write, 404 for an unknown revision, attachment or review (RegistryNotFound), 413 for an oversized body, 422 for an invalid document or fingerprint (RegistryError).
Client (standard library): registry_client.RegistryClient(url, token=None) with status(), list_revisions(), get_manifest(fingerprint), list_reviews(fingerprint), get_attachment(fingerprint, name, max_bytes=16 MiB) (unchecked bytes), get_checked_attachment(fingerprint, name, sha256, size) (checked against the SHA-256 and size of the checked manifest), publish_manifest(manifest), upload_attachment(fingerprint, name, data), publish_review(review) and publish_capsule(capsule, reviews=()) (manifest, attachments, reviews; never the workspace); registry_client.fetch_revision(client, fingerprint) returns the manifest checked against the fingerprint, each review with its status and each declared attachment checked against its size and SHA-256. Each response is read up to a limit and refused beyond it: 16 MiB for a listing (MAX_LISTING_BYTES), the registry's manifest limit for a manifest, the declared size for an attachment fetched by fetch_revision, 64 KiB for other replies and errors (MAX_REPLY_BYTES). Errors raise RegistryClientError.
| File | $id |
Document |
|---|---|---|
schemas/ledger-1.schema.json |
urn:datalab:schema:provenance-ledger:1 |
Workspace ledger |
schemas/report-1.schema.json |
urn:datalab:schema:verification-report:1 |
Verification report |
schemas/review-1.schema.json |
urn:datalab:schema:capsule-review:1 |
Review assertion |
contexts/ro-crate-1.3.jsonld |
(copy of https://w3id.org/ro/crate/1.3/context, CC0) |
Pinned RO-Crate context used for offline validation |