Skip to content

architecture.md

Latest commit

 

History

History
564 lines (460 loc) · 34.1 KB

architecture.md

File metadata and controls

564 lines (460 loc) · 34.1 KB
Raw

DataLab-Web — Architecture

Status: living document. Mirrors the source tree as of the time of writing. When you add a new top-level subsystem, update the diagrams below in the same commit.

DataLab-Web is the browser-native sibling of the DataLab desktop application. It runs the Sigima computation engine inside Pyodide (CPython compiled to WebAssembly) and renders a dedicated React + TypeScript UI inspired by the desktop Qt application. Plotting is delegated to Plotly.js — PlotPy (Qt) is not available in the browser.

The whole stack runs inside a single browser tab: there is no backend. The HDF5 workspace file is the only durable artifact (see Persistence model).

1. General architecture

flowchart TB
    Host["Host page <i>(optional)</i><br/>@datalab-platform/web-sdk"]

    subgraph Tab["Browser tab"]
        direction TB

        subgraph Main["Main thread"]
            direction TB
            UI["React UI<br/>(src/components, src/App.tsx)"]
            Actions["Action registry<br/>(src/actions)"]
            DBridge["DialogBridge.tsx"]
            RT["DataLabRuntime (TS)<br/>(src/runtime/runtime.ts)"]
            Preview["ProcessingPreviewController.ts<br/>bounded preview queue"]
            Py0["Pyodide #0 — main<br/>bootstrap.py · processor.py<br/>dlw_*.py · Sigima · numpy · scipy"]

            UI -->|dispatch| Actions
            Actions -->|typed calls| RT
            RT -->|runPython| Py0
            UI --> Preview
            Preview -->|previewFeature| RT
            Py0 -->|dialog requests| DBridge
            DBridge --> UI
        end

        PB["proxyBridge.ts<br/>(whitelisted command router)"]

        subgraph Workers["Web Workers"]
            direction LR
            Mw["macroWorker.ts<br/>Pyodide #1<br/>macro_proxy.py + user code"]
            Nw["notebookWorker.ts<br/>Pyodide #N (1 per notebook)<br/>notebook_display.py"]
        end

        IDB[("IndexedDB<br/>recentStore.ts")]

        RT --- PB
        PB <-->|postMessage| Mw
        PB <-->|postMessage| Nw
        UI -. autosave / recovery .-> IDB
    end

    Host <-->|postMessage · JSON-RPC<br/>remoteBridge.ts| RT
Loading

Key facts:

  • One Pyodide instance owns the object model (_MODEL in bootstrap.py). All UI reads go through it.
  • Each macro / notebook runs in its own Web Worker with its own Pyodide instance. Workers never touch the main object model directly: they call back to the main runtime through proxyBridge.
  • Processing preview reuses the already-loaded computation runtime. Python deep-copies the source, computes a temporary plotting payload, and never publishes the copy or result to the main object model.
  • The host page (optional) controls an embedded DataLab-Web through the same RPC vocabulary, but transported by remoteBridge over window.postMessage.

2. Layer view

Five layers, left to right — each layer only depends on the ones to its right.

flowchart LR
    L1["<b>L1 — UI layer</b><br/>(React components)<br/><br/>src/components/<br/>src/App.tsx<br/>src/main.tsx<br/><br/><i>Purely presentational.<br/>No Pyodide imports.</i>"]
    L2["<b>L2 — Orchestration</b><br/><br/>src/actions/<br/>(registry, menu builder)<br/>src/macros/<br/>src/notebook/<br/>src/plugins/<br/>src/aiassistant/<br/>src/preferences/"]
    L3["<b>L3 — Runtime / bridge</b><br/>(TypeScript)<br/><br/>RuntimeApi.ts (façade)<br/>runtime.ts<br/>(DataLabRuntime + types)<br/>WorkerRuntimeProxy.ts · kernelWorker.ts<br/>runtimeMode.ts · workerProtocol.ts<br/>ProcessingPreviewController.ts<br/>RuntimeContext.tsx<br/>WorkspaceContext.tsx<br/>MacroRuntime.ts<br/>proxyBridge.ts · remoteBridge.ts<br/>macroWorker.ts · notebookWorker.ts<br/>src/storage/ (OPFS spill stores)"]
    L4["<b>L4 — Python kernel</b><br/>(loaded into Pyodide)<br/><br/>bootstrap.py<br/>processor.py<br/>dlw_main.py<br/>dlw_wheels.py<br/>dlw_plugins.py<br/>dlw_applications.py<br/>dlw_h5browser.py<br/>dlw_interactive_fit.py<br/>dlw_title_format.py<br/>dlw_provenance.py<br/>notebook_display.py<br/>macro_proxy.py<br/>_guidata_*_shim.py"]
    L5["<b>L5 — Computation engine</b><br/><br/>Sigima<br/>+ numpy · scipy<br/>+ scikit-image<br/>+ h5py · pandas …<br/><br/><i>Installed via micropip<br/>on first load.</i>"]

    L1 --> L2 --> L3 --> L4 --> L5
Loading

Inter-layer contracts

Boundary Contract
L1 → L2 useAction(id) / event handlers; no direct runtime calls from components.
L2 → L3 Typed methods on the RuntimeApi façade (useRuntime()); the concrete impl is the in-thread DataLabRuntime or a worker-backed proxy.
L3 → L4 pyodide.runPython() / pyodide.runPythonAsync() calling bootstrap.py helpers.
L3 ↔ Worker postMessage (bridge_call / bridge_reply / stdout / stderr / …).
L3 ↔ Host postMessage JSON-RPC (request / response / event) via remoteBridge.
L4 → L5 Python imports (sigima.proc, sigima.objects, …). Catalog is auto-discovered.

Hard invariants

  • Components must not import from src/runtime/runtime.ts directly; they consume useRuntime() (typed as the RuntimeApi façade) and dispatch through src/actions/. Depending on the façade — not the concrete class — is what lets the runtime run either in-thread or in a Dedicated Web Worker without touching any call site.
  • Python helpers return JSON-friendly dicts / lists (tolist() on arrays), never live PyProxies, to keep the bridge cheap.
  • The pinned PYODIDE_VERSION in runtime.ts and the <script> tag in index.html must always match — the wheel index is version-specific.

3. Component view

3.1 UI layer — src/components/

Presentational React. Notable components:

  • MenuBar.tsx / MenuDropdown.tsx — top menu bar built from the action registry.
  • ObjectTree.tsx, TreeKindSwitcher.tsx, SidePanel.tsx — left panel: hierarchical workspace (groups, signals, images) + properties / metadata / stats / history.
  • SignalPlot.tsx, ImagePlot.tsx, MultiImagePlot.tsx, CentralViewSwitcher.tsx — Plotly-based central plot area.
  • DataSetDialog.tsx, ProcessingDataSetDialog.tsx + DataSetForm/ — auto-generated parameter dialogs from guidata DataSet JSON schemas (do not hand-write a form unless the auto path genuinely cannot express it).
  • ProcessingPreview.tsx + ProcessingPreviewPlot.tsx — optional, non-publishing signal/image result preview for eligible processing dialogs.
  • DialogBridge.tsx — single React entry-point that receives dialog requests from Python via bootstrap.set_dialog_bridge() and routes them to the appropriate React dialog component.
  • ROI: RoiPanel.tsx (non-modal docked editor for signal & image ROIs and for the image Erase area… session — draw / table-edit regions, then apply), RoiGridDialog.tsx, signalRoi.ts, imageRoi.ts.
  • Other domain dialogs: H5BrowserDialog.tsx, InteractiveFitDialog.tsx, TextImportWizard.tsx, ObjectPropertiesDialog.tsx, ProfileDefinitionDialog.tsx, SaveToDirectoryDialog.tsx, SeparateViewDialog.tsx, PluginManagerDialog.tsx, PluginConsentDialog.tsx, ApplicationsDialog.tsx, RecipeInputResolverDialog.tsx, HelpDialog.tsx.
  • CapsuleDialog.tsx (File > Workspace capsule…) — exports a capsule (included or referenced workspace, optionally as a revision of the capsule the workspace was opened from), inspects one without opening it, opens it, replays activities locally, writes reviews and compares revisions. It keeps three results apart: the integrity of a capsule (computed from its files, nothing recomputed), the checks of supplied documents (reports and reviews: their verdicts are their authors' claims) and the local replay of an activity of the open workspace. Light attachments selected at export are declared in the manifest (size, SHA-256) and listed on inspection; one is saved only on request and never opened. A child revision does not copy its parent's attachments: the export names those not selected again (capsule_origin.attachments). Comparisons list storage changes (workspace bytes, locators) apart from scientific ones and state a scientific conclusion, which never claims equality when the workspace files or the attachment contents differ. The optional Registry tab connects to a DataLab-Capsule registry (address kept in localStorage, publishing token in sessionStorage only) to publish a manifest with its attachments and reviews, list revisions and retrieve one; everything it returns is checked by DataLab-Capsule in the runtime (check_registry_revision) and attachments against the checked manifest's SHA-256. Without an address, nothing changes.
  • Macros / notebooks: MacroPanel.tsx, MacroEditorTabs.tsx, MacroConsole.tsx, notebook/.
  • AI assistant: AIAssistant/.

3.2 Orchestration — src/actions/

flowchart TB
    subgraph A["src/actions/"]
        T["types.ts<br/>ActionDescriptor, kinds"]
        R["registry.ts<br/>register / get / list actions"]
        B["buildMenu.ts<br/>merge static entries +<br/>Sigima catalog → menu tree"]
        I["submenuIcons.ts<br/>icon assignment"]
    end

    Cat["processor.py<br/>build_signal_catalog()<br/>build_image_catalog()"] --> B
    R --> B
    T --> R
    B --> MB["MenuBar.tsx<br/>(renders the tree)"]
    MB -->|click| Run["action.run(ctx)"]
    Run --> RT2["runtime.&lt;method&gt;(…)"]
Loading

Most processings are not hand-registered: buildMenu.ts queries processor.py (build_signal_catalog, build_image_catalog) and synthesises menu entries automatically. The static registry only adds DataLab-Web-specific commands (file IO, ROI editor, plugin manager, notebooks, macros, …).

3.3 Runtime / bridge — src/runtime/

DataLabRuntime (runtime.ts)

Single class that owns the main-thread Pyodide instance and exposes a typed surface to the rest of the app:

classDiagram
    class DataLabRuntime {
      +load() / ready
      +runPython() / runPythonAsync()
      +list_signals / get_signal_xy / …  (object CRUD)
      +createSignalTyped / createImageTyped
      +applyFeature(id, oids, params)
      +HDF5 workspace save / load
      +macros / notebooks CRUD
      +plugins (inspect, install, enable, remove, hot-reload)
      +plugin recipes / examples
      +interactive fit, ROI editing, …
    }
    DataLabRuntime ..> Pyodide : runPython
    DataLabRuntime ..> SignalMeta
    DataLabRuntime ..> SignalData
    DataLabRuntime ..> ProcessingDescriptor
    DataLabRuntime ..> FeatureDescriptor
    DataLabRuntime ..> PanelTree
Loading

Companion exports (interfaces / types): SignalMeta, SignalStyle, SignalData, ProcessingDescriptor, LastProcessingInfo, Pattern, FeatureDescriptor, InteractiveFitInfo / Param / Init / Auto, PluginInfoMeta, PluginRecord, PluginMenuAction, PluginRecipePreparation, PluginRecipeCommit, PluginExampleOpenResult, PanelKind, ObjectNode, GroupNode, PanelTree, ObjectMeta, plus thin PyodideAPI / PyProxy aliases.

React contexts

flowchart LR
    RC["<b>RuntimeContext</b><br/>singleton load()<br/>useRuntime()"]
    WC["<b>WorkspaceContext</b><br/>dirty flag<br/>current filename<br/>recovered-from-cache flag"]
    Consumers["L1 / L2 consumers"]
    RC --> Consumers
    WC --> Consumers
Loading

RuntimeContext guarantees Pyodide is loaded exactly once, even across HMR reloads (the Python _MODEL and _CATALOG survive re-execution of bootstrap.py).

Worker subsystem

flowchart LR
    subgraph MT["Main thread"]
        direction TB
        MR["MacroRuntime.ts<br/>spawns workers on demand<br/>keeps a warm standby<br/>postMessage glue"]
        PB2["proxyBridge.ts<br/>whitelisted command router"]
        RT3["DataLabRuntime"]
        PC["ProcessingPreviewController.ts<br/>one active + latest pending"]
        PB2 --> RT3
        PC -->|previewFeature| RT3
    end

    subgraph Workers2["Web Workers"]
        direction TB
        MW2["macroWorker.ts<br/>Pyodide #1<br/>macro_proxy.py<br/>+ user script"]
        NW2["notebookWorker.ts<br/>Pyodide #N (per notebook)<br/>notebook_display.py"]
    end

    MR ==>|postMessage| MW2
    MR ==>|postMessage| NW2
    MW2 -. bridge_call .-> PB2
    NW2 -. bridge_call .-> PB2
Loading

Wire shape (same for both worker kinds):

Direction Message types
main → worker init · run · bridge_reply
worker → main ready · started · finished · stdout · stderr · bridge_call

proxyBridge.ts is a whitelisted command router: it maps the method string in a bridge_call to a specific main-thread runtime call. Workers cannot invoke arbitrary methods — only the ones explicitly listed (add_signal, add_image, list/get/delete object, add/set object from pickle, add_group, get/set selection, etc.).

ProcessingPreviewController coordinates requests per RuntimeApi: one computation may run while a single pending slot retains only the latest parameters. bootstrap.preview_feature deep-copies the source and delegates to the same BaseProcessor.apply path as publication, then returns a binary plotting payload. For a single-source dialog it also retains the complete raw result behind an opaque token. apply_feature consumes that result at most once after checking the feature, source data revision and parameters, then performs the usual publication and history updates. A missing, stale or mismatched token falls back to normal computation. Preview never inserts the temporary object into _MODEL, updates processing history, or dirties the workspace.

Disabling preview, Cancel, Escape, or dialog unmount drops pending work and invalidates late results. A Python call that has already started is allowed to finish; it may delay the next serialised runtime call. Preview controls are therefore exposed only for the default worker-hosted runtime and hidden in the runtime=main fallback, where computation would block the UI thread.

Remote control bridge

flowchart LR
    SDK["@datalab-platform/web-sdk<br/>(packages/sdk/)"]
    RB["remoteBridge.ts<br/>request / response / event"]
    RT4["DataLabRuntime"]
    SDK <-->|postMessage<br/>JSON-RPC| RB
    RB --> RT4
Loading

Worker-hosted runtime (default) + on-disk storage

By default the Pyodide runtime is hosted in its own Dedicated Web Worker, selected by runtimeMode.ts. This keeps the UI thread responsive during heavy work and unlocks synchronous OPFS spills (DEW ADR #2). The legacy in-thread topology — where DataLabRuntime runs on the UI thread — is kept as a backup / escape hatch, selectable with ?runtime=main (or a localStorage key / the VITE_RUNTIME_WORKER=0 build flag). Both paths satisfy the same RuntimeApi façade, so the UI is identical either way.

flowchart LR
    subgraph MT3["Main thread"]
        direction TB
        RC2["RuntimeContext<br/>(picks the mode)"]
        WP["WorkerRuntimeProxy.ts<br/>implements RuntimeApi<br/>RPC over postMessage<br/>+ synchronous mirror cache"]
        RC2 --> WP
    end
    subgraph KW["Kernel worker"]
        direction TB
        KWp["kernelWorker.ts<br/>Pyodide #0 + DataLabRuntime<br/>+ OpfsSyncObjectStore"]
    end
    WP <-->|"workerProtocol.ts<br/>call / result / mirror /<br/>mutation / dialog (transferables)"| KWp
Loading

Why this exists:

  • Synchronous OPFS spills. The on-disk storage mode (StorageMode = "ram" | "disk") spills heavy signal/image arrays out of the WASM heap to the Origin Private File System through src/storage/. The async OpfsObjectStore (createWritable) works on the main thread; the faster OpfsSyncObjectStore (createSyncAccessHandle) is worker-only by spec, so it is selected only when the runtime runs in the worker. Either way HDF5 stays the durable source of truth — OPFS is an ephemeral spill cache.
  • Disk guard policy. In disk mode every object holds a 1-element placeholder at rest, and runtime.ts pages arrays in only around the Python call that reads them. pyCallPolicy.ts declares what each call reads (object ids passed as oid, current_oid, source_oid, selected_oids, candidate_ids, bindings…, or a Python resolver such as last_processing_input_ids, or every object for workspace serialisation and plugin menu actions) and after which calls the OPFS store is reconciled with the model (disk_sweep_state: on-disk copies of objects that left the model are deleted, objects left in memory are spilled, whatever the call returns). A paged-in object is settled after the call (settle_object_array): its on-disk copy is rewritten only when its data revision changed. Two audits keep the policy complete: tests/ts/runtime/pyCallPolicyAudit.test.ts requires every identifier-like kwarg passed to callPy to be classified, and tests/python/test_py_call_policy_audit.py requires every called Python function that adds, deletes or replaces objects to be followed by a reconciliation.
  • Workspace dirty flag. After each successful call listed in MUTATING_PY_FUNCTIONS (pyCallPolicy.ts), the runtime notifies its mutation listeners (forwarded by the kernel worker in worker mode) and App.tsx marks the workspace dirty, which drives the • title marker and the unsaved-changes guard on close or reload. Clean transitions (REPLACES_MODEL) and calls whose caller decides from their result (DIRTY_SET_BY_CALLER, e.g. opening an application example) are listed apart. The same audits keep this list complete: the Python audit requires every call that adds, deletes or replaces objects to be listed, and the TypeScript audit requires every call named like a write (set_*, paste_*, move_*, plot_*…) to be listed or explicitly exempted in NON_DURABLE_WRITE_CALLS.
  • Zero-copy bridge. workerProtocol.collectTransferables passes every ArrayBuffer in call arguments and return values as a postMessage transferable, so binary payloads (image/signal arrays, file/HDF5 bytes) cross the boundary without a structured-clone copy.
  • Synchronous accessors (getStorageMode, getSpilledCount, getMemoryUsage, …) are served from a mirror the worker pushes after every call, preserving their synchronous shape across the boundary.

No COOP/COEP headers are required: the design uses transferables, not SharedArrayBuffer, so it remains deployable to a plain static host (GitHub Pages). Macro / notebook workers are unchanged — their bridge_calls still route through proxyBridge to the (now possibly worker-hosted) runtime.

3.4 Python kernel — src/runtime/*.py

Modules pushed into Pyodide's virtual FS at load time. They are written to be re-executable so HMR keeps _MODEL / _CATALOG alive.

  • bootstrap.py — the kernel. Owns _MODEL (the hierarchical object store: panels → groups → objects), the dialog bridge, HDF5 serialisation, macro / notebook records, and ~150 helper functions consumed by DataLabRuntime. Logical groups:

    • Object CRUD: add_object_pickled, set_object_pickled, duplicate_object, delete_object, move_object[s], rename_object, add_signal_from_arrays, add_image_from_array.
    • Tree / groups: get_panel_tree, create_group, rename_group, delete_group, get_group_titles_with_object_info, resolve_group_oids.
    • Signals & images access: get_signal_xy, get_signals_xy, set_signal_style, get_object_stats, get_object_property_schema, set_object_property_values, metadata helpers.
    • Typed generators: list_signal_creation_types, create_signal_typed, update_signal_creation_params, idem image.
    • IO: list_signal_io_formats, open_signal_from_bytes, save_signal_to_bytes, idem image. HDF5 workspace serialisation (save_workspace_to_bytes / open_workspace_from_bytes, same layout as DataLab desktop, provenance block included). Opening is transactional: the file is fully read and validated before the current workspace is replaced, so an invalid file leaves it untouched.
    • Macros / notebooks: full CRUD + reorder + replace (used by the in-browser recovery cache when restoring from IndexedDB).
    • Dialog bridge: set_dialog_bridge(bridge) registers the JS callable used by Python helpers that need to ask the user something.

  • processor.py (loaded as dlw_processor.py) — introspects Sigima's catalog and exposes build_signal_catalog(), build_image_catalog() and the apply_* dispatchers. Equivalent of desktop DataLab's register_1_to_1 / register_n_to_1 / register_2_to_1. The pattern of a feature and the kind of its result come from the Sigima operation contract of its function (the only source of patterns, shared with DataLab desktop); override hooks hold interface choices only (labels, menus, icons, operand labels, preview) and allow renaming / hiding / regrouping entries or wiring custom result renderers.

  • dlw_main.py — exposes a datalab.* namespace inside Pyodide (panels, current panel, etc.) so that DataLab desktop plugins run unchanged in the browser, provided they use await param.edit_async(...) for parameter dialogs.

  • dlw_wheels.py — non-importing wheel inspector. It binds the rules of datalab.plugins.wheels (a verbatim copy of the DataLab desktop module, also used by the plugin catalog) to the dedicated datalab.web_plugins entry-point group: archive structure, pure-Python tags, Python/dependency compatibility and reserved namespaces.

  • dlw_plugins.py — the single managed plugin registry for bundled/user wheels and Python sources: discovery, activation, import ownership, hot-reload, cleanup, capabilities, recipes, examples, and menu wiring.

  • dlw_applications.py — generic application host. It prepares typed recipe slots, assesses recipe readiness on candidate objects and edited bindings (through the shim's datalab.plugins.recipe_binding, shared with Desktop), bridges guidata parameter schemas, opens packaged/generated examples, and commits recipe objects, results, diagnostics, and provenance transactionally. It contains no Camera- or Pulse-specific paths.

  • dlw_h5browser.py — HDF5 browser backend (consumed by H5BrowserDialog.tsx).

  • dlw_interactive_fit.py — interactive-fit backend (consumed by InteractiveFitDialog.tsx).

  • dlw_title_format.py — central title formatting for computed results (mirrors Sigima's title strategy).

  • dlw_provenance.py — workspace provenance ledger. Signal and image processing (1-to-1, including adopted previews, group-wise runs and in-place re-application from the Processing tab; 2-to-1; n-to-1) and analyses are recorded with fingerprints of their input and output states (analysis results as artifacts naming the object and metadata key), ROI definitions and uncertainty rows included, using the provenance recorder of DataLab-Capsule, shared with DataLab desktop (ledger, fingerprints, capture, replay preparation, verification, reports), and the Sigima description and preparation of calls. Every Sigima operation keeps its identity (operation contract and version), whether or not it is qualified for replay; only functions without a contract (plugin features) are opaque, and their inputs interpolated by the legacy X path are flagged x_interpolated. Sigima prepares the operations whose contract declares an X-alignment policy (the signal difference, and the historical policies of the other two-signal and multi-signal operations): the policy is applied once and recorded with the activity, with the global options read by the computation, and applied again on replay; an unknown policy is refused. Every input pair is aligned before any computation, so a refused pair computes nothing. A refusal by such a rule crosses the JS bridge as data (bridge_apply_feature and the other bridge_* entry points return {"refused": {"kind", "code", "reason"}}; runtime.ts throws a ScientificRefusalError, which the kernel worker forwards with its code), so its reason never depends on Pyodide formatting a Python traceback. replay_activity recomputes a recorded activity as a separate candidate, on copies of its inputs, and compares every output (signals, images, analysis results, absence of result) with the stored one under the comparison rule declared by the qualification of the operation, leaving the workspace unchanged; an operation that is not qualified is refused with its reason (unsupported_contract). DataLab-Capsule is optional: without it, processing is unchanged and get_provenance_ledger reports provenance as unavailable. Spilled (on-disk) arrays are never fingerprinted; capture failures never break processing. The ledger is saved in the workspace file (/DataLab_Provenance, shared with DataLab desktop) and reloaded on open, after checking every located object against its recorded fingerprint and ROI; analysis results record a digest so that a replaced result is reported as such. On open, the processing records (__processing_parameters, __analysis_parameters) of objects unknown to the ledger, written by DataLab desktop without provenance, become legacy_metadata activities with explicit unknowns and import issues (get_provenance_ledger()["legacy"]); they are never replayed. Appending a file (open_workspace_from_bytes(..., replace=False)) imports nothing: the current ledger is kept, the appended objects are passed to the recorder so that an object holding an identifier already known to the ledger (e.g. that of a deleted object) starts a new history instead of inheriting another object's records, and the result carries provenance_notice when the file held provenance (workspace_has_provenance tells beforehand). Processing-tab records are rebuilt from the ledger (Recorder.producing_call) for every catalogue processing (1-to-1, 2-to-1 with source and operand, n-to-1 with every source in order), whether or not its verification is qualified; a deleted input is kept under its UUID and reported as missing, and undecodable parameters keep the record with the reason. describe_object_provenance and describe_provenance_activity return the read-only description of DataLab-Capsule (datalab_capsule.inspection) with the current id and title of each named object, for the Provenance tab: consulting reads the ledger only, and checking the data compares resident arrays with their recorded states, reporting spilled arrays as unavailable (never paged in). export_workspace_capsule / open_workspace_capsule write and read .dlcapsule archives (workspace file plus an RO-Crate manifest of its provenance) through DataLab-Capsule; export_referenced_capsule returns a capsule referencing its workspace at an absolute IRI plus that workspace, which open_workspace_capsule then needs from the user (checked against the manifest, never fetched). inspect_workspace_capsule inspects a capsule without opening it: its integrity and the checks of supplied reports and reviews are returned apart, and nothing is computed. Inspection, opening, attachment download, registry checks (check_registry_revision), diff_capsule_manifests and create_capsule_review delegate to datalab_capsule.exchange, shared with the capsule dialog of DataLab desktop so that both editions give the same answers; this module only adds the browser size limits. The capsule a workspace was opened from is remembered (capsule_origin), saved with the workspace file and restored when it is opened again (replacing the workspace), so that a later export can name it as parent revision.

  • notebook_display.py — implements Jupyter-like display() and cell execution semantics inside the notebook worker.

  • macro_proxy.py — defines the async proxy object exposed to user scripts (macros and notebook cells). Each method ultimately becomes a bridge_call to the main thread.

  • guidata integration — guidata 3.15 provides the native async backend registry and JSON Schema exporter. bootstrap.py registers the browser dialog bridge through that backend API. _guidata_jsonschema_shim.py is a small temporary backport shim that only adds the FloatArrayItem variable-size and min/max hints still missing upstream. It is declared in src/runtime/shims/registry.ts; see shim-registry.md for the audit and removal workflow.

3.5 SDK — packages/sdk/

Shipped as a separate tarball (@datalab-platform/web-sdk). Provides typed TypeScript clients for host pages embedding DataLab-Web in an iframe. Talks to remoteBridge.ts over the wire protocol described above. Its package.json version must stay aligned with the app's.

3.6 Persistence — src/storage/

  • recentStore.ts — IndexedDB-backed cache for the recent-files menu and for crash recovery. Not durable in the project sense: the HDF5 workspace file is the single source of truth.
  • capsuleRevisions.ts — flat IndexedDB list of the capsule revisions exported or inspected in this browser (manifest fingerprint, parent, and the manifest itself, so that two revisions can be compared without their files). A local convenience, never a source of truth.
  • capsuleRegistry.ts — optional HTTP client of a DataLab-Capsule registry (fetch, no Pyodide) and its settings. The registry is never trusted: each response is read up to the limits of DataLab-Capsule's client (REGISTRY_LIMITS; an attachment up to its declared size), manifests and reviews go through the runtime checks, attachments through a size and SHA-256 check.

4. Wire protocols

4.1 Worker ↔ main (proxyBridge)

Implemented in macroWorker.ts / notebookWorker.ts ↔ proxyBridge.ts.

main → worker
  { type: "init" }
  { type: "run", code: string, name?: string }
  { type: "bridge_reply", id: string, ok: boolean, value?: any, error?: string }

worker → main
  { type: "ready" }
  { type: "started", name: string }
  { type: "finished", ok: boolean, error?: string }
  { type: "stdout" | "stderr", text: string }
  { type: "bridge_call", id: string, method: string, payload: any }

method is matched against an explicit allow-list in proxyBridge.ts; unknown methods produce an error reply.

4.2 Iframe ↔ host (remoteBridge)

Small JSON-RPC dialect, versioned (RPC_PROTOCOL_VERSION).

host → iframe       { id, type: "request",  method, params? }
iframe → host       { id, type: "response", result? | error? }
iframe → host       {     type: "event",    name, payload?   }

Synthetic method get_protocol_version lets clients negotiate compatibility independently of the application version.

5. End-to-end flows

5.1 Applying a processing

1. User clicks  Processing › <feature>     in MenuBar
2. action.run(ctx)  (from src/actions/registry.ts)
3. If params needed → ProcessingDataSetDialog (auto-generated from JSON schema)
4. Optional Preview → runtime.previewFeature(featureId, sourceOid, paramValues)
5. bootstrap deep-copies the source and returns a private plotting payload
6. OK detaches a current single-source preview token, if available, and calls runtime.applyFeature(featureId, selectedOids, paramValues, token)
7. bootstrap consumes a matching result once, or calls processor.apply_*(feature, oids, params)
8. The result is published as a new SignalObj/ImageObj in _MODEL; the execution is also recorded in the provenance ledger
9. bootstrap returns new oids; UI refreshes ObjectTree + plots
10. WorkspaceContext is marked dirty

5.2 Running a macro

1. User clicks  Run  in MacroPanel
2. MacroRuntime spawns / reuses a macroWorker
3. main → worker: { type: "run", code, name }
4. worker executes user code; calls `await proxy.<method>(…)`
5. macro_proxy.py emits  { type: "bridge_call", … }
6. proxyBridge dispatches to a whitelisted DataLabRuntime call
7. main → worker: { type: "bridge_reply", … }
8. stdout / stderr stream back to MacroConsole
9. worker → main: { type: "finished", ok, error? }

5.3 Remote control from a host page

1. Host imports @datalab-platform/web-sdk and embeds the iframe
2. Host calls  client.addSignal(x, y, title)
3. SDK posts   { id, type: "request", method: "add_signal", params }
4. remoteBridge.ts dispatches to DataLabRuntime (same allow-list spirit)
5. iframe posts  { id, type: "response", result }
6. Model changes also surface as  { type: "event", name, payload }

6. Testing layers

The architecture is mirrored by three test layers — see doc/testing-strategy.md for the full decision tree.

Layer Tool Scope
Python pytest (in Pyodide) bootstrap.py, processor.py, dlw_*.py logic
TS unit Vitest + RTL runtime types, action registry, components
End-to-end Playwright full UI through real Pyodide round-trips

7. Conventions recap

  • Type-safety end-to-end: never widen a Pyodide return value beyond its declared TS type. Add new fields in the Python helper and the TS interface in the same commit.
  • No business logic in components: orchestration lives in src/actions/ and the runtime.
  • JSON across the bridge: prefer plain dicts/lists in Python over passing PyProxies.
  • base: "./" in Vite config — the build is drop-in deployable to GitHub Pages and other sub-path hosts. Do not introduce absolute URLs.
  • Pinned Pyodide: PYODIDE_VERSION in runtime.ts and the <script> tag in index.html must always be bumped together.