Fix on-disk storage robustness and other pre-release review findings

CHANGELOG.md

@@ -17,6 +17,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 - **On-disk storage — failed writes:** when the browser refused to write an object to disk (storage quota exceeded, write error), the object lost its data: it had already been removed from memory. The object now stays in memory with its data, and the error is reported. An interrupted write no longer replaces the previous on-disk copy either.
 - **On-disk storage — incomplete transfers:** an object whose data the browser wrote to disk only partly was considered stored and removed from memory, so that it was later read back with missing values; a partial read also gave missing values without error. Incomplete writes now fail like other write errors, so that the object keeps its data in memory, and incomplete reads are reported as errors.
 - **On-disk storage — several tabs:** with DataLab-Web open in two tabs in on-disk storage mode, switching one tab back to memory storage or replacing its workspace deleted the data stored on disk by the other tab, whose objects could then no longer be displayed or processed. Each tab now keeps its data in its own storage area; the areas of closed tabs are reclaimed when the on-disk mode is next enabled.
+- **On-disk storage — enabling the mode:** when **Store data on disk** (in the menu of the memory indicator) could not be enabled (storage refused by the browser, as in the private windows of some browsers, or data that could not be written), DataLab-Web was left halfway: the toggle still showed memory storage, some objects were kept on disk, later operations were reported as failed although they had been applied, and a further attempt reported the mode as enabled while nothing was stored on disk. A failed attempt now leaves every object in memory, as before, and the mode can be enabled again once the cause is fixed.
+- **On-disk storage — operations reported as failed:** in on-disk storage mode, when the browser refused to write the result of an operation to disk (storage quota exceeded, write error), the operation was reported as failed although it had been applied, so that running it again applied it twice. The operation now succeeds: the objects that could not be written stay in memory with their data, a warning reports the storage error once, and these objects are stored on disk by a later operation when possible.
 - **HDF5 import — complex and integer data:** single-precision complex signals (`complex64`) lost their imaginary part, and unsigned 32-bit images were clipped to 2 147 483 647. Complex data is now imported as complex objects and integer data keeps its values exactly (images with large values become 64-bit floating-point images); a warning now reports integers beyond 2⁵³, which cannot be represented exactly. Previews show the real part of complex data.
 - **HDF5 browser — broken links:** files containing a link to a missing object or to a missing file (e.g. synchrotron master files without their detector files) could not be opened at all. Broken links are now skipped and the rest of the file can be browsed and imported.
 - **HDF5 browser — link cycles:** files containing a group linked into itself or into one of its subgroups were browsed down to a hundred phantom levels, or took too long to open. Such links back to a parent group are now skipped; other aliases are still shown at each of their paths.
@@ -26,7 +28,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 - **Image viewer — pixel positions:** images were drawn half a pixel off their coordinates: the first pixel started at the image origin instead of being centered on it, as in Sigima and DataLab desktop. Regions of interest, analysis results, the profile crosshair and the statistics rectangle therefore appeared shifted by half a pixel relative to the computed pixels, and the value shown when hovering the image could be that of the neighbouring pixel. Each pixel is now centered on its coordinates in the image viewer, the multi-image views and the processing preview, so overlays, read-outs, cross-sections and statistics match the computations exactly. The default regions of interest and areas to erase are centered on the image, and the preview of _Create ROI grid…_ now shows the same whole-pixel cells as the created regions of interest. Cross-sections and statistics also work on images with non-uniform coordinates.
 - **Image viewer — blurred pixels when zooming in:** since version 0.6.1, zooming in on an image showed blurred pixels, each one fading into its neighbours, so that close values could not be told apart. Zoomed-in pixels are now drawn as sharp squares (nearest pixel, as in DataLab desktop) in the image viewer, the _Spatial_ multi-image view and the processing preview, in every browser and in the images exported from the plot. Zoomed-out images are displayed as before.
 - **AI Assistant — Save to Macros:** since version 0.10.0, clicking **Save to Macros** under a macro proposed by the AI Assistant showed "Saved ✓", but the macro did not appear in the Macros panel; each click added another hidden macro to the workspace. The saved macro now opens in the Macros panel, which is brought to the front, including from a conversation reopened from the history.
-- **AI Assistant — macros written as text:** asked to write a macro, the assistant could answer with the code in its reply, without any **Save to Macros** button, in particular for curves its creation tools cannot produce (super-Gaussian, Lorentzian, polynomial…), which it was told to create without a macro. The assistant now proposes such macros through its macro tool, without running them unless asked (the approval dialog says so, and approving such a macro never auto-approves a macro that runs), and a **Save to Macros** button is also offered under each Python code block of its replies.
+- **AI Assistant — macros written as text:** asked to write a macro, the assistant could answer with the code in its reply, without any **Save to Macros** button, in particular for curves its creation tools cannot produce (super-Gaussian, Lorentzian, polynomial…), which it was told to create without a macro. The assistant now proposes such macros through its macro tool, without running them unless asked (the approval dialog says so, and approving such a macro never auto-approves a macro that runs), and a **Save to Macros** button is also offered under each Python code block of its replies; the macros saved from a reply with several code blocks are numbered after the request they answer (e.g. _Plot two curves (2)_), so they can be told apart in the Macros panel.
 - **AI Assistant — macro output:** the assistant received an empty result when reading back the output of the last macro it ran.
 
 ### Changed

src/App.tsx

@@ -1151,6 +1151,23 @@ export default function App() {
     return unsubscribe;
   }, [runtime, markDirty]);
 
+  // In on-disk storage mode, objects that could not be stored on disk after
+  // an operation (storage quota exceeded, write error) stay in memory: the
+  // operation succeeded, the storage error is reported once.
+  useEffect(() => {
+    if (!runtime) return;
+    return runtime.onStorageError((message) => {
+      void notify({
+        kind: "warning",
+        title: t("Data storage"),
+        message: t(
+          "Some data could not be stored on disk and is kept in memory: {message}",
+          { message },
+        ),
+      });
+    });
+  }, [runtime, notify]);
+
   // Sync the "Paste metadata" availability from the Python clipboard
   // once the runtime is ready. The Python ``_METADATA_CLIPBOARD`` is
   // preserved across HMR reloads, so the React flag (which resets to
@@ -2282,6 +2299,9 @@ export default function App() {
           : t("On-disk storage disabled. Object data is kept in memory."),
       });
     } catch (err) {
+      // The runtime normally stays in its previous mode, but reports the
+      // mode it actually ended in (see ``setStorageMode``).
+      setStoreOnDisk(runtime.getStorageMode() === "disk");
       const message = err instanceof Error ? err.message : String(err);
       await notify({
         kind: "error",

src/components/AIAssistant/AIAssistantPanel.tsx

@@ -950,7 +950,8 @@ function CodeBlockSaveButtons({
       {blocks.map((code, i) => (
         <SaveMacroButton
           key={i}
-          title={title}
+          // Distinct titles for the macros of a reply with several blocks
+          title={blocks.length > 1 ? `${title} (${i + 1})` : title}
           code={code}
           onSaveMacro={onSaveMacro}
           label={

src/locales/fr.json

@@ -1199,6 +1199,7 @@
   "Sobel filter (vertical)": "Filtre de Sobel (vertical)",
   "Software": "Logiciel",
   "Solid": "Continu",
+  "Some data could not be stored on disk and is kept in memory: {message}": "Des données n'ont pas pu être stockées sur le disque et sont conservées en mémoire : {message}",
   "Some integer values could not be represented exactly and were rounded during import.": "Certaines valeurs entières ne pouvaient pas être représentées exactement et ont été arrondies lors de l'import.",
   "Some selected objects have regions of interest. Do you want to delete them as well?": "Certains objets sélectionnés ont des régions d'intérêt. Souhaitez-vous les supprimer également ?",
   "Source": "Source",

src/runtime/WorkerRuntimeProxy.ts

@@ -13,9 +13,9 @@
  *
  *   * **Synchronous accessors** (``getStorageMode`` etc.) are served from a
  *     locally-cached {@link KernelMirror} that the worker pushes on change.
- *   * **Callback registrars** (``setDialogHandler``, ``onWorkspaceMutation``)
- *     keep the JS callback on the main thread and bridge the worker's events
- *     to it — a function can't cross ``postMessage``.
+ *   * **Callback registrars** (``setDialogHandler``, ``onWorkspaceMutation``,
+ *     ``onStorageError``) keep the JS callback on the main thread and bridge
+ *     the worker's events to it — a function can't cross ``postMessage``.
  *   * Every other method is forwarded generically through a ``Proxy`` trap,
  *     so the ~150-method surface needs no hand-written forwarders and any
  *     new ``RuntimeApi`` method works automatically.
@@ -65,6 +65,7 @@ export class WorkerRuntimeProxy {
   private readonly pending = new Map<number, PendingCall>();
   private mirror: KernelMirror = INITIAL_MIRROR;
   private readonly mutationListeners = new Set<(name: string) => void>();
+  private readonly storageErrorListeners = new Set<(message: string) => void>();
   private dialogHandler:
     ((kind: string, payload: unknown) => Promise<unknown>) | null = null;
 
@@ -173,14 +174,18 @@ export class WorkerRuntimeProxy {
         }
         if (prop === "onWorkspaceMutation") {
           return (listener: (name: string) => void) =>
-            this.addMutationListener(listener);
+            this.addListener(this.mutationListeners, listener);
+        }
+        if (prop === "onStorageError") {
+          return (listener: (message: string) => void) =>
+            this.addListener(this.storageErrorListeners, listener);
         }
         // Generic async forward for every other method.
         return (...args: unknown[]) => this.call(prop, args);
       },
     };
     // The Proxy structurally satisfies RuntimeApi: every access yields a
-    // function with the right shape (sync accessors, the two callback
+    // function with the right shape (sync accessors, the callback
     // registrars, or a generic async forward).
     return new Proxy({}, handler) as unknown as RuntimeApi;
   }
@@ -219,13 +224,31 @@ export class WorkerRuntimeProxy {
     });
   }
 
-  private addMutationListener(listener: (name: string) => void): () => void {
-    this.mutationListeners.add(listener);
+  private addListener<T>(
+    listeners: Set<(value: T) => void>,
+    listener: (value: T) => void,
+  ): () => void {
+    listeners.add(listener);
     return () => {
-      this.mutationListeners.delete(listener);
+      listeners.delete(listener);
     };
   }
 
+  /** Call every listener of *listeners* with *value*, logging their errors. */
+  private notify<T>(
+    listeners: Set<(value: T) => void>,
+    value: T,
+    kind: string,
+  ): void {
+    for (const cb of listeners) {
+      try {
+        cb(value);
+      } catch (err) {
+        console.warn(`[workerRuntime] ${kind} listener threw`, err);
+      }
+    }
+  }
+
   private handleEvent(event: KernelEvent): void {
     switch (event.type) {
       case "progress":
@@ -255,13 +278,10 @@ export class WorkerRuntimeProxy {
         break;
       }
       case "mutation":
-        for (const cb of this.mutationListeners) {
-          try {
-            cb(event.name);
-          } catch (err) {
-            console.warn("[workerRuntime] mutation listener threw", err);
-          }
-        }
+        this.notify(this.mutationListeners, event.name, "mutation");
+        break;
+      case "storage-error":
+        this.notify(this.storageErrorListeners, event.message, "storage-error");
         break;
       case "dialog-request":
         void this.handleDialogRequest(event.id, event.kind, event.payload);

src/runtime/dlw_h5browser.py

@@ -426,10 +426,10 @@ def _build_node(
                     )
                     continue
             try:
+                # Path in the opened file: ``child_dset.name`` is the path of an
+                # object reached through an external link in its own file
                 info.children.append(
-                    _build_node(
-                        file_id, child_dset.name, child_dset, _depth + 1, ancestors
-                    )
+                    _build_node(file_id, child_path, child_dset, _depth + 1, ancestors)
                 )
             except (UnicodeDecodeError, TypeError, ValueError, KeyError) as exc:
                 print(f"[h5browser] skip {child_path!r}: {type(exc).__name__}: {exc}")

src/runtime/kernelWorker.ts

@@ -84,6 +84,8 @@ async function boot(lang: string, labels: RuntimeLabels): Promise<void> {
     );
     // Forward workspace mutations so the main thread can flip its dirty flag.
     rt.onWorkspaceMutation((name) => post({ type: "mutation", name }));
+    // Forward on-disk storage errors, which the main thread reports.
+    rt.onStorageError((message) => post({ type: "storage-error", message }));
     // Route Python dialog requests up to the main thread, which owns the UI.
     rt.setDialogHandler(
       (kind, payload) =>

src/runtime/remoteBridge.ts

@@ -21,6 +21,7 @@
  *
  * Responses with binary payloads (e.g. ``Float64Array``) attach those
  * buffers as ``Transferable`` to ``postMessage`` for zero-copy transfer.
+ * Events, sent to every peer, are cloned instead.
  */
 
 import { buildProxyBridge, type BridgeMethod } from "./proxyBridge";
@@ -184,14 +185,18 @@ export function activateRemoteBridge(
         target: MessageEventSource,
         origin: string,
         msg: OutboundMessage,
+        transferBuffers?: boolean,
       ) => void)
     | null = null;
 
+  // Events are cloned, never transferred: they go to every peer (a buffer
+  // transferred to the first one would arrive empty at the others), and
+  // their payload belongs to the caller, whose buffers must stay usable.
   const flushEvents = () => {
     if (!post || queuedEvents.length === 0 || peers.size === 0) return;
     const drained = queuedEvents.splice(0);
     for (const [peer, origin] of peers) {
-      for (const evt of drained) post(peer, origin, evt);
+      for (const evt of drained) post(peer, origin, evt, false);
     }
   };
 
@@ -237,8 +242,13 @@ export function activateRemoteBridge(
     get_protocol_version: async () => RPC_PROTOCOL_VERSION,
   };
 
-  post = (target: MessageEventSource, origin: string, msg: OutboundMessage) => {
-    const transfer = collectTransferables(msg);
+  post = (
+    target: MessageEventSource,
+    origin: string,
+    msg: OutboundMessage,
+    transferBuffers = true,
+  ) => {
+    const transfer = transferBuffers ? collectTransferables(msg) : [];
     const targetOrigin = peerTargetOrigin(origin);
     try {
       // ``MessageEventSource`` has a narrower type than ``Window`` but
@@ -312,7 +322,7 @@ export function activateRemoteBridge(
     },
     emit(name, payload) {
       const evt: RpcEvent = { type: "event", name, payload };
-      for (const [peer, origin] of peers) post(peer, origin, evt);
+      for (const [peer, origin] of peers) post(peer, origin, evt, false);
     },
   };
 }