Repository navigation
Fix data-loss, HDF5 import and origin issues found in the October 2026 review - #65
Merged
Merged
Conversation
… file A soft link to a missing object or an external link to a missing file (e.g. synchrotron master files without their detector files) made open_file fail with AttributeError: the children were iterated with values(), which yields None for unresolved links, and only some errors were caught. Valid datasets of such files could not be browsed. Children are now iterated by name; broken links are skipped with a diagnostic naming the missing target, like in DataLab desktop (SigimaX). open_file also closes the file and removes its temporary copy when the tree cannot be built, instead of only when h5py cannot open it. Assisted-by: Claude Opus 5.5
…them An HDF5 file is a graph: a group can be linked into itself or into one of its subgroups. The browser walked such links until the depth limit (100), building phantom levels such as /entry/self/self/... and classifying their datasets again at each level; with several such links, the walk grew exponentially and the file could not be opened in practice. _build_node now passes the chain of parent groups and skips, with a diagnostic, any child group identical (same file and object) to one of them, as DataLab desktop (SigimaX) does. Other aliases of a group or dataset are still listed at each of their paths. The depth limit remains as a secondary guard. Assisted-by: Claude Opus 5.5
…a file Opening an HDF5 file read every dataset entirely to classify it and build its one-line description, including datasets that cannot be imported (e.g. a 3D cube) or that the user never selects: in WebAssembly, these allocations could make a large file unusable although only a small dataset was wanted (the file itself is already copied to memory). Datasets are now classified from their shape, type and a sample of at most 100 leading elements (the same number along each axis); shapes and signal or image detection use the dataset shape. Mixed compound datasets are checked on their first records only, as in DataLab desktop. Full data is still read on preview, array display and import. A test checks that the nodes classified from samples match those classified from full reads. Assisted-by: Claude Opus 5.5
…HDF5 data The HDF5 browser converted every signal that was neither float64 nor complex128 to float64, so that complex64 signals lost their imaginary part (with only a ComplexWarning), and it clipped uint32 images to the int32 range (2 147 483 647), losing larger counts that float64 represents exactly. DataLab desktop fixed both in 1.4, but DataLab-Web keeps its own adapter. Signals are now converted to complex128 when complex, to float64 otherwise. Images follow the rules of Sigima >= 1.4 (smallest image integer type holding the values, else an exact floating-point type; complex64 to complex128), implemented locally because DataLab-Web still uses Sigima 1.3, whose converter rounds large integers to float32. 2D signal tables with complex X values or uncertainties are rejected instead of truncated. The uint32 clipping warning is replaced by a precision warning (inexact_values), shown when integers beyond 2**53 cannot be represented exactly; French translation added. Previews show the real part of complex data, which cannot be sent to the JavaScript preview. Assisted-by: Claude Opus 5.5
In on-disk storage mode, spilling an object serialized its array and replaced it with a placeholder in Python before the bytes were written to OPFS. When the write failed (quota exceeded, write error...), nothing attached the bytes back: the only copy of a new or changed object was lost, Python considered it spilled while the JavaScript registry did not. The rewrite of a paged-in object changed by a call had the same flaw. The bytes are now attached back when they cannot be persisted, before the error is propagated: the object stays resident with its latest values and both registries agree it is not spilled. attach_object_array gets an on_disk flag so that such an object is serialized again (never just released) when it is next returned to disk, and a partially written copy is deleted. The asynchronous OPFS store also aborts a failed write instead of committing it over the previous copy. Assisted-by: Claude Opus 5.5
All on-disk object stores of an origin used the same OPFS directory (dlw-object-store), and clearing a store removed that whole directory. With two tabs in on-disk storage mode, switching one back to RAM or replacing its workspace deleted the arrays spilled by the other tab, whose objects then could no longer be paged in. Each store now keeps its files in its own session directory under dlw-object-sessions, owned through an exclusive Web Lock held for the lifetime of the page or worker; clear() only resets that directory. When a store is opened, the directories of ended sessions (lock free: closed tab, crash) are removed; the lock of a new session is acquired before its directory is created, so that it is never taken for an abandoned one. Without the Web Locks API, other directories are left in place. The shared directory of earlier versions is not touched, since an earlier version still open in another tab may use it. Assisted-by: Claude Opus 5.5
The SDK accepted every message whose source was the iframe window, without
checking its origin: a window keeps its identity when it navigates, so a
document of another origin loaded in the iframe could resolve pending
requests or emit events. The bridge, for its part, broadcast its events to
the known host windows with the "*" target origin, so that a host window
that had navigated to another document still received them.
The SDK now also requires the origin to be targetOrigin (any origin with
"*", for local development only). The bridge records the origin of the
requests of each host window and sends its responses and events to that
origin only; opaque origins ("null") are only targeted through "*", which
only the "*" wildcard of allowedOrigins accepts. Documented in the SDK
README.
Assisted-by: Claude Opus 5.5
Add the user-visible fixes of this branch to the [Unreleased] section (on-disk storage write failures and multi-tab isolation, HDF5 complex and integer import, broken links, link cycles and large files, embedding message origins). Replace the absolute "your data never leaves your machine" claim of the README by a precise statement: processing never requires sending data anywhere, and the network services that can be enabled (AI assistant provider, capsule registry) are named. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from the cross-repository review of the develop branches (2026-10-24), one commit per finding, each with regression tests.
AttributeError, file unusable);open_filecloses the file and removes its temporary copy on any tree-building error.uint32_clippedreplaced byinexact_values(integers beyond 2⁵³) end-to-end, with i18n. Previews send the real part of complex data.putfails (first spill and rewrite), withon_disk=Falseso the object is re-serialized before any later release; a partial copy is deleted. The async store aborts a failed write instead of committing it.dlw-object-sessions, owned by an exclusive Web Lock;clear()resets only that directory; ended sessions' directories (lock free) are reclaimed; the legacy shared directory is left untouched.event.origin === targetOrigin(unless"*"); the bridge records each peer's request origin and posts responses/events to it (opaque origins only via the"*"wildcard).Validation
530 passed(with local Sigima develop). With the pinned Sigima 1.3.0 wheel: 528 passed, 2 failures intest_provenance*.pythat also fail onorigin/developwith the current local DataLab-Capsule (pre-existing, unrelated).107 files, 892 tests passed;npm run lint,format:check,i18n:check,npm run build(initial JS 388.3 KiB gzip) andsdk:buildpass.