Skip to content

Integrate experimental data: NeXus files, robust HDF5 import and threaded plugin instruments - #422

Merged
PierreRaybaut merged 26 commits into
developfrom
feature/experimental-data-integration
Oct 23, 2026
Merged

PierreRaybaut merged 26 commits into
developfrom
feature/experimental-data-integration

Conversation

@PierreRaybaut

Copy link
Copy Markdown
Contributor

Summary

Closes #421.

Last of three companion PRs for experimental data integration; merge it after:

  1. Record the origin of experimental data and import large integers exactly Sigima#78 — experimental-source API, exact integer conversion, add_group UUID (Sigima 1.4.0)
  2. Browse and import large HDF5 and NeXus experimental files SigimaX#13 — HDF5/NeXus browsing and import
  3. This PR

This branch requires Sigima ≥ 1.4.0 and SigimaX ≥ 1.1.0. The development CI clones the Sigima and SigimaX branches of the same name.

Changes

  • NeXus files (h5/): .nxs/.nx5 proposed in HDF5 dialogs; NeXus NXdata groups imported through the SigimaX browser, with their experimental_source kept in workspaces (types and axes checked after reload).
  • Robust HDF5 import (h5io.py): the other datasets are imported when one fails, failures listed in one message (or raised as ValueError when unattended); HDF5 files always closed; obsolete uint32 clipping warning removed.
  • Threaded plugin instruments (gui/plugins/instrument.py, gui/main.py): calls run one at a time in a per-instrument SerialWorker shared across window reopenings, refresh requests merged, optional open/close lifecycle, open() errors shown and retried, Live mode kept off when unchecked during an acquisition, windows closed while hidden replaced, and a common time limit for closing all instruments when plugins are reloaded or DataLab quits.
  • Remote control: add_group returns the UUID of the created group (main window, LocalProxy, XML-RPC server and client).
  • Release notes 1.4, HDF5 browser and plugin guides, French catalogs; minimum versions Sigima ≥ 1.4.0, SigimaX ≥ 1.1.0.

Validation

  • datalab/tests/features/hdf5, instrument_window_unit_test.py, plugin_tools_app_test.py, remote_requests_unit_test.py, main_app_test.py: 66 passed.
  • Ruff check/format clean, Pylint 10/10 on changed files.

Assisted-by: Claude Opus 5.5
Assisted-by: Claude Opus 5.5
Document NeXus import, experimental origin and preview limits, clarify instrument connector responsibilities, update French translations and require Sigima 1.4.

Assisted-by: GPT-6.1 Sol
DataLab forgot an instrument window as soon as it was closed, while its
worker could still be running a slow preview or acquisition. Opening the
tool again created a second worker calling the same instrument in parallel
(open and preview during the running call), and the old worker then closed
the instrument the new window had just opened.

- DLMainWindow keeps one worker per instrument, shared by its successive
  windows: a reopened window opens the instrument once the closed one has
  closed it, in the same thread.
- Reloading plugins or quitting waits for the instruments of windows already
  closed by the user, then stops the workers.
- Closing a window that was never shown now closes its instrument.

Assisted-by: Claude Opus 5.5
…t call

The first refresh of an instrument window was scheduled independently of
open(): when open() failed, preview() was still called on the instrument,
and its result cleared the message of open() (or its own error replaced
it), although PluginInstrument.open documents a user-facing error.

- The first preview is requested once open() succeeded.
- When open() fails, its message stays in view and the instrument is not
  called; the next request (e.g. after a settings change, or in live mode)
  opens it again before calling preview() or acquire().

Assisted-by: Claude Opus 5.5
…space reload

- The NeXus workspace test imports a detector image with a decreasing axis
  and a dimension without axis, and checks after reloading the workspace
  that the image is reversed, that "." is kept for the missing axis, and
  that the experimental source keeps its JSON types.
- Documentation and release notes (English and French): instrument calls
  across window reopenings and open() failures, NeXus axis handling,
  decimated NeXus/mapped previews, and the complex64 HDF5 import fix.

Assisted-by: Claude Opus 5.5
Importing several datasets from the HDF5 browser stopped at the first
dataset that could not be imported (e.g. a NeXus image axis that is not
monotonic), so the following datasets were silently skipped. The other
datasets are now imported, and the errors are reported in a single
message at the end (raised in unattended mode, as before).

Assisted-by: Claude Opus 5.5
- remote.po: translate the new "UUID of the created group" entry (add_group
  now returns the UUID of the created group).
- installation.po: update the requirement versions (guidata, Sigima,
  SigimaX) and add the datalab-capsule entries.
- proxy.po: synchronize the API reference entries (add_group, capsule
  export/open); this catalog is not translated, as on develop.

Assisted-by: Claude Opus 5.5
When datasets could not be imported in unattended mode, or when opening a
file in the browser failed, the error was raised before the browser was
cleaned up: the HDF5 files stayed open (locked on Windows) and the preview
worker kept running. `import_dataset_from_file` did not close its file on
errors either. Both now close their files in a `finally` clause.

The NeXus import test checks that the file is closed after errors, and no
longer triggers Pylint's unbalanced-tuple-unpacking and unidiomatic-typecheck
messages.

Assisted-by: Claude Opus 5.5
…tion

Live mode was resumed after an acquisition whenever it was on when the
acquisition started, even if the user unchecked the Live button meanwhile.
It is now resumed only if the button is still checked.

Assisted-by: Claude Opus 5.5
Qt does not emit `finished` when a hidden dialog is closed, so such a
window stayed registered and was shown again, with its instrument already
closed, when the tool was launched again. It is now released and replaced
by a new window.

Assisted-by: Claude Opus 5.5
Release notes and HDF5 browser guide, with French translations: files with
broken links can be opened, and selecting an item that cannot be previewed
clears the preview.

Assisted-by: Claude Opus 5.5
Instruments are waited for together when plugins are reloaded or DataLab quits, so that several stuck instruments no longer add up their time limits.

Assisted-by: Claude Opus 5.5
@PierreRaybaut PierreRaybaut added the enhancement New feature or request label Oct 23, 2026
@PierreRaybaut
PierreRaybaut merged commit bee78fd into develop Oct 23, 2026
6 checks passed
@PierreRaybaut
PierreRaybaut deleted the feature/experimental-data-integration branch October 23, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant