Skip to content

Commit 1c8d287

Browse files
Merge pull request #444 from DataLab-Platform/feature/hdf5-nd-views
HDF5 browser: multidimensional views and legacy NeXus scans (tests and docs)
2 parents 1afe5c7 + ef1eaa5 commit 1c8d287

7 files changed

Lines changed: 313 additions & 120 deletions

File tree

‎.github/workflows/test_pyqt5.yml‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -77,8 +77,9 @@ jobs:
7777
pip install -e ../DataLab-Capsule
7878
# Install tomli for TOML parsing (safe if already present)
7979
pip install tomli
80-
# Extract dependencies and save to file, then install
81-
python -c "import tomli; f=open('pyproject.toml','rb'); data=tomli.load(f); deps=[d for d in data['project']['dependencies'] if not any(p in d for p in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
80+
# Extract dependencies of DataLab and SigimaX (installed with --no-deps
81+
# above), save them to file, then install
82+
python -c "import tomli; deps=[d for p in ('pyproject.toml', '../SigimaX/pyproject.toml') for d in tomli.load(open(p,'rb'))['project']['dependencies'] if not any(n in d for n in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
8283
pip install -r deps.txt
8384
# Install DataLab without dependencies
8485
pip install --no-deps .
@@ -105,8 +106,9 @@ jobs:
105106
pip install -e ../DataLab-Capsule
106107
# Install tomli for TOML parsing (safe if already present)
107108
pip install tomli
108-
# Extract dependencies and save to file, then install
109-
python -c "import tomli; f=open('pyproject.toml','rb'); data=tomli.load(f); deps=[d for d in data['project']['dependencies'] if not any(p in d for p in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
109+
# Extract dependencies of DataLab and SigimaX (installed with --no-deps
110+
# above), save them to file, then install
111+
python -c "import tomli; deps=[d for p in ('pyproject.toml', '../SigimaX/pyproject.toml') for d in tomli.load(open(p,'rb'))['project']['dependencies'] if not any(n in d for n in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
110112
pip install -r deps.txt
111113
# Install DataLab without dependencies
112114
pip install --no-deps .
@@ -186,9 +188,10 @@ jobs:
186188
python -m pip install -e ../SigimaX --no-deps
187189
python -m pip install -e ../DataLab-Capsule
188190
189-
# Extract dependency names from pyproject.toml (excluding Sigima and
190-
# SigimaX, installed from local clones above) and install latest versions
191-
python -m pip install -U --upgrade-strategy eager $(python -c "import tomllib, re; print(' '.join(re.sub(r'[\[\]<>=!~,.\s].*$', '', d).strip() for d in tomllib.loads(open('pyproject.toml', 'rb').read().decode())['project']['dependencies'] if not any(p in d for p in ['Sigima', 'SigimaX', 'datalab-capsule'])))")
191+
# Extract dependency names from the pyproject.toml of DataLab and SigimaX
192+
# (excluding Sigima and SigimaX, installed from local clones above) and
193+
# install latest versions
194+
python -m pip install -U --upgrade-strategy eager $(python -c "import tomllib, re; print(' '.join(sorted({re.sub(r'[\[\]<>=!~,.\s].*$', '', d).strip() for p in ('pyproject.toml', '../SigimaX/pyproject.toml') for d in tomllib.loads(open(p, 'rb').read().decode())['project']['dependencies'] if not any(n in d for n in ['Sigima', 'SigimaX', 'datalab-capsule'])})))")
192195
193196
- name: Lint with Ruff (latest)
194197
run: ruff check --output-format=github datalab

‎.github/workflows/test_pyqt6.yml‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -77,8 +77,9 @@ jobs:
7777
pip install -e ../DataLab-Capsule
7878
# Install tomli for TOML parsing (safe if already present)
7979
pip install tomli
80-
# Extract dependencies and save to file, then install
81-
python -c "import tomli; f=open('pyproject.toml','rb'); data=tomli.load(f); deps=[d for d in data['project']['dependencies'] if not any(p in d for p in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
80+
# Extract dependencies of DataLab and SigimaX (installed with --no-deps
81+
# above), save them to file, then install
82+
python -c "import tomli; deps=[d for p in ('pyproject.toml', '../SigimaX/pyproject.toml') for d in tomli.load(open(p,'rb'))['project']['dependencies'] if not any(n in d for n in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
8283
pip install -r deps.txt
8384
# Install DataLab without dependencies
8485
pip install --no-deps .
@@ -105,8 +106,9 @@ jobs:
105106
pip install -e ../DataLab-Capsule
106107
# Install tomli for TOML parsing (safe if already present)
107108
pip install tomli
108-
# Extract dependencies and save to file, then install
109-
python -c "import tomli; f=open('pyproject.toml','rb'); data=tomli.load(f); deps=[d for d in data['project']['dependencies'] if not any(p in d for p in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
109+
# Extract dependencies of DataLab and SigimaX (installed with --no-deps
110+
# above), save them to file, then install
111+
python -c "import tomli; deps=[d for p in ('pyproject.toml', '../SigimaX/pyproject.toml') for d in tomli.load(open(p,'rb'))['project']['dependencies'] if not any(n in d for n in ['guidata','PlotPy','Sigima', 'SigimaX', 'datalab-capsule'])]; open('deps.txt','w').write('\n'.join(deps))"
110112
pip install -r deps.txt
111113
# Install DataLab without dependencies
112114
pip install --no-deps .
@@ -166,9 +168,10 @@ jobs:
166168
python -m pip install -e ../SigimaX --no-deps
167169
python -m pip install -e ../DataLab-Capsule
168170
169-
# Extract dependency names from pyproject.toml (excluding Sigima and
170-
# SigimaX, installed from local clones above) and install latest versions
171-
python -m pip install -U --upgrade-strategy eager $(python -c "import tomllib, re; print(' '.join(re.sub(r'[\[\]<>=!~,.\s].*$', '', d).strip() for d in tomllib.loads(open('pyproject.toml', 'rb').read().decode())['project']['dependencies'] if not any(p in d for p in ['Sigima', 'SigimaX', 'datalab-capsule'])))")
171+
# Extract dependency names from the pyproject.toml of DataLab and SigimaX
172+
# (excluding Sigima and SigimaX, installed from local clones above) and
173+
# install latest versions
174+
python -m pip install -U --upgrade-strategy eager $(python -c "import tomllib, re; print(' '.join(sorted({re.sub(r'[\[\]<>=!~,.\s].*$', '', d).strip() for p in ('pyproject.toml', '../SigimaX/pyproject.toml') for d in tomllib.loads(open(p, 'rb').read().decode())['project']['dependencies'] if not any(n in d for n in ['Sigima', 'SigimaX', 'datalab-capsule'])})))")
172175
173176
- name: Lint with Ruff (latest)
174177
run: ruff check --output-format=github datalab
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Copyright (c) DataLab Platform Developers, BSD 3-Clause license, see LICENSE file.
2+
3+
"""
4+
NeXus scans and multidimensional views test
5+
--------------------------------------------
6+
7+
A scan written with the legacy NeXus conventions (as recent SOLEIL scans) is
8+
imported with its axes: its counter versus the scanned motor and its detector
9+
spectra as an image with the motor along Y. The frames of an image stack whose
10+
view is enabled in the HDF5 browser are imported as images titled and located
11+
by the motor position, and their origin survives a workspace round trip.
12+
"""
13+
14+
# guitest: show
15+
16+
from __future__ import annotations
17+
18+
import json
19+
import os.path as osp
20+
21+
import h5py
22+
import numpy as np
23+
import pytest
24+
from sigima.objects import get_experimental_source
25+
from sigimax.h5.selection import NDView
26+
from sigimax.widgets.h5browser import H5BrowserDialog
27+
28+
from datalab.tests import datalab_test_app_context, helpers
29+
30+
31+
def _write_scan(fname: str) -> None:
32+
"""Write a step scan: legacy axis attributes, a counter, a detector writing a
33+
spectrum per point and a detector writing an image per point"""
34+
with h5py.File(fname, "w") as h5file:
35+
entry = h5file.create_group("scan_0494")
36+
entry.attrs["NX_class"] = "NXentry"
37+
entry["title"] = "LaB6 calibration"
38+
scan = entry.create_group("scan_data")
39+
scan.attrs["NX_class"] = "NXdata"
40+
scan["actuator_1_1"] = np.array([-8.0, -7.5, -7.0])
41+
scan["actuator_1_1"].attrs.update(axis=np.int32(1), primary=np.int32(1))
42+
scan["actuator_1_1"].attrs["units"] = "deg"
43+
scan["counter"] = np.array([5.0, 9.0, 4.0])
44+
scan["spectra"] = np.arange(3 * 16, dtype=np.int32).reshape(3, 16)
45+
scan["frames"] = np.arange(3 * 4 * 5, dtype=np.uint16).reshape(3, 4, 5)
46+
for name in ("counter", "spectra", "frames"):
47+
scan[name].attrs["signal"] = np.int32(1)
48+
49+
50+
def _nexus_objects(panel, signal: str) -> list:
51+
"""Return the objects of a panel imported from a signal of an NXdata group"""
52+
return [
53+
obj
54+
for obj in panel.objmodel.get_all_objects()
55+
if (get_experimental_source(obj) or {}).get("selection", {}).get("signal")
56+
== signal
57+
]
58+
59+
60+
def test_scan_frames_and_workspace(monkeypatch: pytest.MonkeyPatch) -> None:
61+
"""Legacy scan counters and detectors, then frames of a stack enabled in the
62+
browser, and workspace round trip"""
63+
original_open = H5BrowserDialog.open_file
64+
65+
def open_and_enable_frames(dialog: H5BrowserDialog, fname: str) -> None:
66+
"""Open the file, and enable the import of the first two frames"""
67+
original_open(dialog, fname)
68+
tree = dialog.browser.tree
69+
item = tree.find_node_item(tree.topLevelItem(0), "/scan_0494/scan_data::frames")
70+
tree.get_node(item).set_view(NDView((1, 2), (0,), 0, 1))
71+
tree.set_item_importable(item, True)
72+
73+
monkeypatch.setattr(H5BrowserDialog, "open_file", open_and_enable_frames)
74+
with helpers.WorkdirRestoringTempDir() as tmpdir:
75+
nxsname = osp.join(tmpdir, "scan.nxs")
76+
_write_scan(nxsname)
77+
wsname = osp.join(tmpdir, "workspace.h5")
78+
with datalab_test_app_context(console=False, exec_loop=False) as win:
79+
win.h5inputoutput.import_files([nxsname], True, False)
80+
(counter,) = _nexus_objects(win.signalpanel, "counter")
81+
np.testing.assert_array_equal(counter.x, [-8.0, -7.5, -7.0])
82+
assert counter.xunit == "deg"
83+
source = get_experimental_source(counter)
84+
assert source["details"] == {"entry_title": "LaB6 calibration"}
85+
(spectra,) = _nexus_objects(win.imagepanel, "spectra")
86+
assert spectra.data.shape == (3, 16)
87+
assert (spectra.y0, spectra.dy) == (-8.0, 0.5)
88+
frames = _nexus_objects(win.imagepanel, "frames")
89+
assert len(frames) == 2
90+
assert frames[1].title.startswith("frames[1, :, :] (actuator_1_1=-7.5)")
91+
np.testing.assert_array_equal(
92+
frames[1].data, np.arange(20, 40).reshape(4, 5)
93+
)
94+
win.save_h5_workspace(wsname)
95+
96+
win.load_h5_workspace([wsname], reset_all=True)
97+
frame = _nexus_objects(win.imagepanel, "frames")[1]
98+
source = get_experimental_source(frame)
99+
assert source["selection"]["view"] == {"displayed": [1, 2], "indices": [1]}
100+
assert source["selection"]["positions"] == {"actuator_1_1": -7.5}
101+
json.dumps(frame.metadata["experimental_source"])
102+
103+
104+
if __name__ == "__main__":
105+
pytest.main([__file__])

‎doc/features/common/h5browser.rst‎

Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,26 @@ neighbouring datasets. Images need increasing coordinates: a decreasing image
5151
axis is reversed together with the image, and this conversion is recorded,
5252
while an image axis that is not strictly monotonic prevents the import.
5353

54+
Files written with the legacy NeXus conventions, such as the scans of
55+
synchrotron SOLEIL, are imported with their axes too: an axis dataset may give
56+
the dimension it describes with its ``axis`` attribute, the one with the lowest
57+
``primary`` attribute being preferred. The other measured datasets of a group,
58+
such as all the counters and detectors of a scan, appear as additional items
59+
below the group, each with the axes that fit its dimensions: a counter is a
60+
signal versus the scanned motor, a detector writing a spectrum at each point an
61+
image with the motor along Y, and a detector writing an image at each point a
62+
stack of images (see below).
63+
64+
When a NeXus file designates a default measurement (``default`` attributes), the
65+
browser selects and previews it when the file is opened. Compressed detector
66+
data, such as the bitshuffle/LZ4 data of Eiger detectors, is read; a dataset
67+
compressed with an unavailable filter is reported in the tree instead of failing.
68+
5469
Files with several counters or incomplete NeXus descriptions can be interpreted
5570
by a plugin-provided data mapping. Each mapped object appears as an additional
56-
item with its own axes, labels and units. Mappings must be registered before
71+
item with its own axes, labels and units; the axes of a mapped object may
72+
describe its first dimensions only, so that one mapping applies to the counters
73+
and detectors of a scan. Mappings must be registered before
5774
opening the file; DataLab does not automatically discover JSON files beside it.
5875
Plugin authors can follow the
5976
`SigimaX HDF5 import guide <https://sigimax.readthedocs.io/en/latest/user_guide/hdf5_workspace.html>`_.
@@ -77,10 +94,30 @@ limit: compound datasets and signals stored as 2D tables still use complete
7794
objects for their graphical previews. Closing a file or the browser waits for
7895
its preview read to end.
7996

97+
Multidimensional datasets
98+
-------------------------
99+
80100
Datasets with three or more dimensions, such as image stacks and spectral
81-
cubes, are listed but cannot be imported as complete objects. There is no
82-
graphical slice selector yet. Developers can read 1D or 2D views without loading
83-
the whole cube using the selection API described in the SigimaX guide.
101+
cubes, are imported as one image or signal per frame. Selecting such an item
102+
shows a view selector below the preview:
103+
104+
* **View**: images of two axes, or signals along one axis
105+
* **Frames along**: the axis along which frames are taken, with the **First**
106+
and **Last** frame and a **Step**
107+
* **Fixed indices**: the index of the other axes, if any
108+
* **Import these frames**: makes the item checkable and checked; unchecking
109+
the item cancels the import
110+
111+
The first frame of the view is previewed, and the number of frames is shown.
112+
Images of the last two axes, with every frame along the preceding axis, are
113+
proposed by default, unless the NeXus ``interpretation`` attribute of the
114+
dataset designates spectra (signals along the last axis) or RGB images (one
115+
image per channel). DataLab asks for confirmation before importing more than
116+
100 frames. The frames of NeXus datasets keep the axes of their displayed
117+
dimensions; their title gives the frame indices and, if the other axes have
118+
coordinates, the position of the frame (e.g. ``frames[3, :, :] (theta=0.3)``).
119+
Developers can also read views without creating objects, using the selection
120+
API described in the SigimaX guide.
84121

85122
Numerical values and experimental origin
86123
---------------------------------------
@@ -94,8 +131,9 @@ signals keep their imaginary part, including single-precision ones.
94131

95132
NeXus and mapped objects carry an ``experimental_source`` metadata entry
96133
describing the source file, group, measured dataset, axes (``"."`` for a
97-
dimension without axis) and recorded conversions. Mapped objects also identify
98-
their mapping. This entry describes
134+
dimension without axis) and recorded conversions, with the title, start and end
135+
times of the NeXus entry, and, for frames, the frame indices and position.
136+
Mapped objects also identify their mapping. This entry describes
99137
where the measurement came from; it is not the processing provenance shown in
100138
the Properties panel and does not make the experiment reproducible.
101139

0 commit comments

Comments
 (0)