feat(h5): import multidimensional data, legacy NeXus scans and compressed data

doc/release_notes/release_1.01.md

@@ -41,9 +41,12 @@
 
 * Browsing an HDF5 file no longer reads its numeric datasets entirely just to classify them: generic nodes are classified and described from the dataset shape and data type, and value previews read only the first elements (`TEXT_PREVIEW_MAX_ITEMS`). Compound datasets mixing numeric and string fields are checked on their first records only. Full data is read when creating an imported object or an undecimated graphical preview.
 * The HDF5 browser reads previews in a worker thread, so that it stays responsive while a large dataset is read: only the last selected node is shown, the previews of nodes selected meanwhile are not read, selecting a node that cannot be previewed (a group, for example) clears the preview, and closing a file or the browser dialog waits for the preview being read (imported objects are then read without concurrent access to the file). Ordinary 1D arrays and 2D images, NeXus and mapped objects with more than `PlotPreview.MAX_ITEMS` (2²⁴) elements are previewed with a step along each axis; the axes of NeXus images are still read entirely, to check that they are strictly monotonic as on import (they hold far fewer values than the image). Compound datasets and signals stored in 2D tables still use complete objects for graphical previews; this is not a universal memory bound.
-* Numeric datasets with three dimensions or more (image stacks, spectral cubes...) are now listed in the HDF5 browser, with their shape and type. They are not imported as a whole, and there is no graphical slice selector: developers can read their 1D or 2D views with `sigimax.h5.selection.read_selection`, which reads only the selected elements (`NDSelection`: displayed axes, possibly transposed, and an index along each other axis).
-* NeXus `NXdata` groups are shown as single nodes, imported as a signal (1D) or an image (2D) with their axes, units, labels and, for signals, uncertainties: `@signal`/`@axes` attributes with `@<axis>_indices`, legacy `signal`/`axes` dataset attributes, bin edges converted to bin centers, non-uniform image axes. Image axes deviating from even spacing by at most 2.5 units in the last place of their data type (`UNIFORM_AXIS_TOLERANCE`) are imported as uniform coordinates, so that evenly spaced `float32` axes are recognized, and their maximum deviation is recorded among the conversions; other image axes keep their stored coordinates. Decreasing image axes are reversed with the image. Invalid axes (axis count differing from the signal rank, axes designating a missing or already described dimension, axes not holding real numbers, image axes not strictly monotonic) are reported instead of being guessed. Imported objects record their origin in the `experimental_source` metadata entry (file, group, signal and axes, `"."` for a dimension without axis, conversions). `.nxs` and `.nx5` files are proposed in HDF5 file dialogs.
-* Files that do not designate their data with suitable `NXdata` groups (no axes, several counters...) can be described by declarative data mappings (`sigimax.h5.mapping`): JSON-compatible descriptions of the signals and images of the groups having given attributes and paths, with their axes, uncertainties, labels and units, and a name pattern to repeat an object for each counter of a scan. Mapped objects are shown as extra nodes of the HDF5 browser and record the mapping in their `experimental_source`. Applications and plugins register mappings with `register_mapping` (and `unregister_mapping`); `NodeFactory.add_children_provider` lets them add such extra nodes. Blank units and labels are now ignored.
+* Numeric datasets with three dimensions or more (image stacks, spectral cubes...) are now listed in the HDF5 browser, with their shape and type, and imported through a view: one signal or image per frame. When such a dataset (or NeXus signal) is selected, a view selector proposes the displayed axes (images of two axes, or signals along one axis), the axis along which frames are taken with their first, last and step, and the index of the other axes; the first frame is previewed, and the frames are imported once "Import these frames" is checked, the browser asking for confirmation beyond `MAX_FRAMES_WITHOUT_CONFIRMATION` (100) frames. The default view follows the NeXus `interpretation` of the signal (`spectrum`: signals along the last axis, `rgba-image`: one image per channel, otherwise images of the last two axes). Frames of NeXus signals keep the axes of their displayed dimensions, and their title and origin give the frame indices and the coordinates of the frame along the other axes. Developers set views with `NDView`, `default_view` and `set_view`, and `expand_views` replaces a node by the nodes of its frames (`sigimax.h5.views`); `read_selection` reads only the selected elements (`NDSelection`: displayed axes, possibly transposed, and an index along each other axis).
+* NeXus `NXdata` groups are shown as single nodes, imported as a signal (1D) or an image (2D) with their axes, units, labels and, for signals, uncertainties: `@signal`/`@axes` attributes with `@<axis>_indices`, legacy `signal`/`axes` dataset attributes, bin edges converted to bin centers, non-uniform image axes. Image axes deviating from even spacing by at most 2.5 units in the last place of their data type (`UNIFORM_AXIS_TOLERANCE`) are imported as uniform coordinates, so that evenly spaced `float32` axes are recognized, and their maximum deviation is recorded among the conversions; other image axes keep their stored coordinates. Decreasing image axes are reversed with the image. Invalid axes (axis count differing from the signal rank, axes designating a missing or already described dimension, axes not holding real numbers, image axes not strictly monotonic) are reported instead of being guessed. Imported objects record their origin in the `experimental_source` metadata entry (file, group, signal and axes, `"."` for a dimension without axis, conversions, and the title, start and end times of the entry). `.nxs` and `.nx5` files are proposed in HDF5 file dialogs.
+* NeXus files written with the legacy conventions (e.g. scans of synchrotron SOLEIL) are imported with their axes: without `@axes`, the `axis` attribute of axis datasets gives the dimension they describe (from 1), the dataset with the lowest `primary` attribute being preferred, and an axis applies to the dimensions whose size fits. The other signals of an `NXdata` group (`@auxiliary_signals` attribute, or datasets having a legacy `signal` attribute, as counters and detectors of a scan) are shown as nodes of their own, with the axes that fit their dimensions: for example, a 1D counter is a signal versus the scanned motor, a detector writing a spectrum per point is an image with the motor along Y, and a detector writing an image per point is a stack of images, one per motor position.
+* The HDF5 browser selects and expands the default `NXdata` group of a NeXus file (`@default` attributes of the file and of its entry) when the file is opened (`default_nxdata_path`).
+* Compressed detector data (bitshuffle/LZ4 of Eiger detectors, Blosc, Zstandard...) is now read: SigimaX requires hdf5plugin, which provides these HDF5 filters. Datasets compressed with a filter that is still unavailable are reported in the browser (filter name and number) instead of failing when read.
+* Files that do not designate their data with suitable `NXdata` groups (no axes, several counters...) can be described by declarative data mappings (`sigimax.h5.mapping`): JSON-compatible descriptions of the signals and images of the groups having given attributes and paths, with their axes, uncertainties, labels and units, and a name pattern to repeat an object for each counter of a scan. The `axes` of a mapped object describe its first dimensions, the following ones having no axis, so that one pattern applies to counters, spectra and images of a scan, with the scan axis first. Mapped objects are shown as extra nodes of the HDF5 browser and record the mapping in their `experimental_source`. Applications and plugins register mappings with `register_mapping` (and `unregister_mapping`); `NodeFactory.add_children_provider` lets them add such extra nodes. Blank units and labels are now ignored.
 * Dataset attributes with more than `MAX_ATTRIBUTE_ITEMS` (10 000) values are no longer copied to object metadata.
 * Images with unsigned 32-bit values above 2 147 483 647 are no longer clipped to `int32`: they are converted with Sigima's converter, which keeps every value exactly with Sigima ≥ 1.4.0. `BaseNode.uint32_wng` is kept for compatibility and is always `False`.
 * 1D datasets of single-precision complex numbers (`complex64`) are now imported as complex signals: only their real part was kept (with a `ComplexWarning`). Integer signals beyond 2⁵³ now emit Sigima's precision warning instead of being rounded silently, and 2D tables with complex X values or uncertainties are reported as unsupported instead of being truncated.
@@ -55,4 +58,5 @@ See the [HDF5 guide](../user_guide/hdf5_workspace.rst) for a complete JSON mappi
 
 ### Requirements ###
 
-* Sigima 1.4.0 or later is required: the HDF5 import modules use its experimental-source API, integer conversions use its precision-preserving converter, and the tooltips of rectangular ROIs use its pixel-range conversions.
+* Sigima 1.4.0 or later is required: the HDF5 import modules use its experimental-source API, integer conversions use its precision-preserving converter, and the tooltips of rectangular ROIs use its pixel-range conversions.
+* hdf5plugin 4.0 or later is required, for the HDF5 compression filters of detector data.

doc/user_guide/hdf5_workspace.rst

@@ -90,6 +90,27 @@ With Sigima 1.4 or later, an ``NXdata`` group is importable as a 1D signal or
 (``signal=1`` and ``axes``) are supported. Selecting a group imports the
 scientific object; selecting a child dataset still gives access to raw values.
 
+Files written with the legacy NeXus conventions, such as the scans of
+synchrotron SOLEIL, designate axes by dimension: without ``axes`` attribute, the
+``axis`` attribute of an axis dataset gives the dimension it describes (from 1),
+the dataset with the lowest ``primary`` attribute being preferred, and an axis
+applies to a dimension only if its size fits. The other signals of a group, given
+by its ``auxiliary_signals`` attribute or by a legacy ``signal`` attribute of
+their dataset (whatever its value, as all the counters and detectors of a scan),
+are shown as nodes of their own, below the group, with the axes that fit their
+dimensions: a 1D counter is a signal versus the scanned motor, a detector writing
+a spectrum per point an image with the motor along Y, and a detector writing an
+image per point a stack of images (see below).
+
+When a NeXus file is opened, the browser selects its default ``NXdata`` group,
+designated by the ``default`` attributes of the file and of its entry
+(``sigimax.h5.nexus.default_nxdata_path``).
+
+Compressed detector data (bitshuffle/LZ4, Blosc, Zstandard...) is read with the
+HDF5 filters of hdf5plugin, a dependency of SigimaX. A dataset compressed with a
+filter that is still unavailable is reported in the browser, with the filter name
+and number, instead of failing when read.
+
 Axis datasets must be 1D, hold real numbers and contain either one coordinate
 per point or one extra value for bin edges. Edges are converted to centers.
 Image coordinates may be non-uniform; a decreasing image axis is reversed,
@@ -108,7 +129,9 @@ real numbers and their shape matches; image uncertainties are not imported.
 
 Imported objects record their file, group, signal, axes and conversions in
 ``experimental_source`` metadata; a dimension without axis is recorded as
-``"."``, as in NeXus ``axes`` attributes. This describes the measurement
+``"."``, as in NeXus ``axes`` attributes. The title of the ``NXentry`` group is
+recorded in the details (``entry_title``), and its start and end times in the
+timestamps. This describes the measurement
 origin, not a reproducible acquisition or a processing history. The
 application must preserve object metadata in its own workspace serialization.
 
@@ -146,9 +169,11 @@ then required in the object name and is replaced by each dataset name.
 Uncertainty datasets ending in ``_errors`` and declared axes are excluded
 from pattern expansion.
 
-``axes`` contains one entry per signal dimension, in array order (rows then
+``axes`` gives the axes of the first signal dimensions, in array order (rows then
 columns for an image). Each entry is a path, an object with ``path`` and optional
-``label`` and ``units``, or ``null`` to use indices. Omitting ``axes`` uses
+``label`` and ``units``, or ``null`` to use indices. The following dimensions use
+indices, so that a pattern may match signals of several ranks, e.g. the counters,
+spectra and images of a scan, with the scan axis first. Omitting ``axes`` uses
 indices for every dimension. The optional ``errors`` path overrides the
 default ``<signal>_errors`` dataset. Labels and units otherwise come from
 ``long_name`` and ``units`` attributes, with the dataset name as label fallback.
@@ -221,11 +246,57 @@ still read on import. Compound datasets and signals stored in 2D tables still
 use complete objects for graphical previews, so this is not a universal memory
 bound.
 
-Numeric datasets with three or more dimensions are listed but not imported as
-complete objects. There is no graphical slice selector. Use
+Numeric datasets and NeXus signals with three or more dimensions are imported
+through a view, as one signal or image per frame. When such an item is selected,
+the browser shows a view selector: the displayed axes (images of two axes, or
+signals along one axis), the axis along which frames are taken, with the first
+and last frame and a step, and the index of the other axes. The first frame of
+the view is previewed. Checking "Import these frames" makes the item checkable
+and checked; unchecking the item cancels the import. The browser asks for
+confirmation before importing more than 100 frames
+(``sigimax.widgets.h5browser.MAX_FRAMES_WITHOUT_CONFIRMATION``). The default view
+displays the last two axes as images, with every frame along the preceding axis,
+or follows the NeXus ``interpretation`` attribute of the signal: ``spectrum``
+displays signals along the last axis, ``rgba-image`` one image per channel. The
+frames of NeXus signals keep the axes of their displayed dimensions; their title
+and ``experimental_source`` give the frame indices (``view``) and the coordinates
+of the frame along the other axes (``positions``).
+
+Applications set views with :class:`sigimax.h5.selection.NDView`, and replace a
+node by the nodes of its frames with :func:`sigimax.h5.views.expand_views`, which
+the browser applies to the selected nodes:
+
+.. code-block:: python
+
+    from pathlib import Path
+    from tempfile import TemporaryDirectory
+
+    import h5py
+    import numpy as np
+
+    from sigimax.h5 import H5Importer
+    from sigimax.h5.selection import NDView
+    from sigimax.h5.views import expand_views
+
+    cube = np.arange(5 * 3 * 4).reshape(5, 3, 4)
+    with TemporaryDirectory() as directory:
+        filename = Path(directory) / "stack.h5"
+        with h5py.File(filename, "w") as h5file:
+            h5file["frames"] = cube
+        importer = H5Importer(str(filename))
+        try:
+            node = importer.get("/frames")
+            # Images of axes 1 and 2, frames 1 and 3 along axis 0
+            node.set_view(NDView((1, 2), (1,), frame_axis=0, last=3, step=2))
+            images = [frame.get_native_object() for frame in expand_views([node])]
+            np.testing.assert_array_equal(images[1].data, cube[3])
+        finally:
+            importer.close()
+
+To read a view as a NumPy array, without creating Sigima objects, use
 :class:`sigimax.h5.selection.NDSelection` and
-:func:`sigimax.h5.selection.read_selection` to read only a selected 1D or 2D
-view as a NumPy array:
+:func:`sigimax.h5.selection.read_selection`, which read only the selected
+elements:
 
 .. code-block:: python
 

pyproject.toml

@@ -37,6 +37,7 @@ classifiers = [
 requires-python = ">=3.10, <4"
 dependencies = [
     "guidata >= 3.15.0",
+    "hdf5plugin >= 4.0",
     "PlotPy >= 2.11.0",
     "psutil >= 5.7",
     "Sigima >= 1.4.0",

requirements.txt

@@ -3,6 +3,7 @@ PlotPy >= 2.11.0
 PyQt5 >= 5.15.6
 Sigima >= 1.4.0
 guidata >= 3.15.0
+hdf5plugin >= 4.0
 myst-nb
 myst_parser
 psutil >= 5.7

sigimax/h5/__init__.py

@@ -25,12 +25,19 @@
     :members:
 
 NeXus ``NXdata`` groups are shown as single nodes, imported as signals or images
-with their axes (:mod:`sigimax.h5.nexus`). Views of multidimensional datasets are
-read with :func:`sigimax.h5.selection.read_selection`.
+with their axes (:mod:`sigimax.h5.nexus`); their auxiliary signals are shown as
+nodes of their own. Datasets of three dimensions or more are imported through
+views, one signal or image per frame (:mod:`sigimax.h5.views`), and views of
+multidimensional datasets are read with
+:func:`sigimax.h5.selection.read_selection`.
 
 .. autoclass:: sigimax.h5.nexus.NXdataNode
+.. autoclass:: sigimax.h5.nexus.AuxiliarySignalNode
 .. autofunction:: sigimax.h5.nexus.describe_nxdata
+.. autofunction:: sigimax.h5.nexus.describe_auxiliary_signals
 .. autofunction:: sigimax.h5.nexus.describe_signal
+.. autofunction:: sigimax.h5.nexus.default_nxdata_path
+.. autofunction:: sigimax.h5.nexus.entry_information
 .. autodata:: sigimax.h5.nexus.UNIFORM_AXIS_TOLERANCE
 
 Files without suitable ``NXdata`` groups are described by declarative data
@@ -47,8 +54,16 @@
 .. autoclass:: sigimax.h5.mapping.MappedDataNode
 .. autoclass:: sigimax.h5.selection.NDSelection
     :members:
+.. autoclass:: sigimax.h5.selection.NDView
+    :members:
+.. autofunction:: sigimax.h5.selection.default_view
 .. autofunction:: sigimax.h5.selection.read_selection
 .. autofunction:: sigimax.h5.selection.preview_step
+.. autoclass:: sigimax.h5.views.NDViewMixin
+    :members:
+.. autoclass:: sigimax.h5.views.FrameNode
+.. autofunction:: sigimax.h5.views.expand_views
+.. autofunction:: sigimax.h5.utils.unavailable_filters
 """
 
 __all__ = [
@@ -57,6 +72,10 @@
 
 # pylint: disable=unused-import
 
+# Compression filters of detector data (bitshuffle/LZ4, Blosc, Zstandard...),
+# available to h5py once registered by hdf5plugin
+import hdf5plugin  # noqa: F401
+
 # Registering dynamic I/O features:
 from sigimax.h5 import generic, mapping, nexus  # noqa: F401
 from sigimax.h5.common import H5Importer  # noqa: F401

sigimax/h5/common.py

@@ -131,6 +131,11 @@ def is_supported(self) -> bool:
         """Return True if node is associated to supported data"""
         return False
 
+    def can_preview(self) -> bool:
+        """Return True if the node can be previewed (by default, if it is
+        supported: multidimensional nodes are previewed before being enabled)"""
+        return self.is_supported()
+
     def create_native_object(self):
         """Create native object, if supported"""
         return None
@@ -197,11 +202,21 @@ def __process_metadata(self, obj):
     @property
     def object_title(self):
         """Return signal/image object title"""
+        return self.make_title()
+
+    def make_title(self, suffix: str = "") -> str:
+        """Return the title of an object read from the node
+
+        Args:
+            suffix: text following the node name or path (e.g. frame indices),
+             before the file name
+        """
         conf = get_conf()
         if conf.h5_fullpath_in_title.get():
             title = self.id
         else:
             title = self.name
+        title += suffix
         if conf.h5_fname_in_title.get():
             title += f" ({osp.basename(self.h5file.filename)})"
         return title