Read real SPEC and XDI files as reference readers do, with validation corpora

CHANGELOG.md

@@ -8,15 +8,16 @@ Python 3.10 or later is required.
 
 First version of the Synchrotron Beamline Data example plugin: a demonstration of DataLab with beamline data, not a finished or supported feature.
 
-- SPEC reader in pure Python: index of scans with repeated numbers told apart by occurrence, columns in acquisition order, motor positions from the file headers, interrupted scans, MCA spectra with continuation lines, channel range and calibration; SPEC files recognized by their content, whatever their extension.
-- **File > Open** of `.spec` files in the signal panel: one signal per scan, last column versus the first one.
-- **Plugins > Synchrotron Beamline Data > Import SPEC scans...**: selection of scans (`all`, numbers, `<number>.<occurrence>` keys, ranges), X column, counters and MCA spectra, imported in one group per scan; scans lacking a selected column are skipped and reported in one line per reason. Scans are read and added one at a time, in batches, with a progress dialog that cancels the import (the 5414 signals of a scan with 4 MCA spectra per point are added in about 6 s).
-- Origin of each signal in its `experimental_source` metadata (file, scan, columns, command, date, motor positions).
-- HDF5 data mapping "Scan counters", applied by DataLab's HDF5 browser while the plugin is loaded: each counter `data_<n>` of an `NXdata` group with `time_1` time stamps is imported versus time.
+- SPEC reader in pure Python: index of scans with repeated numbers told apart by occurrence, columns in acquisition order, motor positions from the file headers, interrupted scans, MCA spectra of one or several devices (`@A`, `@A1`, `@A2`...) with continuation lines and the channel range and calibration of each device; labels separated by single spaces split according to the column count, repeated labels with a suffix (`I0_1`), missing values (`None`, `nan`) read as NaN, empty motor lines (Bluesky) and other `@` lines accepted; SPEC files recognized by their content, whatever their extension.
+- **File > Open** of `.spec` files in the signal panel: one signal per scan, last column versus the first one; scans that cannot be read are left out.
+- **Plugins > Synchrotron Beamline Data > Import SPEC scans...**: selection of scans (`all`, numbers, `<number>.<occurrence>` keys, ranges), X column, counters and MCA devices, imported in one group per scan: counters as signals, or as images of the two motors for mesh scans (optional, NaN for the missing points of interrupted scans), and the spectra of each MCA device as an image, one spectrum per row, in the image panel. Scans lacking a selected column, or that cannot be read, are skipped and reported in one line per reason. Scans are read and added one at a time, in batches, with a progress dialog that cancels the import.
+- Origin of each object in its `experimental_source` metadata (file, scan, columns, command, date, motor positions), and scan header as metadata (count time, monitor, hkl, geometry, comments, Bluesky metadata).
+- HDF5 data mapping "Scan counters", applied by DataLab's HDF5 browser while the plugin is loaded: each counter or detector `data_<n>` of an `NXdata` group with `time_1` time stamps is imported versus time (first dimension).
 - X-ray absorption analysis core (`core.xas`): absorption from monitor and detector counts in transmission or fluorescence, with invalid points reported; edge energy E0; pre-edge and post-edge normalization and flattening. Results are identical to Larch's on 10 measured spectra (reference foils and samples, transmission and fluorescence, several beamlines), and E0 of reference foils is within 2 eV of tabulated edge energies.
 - **Plugins > Synchrotron Beamline Data > Absorption from counters...** and **Edge normalization...** recipes: `mu(E)` from the selected monitor and detector counts (monitor recognized by its name, counters of different scans refused, invalid points listed); normalized and flattened spectra with a results table (E0, edge step, fit ranges and degree), automatic or given E0 and ranges.
 - Edge normalization of spectra in keV: energies converted to eV, with the results of the same spectrum in eV, instead of fit ranges computed as if keV were eV. Energies in other units, labelled keV with values in eV, or without unit and below 150 (most likely keV) are refused before running, with the unit to set.
-- XDI reader (XAS Data Interchange 1.x) in pure Python, and **File > Open** of `.xdi` files in the signal panel: one signal per column versus the energy, with units and origin (element, edge, beamline, sample...).
+- XDI reader (XAS Data Interchange 1.x) in pure Python, reading files as the XDI library does (invalid data tables refused with their line, invalid header lines recorded, UTF-8 free text, two-dimensional data in blocks), and **File > Open** of `.xdi` files in the signal panel: one signal per column versus the energy (monochromator angles converted with the lattice spacing), with units, all header fields and comments as metadata, and origin (element, edge, beamline, sample...).
 - **Open copper foil example**: measured counts of a copper foil (Larch examples) to try the absorption and normalization recipes.
 - Application tiles on DataLab's welcome page: the plugin, **Import SPEC scans** and **Open copper foil example**, with their icons.
-- Continuous integration: lint and tests on Linux and Windows, including the comparisons with silx and Larch.
+- Continuous integration: lint and tests on Linux and Windows, including the comparisons with silx, spec2nexus (SPEC extracts of real files), the XDI library (its test files) and Larch.
+- File formats removed from Sigima when the plugin is unloaded, and registered again when it is loaded again.

CONTRIBUTING.md

@@ -29,8 +29,8 @@ adapters -> workflow, formats, mappings -> core
 
 - `core` holds the readers (`spec`, `xdi`), their conversion to Sigima signals with their `experimental_source`, the import plans of scans (`convert`), and the X-ray absorption analysis (`xas`). It imports neither DataLab, Qt, nor compiled reader libraries such as silx or FabIO, so that it also runs in DataLab-Web and installs from DataLab's plugin catalog.
 - `workflow` holds the recipes run by DataLab (`xas`) and the examples (`examples`, data in `data/examples` with its source and license): headless, they use DataLab's recipe and example contracts and the core only.
-- `formats.py` registers the Sigima file formats used by **File > Open**, and removes them when the plugin is unloaded.
-- `mappings.py` registers the HDF5 data mappings of `data/mappings` (JSON, see `sigimax.h5.mapping`) when the plugin is loaded, and removes them when it is unloaded. A new mapping is a new JSON file there.
+- `formats.py` registers the Sigima file formats used by **File > Open**, removes them when the plugin is unloaded and registers them again when it is loaded again (Sigima's `register_format` and `unregister_format`).
+- `mappings.py` registers the HDF5 data mappings of `data/mappings` (JSON, see `sigimax.h5.mapping`) when the plugin is loaded, and removes them when it is unloaded. A new mapping is a new JSON file there. Files following the NeXus conventions, legacy ones included (`axis` and `primary` attributes, as recent SOLEIL scans), need no mapping: DataLab's HDF5 browser imports them; mappings describe what NeXus does not, such as the time stamps of SOLEIL scans (`axis` 0).
 - `adapters/desktop.py` declares the plugin and its **Import SPEC scans...** action.
 
 `tests/unit/test_architecture.py` checks the layer boundaries.
@@ -39,7 +39,12 @@ adapters -> workflow, formats, mappings -> core
 
 - `tests/unit`: readers, conversion, formats and shipped mappings, on synthetic files (`tests/support`).
 - `tests/integration`: imports and recipes in DataLab. DataLab's plugin discovery is disabled there: DataLab would otherwise load this plugin, whether installed or found on the Python path, reload the modules patched by the tests and unregister its formats when closing.
-- `tests/validation`: the SPEC reader compared with silx's reader (skipped if silx is not installed), and the XAS analysis on measured spectra (`tests/data/xas`, from Larch's examples, MIT license) compared with Larch's results and with tabulated edge energies. Larch's results are frozen in `tests/data/xas/larch_references.json`; regenerate them with `python tests/validation/make_larch_references.py` in an environment where Larch is installed (`pip install xraylarch`).
+- `tests/validation`: the readers and the XAS analysis on real files, compared with independent readers and analysis software:
+  - SPEC reader on extracts of real files (`tests/data/spec`: several MCA devices, mesh scans, labels separated by single spaces, Bluesky and Sardana files, interrupted scans, repeated scan numbers), from spec2nexus (CC BY 4.0) and PyMca (MIT), compared with the values read by spec2nexus, or by silx for the files spec2nexus cannot read. The extracts and these values are frozen in `tests/data/spec` (sources at pinned commits and SHA-256 in `tests/support/spec_corpus.py`); regenerate them with `python tests/validation/make_spec_corpus.py` in an environment where spec2nexus, from the same commit as the sources, and silx are installed (see the script). The SPEC reader is also compared with silx's reader on synthetic files (skipped if silx is not installed).
+  - XDI reader on the test files of the XDI library (`tests/data/xdi`, public domain), checked against the behaviour of the library described in `BadFiles.txt`.
+  - XAS analysis on measured spectra (`tests/data/xas`, from Larch's examples, MIT license) compared with Larch's results and with tabulated edge energies. Larch's results are frozen in `tests/data/xas/larch_references.json`; regenerate them with `python tests/validation/make_larch_references.py` in an environment where Larch is installed (`pip install xraylarch`).
+
+Test data stays in this repository: these formats are read by the plugin, not by Sigima. Each data folder holds a `SOURCES.txt` file giving the origin, commit, license and changes of its files; files without a license allowing redistribution are not added.
 
 ## Branches
 

README.md

@@ -2,18 +2,18 @@
 
 A [DataLab](https://datalab-platform.com/) plugin to read beamline data and analyze it in DataLab.
 
-> **Example plugin.** This plugin shows how DataLab can read beamline data and run a beamline analysis. It is a demonstration, not a finished or supported feature: its scope is limited to what is described below, its results are validated only by the tests described in [CONTRIBUTING](CONTRIBUTING.md) (SPEC reader compared with silx on synthetic files, X-ray absorption analysis compared with Larch on measured spectra), and its interface may change without notice.
+> **Example plugin.** This plugin shows how DataLab can read beamline data and run a beamline analysis. It is a demonstration, not a finished or supported feature: its scope is limited to what is described below, its results are validated only by the tests described in [CONTRIBUTING](CONTRIBUTING.md) (SPEC reader compared with spec2nexus on extracts of real files, XDI reader on the test files of the XDI library, X-ray absorption analysis compared with Larch on measured spectra), and its interface may change without notice.
 
 ## What it does
 
-- **SPEC files:** **Plugins > Synchrotron Beamline Data > Import SPEC scans...** reads any SPEC file, whatever its extension: choose the scans (`all`, or e.g. `1, 3.2, 10-12`), the X column, the counters and the MCA spectra, and each scan is imported in its own group. Scans lacking a selected column are skipped and reported in one line per reason, and a progress dialog lets you cancel a long import (e.g. thousands of MCA spectra). **File > Open** in the signal panel also reads a `.spec` file, as one signal per scan: the last column (usually the detector counts) versus the first one (usually the scanned motor). Repeated scan numbers are told apart by their occurrence (`12.1`, `12.2`), the acquisition order of the points is kept, incomplete data lines of interrupted scans are skipped and counted, and MCA spectra are read with their channel range and calibration.
-- Each signal records its origin in its `experimental_source` metadata: file, scan, columns, command, date and motor positions. A saved workspace reopens without the SPEC file or the plugin.
-- **XDI files** (XAS Data Interchange, as written by Larch, Athena and many beamlines): **File > Open** reads every column versus the energy, with units; the origin records the element, edge, facility, beamline, monochromator, sample and start time.
+- **SPEC files:** **Plugins > Synchrotron Beamline Data > Import SPEC scans...** reads any SPEC file, whatever its extension: choose the scans (`all`, or e.g. `1, 3.2, 10-12`), the X column, the counters and the MCA devices, and each scan is imported in its own group. Counters are signals versus the X column; counters of mesh scans (`mesh`, `dmesh`) are images of the two motors (optional), with NaN for the points missing in an interrupted scan; the spectra of each MCA device are an image, one spectrum per row, versus the calibrated channels and the X column (or the point number), in the image panel. Scans lacking a selected column, or that cannot be read, are skipped and reported in one line per reason, and a progress dialog lets you cancel a long import. **File > Open** in the signal panel also reads a `.spec` file, as one signal per scan: the last column (usually the detector counts) versus the first one (usually the scanned motor). Repeated scan numbers are told apart by their occurrence (`12.1`, `12.2`), the acquisition order of the points is kept, incomplete data lines of interrupted scans are skipped and counted, missing values (`None`, `nan`) are NaN, labels separated by single spaces are split according to the column count, and MCA spectra of several devices (`@A1`, `@A2`...) are read with their own channel range and calibration.
+- Each object records its origin in its `experimental_source` metadata: file, scan, columns, command, date and motor positions; the scan header (count time, monitor, hkl, geometry, comments, Bluesky metadata) is kept as `SPEC ...` metadata. A saved workspace reopens without the SPEC file or the plugin.
+- **XDI files** (XAS Data Interchange, as written by Larch, Athena and many beamlines): **File > Open** reads every column versus the energy, with units, as the XDI library does: data tables with invalid values are refused with the line number, invalid header lines are recorded (`XDI problems` metadata), free text may be UTF-8, monochromator angles are converted to energies with the lattice spacing (`Mono.d_spacing`), and two-dimensional data (`Outer.value` blocks) gives one signal per block. Every header field and the comments are kept as metadata; the origin records the element, edge, facility, beamline, monochromator, sample and start time.
 - **X-ray absorption (XANES/EXAFS):** **Absorption from counters...** computes `mu(E)` from the selected monitor (I0, recognized by its name) and detector counts, in transmission (`ln(I0/It)`) or fluorescence (`If/I0`); points without valid counts are left out and listed. **Edge normalization...** finds the edge energy E0 of each selected spectrum and normalizes it to a unit edge step, with a pre-edge line and a post-edge polynomial (automatic or given ranges), giving the normalized and flattened spectra and a table of E0, edge step and ranges. The results are those of [Larch](https://xraypy.github.io/xraylarch/) on 10 measured spectra (transmission and fluorescence, several beamlines). Energies in keV are converted to eV, and results are given in eV. Spectra are refused, with a message telling which X unit to set, if their energies are in another unit (e.g. monochromator angles in degrees), labelled keV with values in eV, or without unit and below 150 (most likely keV, as often in SPEC files); energies without unit from 150 up are taken as eV.
 - **Example:** **Open copper foil example** opens the monitor and transmitted counts of a copper foil (Larch examples, MIT license) to try both recipes: E0 is 8980.5 eV, 1.5 eV above the tabulated Cu K edge, and the edge step is 2.73.
-- NeXus files are read by DataLab itself (HDF5 browser), which shows `NXdata` groups as signals or images with their axes. While the plugin is loaded, the HDF5 browser also shows every counter `data_1`, `data_2`... of scan data groups as a signal versus the time stamps `time_1` of the points, with its label and units: such groups designate only their first counter, without axis.
+- NeXus files are read by DataLab itself (HDF5 browser), which shows `NXdata` groups as signals or images with their axes, and stacks of images or spectra as frames, including the legacy conventions of recent SOLEIL scans: counters are signals versus the scanned motor, and detectors images or stacks with the motor first. While the plugin is loaded, the HDF5 browser also shows every counter and detector `data_1`, `data_2`... of scan data groups versus the time stamps `time_1` of the points, with its label and units: such groups (time scans of SOLEIL) designate only their first counter, without axis.
 
-The readers are written in pure Python: the plugin installs from DataLab's plugin catalog, including with the standalone version of DataLab, and its core can run in DataLab-Web. silx is used only by the tests, as a reference reader.
+The readers are written in pure Python: the plugin installs from DataLab's plugin catalog, including with the standalone version of DataLab, and its core can run in DataLab-Web. silx and spec2nexus are used only by the tests, as reference readers.
 
 ## Installation