diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e6d40e..87db10f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,15 +8,16 @@ Python 3.10 or later is required. First version of the Synchrotron Beamline Data example plugin: a demonstration of DataLab with beamline data, not a finished or supported feature. -- SPEC reader in pure Python: index of scans with repeated numbers told apart by occurrence, columns in acquisition order, motor positions from the file headers, interrupted scans, MCA spectra with continuation lines, channel range and calibration; SPEC files recognized by their content, whatever their extension. -- **File > Open** of `.spec` files in the signal panel: one signal per scan, last column versus the first one. -- **Plugins > Synchrotron Beamline Data > Import SPEC scans...**: selection of scans (`all`, numbers, `.` keys, ranges), X column, counters and MCA spectra, imported in one group per scan; scans lacking a selected column are skipped and reported in one line per reason. Scans are read and added one at a time, in batches, with a progress dialog that cancels the import (the 5414 signals of a scan with 4 MCA spectra per point are added in about 6 s). -- Origin of each signal in its `experimental_source` metadata (file, scan, columns, command, date, motor positions). -- HDF5 data mapping "Scan counters", applied by DataLab's HDF5 browser while the plugin is loaded: each counter `data_` of an `NXdata` group with `time_1` time stamps is imported versus time. +- SPEC reader in pure Python: index of scans with repeated numbers told apart by occurrence, columns in acquisition order, motor positions from the file headers, interrupted scans, MCA spectra of one or several devices (`@A`, `@A1`, `@A2`...) with continuation lines and the channel range and calibration of each device; labels separated by single spaces split according to the column count, repeated labels with a suffix (`I0_1`), missing values (`None`, `nan`) read as NaN, empty motor lines (Bluesky) and other `@` lines accepted; SPEC files recognized by their content, whatever their extension. +- **File > Open** of `.spec` files in the signal panel: one signal per scan, last column versus the first one; scans that cannot be read are left out. +- **Plugins > Synchrotron Beamline Data > Import SPEC scans...**: selection of scans (`all`, numbers, `.` keys, ranges), X column, counters and MCA devices, imported in one group per scan: counters as signals, or as images of the two motors for mesh scans (optional, NaN for the missing points of interrupted scans), and the spectra of each MCA device as an image, one spectrum per row, in the image panel. Scans lacking a selected column, or that cannot be read, are skipped and reported in one line per reason. Scans are read and added one at a time, in batches, with a progress dialog that cancels the import. +- Origin of each object in its `experimental_source` metadata (file, scan, columns, command, date, motor positions), and scan header as metadata (count time, monitor, hkl, geometry, comments, Bluesky metadata). +- HDF5 data mapping "Scan counters", applied by DataLab's HDF5 browser while the plugin is loaded: each counter or detector `data_` of an `NXdata` group with `time_1` time stamps is imported versus time (first dimension). - X-ray absorption analysis core (`core.xas`): absorption from monitor and detector counts in transmission or fluorescence, with invalid points reported; edge energy E0; pre-edge and post-edge normalization and flattening. Results are identical to Larch's on 10 measured spectra (reference foils and samples, transmission and fluorescence, several beamlines), and E0 of reference foils is within 2 eV of tabulated edge energies. - **Plugins > Synchrotron Beamline Data > Absorption from counters...** and **Edge normalization...** recipes: `mu(E)` from the selected monitor and detector counts (monitor recognized by its name, counters of different scans refused, invalid points listed); normalized and flattened spectra with a results table (E0, edge step, fit ranges and degree), automatic or given E0 and ranges. - Edge normalization of spectra in keV: energies converted to eV, with the results of the same spectrum in eV, instead of fit ranges computed as if keV were eV. Energies in other units, labelled keV with values in eV, or without unit and below 150 (most likely keV) are refused before running, with the unit to set. -- XDI reader (XAS Data Interchange 1.x) in pure Python, and **File > Open** of `.xdi` files in the signal panel: one signal per column versus the energy, with units and origin (element, edge, beamline, sample...). +- XDI reader (XAS Data Interchange 1.x) in pure Python, reading files as the XDI library does (invalid data tables refused with their line, invalid header lines recorded, UTF-8 free text, two-dimensional data in blocks), and **File > Open** of `.xdi` files in the signal panel: one signal per column versus the energy (monochromator angles converted with the lattice spacing), with units, all header fields and comments as metadata, and origin (element, edge, beamline, sample...). - **Open copper foil example**: measured counts of a copper foil (Larch examples) to try the absorption and normalization recipes. - Application tiles on DataLab's welcome page: the plugin, **Import SPEC scans** and **Open copper foil example**, with their icons. -- Continuous integration: lint and tests on Linux and Windows, including the comparisons with silx and Larch. +- Continuous integration: lint and tests on Linux and Windows, including the comparisons with silx, spec2nexus (SPEC extracts of real files), the XDI library (its test files) and Larch. +- File formats removed from Sigima when the plugin is unloaded, and registered again when it is loaded again. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 364ecdc..ec370d1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,8 +29,8 @@ adapters -> workflow, formats, mappings -> core - `core` holds the readers (`spec`, `xdi`), their conversion to Sigima signals with their `experimental_source`, the import plans of scans (`convert`), and the X-ray absorption analysis (`xas`). It imports neither DataLab, Qt, nor compiled reader libraries such as silx or FabIO, so that it also runs in DataLab-Web and installs from DataLab's plugin catalog. - `workflow` holds the recipes run by DataLab (`xas`) and the examples (`examples`, data in `data/examples` with its source and license): headless, they use DataLab's recipe and example contracts and the core only. -- `formats.py` registers the Sigima file formats used by **File > Open**, and removes them when the plugin is unloaded. -- `mappings.py` registers the HDF5 data mappings of `data/mappings` (JSON, see `sigimax.h5.mapping`) when the plugin is loaded, and removes them when it is unloaded. A new mapping is a new JSON file there. +- `formats.py` registers the Sigima file formats used by **File > Open**, removes them when the plugin is unloaded and registers them again when it is loaded again (Sigima's `register_format` and `unregister_format`). +- `mappings.py` registers the HDF5 data mappings of `data/mappings` (JSON, see `sigimax.h5.mapping`) when the plugin is loaded, and removes them when it is unloaded. A new mapping is a new JSON file there. Files following the NeXus conventions, legacy ones included (`axis` and `primary` attributes, as recent SOLEIL scans), need no mapping: DataLab's HDF5 browser imports them; mappings describe what NeXus does not, such as the time stamps of SOLEIL scans (`axis` 0). - `adapters/desktop.py` declares the plugin and its **Import SPEC scans...** action. `tests/unit/test_architecture.py` checks the layer boundaries. @@ -39,7 +39,12 @@ adapters -> workflow, formats, mappings -> core - `tests/unit`: readers, conversion, formats and shipped mappings, on synthetic files (`tests/support`). - `tests/integration`: imports and recipes in DataLab. DataLab's plugin discovery is disabled there: DataLab would otherwise load this plugin, whether installed or found on the Python path, reload the modules patched by the tests and unregister its formats when closing. -- `tests/validation`: the SPEC reader compared with silx's reader (skipped if silx is not installed), and the XAS analysis on measured spectra (`tests/data/xas`, from Larch's examples, MIT license) compared with Larch's results and with tabulated edge energies. Larch's results are frozen in `tests/data/xas/larch_references.json`; regenerate them with `python tests/validation/make_larch_references.py` in an environment where Larch is installed (`pip install xraylarch`). +- `tests/validation`: the readers and the XAS analysis on real files, compared with independent readers and analysis software: + - SPEC reader on extracts of real files (`tests/data/spec`: several MCA devices, mesh scans, labels separated by single spaces, Bluesky and Sardana files, interrupted scans, repeated scan numbers), from spec2nexus (CC BY 4.0) and PyMca (MIT), compared with the values read by spec2nexus, or by silx for the files spec2nexus cannot read. The extracts and these values are frozen in `tests/data/spec` (sources at pinned commits and SHA-256 in `tests/support/spec_corpus.py`); regenerate them with `python tests/validation/make_spec_corpus.py` in an environment where spec2nexus, from the same commit as the sources, and silx are installed (see the script). The SPEC reader is also compared with silx's reader on synthetic files (skipped if silx is not installed). + - XDI reader on the test files of the XDI library (`tests/data/xdi`, public domain), checked against the behaviour of the library described in `BadFiles.txt`. + - XAS analysis on measured spectra (`tests/data/xas`, from Larch's examples, MIT license) compared with Larch's results and with tabulated edge energies. Larch's results are frozen in `tests/data/xas/larch_references.json`; regenerate them with `python tests/validation/make_larch_references.py` in an environment where Larch is installed (`pip install xraylarch`). + +Test data stays in this repository: these formats are read by the plugin, not by Sigima. Each data folder holds a `SOURCES.txt` file giving the origin, commit, license and changes of its files; files without a license allowing redistribution are not added. ## Branches diff --git a/README.md b/README.md index 10bcf91..f6ae894 100644 --- a/README.md +++ b/README.md @@ -2,18 +2,18 @@ A [DataLab](https://datalab-platform.com/) plugin to read beamline data and analyze it in DataLab. -> **Example plugin.** This plugin shows how DataLab can read beamline data and run a beamline analysis. It is a demonstration, not a finished or supported feature: its scope is limited to what is described below, its results are validated only by the tests described in [CONTRIBUTING](CONTRIBUTING.md) (SPEC reader compared with silx on synthetic files, X-ray absorption analysis compared with Larch on measured spectra), and its interface may change without notice. +> **Example plugin.** This plugin shows how DataLab can read beamline data and run a beamline analysis. It is a demonstration, not a finished or supported feature: its scope is limited to what is described below, its results are validated only by the tests described in [CONTRIBUTING](CONTRIBUTING.md) (SPEC reader compared with spec2nexus on extracts of real files, XDI reader on the test files of the XDI library, X-ray absorption analysis compared with Larch on measured spectra), and its interface may change without notice. ## What it does -- **SPEC files:** **Plugins > Synchrotron Beamline Data > Import SPEC scans...** reads any SPEC file, whatever its extension: choose the scans (`all`, or e.g. `1, 3.2, 10-12`), the X column, the counters and the MCA spectra, and each scan is imported in its own group. Scans lacking a selected column are skipped and reported in one line per reason, and a progress dialog lets you cancel a long import (e.g. thousands of MCA spectra). **File > Open** in the signal panel also reads a `.spec` file, as one signal per scan: the last column (usually the detector counts) versus the first one (usually the scanned motor). Repeated scan numbers are told apart by their occurrence (`12.1`, `12.2`), the acquisition order of the points is kept, incomplete data lines of interrupted scans are skipped and counted, and MCA spectra are read with their channel range and calibration. -- Each signal records its origin in its `experimental_source` metadata: file, scan, columns, command, date and motor positions. A saved workspace reopens without the SPEC file or the plugin. -- **XDI files** (XAS Data Interchange, as written by Larch, Athena and many beamlines): **File > Open** reads every column versus the energy, with units; the origin records the element, edge, facility, beamline, monochromator, sample and start time. +- **SPEC files:** **Plugins > Synchrotron Beamline Data > Import SPEC scans...** reads any SPEC file, whatever its extension: choose the scans (`all`, or e.g. `1, 3.2, 10-12`), the X column, the counters and the MCA devices, and each scan is imported in its own group. Counters are signals versus the X column; counters of mesh scans (`mesh`, `dmesh`) are images of the two motors (optional), with NaN for the points missing in an interrupted scan; the spectra of each MCA device are an image, one spectrum per row, versus the calibrated channels and the X column (or the point number), in the image panel. Scans lacking a selected column, or that cannot be read, are skipped and reported in one line per reason, and a progress dialog lets you cancel a long import. **File > Open** in the signal panel also reads a `.spec` file, as one signal per scan: the last column (usually the detector counts) versus the first one (usually the scanned motor). Repeated scan numbers are told apart by their occurrence (`12.1`, `12.2`), the acquisition order of the points is kept, incomplete data lines of interrupted scans are skipped and counted, missing values (`None`, `nan`) are NaN, labels separated by single spaces are split according to the column count, and MCA spectra of several devices (`@A1`, `@A2`...) are read with their own channel range and calibration. +- Each object records its origin in its `experimental_source` metadata: file, scan, columns, command, date and motor positions; the scan header (count time, monitor, hkl, geometry, comments, Bluesky metadata) is kept as `SPEC ...` metadata. A saved workspace reopens without the SPEC file or the plugin. +- **XDI files** (XAS Data Interchange, as written by Larch, Athena and many beamlines): **File > Open** reads every column versus the energy, with units, as the XDI library does: data tables with invalid values are refused with the line number, invalid header lines are recorded (`XDI problems` metadata), free text may be UTF-8, monochromator angles are converted to energies with the lattice spacing (`Mono.d_spacing`), and two-dimensional data (`Outer.value` blocks) gives one signal per block. Every header field and the comments are kept as metadata; the origin records the element, edge, facility, beamline, monochromator, sample and start time. - **X-ray absorption (XANES/EXAFS):** **Absorption from counters...** computes `mu(E)` from the selected monitor (I0, recognized by its name) and detector counts, in transmission (`ln(I0/It)`) or fluorescence (`If/I0`); points without valid counts are left out and listed. **Edge normalization...** finds the edge energy E0 of each selected spectrum and normalizes it to a unit edge step, with a pre-edge line and a post-edge polynomial (automatic or given ranges), giving the normalized and flattened spectra and a table of E0, edge step and ranges. The results are those of [Larch](https://xraypy.github.io/xraylarch/) on 10 measured spectra (transmission and fluorescence, several beamlines). Energies in keV are converted to eV, and results are given in eV. Spectra are refused, with a message telling which X unit to set, if their energies are in another unit (e.g. monochromator angles in degrees), labelled keV with values in eV, or without unit and below 150 (most likely keV, as often in SPEC files); energies without unit from 150 up are taken as eV. - **Example:** **Open copper foil example** opens the monitor and transmitted counts of a copper foil (Larch examples, MIT license) to try both recipes: E0 is 8980.5 eV, 1.5 eV above the tabulated Cu K edge, and the edge step is 2.73. -- NeXus files are read by DataLab itself (HDF5 browser), which shows `NXdata` groups as signals or images with their axes. While the plugin is loaded, the HDF5 browser also shows every counter `data_1`, `data_2`... of scan data groups as a signal versus the time stamps `time_1` of the points, with its label and units: such groups designate only their first counter, without axis. +- NeXus files are read by DataLab itself (HDF5 browser), which shows `NXdata` groups as signals or images with their axes, and stacks of images or spectra as frames, including the legacy conventions of recent SOLEIL scans: counters are signals versus the scanned motor, and detectors images or stacks with the motor first. While the plugin is loaded, the HDF5 browser also shows every counter and detector `data_1`, `data_2`... of scan data groups versus the time stamps `time_1` of the points, with its label and units: such groups (time scans of SOLEIL) designate only their first counter, without axis. -The readers are written in pure Python: the plugin installs from DataLab's plugin catalog, including with the standalone version of DataLab, and its core can run in DataLab-Web. silx is used only by the tests, as a reference reader. +The readers are written in pure Python: the plugin installs from DataLab's plugin catalog, including with the standalone version of DataLab, and its core can run in DataLab-Web. silx and spec2nexus are used only by the tests, as reference readers. ## Installation diff --git a/src/datalab_synchrotron/adapters/desktop.py b/src/datalab_synchrotron/adapters/desktop.py index ac2922f..a9acafd 100644 --- a/src/datalab_synchrotron/adapters/desktop.py +++ b/src/datalab_synchrotron/adapters/desktop.py @@ -22,10 +22,11 @@ check_scan_selection, import_scan, is_spec_file, + mesh_shape, select_scans, summarize_skipped, ) -from ..formats import unregister_formats +from ..formats import register_formats, unregister_formats from ..mappings import register_mappings, unregister_mappings from ..workflow import ( ABSORPTION_RECIPE, @@ -36,7 +37,7 @@ from ..workflow import EXAMPLES as WORKFLOW_EXAMPLES from ..workflow import RECIPES as WORKFLOW_RECIPES -#: Number of signals added to the workspace at once by the SPEC import +#: Number of objects added to the workspace at once by the SPEC import ADD_BATCH_SIZE = 200 PLUGIN_ICON = "datalab_synchrotron:icons/synchrotron.svg" @@ -45,10 +46,17 @@ def _import_parameters(spec: SpecFile) -> gds.DataSet: - """Return the import parameters of a SPEC file, with its columns as choices.""" + """Return the import parameters of a SPEC file, with its columns and MCA devices + as choices. + + The MCA and mesh items are hidden if no scan has MCA spectra or is a mesh scan. + """ labels: list[str] = [] + devices: list[str] = [] for info in spec.scans: labels.extend(label for label in info.labels if label not in labels) + devices.extend(device for device in info.mca_devices if device not in devices) + has_mesh = any(mesh_shape(info) is not None for info in spec.scans) # Default columns: those of the most frequent scan structure structures = collections.Counter(info.labels for info in spec.scans if info.labels) first = structures.most_common(1)[0][0] if structures else ("",) @@ -67,7 +75,17 @@ class SpecImportParameters(gds.DataSet): counters = gds.MultipleChoiceItem( _("Counters"), choices, default=[first[-1]] ).vertical(3) - mca = gds.BoolItem(_("MCA spectra"), default=False) + mca = gds.MultipleChoiceItem( + _("MCA spectra"), + [(device, device) for device in devices] or [("", "-")], + default=[], + help=_("One image per device and scan: one spectrum per row"), + ).set_prop("display", hide=not devices) + mesh = gds.BoolItem( + _("Mesh scans as images"), + default=True, + help=_("Counters of mesh scans as images of the two motors"), + ).set_prop("display", hide=not has_mesh) return SpecImportParameters(title=osp.basename(spec.path)) @@ -124,7 +142,9 @@ def open_cu_foil_example(self) -> PluginExample | None: return self.launch_example(CU_FOIL_EXAMPLE.id) def register_hooks(self) -> None: - """Apply the shipped HDF5 data mappings.""" + """Register the file formats (again, after an unload) and apply the + shipped HDF5 data mappings.""" + register_formats() register_mappings() def unregister_hooks(self) -> None: @@ -160,24 +180,25 @@ def import_spec_scans(self, filename: str | None = None) -> int: return 0 keys = select_scans(spec, param.scans) counters = list(param.counters) - check_scan_selection(param.x, counters, param.mca) + devices = [device for device in param.mca if device] + check_scan_selection(param.x, counters, devices) except ValueError as exc: self.show_error(str(exc)) return 0 count, skipped, cancelled = self._add_scans( - spec, keys, param.x, counters, param.mca + spec, keys, param.x, counters, devices, param.mesh ) report = summarize_skipped(skipped) if cancelled: self.show_info( - _("Import cancelled: %d signal(s) imported.") % count + _("Import cancelled: %d object(s) imported.") % count + "".join("\n" + line for line in report) ) elif count == 0: - self.show_error(_("No signal was imported:") + "\n" + "\n".join(report)) + self.show_error(_("Nothing was imported:") + "\n" + "\n".join(report)) elif skipped: self.show_warning( - _("Selected columns were not imported from %d of %d scans:") + _("Selected data was not imported from %d of %d scans:") % (len(skipped), len(keys)) + "\n" + "\n".join(report) @@ -190,12 +211,15 @@ def _add_scans( keys: list[str], x: str, counters: list[str], - include_mca: bool, + mca_devices: list[str], + mesh_as_images: bool, ) -> tuple[int, list[SkippedScan], bool]: - """Read and add scans one at a time, in a group each, until cancelled. + """Read and add scans one at a time, until cancelled: signals in a group of + the signal panel, images in a group of the image panel, named after the + scan. Returns: - Number of added signals, skipped scans, and True if cancelled + Number of added objects, skipped scans, and True if cancelled """ count, skipped = 0, [] title = _("Importing SPEC scans") @@ -209,26 +233,31 @@ def _add_scans( QW.QApplication.processEvents() if progress.wasCanceled(): return count, skipped, True - group, reason = import_scan(spec, key, x, counters, include_mca) + group, reason = import_scan( + spec, key, x, counters, mca_devices, mesh_as_images + ) if reason is not None: skipped.append(reason) if group is None: continue - group_id = self.proxy.add_group(group.title, panel="signal") - signals = group.signals - for start in range(0, len(signals), ADD_BATCH_SIZE): - progress.setLabelText( - _("Adding signals of scan %s (%d of %d)") - % (key, start, len(signals)) - ) - QW.QApplication.processEvents() - if progress.wasCanceled(): - return count, skipped, True - batch = signals[start : start + ADD_BATCH_SIZE] - self.signalpanel.add_objects( - batch, group_id=group_id, set_current=False - ) - count += len(batch) + for panel_name, panel, objects in ( + ("signal", self.signalpanel, group.signals), + ("image", self.imagepanel, group.images), + ): + if not objects: + continue + group_id = self.proxy.add_group(group.title, panel=panel_name) + for start in range(0, len(objects), ADD_BATCH_SIZE): + progress.setLabelText( + _("Adding objects of scan %s (%d of %d)") + % (key, start, len(objects)) + ) + QW.QApplication.processEvents() + if progress.wasCanceled(): + return count, skipped, True + batch = objects[start : start + ADD_BATCH_SIZE] + panel.add_objects(batch, group_id=group_id, set_current=False) + count += len(batch) progress.setValue(len(keys)) return count, skipped, False diff --git a/src/datalab_synchrotron/core/__init__.py b/src/datalab_synchrotron/core/__init__.py index 98f40b7..2b0e008 100644 --- a/src/datalab_synchrotron/core/__init__.py +++ b/src/datalab_synchrotron/core/__init__.py @@ -1,19 +1,26 @@ """Host-independent readers of experimental data formats, and XAS analysis.""" from .convert import ( + HC_EV_ANGSTROM, + MeshShape, ScanGroup, ScanImport, SkippedScan, check_scan_selection, import_scan, - mca_signals, + mca_images, + mesh_images, + mesh_shape, plan_scan_import, + scan_metadata, scan_signals, select_scans, summarize_skipped, + xdi_energy, + xdi_metadata, xdi_signals, ) -from .spec import SpecFile, SpecScan, SpecScanInfo, is_spec_file +from .spec import SpecFile, SpecScan, SpecScanInfo, is_spec_file, mca_device from .xas import ( MAX_EDGE_KEV, Absorption, @@ -28,11 +35,13 @@ from .xdi import XdiColumn, XdiFile, is_xdi_file, read_xdi __all__ = [ + "HC_EV_ANGSTROM", "MAX_EDGE_KEV", "Absorption", "AbsorptionMode", "EdgeNormalization", "EnergyUnit", + "MeshShape", "ScanGroup", "ScanImport", "SkippedScan", @@ -48,12 +57,18 @@ "import_scan", "is_spec_file", "is_xdi_file", - "mca_signals", + "mca_device", + "mca_images", + "mesh_images", + "mesh_shape", "normalize_edge", "plan_scan_import", "read_xdi", + "scan_metadata", "scan_signals", "select_scans", "summarize_skipped", + "xdi_energy", + "xdi_metadata", "xdi_signals", ] diff --git a/src/datalab_synchrotron/core/convert.py b/src/datalab_synchrotron/core/convert.py index f6d6a0c..6cb4f42 100644 --- a/src/datalab_synchrotron/core/convert.py +++ b/src/datalab_synchrotron/core/convert.py @@ -1,4 +1,10 @@ -"""Conversion of SPEC scans and XDI files to signals recording their origin.""" +"""Conversion of SPEC scans and XDI files to signals and images recording their +origin. + +The header information of the files (SPEC scan headers, XDI fields and comments) +is kept as object metadata, while the ``experimental_source`` metadata entry +identifies where the data comes from. +""" from __future__ import annotations @@ -9,47 +15,125 @@ import numpy as np from sigima.objects import ( + ImageObj, SignalObj, + create_image, create_signal, make_experimental_source, set_experimental_source, ) -from .spec import SpecFile, SpecScan -from .xdi import XdiFile +from .spec import SpecFile, SpecScan, SpecScanInfo +from .xdi import XdiColumn, XdiFile __all__ = [ + "HC_EV_ANGSTROM", + "MeshShape", "ScanGroup", "ScanImport", "SkippedScan", "check_scan_selection", "import_scan", - "mca_signals", + "mca_images", + "mesh_images", + "mesh_shape", "plan_scan_import", + "scan_metadata", "scan_signals", "select_scans", "summarize_skipped", + "xdi_energy", + "xdi_metadata", "xdi_signals", ] +#: Planck constant times the speed of light, in eV.Å (exact SI values) +HC_EV_ANGSTROM = 12398.419843320026 + +#: Relative tolerance on the steps of evenly spaced image coordinates +_UNIFORM_RTOL = 1e-9 + -def _source(spec: SpecFile, scan: SpecScan, selection: dict, conversions) -> dict: - """Return the experimental source of a signal read from a scan.""" +def _source( + spec: SpecFile, scan: SpecScan, selection: dict, conversions: Sequence[str] +) -> dict: + """Return the experimental source of an object read from a scan.""" details = {"command": scan.info.command, "date": scan.info.date} if scan.motors: details["motors"] = dict(scan.motors) if scan.problems: details["skipped_lines"] = len(scan.problems) + if scan.ignored: + details["ignored_lines"] = dict(scan.ignored) return make_experimental_source( "spec", spec.path, entry=scan.info.key, selection=selection, - conversions=conversions or None, + conversions=list(conversions) or None, details=details, ) +def _value_and_note(text: str) -> tuple[float, str]: + """Return the number and the parenthesized note of a header line, e.g. + ``60 (Clock)``.""" + match = re.match(r"(\S+)\s*(?:\((.*)\))?", text) + if match is None: + raise ValueError(f"No value in '{text}'") + return float(match.group(1)), (match.group(2) or "").strip() + + +def scan_metadata(scan: SpecScan) -> dict[str, object]: + """Return the scan header information kept as object metadata. + + Keys: ``SPEC count time`` and ``SPEC count time unit`` (``#T``), ``SPEC monitor + count`` and ``SPEC monitor counter`` (``#M``), ``SPEC hkl`` (``#Q``), ``SPEC + geometry G`` (``#G``), ``SPEC comments`` (``#C`` lines, one per line), + and ``SPEC MD `` (``#MD = `` lines written by Bluesky). + Header lines that cannot be read are left out. + """ + metadata: dict[str, object] = {} + comments: list[str] = [] + for line in scan.header[1:]: + tag, _sep, text = line.strip().partition(" ") + text = text.strip() + try: + if tag in ("#T", "#M") and text: + value, note = _value_and_note(text) + name, note_name = ( + ("count time", "count time unit") + if tag == "#T" + else ("monitor count", "monitor counter") + ) + metadata[f"SPEC {name}"] = value + if note: + metadata[f"SPEC {note_name}"] = note + elif tag == "#Q" and text: + metadata["SPEC hkl"] = np.array([float(word) for word in text.split()]) + elif re.fullmatch(r"#G\d+", tag) and text: + metadata[f"SPEC geometry {tag[1:]}"] = np.array( + [float(word) for word in text.split()] + ) + elif tag == "#C" and text: + comments.append(text) + elif tag == "#MD" and "=" in text: + name, value = text.split("=", 1) + if name.strip(): + metadata[f"SPEC MD {name.strip()}"] = value.strip() + except ValueError: + continue + if comments: + metadata["SPEC comments"] = "\n".join(comments) + return metadata + + +def _finish(obj: SignalObj | ImageObj, source: dict, metadata: dict) -> None: + """Record the header metadata and the origin of an object.""" + obj.metadata.update(metadata) + set_experimental_source(obj, source) + + def scan_signals( spec: SpecFile, scan: SpecScan, x: str, counters: Sequence[str] ) -> list[SignalObj]: @@ -61,6 +145,7 @@ def scan_signals( ValueError: if a column does not exist """ xdata = scan.column(x) + metadata = scan_metadata(scan) signals = [] for counter in counters: signal = create_signal( @@ -69,51 +154,208 @@ def scan_signals( scan.column(counter).copy(), labels=(x, counter), ) - selection = {"x": x, "y": counter} - set_experimental_source(signal, _source(spec, scan, selection, None)) + source = _source(spec, scan, {"x": x, "y": counter}, ()) + _finish(signal, source, metadata) signals.append(signal) return signals -def mca_signals( - spec: SpecFile, scan: SpecScan, indices: Sequence[int] | None = None -) -> list[SignalObj]: - """Return MCA spectra of a scan, on calibrated X if the scan declares it. +def _is_strictly_monotonic(values: np.ndarray) -> bool: + """Return True if values are finite and strictly increasing or decreasing.""" + if not np.all(np.isfinite(values)): + return False + steps = np.diff(values) + return bool(np.all(steps > 0) or np.all(steps < 0)) - The X axis is the channel number, or ``a + b * channel + c * channel**2`` - with the ``#@CALIB`` coefficients of the scan. + +def _image( + title: str, + data: np.ndarray, + coords: tuple[np.ndarray, np.ndarray], + labels: tuple[str, str, str], + conversions: list[str], +) -> ImageObj: + """Return an image with increasing coordinates, uniform if evenly spaced. + + Args: + title: image title + data: image values, one row per Y coordinate + coords: strictly monotonic X and Y coordinates + labels: X, Y and Z labels + conversions: conversions applied to the values, completed with the axes + reversed to increasing order """ - signals = [] - for index in range(len(scan.mca)) if indices is None else indices: - channels = scan.mca_channels_axis(index) - conversions = [] - xlabel = "Channel" - if scan.mca_calibration is not None: - a, b, c = scan.mca_calibration - x = a + b * channels + c * channels**2 - conversions.append(f"channels -> a + b*ch + c*ch^2 with {a}, {b}, {c}") - xlabel = "Calibrated channel" - else: - x = channels - signal = create_signal( - f"MCA {index} (scan {scan.info.key})", - np.asarray(x, dtype=float), - scan.mca[index].astype(float), - labels=(xlabel, "Counts"), + xvals, yvals = (np.asarray(values, dtype=float) for values in coords) + for axis, (values, label) in enumerate(((yvals, labels[1]), (xvals, labels[0]))): + if values.size > 1 and values[-1] < values[0]: + data = np.flip(data, axis=axis) + conversions.append(f"axis '{label}': reversed to increasing order") + xvals, yvals = np.sort(xvals), np.sort(yvals) + image = create_image(title, np.ascontiguousarray(data), labels=labels) + steps = [np.diff(values) for values in (xvals, yvals)] + if all( + step.size < 2 or np.allclose(step, step[0], rtol=_UNIFORM_RTOL, atol=0.0) + for step in steps + ): + dx, dy = (float(step[0]) if step.size else 1.0 for step in steps) + image.set_uniform_coords(dx, dy, float(xvals[0]), float(yvals[0])) + else: + image.set_coords(xvals, yvals) + return image + + +def mca_images( + spec: SpecFile, scan: SpecScan, x: str = "", devices: Sequence[str] | None = None +) -> list[ImageObj]: + """Return the MCA spectra of a scan as one image per device. + + Each row holds the spectrum of a point. The X axis is the channel number, or + ``a + b * channel + c * channel**2`` with the ``#@CALIB`` coefficients of the + device, if they give strictly monotonic values. The Y axis is the X column + of the scan, if it holds one strictly monotonic value per spectrum, else the + point number. + + Args: + spec: SPEC file + scan: scan read from the file + x: X column of the scan (empty: point numbers) + devices: MCA devices (default: all devices of the scan) + + Raises: + ValueError: if a device has no spectrum, or spectra of several sizes + """ + metadata = scan_metadata(scan) + images = [] + for device in scan.mca if devices is None else devices: + spectra = scan.mca.get(device, ()) + channels = scan.channel_numbers(device) + conversions = [f"spectra -> image rows ({len(spectra)} spectra)"] + xvals, xlabel = channels, "Channel" + calibration = scan.calibration(device) + if calibration is not None: + a, b, c = calibration + calibrated = a + b * channels + c * channels**2 + if _is_strictly_monotonic(calibrated): + xvals, xlabel = calibrated, "Calibrated channel" + conversions.append(f"channels -> a + b*ch + c*ch^2 with {a}, {b}, {c}") + else: + conversions.append("calibration not monotonic: channel numbers kept") + yvals, ylabel = np.arange(len(spectra), dtype=float), "Point" + if x in scan.info.labels: + column = scan.column(x) + if column.size == len(spectra) and _is_strictly_monotonic(column): + yvals, ylabel = column, x + image = _image( + f"{device} (scan {scan.info.key})", + np.vstack(spectra), + (xvals, yvals), + (xlabel, ylabel, "Counts"), + conversions, ) - set_experimental_source( - signal, _source(spec, scan, {"mca": index}, conversions) + selection = {"mca": device, "y": ylabel if ylabel == x else "point"} + _finish(image, _source(spec, scan, selection, conversions), metadata) + images.append(image) + return images + + +@dataclasses.dataclass(frozen=True) +class MeshShape: + """Grid of a mesh scan, as given by its command. + + Args: + columns: number of points along the first motor (fastest) + rows: number of points along the second motor + steps: steps of the first and second motors + """ + + columns: int + rows: int + steps: tuple[float, float] + + +def mesh_shape(info: SpecScanInfo) -> MeshShape | None: + """Return the grid of a mesh scan, or None if the scan is not a mesh scan. + + Mesh scans are written by ``mesh`` and ``dmesh`` commands: + ``mesh +