Skip to content

Read real SPEC and XDI files as reference readers do, with validation corpora - #3

Merged
PierreRaybaut merged 4 commits into
developfrom
feature/robust-spec-xdi
Oct 27, 2026
Merged

PierreRaybaut merged 4 commits into
developfrom
feature/robust-spec-xdi

Conversation

@PierreRaybaut

@PierreRaybaut PierreRaybaut commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Reading of real SPEC and XDI files as the reference readers do, and validation corpora. Depends on DataLab-Platform/Sigima#109 (format registration API) and DataLab-Platform/SigimaX#23 (legacy NeXus axes, mapping axes on the first dimensions); DataLab side: DataLab-Platform/DataLab#444.

SPEC

  • MCA spectra grouped by device (@A1, @A2...), each device with its own #@CALIB/#@CHANN. Before, devices were mixed and the device number was read as a first channel, shifting every spectrum by one channel (silx has the same behaviour).
  • Labels separated by single spaces are split according to #N, or to the width of the data when #N is wrong (Bluesky). Repeated labels get a suffix (I0_1), as with spec2nexus.
  • Empty #O0 lines (Bluesky) no longer crash. None values are NaN. Other @ lines and their continuations are ignored and counted.
  • A scan that cannot be read is skipped with the error, in the import and in File > Open, instead of stopping everything.
  • MCA spectra are imported as one image per device and scan (one spectrum per row, calibrated channels along X, X column along Y if strictly monotonic). Mesh scans are imported as images of the two motors (optional), with NaN for missing points. The scan header (#T, #M, #Q, #G, #C, #MD) is kept as SPEC ... metadata.

XDI

  • UTF-8 free text, with a Latin-1 fallback.
  • Invalid data tables are refused with their line, as the XDI library does. Invalid header lines are recorded (XDI problems). Comments keep their inner spaces.
  • Every header field and the comments are kept as object metadata.
  • Angles are converted to energies with Mono.d_spacing. Two-dimensional data (Outer.value) gives one signal per block.

NeXus (SOLEIL)

  • The "Scan counters" mapping now also applies to the detectors of time scans, thanks to mapping axes describing the first dimensions.
  • Step scans need no mapping: SigimaX reads their legacy axes. An integration test checks it.

Formats

  • Formats are removed and registered again with Sigima's public API (unregister_format/register_format) instead of editing the registry's private list.

Validation (data next to the code, not in Sigima)

  • tests/data/spec: extracts of real files from spec2nexus (CC BY 4.0, reductions listed) and PyMca (MIT), at pinned commits with SHA-256. They cover several MCA devices, mesh, single-space labels, Bluesky, Sardana, interrupted scans and repeated numbers. They are compared with the values read by spec2nexus (silx for the files spec2nexus cannot read), frozen in references.json by tests/validation/make_spec_corpus.py.
  • tests/data/xdi: the test files of the XDI library (public domain), checked against BadFiles.txt.
  • 202 tests passed; Ruff clean. On the full spec2nexus corpus (24 files), no crash, and the same values as silx except None/nan, which silx reads as 0.

Documentation

README, CHANGELOG (first version, unreleased), CONTRIBUTING (corpora and how to regenerate the references).

SPEC: MCA spectra are grouped by device (@A1, @a2...) with their own
calibration and channels, instead of mixing devices and reading the device
number as a first channel. Labels separated by single spaces are split
according to the column count, repeated labels get a suffix, empty motor
lines and None values (Bluesky) are accepted, and other '@' lines are
ignored. A scan that cannot be read is skipped with the error instead of
stopping the import or File > Open.

MCA spectra are imported as one image per device (one spectrum per row),
mesh scans as images of the two motors, and scan headers are kept as
metadata.

XDI: files are read as UTF-8, invalid data tables are refused with their
line as the XDI library does, invalid header lines are recorded, comments
and all fields are kept as metadata, angles are converted to energies with
the lattice spacing, and outer scans give one signal per block.

The formats are removed and registered again with Sigima's public API.

Assisted-by: Claude Opus 5.5
SPEC: extracts of spec2nexus and PyMca files covering several MCA devices,
mesh scans, single-space labels, Bluesky and Sardana files, interrupted
scans and repeated scan numbers, compared with the values read by
spec2nexus (or silx). A script builds the extracts from pinned sources and
writes the references.

XDI: the test files of the XDI library, checked against its expected
behaviour.

Assisted-by: Claude Opus 5.5
The "Scan counters" mapping applies to detectors as well (spectra and
images per point), the time stamps describing their first dimension. Step
scans of SOLEIL, which follow the legacy NeXus conventions, are imported by
DataLab's HDF5 browser without mapping: their integration test checks it.

Assisted-by: Claude Opus 5.5
@PierreRaybaut
PierreRaybaut marked this pull request as ready for review October 27, 2026 11:57
@PierreRaybaut
PierreRaybaut merged commit 2b98872 into develop Oct 27, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant