Repository navigation
Read real SPEC and XDI files as reference readers do, with validation corpora - #3
Merged
Merged
Conversation
SPEC: MCA spectra are grouped by device (@A1, @a2...) with their own calibration and channels, instead of mixing devices and reading the device number as a first channel. Labels separated by single spaces are split according to the column count, repeated labels get a suffix, empty motor lines and None values (Bluesky) are accepted, and other '@' lines are ignored. A scan that cannot be read is skipped with the error instead of stopping the import or File > Open. MCA spectra are imported as one image per device (one spectrum per row), mesh scans as images of the two motors, and scan headers are kept as metadata. XDI: files are read as UTF-8, invalid data tables are refused with their line as the XDI library does, invalid header lines are recorded, comments and all fields are kept as metadata, angles are converted to energies with the lattice spacing, and outer scans give one signal per block. The formats are removed and registered again with Sigima's public API. Assisted-by: Claude Opus 5.5
SPEC: extracts of spec2nexus and PyMca files covering several MCA devices, mesh scans, single-space labels, Bluesky and Sardana files, interrupted scans and repeated scan numbers, compared with the values read by spec2nexus (or silx). A script builds the extracts from pinned sources and writes the references. XDI: the test files of the XDI library, checked against its expected behaviour. Assisted-by: Claude Opus 5.5
The "Scan counters" mapping applies to detectors as well (spectra and images per point), the time stamps describing their first dimension. Step scans of SOLEIL, which follow the legacy NeXus conventions, are imported by DataLab's HDF5 browser without mapping: their integration test checks it. Assisted-by: Claude Opus 5.5
Assisted-by: Claude Opus 5.5
This was referenced Oct 27, 2026
PierreRaybaut
marked this pull request as ready for review
October 27, 2026 11:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reading of real SPEC and XDI files as the reference readers do, and validation corpora. Depends on DataLab-Platform/Sigima#109 (format registration API) and DataLab-Platform/SigimaX#23 (legacy NeXus axes, mapping axes on the first dimensions); DataLab side: DataLab-Platform/DataLab#444.
SPEC
@A1,@A2...), each device with its own#@CALIB/#@CHANN. Before, devices were mixed and the device number was read as a first channel, shifting every spectrum by one channel (silx has the same behaviour).#N, or to the width of the data when#Nis wrong (Bluesky). Repeated labels get a suffix (I0_1), as with spec2nexus.#O0lines (Bluesky) no longer crash.Nonevalues are NaN. Other@lines and their continuations are ignored and counted.#T,#M,#Q,#G,#C,#MD) is kept asSPEC ...metadata.XDI
XDI problems). Comments keep their inner spaces.Mono.d_spacing. Two-dimensional data (Outer.value) gives one signal per block.NeXus (SOLEIL)
Formats
unregister_format/register_format) instead of editing the registry's private list.Validation (data next to the code, not in Sigima)
tests/data/spec: extracts of real files from spec2nexus (CC BY 4.0, reductions listed) and PyMca (MIT), at pinned commits with SHA-256. They cover several MCA devices, mesh, single-space labels, Bluesky, Sardana, interrupted scans and repeated numbers. They are compared with the values read by spec2nexus (silx for the files spec2nexus cannot read), frozen inreferences.jsonbytests/validation/make_spec_corpus.py.tests/data/xdi: the test files of the XDI library (public domain), checked againstBadFiles.txt.None/nan, which silx reads as 0.Documentation
README, CHANGELOG (first version, unreleased), CONTRIBUTING (corpora and how to regenerate the references).