Design record: serializing and exchanging quantities across MOLI and MolSysSuite

[dprada]

This issue is the design record for how MOLI and MolSysSuite components serialize and exchange physical quantities. It records why the discussion happened, what was measured, what was discarded, what was chosen and what remains open. Questions or new proposals about quantity serialization in any MOLI or MolSysSuite component belong here, or in the implementation issue #82. They should not restart in a consumer repository. The durable form of this record should become a PyUnitWizard devguide page shipped with the implementation (#82), so that the reasoning outlives this thread.

The discussion took place on 2026-09-24 between the Sabueso lead (Diego) and an agent working on Sabueso. Diego took the decisions below.

1. Why the discussion happened

The failure to rule out: one tool stores or computes 3 nanomolar and the next reads 3 picomolar. Nothing fails; the number is just wrong. A dimension check cannot catch it, because both are concentrations. Only a unit that never leaves the value, and a reader that never assumes one, prevent it.

Evidence found while studying the components (origin/main, 2026-09-24):

• MolSysViewer (Standardized lengths are sent to the frontend as nm, 10x off under a user Å policy molsysviewer#96). It takes a standardized value from MolSysMT, strips the unit and assumes nm. Under a user session policy in Å, the simulation box is drawn 10× too large, silently. Measured on 1tcd.h5msm: msm.get(..., box=True) returns 4.371 under MolSysMT's policy and 43.71 under a user Å policy, and both are treated as nm.
• MolSysMT H5MSM (uibcdf/molsysmt, reported alongside). The unit of the same data lives in three places: root attributes, the structures group and each dataset. Different readers consult different places: get_structural_attributes reads the root, to_molsysmt_Structures reads the dataset, then the root, then a literal default ("nm", "ps", "nanometer**2"). The writer writes the values and their unit in separate statements.
• Sabueso (Decide the serialized representation of physical quantities in cards and SourceAssertions sabueso#32). It expresses units in four different ways:
  • in the field name (resolution_angstrom, test_concentration_uM);
  • in metadata ({"unit": "Da"});
  • as the source's verbatim string (ChEMBL units: "nM");
  • nowhere at all (tpsa, where Ų is implicit).
• Unit symbols are case-sensitive and collide across dimensions. nM is nanomolar and nm is nanometre. NM is rejected, correctly. Source notations such as ChEMBL's ug.mL-1 fail in pint, which reads -1 as a subtraction.
• The string form does not round-trip arrays (The string form of an array quantity cannot be parsed back #81).

Related platform work: uibcdf/moli#11 (who configures the shared kernel), uibcdf/moli#12 (quantity interchange contract), uibcdf/molsyssuite#18 (unit-configuration authority).

2. What was measured

Environment: Python 3.13.14, pint 0.25.3, numpy 2.4.6, pyunitwizard 0.25.0+7.g00d756c, Linux, Xeon E5-2630 v4. Timings vary by about 2× between runs on this machine, so treat them as orders of magnitude. The data is 100,000 concentrations unless stated otherwise.

Representation	JSON size	gzip	Load	Memory per value
bare numbers (unit implied)	0.98 MB	0.42 MB	12 ms	24 B (Python float)
{value, unit} per value	4.18 MB	0.50 MB	58 ms	192 B (dict)
unit once per container	0.98 MB	0.42 MB	12 ms	≈ bare
unit-tagged array (per-value 1-byte code + local unit table), text	1.28 MB	0.46 MB	25 ms	9 B
unit-tagged array, base64 binary	1.20 MB	—	2.9 ms	9 B

Operation	Cost
puw.quantity(value, unit) for one scalar	17 µs (100k scalars: 1.7 s)
puw.quantity for one 100k array	4.4 ms; conversion of that array: 0.13 ms
get_value(q, to_unit=...) for a scalar	25 µs
puw.check(q, dimensionality=...)	14 µs
puw.conversion_factor	1.8 µs
decode and convert 100k records with a factor cached per distinct unit	19 ms (0.19 µs/value)
comparing two scalar quantities vs two floats	28 µs vs 45 ns
pint Quantity.from_tuple vs parsing "3.0 nanomolar"	5.3 µs vs 13.8 µs
first quantity in a process (registry build)	490 ms; 311 ms with set_pint_registry_cache(True). MolSysMT's _pyunitwizard.py comments claim 180 → 17 ms; worth re-measuring.
blake2b-128 digest, 100k float64 (0.8 MB)	1–3 ms
blake2b-128 digest, 48 MB float32 (1000 frames × 4000 atoms × 3)	80–150 ms (≈0.3–0.6 GB/s); sha256 similar; crc32 3–5× faster

Other facts checked:

• Using PyUnitWizard without configuring it does not activate a policy (has_active_policy() stays false).
• pint quantities from different UnitRegistry instances cannot be combined (ValueError), so every component must obtain quantities through PyUnitWizard's registry.
• pint normalizes spellings: N*s/m**2, s*N/m^2 and newton*second/meter**2 all become 'newton * second / meter ** 2'. Dimensionally equal units stay distinct (kg/(m*s), Pa*s), with a factor of 1 between them.
• conversion_factor("degC", "K") raises: affine units fail loudly instead of multiplying.
• -log10(M) (pIC50, pChEMBL, pKa) is not a unit.
• Floating-point noise: 5 pM → 4.9999999999999996e-06 µM.

3. What was discarded, and why

• {value, unit} on every value, for collections. 4× the size and 5× the load time, 192 B per value, and clumsy for arrays. It is still fine for a lone scalar.
• The string form as the canonical format. Parsing costs about 14 µs per value, it cannot be indexed numerically, and it does not round-trip arrays today (The string form of an array quantity cannot be parsed back #81).
• pint's structured tuple as the canonical format. It is fast for one scalar, but it creates one object per value and ties the format to pint.
• A unit manifest per container with nothing enforcing it. It costs nothing, but it permits exactly the H5MSM mechanisms: separate writes, several unit sources and silent defaults. Rejected as the contract.
• A global catalogue of unit tags. If code 7 changed meaning between versions, old files would be misread silently. Unit tables must be local to each object or document.
• Treating logarithmic scales as units. They are not; see open question 3.
• crc32 or xxhash for the digest. crc32 is not collision-resistant, and xxhash is an extra dependency. blake2b ships with Python and is fast enough.

4. What was chosen: negotiated containers with an integrity digest

Diego's decisions:

• Quantity-valued answers are returned as quantities.
• Canonical unit strings are ones PyUnitWizard parses and that round-trip, in the unambiguous long form.
• The balance of memory, speed and safety favours negotiated containers with a digest.

The design has three layers, each with a clear owner:

Layer	Guarantee	Owner
API contract	Writers hand over a quantity with the expected dimensionality, never a bare number	ArgDigest (contrib/pyunitwizard_support.check)
Codec	The only writer and reader. It writes the manifest next to the values, converts with an explicit to_unit, and reads with a handshake	PyUnitWizard (#82)
Integrity digest	Any change made outside the codec fails on read	the codec (hashlib)

Rules:

1. The negotiated unit is written in the document, inside the same serialized object as the values, never only in a schema.
2. Each field has one unit source, with no root-level fallback.
3. There are no defaults. A missing manifest is an error.
4. Values enter and leave only through the codec, from and to quantities. Appends convert incoming quantities to the negotiated unit.
5. Reader handshake. The reader declares the field and the expected unit or dimensionality: an exact match, or an explicit conversion. A dimension mismatch is an error.
6. Digest. blake2b-128 over the canonical manifest (format, field, canonical unit, dtype, shape) plus the values' little-endian bytes, not the JSON text. Reformatting a document is harmless; changing a number is not.
7. Tests:
   • conformance under a non-default session policy in every component (it would have caught molsysviewer#96);
   • cross-component canaries (one component writes 3 nM, another asserts that it reads 3 nM);
   • a static guard against low-level writes (h5py, SQL, raw JSON) outside the codec module.

A prototype subjected to deliberate slips detected 9 of 9:

• a raw append;
• a hand edit of a value;
• a unit changed in the manifest (nM → pM);
• a unit respelled by hand;
• a deleted manifest;
• reordered values;
• truncated values;
• a record copied into another field;
• length values pasted into a concentration field.

It verified an indented re-serialization. It cannot detect a writer that is consistently wrong, for example one that meant nM but built and labelled pM. ArgDigest covers that case when the dimension differs; the canaries and conformance tests cover it when the dimension is the same. The threat model is mistakes, not adversaries: someone who recomputes the digest is out of scope, and that would need signatures.

The digest detects more than per-value tags do. With {value, unit} or tags without a digest, a hand edit, a reorder, a truncation or pasted values of the same unit all pass unnoticed.

Unit-tagged arrays (working name TaggedQuantities: values plus a small per-value code into a local unit table) remain useful for genuinely heterogeneous collections, protected by the same digest. At 9 B per value, conversion of mixed units is vectorized (0.38 ms for 100k across 4 units), and the round trip is bit-identical. The layout maps onto Arrow and Parquet dictionary columns and onto HDF5.

Prototype codec used for the slip tests (sketch, not the API)

def write(field, quantity, unit):
    if not puw.is_quantity(quantity):
        raise TypeError("the codec writes quantities, not bare numbers")
    values = np.asarray(puw.get_value(quantity, to_unit=unit))
    manifest = {"format": 1, "field": field, "unit": canonical(unit),
                "dtype": values.dtype.str.lstrip("<>=|"), "shape": list(values.shape)}
    return {"manifest": manifest, "values": values.tolist(), "digest": digest(manifest, values)}

def read(record, field, expected_unit=None, dimensionality=None):
    m = record.get("manifest") or error("no manifest; there is no default unit")
    m["field"] == field or error("record belongs to another field")
    values = np.asarray(record["values"], dtype=m["dtype"]).reshape(m["shape"])
    digest(m, values) == record["digest"] or error("changed outside the codec")
    q = puw.quantity(values, m["unit"])   # then check dimensionality / convert to expected_unit

def digest(manifest, values):  # blake2b-128 over canonical JSON manifest + little-endian bytes

5. Open questions (decide here, not downstream)

1. Broadcast codes. One code for a whole array, or per column, avoids +25 % memory on float32 trajectories. It stays inside the same object, always written explicitly and never absent. The alternative is always one code per value. Leaning towards broadcast, but this touches the core concern and needs an explicit decision.
2. Offset units (°C). Affine conversion in the codec, or a policy of storing absolute temperatures in K (lossy relative to the source)?
3. Logarithmic scales (pChEMBL, pIC50, pKa). Store the underlying quantity and derive the scale? Some sources give only pKa. The alternative is a named quantity kind with its definition ("kind": "pchembl"), never a bare dimensionless number.
4. Granularity of the digest. Per record for write-once data. Per block plus a top digest over the block list for growing data (frames, rows). Per document for small scalar sets (a card), where a per-scalar digest would outweigh the value.
5. Canonicalization. NaN payloads, −0.0, float32 through JSON text (the dtype in the manifest makes the recast exact), and stable canonical spelling owned by PyUnitWizard even if pint changes its formatting.
6. Verification policy. Always on read, possibly lazily on first access. Never disabled silently: an explicit opt-out emits an SMonitor warning.
7. Legacy data and tooling. A marked legacy read mode for files without a digest, and an explicit "reseal" tool for deliberate edits (e.g. fixtures).
8. Stated precision. A float does not keep the decimals a source stated (33.0 vs 33). Consumers that need them, such as Sabueso's asserted_value, keep the source text themselves.

6. Where things go

• Implementation: QuantityRecord: PyUnitWizard's native inert interchange form (verified quantity records) #82. The codec, manifest and digest, the handshake, TaggedQuantities, and the devguide page that turns this record into documentation.
• Array string round trip: The string form of an array quantity cannot be parsed back #81.
• Platform contract: Quantity interchange contract: a unit is never lost when a value leaves a component moli#12.
• Routing, so the discussion stays here: the uibcdf/moli and uibcdf/molsyssuite issues linked below.
• First consumer: Sabueso (Decide the serialized representation of physical quantities in cards and SourceAssertions sabueso#32).
• Known cases to migrate: Standardized lengths are sent to the frontend as nm, 10x off under a user Å policy molsysviewer#96, and MolSysMT H5MSM (linked below).

[dprada]

Prior art: the ArgDigest "passport" and value certification (read before extending this design)

Diego pointed out that a "passport" for quantities was attempted before. It was, and its history bears directly on this design:

• ValidatedPayload (the passport) was removed from ArgDigest on 2026-08-13 (b402d71; MolSysMT 2eca46926). The defects are recorded in uibcdf/argdigest devguide/solved_bugs/the_passport_is_two_classes_and_admits_too_much.md:
  1. The class existed twice, in core/contract.py and in contrib/pyunitwizard_support.py. The decorator could never recognise payloads emitted by the PyUnitWizard pipelines, so the wrapper reached function bodies in place of the array.
  2. It was honoured by argument name alone, with no record of which verification had been performed. One library's claim silenced another library's check for the same name.
• Value certification (certify(value, by=..., guard=...)) replaced it as a design and was built, measured and declined. It is kept in uibcdf/argdigest devguide/pending_proposals/value_certification/. There were three reasons:
  • it asked every digester author to learn three concepts;
  • no consumer had the problem;
  • the real cause was that asking "is this quantity already canonical?" was expensive. PyUnitWizard fixed that with a cheap predicate (puw.check(q, unit=...) went from 887 µs to 10 µs).
• standards/PYUNITWIZARD_GUIDE.md now lists as an anti-pattern: "identity registries, passports, or mutable-object caches whose only purpose is to remember that a quantity was canonical."

How this design differs, and must keep differing:

	Passport / certification	This design (#83, #82)
Problem	performance: avoid re-validating a value in memory	correctness: a value at rest or crossing a boundary must never be read in the wrong unit
What it does with a claim	skips a check when a claim is present	never skips anything: every read recomputes the digest and runs the handshake
Where it lives	registry or wrapper attached to live Python objects	inside the serialized record, next to the values
Evidence of need	no consumer used it	a measured silent 10× error (uibcdf/molsysviewer#96), latent H5MSM misreads (uibcdf/molsysmt#240), four inconsistent unit styles in Sabueso (uibcdf/sabueso#32)

The digest remembers nothing about canonicity. It protects data against changes made outside the codec, so it does not fall under the guide's anti-pattern. The guide should say so explicitly when #82 lands, so that the two are not conflated.

Lessons carried over, as requirements for #82:

1. A claim must name what it certifies (passport defect 2). The digest covers the field path, the canonical unit, dtype, shape and format version, not just the bytes. A record copied into another field fails; the prototype checked this.
2. One implementation, one place (passport defect 1). The codec lives in PyUnitWizard only. ArgDigest's contrib/pyunitwizard_support and every consumer call it and never re-implement it.
3. Nothing new for digester authors to learn (certification's first reason). Writers call one function with a quantity; readers call one function with the expected field and unit. The API contract uses ArgDigest's existing check pipeline.
4. Start from a measured consumer case (certification's second reason). The cases are the three listed above. If none of them needs a feature, it does not go in.

[dprada]

Routing and adoption, so that this remains the single home of the topic:

• Route every question about serializing quantities to PyUnitWizard's design record moli#13: MOLI points every question about serializing quantities here.
• Route quantity serialization to PyUnitWizard's design record and track member adoption molsyssuite#46: MolSysSuite does the same and tracks member adoption.
• Cases to migrate: H5MSM: three unit locations, readers that disagree, and silent unit defaults molsysmt#240 (H5MSM unit locations and silent defaults) and Standardized lengths are sent to the frontend as nm, 10x off under a user Å policy molsysviewer#96 (standardized lengths assumed to be nm).
• First consumer: Decide the serialized representation of physical quantities in cards and SourceAssertions sabueso#32.
• Platform contract: Quantity interchange contract: a unit is never lost when a value leaves a component moli#12, amended to this design.

[dprada]

Alternatives and prior experience (review, 2026-09-24)

This checks whether others have met the same problem, and how they solved it, before building our own. Each claim below was checked against its source. Where it was not, this says so.

The problem is real, and it happens at interfaces

• Mars Climate Orbiter (1999). The Software Interface Specification required impulse in N·s, and the ground software wrote lbf·s into the "Small Forces" file. That is a factor of 4.45, and the spacecraft was lost. The unit was agreed in a document, not carried by the data. This is exactly why rule 1 of this design puts the negotiated unit inside the serialized object. (NASA LLIS 641, MIT summary)

• ChEMBL itself sees nM/µM confusions in its data. It flags "otherwise identical entries whose activity values differ by exactly 3 or 6 orders of magnitude" as a Potential transcription error. data_validity_comment also has Non standard unit for type. (ChEMBL FAQ)

  This matters because it is the one error class no format can detect: a source that is wrong and consistent. Sabueso already keeps data_validity_comment. A cross-source "3 or 6 orders of magnitude" check is a candidate derived check.

• Units libraries are under-adopted. A survey of scientific developers reports limitations that impede adoption (McKeever 2021, SPE; ICSE-SEIP 2020). I could not retrieve the full text, so the specific reasons are not verified here. This supports the "nothing new to learn, light and fast" requirement.

What exists, and what each one covers

Solution	What it does	Units	Integrity	Handshake / no defaults	Notes
pint	str(q) or to_tuple() (docs); no format	pint names	—	—	Quantity kinds requested in pint#676, closed without implementation
OpenFF (molecular simulation)	pydantic models serialize {"val": …, "unit": "…"} (openff-units, openff-models)	pint	—	—	Per-value records, the closest precedent in our domain
ASDF (astronomy)	quantity tag with value + unit (schema); binary blocks with a checksum (layout)	astropy (the schema does not name the standard)	MD5 optional (all zeros = skip), over the block bytes only, behaviour on mismatch undefined	—	Closest overall; its checksum does not protect the unit metadata
CF / UDUNITS (NetCDF, and Zarr via convention)	A units attribute is required for dimensional variables and is case-significant; standard_name gives the kind (CF units, Zarr CF)	UDUNITS	—	required attribute, but no reader handshake	Container manifest, decades of use: the H5MSM model done well
HDF5 Fletcher32 / Parquet page CRC / BagIt	Checksums of chunks, pages or files (h5py, BagIt)	—	recomputed by any writer: detects corruption, not writes that bypass the codec	—	Cannot replace a digest only the codec produces
UCUM (HL7/FHIR, LOINC)	Formal, case-sensitive grammar, with a case-insensitive variant "when there is a risk of upper and lower case to be confused"; mol/l and mol/L; %; no M for molar (M is mega) (ucum.org)	UCUM	—	—	The best portable spelling. Ångström and pH: not found in the text consulted; to confirm
QUDT (NASA origin)	Each unit has a conversion multiplier to SI, a dimension vector and a quantity kind (overview)	QUDT IRIs	—	—	Our si block is what QUDT standardizes; kinds solve Hz/Bq, J/N·m
UO (OBO Units Ontology)	Biomedical unit terms, used by OBI and others (OBO Foundry, paper)	UO IDs	—	—	Relevant to Sabueso's biological annotations
D-SI (PTB/EMPIR SmartCom)	Metrology exchange format, strict SI, with XML / JSON-LD / ontology implementations (PTB, v2.2.1)	SI syntax	(signatures at the calibration-certificate level)	—	Heavy; built for certificates, not large arrays

Conclusion

Nothing found combines the three pieces this design needs:

• a digest over units and values together, produced only by the codec;
• a mandatory reader handshake;
• no defaults.

The absence from this search is not proof that nothing exists, but ASDF, CF and OpenFF each cover one piece.

Changes to adopt instead of reinventing:

1. Spelling. Record a UCUM code for third parties next to PyUnitWizard's canonical name (open question 5). Readers cross-check both, as with si.
2. Kinds. An optional kind with a QUDT quantity-kind (or UO) identifier resolves the Hz/Bq and J/N·m ambiguity that SI exponents cannot.
3. si block. Keep its semantics identical to QUDT's multiplier and dimension vector, so that it is standard rather than ours.
4. Bindings. Layer on existing containers rather than defining a new file format. In HDF5, NetCDF and Zarr, also write the CF units attribute, so that every CF tool still reads the unit, and add our manifest and digest beside it. That gives CF-compatible and verified data.
5. Domain checks. For sources that are consistently wrong, adopt heuristics such as ChEMBL's 3-or-6-orders check. Sabueso already keeps data_validity_comment.

A prototype of the format draft, with a standard-library-only third-party reader and 40 tests, exists and will be attached to #82 when the revised draft (a layer over containers, with UCUM, kinds and the CF binding) is ready.

[dprada]

Follow-ups from the review: #85 (interoperability forms and unit dialects: openff-units, ASDF, CF/UDUNITS, UCUM, QUDT) and #84 (a defect found while checking OpenFF: quantities from another pint registry are not recognized and cannot be converted).

[dprada]

Design direction: a native inert interchange form (Diego, 2026-09-24)

Decision. The codec of #82 becomes PyUnitWizard's native interchange form: values plus a verified unit descriptor, made up of

• the canonical name;
• a UCUM code;
• the SI factor, offset and exponents (semantics aligned with QUDT);
• an optional kind (QUDT or UO);
• an optional digest.

It is a form in PyUnitWizard's sense, like "string". It is inert: it does not compute; to compute, you convert to pint, openmm.unit, astropy or unyt. TaggedQuantities is its heterogeneous variant.

What it gives that no current form does:

1. Serialization stops being a separate API. Writing is puw.convert(q, to_form=<native>) and reading is puw.convert(rec, to_form="pint"). There is nothing new to learn, which was the lesson of the passport.
2. A translation hub. Today translations between forms are defined pairwise (dict_translate_quantity[from][to]). With a complete SI descriptor, every form can reach every other through the native one: 2N translators instead of N², without depending on pint for, say, openmm → astropy. This also addresses Quantities from another pint UnitRegistry are not recognized and cannot be converted #84 at the root and eases OpenFF compatibility: openff-units form, OpenMM consistency and serialization interop #86 (OpenFF).
3. Kinds, UCUM and verification, which pint lacks.
4. Readable by third parties with no dependencies.
5. Light: zero-copy numpy arrays and a small descriptor.

Risks, as requirements for #82:

• There are two representations of the unit, the descriptor and the backend's unit. Every conversion cross-checks them, and a disagreement is an error.
• The form stays inert on purpose.
• It is not a "passport". It remembers nothing about live objects and is verified on every read. The guide should say so explicitly.

Not now: a live, computing native form. The reasons are the declined rusterization, the fragmentation of unit libraries, and the fact that computation is already solved elsewhere. Future re-evaluation is recorded in #87, with the problems of existing libraries it would have to solve and the conditions to reopen it.

[dprada]

Naming: the native inert interchange form is QuantityRecord (form name "record", format qrec/<version>). Details in #82.

[dprada]

Decision: TaggedQuantities becomes the tagged layout of QuantityRecord, not a separate class. This settles open question 1 (broadcast codes): the homogeneous layout is the broadcast case. Specified now; implemented when a consumer needs it. Details in #82.