diff --git a/GLOSSARY.md b/GLOSSARY.md index 6043bc0..dc87f26 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -5,10 +5,12 @@ The words this repository uses, each in one line. [SPEC.md](SPEC.md) says the re - **Item**: one thing that has results, such as a product or a dataset. Its file calls it by its own word (`item.one`). - **Indicator**: one impact category, named by an exact id. - **Reference unit**: the unit an item's result is per (kg, MJ, machine hour). Floors are kept apart per unit. -- **State**: one set of results. The first may be the published results; the others are builds. -- **Step**: from one state to the next, with its kind, claim, tolerance and noise. +- **State**: one set of results. The first may be the published results (exact: no builds); the others are builds, each with its noise. +- **Step**: from one state to the next, with its kind, claim, tolerance and noise, the sum of its two states'. - **Step 0**: the reproduction of the published results. The start, never the target. A chain that does not start from the published results has no step 0: its steps are numbered from 1. - **Chain**: the states in order and the steps between them. +- **Whole way**: from the first state to the last, every step together. +- **Run of steps**: two or more steps that follow each other, short of the whole way; compared as the two built states it lies between. Steps apart are not compared on a chain. - **Pair**: one item in one indicator, before and after a step. - **Class**: what a pair is, by one rule: agrees, within noise, differs, beyond 10x, zero reference, zero result, sign change, near zero, missing. - **Moved**: a pair of class differs, beyond 10x, zero reference, zero result or sign change. Only these have a direction. @@ -16,7 +18,7 @@ The words this repository uses, each in one line. [SPEC.md](SPEC.md) says the re - **Band**: the direction and the factor between the two sizes of a pair that differs. - **Floor**: the size below which a result counts as near zero; one per indicator and reference unit. - **Tolerance**: how far two results may differ, relative to the larger, and still agree. -- **Noise**: how far two builds of one state differ, measured. +- **Noise**: how far two builds of one state differ, measured; it belongs to the state. A comparison of two states takes the sum of theirs, never that of a state between them. - **Size filter**: a reader's choice on the page that hides small changes. It changes no count. - **Claim**: what a step says it moves; its **scope entries** say which indicators and items, which way, how far, and whether they **must** or **may** move. - **Verdict**: how a pair of a step holds against the step's claim. diff --git a/SPEC.md b/SPEC.md index 20d5d03..31255ca 100644 --- a/SPEC.md +++ b/SPEC.md @@ -56,14 +56,17 @@ step 0 is counted as movement, never as better (see [A22](#11-acceptance-cases)) A **state** is one set of results: an item's result per indicator. The first state may be the published results (`kind: "published"`); every other state is a build (`kind: "build"`). A state's `identity` names what makes it: commit, release, machine, and the solver, since -two solvers on one package differ by about 3e-7. +two solvers on one package differ by about 3e-7. A build has a **noise**: how far two builds +of it differ, measured (section 5.3), or "not measured". The published results have no +builds and carry none: they are exact. A **step** goes from one state to the next. Its kind is free text that the writer chooses (`reproduction`, `fix`, `mapping`, `contribution`, `rebuild`, ...). A step has: - a **claim** (optional): what it says it moves (section 6); - a **tolerance**: how far two results may differ and still agree; -- a **noise** (optional): how far two builds of one state differ, measured; +- a **noise**: the sum of the noise of its two states (section 5.3), worked out, never + declared on the step; - `explained`: `false` for a step that no cause explains (a rebuild on another machine, a change nobody can name). Its moves are counted as what nothing explains. @@ -88,6 +91,25 @@ that step, made up, and a test holds what it shows. **Order matters.** Where two steps touch the same results, each one's share depends on the order they were applied in. The file keeps the order; the page says so. +**A run of steps.** A file holds one comparison per step and one for the whole way. A run +of two or more consecutive steps, short of every step (steps 3 to 9), is not in the file: +the page judges it from every item's results (`together.py` is the reference). Its pair is +the item's result in the state before the run's first step and in the state after its last: +two built states, never a sum of steps. It is judged by the one rule (section 5) with the +largest tolerance of its steps and, per indicator, the sum of the noise of those two states +(5.3): the pair holds no result of a state inside the run, so neither does its noise. A run +has no verdict: each claim is judged +in its own step. A run needs every item, so it needs the matrix, or a file that lists every +item. Before it draws a run, the page judges each of its steps on its own from the same +results and holds that to the file's class and direction of the step for every item; a run +whose steps disagree with the file is refused, not drawn. + +**Steps apart are not compared on a chain.** A step's difference was measured with every +step before it applied, the left-out ones too, and an LCA result is not linear in the +technosphere, so adding up steps apart would use differences measured in a state the set +never had. The page refuses them and says why. Any set of fixes is the business of a file +of fixes measured on one base (section 16, proposed in pull request 23). + **One chain per reference.** A database with two references (sentier-bafu-agribalyse has ADEME's published results on ecoinvent and its own build on ecoinvent) gets one file per reference. A joint view is an open question (section 15). @@ -121,9 +143,9 @@ control characters, bounded lengths. | `item` | `{"one": "product", "many": "products"}`: the word for an item | | `indicator` | Optional. `{"one": "impact category", "many": "impact categories"}`: the word for an indicator, which the page and the summary use; null or absent for "indicator" | | `indicators` | `[{id, name, short, unit}]`. An indicator is matched by its exact id, never by a prefix | -| `rule` | The floors and the tolerance of the whole way (section 5) | -| `states` | `[{id, label, about, kind, identity, join}]` in order. `identity` is a short table of texts (commit, release, machine); `join` is what the reader counted when it matched its rows to items: `{compared, unmatched, ambiguous, note}`, or null | -| `steps` | `[{id, from, to, label, about, kind, explained, link, claim, tolerance, noise}]`, one fewer than the states, each from one state to the next | +| `rule` | The floors, the tolerance of the whole way and its noise, `total_noise`: the sum of the noise of the first and last state (section 5.3) | +| `states` | `[{id, label, about, kind, identity, join, noise}]` in order. `identity` is a short table of texts (commit, release, machine); `join` is what the reader counted when it matched its rows to items: `{compared, unmatched, ambiguous, note}`, or null. `noise` is `{measured, between, by, note}`: `between` names two builds of that state and `by` holds a share per indicator measured; every build carries it, measured or not, and the published results never do (exact). A file made before noise belonged to a state has it on no state | +| `steps` | `[{id, from, to, label, about, kind, explained, link, claim, tolerance, noise}]`, one fewer than the states, each from one state to the next. Where the states carry their noise, a step's `noise` is the sum of its two states', `between` null, and the validator holds it to them; it reads `measured: true` whether one end measured or both, and which did is in the states. In a file made before, it is the step's own, with its two builds | | `counts` | `{steps: {step: {indicator: block}}, total: {indicator: block}}` | | `attribution` | `{indicator: {dominant: {step: n}, not_attributable: n, spread: n, unexplained_moved: n}}` (section 7) | | `items` | The listed items, worst first (section 10.4) | @@ -191,6 +213,9 @@ many items as a file may (5,000); a state's identity has up to 12 entries. Ids a 100 characters, labels 80, texts 600 and an explanation's text 2,000. The limits are in `fileformat.py`, and a refusal names the limit it met. +Thirty states is 29 steps. A chain is the built states in order; hundreds of fixes, chosen +in any set, are a file of fixes measured on one base (section 16), not hundreds of states. + ### 4.5 The matrix of every item A file lists the items worth a look, up to `listed`. A project looking at its own build @@ -213,7 +238,10 @@ One row per item, in the order of the listed items (worst first): `[id, name, pl group, unit]`; the results per indicator and state, indicator after indicator; per indicator one code per step and one for the whole way; and, where a step has a claim, per indicator one verdict code per step (`.` for a step without a claim; `verdicts` is null when -no step has one). The class codes are `=` agrees, `~` within noise, `0` near zero, `-` +no step has one). A result equal to the one before it in the same indicator, bit for bit, is +written `"="`: in a chain of many steps most results do not change, and this keeps the +matrix small. A `"="` that stands first in an indicator or after a missing result is +refused. The class codes are `=` agrees, `~` within noise, `0` near zero, `-` missing, and for a pair that moved its class in upper case up and lower case down: `D` differs, `B` beyond 10x, `R` zero reference, `T` zero result, `S` sign change. The verdict codes are `a` as claimed, `o` off claim, `n` not claimed, `c` claimed, not moved, `~` within @@ -293,6 +321,34 @@ it. Two words, two jobs: tolerance (agreement by rule) and noise (measured instability). A third, the reader's **size filter**, hides small changes on the page and changes no count (section 10.5). +**Noise belongs to a state** (decided by Manuel on 17 November 2026). A state's noise is how +far two builds of it differ; the published results have no builds and are exact. A step, a +run of steps and the whole way are each a comparison of two states, and each is judged with +the **sum** of the noise of its two ends, per indicator: rebuilding both ends moves the +difference between them by up to the sum of their noise, and nothing cancels between two +ends. A state inside a run adds nothing: the pair holds none of its results. So a step takes +the noise of its two states, a run that of the state before its first step and the state after +its last, and the whole way (`rule.total_noise`) that of the first and last state. A set of +fixes measured on one base takes the same rule, repeated (section 16). + +An end that did not measure an indicator adds nothing to it, and the page says at how many of +the two ends the noise was measured ("measured at 1 of 2 ends"): a noise measured at one end +is never a bound for the comparison. An indicator measured at no end has no noise, and the page +says "not measured", never "identical"; the published results read "exact". + +Why: sentier-agribalyse measured noise in upstream main, whose random identifiers make water +use differ by up to 21 % between two builds. The rule before this one took the largest noise +of any step for the whole way and for a run, so main's noise, a state inside them, put 1,504 +of 2,451 water results "within noise" where the difference from the published results, exact +at one end and rebuilt bit-identically at the other, was real (A30). The rule before that +judged the whole way without noise, which called a move within the noise of its last state +"differs" (A28). Each comparison now carries the noise of what it compares, no more and no +less. + +A file made before noise belonged to a state says a noise per step and never which state it was +measured on: its steps are judged with their own noise, its runs and its whole way without +noise, and the page says so. The validator refuses a `total_noise` in such a file. + One number per indicator is coarse: a few unstable items make every smaller difference of that indicator "within noise". Two builds that are identical are a finding, shown as "identical", and never as "not measured". Large noise is a defect to fix (sentier-agribalyse's @@ -474,11 +530,16 @@ would. It opens from disk; no build step, no server, no network, no font or scri elsewhere. Where the file names a matrix that lies beside it, the page writes it as `matrix.js`, which loads from disk too; a site serving `matrix.json` needs nothing else, as the page fetches it over HTTP. Served over HTTP the page can also read `?file=...`, and it -opens a file a reader chooses (read in the browser, sent nowhere). **The page works out no -class, no direction and no count of its own.** It reads them from the file, which the -validator has checked, and only adds the file's counts up: the classes that moved into -"moved", and a step's counts over its indicators. The one number it works out itself is -the median and percentiles of a distribution (10.3), and only from every item. A file read by `?file=` or chosen by a +opens a file a reader chooses (read in the browser, sent nowhere). **For one step and the whole way, the page works out no class, no direction and no count +of its own.** It reads them from the file, which the validator has checked, and only adds +the file's counts up: the classes that moved into "moved", and a step's counts over its +indicators. Two things it works out itself, and only from every item: the median and +percentiles of a distribution (10.3), and a run of steps (section 3), the one place where +it judges a pair itself. It does so with the rule ported from `rule.py`, held to it by the +shared acceptance cases and by 813 pairs on every edge of the rule that the Python writes +(`tests/test_boundaries.py`), after holding each step of the run to the file's own classes. +`tests/fixtures/together.json` holds the counts `together.py` gives for every run of the +local example, and the page is held to them. A file read by `?file=` or chosen by a reader has not passed through the validator: the page checks its shape only, says so above everything it draws, and refuses a file it cannot draw instead of drawing part of it. A matrix is held to the file it says it is of (title, states, indicators, steps, number of @@ -489,16 +550,19 @@ items, the length of every row) before a row of it is drawn; its codes are check The text a page renders comes from the file or from words chosen to be neutral: it avoids stage, triage, ledger, layer, upstream, provider and candidate (a test holds this). -**The page is a matrix page.** From the top: the header (10.1); a row of chips that -chooses the comparison, the whole way first and then each step with its number, the values -that moved in it, a `!` where nothing explains it and a mark for each verdict worth a look -(the arrow keys move between them); under the chips one line for the chosen step: the -states it goes between, what it is, its claim with who made it, what the claim found, its -tolerance and its noise; one row of controls (name, group, class, verdict, the size +**The page is a matrix page.** From the top: the header (10.1); the steps (10.1a), which +choose what is compared: the whole way, one step, or a run of steps; under them one line +for what is chosen: for a step, the states it goes between, what it is, its claim with who +made it, what the claim found, its tolerance and its noise; for a run, the two states it +lies between, each step's claim and what moved in it alone (the quiet ones folded into one +line), its tolerance and its noise; one row of controls (name, group, class, verdict, the size filter) with the count and the legend in the row under it; then the matrix (10.5), which opens an item in a drawer (10.6); directly under it, always open, the distributions of the -chosen comparison (10.3). The address keeps the chosen step and an open item -(`#step=&item=`), so that a reviewer can send someone to exactly that view. Below +chosen comparison (10.3). The address keeps what is chosen and an open item, so that a +reviewer can send someone to exactly that view: nothing for the whole way, `#step=` for +one step, `#steps=:` for a run (each id URL-encoded), and `&item=`. An +address naming a step the file does not have, steps apart, or a run the page cannot judge +(no matrix), says so and shows the whole way. Below them, each closed until a reader opens it: the steps and the whole way (10.2), the worst values (10.4), the explanations and the rule (10.7). A host's page never offers to open a file; it says only why nothing is drawn. @@ -514,6 +578,20 @@ it holds of how many worth a look, and, for a local file, that its writer has no may be published (a quiet part of the line, as a site's own notice, not a box). The states in order, with their identity and join counts, head the steps (10.2). +### 10.1a Choosing steps +**A strip**: "The whole way", then one thin cell per step whose height is how many values +moved in it alone (on a log scale, so one value still shows), red at the top where nothing +explains it, a dot where a verdict is worth a look; numbered up to 40 steps, 2 px wide past +60, and scrolling inside its frame. A click chooses one step; Shift (or ⌘ or Ctrl) chooses +every step from the one chosen before, a run. The arrow keys move a step at a time; with +Shift they grow or shrink the run. Chosen steps are filled and underlined. + +**A list**, opened by "Find and choose steps": every step with its number, label, kind and +what moved in it; a click chooses it, Shift a run; consecutive steps of one kind under one +heading that chooses their run; a search by number, name or kind; "Only steps where +something moved"; "Every step: the whole way". Without every item the list says why a run +cannot be judged yet, and one step and the whole way remain. + ### 10.2 Steps and the whole way Steps are numbered from 0 where the first state is `published`: step 0 is then the reproduction (rule 1 of section 1). A chain that starts from a build of its own has no @@ -525,8 +603,8 @@ every class, up, down, the step that carries most of the difference (section 7), over steps, not attributable, and moved by what nothing explains. ### 10.3 Distributions -One card per indicator, in the order of the matrix's columns (10.5), for the comparison the -chips have chosen: a histogram of the file's own counts, from left to right a sign change, +One card per indicator, in the order of the matrix's columns (10.5), for the comparison +chosen above (one step, a run of steps or the whole way): a histogram of the file's own counts, from left to right a sign change, the bands down from far to near, agrees and within noise (two bars, the band around 1), the bands up from near to far, then zero reference, zero result, near zero and missing. Every count is in its bar's title. @@ -561,7 +639,7 @@ ratio). The page always says how many it shows of how many, and how many items w look the file does not hold. ### 10.5 Matrix -Items by indicator for the comparison the chips have chosen. **The indicators in which that +Items by indicator for the comparison chosen above. **The indicators in which that comparison moved most come first**, so that what a step did is in view without scrolling; an indicator in which it moved nothing and nothing is missing comes last, quietened, and keeps its column (a missing value counts with what moved: it is a finding, not stillness). @@ -641,6 +719,9 @@ the page script (`tests/js/page.test.js`), so the two cannot drift apart unnotic | A25 | A file is local unless its writer says otherwise | sentier-bafu, sentier-bafu-agribalyse | `test_a25` | | A26 | A pair on both sides of the floor that agrees was a zero result, and listed second worst | review of PR 2 | cases A26-* | | A27 | A factor too large for a float: Python and the page ordered the pair differently, the page wrote "×Infinity" | review of PR 2 | case A27 | +| A28 | A move within the noise of its step was counted as "differs" over the whole way, which judged without noise | sentier-agribalyse | `StateNoise` (a move within the noise of an end), node "the whole way and a run take the noise of their two ends" | +| A29 | A run of steps judged by the page has the classes and counts of the Python reference, item by item, is refused where its steps disagree with the file, and steps apart are refused; the page's rule agrees with the Python's on every edge, to the last bit | review of PR 22 | `Together`, `Boundaries`, `tests/fixtures/together.json`, `tests/fixtures/boundaries.json`, node "a run judged by the page", "every edge of the rule", "refuses a run", "steps apart are refused" | +| A30 | The noise of a state inside a run or the whole way, taken as the largest of any step, put 1,504 of 2,451 real differences "within noise"; a comparison now takes the sum of the noise of its two ends, never an inner state's | sentier-agribalyse | `StateNoise`, `Together` (sum of two measured ends), node "two states are compared with the sum of their noise", "a run is judged with the sum of the noise of its two ends" | Also tested: the validator refuses a class, direction or verdict that contradicts the values, items out of order, counts that do not add up, unknown keys and other formats diff --git a/examples/diagnostics.json b/examples/diagnostics.json index ebc727d..76ee145 100644 --- a/examples/diagnostics.json +++ b/examples/diagnostics.json @@ -33,6 +33,7 @@ "floors_from": "published", "beyond": 10.0, "total_tolerance": 0.0005, + "total_noise": {}, "floors": [ { "indicator": "x:climate", @@ -123,7 +124,13 @@ "identity": { "commit": "0000001" }, - "join": null + "join": null, + "noise": { + "measured": false, + "between": null, + "by": null, + "note": null + } }, { "id": "fix-1", @@ -133,7 +140,20 @@ "identity": { "commit": "0000002" }, - "join": null + "join": null, + "noise": { + "measured": true, + "between": [ + "fix-1", + "rebuilt" + ], + "by": { + "x:climate": 0.0, + "x:water": 0.02439024390243894, + "x:tox": 0.0 + }, + "note": "Two builds of commit 0000002, on two machines." + } }, { "id": "rebuilt", @@ -144,7 +164,20 @@ "commit": "0000002", "machine": "second" }, - "join": null + "join": null, + "noise": { + "measured": true, + "between": [ + "fix-1", + "rebuilt" + ], + "by": { + "x:climate": 0.0, + "x:water": 0.02439024390243894, + "x:tox": 0.0 + }, + "note": "Two builds of commit 0000002, on two machines." + } }, { "id": "mapping-2", @@ -154,7 +187,13 @@ "identity": { "commit": "0000003" }, - "join": null + "join": null, + "noise": { + "measured": false, + "between": null, + "by": null, + "note": null + } } ], "steps": [ @@ -229,9 +268,13 @@ "note": "one order of magnitude above the solver noise of two builds" }, "noise": { - "measured": false, + "measured": true, "between": null, - "by": null, + "by": { + "x:climate": 0.0, + "x:water": 0.02439024390243894, + "x:tox": 0.0 + }, "note": null } }, @@ -251,9 +294,13 @@ "note": "one order of magnitude above the solver noise of two builds" }, "noise": { - "measured": false, + "measured": true, "between": null, - "by": null, + "by": { + "x:climate": 0.0, + "x:water": 0.04878048780487788, + "x:tox": 0.0 + }, "note": null } }, @@ -310,16 +357,13 @@ }, "noise": { "measured": true, - "between": [ - "fix-1", - "rebuilt" - ], + "between": null, "by": { "x:climate": 0.0, "x:water": 0.02439024390243894, "x:tox": 0.0 }, - "note": "One pair of builds of one state." + "note": null } } ], @@ -648,8 +692,8 @@ "pairs": 10, "classes": { "agrees": 8, - "within noise": 0, - "differs": 1, + "within noise": 1, + "differs": 0, "beyond 10x": 0, "zero reference": 0, "zero result": 0, @@ -663,7 +707,7 @@ "both": 0 }, "directions": { - "up": 1, + "up": 0, "down": 0 }, "bands": { @@ -672,7 +716,7 @@ "down, factor 1.25-2": 0, "down, factor 1.1-1.25": 0, "down, factor 1-1.1": 0, - "up, factor 1-1.1": 1, + "up, factor 1-1.1": 0, "up, factor 1.1-1.25": 0, "up, factor 1.25-2": 0, "up, factor 2-10": 0, @@ -680,7 +724,7 @@ }, "verdicts": null, "closer": 0, - "further": 1 + "further": 0 }, "x:tox": { "pairs": 10, @@ -995,12 +1039,12 @@ "dominant": { "step-0": 0, "fix-1": 0, - "rebuilt": 1, + "rebuilt": 0, "mapping-2": 1 }, "not_attributable": 1, - "spread": 0, - "unexplained_moved": 1 + "spread": 1, + "unexplained_moved": 0 }, "x:tox": { "dominant": { @@ -1022,7 +1066,7 @@ {"id": "p05", "name": "Rice", "place": "IT", "group": "crops", "unit": "kg", "values": {"x:climate": [1.9, 1.9, 0.95, 0.95, 0.95], "x:water": [3.4, 3.4, 3.4, 3.4, 3.4], "x:tox": [1e-12, 1e-12, 1e-12, 1e-12, 3e-12]}, "classes": {"x:climate": ["agrees", "differs", "agrees", "agrees", "differs"], "x:water": ["agrees", "agrees", "agrees", "agrees", "agrees"], "x:tox": ["near zero", "near zero", "near zero", "near zero", "near zero"]}, "directions": {"x:climate": [null, "down", null, null, "down"], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "bands": {"x:climate": [null, "down, factor 1.25-2", null, null, "down, factor 1.25-2"], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "verdicts": {"x:climate": ["unchanged", "off claim", null, "unchanged"], "x:water": ["unchanged", "unchanged", null, "unchanged"], "x:tox": ["not judged", "not judged", null, "not judged"]}, "breakdowns": null}, {"id": "p03", "name": "Barley", "place": "DE", "group": "crops", "unit": "kg", "values": {"x:climate": [0.48, 0.47999, 0.575988, 0.575988, 0.575988], "x:water": [1.1, 1.1, 1.1, null, 1.3], "x:tox": [1.9e-07, 1.9e-07, 1.9e-07, 1.9e-07, 1.9e-07]}, "classes": {"x:climate": ["agrees", "differs", "agrees", "agrees", "differs"], "x:water": ["agrees", "agrees", "missing", "missing", "differs"], "x:tox": ["agrees", "agrees", "agrees", "agrees", "agrees"]}, "directions": {"x:climate": [null, "up", null, null, "up"], "x:water": [null, null, null, null, "up"], "x:tox": [null, null, null, null, null]}, "bands": {"x:climate": [null, "up, factor 1.1-1.25", null, null, "up, factor 1.1-1.25"], "x:water": [null, null, null, null, "up, factor 1.1-1.25"], "x:tox": [null, null, null, null, null]}, "verdicts": {"x:climate": ["unchanged", "not claimed", null, "unchanged"], "x:water": ["unchanged", "unchanged", null, "not judged"], "x:tox": ["unchanged", "unchanged", null, "unchanged"]}, "breakdowns": null}, {"id": "p01", "name": "Wheat", "place": "FR", "group": "crops", "unit": "kg", "values": {"x:climate": [0.52, 0.520013, 0.468012, 0.468012, 0.468012], "x:water": [1.2, 1.2, 1.2, 1.2000001, 1.2000001], "x:tox": [2e-07, 2.0000002e-07, 2.0000002e-07, 2.0000002e-07, 2.0000002e-07]}, "classes": {"x:climate": ["agrees", "differs", "agrees", "agrees", "differs"], "x:water": ["agrees", "agrees", "agrees", "agrees", "agrees"], "x:tox": ["agrees", "agrees", "agrees", "agrees", "agrees"]}, "directions": {"x:climate": [null, "down", null, null, "down"], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "bands": {"x:climate": [null, "down, factor 1.1-1.25", null, null, "down, factor 1.1-1.25"], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "verdicts": {"x:climate": ["unchanged", "as claimed", null, "unchanged"], "x:water": ["unchanged", "unchanged", null, "unchanged"], "x:tox": ["unchanged", "unchanged", null, "unchanged"]}, "breakdowns": {"fix-1": "p01-fix-1.json"}}, - {"id": "p04", "name": "Oats", "place": "DE", "group": "crops", "unit": "kg", "values": {"x:climate": [0.45, 0.45, 0.45, 0.45, 0.45], "x:water": [1.0, 1.0, 1.0, 1.025, 1.0455], "x:tox": [1.7e-07, 1.7e-07, 1.7e-07, 1.7e-07, 1.7e-07]}, "classes": {"x:climate": ["agrees", "agrees", "agrees", "agrees", "agrees"], "x:water": ["agrees", "agrees", "differs", "within noise", "differs"], "x:tox": ["agrees", "agrees", "agrees", "agrees", "agrees"]}, "directions": {"x:climate": [null, null, null, null, null], "x:water": [null, null, "up", null, "up"], "x:tox": [null, null, null, null, null]}, "bands": {"x:climate": [null, null, null, null, null], "x:water": [null, null, "up, factor 1-1.1", null, "up, factor 1-1.1"], "x:tox": [null, null, null, null, null]}, "verdicts": {"x:climate": ["unchanged", "unchanged", null, "unchanged"], "x:water": ["unchanged", "unchanged", null, "within noise"], "x:tox": ["unchanged", "unchanged", null, "unchanged"]}, "breakdowns": null}, + {"id": "p04", "name": "Oats", "place": "DE", "group": "crops", "unit": "kg", "values": {"x:climate": [0.45, 0.45, 0.45, 0.45, 0.45], "x:water": [1.0, 1.0, 1.0, 1.025, 1.0455], "x:tox": [1.7e-07, 1.7e-07, 1.7e-07, 1.7e-07, 1.7e-07]}, "classes": {"x:climate": ["agrees", "agrees", "agrees", "agrees", "agrees"], "x:water": ["agrees", "agrees", "within noise", "within noise", "differs"], "x:tox": ["agrees", "agrees", "agrees", "agrees", "agrees"]}, "directions": {"x:climate": [null, null, null, null, null], "x:water": [null, null, null, null, "up"], "x:tox": [null, null, null, null, null]}, "bands": {"x:climate": [null, null, null, null, null], "x:water": [null, null, null, null, "up, factor 1-1.1"], "x:tox": [null, null, null, null, null]}, "verdicts": {"x:climate": ["unchanged", "unchanged", null, "unchanged"], "x:water": ["unchanged", "unchanged", null, "within noise"], "x:tox": ["unchanged", "unchanged", null, "unchanged"]}, "breakdowns": null}, {"id": "p06", "name": "Rye", "place": "PL", "group": "crops", "unit": "kg", "values": {"x:climate": [0.43, 0.43, 0.43, 0.43, null], "x:water": [null, 1.15, 1.15, 1.15, 1.15], "x:tox": [2.1e-07, 2.1e-07, 2.1e-07, 2.1e-07, 2.1e-07]}, "classes": {"x:climate": ["agrees", "agrees", "agrees", "missing", "missing"], "x:water": ["missing", "agrees", "agrees", "agrees", "missing"], "x:tox": ["agrees", "agrees", "agrees", "agrees", "agrees"]}, "directions": {"x:climate": [null, null, null, null, null], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "bands": {"x:climate": [null, null, null, null, null], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "verdicts": {"x:climate": ["unchanged", "unchanged", null, "not judged"], "x:water": ["not judged", "unchanged", null, "unchanged"], "x:tox": ["unchanged", "unchanged", null, "unchanged"]}, "breakdowns": null}, {"id": "p02", "name": "Maize", "place": "FR", "group": "crops", "unit": "kg", "values": {"x:climate": [0.41, 0.41, 0.41, 0.41, 0.41], "x:water": [2.5, 2.500025, 2.500025, 2.500025, 2.500025], "x:tox": [1.8e-07, 1.8e-07, 1.8e-07, 1.8e-07, 1.8e-07]}, "classes": {"x:climate": ["agrees", "agrees", "agrees", "agrees", "agrees"], "x:water": ["agrees", "agrees", "agrees", "agrees", "agrees"], "x:tox": ["agrees", "agrees", "agrees", "agrees", "agrees"]}, "directions": {"x:climate": [null, null, null, null, null], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "bands": {"x:climate": [null, null, null, null, null], "x:water": [null, null, null, null, null], "x:tox": [null, null, null, null, null]}, "verdicts": {"x:climate": ["unchanged", "unchanged", null, "unchanged"], "x:water": ["unchanged", "unchanged", null, "claimed, not moved"], "x:tox": ["unchanged", "unchanged", null, "unchanged"]}, "breakdowns": null} ], diff --git a/examples/local/diagnostics.json b/examples/local/diagnostics.json index def56d0..ffeea12 100644 --- a/examples/local/diagnostics.json +++ b/examples/local/diagnostics.json @@ -33,6 +33,7 @@ "floors_from": "published", "beyond": 10.0, "total_tolerance": 0.0005, + "total_noise": {}, "floors": [ { "indicator": "x:climate", @@ -123,7 +124,13 @@ "identity": { "commit": "0000001" }, - "join": null + "join": null, + "noise": { + "measured": false, + "between": null, + "by": null, + "note": null + } }, { "id": "fix-1", @@ -133,7 +140,20 @@ "identity": { "commit": "0000002" }, - "join": null + "join": null, + "noise": { + "measured": true, + "between": [ + "fix-1", + "rebuilt" + ], + "by": { + "x:climate": 0.0, + "x:water": 0.02439024390243894, + "x:tox": 0.0 + }, + "note": "Two builds of commit 0000002, on two machines." + } }, { "id": "rebuilt", @@ -144,7 +164,20 @@ "commit": "0000002", "machine": "second" }, - "join": null + "join": null, + "noise": { + "measured": true, + "between": [ + "fix-1", + "rebuilt" + ], + "by": { + "x:climate": 0.0, + "x:water": 0.02439024390243894, + "x:tox": 0.0 + }, + "note": "Two builds of commit 0000002, on two machines." + } }, { "id": "mapping-2", @@ -154,7 +187,13 @@ "identity": { "commit": "0000003" }, - "join": null + "join": null, + "noise": { + "measured": false, + "between": null, + "by": null, + "note": null + } } ], "steps": [ @@ -229,9 +268,13 @@ "note": "one order of magnitude above the solver noise of two builds" }, "noise": { - "measured": false, + "measured": true, "between": null, - "by": null, + "by": { + "x:climate": 0.0, + "x:water": 0.02439024390243894, + "x:tox": 0.0 + }, "note": null } }, @@ -251,9 +294,13 @@ "note": "one order of magnitude above the solver noise of two builds" }, "noise": { - "measured": false, + "measured": true, "between": null, - "by": null, + "by": { + "x:climate": 0.0, + "x:water": 0.04878048780487788, + "x:tox": 0.0 + }, "note": null } }, @@ -310,16 +357,13 @@ }, "noise": { "measured": true, - "between": [ - "fix-1", - "rebuilt" - ], + "between": null, "by": { "x:climate": 0.0, "x:water": 0.02439024390243894, "x:tox": 0.0 }, - "note": "One pair of builds of one state." + "note": null } } ], @@ -648,8 +692,8 @@ "pairs": 10, "classes": { "agrees": 8, - "within noise": 0, - "differs": 1, + "within noise": 1, + "differs": 0, "beyond 10x": 0, "zero reference": 0, "zero result": 0, @@ -663,7 +707,7 @@ "both": 0 }, "directions": { - "up": 1, + "up": 0, "down": 0 }, "bands": { @@ -672,7 +716,7 @@ "down, factor 1.25-2": 0, "down, factor 1.1-1.25": 0, "down, factor 1-1.1": 0, - "up, factor 1-1.1": 1, + "up, factor 1-1.1": 0, "up, factor 1.1-1.25": 0, "up, factor 1.25-2": 0, "up, factor 2-10": 0, @@ -680,7 +724,7 @@ }, "verdicts": null, "closer": 0, - "further": 1 + "further": 0 }, "x:tox": { "pairs": 10, @@ -995,12 +1039,12 @@ "dominant": { "step-0": 0, "fix-1": 0, - "rebuilt": 1, + "rebuilt": 0, "mapping-2": 1 }, "not_attributable": 1, - "spread": 0, - "unexplained_moved": 1 + "spread": 1, + "unexplained_moved": 0 }, "x:tox": { "dominant": { diff --git a/examples/local/matrix.json b/examples/local/matrix.json index 7945f66..e2710c6 100644 --- a/examples/local/matrix.json +++ b/examples/local/matrix.json @@ -12,16 +12,16 @@ ["p02","Maize","FR","crops","kg"] ], "values": [ - [-0.5,-0.5,-0.5,-0.5,0.2,0.05,0.05,0.05,0.05,0.05,-2e-06,-2e-06,-2e-06,-2e-06,-4e-06], - [42.0,42.00042,42.00042,42.00042,42.00042,0.9,0.9,0.9,0.9,0.9,0.0,0.0,0.0,0.0,4e-07], - [0.03,0.03,0.03,0.03,0.0,0.0004,0.0004,0.0004,0.0004,0.0004,3e-09,3e-09,3e-09,3e-09,3e-09], - [1.3,1.3,1.3,1.3,1.3,0.02,0.02,0.02,0.02,0.5,2.1e-07,2.3e-07,2.3e-07,2.3e-07,2.3e-07], - [1.9,1.9,0.95,0.95,0.95,3.4,3.4,3.4,3.4,3.4,1e-12,1e-12,1e-12,1e-12,3e-12], - [0.48,0.47999,0.575988,0.575988,0.575988,1.1,1.1,1.1,null,1.3,1.9e-07,1.9e-07,1.9e-07,1.9e-07,1.9e-07], - [0.52,0.520013,0.468012,0.468012,0.468012,1.2,1.2,1.2,1.2000001,1.2000001,2e-07,2.0000002e-07,2.0000002e-07,2.0000002e-07,2.0000002e-07], - [0.45,0.45,0.45,0.45,0.45,1.0,1.0,1.0,1.025,1.0455,1.7e-07,1.7e-07,1.7e-07,1.7e-07,1.7e-07], - [0.43,0.43,0.43,0.43,null,null,1.15,1.15,1.15,1.15,2.1e-07,2.1e-07,2.1e-07,2.1e-07,2.1e-07], - [0.41,0.41,0.41,0.41,0.41,2.5,2.500025,2.500025,2.500025,2.500025,1.8e-07,1.8e-07,1.8e-07,1.8e-07,1.8e-07] + [-0.5,"=","=","=",0.2,0.05,"=","=","=","=",-2e-06,"=","=","=",-4e-06], + [42.0,42.00042,"=","=","=",0.9,"=","=","=","=",0.0,"=","=","=",4e-07], + [0.03,"=","=","=",0.0,0.0004,"=","=","=","=",3e-09,"=","=","=","="], + [1.3,"=","=","=","=",0.02,"=","=","=",0.5,2.1e-07,2.3e-07,"=","=","="], + [1.9,"=",0.95,"=","=",3.4,"=","=","=","=",1e-12,"=","=","=",3e-12], + [0.48,0.47999,0.575988,"=","=",1.1,"=","=",null,1.3,1.9e-07,"=","=","=","="], + [0.52,0.520013,0.468012,"=","=",1.2,"=","=",1.2000001,"=",2e-07,2.0000002e-07,"=","=","="], + [0.45,"=","=","=","=",1.0,"=","=",1.025,1.0455,1.7e-07,"=","=","=","="], + [0.43,"=","=","=",null,null,1.15,"=","=","=",2.1e-07,"=","=","=","="], + [0.41,"=","=","=","=",2.5,2.500025,"=","=","=",1.8e-07,"=","=","=","="] ], "classes": [ "===SS========dd", @@ -31,7 +31,7 @@ "=d==d=====00000", "=D==D==--D=====", "=d==d==========", - "=======D~D=====", + "=======~~D=====", "===---===-=====", "===============" ], diff --git a/src/sentier_diagnostics/chain.py b/src/sentier_diagnostics/chain.py index 905b12a..99e35cc 100644 --- a/src/sentier_diagnostics/chain.py +++ b/src/sentier_diagnostics/chain.py @@ -44,8 +44,10 @@ def tolerance(value=None, basis=None, note=None) -> dict: def noise(between=None, by=None, note=None) -> dict: """What two builds of one state differ by, per indicator; not measured without `between`. - An indicator whose noise is None (`noise_of` found no pair to take it from) is left out: - its noise was not measured, which is not the same as two identical builds.""" + It belongs to the state that was built twice (`state(..., noise=...)`), never to a step: a + step and a run are compared with the noise of the two states they lie between (SPEC + section 5.3). An indicator whose noise is None (`noise_of` found no pair to take it from) is + left out: its noise was not measured, which is not the same as two identical builds.""" if not between: return {"measured": False, "between": None, "by": None, "note": note} found = {k: v for k, v in (by or {}).items() if v is not None} @@ -88,7 +90,11 @@ def beyond(item, value): def step(id: str, label: str, *, kind: str, explained: bool = True, about: str | None = None, link: str | None = None, claim: dict | None = None, tolerance: dict | None = None, noise: dict | None = None) -> dict: # fmt: skip - """One step between two states, as `build` takes it.""" + """One step between two states, as `build` takes it. Its noise is that of its two states, + worked out by `build`; a noise given here is refused, since it cannot say which state was + built twice.""" + if noise is not None: + raise ValueError(NOISE_ON_A_STEP.format(id=id)) return { "id": id, "label": label, @@ -98,14 +104,26 @@ def step(id: str, label: str, *, kind: str, explained: bool = True, about: str | "link": link, "claim": claim, "tolerance": tolerance, - "noise": noise, } +#: Why a step takes no noise of its own. +NOISE_ON_A_STEP = ( + "step {id!r}: noise belongs to the state that was built twice: give it to `chain.state`; " + "a step is compared with the noise of its two states (SPEC section 5.3)" +) + + def state(id: str, label: str, *, kind: str = "build", about: str | None = None, - identity: Mapping[str, str | None] | None = None, join: Mapping | None = None) -> dict: # fmt: skip + identity: Mapping[str, str | None] | None = None, join: Mapping | None = None, + noise: dict | None = None) -> dict: # fmt: skip """One state: a set of results. `kind` is "published" for the results its publisher released. + `noise` is what two builds of this state differ by (`chain.noise`); without it the + state's noise was not measured. A step, a run of steps and the whole way are compared with + the sum of the noise of the two states they lie between, never with that of a state inside + them. The published results have no builds: they are exact and take no noise. + `identity` is a short table of texts (commit, release, machine); an entry whose value is None is left out, so that a writer can pass what it may not know. @@ -115,11 +133,14 @@ def state(id: str, label: str, *, kind: str = "build", about: str | None = None, >>> state("b", "Build", identity={"commit": "0e1f5be", "machine": None})["identity"] {'commit': '0e1f5be'} """ + if kind == "published" and noise is not None: + raise ValueError(f"state {id!r}: the published results have no builds and take no noise: they are exact") if join is not None: join = {"compared": join["compared"], "unmatched": join.get("unmatched", 0), "ambiguous": join.get("ambiguous", 0), "note": join.get("note")} # fmt: skip return {"id": id, "label": label, "about": about, "kind": kind, - "identity": {k: v for k, v in (identity or {}).items() if v is not None}, "join": join} # fmt: skip + "identity": {k: v for k, v in (identity or {}).items() if v is not None}, "join": join, + "noise": noise} # fmt: skip def explanation(text: str, by: dict, *, step: str | None = None, indicator: str | None = None, @@ -272,9 +293,23 @@ def _build( floor_rows.append({"indicator": indicator, "unit": unit, "value": value, "basis": basis, "n": n}) # fmt: skip + # the states, normalised: each with its noise, measured or not + out_states = [] + for s in states: + out = {"id": s["id"], "label": s["label"], "about": s.get("about"), + "kind": s.get("kind", "build"), "identity": dict(s.get("identity") or {}), "join": s.get("join")} # fmt: skip + if out["kind"] == "published": + if s.get("noise") is not None: + raise ValueError(f"state {s['id']!r}: the published results have no builds and take no noise: they are exact") + else: + out["noise"] = s.get("noise") or noise() + out_states.append(out) + # the steps, normalised out_steps = [] for k, one in enumerate(steps): + if one.get("noise") is not None: + raise ValueError(NOISE_ON_A_STEP.format(id=one["id"])) found_claim = one.get("claim") out_steps.append({ "id": one["id"], @@ -287,9 +322,10 @@ def _build( "link": one.get("link"), "claim": claims.published(found_claim, tier == "items" and claim_lists) if found_claim else None, "tolerance": one.get("tolerance") or tolerance(), - "noise": one.get("noise") or noise(), + "noise": fileformat.noise_of_states(out_states[k], out_states[k + 1]), }) # fmt: skip total_tolerance = max(s["tolerance"]["value"] for s in out_steps) + total_noise = fileformat.noise_of_states(out_states[0], out_states[-1])["by"] or {} published_first = states[0].get("kind") == "published" def judged(k, indicator, item): @@ -297,7 +333,7 @@ def judged(k, indicator, item): first, last = (order[0], order[-1]) if k is None else (order[k], order[k + 1]) before, after = table[first][indicator].get(item), table[last][indicator].get(item) if k is None: - level, tol = None, total_tolerance + level, tol = total_noise.get(indicator), total_tolerance else: said = out_steps[k]["noise"] level = (said["by"] or {}).get(indicator) if said["measured"] else None @@ -438,14 +474,10 @@ def record_of(item): "floors_from": order[0], "beyond": rule.BEYOND_FACTOR, "total_tolerance": total_tolerance, + "total_noise": total_noise, "floors": floor_rows, }, - "states": [ - {"id": s["id"], "label": s["label"], "about": s.get("about"), - "kind": s.get("kind", "build"), "identity": dict(s.get("identity") or {}), - "join": s.get("join")} - for s in states - ], # fmt: skip + "states": out_states, "steps": out_steps, "counts": counts, "attribution": attribution, diff --git a/src/sentier_diagnostics/example.py b/src/sentier_diagnostics/example.py index e6719bf..219b5b2 100644 --- a/src/sentier_diagnostics/example.py +++ b/src/sentier_diagnostics/example.py @@ -11,7 +11,7 @@ What it must show, at least once each: agrees most values in every step - within noise Oats, water use, in mapping-2: less than the rebuild differs + within noise Oats, water use, in rebuilt and mapping-2: less than two builds differ differs Wheat, climate change, in fix-1: down 10 % beyond 10x Tomato, water use, in mapping-2: 25 times as much zero reference Machine hour, toxicity, in mapping-2: from 0 to a result @@ -103,15 +103,21 @@ def _with(base, **changes): } -def steps() -> list[dict]: - # noise beyond the floors the chain counts with: those of its first state +def rebuild_noise() -> dict: + """The noise of commit 0000002, from its two builds (With fix 1 and Rebuilt): it belongs to + both states, each a build of that commit. Taken beyond the floors the chain counts with, + those of its first state.""" units = {item: about[1] for item, about in ITEMS.items()} floors = rule.floors_of(PUBLISHED, units) - noise = { + by = { key: chain.noise_of(FIX_1[key], REBUILT[key], floor={item: floors[(key, unit)][0] for item, unit in units.items()}) # fmt: skip for key in FIX_1 } + return chain.noise(("fix-1", "rebuilt"), by, "Two builds of commit 0000002, on two machines.") + + +def steps() -> list[dict]: return [ chain.step( "step-0", @@ -157,7 +163,6 @@ def steps() -> list[dict]: ], by=claims.origin("code", "reach of the changed links"), ), - noise=chain.noise(("fix-1", "rebuilt"), noise, "One pair of builds of one state."), ), ] @@ -222,8 +227,9 @@ def _made(builder, **more): join={"compared": 10, "unmatched": 1, "ambiguous": 0, "note": "One published row names no item of the build."}), chain.state("repro", "Recomputed", identity={"commit": "0000001"}), - chain.state("fix-1", "With fix 1", identity={"commit": "0000002"}), - chain.state("rebuilt", "Rebuilt", identity={"commit": "0000002", "machine": "second"}), + chain.state("fix-1", "With fix 1", identity={"commit": "0000002"}, noise=rebuild_noise()), + chain.state("rebuilt", "Rebuilt", identity={"commit": "0000002", "machine": "second"}, + noise=rebuild_noise()), # fmt: skip chain.state("mapping-2", "With mapping 2", identity={"commit": "0000003"}), ], # fmt: skip steps=steps(), diff --git a/src/sentier_diagnostics/fileformat.py b/src/sentier_diagnostics/fileformat.py index e00b03d..bce10fa 100644 --- a/src/sentier_diagnostics/fileformat.py +++ b/src/sentier_diagnostics/fileformat.py @@ -57,8 +57,15 @@ } # fmt: skip INDICATOR_KEYS = {"id", "name", "short", "unit"} RULE_KEYS = {"floor_factor", "floors_from", "beyond", "total_tolerance", "floors"} +#: Left out only by a file made before noise belonged to a state: such a file judges the +#: whole way without noise. +RULE_OPTIONAL = {"total_noise"} FLOOR_KEYS = {"indicator", "unit", "value", "basis", "n"} STATE_KEYS = {"id", "label", "about", "kind", "identity", "join"} +#: Carried by every built state of a file made since noise belongs to a state (SPEC section +#: 5.3); never by the published results, which have no builds and are exact. +STATE_OPTIONAL = {"noise"} +NOISE_KEYS = {"measured", "between", "by", "note"} STEP_KEYS = {"id", "from", "to", "label", "about", "kind", "explained", "link", "claim", "tolerance", "noise"} # fmt: skip CLAIM_KEYS = {"text", "by", "reach", "scope"} @@ -219,15 +226,19 @@ def _indicators(data, found) -> list[str]: return ids -def _states(data, found) -> list[str]: +def _states(data, found, indicators) -> list[str]: states = data.get("states") if not isinstance(states, list) or not 2 <= len(states) <= MAX_STATES: found.append(f"`states` is not a list of 2 to {MAX_STATES}") return [] ids = [] for k, one in enumerate(states): - if not isinstance(one, dict) or set(one) != STATE_KEYS or not is_text(one["id"], MAX_ID): - found.append(f"a state is not {{{', '.join(sorted(STATE_KEYS))}}}") + if ( + not isinstance(one, dict) + or not STATE_KEYS <= set(one) <= STATE_KEYS | STATE_OPTIONAL + or not is_text(one["id"], MAX_ID) + ): + found.append(f"a state is not {{{', '.join(sorted(STATE_KEYS))}}}, with `noise`") return [] where = f"state {one['id']!r}" if not is_text(one["label"], MAX_LABEL) or not is_optional(one["about"]): @@ -236,6 +247,14 @@ def _states(data, found) -> list[str]: found.append(f"{where}: `kind` is not one of {', '.join(STATE_KINDS)}") elif one["kind"] == "published" and k != 0: found.append(f"{where}: only the first state can be the published results") + if "noise" in one: + if one["kind"] == "published": + found.append(f"{where}: the published results have no builds and carry no noise: they are exact") + return [] + shape = noise_errors(one["noise"], where, indicators) + if shape: + found += shape + return [] identity = one["identity"] if not ( isinstance(identity, dict) @@ -257,6 +276,9 @@ def _states(data, found) -> list[str]: if len(set(ids)) != len(ids): found.append("two states have the same id") return [] + if len({"noise" in one for one in states if one["kind"] != "published"}) > 1: + found.append("some built states carry their noise and some do not: all do, or none (a file made before noise belonged to a state)") + return [] return ids @@ -349,28 +371,16 @@ def _steps(data, found, states, indicators, tier) -> list[dict]: ): found.append(f"{where}: `tolerance` is not {{value, basis, note}}") return [] - noise = one["noise"] - if not ( - isinstance(noise, dict) - and set(noise) == {"measured", "between", "by", "note"} - and isinstance(noise["measured"], bool) - and is_optional(noise["note"]) - ): - found.append(f"{where}: `noise` is not {{measured, between, by, note}}") - return [] - if noise["measured"]: - if not ( - isinstance(noise["between"], list) - and len(noise["between"]) == 2 - and all(is_text(n, MAX_LABEL) for n in noise["between"]) - and isinstance(noise["by"], dict) - and set(noise["by"]) <= set(indicators) - and all(is_share(v) for v in noise["by"].values()) - ): - found.append(f"{where}: measured noise names the two builds and a size per indicator") + if states_carry_noise(data): + # the step's noise is its two states', the larger per indicator (SPEC section 5.3) + if one["noise"] != noise_of_states(data["states"][k], data["states"][k + 1]): + found.append(f"{where}: `noise` is not the sum of the noise of its two states, per indicator") + return [] + else: + shape = noise_errors(one["noise"], where, indicators) + if shape: + found += shape return [] - elif noise["between"] is not None or noise["by"] is not None: - found.append(f"{where}: noise that was not measured has no builds and no sizes") ids.append(one["id"]) if len(set(ids)) != len(ids): found.append("two steps have the same id") @@ -378,10 +388,106 @@ def _steps(data, found, states, indicators, tier) -> list[dict]: return steps +def noise_errors(noise, where: str, indicators) -> list[str]: + """What is wrong with a measured noise: of a state, or of a step in a file made before noise + belonged to a state. Measured noise names its two builds and a size per indicator; noise + that was not measured has neither.""" + if not ( + isinstance(noise, dict) + and set(noise) == NOISE_KEYS + and isinstance(noise["measured"], bool) + and is_optional(noise["note"]) + ): + return [f"{where}: `noise` is not {{measured, between, by, note}}"] + if noise["measured"]: + if not ( + isinstance(noise["between"], list) + and len(noise["between"]) == 2 + and all(is_text(n, MAX_LABEL) for n in noise["between"]) + and isinstance(noise["by"], dict) + and set(noise["by"]) <= set(indicators) + and all(is_share(v) for v in noise["by"].values()) + ): + return [f"{where}: measured noise names the two builds and a size per indicator"] + elif noise["between"] is not None or noise["by"] is not None: + return [f"{where}: noise that was not measured has no builds and no sizes"] + return [] + + +def states_carry_noise(data: Mapping) -> bool: + """Whether the file says the noise of each built state (SPEC section 5.3). A file made before + noise belonged to a state says a noise per step only, and never which state it was measured + on. The published results carry none in either: they are exact. + + >>> built = {"kind": "build", "noise": {}} + >>> states_carry_noise({"states": [{"kind": "published"}, built]}), states_carry_noise({"states": [{"kind": "build"}] * 2}) + (True, False) + """ + return all("noise" in one for one in data["states"] if one["kind"] != "published") + + +def noise_of_states(first: Mapping, last: Mapping) -> dict: + """The noise two states are compared with: per indicator, the sum of the noise of the two + (SPEC section 5.3). Two builds of a state differ by its noise, so rebuilding both ends moves + their difference by up to the sum of theirs. The pair holds the results of those two states + only, so the noise of any state between them is not in it. The published results are exact + and add nothing; a built state that did not measure an indicator adds nothing to it, and the + page says at how many ends the noise was measured. An indicator measured at no end is left + out: not measured, which is not the same as two identical builds. The noise of a step, in a + file whose states carry their noise. + + >>> built = lambda by: {"kind": "build", "noise": {"measured": by is not None, "between": by and ["x", "y"], "by": by, "note": None}} + >>> noise_of_states(built({"a": 1e-9}), built({"a": 3e-9, "b": 2e-9}))["by"] + {'a': 4e-09, 'b': 2e-09} + >>> noise_of_states({"kind": "published"}, built({"a": 0.0})) + {'measured': True, 'between': None, 'by': {'a': 0.0}, 'note': None} + >>> noise_of_states({"kind": "published"}, built(None))["measured"] + False + """ + measured = [one["noise"] for one in (first, last) if one["kind"] != "published" and one["noise"]["measured"]] + if not measured: + return {"measured": False, "between": None, "by": None, "note": None} + by: dict[str, float] = {} + for said in measured: + for indicator, value in said["by"].items(): + by[indicator] = by.get(indicator, 0.0) + value + return {"measured": True, "between": None, "by": by, "note": None} + + +def ends_known(first: Mapping, last: Mapping, indicator: str) -> int: + """At how many of the two ends the noise of `indicator` is known: an end is known where it + is the published results (exact) or a build that measured it. + + >>> built = lambda by: {"kind": "build", "noise": {"measured": by is not None, "between": None, "by": by, "note": None}} + >>> ends_known({"kind": "published"}, built({"a": 0.1}), "a"), ends_known(built(None), built({"a": 0.1}), "a") + (2, 1) + """ + return sum( + 1 for one in (first, last) + if one["kind"] == "published" or (one["noise"]["measured"] and indicator in one["noise"]["by"]) + ) # fmt: skip + + +def pair_noise(data: Mapping, first: int, last: int) -> dict[str, float]: + """The noise per indicator that the states at places `first` and `last` are compared with: + that of the two states (`noise_of_states`); nothing in a file whose states carry no noise, + which never says which state its noise belongs to.""" + if not states_carry_noise(data): + return {} + return noise_of_states(data["states"][first], data["states"][last])["by"] or {} + + +def total_noise_level(data: Mapping, indicator: str) -> float | None: + """The noise the whole way is judged with in one indicator: that of its first and last + state; None where neither measured it, or the file was made before noise belonged to a + state.""" + return (data["rule"].get("total_noise") or {}).get(indicator) + + def _rule(data, found, indicators, states, steps) -> dict | None: said = data.get("rule") - if not isinstance(said, dict) or set(said) != RULE_KEYS: - found.append(f"`rule` is not {{{', '.join(sorted(RULE_KEYS))}}}") + if not isinstance(said, dict) or not RULE_KEYS <= set(said) <= RULE_KEYS | RULE_OPTIONAL: + found.append(f"`rule` is not {{{', '.join(sorted(RULE_KEYS))}}}, with `total_noise`") return None if not is_share(said["floor_factor"]): found.append("`rule`: the floor factor is not a share") @@ -394,6 +500,18 @@ def _rule(data, found, indicators, states, steps) -> dict | None: return None if steps and said["total_tolerance"] != max(s["tolerance"]["value"] for s in steps): found.append("`rule.total_tolerance` is not the largest tolerance of the steps") + if "total_noise" in said: + given = said["total_noise"] + if not (isinstance(given, dict) and set(given) <= set(indicators) + and all(is_share(v) for v in given.values())): # fmt: skip + found.append("`rule.total_noise` is not a share per indicator") + return None + if not states_carry_noise(data): + found.append("`rule.total_noise` is the noise of the first and last state, and this file's states carry none") + elif given != pair_noise(data, 0, len(data["states"]) - 1): + found.append("`rule.total_noise` is not the sum of the noise of the first and last state, per indicator") + elif states and states_carry_noise(data): + found.append("`rule.total_noise` is required where the states carry their noise") floors = said["floors"] if not isinstance(floors, list): found.append("`rule.floors` is not a list") @@ -622,7 +740,7 @@ def _items(data, found, indicators, states, steps, floors, tier, blocks): name = TOTAL if k is None else steps[k]["id"] before, after = (values[0], values[-1]) if k is None else (values[k], values[k + 1]) if k is None: - tol, level = data["rule"]["total_tolerance"], None + tol, level = data["rule"]["total_tolerance"], total_noise_level(data, indicator) else: tol = steps[k]["tolerance"]["value"] noise = steps[k]["noise"] @@ -843,7 +961,7 @@ def _errors(data, found: list[str]) -> list[str]: if not isinstance(notes, list) or len(notes) > MAX_NOTES or not all(map(is_text, notes)): found.append(f"`notes` is not a list of at most {MAX_NOTES} texts") indicators = _indicators(data, found) - states = _states(data, found) + states = _states(data, found, indicators) if not indicators or not states: return found steps = _steps(data, found, states, indicators, tier) diff --git a/src/sentier_diagnostics/from_tables.py b/src/sentier_diagnostics/from_tables.py index 538dd0a..c775955 100644 --- a/src/sentier_diagnostics/from_tables.py +++ b/src/sentier_diagnostics/from_tables.py @@ -24,11 +24,13 @@ with `"audience": "local"`): `build_with_matrix` then gives it, and `sentier-diagnostics build` writes it beside the file. -Paths are relative to the description. A step's `noise` is -`{"states": ["a", "b"]}` (two states of this chain that are builds of one state) or +Paths are relative to the description. A built state's `noise` is what two builds of it +differ by: `{"states": ["a", "b"]}` (two states of this chain that are builds of it) or `{"tables": [{"label": "...", "results": "a.csv"}, {"label": "...", "results": "b.csv"}]}`. -It is taken over the results beyond the floors of the chain's first state. A scope entry -without `expect` says what may move (`"may"`), as SPEC section 6 says. +It is taken over the results beyond the floors of the chain's first state. A step takes no +noise of its own: it is compared with the sum of the noise of its two states, and the +published results are exact (SPEC section 5.3). A scope entry without `expect` says what +may move (`"may"`), as SPEC section 6 says. """ from __future__ import annotations @@ -106,6 +108,14 @@ def description_errors(spec) -> list[str]: found.append("`states` is a list of at least two {id, label, results}, `results` naming a table") states = [] known = {s["id"] for s in states if isinstance(s.get("id"), str)} + for one in states: + said = one.get("noise") + if said is not None: + where = f"state {one['id']!r}" + if one.get("kind") == "published": + found.append(f"{where}: the published results have no builds and take no noise: they are exact") + else: + found += _noise_errors(said, where, known) steps = spec.get("steps") if not (isinstance(steps, list) and all(_table(s, ("id", "label", "kind")) for s in steps)): return [*found, "`steps` is a list of {id, label, kind}"] @@ -125,28 +135,27 @@ def description_errors(spec) -> list[str]: said = one.get("tolerance") if said is not None and not (_table(said, ("value",)) and _number(said["value"])): found.append(f"{where}: a tolerance is {{value, basis, note}} with a number as its value") - said = one.get("noise") - if said is not None: - pair = said.get("states") if isinstance(said, dict) else None - tables_ = said.get("tables") if isinstance(said, dict) else None - if pair is not None: - if not ( - isinstance(pair, list) - and len(pair) == 2 - and all(isinstance(s, str) and s in known for s in pair) - ): - found.append(f"{where}: a noise's `states` names two states of this chain") - elif not (isinstance(tables_, list) and len(tables_) == 2 - and all(_table(t, ("label", "results")) and _text(t["results"]) for t in tables_)): # fmt: skip - found.append( - f"{where}: a noise is {{states: [a, b]}} or {{tables: [{{label, results}}, {{label, results}}]}}" - ) + if one.get("noise") is not None: + found.append(f"{where}: noise belongs to the state that was built twice: move it to that state " + "(a step is compared with the sum of the noise of its two states)") # fmt: skip explanations = spec.get("explanations", []) if not (isinstance(explanations, list) and all(_table(e, ("text",)) for e in explanations)): found.append("`explanations` is a list of {text, by, ...}") return found +def _noise_errors(said, where: str, known: set) -> list[str]: + pair = said.get("states") if isinstance(said, dict) else None + tables_ = said.get("tables") if isinstance(said, dict) else None + if pair is not None: + if not (isinstance(pair, list) and len(pair) == 2 and all(isinstance(s, str) and s in known for s in pair)): + return [f"{where}: a noise's `states` names two states of this chain"] + elif not (isinstance(tables_, list) and len(tables_) == 2 + and all(_table(t, ("label", "results")) and _text(t["results"]) for t in tables_)): # fmt: skip + return [f"{where}: a noise is {{states: [a, b]}} or {{tables: [{{label, results}}, {{label, results}}]}}"] + return [] + + def build(path: Path) -> dict: """The diagnostics file a description and its tables give, checked. Every mistake is a ValueError that says what is wrong (`tables.TableError` for a table).""" @@ -188,6 +197,24 @@ def build_with_matrix(path: Path) -> tuple[dict, dict | None]: def floor_by_item(indicator): return {item: floors.get((indicator, unit), (0.0, 0))[0] for item, unit in units.items()} + def noise_of_state(said): + """What two builds of a state differ by, from two states of this chain or two tables.""" + if not said: + return None + if "states" in said: + a, b = said["states"] + pair, labels = (values[a], values[b]), (a, b) + else: + first, second = said["tables"] + pair = (tables.read_results(here / first["results"], ids), + tables.read_results(here / second["results"], ids)) # fmt: skip + labels = (first["label"], second["label"]) + return chain.noise( + labels, + {i: chain.noise_of(pair[0][i], pair[1][i], floor=floor_by_item(i)) for i in ids}, + said.get("note"), + ) + steps = [] for one in spec["steps"]: said_claim = one.get("claim") @@ -208,22 +235,6 @@ def floor_by_item(indicator): by=_origin(said_claim.get("by")), reach=said_claim.get("reach"), ) - said_noise = one.get("noise") - found_noise = None - if said_noise: - if "states" in said_noise: - a, b = said_noise["states"] - pair, labels = (values[a], values[b]), (a, b) - else: - first, second = said_noise["tables"] - pair = (tables.read_results(here / first["results"], ids), - tables.read_results(here / second["results"], ids)) # fmt: skip - labels = (first["label"], second["label"]) - found_noise = chain.noise( - labels, - {i: chain.noise_of(pair[0][i], pair[1][i], floor=floor_by_item(i)) for i in ids}, - said_noise.get("note"), - ) said_tol = one.get("tolerance") steps.append( chain.step( @@ -237,7 +248,6 @@ def floor_by_item(indicator): tolerance=chain.tolerance(said_tol["value"], said_tol.get("basis"), said_tol.get("note")) if said_tol else None, - noise=found_noise, ) # fmt: skip ) explanations = [ @@ -273,6 +283,7 @@ def floor_by_item(indicator): about=s.get("about"), identity=s.get("identity"), join=s.get("join"), + noise=noise_of_state(s.get("noise")), ) # fmt: skip for s in spec["states"] ], diff --git a/src/sentier_diagnostics/matrix.py b/src/sentier_diagnostics/matrix.py index f7c3e0d..1f714cc 100644 --- a/src/sentier_diagnostics/matrix.py +++ b/src/sentier_diagnostics/matrix.py @@ -6,7 +6,9 @@ build wants every item: the matrix is that, in a second file the page loads after it has drawn. It is compact (one row per item, the indicators and states named once) and it holds only what the rule gives: the results, and one code per pair for its class and direction, -and per step with a claim one code for its verdict. +and per step with a claim one code for its verdict. A result equal to the one of the state +before it, in the same indicator, is written `"="`: most steps of a chain of fixes leave most +results as they were, and a chain of 16 states would otherwise hold every number 16 times. Only a local file may name a matrix (`"matrix": {"file": ..., "items": n}`); a public file keeps the listed items only. `errors(m)` checks the shape; `against(m, data)` works every @@ -28,6 +30,9 @@ KEYS = {"format", "of", "items", "values", "classes", "verdicts"} OF_KEYS = {"title", "states", "indicators", "steps"} +#: A result written as the same number as the state before it, in the same indicator. +SAME = "=" + #: A matrix is loaded after the page is drawn, not held inside it: it may be larger. MAX_BYTES = 64 * 1024 * 1024 MAX_ITEMS = 200_000 @@ -75,6 +80,37 @@ def code_of(kind: str, way: str | None) -> str: return CODES[(kind, way)] +def _same(a, b) -> bool: + """Two results that are the same number, bit for bit (0.0 and -0.0 are not).""" + return a is not None and b is not None and float(a).hex() == float(b).hex() + + +def compact(values: list) -> list: + """One indicator's results over the states, each equal to the one before it as `SAME`. + + >>> compact([1.5, 1.5, None, None, 2.0, 2.0, -0.0, 0.0]) + [1.5, '=', None, None, 2.0, '=', -0.0, 0.0] + """ + return [SAME if n and _same(v, values[n - 1]) else v for n, v in enumerate(values)] + + +def expand(values: list, nstates: int) -> list: + """The results of one item, every `SAME` written out again; raises ValueError where one + stands first in its indicator or after a missing result. + + >>> expand([1.5, "=", None, 2.0], 2) + [1.5, 1.5, None, 2.0] + """ + out = [] + for n, v in enumerate(values): + if v == SAME: + if n % nstates == 0 or out[-1] is None: + raise ValueError("`=` stands first in an indicator, or after a missing result") + v = out[-1] + out.append(v) + return out + + def make(data: Mapping, rows: list[dict]) -> dict: """A matrix of a diagnostics file from its rows, as `chain.build_with_matrix` gives them: `{id, name, place, group, unit, values: {indicator: [per state]}, classes, directions, @@ -96,7 +132,7 @@ def make(data: Mapping, rows: list[dict]) -> dict: } for row in rows: out["items"].append([row["id"], row["name"], row["place"], row["group"], row["unit"]]) - out["values"].append([v for i in ids for v in row["values"][i]]) + out["values"].append([v for i in ids for v in compact(row["values"][i])]) out["classes"].append( "".join( code_of(kind, way) for i in ids for kind, way in zip(row["classes"][i], row["directions"][i]) @@ -223,8 +259,12 @@ def _errors(m) -> list[str]: seen.add(row[0]) values = m["values"][at] if not (isinstance(values, list) and len(values) == nind * nstates - and all(v is None or rule.is_value(v) for v in values)): # fmt: skip - return [f"item {row[0]!r}: not one finite number or null per indicator and state"] + and all(v is None or v == SAME or rule.is_value(v) for v in values)): # fmt: skip + return [f"item {row[0]!r}: not one finite number, `=` or null per indicator and state"] + try: + expand(values, nstates) + except ValueError as error: + return [f"item {row[0]!r}: {error}"] codes = m["classes"][at] if not ( isinstance(codes, str) and len(codes) == nind * (nsteps + 1) and all(c in DECODE for c in codes) @@ -296,7 +336,7 @@ def empty(k): qualifying = 0 for at, (item_id, name, place, group, unit) in enumerate(m["items"]): where = f"item {item_id!r}" - flat, codes = m["values"][at], m["classes"][at] + flat, codes = expand(m["values"][at], nstates), m["classes"][at] said_verdicts = m["verdicts"][at] if m["verdicts"] is not None else None worth = False values_of = {} @@ -311,7 +351,7 @@ def empty(k): at_k = nsteps if k is None else k before, after = (values[0], values[-1]) if k is None else (values[k], values[k + 1]) if k is None: - tol, level = data["rule"]["total_tolerance"], None + tol, level = data["rule"]["total_tolerance"], fileformat.total_noise_level(data, indicator) else: tol = steps[k]["tolerance"]["value"] noise = steps[k]["noise"] diff --git a/src/sentier_diagnostics/page/body.html b/src/sentier_diagnostics/page/body.html index 0c8b053..0a54ee5 100644 --- a/src/sentier_diagnostics/page/body.html +++ b/src/sentier_diagnostics/page/body.html @@ -18,9 +18,25 @@

Open a file