fix(novelty): a capped run probes atoms spread over the whole input

dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md

@@ -0,0 +1,32 @@
+### Changed
+
+- **A capped novelty run probes atoms spread over the whole input** — under
+  `--max-atoms` or a tier's cap (25 at `triage`, 60 at `standard`, 120 at
+  `study`), extraction no longer stops at the cap: it reads every span, and of
+  the M atoms it extracts the run probes n at indices
+  `floor((i + 0.5) * M / n)`, every (M/n)-th in document order by the earliest
+  span each atom cites (on a `--include-resources` run, across the files in
+  file order, so a file with few atoms may get no probe), so a capped rate
+  estimates the whole input's per-atom rate instead of describing its opening.
+  Like any fixed stride it can fall in step with repeated, same-shaped
+  sections and miss or over-count what each one adds; for such an artifact, or
+  a rate that decides something, pass `--all-atoms`. The selection takes no
+  seed: the same extracted atom list gives the same sample. `novelty analyze`
+  and `novelty atoms` (whose `atoms.json` `probe` and `shorten` read) probe the
+  same atoms. A cap below 1 is refused before any call. New runs record
+  `extraction.reached` `null`; when the cap leaves atoms out, `atoms.sampling`
+  is `cap-spread:<n>` and `extraction.sampled` `{"atoms", "of", "rule"}`, and
+  when it keeps every atom (M ≤ n), `full` and `null`.
+  Only a cap that leaves atoms out sets `spans.capped_atom_set` and brings the
+  cap's forfeits: an input whose atoms exactly meet the cap no longer counts as
+  capped. The cap warning, the `novelty.md` header and the headline's
+  `coverage:` line say how many of the input's atoms were probed and with what
+  stride, or, when some spans went unread, that the sample is spread over the
+  atoms of the spans that were read and how many were not. Artifacts written
+  with `cap-per-artifact:<n>` and a `reached` record still validate and render
+  as before. Probing still stops at the cap, but a run whose cap was reached
+  before its last extraction batch now asks the remaining batches, and
+  `--estimate-only` counts them, one call per batch of 25 spans; the tier profiles were measured with the old
+  order (decision D017). The
+  shortening forfeit now says candidates are scored against a spread sample of
+  the cap's atoms, so cutting a span with no sampled atom looks free.

dokimi/docs/reviewing.md

@@ -83,13 +83,15 @@ self-judge; `arms run` does not check, so name
   times arms, or tasks times harnesses. One measured example: triage on a
   19k-token skill with a mid-size model was 103 calls, about $1.80; `study` on
   the same skill was 1,900 calls, about $14.50.
-- **At triage** with `--include-resources`, the 25-atom cap takes atoms in
-  document order, and the entry point comes first: the flag reaches
-  `references/` only if `SKILL.md` has fewer than 25 atoms, and the run's cap
-  warning prints how far it reached through the input, the entry point and the files the budgets selected (span,
-  spans and token share). For such skills,
-  use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for
-  a full sweep; run `--estimate-only` first, it costs more.
+- **At triage** with `--include-resources`, the 25-atom cap strides over the
+  concatenated atom list of the files the budgets selected, in file order, so
+  `references/` can be probed, but a file with few atoms may get no probe: with
+  one atom in each of 100 files, triage probes 25 of them. The run's cap warning
+  says how many of the input's atoms it probed. Extraction still reads every
+  file, so a large bundle costs more extraction calls than its 25 probes suggest.
+  `--tier standard` (60 atoms) spreads more probes without promising each file
+  one; add `--all-atoms` to the tier command to probe every extracted atom, and
+  run `--estimate-only` first, it costs more.
 - Step 5 needs evaluation content: a golden question set for 5a, executable
   tasks for 5b. `arms run` takes no tasks. That content lives outside this repository
   ([external-content.md](external-content.md)). Without it, the

dokimi/docs/skill-novelty-walkthrough.md

@@ -94,9 +94,10 @@ comparable with another artifact's.
 
 If the skill is a thin `SKILL.md` over a `references/` tree, add `--include-resources`.
 Without it the run measures the entry point and prints the share it is skipping — in the
-study corpus that share was 0.72 of all words. At `triage` the 25-atom cap takes atoms in
-document order and the entry point comes first, so the flag reaches `references/` only if
-`SKILL.md` has fewer than 25 atoms; otherwise use `--tier standard` or higher.
+study corpus that share was 0.72 of all words. At `triage` the 25-atom cap strides over the
+concatenated atom list of the selected files, in file order, so a file with few atoms may get no
+probe. A higher tier spreads more probes without promising each file one; `--all-atoms` probes
+every extracted atom.
 
 ## 4–5. Extract once, probe per model
 

dokimi/docs/skill-novelty.md

@@ -131,10 +131,10 @@ marked. Read it as a review queue:
 - **Internal only.** It says the two statements disagree, not which is wrong. That needs a
   primary source, which this layer does not consult.
 - **Lower bound.** Only atoms in the same call are compared. `analyze` sends the whole
-  extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared
+  probed atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared
   unless a call fails or its reply has to be split. The artifact records the share as
   `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of
-  Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which are only the ones the cap kept when
+  Y atom pairs` when it is short. `pairs_total` is over the probed atoms, which are only the ones the cap kept when
   `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only
   when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped;
   otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A
@@ -885,43 +885,79 @@ tokens: above that the dollar figure extrapolates, and the payload says so
 arm that carries the artifact (every arm but `no_artifact`), so a three-arm run costs more
 per answerer call than a two-arm one; a `--placebo-text` of another length is not read. Only
 three of the 12 are standard runs, so the standard profile rests on little data.
+The measured runs, the tier table above among them, predate the spread cap: their extraction
+stopped at the cap. A capped run now asks every extraction batch, and the projection counts
+one extraction call per batch of 25 spans, as the run asks them, while the probes stay at the
+cap. On an
+artifact with more atoms than the cap that adds extractor calls no measured run has checked
+yet; the held-out figures above are for the runs as they were measured.
 An explicit flag always beats the tier, and the override is recorded — a
 `--tier triage --max-atoms 90` run is not a stock triage run and does not get to be
 read as one.
 
 Each tier carries the **claims it forfeits**, and those strings are written into
 `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the
 run used and the atoms it extracted, not the tier's name: `--no-verify` adds the
-faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the
-cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and
+faithfulness forfeit, and `--all-atoms` or a skill with no more atoms than the cap (which
+keeps every atom: `extraction.sampled` is `null` and `spans.capped_atom_set` false) drops the
+cap's ones (that `added` is a rate over the sample of extracted atoms the cap kept, not over
+every atom, and
 that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on
-how far the cap reached (`--estimate-only` lists them before it is known), and neither does
-the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`).
-`--estimate-only` cannot know the atom count yet, so it lists them:
+which atoms the cap kept (`--estimate-only` lists them before it is known), and neither does
+the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). With shortening on, a
+cap that left atoms out adds that `shortening candidates are scored against a spread sample of 60 of the
+extracted atoms: a cut of a span with no sampled atom looks free` (60 at `standard`, 120 at
+`study`). `--estimate-only` cannot know the atom count yet, so it lists them:
 
 > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected
 > against the model matching the artifact's own wording
 
-The cap is not a sample: extraction walks the spans in document order and stops at it, so a
-capped run measures the input up to the span it stopped at. The input is the text that was segmented: the
-entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file
-left out is in none of the reach's spans or tokens (the `coverage:` line says how much of the
-artifact went into the input). The warning says how far the cap reached, as in `atom
-extraction hit the cap of 25 at span s0012 (12 of the input's 61 spans, 23% of its tokens):
-rates describe the opening of the input, not all of it; pass --all-atoms for the full sweep`,
-`extraction.reached` records it, and the headline's `coverage:` line then says the words
-went into the input and the capped atoms come from the first 23% of the input's tokens (the
-warning under that line, when half or less of the words went in, says "went into the input"
-too). `novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its
-`atoms.json`, do not repeat it.
-
-A cap met on the input's last span leaves no stretch of the input out: what it cut is any atoms
-past the cap in that last reply. The warning then reads `atom extraction hit the cap of 30 at
-span s0010 (10 of the input's 10 spans, 100% of its tokens): any atoms past the cap in its last
-reply were dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the
-atoms reach the input's last span and the cap dropped any atoms past it in the last reply, and
-the `coverage:` line ends `the capped atoms reach the input's last span, and the cap dropped
-any atoms past it`; none of them says "the opening".
+The cap is a spread sample, not the opening. Extraction reads every span of the input, and when
+it yields M atoms, more than the cap n, the run probes the atoms at indices
+`floor((i + 0.5) * M / n)` for `i` in `0..n-1` of the extracted list sorted by the earliest span
+each atom cites (`spread_sample` in `dokimi/novelty/atoms.py`): every (M/n)-th atom in document
+order, starting in the middle of the first stride, so the sample estimates the per-atom rate an
+uncapped run reports. Like any fixed stride, it can fall in step with a repeated structure: with
+M = 100 and n = 25 it always takes atoms 2, 6, ..., 98 (from 0), so an input of four-atom
+sections of the same shape, each with its new fact in the same place, can give a sampled rate
+far from the whole input's, and re-extracting the same atoms gives the same sample, so it cannot
+show the gap. For an artifact of repeated, same-shaped sections (a template per item), or a
+rate that decides something, pass `--all-atoms`. The sort is needed because the extractor's
+reply lists a batch's atoms in its own order; it is stable, so atoms citing the same span keep
+that order, and with M ≤ n nothing is sorted and every atom is probed in extraction order. A
+stride needs no seed: the same extracted atom list gives the same sample. A fresh extraction is new model calls, so the
+same text can give other atoms and so another sample; to probe identical atoms across runs or
+models, extract once with `novelty atoms` and pass its `atoms.json` to `novelty probe --atoms`
+(and `novelty shorten --atoms`), which probe the saved atoms without sampling again. The input
+is the text that was segmented: the entry point, plus on a bundle run the files its roles and
+budgets selected, in that order. The stride runs over the concatenated atom list of those
+files, so a file's expected share of the probes is its share of the atoms, but a file with few
+atoms can fall between two sampled indices and get no probe; a higher cap spreads more probes
+without promising each file one, and `--all-atoms` probes every extracted atom. A bundled file
+left out has no atom in it (the `coverage:` line says how much of the artifact went into the
+input). The run records `atoms.sampling` as `cap-spread:<n>` and `extraction.sampled` as
+`{"atoms": n, "of": M, "rule": "cap-spread"}`; a run whose cap kept every atom (M ≤ n) records
+`full`, as an uncapped run does, and `null`. When every span was read it warns `atom
+extraction read the whole input, and the atom cap probes 25 of the input's 61 atoms, spread over
+all of it (a stride of 2.44 atoms in document order): rates are estimated from that sample; pass
+--all-atoms to probe every atom`. When some spans were left unread (`extraction.unread_span_ids`:
+a failed call, an empty or unparseable reply), M counts only the atoms of the spans that were
+read, and the warning says so instead: `atom extraction could not read 25 of the input's spans,
+and the atom cap probes 2 of the 25 atoms extracted from the spans that were read, spread over
+those (a stride of 12.5 atoms in document order; 25 of the input's spans were not read, see
+extraction.unread_span_ids): ...`. The `novelty.md` header says the same, and the headline's
+`coverage:` line says the words went into the input and the cap probed 25 of the input's 61
+atoms, or of the atoms extracted from the spans that were read (the warning under that line,
+when half or less of the words went in, says "went into the input" too). `novelty atoms` prints
+the same warning on stderr and saves the same sample in `atoms.json`; `probe` and `shorten`,
+which read it, do not repeat the warning.
+
+Runs written before the spread rule recorded `cap-per-artifact:<n>`: extraction walked the spans
+in document order and stopped at the cap, so their atoms cover the input up to the span it
+stopped at, and `extraction.reached` says how far (`extraction.sampled` is absent). Their
+`novelty.md` header and `coverage:` line still say so, as in `the capped atoms come from the
+first 23% of the input's tokens`, or, for a cap met on the input's last span, `the capped atoms
+reach the input's last span, and the cap dropped any atoms past it`.
 
 That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a
 sweep is not.
@@ -968,7 +1004,7 @@ at exactly 4096 tokens, cut mid-array. Extraction recovers by splitting an unpar
 all unusable) in half and re-asking, down to one span per call, so a truncation costs extra calls rather than
 atoms: up to 2n−1 calls for an n-span batch that never parses. The artifact's `extraction`
 block counts them (`calls` against `batches`, the calls the run would make with no splitting; a
-capped run counts only the batches it reached), and a warning (`atom extraction re-asked N
+capped run asks every batch too), and a warning (`atom extraction re-asked N
 unparseable batches as halves`) says what they cost; `--max-tokens 16000` avoids the truncated ones, not those with unusable rows. A call
 that fails or returns an empty reply is not split, since half a batch fails the same way.
 
@@ -989,7 +1025,8 @@ lost to a failed call (with its error). The
 | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked |
 | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting |
 | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded |
-| `reached` | on a capped run, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind |
+| `sampled` | on a run whose cap left atoms out, what it probed: `atoms`, the cap; `of`, the atoms extracted from every span some call read; `rule`, `cap-spread`. `null` when every atom was probed |
+| `reached` | `null` on a `cap-spread` run. On a run before it, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind |
 
 `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out
 of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as

dokimi/dokimi/commands/novelty.py

@@ -158,16 +158,16 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None:
     except ExtractionError as exc:
         raise SystemExit(str(exc)) from None
     # The cap warning analyze records; probe and shorten, which read atoms.json, do not repeat it.
-    reach = [cap_warning(cap, extraction.reached)] if extraction.reached else []
-    for warning in extraction.warnings(len(spans)) + reach:
+    sampled = [cap_warning(extraction.as_dict())] if extraction.sampled else []
+    for warning in extraction.warnings(len(spans)) + sampled:
         print(f"WARNING {warning}", file=sys.stderr)
     if settings.values["verify_faithfulness"]:
         verify_atoms(atoms, spans, client=extractor)
     out = Path(args.out) if args.out else _default_dir(args.artifact) / "atoms.json"
     save_atoms(atoms, spans, out, extractor=extractor.model, cap=cap,
                bundle=artifact_bundle.summary(), extraction=extraction)
     print(f"OK Saved {len(atoms)} atoms over {len(spans)} spans: {out}")
-    print(json.dumps(span_summary(spans, atoms, atom_cap=cap), indent=2))
+    print(json.dumps(span_summary(spans, atoms, capped=extraction.sampled is not None), indent=2))
 
 
 def cmd_novelty_probe(args: argparse.Namespace) -> None:
@@ -948,7 +948,7 @@ def _add_common(parser: argparse.ArgumentParser, positive_int) -> None:
                              "Unset by default so that passing the default value explicitly "
                              "still overrides a --tier cap")
     parser.add_argument("--all-atoms", action="store_true", dest="all_atoms",
-                        help="Extract every atom (opt-in: this is the expensive sweep)")
+                        help="Probe every extracted atom (opt-in: this is the expensive sweep)")
     parser.add_argument("--budgets", type=float, nargs="+", default=[0.75, 0.5],
                         help="Target token fractions for shortening candidates")
     parser.add_argument("--cache-dir", default=None, dest="cache_dir",
@@ -1159,7 +1159,7 @@ def _print_coverage(bundle: dict, extraction: dict | None = None) -> None:
     The share is stated on stdout because the headline is what a user copies. A run under
     the floor also says so on stdout: a stderr-only warning is one a user piping the
     headline never sees, and a rate quoted off half-read skill is not a property of it.
-    A capped run says the words went into the input, and how far its atoms reach.
+    A capped run says the words went into the input, and how many of its atoms were probed.
     """
     from dokimi.novelty.bundle import MEASURED_SHARE_WARN_AT, Coverage
     from dokimi.novelty.report import coverage_reach, floored_percent