From ecdb5eb9696b394f236e6daa43469f14493c6a05 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Sun, 1 Nov 2026 18:19:53 +0000 Subject: [PATCH 1/8] fix(novelty): spread the atom cap over the whole input Settles intel/dokimi#293. A capped novelty run took atoms in document order and stopped extraction at the cap, so its rates described the opening of the input (at triage, about a quarter of the root AGENTS.md of the dpnp guidance) and moved when sections were reordered. Extraction now reads every span. Of the M atoms extracted, a cap of n keeps those at floor((i + 0.5) * M / n), in extraction order (document order; on a bundle run, the files in bundle order), via the new atoms.spread_sample. There is no seed: the same extracted list gives the same sample. With M <= n every atom is kept, as before. novelty analyze and novelty atoms both go through extract_atoms, so probe and shorten read the same sample from atoms.json. - sampling is cap-spread: for new runs; extraction.sampled records {"atoms", "of", "rule"} and extraction.reached is null. novelty.v1 is widened in place: older cap-per-artifact artifacts with a reach still validate and render as before. - The cap warning, the novelty.md header and the coverage: line say how many of the input's atoms were probed and at what stride. - The shortening forfeit says candidates are scored against a spread sample of the cap's atoms, so cutting a span with no sampled atom looks free. - --estimate-only counts every extraction batch over the expected uncapped atoms; probing stays at the cap. The held-out calibration checks still project the runs as they were measured. - Decision D017 records the rule and leaves re-measuring the capped tier profiles to follow-up work. Docs and a changelog fragment (Changed) follow the code. Signed-off-by: Anton Volkov --- ...01_120000_antonwolfy_novelty-cap-spread.md | 21 +++ dokimi/docs/reviewing.md | 12 +- dokimi/docs/skill-novelty-walkthrough.md | 6 +- dokimi/docs/skill-novelty.md | 67 +++++---- dokimi/dokimi/commands/novelty.py | 8 +- dokimi/dokimi/novelty/analyze.py | 9 +- dokimi/dokimi/novelty/atoms.py | 84 +++++------ dokimi/dokimi/novelty/report.py | 35 +++-- dokimi/dokimi/novelty/tiers.py | 26 ++-- dokimi/dokimi/schemas/novelty.v1.json | 18 ++- dokimi/research/records/decision/D017.md | 21 +++ dokimi/scripts/growth_baseline.json | 2 +- dokimi/tests/test_novelty_analyze.py | 109 ++++++++------ dokimi/tests/test_novelty_atoms.py | 137 +++++++++++------- dokimi/tests/test_novelty_cli.py | 11 ++ dokimi/tests/test_novelty_report.py | 58 ++++++-- dokimi/tests/test_novelty_settings_parity.py | 46 +++++- dokimi/tests/test_novelty_tiers.py | 32 ++-- 18 files changed, 459 insertions(+), 243 deletions(-) create mode 100644 dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md create mode 100644 dokimi/research/records/decision/D017.md diff --git a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md new file mode 100644 index 000000000..6716f588c --- /dev/null +++ b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md @@ -0,0 +1,21 @@ +### Changed + +- **A capped novelty run probes atoms spread over the whole input** — under + `--max-atoms` or a tier's cap (25 at `triage`, 60 at `standard`, 120 at + `study`), extraction no longer stops at the cap: it reads every span, and of + the M atoms it extracts the run probes n at indices + `floor((i + 0.5) * M / n)`, every (M/n)-th in document order (on a + `--include-resources` run, across the files in file order), so a capped rate + estimates the whole input's per-atom rate instead of describing its opening. + The selection takes no seed. `novelty analyze` and `novelty atoms` (whose + `atoms.json` `probe` and `shorten` read) probe the same atoms. New runs record + `atoms.sampling` as `cap-spread:` and `extraction.sampled` as + `{"atoms", "of", "rule"}`, with `extraction.reached` `null`; the cap + warning, the `novelty.md` header and the headline's `coverage:` line say how + many of the input's atoms were probed and with what stride. Artifacts written + with `cap-per-artifact:` and a `reached` record still validate and render + as before. Probing still stops at the cap, but on an artifact with more atoms + than the cap extraction now makes more calls, and `--estimate-only` counts + them; the tier profiles were measured with the old order (decision D017). The + shortening forfeit now says candidates are scored against a spread sample of + the cap's atoms, so cutting a span with no sampled atom looks free. diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 6fa1a3ff1..30c0f851b 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -83,12 +83,12 @@ self-judge; `arms run` does not check, so name times arms, or tasks times harnesses. One measured example: triage on a 19k-token skill with a mid-size model was 103 calls, about $1.80; `study` on the same skill was 1,900 calls, about $14.50. -- **At triage** with `--include-resources`, the 25-atom cap takes atoms in - document order, and the entry point comes first: the flag reaches - `references/` only if `SKILL.md` has fewer than 25 atoms, and the run's cap - warning prints how far it reached through the input, the entry point and the files the budgets selected (span, - spans and token share). For such skills, - use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for +- **At triage** with `--include-resources`, the 25-atom cap probes atoms spread + over every file the budgets selected, in file order, so `references/` gets its + share by atom count; the run's cap warning says how many of the input's atoms + it probed. Extraction still reads every file, so a large bundle costs more + extraction calls than its 25 probes suggest. Spread over several files, 25 + atoms leave each file few; for a figure per file, use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for a full sweep; run `--estimate-only` first, it costs more. - Step 5 needs evaluation content: a golden question set for 5a, executable tasks for 5b. `arms run` takes no tasks. That content lives outside this repository diff --git a/dokimi/docs/skill-novelty-walkthrough.md b/dokimi/docs/skill-novelty-walkthrough.md index 5ad273f62..b17b52630 100644 --- a/dokimi/docs/skill-novelty-walkthrough.md +++ b/dokimi/docs/skill-novelty-walkthrough.md @@ -94,9 +94,9 @@ comparable with another artifact's. If the skill is a thin `SKILL.md` over a `references/` tree, add `--include-resources`. Without it the run measures the entry point and prints the share it is skipping — in the -study corpus that share was 0.72 of all words. At `triage` the 25-atom cap takes atoms in -document order and the entry point comes first, so the flag reaches `references/` only if -`SKILL.md` has fewer than 25 atoms; otherwise use `--tier standard` or higher. +study corpus that share was 0.72 of all words. At `triage` the 25-atom cap probes atoms spread +over every selected file in file order, so `references/` gets its share by atom count; 25 atoms +over several files leave each one few, so for a figure per file use `--tier standard` or higher. ## 4–5. Extract once, probe per model diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 0894076c7..7a3f93ec5 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -131,10 +131,10 @@ marked. Read it as a review queue: - **Internal only.** It says the two statements disagree, not which is wrong. That needs a primary source, which this layer does not consult. - **Lower bound.** Only atoms in the same call are compared. `analyze` sends the whole - extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared + probed atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared unless a call fails or its reply has to be split. The artifact records the share as `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of - Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which are only the ones the cap kept when + Y atom pairs` when it is short. `pairs_total` is over the probed atoms, which are only the ones the cap kept when `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped; otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A @@ -885,6 +885,11 @@ tokens: above that the dollar figure extrapolates, and the payload says so arm that carries the artifact (every arm but `no_artifact`), so a three-arm run costs more per answerer call than a two-arm one; a `--placebo-text` of another length is not read. Only three of the 12 are standard runs, so the standard profile rests on little data. +The measured runs, the tier table above among them, predate the spread cap: their extraction +stopped at the cap. A capped run now asks every extraction batch, and the projection counts +those batches over the atoms expected before the cap, while the probes stay at the cap. On an +artifact with more atoms than the cap that adds extractor calls no measured run has checked +yet; the held-out figures above are for the runs as they were measured. An explicit flag always beats the tier, and the override is recorded — a `--tier triage --max-atoms 90` run is not a stock triage run and does not get to be read as one. @@ -895,33 +900,40 @@ run used and the atoms it extracted, not the tier's name: `--no-verify` adds the faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on -how far the cap reached (`--estimate-only` lists them before it is known), and neither does -the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). -`--estimate-only` cannot know the atom count yet, so it lists them: +which atoms the cap kept (`--estimate-only` lists them before it is known), and neither does +the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). With shortening on, a +reached cap adds that `shortening candidates are scored against a spread sample of 60 of the +extracted atoms: a cut of a span with no sampled atom looks free` (60 at `standard`, 120 at +`study`). `--estimate-only` cannot know the atom count yet, so it lists them: > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected > against the model matching the artifact's own wording -The cap is not a sample: extraction walks the spans in document order and stops at it, so a -capped run measures the input up to the span it stopped at. The input is the text that was segmented: the -entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file -left out is in none of the reach's spans or tokens (the `coverage:` line says how much of the -artifact went into the input). The warning says how far the cap reached, as in `atom -extraction hit the cap of 25 at span s0012 (12 of the input's 61 spans, 23% of its tokens): -rates describe the opening of the input, not all of it; pass --all-atoms for the full sweep`, -`extraction.reached` records it, and the headline's `coverage:` line then says the words -went into the input and the capped atoms come from the first 23% of the input's tokens (the -warning under that line, when half or less of the words went in, says "went into the input" -too). `novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its -`atoms.json`, do not repeat it. - -A cap met on the input's last span leaves no stretch of the input out: what it cut is any atoms -past the cap in that last reply. The warning then reads `atom extraction hit the cap of 30 at -span s0010 (10 of the input's 10 spans, 100% of its tokens): any atoms past the cap in its last -reply were dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the -atoms reach the input's last span and the cap dropped any atoms past it in the last reply, and -the `coverage:` line ends `the capped atoms reach the input's last span, and the cap dropped -any atoms past it`; none of them says "the opening". +The cap is a spread sample, not the opening. Extraction reads every span of the input, and when +it yields M atoms, more than the cap n, the run probes the atoms at indices +`floor((i + 0.5) * M / n)` for `i` in `0..n-1` of the extracted list (`spread_sample` in +`dokimi/novelty/atoms.py`): every (M/n)-th atom in document order, starting in the middle of the +first stride, so the sample estimates the per-atom rate an uncapped run reports. A stride needs no +seed: the same text gives the same sample. With M ≤ n every atom is probed. The input is the text that was +segmented: the entry point, plus on a bundle run the files its roles and budgets selected, in that +order, so the stride gives each selected file its share by atom count, and a bundled file left +out has no atom in it (the `coverage:` line says how much of the artifact went into the input). +The run records `atoms.sampling` as `cap-spread:` and `extraction.sampled` as +`{"atoms": n, "of": M, "rule": "cap-spread"}`, and warns `atom extraction read the whole input, +and the atom cap probes 25 of the input's 61 atoms, spread over all of it (a stride of 2.44 +atoms in document order): rates are estimated from that sample; pass --all-atoms to probe every +atom`. The `novelty.md` header says the same, and the headline's `coverage:` line says the +words went into the input and the cap probed 25 of the input's 61 atoms (the warning under that +line, when half or less of the words went in, says "went into the input" too). `novelty atoms` +prints the same warning on stderr and saves the same sample in `atoms.json`; `probe` and +`shorten`, which read it, do not repeat the warning. + +Runs written before the spread rule recorded `cap-per-artifact:`: extraction walked the spans +in document order and stopped at the cap, so their atoms cover the input up to the span it +stopped at, and `extraction.reached` says how far (`extraction.sampled` is absent). Their +`novelty.md` header and `coverage:` line still say so, as in `the capped atoms come from the +first 23% of the input's tokens`, or, for a cap met on the input's last span, `the capped atoms +reach the input's last span, and the cap dropped any atoms past it`. That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a sweep is not. @@ -968,7 +980,7 @@ at exactly 4096 tokens, cut mid-array. Extraction recovers by splitting an unpar all unusable) in half and re-asking, down to one span per call, so a truncation costs extra calls rather than atoms: up to 2n−1 calls for an n-span batch that never parses. The artifact's `extraction` block counts them (`calls` against `batches`, the calls the run would make with no splitting; a -capped run counts only the batches it reached), and a warning (`atom extraction re-asked N +capped run asks every batch too), and a warning (`atom extraction re-asked N unparseable batches as halves`) says what they cost; `--max-tokens 16000` avoids the truncated ones, not those with unusable rows. A call that fails or returns an empty reply is not split, since half a batch fails the same way. @@ -989,7 +1001,8 @@ lost to a failed call (with its error). The | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked | | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting | | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded | -| `reached` | on a capped run, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind | +| `sampled` | on a run whose cap left atoms out, what it probed: `atoms`, the cap; `of`, the atoms extracted from every span; `rule`, `cap-spread`. `null` when every atom was probed | +| `reached` | `null` on a `cap-spread` run. On a run before it, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind | `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index ae625fd9c..3bd0f1310 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -158,8 +158,8 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None: except ExtractionError as exc: raise SystemExit(str(exc)) from None # The cap warning analyze records; probe and shorten, which read atoms.json, do not repeat it. - reach = [cap_warning(cap, extraction.reached)] if extraction.reached else [] - for warning in extraction.warnings(len(spans)) + reach: + sampled = [cap_warning(extraction.sampled)] if extraction.sampled else [] + for warning in extraction.warnings(len(spans)) + sampled: print(f"WARNING {warning}", file=sys.stderr) if settings.values["verify_faithfulness"]: verify_atoms(atoms, spans, client=extractor) @@ -948,7 +948,7 @@ def _add_common(parser: argparse.ArgumentParser, positive_int) -> None: "Unset by default so that passing the default value explicitly " "still overrides a --tier cap") parser.add_argument("--all-atoms", action="store_true", dest="all_atoms", - help="Extract every atom (opt-in: this is the expensive sweep)") + help="Probe every extracted atom (opt-in: this is the expensive sweep)") parser.add_argument("--budgets", type=float, nargs="+", default=[0.75, 0.5], help="Target token fractions for shortening candidates") parser.add_argument("--cache-dir", default=None, dest="cache_dir", @@ -1159,7 +1159,7 @@ def _print_coverage(bundle: dict, extraction: dict | None = None) -> None: The share is stated on stdout because the headline is what a user copies. A run under the floor also says so on stdout: a stderr-only warning is one a user piping the headline never sees, and a rate quoted off half-read skill is not a property of it. - A capped run says the words went into the input, and how far its atoms reach. + A capped run says the words went into the input, and how many of its atoms were probed. """ from dokimi.novelty.bundle import MEASURED_SHARE_WARN_AT, Coverage from dokimi.novelty.report import coverage_reach, floored_percent diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index 4621f1b64..158708524 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -180,11 +180,10 @@ def analyze( # From what this run did, not the tier's text: an override or an atom set under the # cap changes which claims are forfeit. warnings.extend(settings.warnings(n_atoms=len(atom_list))) - sampling = ( - f"cap-per-artifact:{config.max_atoms}" if config.max_atoms is not None else "full" - ) - if extraction.reached and config.max_atoms is not None: # reached is set only under a cap - warnings.append(report.cap_warning(config.max_atoms, extraction.reached)) + # Older runs recorded `cap-per-artifact:`, the first n atoms in document order (#293). + sampling = f"cap-spread:{config.max_atoms}" if config.max_atoms is not None else "full" + if extraction.sampled: # set only when the cap left atoms out + warnings.append(report.cap_warning(extraction.sampled)) if not atom_list: warnings.append("no atoms with span provenance were extracted; nothing to measure") diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index f4bf27991..208412c7e 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -193,14 +193,13 @@ class ExtractionReport: cut-off ones are cured by ``--max-tokens``. Kept for the warning. calls: Extraction calls made, splits included, so the cost of re-asking is visible. batches: Calls the same run would make with no splitting: a batch counts once it - is asked, and its halves count at its level, so a run the cap stopped early - counts only the batches it reached. + is asked, and its halves count at its level. A capped run asks every batch. parse_error: The last parse failure on a span that stayed unread. call_error: The last failed call on a span that stayed unread. Errors that splitting recovered from are not recorded: they did not lose anything, and quoting one would blame the wrong cause. - reached: How far through the input a capped atom set reached (see - :func:`_cap_reach`); ``None`` when the cap did not bind. + sampled: What a binding cap kept (see :func:`sample_record`); ``None`` when every + extracted atom was kept. """ unread_span_ids: list[str] = field(default_factory=list) @@ -215,7 +214,7 @@ class ExtractionReport: parse_error: str = "" call_error: str = "" unusable_error: str = "" - reached: dict | None = None + sampled: dict | None = None @property def last_error(self) -> str: @@ -291,7 +290,9 @@ def as_dict(self) -> dict: "calls": self.calls, "batches": self.batches, "last_error": self.last_error, - "reached": self.reached, + # How far a document-order cap reached (#299); a spread sample has no reach. + "reached": None, + "sampled": self.sampled, } @@ -308,11 +309,11 @@ def extract_atoms( A batch whose reply does not parse, or parses to rows none of which is a usable atom (see :func:`_atom_from_row`), is split in half and re-asked, down to one span per call; the halves go back to the front of the queue, so atoms keep document order - (which is what ``cap`` cuts on). The common cause is an output ceiling: a provider + (which is what ``cap`` strides over). The common cause is an output ceiling: a provider that defaults to 4096 output tokens truncates a 25-span reply mid-array, and a truncated array does not parse. Splitting recovers without knowing the ceiling, at a cost of up to 2n-1 calls for an n-span batch that never parses; ``report.calls`` - counts them, against ``report.batches`` for the batches the run reached. A call that + counts them, against ``report.batches`` for the batches of the input. A call that raises (transport, auth) or returns nothing is not split, because re-asking smaller would only multiply it. @@ -321,14 +322,15 @@ def extract_atoms( client: Extractor client (must not be the judge — see :mod:`.measure`). batch_spans: Spans per extraction call. Batching is what keeps extraction affordable; too large and the model starts skipping spans. - cap: Stop after this many atoms. Recorded in the artifact as the sampling + cap: Keep this many atoms, spread over the whole input (:func:`spread_sample`); + every span is extracted either way. Recorded in the artifact as the sampling rule, so a capped run is never quoted as a full sweep. report: Filled in place with what could not be read and what re-asking cost, - and how far a binding cap reached (``reached``). Spans no call could read - are also marked ``unread`` in place. + and what a binding cap kept (``sampled``). Spans no call could read are also + marked ``unread`` in place. Returns: - Atoms in extraction order, de-duplicated by id. + Atoms in extraction order, de-duplicated by id: all of them, or the cap's sample. """ from dokimi.llm import extract_json_array @@ -346,7 +348,6 @@ def parse(reply: str) -> list: for span in spans: span.unread = False seen: set[str] = set() - sent: set[str] = set() # span ids some call was shown, which bound the reach atoms: list[Atom] = [] lost: Exception | None = None # (spans, is a half): a half was counted in ``batches`` with the batch it came from. @@ -355,10 +356,7 @@ def parse(reply: str) -> list: for start in range(0, len(spans), batch_spans) ] while pending: - if cap is not None and len(atoms) >= cap: - break batch, is_half = pending.pop(0) - sent.update(span.id for span in batch) if not is_half: report.batches += 1 prompt = _EXTRACT_PROMPT.format( @@ -408,45 +406,33 @@ def parse(reply: str) -> list: continue seen.add(atom.id) atoms.append(atom) - if cap is not None and len(atoms) >= cap: - break if not atoms and lost is not None and all(span.unread for span in spans): # Only when nothing was read: spans that were read and asserted nothing are a # result, and a recovered parse error is not a failure. raise ExtractionError(f"no atoms: every extraction call failed ({lost})") from lost - report.reached = _cap_reach(spans, atoms, cap, sent) - return atoms - - -def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None, - sent: set[str]) -> dict | None: - """How far through the input (``spans``) a capped atom set reached; ``None`` under the cap. - - Extraction walks the spans in document order and stops at the cap, so a capped set - covers the input up to the span it stopped at, not a sample of it. The input is the text segmented, - not the artifact: a bundled file it left out counts in no span. The reach is the last span any - kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole, - but its later spans gave no atom. Only the ``sent`` spans count: an atom may cite a span - of the document its call was never shown, and that made-up id must not move the reach - (an atom set citing nothing sent reaches the last span sent). ``spans`` counts the spans - up to and including it, ``of`` all of them, and ``tokens_share`` is their share of the - span tokens. + report.sampled = sample_record(len(atoms), cap) + return spread_sample(atoms, cap) + + +def spread_sample(atoms: list, cap: int | None) -> list: + """The ``cap`` atoms a capped run probes: every (M/n)-th of the M extracted, in their order. + + Index ``floor((i + 0.5) * M / n)`` for ``i`` in ``0..n-1``, so the sample starts at the + middle of the first stride and estimates the per-atom rate an uncapped run reports. A + pure function of the list, with no seed: the same atoms give the same sample. With no + cap, or ``M <= n``, every atom. https://github.com/intel/dokimi/issues/293 """ - if cap is None or not atoms or len(atoms) < cap: + if cap is None or len(atoms) <= cap: + return list(atoms) + return [atoms[(2 * i + 1) * len(atoms) // (2 * cap)] for i in range(cap)] + + +def sample_record(n_extracted: int, cap: int | None) -> dict | None: + """``extraction.sampled``: the cap's ``atoms`` of the ``of`` extracted, and the rule; ``None`` + when the cap kept every atom.""" + if cap is None or n_extracted <= cap: return None - order = {span.id: index for index, span in enumerate(spans) if span.id in sent} - last = max((order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order), - default=max(order.values())) - reached = sum(span.tokens for span in spans[:last + 1]) - total = sum(span.tokens for span in spans) - # Floored in integers, as floored_percent prints it: a cap one span short of the end must not round to 1.0 - share = reached * 10000 // total - return { - "last_span": spans[last].id, - "spans": last + 1, - "of": len(spans), - "tokens_share": (max(share, 1) if reached else 0) / 10000, - } + return {"atoms": cap, "of": n_extracted, "rule": "cap-spread"} def _mark_unread(batch: list[Span], report: ExtractionReport) -> None: diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index b1c7d5a4b..1882f9dd4 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -257,21 +257,31 @@ def reach_is_end(reached: dict) -> bool: return reached["spans"] == reached["of"] -def cap_warning(cap: int, reached: dict) -> str: - """The warning for a binding cap: where extraction stopped, and what that leaves out.""" - left_out = ("any atoms past the cap in its last reply were dropped" if reach_is_end(reached) - else "rates describe the opening of the input, not all of it") - return (f"atom extraction hit the cap of {cap} at {describe_reach(reached)}: {left_out}; " - "pass --all-atoms for the full sweep") +def describe_sample(sampled: dict) -> str: + """What a binding cap probed, as ``25 of the input's 61 atoms, spread over all of it (a + stride of 2.44 atoms in document order)``: ``extraction.sampled`` of a ``cap-spread`` run.""" + stride = sampled["of"] / sampled["atoms"] + return (f"{sampled['atoms']} of the input's {sampled['of']} atoms, spread over all of it " + f"(a stride of {stride:.3g} atoms in document order)") + + +def cap_warning(sampled: dict) -> str: + """The warning for a binding cap: which atoms it probed, and how to probe them all.""" + return (f"atom extraction read the whole input, and the atom cap probes " + f"{describe_sample(sampled)}: rates are estimated from that sample; pass " + "--all-atoms to probe every atom") def coverage_reach(extraction: dict | None) -> tuple[str, str]: """The verb and tail of the headline's coverage line, from the ``extraction`` record. - The coverage share counts the words that went into the input. Under a binding cap the - atoms come from those words up to the span the cap stopped at, so the line must not say - they were measured, and it says how far the atoms reach. + The coverage share counts the words that went into the input. Under a binding cap only + the cap's atoms were probed, so the line must not say the words were measured: it says + which atoms were (``sampled``), or for a run before #293, how far the atoms reach. """ + sampled = (extraction or {}).get("sampled") + if sampled: + return "went into the input", f"; the cap probed {describe_sample(sampled)}" reached = (extraction or {}).get("reached") if not reached: return "were measured", "" @@ -283,7 +293,12 @@ def coverage_reach(extraction: dict | None) -> tuple[str, str]: def _capped_at(artifact: dict) -> str: - """The report header's sentence on where a binding cap stopped extraction, or ``""``.""" + """The report header's sentence on what a binding cap probed (or, before #293, where it + stopped extraction), or ``""``.""" + sampled = (artifact.get("extraction") or {}).get("sampled") + if sampled: + return (f" Extraction read the whole input; the atom cap probed " + f"{describe_sample(sampled)}.") reached = (artifact.get("extraction") or {}).get("reached") if not reached: return "" diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 052481887..7f0c1e9ca 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -24,8 +24,8 @@ from dataclasses import dataclass, field from typing import Any -#: What a run gives up, keyed by the setting that is off. "sampled" is a reached atom cap; the -#: name is old, the cap takes atoms in document order: https://github.com/intel/dokimi/pull/299 +#: What a run gives up, keyed by the setting that is off. "sampled" is a reached atom cap, which +#: probes atoms spread over the input: https://github.com/intel/dokimi/issues/293 FORFEITS: dict[str, list[str]] = { "sampled": [ "`added` is a rate over the atoms the cap kept, not the whole input: usable as a " @@ -69,8 +69,8 @@ def forfeits(settings: dict, *, n_atoms: int | None = None) -> list[str]: out = list(FORFEITS["sampled"]) if capped else [] if capped and settings.get("shorten_artifact"): out.append( - "shortening candidates are scored only against the capped atoms, i.e. the " - "opening of the artifact: a cut of the uncovered tail looks free" + f"shortening candidates are scored against a spread sample of {cap} of the " + "extracted atoms: a cut of a span with no sampled atom looks free" ) for setting, lines in FORFEITS.items(): if setting != "sampled" and not settings.get(setting): @@ -212,10 +212,10 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, The atom count is the larger of one atom per two spans and one per :data:`TOKENS_PER_ATOM` tokens (spans alone undercount: short spans, such as table - rows, yield an atom each), capped at ``max_atoms``. Without ``n_tokens`` only the - span rule applies. The forfeits assume the cap is reached, since the atom count is - a guess until extraction runs: an artifact with fewer atoms than the cap drops the - cap's forfeits. + rows, yield an atom each), capped at ``max_atoms`` for every stage but extraction, + which reads the whole input. Without ``n_tokens`` only the span rule applies. The + forfeits assume the cap is reached, since the atom count is a guess until extraction + runs: an artifact with fewer atoms than the cap drops the cap's forfeits. """ cap = self.values["max_atoms"] expected = max(1, n_spans // 2) @@ -229,7 +229,7 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, # The full resolved settings, so the projection is the model `estimate_calls` # implements rather than a subset of it. "estimate": estimate_calls(n_atoms, dict(self.values, budgets=tuple(budgets)), - n_arms=n_arms), + n_arms=n_arms, n_extracted=expected), "tier": self.tier_record(), "caveat": ( "forfeits assume the atom cap is reached; the run recomputes them from the " @@ -372,7 +372,8 @@ def estimate_usd( return round(total, 6) -def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2) -> dict: +def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2, + n_extracted: int | None = None) -> dict: """Project the model-call bill before spending it. From the instrumented runs: every probe, reversal included, costs an answer and a grade @@ -390,6 +391,8 @@ def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2) -> dict: n_atoms: Atoms expected after the cap. settings: Resolved tier settings (``paraphrases``, ``reversals``, stage switches). n_arms: Arms in the run, including the baseline. + n_extracted: Atoms expected before the cap (default ``n_atoms``). Extraction asks + every batch whatever the cap, so its calls follow this count. Returns: ``{"n_atoms", "paired_probes", "reversal_probes", "probe_calls", "stage_calls", @@ -408,8 +411,9 @@ def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2) -> dict: # Reversals are graded in every arm too: the baseline arm's resistance to a planted # inversion is a reported control, not a by-product. probe_calls = (paired + reversals) * n_arms * 2 + extracted = n_atoms if n_extracted is None else n_extracted stage_calls = ( - max(1, round(n_atoms / 12)) # extraction batches + max(1, round(extracted / 12)) # extraction batches + (n_atoms if settings.get("verify_faithfulness") else 0) # one per atom + (1 if settings.get("paraphrases") else 0) + (1 if settings.get("reversals") else 0) diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index 9bd471137..c1c909304 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -71,7 +71,7 @@ }, "sampling": { "type": "string", - "description": "How the probed subset was chosen (e.g. cap-per-artifact:120). Present so a capped run is never read as a full sweep." + "description": "How the probed subset was chosen: full, or cap-spread: (e.g. cap-spread:120), n atoms spread over every extracted atom in document order (extraction.sampled). Runs before cap-spread wrote cap-per-artifact:, the first n atoms in document order. Present so a capped run is never read as a full sweep." }, "faithfulness": { "type": ["object", "null"], @@ -270,7 +270,7 @@ }, "conflict_coverage": { "type": "object", - "description": "How much of the atom-pair space the conflict check compared. pairs_compared counts pairs that shared an answered call; pairs_total is every pair among the extracted atoms, which is the capped sample when spans.capped_atom_set is true. pairs_compared <= pairs_total is not expressible here and is enforced by the gate. A run that left pairs_compared below pairs_total, or capped the atom set, does not clear harmful_atoms at a count of 0.", + "description": "How much of the atom-pair space the conflict check compared. pairs_compared counts pairs that shared an answered call; pairs_total is every pair among the probed atoms, which is the capped sample when spans.capped_atom_set is true. pairs_compared <= pairs_total is not expressible here and is enforced by the gate. A run that left pairs_compared below pairs_total, or capped the atom set, does not clear harmful_atoms at a count of 0.", "additionalProperties": false, "required": ["pairs_compared", "pairs_total"], "properties": { @@ -334,11 +334,11 @@ }, "retried_batches": {"type": "integer", "minimum": 0, "description": "Batches whose reply did not parse, or had no usable atom, and were split and re-asked."}, "calls": {"type": "integer", "minimum": 0, "description": "Extraction calls made, splits included. An n-span batch that never parses costs up to 2n-1 calls."}, - "batches": {"type": "integer", "minimum": 0, "description": "Calls the same extraction would make with no splitting: batches the run reached (a capped run stops early), each counted once whatever its halves cost."}, + "batches": {"type": "integer", "minimum": 0, "description": "Calls the same extraction would make with no splitting: every batch of the input (a capped run before cap-spread stopped early and counted only the batches it reached), each counted once whatever its halves cost."}, "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."}, "reached": { "type": ["object", "null"], - "description": "How far through the input a capped atom set reached; null when the cap did not bind. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). Extraction walks the spans in document order and stops at the cap, so a capped set covers the input up to the span it stopped at, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", + "description": "Written by runs before cap-spread (atoms.sampling cap-per-artifact:): how far through the input a capped atom set reached; null when the cap did not bind. A cap-spread run extracts every span and writes null here, and says what it probed in sampled. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). On those older runs extraction walked the spans in document order and stopped at the cap, so a capped set covers the input up to the span it stopped at, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", "properties": { "last_span": {"type": "string"}, "spans": {"type": "integer", "minimum": 1}, @@ -346,6 +346,16 @@ "tokens_share": {"type": "number", "minimum": 0, "maximum": 1} }, "required": ["last_span", "spans", "of", "tokens_share"] + }, + "sampled": { + "type": ["object", "null"], + "description": "What a binding cap probed (atoms.sampling cap-spread:): atoms of the of atoms extracted from every span, at indices floor((i + 0.5) * of / atoms) for i in 0..atoms-1 of the extracted list (document order; on a bundle run, the files in bundle order). No seed: the same extracted list gives the same sample. null when the cap kept every atom. Absent from artifacts written before it existed.", + "properties": { + "atoms": {"type": "integer", "minimum": 0}, + "of": {"type": "integer", "minimum": 1}, + "rule": {"type": "string"} + }, + "required": ["atoms", "of", "rule"] } } }, diff --git a/dokimi/research/records/decision/D017.md b/dokimi/research/records/decision/D017.md new file mode 100644 index 000000000..2a91e58a2 --- /dev/null +++ b/dokimi/research/records/decision/D017.md @@ -0,0 +1,21 @@ +--- +id: D017 +type: decision +date: '2026-11-01' +title: A capped novelty run probes atoms spread over the input +ideas: +- I005 +state: accepted +links: +- https://github.com/intel/dokimi/issues/293 +--- + +## Rationale +A cap that takes atoms in document order measures the opening of the input, not the input. At `--tier triage` on the dpnp repository's agent guidance, the 25 capped atoms of the root `AGENTS.md` (61 spans) came from spans s0002 to s0012, about a quarter of its tokens, and those of `dpnp/tests/AGENTS.md` (32 spans) from s0003 to s0013, about a third. Two artifacts' capped rates then describe different shares of their text, and a file's rate moves when its sections are reordered. + +A capped run now extracts every span and, of the M atoms extracted, probes the n at indices `floor((i + 0.5) * M / n)`, in document order (on a bundle run, in file order). A per-atom stride estimates the same per-atom rate an uncapped run reports. Equal token strata would not: they would weight a sparse introduction like a dense rule list. The stride needs no seed, so the same text gives the same sample. Probing stays at the cap; extraction grows to every batch of the input. + +The tier profiles (the call and token figures `--estimate-only` is fitted and checked on, and the per-tier call counts in `docs/skill-novelty.md`) were measured with the document-order cap. Only artifacts with more atoms than a tier's cap (25 at triage, 60 at standard, 120 at study) are affected; below the cap every atom is probed, as before. Re-measuring those artifacts, and refitting the extraction profile to a run that asks every batch, is follow-up work this decision does not do. diff --git a/dokimi/scripts/growth_baseline.json b/dokimi/scripts/growth_baseline.json index d751c4643..8721f233e 100644 --- a/dokimi/scripts/growth_baseline.json +++ b/dokimi/scripts/growth_baseline.json @@ -49,7 +49,7 @@ "dokimi/harnesses/tasks.py::harbor_task_problem": {"complexity": 11}, "dokimi/mcp/context7.py::Context7Client.get_library_docs": {"complexity": 12}, "dokimi/novelty/analyze.py::analyze": {"complexity": 22}, - "dokimi/novelty/atoms.py::extract_atoms": {"complexity": 16}, + "dokimi/novelty/atoms.py::extract_atoms": {"complexity": 14}, "dokimi/novelty/bundle.py::load_bundle": {"complexity": 14}, "dokimi/novelty/items.py::discrimination": {"complexity": 11}, "dokimi/novelty/items.py::load_probe_cells": {"complexity": 11, "rationale": "pre-existing when the gate was introduced"}, diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index c814879bc..6284a725d 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -614,48 +614,71 @@ def test_a_tier_writes_its_forfeits_into_the_run_it_produced(tmp_path): validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) -def test_a_capped_run_says_how_far_through_the_artifact_the_cap_reached(tmp_path): - # One extraction batch sends all five spans; the cap keeps the first atom, from s0002. - artifact = _analyze(tmp_path, max_atoms=1).artifact - # s0001 and s0002 are 15 of the body's 42 span tokens. - reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} - assert artifact["extraction"]["reached"] == reached - warning = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " - "of its tokens): rates describe the opening of the input, not all of it; pass " - "--all-atoms for the full sweep") +def test_a_capped_run_probes_a_spread_sample_and_says_so(tmp_path, monkeypatch): + # One batch, two atoms (s0002, s0003); a cap of 1 keeps the middle of its one stride, the + # second, where the document-order cap kept the first. + from dokimi.novelty import shorten + + scored: list = [] + build = shorten.build_candidates + + def recording(span_list, atom_list, **kwargs): + scored.append([atom.span_ids for atom in atom_list]) + return build(span_list, atom_list, **kwargs) + + monkeypatch.setattr(shorten, "build_candidates", recording) + result = _analyze(tmp_path, max_atoms=1) + artifact = result.artifact + assert [atom.span_ids for atom in result.atoms] == [("s0003",)] + # Shortening candidates are judged against the sampled atom only. + assert scored == [[("s0003",)]] + assert artifact["atoms"]["sampling"] == "cap-spread:1" + assert artifact["extraction"]["sampled"] == {"atoms": 1, "of": 2, "rule": "cap-spread"} + assert artifact["extraction"]["reached"] is None + warning = ("atom extraction read the whole input, and the atom cap probes 1 of the input's 2 " + "atoms, spread over all of it (a stride of 2 atoms in document order): rates are " + "estimated from that sample; pass --all-atoms to probe every atom") assert warning in artifact["warnings"] - assert not any("sampled atoms" in w for w in artifact["warnings"]) - assert ("1 atoms (cap-per-artifact:1), 1 probed. The atom cap stopped extraction at span " - "s0002 (2 of the input's 5 spans, 35% of its tokens): the atoms come from the " - "opening of the input, not all of it.") in render_novelty_markdown(artifact) + assert not any("opening" in w for w in artifact["warnings"]) + assert ("1 atoms (cap-spread:1), 1 probed. Extraction read the whole input; the atom cap " + "probed 1 of the input's 2 atoms, spread over all of it (a stride of 2 atoms in " + "document order).") in render_novelty_markdown(artifact) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + # The renderers read every key of the sample, so a partial one does not validate. + partial = dict(artifact["extraction"], sampled={"atoms": 1}) + with pytest.raises(ArtifactValidationError, match="extraction/sampled: 'of' is a required"): + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=partial)) + + +def test_an_artifact_written_under_the_document_order_cap_still_validates(tmp_path): + artifact = _analyze(tmp_path, max_atoms=1).artifact + old = {key: value for key, value in artifact["extraction"].items() if key != "sampled"} + old["reached"] = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=old, + atoms=dict(artifact["atoms"], sampling="cap-per-artifact:1"))) # The renderers read every key of the reach, so a partial one does not validate. - partial = dict(artifact["extraction"], reached={"spans": 3}) + partial = dict(old, reached={"spans": 3}) with pytest.raises(ArtifactValidationError, match="extraction/reached: 'last_span' is a required"): validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=partial)) -def test_the_reach_is_null_only_when_the_cap_did_not_bind(tmp_path): - # A cap the last atom meets binds, and reaches that atom's span; one above it does not. +def test_a_cap_the_atoms_meet_samples_nothing(tmp_path): + # Two atoms under a cap of 2: every atom is probed, exactly as with no cap. artifact = _analyze(tmp_path, max_atoms=2).artifact - assert artifact["atoms"]["n"] == 2 and artifact["spans"]["capped_atom_set"] is True - assert artifact["extraction"]["reached"]["last_span"] == "s0003" - uncapped = _analyze(tmp_path / "again", max_atoms=3).artifact - assert uncapped["extraction"]["reached"] is None - assert not any("hit the cap" in w for w in uncapped["warnings"]) - assert "atom cap stopped" not in render_novelty_markdown(uncapped) - validate_artifact("novelty", dict(uncapped, schema_version="novelty.v1")) - - -@pytest.mark.parametrize("cap, header, tail", [ - (2, "s0002 (2 of the input's 8 spans, 25% of its tokens): the atoms come from the opening " - "of the input, not all of it.", "come from the first 25% of the input's tokens"), - (8, "s0008 (8 of the input's 8 spans, 100% of its tokens): the atoms reach the input's last " - "span, and the cap dropped any atoms past it in the last reply.", - "reach the input's last span, and the cap dropped any atoms past it"), -], ids=["mid-input", "last-span"]) -def test_a_capped_bundle_run_reaches_through_its_input_not_the_artifact(tmp_path, capsys, cap, - header, tail): + assert artifact["atoms"]["n"] == 2 and artifact["atoms"]["sampling"] == "cap-spread:2" + assert artifact["extraction"]["sampled"] is None + assert not any("atom cap probes" in w for w in artifact["warnings"]) + assert "atom cap" not in render_novelty_markdown(artifact) + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + + +@pytest.mark.parametrize("cap, probed, by_file", [ + # 8 atoms, one per span; SKILL.md holds s0001-s0005, references/tuning.md s0006-s0008. + (2, ["s0003", "s0007"], {"SKILL.md": 1, "references/tuning.md": 1}), + (3, ["s0002", "s0005", "s0007"], {"SKILL.md": 2, "references/tuning.md": 1}), +], ids=["cap-2", "cap-3"]) +def test_a_capped_bundle_run_strides_across_its_files_in_file_order(tmp_path, capsys, cap, probed, + by_file): from dokimi.commands import novelty as novelty_cmd from dokimi.novelty.bundle import load_bundle @@ -664,7 +687,7 @@ def test_a_capped_bundle_run_reaches_through_its_input_not_the_artifact(tmp_path (root / "SKILL.md").write_text(SKILL, encoding="utf-8") (root / "references" / "tuning.md").write_text( "# Tuning\n\nSet `--threads` to the core count.\n", encoding="utf-8") - # 500 words over the 100-word file budget: on disk, in no span the reach counts. + # 500 words over the 100-word file budget: on disk, in no span. (root / "references" / "zlarge.md").write_text( " ".join(f"word{i}" for i in range(500)), encoding="utf-8") bundle = load_bundle(f"skill:{root}", include_resources=True, max_file_words=100) @@ -685,15 +708,17 @@ def per_span(prompt, system): ) _, answerer, judge = _clients() extractor = LLMClient(model="extractor-m", role="extractor", transport=per_span) - artifact = analyze(bundle.text, config, extractor=extractor, answerer=answerer, - judge=judge, artifact_bundle=bundle).artifact - assert artifact["extraction"]["reached"]["of"] == 8 - assert f"The atom cap stopped extraction at span {header}" in render_novelty_markdown(artifact) + result = analyze(bundle.text, config, extractor=extractor, answerer=answerer, + judge=judge, artifact_bundle=bundle) + artifact = result.artifact + assert [atom.span_ids[0] for atom in result.atoms] == probed + assert {row["source"]: row["n_atoms"] for row in artifact["bundle"]["by_source"]} == by_file + assert artifact["extraction"]["sampled"] == {"atoms": cap, "of": 8, "rule": "cap-spread"} novelty_cmd._print_coverage(artifact["bundle"], artifact["extraction"]) - # 6% of the artifact went into the input; the reach is a share of the input alone. + # 6% of the artifact went into the input; the sample is of the input's atoms. assert capsys.readouterr().out.startswith( - "coverage: 6% of the artifact's readable words went into the input; the capped atoms " - f"{tail}\n") + "coverage: 6% of the artifact's readable words went into the input; the cap probed " + f"{cap} of the input's 8 atoms, spread over all of it") def test_skipping_the_faithfulness_check_is_recorded_as_a_warning(tmp_path): diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 8851908e5..4802d9193 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -7,13 +7,11 @@ import json import re -from types import SimpleNamespace import pytest -from dokimi.novelty.atoms import ExtractionError, ExtractionReport, _cap_reach, extract_atoms +from dokimi.novelty.atoms import ExtractionError, ExtractionReport, extract_atoms, spread_sample from dokimi.novelty.client import LLMClient -from dokimi.novelty.report import floored_percent from dokimi.novelty.spans import segment DOC = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 11)) @@ -198,7 +196,7 @@ def starved_first(prompt, system): assert report.as_dict()["last_error"] == report.last_error -def test_a_capped_extraction_counts_only_the_batches_it_reached(): +def test_a_capped_extraction_sends_every_batch(): """200 spans, 3 atoms each; every 25-span reply is cut and both halves parse.""" doc = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 201)) spans = segment(doc) @@ -215,12 +213,33 @@ def three_per_span_cut_above_13(prompt, system): client = LLMClient(model="x", transport=three_per_span_cut_above_13) atoms = extract_atoms(spans, client=client, cap=120, report=report) assert len(atoms) == 120 - # Two 25-span batches reached the cap: each cost one cut call and two halves. - assert (report.calls, report.batches, report.retried_batches) == (6, 2, 2) + # All eight 25-span batches were asked, though the first two held 150 atoms: each cost + # one cut call and two halves. + assert (report.calls, report.batches, report.retried_batches) == (24, 8, 8) assert ( - "atom extraction re-asked 2 unparseable batches as halves: 6 extraction calls where " - "2 would do without splitting; a higher --max-tokens avoids the extra calls" + "atom extraction re-asked 8 unparseable batches as halves: 24 extraction calls where " + "8 would do without splitting; a higher --max-tokens avoids the extra calls" ) in report.warnings(len(spans)) + # The 120 kept are spread over all 600, one per 5, from the middle of the first stride. + assert [a.span_ids[0] for a in atoms[:3]] == ["s0001", "s0003", "s0005"] + assert atoms[-1].span_ids[0] == "s0200" + + +# The issue's root AGENTS.md: 61 atoms under the triage cap of 25. +SPREAD_61_25 = [1, 3, 6, 8, 10, 13, 15, 18, 20, 23, 25, 28, 30, 32, 35, 37, 40, 42, 45, 47, + 50, 52, 54, 57, 59] + + +@pytest.mark.parametrize("n_atoms, cap, kept", [ + (61, 25, SPREAD_61_25), + (61, 1, [30]), # one atom: the middle one, not the first + (25, 25, list(range(25))), # a cap the atoms meet exactly keeps them all + (10, 25, list(range(10))), + (10, None, list(range(10))), +], ids=["61-of-25", "cap-1", "at-cap", "under-cap", "no-cap"]) +def test_a_cap_keeps_every_kth_atom_from_the_middle_of_the_first_stride(n_atoms, cap, kept): + """``floor((i + 0.5) * M / n)``: the stride rule of https://github.com/intel/dokimi/issues/293.""" + assert spread_sample([f"atom{i}" for i in range(n_atoms)], cap) == [f"atom{i}" for i in kept] def _three_per_span(prompt, system): @@ -230,60 +249,47 @@ def _three_per_span(prompt, system): return json.dumps(rows), None -def test_a_capped_extraction_records_the_last_span_its_atoms_reached(tmp_path): - """60 spans, 3 atoms each, 10 per batch: the 25th atom is span 9's, mid-batch.""" +def test_a_capped_extraction_reads_every_span_and_records_the_sample(tmp_path): + """60 spans, 3 atoms each, 10 per batch: a cap of 25 keeps 25 of 180, from every part.""" from dokimi.novelty.atoms import save_atoms doc = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61)) spans = segment(doc) + calls: list[str] = [] + + def counted(prompt, system): + calls.append(prompt) + return _three_per_span(prompt, system) + report = ExtractionReport() - atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=counted), batch_spans=10, cap=25, report=report) - assert len(atoms) == 25 and report.batches == 1 - # The batch sent s0001-s0010; the cap cut its reply inside s0009's atoms. - # Nine spans of 8 tokens, of 480 tokens in all: 72 / 480. - assert [s.tokens for s in spans[:9]] == [8] * 9 and sum(s.tokens for s in spans) == 480 - assert report.reached == {"last_span": "s0009", "spans": 9, "of": 60, "tokens_share": 0.15} + assert (len(calls), report.batches, report.calls) == (6, 6, 6) + assert len(atoms) == 25 + # Every batch gives sampled atoms, the first from s0002 (atom 3) and the last from s0059 (176). + assert {(int(a.span_ids[0][1:]) - 1) // 10 for a in atoms} == set(range(6)) + assert atoms[0].span_ids[0] == "s0002" and atoms[-1].span_ids[0] == "s0059" + assert report.sampled == {"atoms": 25, "of": 180, "rule": "cap-spread"} path = tmp_path / "atoms.json" save_atoms(atoms, spans, path, cap=25, extraction=report) - assert json.loads(path.read_text())["extraction"]["reached"] == report.reached - + saved = json.loads(path.read_text())["extraction"] + assert (saved["sampled"], saved["reached"]) == (report.sampled, None) + # Selection is a pure function of the extracted list: a second run picks the same atoms. + again = extract_atoms(segment(doc), client=LLMClient(model="x", transport=_three_per_span), + batch_spans=10, cap=25) + assert [a.id for a in again] == [a.id for a in atoms] -def test_a_span_id_no_call_was_shown_does_not_move_the_reach(): - """The cap binds on the first batch's last span (s0010); one atom also cites s0055.""" - def made_up(prompt, system): - # Two atoms for each of s0001-s0005 and three for s0006-s0010: 25 in all. - rows = [dict(row, statement=f"{row['statement']} ({k})", question=f"q{k} {row['span_ids'][0]}") - for row in _rows(prompt) for k in range(2 if row["span_ids"][0] <= "s0005" else 3)] - rows[4]["span_ids"] = ["s0003", "s0055"] - return json.dumps(rows), None - spans = segment("\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61))) - report = ExtractionReport() - atoms = extract_atoms(spans, client=LLMClient(model="x", transport=made_up), - batch_spans=10, cap=25, report=report) - assert "s0055" in atoms[4].span_ids and report.batches == 1 - assert report.reached == {"last_span": "s0010", "spans": 10, "of": 60, - "tokens_share": 0.1666} - # An atom set that cites nothing it was shown reaches the last span sent. - first_ten = {span.id for span in spans[:10]} - assert _cap_reach(spans, [SimpleNamespace(span_ids=["s0055"])], 1, first_ten)["spans"] == 10 - - -@pytest.mark.parametrize("cap, reached", [ - (None, None), - (31, None), - # Met on the last span: still a capped set, and its reach is the whole artifact. - (30, {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0}), -]) -def test_an_extraction_the_cap_did_not_stop_reaches_nothing_or_the_end(cap, reached): +@pytest.mark.parametrize("cap", [None, 31, 30], ids=["no-cap", "above", "at-cap"]) +def test_a_cap_the_atoms_do_not_exceed_keeps_them_all_and_records_no_sample(cap): """No cap, a cap above the 30 atoms, and a cap the last atom meets exactly.""" spans = segment(DOC) report = ExtractionReport() atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), cap=cap, report=report) assert len(atoms) == 30 and len(spans) == 10 - assert report.reached == report.as_dict()["reached"] == reached + assert report.sampled is None + assert (report.as_dict()["sampled"], report.as_dict()["reached"]) == (None, None) def _half_lost(n: int = 50, silent: frozenset[int] = frozenset()): @@ -567,12 +573,33 @@ def transport(prompt, system): ) -@pytest.mark.parametrize("first, share, shown", [ - (29_999, 0.9999, "99%"), # one span short of the end: floored, never rounded up to 100% - (1, 0.0001, "<1%"), # a reach above zero is never 0% -]) -def test_a_cap_reach_share_is_floored_not_rounded(first, share, shown): - spans = [SimpleNamespace(id="s0001", tokens=first), SimpleNamespace(id="s0002", tokens=30_000 - first)] - reach = _cap_reach(spans, [SimpleNamespace(span_ids=["s0001"])], cap=1, sent={"s0001", "s0002"}) - assert reach == {"last_span": "s0001", "spans": 1, "of": 2, "tokens_share": share} - assert floored_percent(reach["tokens_share"]) == shown +def test_novelty_atoms_saves_the_same_spread_sample_analyze_probes(tmp_path, monkeypatch, capsys): + """Two batches of 25 spans, one atom each: a cap of 2 keeps s0013 and s0038 for probe and shorten.""" + from tests import test_novelty_cli as novelty_cli + + skill = tmp_path / "s" / "SKILL.md" + skill.parent.mkdir(parents=True) + skill.write_text("---\nname: s\ndescription: d\n---\n\n" + "\n\n".join( + f"Setting {i} defaults to {i}." for i in range(1, 51)) + "\n", encoding="utf-8") + batches: list = [] + + def transport(prompt, system=None): + ids = re.findall(r"^\[(s\d+)\]", prompt, flags=re.M) + batches.append(ids) + return json.dumps([{"statement": f"{sid} holds.", "question": f"What of {sid}?", + "answer": "a value", "span_ids": [sid]} for sid in ids]), None + + real_client = novelty_cli.novelty_cmd._client + monkeypatch.setattr(novelty_cli.novelty_cmd, "_client", + lambda model, provider, role, args: novelty_cli._with_transport( + real_client(model, provider, role, args), transport)) + out = tmp_path / "atoms.json" + args = novelty_cli._parse("novelty", "atoms", "--artifact", f"skill:{skill}", "--out", + str(out), "--max-atoms", "2", "--no-verify") + args.func(args) + payload = json.loads(out.read_text()) + assert len(batches) == 2 # the cap no longer stops extraction after the first batch + assert [atom["span_ids"] for atom in payload["atoms"]] == [["s0013"], ["s0038"]] + assert payload["extraction"]["sampled"] == {"atoms": 2, "of": 50, "rule": "cap-spread"} + assert ("WARNING atom extraction read the whole input, and the atom cap probes 2 of the " + "input's 50 atoms") in capsys.readouterr().err diff --git a/dokimi/tests/test_novelty_cli.py b/dokimi/tests/test_novelty_cli.py index f78f6351e..0df58936b 100644 --- a/dokimi/tests/test_novelty_cli.py +++ b/dokimi/tests/test_novelty_cli.py @@ -875,6 +875,17 @@ def test_a_capped_run_s_headline_does_not_say_the_whole_artifact_was_measured(ca in capsys.readouterr().out +def test_a_spread_run_s_headline_says_how_many_atoms_the_cap_probed(capsys): + sampled = {"atoms": 25, "of": 61, "rule": "cap-spread"} + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=_readable_record(resource_words_on_disk=0), + extraction={"reached": None, "sampled": sampled})) + out = capsys.readouterr().out + assert ("coverage: 100% of the artifact's readable words went into the input (entry point " + "only); the cap probed 25 of the input's 61 atoms, spread over all of it (a stride of " + "2.44 atoms in document order)\n") in out + assert "measured" not in out and "opening" not in out + + def test_a_capped_run_under_the_floor_warns_in_the_coverage_line_s_words(capsys): # 10 of the 40 readable words went into the input, and the atoms stopped short of them. reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index 99881e1e2..3457319f8 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -6,7 +6,13 @@ import pytest from dokimi.gate.novelty_gate import uncleared_conflicts -from dokimi.novelty.report import _capped_at, cap_warning, coverage_reach, render_novelty_markdown +from dokimi.novelty.report import ( + _capped_at, + cap_warning, + coverage_reach, + floored_percent, + render_novelty_markdown, +) def _artifact(**overrides): @@ -72,27 +78,61 @@ def test_the_cost_line_names_replayed_calls_only_when_there_are_some(cost, line) assert line in render_novelty_markdown(_artifact(cost=cost)).splitlines() -def test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): +SAMPLED = {"atoms": 25, "of": 61, "rule": "cap-spread"} + + +def test_a_spread_run_says_how_many_atoms_it_probed_spread_over_the_whole_input(): + assert cap_warning(SAMPLED) == ( + "atom extraction read the whole input, and the atom cap probes 25 of the input's 61 atoms, " + "spread over all of it (a stride of 2.44 atoms in document order): rates are estimated from " + "that sample; pass --all-atoms to probe every atom") + assert coverage_reach({"reached": None, "sampled": SAMPLED}) == ( + "went into the input", "; the cap probed 25 of the input's 61 atoms, spread over all of it " + "(a stride of 2.44 atoms in document order)") + header = _capped_at({"extraction": {"reached": None, "sampled": SAMPLED}}) + assert header == (" Extraction read the whole input; the atom cap probed 25 of the input's 61 " + "atoms, spread over all of it (a stride of 2.44 atoms in document order).") + assert "opening" not in header + + +def test_an_uncapped_record_says_the_words_were_measured_and_adds_no_header(): + assert coverage_reach({"reached": None, "sampled": None}) == ("were measured", "") + assert _capped_at({"extraction": {"reached": None, "sampled": None}}) == "" + + +def test_an_old_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): + # A `cap-per-artifact` artifact, written before the spread rule: `reached`, no `sampled`. end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} assert coverage_reach({"reached": end}) == ( "went into the input", "; the capped atoms reach the input's last span, and the cap dropped any atoms " "past it") assert "opening" not in _capped_at({"extraction": {"reached": end}}) - assert cap_warning(30, end) == ( - "atom extraction hit the cap of 30 at span s0010 (10 of the input's 10 spans, 100% of its tokens): " - "any atoms past the cap in its last reply were dropped; pass --all-atoms for the full sweep") -def test_a_cap_that_stopped_inside_the_input_still_says_the_rates_describe_the_opening(): +def test_an_old_cap_that_stopped_inside_the_input_still_says_the_rates_describe_the_opening(): inside = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} assert coverage_reach({"reached": inside}) == ( "went into the input", "; the capped atoms come from the first 35% of the input's tokens") assert _capped_at({"extraction": {"reached": inside}}) == ( " The atom cap stopped extraction at span s0002 (2 of the input's 5 spans, 35% of its tokens): the " "atoms come from the opening of the input, not all of it.") - assert cap_warning(1, inside) == ( - "atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% of its tokens): rates " - "describe the opening of the input, not all of it; pass --all-atoms for the full sweep") + + +def test_an_old_artifact_with_a_reach_renders_its_header_as_before(): + artifact = _artifact(atoms={"n": 1, "n_probed": 1, "sampling": "cap-per-artifact:1"}, + extraction={"reached": {"last_span": "s0002", "spans": 2, "of": 5, + "tokens_share": 0.3571}}) + assert ("1 atoms (cap-per-artifact:1), 1 probed. The atom cap stopped extraction at span " + "s0002 (2 of the input's 5 spans, 35% of its tokens): the atoms come from the " + "opening of the input, not all of it.") in render_novelty_markdown(artifact) + + +@pytest.mark.parametrize("share, shown", [ + (0.9999, "99%"), # one span short of the end: floored, never rounded up to 100% + (0.0001, "<1%"), # a reach above zero is never 0% +]) +def test_an_old_reach_share_is_floored_not_rounded(share, shown): + assert floored_percent(share) == shown def test_a_cap_met_on_the_last_span_leaves_the_zero_uncleared_without_naming_an_opening(): diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 713ab052b..32bc21e4a 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -27,6 +27,7 @@ import io import json import os +import re import sys from decimal import ROUND_FLOOR, Decimal from pathlib import Path @@ -136,8 +137,9 @@ def _headline_coverage(text: str, work: Path) -> dict[str, Any]: printed = [line for line in text.splitlines() if line.startswith(COVERAGE_LINES)] record = work / "out" / "novelty.json" saved = json.loads(record.read_text(encoding="utf-8")) if record.exists() else {} + extraction = saved.get("extraction") or {} return {"printed": printed, "measured_share": (saved.get("bundle") or {}).get("measured_share"), - "reached": (saved.get("extraction") or {}).get("reached")} + "capped": bool(extraction.get("reached") or extraction.get("sampled"))} def _check_headline(coverage: dict[str, Any]) -> None: @@ -151,7 +153,7 @@ def _check_headline(coverage: dict[str, Any]) -> None: pct = f"{int(Decimal(str(share)).scaleb(2).to_integral_value(ROUND_FLOOR))}%" assert printed, "a run with a record must print its coverage line" # A capped run's words went into the input; its atoms do not cover all of them. - went = "went into the input" if coverage["reached"] else "were measured" + went = "went into the input" if coverage["capped"] else "were measured" assert printed[0].startswith( f"coverage: {pct} of the artifact's readable words {went}" ), printed @@ -215,6 +217,8 @@ def scripted(model, provider, role, args): "n_atoms", "total_calls", "total_calls_if_every_pair_escalates", + # Extraction asks every batch under a cap too (#293); tests/test_novelty_tiers.py pins it. + "stage_calls", ) @@ -259,20 +263,45 @@ def _refusal_as_exit_2(step: dict[str, Any]) -> dict[str, Any]: # atoms a sample. In the warnings and on stderr alike. CAP_WARNING_ON_MAIN = ("atom extraction hit the cap of 1; rates describe the sampled atoms, " "not the whole artifact") -CAP_WARNING = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " - "of its tokens): rates describe the opening of the input, not all of it; pass " - "--all-atoms for the full sweep") +# Intended change (#293): the cap probes atoms spread over the input, and the warning says so. +CAP_WARNING = ("atom extraction read the whole input, and the atom cap probes 1 of the input's 2 " + "atoms, spread over all of it (a stride of 2 atoms in document order): rates are " + "estimated from that sample; pass --all-atoms to probe every atom") # Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier # record, the warnings and the estimate's stderr. CAP_FORFEIT_ON_MAIN = "`added` is a sampled rate over the atom cap: usable" CAP_FORFEIT = "`added` is a rate over the atoms the cap kept, not the whole input: usable" +# Intended change (#293): shortening is scored against a spread sample, not the opening. The +# forfeit names the cap, which tests/test_novelty_tiers.py pins; here both sides read . +SHORTEN_FORFEIT_ON_MAIN = ("shortening candidates are scored only against the capped atoms, i.e. " + "the opening of the artifact: a cut of the uncovered tail looks free") +SHORTEN_FORFEIT = ("shortening candidates are scored against a spread sample of of the " + "extracted atoms: a cut of a span with no sampled atom looks free") + + +# Intended change (#293): a cap of 1 on two atoms probes the second (s0003, mid-stride), not the first, +# so what follows the atom (dead tokens, oracle calls, the warnings on it) is not main's. +SPREAD_RUNS = {"run/max-atoms-1", "run/triage-max-atoms-1"} + + +def _settings_only(step: dict) -> dict: + """What the flags decide in a run step: its status, run_config and tier and cap warnings.""" + return {"status": step["status"], "run_config": step.get("run_config"), + "warnings": [w for w in step.get("warnings", []) + if w.startswith("tier `") or w == CAP_WARNING]} + + +def _cap_named(steps: list) -> list: + return json.loads(re.sub(r"spread sample of \d+ of", "spread sample of of", + json.dumps(steps))) @pytest.mark.parametrize("name", list(CASES)) def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): golden = json.loads(json.dumps(json.loads(GOLDEN.read_text(encoding="utf-8"))) - .replace(CAP_FORFEIT_ON_MAIN, CAP_FORFEIT)) - steps = _capture(name, tmp_path) + .replace(CAP_FORFEIT_ON_MAIN, CAP_FORFEIT) + .replace(SHORTEN_FORFEIT_ON_MAIN, SHORTEN_FORFEIT)) + steps = _cap_named(_capture(name, tmp_path)) coverage = [step.pop("coverage") for step in steps] for step_coverage in coverage: if step_coverage is not None: @@ -292,6 +321,9 @@ def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): # Intended change: `novelty atoms` prints the cap warning `analyze` records. expected = [dict(step, stderr=f"{step['stderr']}WARNING {CAP_WARNING}\n") for step in expected] + if name in SPREAD_RUNS: + steps, expected = ([_settings_only(step) for step in side] for side in (steps, expected)) + assert any(CAP_WARNING in step["warnings"] for step in steps) assert steps == expected diff --git a/dokimi/tests/test_novelty_tiers.py b/dokimi/tests/test_novelty_tiers.py index 8133eee3d..0ef45da39 100644 --- a/dokimi/tests/test_novelty_tiers.py +++ b/dokimi/tests/test_novelty_tiers.py @@ -271,15 +271,18 @@ def test_flags_that_drop_the_probe_kinds_warn_without_a_tier(tmp_path): assert not any("no paraphrase probe" in w or "no reversal probe" in w for w in full) -def test_shortening_under_a_reached_cap_says_it_saw_only_the_opening(): +def test_shortening_under_a_reached_cap_says_it_saw_only_a_spread_sample(): from dokimi.novelty.tiers import forfeits on = {"max_atoms": 60, "shorten_artifact": True, "paraphrases": 1, "reversals": True, "verify_faithfulness": True, "classify_content": True, "confirm_carriers": True} - assert any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=60)) - assert not any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=10)) - assert not any("shortening candidates are scored only" in f for f in forfeits(dict(on, max_atoms=None))) - assert not any("shortening candidates are scored only" in f + line = ("shortening candidates are scored against a spread sample of 60 of the extracted " + "atoms: a cut of a span with no sampled atom looks free") + assert line in forfeits(on, n_atoms=60) + assert not any("opening" in f or "tail" in f for f in forfeits(on, n_atoms=60)) + assert not any("spread sample" in f for f in forfeits(on, n_atoms=10)) + assert not any("spread sample" in f for f in forfeits(dict(on, max_atoms=None))) + assert not any("spread sample" in f for f in forfeits(dict(on, shorten_artifact=False), n_atoms=60)) @@ -315,8 +318,13 @@ def test_the_estimate_caps_the_expected_atoms_and_records_the_tier(): study = resolve_settings("study", {}) bill = study.estimate(300, budgets=(0.5,), n_arms=3) assert (bill["spans"], bill["atom_cap"]) == (300, 120) + # Probing stops at the cap; extraction reads all 150 expected atoms' spans. assert bill["estimate"] == estimate_calls( - 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3) + 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3, n_extracted=150) + assert bill["estimate"]["n_atoms"] == 120 + # 12 extraction batches (150 / 12, rounded half to even) where the cap's 120 atoms take 10. + assert bill["estimate"]["stage_calls"] - estimate_calls( + 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3)["stage_calls"] == 2 assert bill["tier"] == {"name": "study", "forfeits": TIERS["study"]["forfeits"], "overrides": {}} assert "cap is reached" in bill["caveat"] @@ -412,10 +420,14 @@ def pinned_list_prices(monkeypatch): def _projection(tier, spans, tokens): - """The estimate a run at this tier on this artifact prints, and its dollar figure.""" - estimate = resolve_settings(tier, {}).estimate( - spans, n_tokens=tokens, budgets=_BUDGETS, n_arms=2 - )["estimate"] + """The estimate of this run as it was measured, and its dollar figure. + + The runs predate the spread cap (#293): extraction stopped at the cap, so its batches follow + the capped atoms here. The estimate a run prints now counts every batch (no measured run yet). + """ + settings = resolve_settings(tier, {}) + n_atoms = settings.estimate(spans, n_tokens=tokens, budgets=_BUDGETS)["estimate"]["n_atoms"] + estimate = estimate_calls(n_atoms, dict(settings.values, budgets=_BUDGETS), n_arms=2) return estimate, estimate_usd(estimate["by_role"], tokens, _MODELS, tier) From e8fdf9529f19bdc187e79b1242738b55e1444aea Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Sun, 1 Nov 2026 19:52:05 +0000 Subject: [PATCH 2/8] fix(novelty): stride in document order; qualify partial-read samples Settles the review of the cap-spread change (#293): - When the cap binds, the atoms are sorted (stably) by the input position of the earliest span each cites before the stride is taken, so the sample is in document order whatever order the extractor's reply lists a batch's atoms in. A cap that does not bind keeps the extraction order, so atoms.json is unchanged for those runs. - extract_atoms refuses a cap below 1 with ValueError before any call, and extraction.sampled.atoms has a minimum of 1, so no record asks the renderers to describe a stride over zero atoms. - When spans went unread, the cap warning, the novelty.md header and the coverage line say the sample is spread over the atoms extracted from the spans that were read and how many spans were not, instead of claiming the whole input was read. cap_warning takes the extraction record. - The docs and D017 say the stride runs over the concatenated atom list of the selected files, so a file with few atoms may get no probe; that the same extracted atom list, not the same text, gives the same sample, and that reusing atoms.json probes identical atoms; and that extraction cost changes for every run whose cap was reached before its last batch, an exact-cap artifact included. Signed-off-by: Anton Volkov --- ...01_120000_antonwolfy_novelty-cap-spread.md | 27 +++++--- dokimi/docs/reviewing.md | 16 +++-- dokimi/docs/skill-novelty-walkthrough.md | 7 +- dokimi/docs/skill-novelty.md | 49 ++++++++----- dokimi/dokimi/commands/novelty.py | 2 +- dokimi/dokimi/novelty/analyze.py | 2 +- dokimi/dokimi/novelty/atoms.py | 54 +++++++++++---- dokimi/dokimi/novelty/report.py | 44 ++++++++---- dokimi/dokimi/schemas/novelty.v1.json | 4 +- dokimi/research/records/decision/D017.md | 4 +- dokimi/tests/test_novelty_analyze.py | 4 ++ dokimi/tests/test_novelty_atoms.py | 68 +++++++++++++++++++ dokimi/tests/test_novelty_report.py | 2 +- 13 files changed, 211 insertions(+), 72 deletions(-) diff --git a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md index 6716f588c..712778d02 100644 --- a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md +++ b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md @@ -4,18 +4,23 @@ `--max-atoms` or a tier's cap (25 at `triage`, 60 at `standard`, 120 at `study`), extraction no longer stops at the cap: it reads every span, and of the M atoms it extracts the run probes n at indices - `floor((i + 0.5) * M / n)`, every (M/n)-th in document order (on a - `--include-resources` run, across the files in file order), so a capped rate + `floor((i + 0.5) * M / n)`, every (M/n)-th in document order by the earliest + span each atom cites (on a `--include-resources` run, across the files in + file order, so a file with few atoms may get no probe), so a capped rate estimates the whole input's per-atom rate instead of describing its opening. - The selection takes no seed. `novelty analyze` and `novelty atoms` (whose - `atoms.json` `probe` and `shorten` read) probe the same atoms. New runs record - `atoms.sampling` as `cap-spread:` and `extraction.sampled` as - `{"atoms", "of", "rule"}`, with `extraction.reached` `null`; the cap - warning, the `novelty.md` header and the headline's `coverage:` line say how - many of the input's atoms were probed and with what stride. Artifacts written + The selection takes no seed: the same extracted atom list gives the same + sample. `novelty analyze` and `novelty atoms` (whose `atoms.json` `probe` and + `shorten` read) probe the same atoms. A cap below 1 is refused before any + call. New runs record `atoms.sampling` as `cap-spread:` and + `extraction.sampled` as `{"atoms", "of", "rule"}`, with `extraction.reached` + `null`; the cap warning, the `novelty.md` header and the headline's + `coverage:` line say how many of the input's atoms were probed and with what + stride, or, when some spans went unread, that the sample is spread over the + atoms of the spans that were read and how many were not. Artifacts written with `cap-per-artifact:` and a `reached` record still validate and render - as before. Probing still stops at the cap, but on an artifact with more atoms - than the cap extraction now makes more calls, and `--estimate-only` counts - them; the tier profiles were measured with the old order (decision D017). The + as before. Probing still stops at the cap, but a run whose cap was reached + before its last extraction batch now asks the remaining batches, and + `--estimate-only` counts them; the tier profiles were measured with the old + order (decision D017). The shortening forfeit now says candidates are scored against a spread sample of the cap's atoms, so cutting a span with no sampled atom looks free. diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 30c0f851b..18333de73 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -83,13 +83,15 @@ self-judge; `arms run` does not check, so name times arms, or tasks times harnesses. One measured example: triage on a 19k-token skill with a mid-size model was 103 calls, about $1.80; `study` on the same skill was 1,900 calls, about $14.50. -- **At triage** with `--include-resources`, the 25-atom cap probes atoms spread - over every file the budgets selected, in file order, so `references/` gets its - share by atom count; the run's cap warning says how many of the input's atoms - it probed. Extraction still reads every file, so a large bundle costs more - extraction calls than its 25 probes suggest. Spread over several files, 25 - atoms leave each file few; for a figure per file, use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for - a full sweep; run `--estimate-only` first, it costs more. +- **At triage** with `--include-resources`, the 25-atom cap strides over the + concatenated atom list of the files the budgets selected, in file order, so + `references/` can be probed, but a file with few atoms may get no probe: with + one atom in each of 100 files, triage probes 25 of them. The run's cap warning + says how many of the input's atoms it probed. Extraction still reads every + file, so a large bundle costs more extraction calls than its 25 probes suggest. + `--tier standard` (60 atoms) spreads more probes without promising each file + one; add `--all-atoms` to the tier command to probe every extracted atom, and + run `--estimate-only` first, it costs more. - Step 5 needs evaluation content: a golden question set for 5a, executable tasks for 5b. `arms run` takes no tasks. That content lives outside this repository ([external-content.md](external-content.md)). Without it, the diff --git a/dokimi/docs/skill-novelty-walkthrough.md b/dokimi/docs/skill-novelty-walkthrough.md index b17b52630..21ffa9195 100644 --- a/dokimi/docs/skill-novelty-walkthrough.md +++ b/dokimi/docs/skill-novelty-walkthrough.md @@ -94,9 +94,10 @@ comparable with another artifact's. If the skill is a thin `SKILL.md` over a `references/` tree, add `--include-resources`. Without it the run measures the entry point and prints the share it is skipping — in the -study corpus that share was 0.72 of all words. At `triage` the 25-atom cap probes atoms spread -over every selected file in file order, so `references/` gets its share by atom count; 25 atoms -over several files leave each one few, so for a figure per file use `--tier standard` or higher. +study corpus that share was 0.72 of all words. At `triage` the 25-atom cap strides over the +concatenated atom list of the selected files, in file order, so a file with few atoms may get no +probe. A higher tier spreads more probes without promising each file one; `--all-atoms` probes +every extracted atom. ## 4–5. Extract once, probe per model diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 7a3f93ec5..d8560bea3 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -911,22 +911,37 @@ extracted atoms: a cut of a span with no sampled atom looks free` (60 at `standa The cap is a spread sample, not the opening. Extraction reads every span of the input, and when it yields M atoms, more than the cap n, the run probes the atoms at indices -`floor((i + 0.5) * M / n)` for `i` in `0..n-1` of the extracted list (`spread_sample` in -`dokimi/novelty/atoms.py`): every (M/n)-th atom in document order, starting in the middle of the -first stride, so the sample estimates the per-atom rate an uncapped run reports. A stride needs no -seed: the same text gives the same sample. With M ≤ n every atom is probed. The input is the text that was -segmented: the entry point, plus on a bundle run the files its roles and budgets selected, in that -order, so the stride gives each selected file its share by atom count, and a bundled file left -out has no atom in it (the `coverage:` line says how much of the artifact went into the input). -The run records `atoms.sampling` as `cap-spread:` and `extraction.sampled` as -`{"atoms": n, "of": M, "rule": "cap-spread"}`, and warns `atom extraction read the whole input, -and the atom cap probes 25 of the input's 61 atoms, spread over all of it (a stride of 2.44 -atoms in document order): rates are estimated from that sample; pass --all-atoms to probe every -atom`. The `novelty.md` header says the same, and the headline's `coverage:` line says the -words went into the input and the cap probed 25 of the input's 61 atoms (the warning under that -line, when half or less of the words went in, says "went into the input" too). `novelty atoms` -prints the same warning on stderr and saves the same sample in `atoms.json`; `probe` and -`shorten`, which read it, do not repeat the warning. +`floor((i + 0.5) * M / n)` for `i` in `0..n-1` of the extracted list sorted by the earliest span +each atom cites (`spread_sample` in `dokimi/novelty/atoms.py`): every (M/n)-th atom in document +order, starting in the middle of the first stride, so the sample estimates the per-atom rate an +uncapped run reports. The sort is needed because the extractor's reply lists a batch's atoms in +its own order; it is stable, so atoms citing the same span keep that order, and with M ≤ n +nothing is sorted and every atom is probed in extraction order. A stride needs no seed: the +same extracted atom list gives the same sample. A fresh extraction is new model calls, so the +same text can give other atoms and so another sample; to probe identical atoms across runs or +models, extract once with `novelty atoms` and pass its `atoms.json` to `novelty probe --atoms` +(and `novelty shorten --atoms`), which probe the saved atoms without sampling again. The input +is the text that was segmented: the entry point, plus on a bundle run the files its roles and +budgets selected, in that order. The stride runs over the concatenated atom list of those +files, so a file's expected share of the probes is its share of the atoms, but a file with few +atoms can fall between two sampled indices and get no probe; a higher cap spreads more probes +without promising each file one, and `--all-atoms` probes every extracted atom. A bundled file +left out has no atom in it (the `coverage:` line says how much of the artifact went into the +input). The run records `atoms.sampling` as `cap-spread:` and `extraction.sampled` as +`{"atoms": n, "of": M, "rule": "cap-spread"}`. When every span was read it warns `atom +extraction read the whole input, and the atom cap probes 25 of the input's 61 atoms, spread over +all of it (a stride of 2.44 atoms in document order): rates are estimated from that sample; pass +--all-atoms to probe every atom`. When some spans were left unread (`extraction.unread_span_ids`: +a failed call, an empty or unparseable reply), M counts only the atoms of the spans that were +read, and the warning says so instead: `atom extraction could not read 25 of the input's spans, +and the atom cap probes 2 of the 25 atoms extracted from the spans that were read, spread over +those (a stride of 12.5 atoms in document order; 25 of the input's spans were not read, see +extraction.unread_span_ids): ...`. The `novelty.md` header says the same, and the headline's +`coverage:` line says the words went into the input and the cap probed 25 of the input's 61 +atoms, or of the atoms extracted from the spans that were read (the warning under that line, +when half or less of the words went in, says "went into the input" too). `novelty atoms` prints +the same warning on stderr and saves the same sample in `atoms.json`; `probe` and `shorten`, +which read it, do not repeat the warning. Runs written before the spread rule recorded `cap-per-artifact:`: extraction walked the spans in document order and stopped at the cap, so their atoms cover the input up to the span it @@ -1001,7 +1016,7 @@ lost to a failed call (with its error). The | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked | | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting | | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded | -| `sampled` | on a run whose cap left atoms out, what it probed: `atoms`, the cap; `of`, the atoms extracted from every span; `rule`, `cap-spread`. `null` when every atom was probed | +| `sampled` | on a run whose cap left atoms out, what it probed: `atoms`, the cap; `of`, the atoms extracted from every span some call read; `rule`, `cap-spread`. `null` when every atom was probed | | `reached` | `null` on a `cap-spread` run. On a run before it, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind | `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index 3bd0f1310..bb12006da 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -158,7 +158,7 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None: except ExtractionError as exc: raise SystemExit(str(exc)) from None # The cap warning analyze records; probe and shorten, which read atoms.json, do not repeat it. - sampled = [cap_warning(extraction.sampled)] if extraction.sampled else [] + sampled = [cap_warning(extraction.as_dict())] if extraction.sampled else [] for warning in extraction.warnings(len(spans)) + sampled: print(f"WARNING {warning}", file=sys.stderr) if settings.values["verify_faithfulness"]: diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index 158708524..226d6665c 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -183,7 +183,7 @@ def analyze( # Older runs recorded `cap-per-artifact:`, the first n atoms in document order (#293). sampling = f"cap-spread:{config.max_atoms}" if config.max_atoms is not None else "full" if extraction.sampled: # set only when the cap left atoms out - warnings.append(report.cap_warning(extraction.sampled)) + warnings.append(report.cap_warning(extraction.as_dict())) if not atom_list: warnings.append("no atoms with span provenance were extracted; nothing to measure") diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index 208412c7e..65b4df374 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -308,9 +308,10 @@ def extract_atoms( A batch whose reply does not parse, or parses to rows none of which is a usable atom (see :func:`_atom_from_row`), is split in half and re-asked, down to one span - per call; the halves go back to the front of the queue, so atoms keep document order - (which is what ``cap`` strides over). The common cause is an output ceiling: a provider - that defaults to 4096 output tokens truncates a 25-span reply mid-array, and a + per call; the halves go back to the front of the queue, so batches keep document order + (within a batch, atoms come in the reply's order). The common cause is an output + ceiling: a provider that defaults to 4096 output tokens truncates a 25-span reply + mid-array, and a truncated array does not parse. Splitting recovers without knowing the ceiling, at a cost of up to 2n-1 calls for an n-span batch that never parses; ``report.calls`` counts them, against ``report.batches`` for the batches of the input. A call that @@ -322,15 +323,19 @@ def extract_atoms( client: Extractor client (must not be the judge — see :mod:`.measure`). batch_spans: Spans per extraction call. Batching is what keeps extraction affordable; too large and the model starts skipping spans. - cap: Keep this many atoms, spread over the whole input (:func:`spread_sample`); - every span is extracted either way. Recorded in the artifact as the sampling - rule, so a capped run is never quoted as a full sweep. + cap: Keep this many atoms, spread over the extracted atoms in document order + (:func:`spread_sample`); every span is extracted either way. At least 1: a + smaller cap raises ``ValueError`` before any call. Recorded in the artifact as + the sampling rule, so a capped run is never quoted as a full sweep. report: Filled in place with what could not be read and what re-asking cost, and what a binding cap kept (``sampled``). Spans no call could read are also marked ``unread`` in place. Returns: - Atoms in extraction order, de-duplicated by id: all of them, or the cap's sample. + Atoms de-duplicated by id. When the cap keeps them all (no cap, or ``M <= n``): + every atom, in extraction order (batches in document order, each in its reply's + order). When it binds: the cap's sample, in document order, by the earliest span + each atom cites (atoms citing the same earliest span keep extraction order). """ from dokimi.llm import extract_json_array @@ -355,6 +360,7 @@ def parse(reply: str) -> list: (spans[start:start + batch_spans], False) for start in range(0, len(spans), batch_spans) ] + _check_cap(cap) # before any call while pending: batch, is_half = pending.pop(0) if not is_half: @@ -411,20 +417,38 @@ def parse(reply: str) -> list: # result, and a recovered parse error is not a failure. raise ExtractionError(f"no atoms: every extraction call failed ({lost})") from lost report.sampled = sample_record(len(atoms), cap) - return spread_sample(atoms, cap) + return spread_sample(atoms, cap, key=_earliest_span(spans)) -def spread_sample(atoms: list, cap: int | None) -> list: - """The ``cap`` atoms a capped run probes: every (M/n)-th of the M extracted, in their order. +def _check_cap(cap: int | None) -> None: + """A cap probes at least one atom: ``0`` would record a sample of nothing (the CLI's + ``--max-atoms`` takes positive ints only; this guards the Python API).""" + if cap is not None and cap < 1: + raise ValueError(f"the atom cap must be at least 1, or None for no cap; got {cap}") - Index ``floor((i + 0.5) * M / n)`` for ``i`` in ``0..n-1``, so the sample starts at the - middle of the first stride and estimates the per-atom rate an uncapped run reports. A - pure function of the list, with no seed: the same atoms give the same sample. With no - cap, or ``M <= n``, every atom. https://github.com/intel/dokimi/issues/293 + +def _earliest_span(spans: list[Span]): + """Sort key: the input position of the earliest span an atom cites. Within a batch the + reply's order is the model's, which the prompt does not fix.""" + position = {span.id: i for i, span in enumerate(spans)} + return lambda atom: min(position[span_id] for span_id in atom.span_ids) + + +def spread_sample(atoms: list, cap: int | None, *, key=None) -> list: + """The ``cap`` atoms a capped run probes: every (M/n)-th of the M extracted. + + Index ``floor((i + 0.5) * M / n)`` for ``i`` in ``0..n-1`` of the atoms sorted by ``key`` + (stably, so atoms with equal keys keep their order; ``None``: as given), so the sample + starts at the middle of the first stride and estimates the per-atom rate an uncapped run + reports. :func:`extract_atoms` sorts by the earliest cited span, i.e. document order + (on a bundle run, the files in bundle order). A pure function of the list, with no seed: + the same extracted list gives the same sample. With no cap, or ``M <= n``, every atom, + in the order given and unsorted. https://github.com/intel/dokimi/issues/293 """ if cap is None or len(atoms) <= cap: return list(atoms) - return [atoms[(2 * i + 1) * len(atoms) // (2 * cap)] for i in range(cap)] + ordered = sorted(atoms, key=key) if key is not None else list(atoms) + return [ordered[(2 * i + 1) * len(ordered) // (2 * cap)] for i in range(cap)] def sample_record(n_extracted: int, cap: int | None) -> dict | None: diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index 1882f9dd4..35f753570 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -257,18 +257,35 @@ def reach_is_end(reached: dict) -> bool: return reached["spans"] == reached["of"] -def describe_sample(sampled: dict) -> str: +def describe_sample(sampled: dict, n_unread: int = 0) -> str: """What a binding cap probed, as ``25 of the input's 61 atoms, spread over all of it (a - stride of 2.44 atoms in document order)``: ``extraction.sampled`` of a ``cap-spread`` run.""" - stride = sampled["of"] / sampled["atoms"] + stride of 2.44 atoms in document order)``: ``extraction.sampled`` of a ``cap-spread`` run. + + With ``n_unread`` spans no extraction call read, the atoms are only those of the spans + that were read, so the sample is spread over those, not over the whole input. + """ + stride = f"a stride of {sampled['of'] / sampled['atoms']:.3g} atoms in document order" + if n_unread: + return (f"{sampled['atoms']} of the {sampled['of']} atoms extracted from the spans that " + f"were read, spread over those ({stride}; {n_unread} of the input's spans were " + "not read, see extraction.unread_span_ids)") return (f"{sampled['atoms']} of the input's {sampled['of']} atoms, spread over all of it " - f"(a stride of {stride:.3g} atoms in document order)") + f"({stride})") + + +def _what_extraction_read(n_unread: int) -> str: + """What extraction read, after "extraction" in a cap sentence: every span, or not.""" + if n_unread: + return f"could not read {n_unread} of the input's spans" + return "read the whole input" -def cap_warning(sampled: dict) -> str: - """The warning for a binding cap: which atoms it probed, and how to probe them all.""" - return (f"atom extraction read the whole input, and the atom cap probes " - f"{describe_sample(sampled)}: rates are estimated from that sample; pass " +def cap_warning(extraction: dict) -> str: + """The warning for a binding cap, from the ``extraction`` record of a run whose ``sampled`` + is set: which atoms it probed, and how to probe them all.""" + sampled, n_unread = extraction["sampled"], len(extraction.get("unread_span_ids") or []) + return (f"atom extraction {_what_extraction_read(n_unread)}, and the atom cap probes " + f"{describe_sample(sampled, n_unread)}: rates are estimated from that sample; pass " "--all-atoms to probe every atom") @@ -277,11 +294,13 @@ def coverage_reach(extraction: dict | None) -> tuple[str, str]: The coverage share counts the words that went into the input. Under a binding cap only the cap's atoms were probed, so the line must not say the words were measured: it says - which atoms were (``sampled``), or for a run before #293, how far the atoms reach. + which atoms were (``sampled``; over the read spans only, when some went unread), or for a + run before #293, how far the atoms reach. """ sampled = (extraction or {}).get("sampled") if sampled: - return "went into the input", f"; the cap probed {describe_sample(sampled)}" + n_unread = len((extraction or {}).get("unread_span_ids") or []) + return "went into the input", f"; the cap probed {describe_sample(sampled, n_unread)}" reached = (extraction or {}).get("reached") if not reached: return "were measured", "" @@ -297,8 +316,9 @@ def _capped_at(artifact: dict) -> str: stopped extraction), or ``""``.""" sampled = (artifact.get("extraction") or {}).get("sampled") if sampled: - return (f" Extraction read the whole input; the atom cap probed " - f"{describe_sample(sampled)}.") + n_unread = len(artifact["extraction"].get("unread_span_ids") or []) + return (f" Extraction {_what_extraction_read(n_unread)}; the atom cap probed " + f"{describe_sample(sampled, n_unread)}.") reached = (artifact.get("extraction") or {}).get("reached") if not reached: return "" diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index c1c909304..a3f9be08d 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -349,9 +349,9 @@ }, "sampled": { "type": ["object", "null"], - "description": "What a binding cap probed (atoms.sampling cap-spread:): atoms of the of atoms extracted from every span, at indices floor((i + 0.5) * of / atoms) for i in 0..atoms-1 of the extracted list (document order; on a bundle run, the files in bundle order). No seed: the same extracted list gives the same sample. null when the cap kept every atom. Absent from artifacts written before it existed.", + "description": "What a binding cap probed (atoms.sampling cap-spread:): atoms of the of atoms extracted from every span some call read (extraction.unread_span_ids lists the rest), at indices floor((i + 0.5) * of / atoms) for i in 0..atoms-1 of the extracted list sorted by the earliest span each atom cites (document order; on a bundle run, the files in bundle order). No seed: the same extracted list gives the same sample. null when the cap kept every atom. Absent from artifacts written before it existed.", "properties": { - "atoms": {"type": "integer", "minimum": 0}, + "atoms": {"type": "integer", "minimum": 1}, "of": {"type": "integer", "minimum": 1}, "rule": {"type": "string"} }, diff --git a/dokimi/research/records/decision/D017.md b/dokimi/research/records/decision/D017.md index 2a91e58a2..c948b89a9 100644 --- a/dokimi/research/records/decision/D017.md +++ b/dokimi/research/records/decision/D017.md @@ -16,6 +16,6 @@ SPDX-License-Identifier: Apache-2.0 ## Rationale A cap that takes atoms in document order measures the opening of the input, not the input. At `--tier triage` on the dpnp repository's agent guidance, the 25 capped atoms of the root `AGENTS.md` (61 spans) came from spans s0002 to s0012, about a quarter of its tokens, and those of `dpnp/tests/AGENTS.md` (32 spans) from s0003 to s0013, about a third. Two artifacts' capped rates then describe different shares of their text, and a file's rate moves when its sections are reordered. -A capped run now extracts every span and, of the M atoms extracted, probes the n at indices `floor((i + 0.5) * M / n)`, in document order (on a bundle run, in file order). A per-atom stride estimates the same per-atom rate an uncapped run reports. Equal token strata would not: they would weight a sparse introduction like a dense rule list. The stride needs no seed, so the same text gives the same sample. Probing stays at the cap; extraction grows to every batch of the input. +A capped run now extracts every span and, of the M atoms extracted, probes the n at indices `floor((i + 0.5) * M / n)` of the atoms in document order, by the earliest span each cites (on a bundle run, in file order). A per-atom stride estimates the same per-atom rate an uncapped run reports. Equal token strata would not: they would weight a sparse introduction like a dense rule list. The stride needs no seed, so the same extracted atom list gives the same sample; a fresh extraction is new model calls and can give other atoms, so runs that must probe identical atoms reuse one `atoms.json` through `novelty probe --atoms`. Probing stays at the cap; extraction grows to every batch of the input. -The tier profiles (the call and token figures `--estimate-only` is fitted and checked on, and the per-tier call counts in `docs/skill-novelty.md`) were measured with the document-order cap. Only artifacts with more atoms than a tier's cap (25 at triage, 60 at standard, 120 at study) are affected; below the cap every atom is probed, as before. Re-measuring those artifacts, and refitting the extraction profile to a run that asks every batch, is follow-up work this decision does not do. +The tier profiles (the call and token figures `--estimate-only` is fitted and checked on, and the per-tier call counts in `docs/skill-novelty.md`) were measured with the document-order cap. Which atoms are probed changes only for artifacts with more atoms than a tier's cap (25 at triage, 60 at standard, 120 at study); at or below the cap every atom is probed, as before. Extraction cost changes for more runs: any run whose cap was reached before its last batch, an artifact whose atoms exactly meet the cap included, now asks the remaining batches. Re-measuring those runs, and refitting the extraction profile to a run that asks every batch, is follow-up work this decision does not do. diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 6284a725d..2fa22efb1 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -648,6 +648,10 @@ def recording(span_list, atom_list, **kwargs): partial = dict(artifact["extraction"], sampled={"atoms": 1}) with pytest.raises(ArtifactValidationError, match="extraction/sampled: 'of' is a required"): validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=partial)) + # A sample of no atoms has no stride to describe; `null` is the absence of a sample. + empty = dict(artifact["extraction"], sampled={"atoms": 0, "of": 2, "rule": "cap-spread"}) + with pytest.raises(ArtifactValidationError, match="extraction/sampled/atoms: 0 is less than"): + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=empty)) def test_an_artifact_written_under_the_document_order_cap_still_validates(tmp_path): diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 4802d9193..a205f55a1 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -242,6 +242,74 @@ def test_a_cap_keeps_every_kth_atom_from_the_middle_of_the_first_stride(n_atoms, assert spread_sample([f"atom{i}" for i in range(n_atoms)], cap) == [f"atom{i}" for i in kept] +def _cited(*span_ids: str) -> dict: + return {"statement": f"{'+'.join(span_ids)} holds a default.", "question": f"q {span_ids}", + "answer": "a value", "kind": "proposition", "tags": [], "span_ids": list(span_ids)} + + +@pytest.mark.parametrize("reply, cap, probed", [ + # Sorted s0001 s0002 s0003; the one stride's middle is s0002, where reply order gives s0003. + ([("s0001",), ("s0003",), ("s0002",)], 1, [("s0002",)]), + ([("s0001",), ("s0003",), ("s0002",), ("s0004",)], 2, [("s0002",), ("s0004",)]), + # By the earliest span an atom cites, not its first: (s0004, s0001) sorts first. + ([("s0003",), ("s0004", "s0001"), ("s0002",)], 1, [("s0002",)]), + # A cap that does not bind keeps the reply's order: atoms.json is as before #293. + ([("s0002",), ("s0001",)], 2, [("s0002",), ("s0001",)]), + ([("s0001",), ("s0003",), ("s0002",)], None, [("s0001",), ("s0003",), ("s0002",)]), +], ids=["cap-1", "cap-2-of-4", "earliest-cited", "at-cap", "no-cap"]) +def test_a_binding_cap_strides_in_document_order_whatever_the_reply_s_order(reply, cap, probed): + def out_of_order(prompt, system): + return json.dumps([_cited(*span_ids) for span_ids in reply]), None + + atoms = extract_atoms(segment(DOC), client=LLMClient(model="x", transport=out_of_order), + cap=cap) + assert [atom.span_ids for atom in atoms] == probed + + +def test_a_cap_below_one_is_refused_before_any_call(): + calls: list[str] = [] + + def counted(prompt, system): + calls.append(prompt) + return _three_per_span(prompt, system) + + with pytest.raises(ValueError, match="the atom cap must be at least 1"): + extract_atoms(segment(DOC), client=LLMClient(model="x", transport=counted), cap=0) + assert calls == [] + + +def test_a_binding_cap_over_a_partly_read_input_makes_no_whole_input_claim(): + """50 spans, cap 2: the first batch gives 25 atoms and the second an empty reply.""" + from dokimi.novelty.report import _capped_at, cap_warning, coverage_reach + + spans = segment("\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 51))) + second_batch = {s.id for s in spans[25:]} + + def second_empty(prompt, system): + if second_batch & set(re.findall(r"^\[(s\d+)\]", prompt, flags=re.M)): + return "", None + return json.dumps(_rows(prompt)), None + + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=second_empty), cap=2, + report=report) + assert [a.span_ids[0] for a in atoms] == ["s0007", "s0019"] # both from the opening + assert report.sampled == {"atoms": 2, "of": 25, "rule": "cap-spread"} + assert len(report.unread_span_ids) == 25 + record = report.as_dict() + sample = ("2 of the 25 atoms extracted from the spans that were read, spread over those (a " + "stride of 12.5 atoms in document order; 25 of the input's spans were not read, see " + "extraction.unread_span_ids)") + texts = [cap_warning(record), _capped_at({"extraction": record}), coverage_reach(record)[1]] + assert texts == [ + f"atom extraction could not read 25 of the input's spans, and the atom cap probes {sample}: " + "rates are estimated from that sample; pass --all-atoms to probe every atom", + f" Extraction could not read 25 of the input's spans; the atom cap probed {sample}.", + f"; the cap probed {sample}", + ] + assert not any("whole input" in text or "all of it" in text for text in texts) + + def _three_per_span(prompt, system): rows = [dict(row, statement=f"{row['statement']} ({k})", question=f"q{k} {row['span_ids'][0]}") diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index 3457319f8..35e939184 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -82,7 +82,7 @@ def test_the_cost_line_names_replayed_calls_only_when_there_are_some(cost, line) def test_a_spread_run_says_how_many_atoms_it_probed_spread_over_the_whole_input(): - assert cap_warning(SAMPLED) == ( + assert cap_warning({"sampled": SAMPLED, "unread_span_ids": []}) == ( "atom extraction read the whole input, and the atom cap probes 25 of the input's 61 atoms, " "spread over all of it (a stride of 2.44 atoms in document order): rates are estimated from " "that sample; pass --all-atoms to probe every atom") From c49b22125ca8ba5fc2dd02cff1a1f5bf625c9994 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Sun, 1 Nov 2026 20:55:48 +0000 Subject: [PATCH 3/8] fix(novelty): count a run as capped only when the cap left atoms out Settles the two review comments on e8fdf95 in #358 (issue #293). Extraction now reads every span, so a run whose atoms exactly meet the cap keeps every atom and records `extraction.sampled` as null. Two places still read `n_atoms >= cap` as capped: - `tiers.forfeits` took the post-cap atom count, so such a run carried the `sampled` forfeits and the spread-sample shortening forfeit in its warnings and `run_config.tier`. It now takes `sampled`, whether the cap left atoms out; `analyze` passes `extraction.sampled is not None` to `settings.warnings` and `tier_record`. Before extraction (estimates, the CLI's up-front tier listing) `sampled` is None and the cap is still assumed to bind. - `spans.span_summary` set `capped_atom_set` from the same count. It now takes `capped` from the same state, in `analyze` and in `novelty atoms`. Artifacts already written keep what they stored. The `capped_atom_set` schema description and docs/skill-novelty.md say when the flag and the forfeits apply. The changelog fragment says `extraction.sampled` is the {"atoms", "of", "rule"} object only when the cap leaves atoms out, and null when it keeps every atom. Tests: a study run whose atoms meet the cap has no sampled or shortening forfeit and `capped_atom_set` false, one atom over the cap has all three; `novelty atoms` flags the set the same way; the shortening-forfeit test no longer expects the warning at equality. Signed-off-by: Anton Volkov --- ...01_120000_antonwolfy_novelty-cap-spread.md | 7 +++-- dokimi/docs/skill-novelty.md | 5 ++-- dokimi/dokimi/commands/novelty.py | 2 +- dokimi/dokimi/novelty/analyze.py | 11 +++---- dokimi/dokimi/novelty/spans.py | 9 +++--- dokimi/dokimi/novelty/tiers.py | 30 +++++++++++-------- dokimi/dokimi/schemas/novelty.v1.json | 2 +- dokimi/tests/test_novelty_analyze.py | 20 +++++++++++++ dokimi/tests/test_novelty_atoms.py | 29 ++++++++++++++++++ dokimi/tests/test_novelty_spans.py | 5 ++-- dokimi/tests/test_novelty_tiers.py | 25 +++++++++------- 11 files changed, 104 insertions(+), 41 deletions(-) diff --git a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md index 712778d02..bbd67e6d6 100644 --- a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md +++ b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md @@ -12,8 +12,11 @@ sample. `novelty analyze` and `novelty atoms` (whose `atoms.json` `probe` and `shorten` read) probe the same atoms. A cap below 1 is refused before any call. New runs record `atoms.sampling` as `cap-spread:` and - `extraction.sampled` as `{"atoms", "of", "rule"}`, with `extraction.reached` - `null`; the cap warning, the `novelty.md` header and the headline's + `extraction.reached` `null`; `extraction.sampled` is `{"atoms", "of", "rule"}` + when the cap leaves atoms out, and `null` when it keeps every atom (M ≤ n). + Only a cap that leaves atoms out sets `spans.capped_atom_set` and brings the + cap's forfeits: an input whose atoms exactly meet the cap no longer counts as + capped. The cap warning, the `novelty.md` header and the headline's `coverage:` line say how many of the input's atoms were probed and with what stride, or, when some spans went unread, that the sample is spread over the atoms of the spans that were read and how many were not. Artifacts written diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index d8560bea3..630ebd852 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -897,12 +897,13 @@ read as one. Each tier carries the **claims it forfeits**, and those strings are written into `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the run used and the atoms it extracted, not the tier's name: `--no-verify` adds the -faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the +faithfulness forfeit, and `--all-atoms` or a skill with no more atoms than the cap (which +keeps every atom: `extraction.sampled` is `null` and `spans.capped_atom_set` false) drops the cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on which atoms the cap kept (`--estimate-only` lists them before it is known), and neither does the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). With shortening on, a -reached cap adds that `shortening candidates are scored against a spread sample of 60 of the +cap that left atoms out adds that `shortening candidates are scored against a spread sample of 60 of the extracted atoms: a cut of a span with no sampled atom looks free` (60 at `standard`, 120 at `study`). `--estimate-only` cannot know the atom count yet, so it lists them: diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index bb12006da..61d90c49b 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -167,7 +167,7 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None: save_atoms(atoms, spans, out, extractor=extractor.model, cap=cap, bundle=artifact_bundle.summary(), extraction=extraction) print(f"OK Saved {len(atoms)} atoms over {len(spans)} spans: {out}") - print(json.dumps(span_summary(spans, atoms, atom_cap=cap), indent=2)) + print(json.dumps(span_summary(spans, atoms, capped=extraction.sampled is not None), indent=2)) def cmd_novelty_probe(args: argparse.Namespace) -> None: diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index 226d6665c..b30a516cb 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -177,9 +177,10 @@ def analyze( if tier: settings = replace(settings, tier=tier["name"], overrides=dict(tier.get("overrides") or {})) - # From what this run did, not the tier's text: an override or an atom set under the - # cap changes which claims are forfeit. - warnings.extend(settings.warnings(n_atoms=len(atom_list))) + # From what this run did, not the tier's text: an override, or a cap that kept every atom + # (M <= n, so `sampled` is unset), changes which claims are forfeit. + capped = extraction.sampled is not None + warnings.extend(settings.warnings(sampled=capped)) # Older runs recorded `cap-per-artifact:`, the first n atoms in document order (#293). sampling = f"cap-spread:{config.max_atoms}" if config.max_atoms is not None else "full" if extraction.sampled: # set only when the cap left atoms out @@ -360,7 +361,7 @@ def analyze( atoms=atom_list, sampling=sampling, measure_result=measure_result, - span_summary=spans.span_summary(span_list, atom_list, atom_cap=config.max_atoms), + span_summary=spans.span_summary(span_list, atom_list, capped=capped), by_knowledge_kind=by_knowledge_kind, probe_fidelity=probe_set.as_dict(), items=item_stats, @@ -384,7 +385,7 @@ def analyze( warnings=warnings, ) payload["run_config"] = _run_config(config) - payload["run_config"]["tier"] = settings.tier_record(n_atoms=len(atom_list)) + payload["run_config"]["tier"] = settings.tier_record(sampled=capped) return AnalyzeResult( artifact=payload, attributions=attributions, diff --git a/dokimi/dokimi/novelty/spans.py b/dokimi/dokimi/novelty/spans.py index c091d1611..292d64279 100644 --- a/dokimi/dokimi/novelty/spans.py +++ b/dokimi/dokimi/novelty/spans.py @@ -254,7 +254,7 @@ def index_atoms(spans: list[Span], atoms: list) -> None: target.atom_ids.append(atom.id) -def span_summary(spans: list[Span], atoms: list | None = None, *, atom_cap: int | None = None) -> dict: +def span_summary(spans: list[Span], atoms: list | None = None, *, capped: bool = False) -> dict: """Aggregate span statistics for the ``novelty.v1`` ``spans{}`` block. ``tokens_carrying_nothing`` counts tokens in spans from which the extractor drew no @@ -281,8 +281,9 @@ def span_summary(spans: list[Span], atoms: list | None = None, *, atom_cap: int Args: spans: Segmented artifact spans. atoms: Atoms to index onto the spans. Pass the full extracted set. - atom_cap: The cap the extraction ran under, if any, so a capped set is flagged - rather than silently averaged. + capped: Whether the cap left atoms out (``ExtractionReport.sampled`` is set), so a + sampled set is flagged rather than silently averaged. An input with as many atoms + as the cap, or fewer, keeps every atom and is not capped. """ if atoms is not None: index_atoms(spans, atoms) @@ -305,7 +306,7 @@ def span_summary(spans: list[Span], atoms: list | None = None, *, atom_cap: int "n_unread": len(unknown), "tokens_unread": unknown_tokens, "n_atoms_indexed": len(atoms) if atoms is not None else None, - "capped_atom_set": bool(atom_cap and atoms is not None and len(atoms) >= atom_cap), + "capped_atom_set": capped, "sole_carrier": len(sole), "protected": sum(1 for s in spans if s.protected), "by_kind": { diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 7f0c1e9ca..20b23a567 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -24,8 +24,8 @@ from dataclasses import dataclass, field from typing import Any -#: What a run gives up, keyed by the setting that is off. "sampled" is a reached atom cap, which -#: probes atoms spread over the input: https://github.com/intel/dokimi/issues/293 +#: What a run gives up, keyed by the setting that is off. "sampled" is an atom cap that left atoms +#: out, which probes atoms spread over the input: https://github.com/intel/dokimi/issues/293 FORFEITS: dict[str, list[str]] = { "sampled": [ "`added` is a rate over the atoms the cap kept, not the whole input: usable as a " @@ -57,15 +57,17 @@ } -def forfeits(settings: dict, *, n_atoms: int | None = None) -> list[str]: +def forfeits(settings: dict, *, sampled: bool | None = None) -> list[str]: """The claims a run with these settings may not make. Args: settings: Resolved ``AnalyzeConfig`` fields; a stage switch that is absent is off. - n_atoms: Atoms extracted, when known. Before extraction a cap is assumed reached. + sampled: Whether the cap left atoms out (``ExtractionReport.sampled`` is set), once + extraction has run: an input with as many atoms as the cap, or fewer, keeps every + atom. ``None`` before extraction, when a cap is assumed to bind. """ cap = settings.get("max_atoms") - capped = cap is not None and (n_atoms is None or n_atoms >= cap) + capped = cap is not None and sampled is not False out = list(FORFEITS["sampled"]) if capped else [] if capped and settings.get("shorten_artifact"): out.append( @@ -154,7 +156,8 @@ class RunSettings: ``values`` holds every :data:`SETTING_NAMES` entry, resolved; ``tier`` and ``overrides`` say how they were reached, because a ``--tier triage --max-atoms 90`` run is not a stock triage run and must not be read as one. The forfeits and warnings - follow ``values`` (and, once known, the atom count) -- never the tier's name. + follow ``values`` (and, once known, whether the cap left atoms out) -- never the tier's + name. """ values: dict[str, Any] @@ -172,18 +175,18 @@ def config_fields(self) -> dict[str, Any]: """The ``AnalyzeConfig`` keyword arguments that carry these settings.""" return {**self.values, "tier": self.tier, "tier_overrides": dict(self.overrides)} - def forfeits(self, n_atoms: int | None = None) -> list[str]: + def forfeits(self, sampled: bool | None = None) -> list[str]: """The claims a run at these settings may not make (see :func:`forfeits`).""" - return forfeits(self.values, n_atoms=n_atoms) + return forfeits(self.values, sampled=sampled) - def tier_record(self, n_atoms: int | None = None) -> dict | None: + def tier_record(self, sampled: bool | None = None) -> dict | None: """``run_config.tier``: name, forfeits, overrides; ``None`` without a tier.""" if self.tier is None: return None - return {"name": self.tier, "forfeits": self.forfeits(n_atoms), + return {"name": self.tier, "forfeits": self.forfeits(sampled), "overrides": dict(self.overrides)} - def warnings(self, n_atoms: int | None = None) -> list[str]: + def warnings(self, sampled: bool | None = None) -> list[str]: """The warnings a result at these settings carries. With a tier: its forfeits, recomputed from what the run did, and the overrides. @@ -197,7 +200,7 @@ def warnings(self, n_atoms: int | None = None) -> list[str]: if not self.values["reversals"]: out.extend(FORFEITS["reversals"]) return out - out = [f"tier `{self.tier}` forfeits: {line}" for line in self.forfeits(n_atoms)] + out = [f"tier `{self.tier}` forfeits: {line}" for line in self.forfeits(sampled)] if self.overrides: out.append( f"tier `{self.tier}` was overridden by flags " @@ -215,7 +218,8 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, rows, yield an atom each), capped at ``max_atoms`` for every stage but extraction, which reads the whole input. Without ``n_tokens`` only the span rule applies. The forfeits assume the cap is reached, since the atom count is a guess until extraction - runs: an artifact with fewer atoms than the cap drops the cap's forfeits. + runs: an artifact with no more atoms than the cap keeps them all and drops the cap's + forfeits. """ cap = self.values["max_atoms"] expected = max(1, n_spans // 2) diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index a3f9be08d..175a376fa 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -302,7 +302,7 @@ }, "capped_atom_set": { "type": "boolean", - "description": "True when the indexed atom set hit the extraction cap. The dead-token share is then a property of the sample, not of the artifact, and must not be quoted as the latter." + "description": "True when the atom cap left extracted atoms out, so the indexed atoms are a sample (extraction.sampled is set); an input with as many atoms as the cap, or fewer, keeps every atom and is false. Runs written before the spread rule (#293) set it when the indexed atoms reached the cap. The dead-token share is then a property of the sample, not of the artifact, and must not be quoted as the latter." }, "sole_carrier": {"type": "integer", "minimum": 0}, "protected": {"type": "integer", "minimum": 0}, diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 2fa22efb1..03d1589dd 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -676,6 +676,26 @@ def test_a_cap_the_atoms_meet_samples_nothing(tmp_path): validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) +@pytest.mark.parametrize("cap, capped", [(2, False), (1, True)], ids=["cap-met", "cap-binds"]) +def test_only_a_cap_that_leaves_atoms_out_counts_as_capped(tmp_path, cap, capped): + # Two atoms at `study`: a cap they meet keeps them all, so no cap forfeit and no flag; + # one atom more than the cap is a sample, and says so (#358). + from dokimi.novelty.tiers import FORFEITS, resolve_settings + + settings = resolve_settings("study", {"max_atoms": cap}) + artifact = _analyze(tmp_path, **settings.config_fields()).artifact + assert (artifact["extraction"]["sampled"] is not None) is capped + assert artifact["spans"]["capped_atom_set"] is capped + shortening = (f"shortening candidates are scored against a spread sample of {cap} of the " + "extracted atoms: a cut of a span with no sampled atom looks free") + warned = "\n".join(artifact["warnings"]) + recorded = artifact["run_config"]["tier"]["forfeits"] + for line in [*FORFEITS["sampled"], shortening]: + assert (line in warned) is capped, line + assert (line in recorded) is capped, line + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + + @pytest.mark.parametrize("cap, probed, by_file", [ # 8 atoms, one per span; SKILL.md holds s0001-s0005, references/tuning.md s0006-s0008. (2, ["s0003", "s0007"], {"SKILL.md": 1, "references/tuning.md": 1}), diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index a205f55a1..62fc3e2d1 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -641,6 +641,35 @@ def transport(prompt, system): ) +@pytest.mark.parametrize("cap, capped", [(3, False), (2, True)], ids=["cap-met", "cap-binds"]) +def test_novelty_atoms_flags_the_atom_set_only_when_the_cap_left_atoms_out( + tmp_path, monkeypatch, capsys, cap, capped): + """Three spans, one atom each: a cap of 3 keeps every atom, a cap of 2 samples them.""" + from tests import test_novelty_cli as novelty_cli + + skill = tmp_path / "s" / "SKILL.md" + skill.parent.mkdir(parents=True) + skill.write_text("---\nname: s\ndescription: d\n---\n\n" + "\n\n".join( + f"Setting {i} defaults to {i}." for i in range(1, 4)) + "\n", encoding="utf-8") + + def transport(prompt, system=None): + ids = re.findall(r"^\[(s\d+)\]", prompt, flags=re.M) + return json.dumps([{"statement": f"{sid} holds.", "question": f"What of {sid}?", + "answer": "a value", "span_ids": [sid]} for sid in ids]), None + + real_client = novelty_cli.novelty_cmd._client + monkeypatch.setattr(novelty_cli.novelty_cmd, "_client", + lambda model, provider, role, args: novelty_cli._with_transport( + real_client(model, provider, role, args), transport)) + args = novelty_cli._parse("novelty", "atoms", "--artifact", f"skill:{skill}", "--out", + str(tmp_path / "atoms.json"), "--max-atoms", str(cap), "--no-verify") + args.func(args) + out = capsys.readouterr().out + summary = json.loads(out[out.index("{"):]) + assert summary["n_atoms_indexed"] == cap + assert summary["capped_atom_set"] is capped + + def test_novelty_atoms_saves_the_same_spread_sample_analyze_probes(tmp_path, monkeypatch, capsys): """Two batches of 25 spans, one atom each: a cap of 2 keeps s0013 and s0038 for probe and shorten.""" from tests import test_novelty_cli as novelty_cli diff --git a/dokimi/tests/test_novelty_spans.py b/dokimi/tests/test_novelty_spans.py index ad51f4894..a4de2a034 100644 --- a/dokimi/tests/test_novelty_spans.py +++ b/dokimi/tests/test_novelty_spans.py @@ -90,6 +90,7 @@ def test_span_summary_flags_a_capped_atom_set_because_the_cap_inverts_the_metric Atom(statement=f"claim {i}", question="q", answer="a", span_ids=(spans[1].id,)) for i in range(3) ] - assert span_summary(spans, atoms, atom_cap=3)["capped_atom_set"] is True - assert span_summary(spans, atoms, atom_cap=4)["capped_atom_set"] is False + # The run says whether its cap left atoms out; three atoms alone do not say it. + assert span_summary(spans, atoms, capped=True)["capped_atom_set"] is True + assert span_summary(spans, atoms)["capped_atom_set"] is False assert span_summary(spans)["n_atoms_indexed"] is None diff --git a/dokimi/tests/test_novelty_tiers.py b/dokimi/tests/test_novelty_tiers.py index 0ef45da39..9e2d319c6 100644 --- a/dokimi/tests/test_novelty_tiers.py +++ b/dokimi/tests/test_novelty_tiers.py @@ -222,10 +222,10 @@ def test_forfeits_follow_the_resolved_settings(): sampled = FORFEITS["sampled"][0] standard = TIERS["standard"]["settings"] - # Before extraction the cap is assumed reached; an atom set under it is not sampled. + # Before extraction the cap is assumed to bind; a cap that kept every atom sampled none. assert sampled in forfeits(standard) - assert sampled not in forfeits(standard, n_atoms=15) - assert sampled in forfeits(standard, n_atoms=60) + assert sampled not in forfeits(standard, sampled=False) + assert sampled in forfeits(standard, sampled=True) # --all-atoms drops the cap's forfeit, triage included. args = _parse("novelty", "analyze", "--artifact", "skill:x", "--tier", "triage", "--all-atoms") @@ -271,19 +271,22 @@ def test_flags_that_drop_the_probe_kinds_warn_without_a_tier(tmp_path): assert not any("no paraphrase probe" in w or "no reversal probe" in w for w in full) -def test_shortening_under_a_reached_cap_says_it_saw_only_a_spread_sample(): +def test_shortening_under_a_cap_that_left_atoms_out_says_it_saw_only_a_spread_sample(): from dokimi.novelty.tiers import forfeits on = {"max_atoms": 60, "shorten_artifact": True, "paraphrases": 1, "reversals": True, "verify_faithfulness": True, "classify_content": True, "confirm_carriers": True} line = ("shortening candidates are scored against a spread sample of 60 of the extracted " "atoms: a cut of a span with no sampled atom looks free") - assert line in forfeits(on, n_atoms=60) - assert not any("opening" in f or "tail" in f for f in forfeits(on, n_atoms=60)) - assert not any("spread sample" in f for f in forfeits(on, n_atoms=10)) + assert line in forfeits(on, sampled=True) + # Before extraction (an estimate) the cap is assumed to leave atoms out. + assert line in forfeits(on) + assert not any("opening" in f or "tail" in f for f in forfeits(on, sampled=True)) + # A cap that kept every atom (M <= n, the atoms meeting it exactly included) sampled none. + assert not any("spread sample" in f for f in forfeits(on, sampled=False)) assert not any("spread sample" in f for f in forfeits(dict(on, max_atoms=None))) assert not any("spread sample" in f - for f in forfeits(dict(on, shorten_artifact=False), n_atoms=60)) + for f in forfeits(dict(on, shorten_artifact=False), sampled=True)) def test_the_settings_travel_through_the_config_unchanged(): @@ -336,11 +339,11 @@ def test_the_estimate_caps_the_expected_atoms_and_records_the_tier(): def test_the_warnings_follow_the_settings_and_the_atom_count(): standard = resolve_settings("standard", {"no_verify": True}) - warned = standard.warnings(n_atoms=60) + warned = standard.warnings(sampled=True) assert all(w.startswith("tier `standard`") for w in warned) assert any(FORFEITS["verify_faithfulness"][0] in w for w in warned) assert "overridden by flags (no_verify)" in warned[-1] - assert not any(FORFEITS["sampled"][0] in w for w in standard.warnings(n_atoms=10)) + assert not any(FORFEITS["sampled"][0] in w for w in standard.warnings(sampled=False)) # Without a tier only the dropped probe kinds warn; the stages warn in the analyzer. bare = resolve_settings(None, {"no_paraphrase": True, "no_classify": True}) assert bare.warnings() == FORFEITS["paraphrases"] @@ -362,7 +365,7 @@ def test_a_tier_on_the_config_and_the_older_tier_argument_record_the_same(tmp_pa judge=_clients()[2], tier={"name": "triage", "overrides": {"max_atoms": 1}}).artifact assert on_config["run_config"]["tier"] == as_argument["run_config"]["tier"] == ( - settings.tier_record(n_atoms=1)) + settings.tier_record(sampled=True)) assert on_config["warnings"] == as_argument["warnings"] assert any("tier `triage` was overridden by flags (max_atoms)" in w for w in on_config["warnings"]) From f9f9a8a6e8420a553c91272c91e000760f6eec1e Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Sun, 1 Nov 2026 21:17:09 +0000 Subject: [PATCH 4/8] docs(novelty): name the fixed stride's periodic-order limit Settles the review point on intel/dokimi#358 that a spread sample's estimate of the whole input's per-atom rate went unqualified. A fixed stride can fall in step with a repeated structure: with M = 100 atoms and a cap of 25 it always takes atoms 2, 6, ..., 98, so an input of same-shaped four-atom sections, each with its new fact in the same place, can give a sampled rate far from the whole input's, and re-extracting the same atoms gives the same sample. The cap passage of docs/skill-novelty.md, the changelog fragment and decision D017's rationale now say so, and recommend --all-atoms for an artifact of repeated, same-shaped sections or a rate that decides something. The stride stays seedless, as D017 decided; no code changes. Signed-off-by: Anton Volkov --- ...20261101_120000_antonwolfy_novelty-cap-spread.md | 10 ++++++---- dokimi/docs/skill-novelty.md | 13 +++++++++---- dokimi/research/records/decision/D017.md | 2 +- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md index bbd67e6d6..6072c0544 100644 --- a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md +++ b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md @@ -8,10 +8,12 @@ span each atom cites (on a `--include-resources` run, across the files in file order, so a file with few atoms may get no probe), so a capped rate estimates the whole input's per-atom rate instead of describing its opening. - The selection takes no seed: the same extracted atom list gives the same - sample. `novelty analyze` and `novelty atoms` (whose `atoms.json` `probe` and - `shorten` read) probe the same atoms. A cap below 1 is refused before any - call. New runs record `atoms.sampling` as `cap-spread:` and + Like any fixed stride it can fall in step with repeated, same-shaped + sections and miss or over-count what each one adds; for such an artifact, or + a rate that decides something, pass `--all-atoms`. The selection takes no + seed: the same extracted atom list gives the same sample. `novelty analyze` + and `novelty atoms` (whose `atoms.json` `probe` and `shorten` read) probe the + same atoms. A cap below 1 is refused before any call. New runs record `atoms.sampling` as `cap-spread:` and `extraction.reached` `null`; `extraction.sampled` is `{"atoms", "of", "rule"}` when the cap leaves atoms out, and `null` when it keeps every atom (M ≤ n). Only a cap that leaves atoms out sets `spans.capped_atom_set` and brings the diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 630ebd852..3ec9cda10 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -915,10 +915,15 @@ it yields M atoms, more than the cap n, the run probes the atoms at indices `floor((i + 0.5) * M / n)` for `i` in `0..n-1` of the extracted list sorted by the earliest span each atom cites (`spread_sample` in `dokimi/novelty/atoms.py`): every (M/n)-th atom in document order, starting in the middle of the first stride, so the sample estimates the per-atom rate an -uncapped run reports. The sort is needed because the extractor's reply lists a batch's atoms in -its own order; it is stable, so atoms citing the same span keep that order, and with M ≤ n -nothing is sorted and every atom is probed in extraction order. A stride needs no seed: the -same extracted atom list gives the same sample. A fresh extraction is new model calls, so the +uncapped run reports. Like any fixed stride, it can fall in step with a repeated structure: with +M = 100 and n = 25 it always takes atoms 2, 6, ..., 98 (from 0), so an input of four-atom +sections of the same shape, each with its new fact in the same place, can give a sampled rate +far from the whole input's, and re-extracting the same atoms gives the same sample, so it cannot +show the gap. For an artifact of repeated, same-shaped sections (a template per item), or a +rate that decides something, pass `--all-atoms`. The sort is needed because the extractor's +reply lists a batch's atoms in its own order; it is stable, so atoms citing the same span keep +that order, and with M ≤ n nothing is sorted and every atom is probed in extraction order. A +stride needs no seed: the same extracted atom list gives the same sample. A fresh extraction is new model calls, so the same text can give other atoms and so another sample; to probe identical atoms across runs or models, extract once with `novelty atoms` and pass its `atoms.json` to `novelty probe --atoms` (and `novelty shorten --atoms`), which probe the saved atoms without sampling again. The input diff --git a/dokimi/research/records/decision/D017.md b/dokimi/research/records/decision/D017.md index c948b89a9..88bd29aa1 100644 --- a/dokimi/research/records/decision/D017.md +++ b/dokimi/research/records/decision/D017.md @@ -16,6 +16,6 @@ SPDX-License-Identifier: Apache-2.0 ## Rationale A cap that takes atoms in document order measures the opening of the input, not the input. At `--tier triage` on the dpnp repository's agent guidance, the 25 capped atoms of the root `AGENTS.md` (61 spans) came from spans s0002 to s0012, about a quarter of its tokens, and those of `dpnp/tests/AGENTS.md` (32 spans) from s0003 to s0013, about a third. Two artifacts' capped rates then describe different shares of their text, and a file's rate moves when its sections are reordered. -A capped run now extracts every span and, of the M atoms extracted, probes the n at indices `floor((i + 0.5) * M / n)` of the atoms in document order, by the earliest span each cites (on a bundle run, in file order). A per-atom stride estimates the same per-atom rate an uncapped run reports. Equal token strata would not: they would weight a sparse introduction like a dense rule list. The stride needs no seed, so the same extracted atom list gives the same sample; a fresh extraction is new model calls and can give other atoms, so runs that must probe identical atoms reuse one `atoms.json` through `novelty probe --atoms`. Probing stays at the cap; extraction grows to every batch of the input. +A capped run now extracts every span and, of the M atoms extracted, probes the n at indices `floor((i + 0.5) * M / n)` of the atoms in document order, by the earliest span each cites (on a bundle run, in file order). A per-atom stride estimates the same per-atom rate an uncapped run reports. Equal token strata would not: they would weight a sparse introduction like a dense rule list. Like any fixed stride, it can fall in step with a repeated structure: with M = 100 and n = 25 it always takes atoms 2, 6, ..., 98 (from 0), so an input of four-atom sections of the same shape, each with its new fact in the same place, can give a sampled rate far from the whole input's, and re-extracting the same atoms cannot show the gap. A run on an artifact of repeated, same-shaped sections, or whose rate decides something, passes `--all-atoms`. The stride needs no seed, so the same extracted atom list gives the same sample; a fresh extraction is new model calls and can give other atoms, so runs that must probe identical atoms reuse one `atoms.json` through `novelty probe --atoms`. Probing stays at the cap; extraction grows to every batch of the input. The tier profiles (the call and token figures `--estimate-only` is fitted and checked on, and the per-tier call counts in `docs/skill-novelty.md`) were measured with the document-order cap. Which atoms are probed changes only for artifacts with more atoms than a tier's cap (25 at triage, 60 at standard, 120 at study); at or below the cap every atom is probed, as before. Extraction cost changes for more runs: any run whose cap was reached before its last batch, an artifact whose atoms exactly meet the cap included, now asks the remaining batches. Re-measuring those runs, and refitting the extraction profile to a run that asks every batch, is follow-up work this decision does not do. From 82d576a3d2f1ecc56bf090a45547424f197c7338 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 10:44:09 +0000 Subject: [PATCH 5/8] fix(novelty): label a run cap-spread only when the cap left atoms out Review of #358 (comment 4438896876): `atoms.sampling` came from `max_atoms` alone, so a run whose atoms met or fell under the cap (`extraction.sampled` null, every atom probed) still read `cap-spread:`. A run with 10 atoms under the triage cap of 25 was labelled `cap-spread:25`. `analyze` now derives the label from the same state as the forfeits and `spans.capped_atom_set`: `cap-spread:` when `extraction.sampled` is set, `full` otherwise. Readers of the label only print or copy it, so such a run renders as a full one. The schema description, the cap passage of docs/skill-novelty.md and the changelog fragment say when each label is written. The review also named the `sampled` forfeit, which called `added` a rate over "the atoms the cap kept, not the whole input", although a spread run reads every span. It now says the rate is over the sample of extracted atoms the cap kept, not over every atom; the docs quote and the settings-parity test follow. Tests: the cap-met run is labelled `full`; a 10-span input under cap 25 is labelled `full` with `sampled` null and renders "10 atoms (full)"; the cap-met/cap-binds pair pins both labels. Signed-off-by: Anton Volkov --- ...01_120000_antonwolfy_novelty-cap-spread.md | 7 ++-- dokimi/docs/skill-novelty.md | 6 ++- dokimi/dokimi/novelty/analyze.py | 3 +- dokimi/dokimi/novelty/tiers.py | 6 +-- dokimi/dokimi/schemas/novelty.v1.json | 2 +- dokimi/tests/test_novelty_analyze.py | 39 +++++++++++++++++-- dokimi/tests/test_novelty_settings_parity.py | 3 +- 7 files changed, 52 insertions(+), 14 deletions(-) diff --git a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md index 6072c0544..5df973396 100644 --- a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md +++ b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md @@ -13,9 +13,10 @@ a rate that decides something, pass `--all-atoms`. The selection takes no seed: the same extracted atom list gives the same sample. `novelty analyze` and `novelty atoms` (whose `atoms.json` `probe` and `shorten` read) probe the - same atoms. A cap below 1 is refused before any call. New runs record `atoms.sampling` as `cap-spread:` and - `extraction.reached` `null`; `extraction.sampled` is `{"atoms", "of", "rule"}` - when the cap leaves atoms out, and `null` when it keeps every atom (M ≤ n). + same atoms. A cap below 1 is refused before any call. New runs record + `extraction.reached` `null`; when the cap leaves atoms out, `atoms.sampling` + is `cap-spread:` and `extraction.sampled` `{"atoms", "of", "rule"}`, and + when it keeps every atom (M ≤ n), `full` and `null`. Only a cap that leaves atoms out sets `spans.capped_atom_set` and brings the cap's forfeits: an input whose atoms exactly meet the cap no longer counts as capped. The cap warning, the `novelty.md` header and the headline's diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 3ec9cda10..b00c7190c 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -899,7 +899,8 @@ Each tier carries the **claims it forfeits**, and those strings are written into run used and the atoms it extracted, not the tier's name: `--no-verify` adds the faithfulness forfeit, and `--all-atoms` or a skill with no more atoms than the cap (which keeps every atom: `extraction.sampled` is `null` and `spans.capped_atom_set` false) drops the -cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and +cap's ones (that `added` is a rate over the sample of extracted atoms the cap kept, not over +every atom, and that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on which atoms the cap kept (`--estimate-only` lists them before it is known), and neither does the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). With shortening on, a @@ -934,7 +935,8 @@ atoms can fall between two sampled indices and get no probe; a higher cap spread without promising each file one, and `--all-atoms` probes every extracted atom. A bundled file left out has no atom in it (the `coverage:` line says how much of the artifact went into the input). The run records `atoms.sampling` as `cap-spread:` and `extraction.sampled` as -`{"atoms": n, "of": M, "rule": "cap-spread"}`. When every span was read it warns `atom +`{"atoms": n, "of": M, "rule": "cap-spread"}`; a run whose cap kept every atom (M ≤ n) records +`full`, as an uncapped run does, and `null`. When every span was read it warns `atom extraction read the whole input, and the atom cap probes 25 of the input's 61 atoms, spread over all of it (a stride of 2.44 atoms in document order): rates are estimated from that sample; pass --all-atoms to probe every atom`. When some spans were left unread (`extraction.unread_span_ids`: diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index b30a516cb..32d21b560 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -181,8 +181,9 @@ def analyze( # (M <= n, so `sampled` is unset), changes which claims are forfeit. capped = extraction.sampled is not None warnings.extend(settings.warnings(sampled=capped)) + # A label for a sample only when the cap left atoms out; a cap the atoms meet is a full run. # Older runs recorded `cap-per-artifact:`, the first n atoms in document order (#293). - sampling = f"cap-spread:{config.max_atoms}" if config.max_atoms is not None else "full" + sampling = f"cap-spread:{config.max_atoms}" if capped else "full" if extraction.sampled: # set only when the cap left atoms out warnings.append(report.cap_warning(extraction.as_dict())) if not atom_list: diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 20b23a567..8180dd284 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -28,9 +28,9 @@ #: out, which probes atoms spread over the input: https://github.com/intel/dokimi/issues/293 FORFEITS: dict[str, list[str]] = { "sampled": [ - "`added` is a rate over the atoms the cap kept, not the whole input: usable as a " - "sign, not as a figure, and not comparable with another artifact's; pass " - "--all-atoms for the full sweep", + "`added` is a rate over the sample of extracted atoms the cap kept, not over every " + "atom: usable as a sign, not as a figure, and not comparable with another " + "artifact's; pass --all-atoms for the full sweep", "`frac_carrying_nothing` is not interpretable on a capped atom set (0.064 read " "as 0.743 on the study's largest artifact)", ], diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index 175a376fa..8f2a62bad 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -71,7 +71,7 @@ }, "sampling": { "type": "string", - "description": "How the probed subset was chosen: full, or cap-spread: (e.g. cap-spread:120), n atoms spread over every extracted atom in document order (extraction.sampled). Runs before cap-spread wrote cap-per-artifact:, the first n atoms in document order. Present so a capped run is never read as a full sweep." + "description": "How the probed subset was chosen: full, every extracted atom (also a cap the atoms meet or fall under), or cap-spread: (e.g. cap-spread:120), only when the cap left atoms out: n atoms spread over every extracted atom in document order (extraction.sampled). Runs before cap-spread wrote cap-per-artifact:, the first n atoms in document order. Present so a capped run is never read as a full sweep." }, "faithfulness": { "type": ["object", "null"], diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 03d1589dd..edacb94a3 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -610,7 +610,7 @@ def test_a_tier_writes_its_forfeits_into_the_run_it_produced(tmp_path): assert len(forfeits) == len(TIERS["triage"]["forfeits"]) - len(FORFEITS["sampled"]) assert sorted(artifact["run_config"]["tier"]["forfeits"]) == sorted( w.split("forfeits: ", 1)[1] for w in forfeits) - assert not any("rate over the atoms the cap kept" in w for w in forfeits) + assert not any("atoms the cap kept" in w for w in forfeits) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) @@ -667,15 +667,47 @@ def test_an_artifact_written_under_the_document_order_cap_still_validates(tmp_pa def test_a_cap_the_atoms_meet_samples_nothing(tmp_path): - # Two atoms under a cap of 2: every atom is probed, exactly as with no cap. + # Two atoms under a cap of 2: every atom is probed, exactly as with no cap, and the label + # says so: `cap-spread:` names a sample only when the cap left atoms out (#358). artifact = _analyze(tmp_path, max_atoms=2).artifact - assert artifact["atoms"]["n"] == 2 and artifact["atoms"]["sampling"] == "cap-spread:2" + assert artifact["atoms"]["n"] == 2 and artifact["atoms"]["sampling"] == "full" assert artifact["extraction"]["sampled"] is None assert not any("atom cap probes" in w for w in artifact["warnings"]) assert "atom cap" not in render_novelty_markdown(artifact) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) +def test_a_cap_above_the_atom_count_labels_the_run_full(tmp_path): + # Ten spans, one atom each, under the triage cap of 25: nothing is left out, so the run is + # a full one in its label as in `extraction.sampled` and the report header. + path = tmp_path / "ten" / "SKILL.md" + path.parent.mkdir(parents=True) + rules = "\n\n".join(f"Rule {i}: set `--opt{i}` to {i}." for i in range(10)) + path.write_text(f"---\nname: ten\ndescription: Ten rules\n---\n\n{rules}\n", encoding="utf-8") + kind, name, description, frontmatter, body = load_artifact_text(f"skill:{path}") + scripted = _scripted("informed") + + def per_span(prompt, system): + if "Extract the checkable knowledge" not in prompt: + return scripted(prompt, system) + return json.dumps([dict(_ATOMS[0], statement=f"{sid} holds.", question=f"What of {sid}?", + span_ids=[sid]) + for sid in re.findall(r"^\[(s\d+)\]", prompt, flags=re.M)]), None + + config = AnalyzeConfig(artifact_ref=f"skill:{path}", artifact_kind=kind, name=name, + description=description, frontmatter=frontmatter, concurrency=1, + max_atoms=25) + _, answerer, judge = _clients() + extractor = LLMClient(model="extractor-m", role="extractor", transport=per_span) + artifact = analyze(body, config, extractor=extractor, answerer=answerer, + judge=judge).artifact + assert artifact["atoms"]["n"] == 10 + assert artifact["atoms"]["sampling"] == "full" + assert artifact["extraction"]["sampled"] is None + assert "10 atoms (full), 10 probed." in render_novelty_markdown(artifact) + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + + @pytest.mark.parametrize("cap, capped", [(2, False), (1, True)], ids=["cap-met", "cap-binds"]) def test_only_a_cap_that_leaves_atoms_out_counts_as_capped(tmp_path, cap, capped): # Two atoms at `study`: a cap they meet keeps them all, so no cap forfeit and no flag; @@ -686,6 +718,7 @@ def test_only_a_cap_that_leaves_atoms_out_counts_as_capped(tmp_path, cap, capped artifact = _analyze(tmp_path, **settings.config_fields()).artifact assert (artifact["extraction"]["sampled"] is not None) is capped assert artifact["spans"]["capped_atom_set"] is capped + assert artifact["atoms"]["sampling"] == (f"cap-spread:{cap}" if capped else "full") shortening = (f"shortening candidates are scored against a spread sample of {cap} of the " "extracted atoms: a cut of a span with no sampled atom looks free") warned = "\n".join(artifact["warnings"]) diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 32bc21e4a..ba575660a 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -270,7 +270,8 @@ def _refusal_as_exit_2(step: dict[str, Any]) -> dict[str, Any]: # Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier # record, the warnings and the estimate's stderr. CAP_FORFEIT_ON_MAIN = "`added` is a sampled rate over the atom cap: usable" -CAP_FORFEIT = "`added` is a rate over the atoms the cap kept, not the whole input: usable" +CAP_FORFEIT = ("`added` is a rate over the sample of extracted atoms the cap kept, not over every " + "atom: usable") # Intended change (#293): shortening is scored against a spread sample, not the opening. The # forfeit names the cap, which tests/test_novelty_tiers.py pins; here both sides read . SHORTEN_FORFEIT_ON_MAIN = ("shortening candidates are scored only against the capped atoms, i.e. " From 285e29e0b7aa132e6fba29c067f2311868c8a95c Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 19:00:21 +0000 Subject: [PATCH 6/8] fix(novelty): refuse a bad atom cap before touching the spans extract_atoms checked the cap after the loop that resets every span's unread mark, so a cap below 1 raised before any model call but after it had changed the caller's spans. The check now runs first, and the cap-0 test asserts the spans keep the unread marks they came with. Settles the review comment on intel/dokimi#358 that _check_cap runs after the side effect it is meant to precede. Signed-off-by: Anton Volkov --- dokimi/dokimi/novelty/atoms.py | 4 ++-- dokimi/tests/test_novelty_atoms.py | 7 +++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index 65b4df374..ee564cdae 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -325,7 +325,7 @@ def extract_atoms( affordable; too large and the model starts skipping spans. cap: Keep this many atoms, spread over the extracted atoms in document order (:func:`spread_sample`); every span is extracted either way. At least 1: a - smaller cap raises ``ValueError`` before any call. Recorded in the artifact as + smaller cap raises ``ValueError`` before any call or change to ``spans``. Recorded in the artifact as the sampling rule, so a capped run is never quoted as a full sweep. report: Filled in place with what could not be read and what re-asking cost, and what a binding cap kept (``sampled``). Spans no call could read are also @@ -339,6 +339,7 @@ def extract_atoms( """ from dokimi.llm import extract_json_array + _check_cap(cap) # before any call, and before the spans' unread marks are reset report = report if report is not None else ExtractionReport() valid_ids = {span.id for span in spans} @@ -360,7 +361,6 @@ def parse(reply: str) -> list: (spans[start:start + batch_spans], False) for start in range(0, len(spans), batch_spans) ] - _check_cap(cap) # before any call while pending: batch, is_half = pending.pop(0) if not is_half: diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 62fc3e2d1..aed6790f4 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -266,16 +266,19 @@ def out_of_order(prompt, system): assert [atom.span_ids for atom in atoms] == probed -def test_a_cap_below_one_is_refused_before_any_call(): +def test_a_cap_below_one_is_refused_before_any_call_or_change_to_the_spans(): calls: list[str] = [] def counted(prompt, system): calls.append(prompt) return _three_per_span(prompt, system) + spans = segment(DOC) + spans[0].unread = True # the caller's marks, which a refused cap must leave as they were with pytest.raises(ValueError, match="the atom cap must be at least 1"): - extract_atoms(segment(DOC), client=LLMClient(model="x", transport=counted), cap=0) + extract_atoms(spans, client=LLMClient(model="x", transport=counted), cap=0) assert calls == [] + assert [span.unread for span in spans] == [True] + [False] * (len(spans) - 1) def test_a_binding_cap_over_a_partly_read_input_makes_no_whole_input_claim(): From 3606282e45eff6971cadc8fd627ab0b6048f55ed Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 19:02:24 +0000 Subject: [PATCH 7/8] fix(novelty): estimate extraction calls from the span batches --estimate-only counted extraction as one call per 12 atoms expected before the cap, and the token rule expects one atom per 23 tokens. For triage on 300 spans and 19,000 tokens that gave 69 extraction calls, while the run asks one call per 25-span batch: 12. estimate_calls now takes n_spans in place of n_extracted and counts ceil(n_spans / BATCH_SPANS) extraction calls, with BATCH_SPANS the default batch size of extract_atoms, now a shared constant. RunSettings.estimate passes the span count. Without n_spans the old per-12-atoms rule stays, which is how the held-out calibration runs, whose extraction stopped at the cap, are projected. The per-call token profiles are unchanged. Settles the review comment on intel/dokimi#358 that the estimate counts extraction calls from the pre-cap atom guess. Signed-off-by: Anton Volkov --- ...01_120000_antonwolfy_novelty-cap-spread.md | 2 +- dokimi/docs/skill-novelty.md | 3 ++- dokimi/dokimi/novelty/atoms.py | 6 ++++- dokimi/dokimi/novelty/tiers.py | 24 ++++++++++++------- dokimi/tests/test_novelty_tiers.py | 17 +++++++++---- 5 files changed, 36 insertions(+), 16 deletions(-) diff --git a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md index 5df973396..0102a9a72 100644 --- a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md +++ b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md @@ -26,7 +26,7 @@ with `cap-per-artifact:` and a `reached` record still validate and render as before. Probing still stops at the cap, but a run whose cap was reached before its last extraction batch now asks the remaining batches, and - `--estimate-only` counts them; the tier profiles were measured with the old + `--estimate-only` counts them, one call per batch of 25 spans; the tier profiles were measured with the old order (decision D017). The shortening forfeit now says candidates are scored against a spread sample of the cap's atoms, so cutting a span with no sampled atom looks free. diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index b00c7190c..87ac03d14 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -887,7 +887,8 @@ per answerer call than a two-arm one; a `--placebo-text` of another length is no three of the 12 are standard runs, so the standard profile rests on little data. The measured runs, the tier table above among them, predate the spread cap: their extraction stopped at the cap. A capped run now asks every extraction batch, and the projection counts -those batches over the atoms expected before the cap, while the probes stay at the cap. On an +one extraction call per batch of 25 spans, as the run asks them, while the probes stay at the +cap. On an artifact with more atoms than the cap that adds extractor calls no measured run has checked yet; the held-out figures above are for the runs as they were measured. An explicit flag always beats the tier, and the override is recorded — a diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index ee564cdae..8ceae31cc 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -35,6 +35,10 @@ "rare_critical", "procedural", "routing", "evidence", ) +#: Spans per extraction call when the caller does not say; ``tiers.estimate_calls`` counts +#: extraction calls with it, one per batch of this many spans. +BATCH_SPANS = 25 + _EXTRACT_SYSTEM = ( "You extract checkable knowledge from technical documents. You never invent facts " "that the text does not assert, and you never soften a claim the text makes." @@ -300,7 +304,7 @@ def extract_atoms( spans: list[Span], *, client: LLMClient, - batch_spans: int = 25, + batch_spans: int = BATCH_SPANS, cap: int | None = None, report: ExtractionReport | None = None, ) -> list[Atom]: diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 8180dd284..52e223c18 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -21,9 +21,12 @@ from __future__ import annotations +import math from dataclasses import dataclass, field from typing import Any +from .atoms import BATCH_SPANS + #: What a run gives up, keyed by the setting that is off. "sampled" is an atom cap that left atoms #: out, which probes atoms spread over the input: https://github.com/intel/dokimi/issues/293 FORFEITS: dict[str, list[str]] = { @@ -215,8 +218,9 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, The atom count is the larger of one atom per two spans and one per :data:`TOKENS_PER_ATOM` tokens (spans alone undercount: short spans, such as table - rows, yield an atom each), capped at ``max_atoms`` for every stage but extraction, - which reads the whole input. Without ``n_tokens`` only the span rule applies. The + rows, yield an atom each), capped at ``max_atoms``. Extraction reads the whole input + whatever the cap, so its calls are the span batches it asks (see + :func:`estimate_calls`). Without ``n_tokens`` only the span rule applies. The forfeits assume the cap is reached, since the atom count is a guess until extraction runs: an artifact with no more atoms than the cap keeps them all and drops the cap's forfeits. @@ -233,7 +237,7 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, # The full resolved settings, so the projection is the model `estimate_calls` # implements rather than a subset of it. "estimate": estimate_calls(n_atoms, dict(self.values, budgets=tuple(budgets)), - n_arms=n_arms, n_extracted=expected), + n_arms=n_arms, n_spans=n_spans), "tier": self.tier_record(), "caveat": ( "forfeits assume the atom cap is reached; the run recomputes them from the " @@ -377,7 +381,7 @@ def estimate_usd( def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2, - n_extracted: int | None = None) -> dict: + n_spans: int | None = None) -> dict: """Project the model-call bill before spending it. From the instrumented runs: every probe, reversal included, costs an answer and a grade @@ -395,8 +399,11 @@ def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2, n_atoms: Atoms expected after the cap. settings: Resolved tier settings (``paraphrases``, ``reversals``, stage switches). n_arms: Arms in the run, including the baseline. - n_extracted: Atoms expected before the cap (default ``n_atoms``). Extraction asks - every batch whatever the cap, so its calls follow this count. + n_spans: Spans of the input. Extraction asks every batch of + :data:`~dokimi.novelty.atoms.BATCH_SPANS` spans whatever the cap, so it makes + ``ceil(n_spans / BATCH_SPANS)`` calls (splits of an unparseable reply aside). + Without it, one call per 12 atoms of ``n_atoms``: the rule of the measured runs, + whose extraction stopped at the cap. Returns: ``{"n_atoms", "paired_probes", "reversal_probes", "probe_calls", "stage_calls", @@ -415,9 +422,10 @@ def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2, # Reversals are graded in every arm too: the baseline arm's resistance to a planted # inversion is a reported control, not a by-product. probe_calls = (paired + reversals) * n_arms * 2 - extracted = n_atoms if n_extracted is None else n_extracted + batches = (max(1, round(n_atoms / 12)) if n_spans is None + else math.ceil(n_spans / BATCH_SPANS)) stage_calls = ( - max(1, round(extracted / 12)) # extraction batches + batches # extraction batches + (n_atoms if settings.get("verify_faithfulness") else 0) # one per atom + (1 if settings.get("paraphrases") else 0) + (1 if settings.get("reversals") else 0) diff --git a/dokimi/tests/test_novelty_tiers.py b/dokimi/tests/test_novelty_tiers.py index 9e2d319c6..0eb51da3d 100644 --- a/dokimi/tests/test_novelty_tiers.py +++ b/dokimi/tests/test_novelty_tiers.py @@ -321,13 +321,10 @@ def test_the_estimate_caps_the_expected_atoms_and_records_the_tier(): study = resolve_settings("study", {}) bill = study.estimate(300, budgets=(0.5,), n_arms=3) assert (bill["spans"], bill["atom_cap"]) == (300, 120) - # Probing stops at the cap; extraction reads all 150 expected atoms' spans. + # Probing stops at the cap; extraction asks every batch of the 300 spans. assert bill["estimate"] == estimate_calls( - 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3, n_extracted=150) + 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3, n_spans=300) assert bill["estimate"]["n_atoms"] == 120 - # 12 extraction batches (150 / 12, rounded half to even) where the cap's 120 atoms take 10. - assert bill["estimate"]["stage_calls"] - estimate_calls( - 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3)["stage_calls"] == 2 assert bill["tier"] == {"name": "study", "forfeits": TIERS["study"]["forfeits"], "overrides": {}} assert "cap is reached" in bill["caveat"] @@ -337,6 +334,16 @@ def test_the_estimate_caps_the_expected_atoms_and_records_the_tier(): assert (sweep["caveat"], sweep["tier"]) == (None, None) +@pytest.mark.parametrize(("spans", "tokens", "batches"), [(300, 19000, 12), (26, None, 2), + (25, None, 1)]) +def test_the_estimate_counts_one_extraction_call_per_span_batch(spans, tokens, batches): + """A capped triage run asks ceil(spans / 25) extraction calls, not one per 12 atoms + expected before the cap (826 at 19,000 tokens, which counted 69).""" + bill = resolve_settings("triage", {}).estimate(spans, n_tokens=tokens)["estimate"] + # Triage runs no stage but extraction and the need graph. + assert bill["stage_calls"] == bill["by_role"]["extractor"] == batches + 1 + + def test_the_warnings_follow_the_settings_and_the_atom_count(): standard = resolve_settings("standard", {"no_verify": True}) warned = standard.warnings(sampled=True) From 9f8bdf84f904b678c89a1f04de06ee055f4d99c9 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 20:08:14 +0100 Subject: [PATCH 8/8] docs(research): D017 links the follow-up for spans indexed from the sample A capped run extracts every span but indexes spans, writes atoms.json and shortens from the sampled atoms only. D017 now names that gap and links the issue that tracks it. Settles the review comment on intel/dokimi#358 that asked for the issue to be linked from D017. Signed-off-by: Anton Volkov --- dokimi/research/records/decision/D017.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/dokimi/research/records/decision/D017.md b/dokimi/research/records/decision/D017.md index 88bd29aa1..1db6ab037 100644 --- a/dokimi/research/records/decision/D017.md +++ b/dokimi/research/records/decision/D017.md @@ -8,6 +8,7 @@ ideas: state: accepted links: - https://github.com/intel/dokimi/issues/293 +- https://github.com/intel/dokimi/issues/371 ---