diff --git a/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md new file mode 100644 index 000000000..0102a9a72 --- /dev/null +++ b/dokimi/changelog.d/20261101_120000_antonwolfy_novelty-cap-spread.md @@ -0,0 +1,32 @@ +### Changed + +- **A capped novelty run probes atoms spread over the whole input** — under + `--max-atoms` or a tier's cap (25 at `triage`, 60 at `standard`, 120 at + `study`), extraction no longer stops at the cap: it reads every span, and of + the M atoms it extracts the run probes n at indices + `floor((i + 0.5) * M / n)`, every (M/n)-th in document order by the earliest + span each atom cites (on a `--include-resources` run, across the files in + file order, so a file with few atoms may get no probe), so a capped rate + estimates the whole input's per-atom rate instead of describing its opening. + Like any fixed stride it can fall in step with repeated, same-shaped + sections and miss or over-count what each one adds; for such an artifact, or + a rate that decides something, pass `--all-atoms`. The selection takes no + seed: the same extracted atom list gives the same sample. `novelty analyze` + and `novelty atoms` (whose `atoms.json` `probe` and `shorten` read) probe the + same atoms. A cap below 1 is refused before any call. New runs record + `extraction.reached` `null`; when the cap leaves atoms out, `atoms.sampling` + is `cap-spread:` and `extraction.sampled` `{"atoms", "of", "rule"}`, and + when it keeps every atom (M ≤ n), `full` and `null`. + Only a cap that leaves atoms out sets `spans.capped_atom_set` and brings the + cap's forfeits: an input whose atoms exactly meet the cap no longer counts as + capped. The cap warning, the `novelty.md` header and the headline's + `coverage:` line say how many of the input's atoms were probed and with what + stride, or, when some spans went unread, that the sample is spread over the + atoms of the spans that were read and how many were not. Artifacts written + with `cap-per-artifact:` and a `reached` record still validate and render + as before. Probing still stops at the cap, but a run whose cap was reached + before its last extraction batch now asks the remaining batches, and + `--estimate-only` counts them, one call per batch of 25 spans; the tier profiles were measured with the old + order (decision D017). The + shortening forfeit now says candidates are scored against a spread sample of + the cap's atoms, so cutting a span with no sampled atom looks free. diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 6fa1a3ff1..18333de73 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -83,13 +83,15 @@ self-judge; `arms run` does not check, so name times arms, or tasks times harnesses. One measured example: triage on a 19k-token skill with a mid-size model was 103 calls, about $1.80; `study` on the same skill was 1,900 calls, about $14.50. -- **At triage** with `--include-resources`, the 25-atom cap takes atoms in - document order, and the entry point comes first: the flag reaches - `references/` only if `SKILL.md` has fewer than 25 atoms, and the run's cap - warning prints how far it reached through the input, the entry point and the files the budgets selected (span, - spans and token share). For such skills, - use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for - a full sweep; run `--estimate-only` first, it costs more. +- **At triage** with `--include-resources`, the 25-atom cap strides over the + concatenated atom list of the files the budgets selected, in file order, so + `references/` can be probed, but a file with few atoms may get no probe: with + one atom in each of 100 files, triage probes 25 of them. The run's cap warning + says how many of the input's atoms it probed. Extraction still reads every + file, so a large bundle costs more extraction calls than its 25 probes suggest. + `--tier standard` (60 atoms) spreads more probes without promising each file + one; add `--all-atoms` to the tier command to probe every extracted atom, and + run `--estimate-only` first, it costs more. - Step 5 needs evaluation content: a golden question set for 5a, executable tasks for 5b. `arms run` takes no tasks. That content lives outside this repository ([external-content.md](external-content.md)). Without it, the diff --git a/dokimi/docs/skill-novelty-walkthrough.md b/dokimi/docs/skill-novelty-walkthrough.md index 5ad273f62..21ffa9195 100644 --- a/dokimi/docs/skill-novelty-walkthrough.md +++ b/dokimi/docs/skill-novelty-walkthrough.md @@ -94,9 +94,10 @@ comparable with another artifact's. If the skill is a thin `SKILL.md` over a `references/` tree, add `--include-resources`. Without it the run measures the entry point and prints the share it is skipping — in the -study corpus that share was 0.72 of all words. At `triage` the 25-atom cap takes atoms in -document order and the entry point comes first, so the flag reaches `references/` only if -`SKILL.md` has fewer than 25 atoms; otherwise use `--tier standard` or higher. +study corpus that share was 0.72 of all words. At `triage` the 25-atom cap strides over the +concatenated atom list of the selected files, in file order, so a file with few atoms may get no +probe. A higher tier spreads more probes without promising each file one; `--all-atoms` probes +every extracted atom. ## 4–5. Extract once, probe per model diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 0894076c7..87ac03d14 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -131,10 +131,10 @@ marked. Read it as a review queue: - **Internal only.** It says the two statements disagree, not which is wrong. That needs a primary source, which this layer does not consult. - **Lower bound.** Only atoms in the same call are compared. `analyze` sends the whole - extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared + probed atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared unless a call fails or its reply has to be split. The artifact records the share as `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of - Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which are only the ones the cap kept when + Y atom pairs` when it is short. `pairs_total` is over the probed atoms, which are only the ones the cap kept when `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped; otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A @@ -885,6 +885,12 @@ tokens: above that the dollar figure extrapolates, and the payload says so arm that carries the artifact (every arm but `no_artifact`), so a three-arm run costs more per answerer call than a two-arm one; a `--placebo-text` of another length is not read. Only three of the 12 are standard runs, so the standard profile rests on little data. +The measured runs, the tier table above among them, predate the spread cap: their extraction +stopped at the cap. A capped run now asks every extraction batch, and the projection counts +one extraction call per batch of 25 spans, as the run asks them, while the probes stay at the +cap. On an +artifact with more atoms than the cap that adds extractor calls no measured run has checked +yet; the held-out figures above are for the runs as they were measured. An explicit flag always beats the tier, and the override is recorded — a `--tier triage --max-atoms 90` run is not a stock triage run and does not get to be read as one. @@ -892,36 +898,66 @@ read as one. Each tier carries the **claims it forfeits**, and those strings are written into `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the run used and the atoms it extracted, not the tier's name: `--no-verify` adds the -faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the -cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and +faithfulness forfeit, and `--all-atoms` or a skill with no more atoms than the cap (which +keeps every atom: `extraction.sampled` is `null` and `spans.capped_atom_set` false) drops the +cap's ones (that `added` is a rate over the sample of extracted atoms the cap kept, not over +every atom, and that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on -how far the cap reached (`--estimate-only` lists them before it is known), and neither does -the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). -`--estimate-only` cannot know the atom count yet, so it lists them: +which atoms the cap kept (`--estimate-only` lists them before it is known), and neither does +the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). With shortening on, a +cap that left atoms out adds that `shortening candidates are scored against a spread sample of 60 of the +extracted atoms: a cut of a span with no sampled atom looks free` (60 at `standard`, 120 at +`study`). `--estimate-only` cannot know the atom count yet, so it lists them: > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected > against the model matching the artifact's own wording -The cap is not a sample: extraction walks the spans in document order and stops at it, so a -capped run measures the input up to the span it stopped at. The input is the text that was segmented: the -entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file -left out is in none of the reach's spans or tokens (the `coverage:` line says how much of the -artifact went into the input). The warning says how far the cap reached, as in `atom -extraction hit the cap of 25 at span s0012 (12 of the input's 61 spans, 23% of its tokens): -rates describe the opening of the input, not all of it; pass --all-atoms for the full sweep`, -`extraction.reached` records it, and the headline's `coverage:` line then says the words -went into the input and the capped atoms come from the first 23% of the input's tokens (the -warning under that line, when half or less of the words went in, says "went into the input" -too). `novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its -`atoms.json`, do not repeat it. - -A cap met on the input's last span leaves no stretch of the input out: what it cut is any atoms -past the cap in that last reply. The warning then reads `atom extraction hit the cap of 30 at -span s0010 (10 of the input's 10 spans, 100% of its tokens): any atoms past the cap in its last -reply were dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the -atoms reach the input's last span and the cap dropped any atoms past it in the last reply, and -the `coverage:` line ends `the capped atoms reach the input's last span, and the cap dropped -any atoms past it`; none of them says "the opening". +The cap is a spread sample, not the opening. Extraction reads every span of the input, and when +it yields M atoms, more than the cap n, the run probes the atoms at indices +`floor((i + 0.5) * M / n)` for `i` in `0..n-1` of the extracted list sorted by the earliest span +each atom cites (`spread_sample` in `dokimi/novelty/atoms.py`): every (M/n)-th atom in document +order, starting in the middle of the first stride, so the sample estimates the per-atom rate an +uncapped run reports. Like any fixed stride, it can fall in step with a repeated structure: with +M = 100 and n = 25 it always takes atoms 2, 6, ..., 98 (from 0), so an input of four-atom +sections of the same shape, each with its new fact in the same place, can give a sampled rate +far from the whole input's, and re-extracting the same atoms gives the same sample, so it cannot +show the gap. For an artifact of repeated, same-shaped sections (a template per item), or a +rate that decides something, pass `--all-atoms`. The sort is needed because the extractor's +reply lists a batch's atoms in its own order; it is stable, so atoms citing the same span keep +that order, and with M ≤ n nothing is sorted and every atom is probed in extraction order. A +stride needs no seed: the same extracted atom list gives the same sample. A fresh extraction is new model calls, so the +same text can give other atoms and so another sample; to probe identical atoms across runs or +models, extract once with `novelty atoms` and pass its `atoms.json` to `novelty probe --atoms` +(and `novelty shorten --atoms`), which probe the saved atoms without sampling again. The input +is the text that was segmented: the entry point, plus on a bundle run the files its roles and +budgets selected, in that order. The stride runs over the concatenated atom list of those +files, so a file's expected share of the probes is its share of the atoms, but a file with few +atoms can fall between two sampled indices and get no probe; a higher cap spreads more probes +without promising each file one, and `--all-atoms` probes every extracted atom. A bundled file +left out has no atom in it (the `coverage:` line says how much of the artifact went into the +input). The run records `atoms.sampling` as `cap-spread:` and `extraction.sampled` as +`{"atoms": n, "of": M, "rule": "cap-spread"}`; a run whose cap kept every atom (M ≤ n) records +`full`, as an uncapped run does, and `null`. When every span was read it warns `atom +extraction read the whole input, and the atom cap probes 25 of the input's 61 atoms, spread over +all of it (a stride of 2.44 atoms in document order): rates are estimated from that sample; pass +--all-atoms to probe every atom`. When some spans were left unread (`extraction.unread_span_ids`: +a failed call, an empty or unparseable reply), M counts only the atoms of the spans that were +read, and the warning says so instead: `atom extraction could not read 25 of the input's spans, +and the atom cap probes 2 of the 25 atoms extracted from the spans that were read, spread over +those (a stride of 12.5 atoms in document order; 25 of the input's spans were not read, see +extraction.unread_span_ids): ...`. The `novelty.md` header says the same, and the headline's +`coverage:` line says the words went into the input and the cap probed 25 of the input's 61 +atoms, or of the atoms extracted from the spans that were read (the warning under that line, +when half or less of the words went in, says "went into the input" too). `novelty atoms` prints +the same warning on stderr and saves the same sample in `atoms.json`; `probe` and `shorten`, +which read it, do not repeat the warning. + +Runs written before the spread rule recorded `cap-per-artifact:`: extraction walked the spans +in document order and stopped at the cap, so their atoms cover the input up to the span it +stopped at, and `extraction.reached` says how far (`extraction.sampled` is absent). Their +`novelty.md` header and `coverage:` line still say so, as in `the capped atoms come from the +first 23% of the input's tokens`, or, for a cap met on the input's last span, `the capped atoms +reach the input's last span, and the cap dropped any atoms past it`. That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a sweep is not. @@ -968,7 +1004,7 @@ at exactly 4096 tokens, cut mid-array. Extraction recovers by splitting an unpar all unusable) in half and re-asking, down to one span per call, so a truncation costs extra calls rather than atoms: up to 2n−1 calls for an n-span batch that never parses. The artifact's `extraction` block counts them (`calls` against `batches`, the calls the run would make with no splitting; a -capped run counts only the batches it reached), and a warning (`atom extraction re-asked N +capped run asks every batch too), and a warning (`atom extraction re-asked N unparseable batches as halves`) says what they cost; `--max-tokens 16000` avoids the truncated ones, not those with unusable rows. A call that fails or returns an empty reply is not split, since half a batch fails the same way. @@ -989,7 +1025,8 @@ lost to a failed call (with its error). The | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked | | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting | | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded | -| `reached` | on a capped run, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind | +| `sampled` | on a run whose cap left atoms out, what it probed: `atoms`, the cap; `of`, the atoms extracted from every span some call read; `rule`, `cap-spread`. `null` when every atom was probed | +| `reached` | `null` on a `cap-spread` run. On a run before it, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind | `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index ae625fd9c..61d90c49b 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -158,8 +158,8 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None: except ExtractionError as exc: raise SystemExit(str(exc)) from None # The cap warning analyze records; probe and shorten, which read atoms.json, do not repeat it. - reach = [cap_warning(cap, extraction.reached)] if extraction.reached else [] - for warning in extraction.warnings(len(spans)) + reach: + sampled = [cap_warning(extraction.as_dict())] if extraction.sampled else [] + for warning in extraction.warnings(len(spans)) + sampled: print(f"WARNING {warning}", file=sys.stderr) if settings.values["verify_faithfulness"]: verify_atoms(atoms, spans, client=extractor) @@ -167,7 +167,7 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None: save_atoms(atoms, spans, out, extractor=extractor.model, cap=cap, bundle=artifact_bundle.summary(), extraction=extraction) print(f"OK Saved {len(atoms)} atoms over {len(spans)} spans: {out}") - print(json.dumps(span_summary(spans, atoms, atom_cap=cap), indent=2)) + print(json.dumps(span_summary(spans, atoms, capped=extraction.sampled is not None), indent=2)) def cmd_novelty_probe(args: argparse.Namespace) -> None: @@ -948,7 +948,7 @@ def _add_common(parser: argparse.ArgumentParser, positive_int) -> None: "Unset by default so that passing the default value explicitly " "still overrides a --tier cap") parser.add_argument("--all-atoms", action="store_true", dest="all_atoms", - help="Extract every atom (opt-in: this is the expensive sweep)") + help="Probe every extracted atom (opt-in: this is the expensive sweep)") parser.add_argument("--budgets", type=float, nargs="+", default=[0.75, 0.5], help="Target token fractions for shortening candidates") parser.add_argument("--cache-dir", default=None, dest="cache_dir", @@ -1159,7 +1159,7 @@ def _print_coverage(bundle: dict, extraction: dict | None = None) -> None: The share is stated on stdout because the headline is what a user copies. A run under the floor also says so on stdout: a stderr-only warning is one a user piping the headline never sees, and a rate quoted off half-read skill is not a property of it. - A capped run says the words went into the input, and how far its atoms reach. + A capped run says the words went into the input, and how many of its atoms were probed. """ from dokimi.novelty.bundle import MEASURED_SHARE_WARN_AT, Coverage from dokimi.novelty.report import coverage_reach, floored_percent diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index 4621f1b64..32d21b560 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -177,14 +177,15 @@ def analyze( if tier: settings = replace(settings, tier=tier["name"], overrides=dict(tier.get("overrides") or {})) - # From what this run did, not the tier's text: an override or an atom set under the - # cap changes which claims are forfeit. - warnings.extend(settings.warnings(n_atoms=len(atom_list))) - sampling = ( - f"cap-per-artifact:{config.max_atoms}" if config.max_atoms is not None else "full" - ) - if extraction.reached and config.max_atoms is not None: # reached is set only under a cap - warnings.append(report.cap_warning(config.max_atoms, extraction.reached)) + # From what this run did, not the tier's text: an override, or a cap that kept every atom + # (M <= n, so `sampled` is unset), changes which claims are forfeit. + capped = extraction.sampled is not None + warnings.extend(settings.warnings(sampled=capped)) + # A label for a sample only when the cap left atoms out; a cap the atoms meet is a full run. + # Older runs recorded `cap-per-artifact:`, the first n atoms in document order (#293). + sampling = f"cap-spread:{config.max_atoms}" if capped else "full" + if extraction.sampled: # set only when the cap left atoms out + warnings.append(report.cap_warning(extraction.as_dict())) if not atom_list: warnings.append("no atoms with span provenance were extracted; nothing to measure") @@ -361,7 +362,7 @@ def analyze( atoms=atom_list, sampling=sampling, measure_result=measure_result, - span_summary=spans.span_summary(span_list, atom_list, atom_cap=config.max_atoms), + span_summary=spans.span_summary(span_list, atom_list, capped=capped), by_knowledge_kind=by_knowledge_kind, probe_fidelity=probe_set.as_dict(), items=item_stats, @@ -385,7 +386,7 @@ def analyze( warnings=warnings, ) payload["run_config"] = _run_config(config) - payload["run_config"]["tier"] = settings.tier_record(n_atoms=len(atom_list)) + payload["run_config"]["tier"] = settings.tier_record(sampled=capped) return AnalyzeResult( artifact=payload, attributions=attributions, diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index f4bf27991..8ceae31cc 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -35,6 +35,10 @@ "rare_critical", "procedural", "routing", "evidence", ) +#: Spans per extraction call when the caller does not say; ``tiers.estimate_calls`` counts +#: extraction calls with it, one per batch of this many spans. +BATCH_SPANS = 25 + _EXTRACT_SYSTEM = ( "You extract checkable knowledge from technical documents. You never invent facts " "that the text does not assert, and you never soften a claim the text makes." @@ -193,14 +197,13 @@ class ExtractionReport: cut-off ones are cured by ``--max-tokens``. Kept for the warning. calls: Extraction calls made, splits included, so the cost of re-asking is visible. batches: Calls the same run would make with no splitting: a batch counts once it - is asked, and its halves count at its level, so a run the cap stopped early - counts only the batches it reached. + is asked, and its halves count at its level. A capped run asks every batch. parse_error: The last parse failure on a span that stayed unread. call_error: The last failed call on a span that stayed unread. Errors that splitting recovered from are not recorded: they did not lose anything, and quoting one would blame the wrong cause. - reached: How far through the input a capped atom set reached (see - :func:`_cap_reach`); ``None`` when the cap did not bind. + sampled: What a binding cap kept (see :func:`sample_record`); ``None`` when every + extracted atom was kept. """ unread_span_ids: list[str] = field(default_factory=list) @@ -215,7 +218,7 @@ class ExtractionReport: parse_error: str = "" call_error: str = "" unusable_error: str = "" - reached: dict | None = None + sampled: dict | None = None @property def last_error(self) -> str: @@ -291,7 +294,9 @@ def as_dict(self) -> dict: "calls": self.calls, "batches": self.batches, "last_error": self.last_error, - "reached": self.reached, + # How far a document-order cap reached (#299); a spread sample has no reach. + "reached": None, + "sampled": self.sampled, } @@ -299,7 +304,7 @@ def extract_atoms( spans: list[Span], *, client: LLMClient, - batch_spans: int = 25, + batch_spans: int = BATCH_SPANS, cap: int | None = None, report: ExtractionReport | None = None, ) -> list[Atom]: @@ -307,12 +312,13 @@ def extract_atoms( A batch whose reply does not parse, or parses to rows none of which is a usable atom (see :func:`_atom_from_row`), is split in half and re-asked, down to one span - per call; the halves go back to the front of the queue, so atoms keep document order - (which is what ``cap`` cuts on). The common cause is an output ceiling: a provider - that defaults to 4096 output tokens truncates a 25-span reply mid-array, and a + per call; the halves go back to the front of the queue, so batches keep document order + (within a batch, atoms come in the reply's order). The common cause is an output + ceiling: a provider that defaults to 4096 output tokens truncates a 25-span reply + mid-array, and a truncated array does not parse. Splitting recovers without knowing the ceiling, at a cost of up to 2n-1 calls for an n-span batch that never parses; ``report.calls`` - counts them, against ``report.batches`` for the batches the run reached. A call that + counts them, against ``report.batches`` for the batches of the input. A call that raises (transport, auth) or returns nothing is not split, because re-asking smaller would only multiply it. @@ -321,17 +327,23 @@ def extract_atoms( client: Extractor client (must not be the judge — see :mod:`.measure`). batch_spans: Spans per extraction call. Batching is what keeps extraction affordable; too large and the model starts skipping spans. - cap: Stop after this many atoms. Recorded in the artifact as the sampling - rule, so a capped run is never quoted as a full sweep. + cap: Keep this many atoms, spread over the extracted atoms in document order + (:func:`spread_sample`); every span is extracted either way. At least 1: a + smaller cap raises ``ValueError`` before any call or change to ``spans``. Recorded in the artifact as + the sampling rule, so a capped run is never quoted as a full sweep. report: Filled in place with what could not be read and what re-asking cost, - and how far a binding cap reached (``reached``). Spans no call could read - are also marked ``unread`` in place. + and what a binding cap kept (``sampled``). Spans no call could read are also + marked ``unread`` in place. Returns: - Atoms in extraction order, de-duplicated by id. + Atoms de-duplicated by id. When the cap keeps them all (no cap, or ``M <= n``): + every atom, in extraction order (batches in document order, each in its reply's + order). When it binds: the cap's sample, in document order, by the earliest span + each atom cites (atoms citing the same earliest span keep extraction order). """ from dokimi.llm import extract_json_array + _check_cap(cap) # before any call, and before the spans' unread marks are reset report = report if report is not None else ExtractionReport() valid_ids = {span.id for span in spans} @@ -346,7 +358,6 @@ def parse(reply: str) -> list: for span in spans: span.unread = False seen: set[str] = set() - sent: set[str] = set() # span ids some call was shown, which bound the reach atoms: list[Atom] = [] lost: Exception | None = None # (spans, is a half): a half was counted in ``batches`` with the batch it came from. @@ -355,10 +366,7 @@ def parse(reply: str) -> list: for start in range(0, len(spans), batch_spans) ] while pending: - if cap is not None and len(atoms) >= cap: - break batch, is_half = pending.pop(0) - sent.update(span.id for span in batch) if not is_half: report.batches += 1 prompt = _EXTRACT_PROMPT.format( @@ -408,45 +416,51 @@ def parse(reply: str) -> list: continue seen.add(atom.id) atoms.append(atom) - if cap is not None and len(atoms) >= cap: - break if not atoms and lost is not None and all(span.unread for span in spans): # Only when nothing was read: spans that were read and asserted nothing are a # result, and a recovered parse error is not a failure. raise ExtractionError(f"no atoms: every extraction call failed ({lost})") from lost - report.reached = _cap_reach(spans, atoms, cap, sent) - return atoms - - -def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None, - sent: set[str]) -> dict | None: - """How far through the input (``spans``) a capped atom set reached; ``None`` under the cap. - - Extraction walks the spans in document order and stops at the cap, so a capped set - covers the input up to the span it stopped at, not a sample of it. The input is the text segmented, - not the artifact: a bundled file it left out counts in no span. The reach is the last span any - kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole, - but its later spans gave no atom. Only the ``sent`` spans count: an atom may cite a span - of the document its call was never shown, and that made-up id must not move the reach - (an atom set citing nothing sent reaches the last span sent). ``spans`` counts the spans - up to and including it, ``of`` all of them, and ``tokens_share`` is their share of the - span tokens. + report.sampled = sample_record(len(atoms), cap) + return spread_sample(atoms, cap, key=_earliest_span(spans)) + + +def _check_cap(cap: int | None) -> None: + """A cap probes at least one atom: ``0`` would record a sample of nothing (the CLI's + ``--max-atoms`` takes positive ints only; this guards the Python API).""" + if cap is not None and cap < 1: + raise ValueError(f"the atom cap must be at least 1, or None for no cap; got {cap}") + + +def _earliest_span(spans: list[Span]): + """Sort key: the input position of the earliest span an atom cites. Within a batch the + reply's order is the model's, which the prompt does not fix.""" + position = {span.id: i for i, span in enumerate(spans)} + return lambda atom: min(position[span_id] for span_id in atom.span_ids) + + +def spread_sample(atoms: list, cap: int | None, *, key=None) -> list: + """The ``cap`` atoms a capped run probes: every (M/n)-th of the M extracted. + + Index ``floor((i + 0.5) * M / n)`` for ``i`` in ``0..n-1`` of the atoms sorted by ``key`` + (stably, so atoms with equal keys keep their order; ``None``: as given), so the sample + starts at the middle of the first stride and estimates the per-atom rate an uncapped run + reports. :func:`extract_atoms` sorts by the earliest cited span, i.e. document order + (on a bundle run, the files in bundle order). A pure function of the list, with no seed: + the same extracted list gives the same sample. With no cap, or ``M <= n``, every atom, + in the order given and unsorted. https://github.com/intel/dokimi/issues/293 """ - if cap is None or not atoms or len(atoms) < cap: + if cap is None or len(atoms) <= cap: + return list(atoms) + ordered = sorted(atoms, key=key) if key is not None else list(atoms) + return [ordered[(2 * i + 1) * len(ordered) // (2 * cap)] for i in range(cap)] + + +def sample_record(n_extracted: int, cap: int | None) -> dict | None: + """``extraction.sampled``: the cap's ``atoms`` of the ``of`` extracted, and the rule; ``None`` + when the cap kept every atom.""" + if cap is None or n_extracted <= cap: return None - order = {span.id: index for index, span in enumerate(spans) if span.id in sent} - last = max((order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order), - default=max(order.values())) - reached = sum(span.tokens for span in spans[:last + 1]) - total = sum(span.tokens for span in spans) - # Floored in integers, as floored_percent prints it: a cap one span short of the end must not round to 1.0 - share = reached * 10000 // total - return { - "last_span": spans[last].id, - "spans": last + 1, - "of": len(spans), - "tokens_share": (max(share, 1) if reached else 0) / 10000, - } + return {"atoms": cap, "of": n_extracted, "rule": "cap-spread"} def _mark_unread(batch: list[Span], report: ExtractionReport) -> None: diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index b1c7d5a4b..35f753570 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -257,21 +257,50 @@ def reach_is_end(reached: dict) -> bool: return reached["spans"] == reached["of"] -def cap_warning(cap: int, reached: dict) -> str: - """The warning for a binding cap: where extraction stopped, and what that leaves out.""" - left_out = ("any atoms past the cap in its last reply were dropped" if reach_is_end(reached) - else "rates describe the opening of the input, not all of it") - return (f"atom extraction hit the cap of {cap} at {describe_reach(reached)}: {left_out}; " - "pass --all-atoms for the full sweep") +def describe_sample(sampled: dict, n_unread: int = 0) -> str: + """What a binding cap probed, as ``25 of the input's 61 atoms, spread over all of it (a + stride of 2.44 atoms in document order)``: ``extraction.sampled`` of a ``cap-spread`` run. + + With ``n_unread`` spans no extraction call read, the atoms are only those of the spans + that were read, so the sample is spread over those, not over the whole input. + """ + stride = f"a stride of {sampled['of'] / sampled['atoms']:.3g} atoms in document order" + if n_unread: + return (f"{sampled['atoms']} of the {sampled['of']} atoms extracted from the spans that " + f"were read, spread over those ({stride}; {n_unread} of the input's spans were " + "not read, see extraction.unread_span_ids)") + return (f"{sampled['atoms']} of the input's {sampled['of']} atoms, spread over all of it " + f"({stride})") + + +def _what_extraction_read(n_unread: int) -> str: + """What extraction read, after "extraction" in a cap sentence: every span, or not.""" + if n_unread: + return f"could not read {n_unread} of the input's spans" + return "read the whole input" + + +def cap_warning(extraction: dict) -> str: + """The warning for a binding cap, from the ``extraction`` record of a run whose ``sampled`` + is set: which atoms it probed, and how to probe them all.""" + sampled, n_unread = extraction["sampled"], len(extraction.get("unread_span_ids") or []) + return (f"atom extraction {_what_extraction_read(n_unread)}, and the atom cap probes " + f"{describe_sample(sampled, n_unread)}: rates are estimated from that sample; pass " + "--all-atoms to probe every atom") def coverage_reach(extraction: dict | None) -> tuple[str, str]: """The verb and tail of the headline's coverage line, from the ``extraction`` record. - The coverage share counts the words that went into the input. Under a binding cap the - atoms come from those words up to the span the cap stopped at, so the line must not say - they were measured, and it says how far the atoms reach. + The coverage share counts the words that went into the input. Under a binding cap only + the cap's atoms were probed, so the line must not say the words were measured: it says + which atoms were (``sampled``; over the read spans only, when some went unread), or for a + run before #293, how far the atoms reach. """ + sampled = (extraction or {}).get("sampled") + if sampled: + n_unread = len((extraction or {}).get("unread_span_ids") or []) + return "went into the input", f"; the cap probed {describe_sample(sampled, n_unread)}" reached = (extraction or {}).get("reached") if not reached: return "were measured", "" @@ -283,7 +312,13 @@ def coverage_reach(extraction: dict | None) -> tuple[str, str]: def _capped_at(artifact: dict) -> str: - """The report header's sentence on where a binding cap stopped extraction, or ``""``.""" + """The report header's sentence on what a binding cap probed (or, before #293, where it + stopped extraction), or ``""``.""" + sampled = (artifact.get("extraction") or {}).get("sampled") + if sampled: + n_unread = len(artifact["extraction"].get("unread_span_ids") or []) + return (f" Extraction {_what_extraction_read(n_unread)}; the atom cap probed " + f"{describe_sample(sampled, n_unread)}.") reached = (artifact.get("extraction") or {}).get("reached") if not reached: return "" diff --git a/dokimi/dokimi/novelty/spans.py b/dokimi/dokimi/novelty/spans.py index c091d1611..292d64279 100644 --- a/dokimi/dokimi/novelty/spans.py +++ b/dokimi/dokimi/novelty/spans.py @@ -254,7 +254,7 @@ def index_atoms(spans: list[Span], atoms: list) -> None: target.atom_ids.append(atom.id) -def span_summary(spans: list[Span], atoms: list | None = None, *, atom_cap: int | None = None) -> dict: +def span_summary(spans: list[Span], atoms: list | None = None, *, capped: bool = False) -> dict: """Aggregate span statistics for the ``novelty.v1`` ``spans{}`` block. ``tokens_carrying_nothing`` counts tokens in spans from which the extractor drew no @@ -281,8 +281,9 @@ def span_summary(spans: list[Span], atoms: list | None = None, *, atom_cap: int Args: spans: Segmented artifact spans. atoms: Atoms to index onto the spans. Pass the full extracted set. - atom_cap: The cap the extraction ran under, if any, so a capped set is flagged - rather than silently averaged. + capped: Whether the cap left atoms out (``ExtractionReport.sampled`` is set), so a + sampled set is flagged rather than silently averaged. An input with as many atoms + as the cap, or fewer, keeps every atom and is not capped. """ if atoms is not None: index_atoms(spans, atoms) @@ -305,7 +306,7 @@ def span_summary(spans: list[Span], atoms: list | None = None, *, atom_cap: int "n_unread": len(unknown), "tokens_unread": unknown_tokens, "n_atoms_indexed": len(atoms) if atoms is not None else None, - "capped_atom_set": bool(atom_cap and atoms is not None and len(atoms) >= atom_cap), + "capped_atom_set": capped, "sole_carrier": len(sole), "protected": sum(1 for s in spans if s.protected), "by_kind": { diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 052481887..52e223c18 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -21,16 +21,19 @@ from __future__ import annotations +import math from dataclasses import dataclass, field from typing import Any -#: What a run gives up, keyed by the setting that is off. "sampled" is a reached atom cap; the -#: name is old, the cap takes atoms in document order: https://github.com/intel/dokimi/pull/299 +from .atoms import BATCH_SPANS + +#: What a run gives up, keyed by the setting that is off. "sampled" is an atom cap that left atoms +#: out, which probes atoms spread over the input: https://github.com/intel/dokimi/issues/293 FORFEITS: dict[str, list[str]] = { "sampled": [ - "`added` is a rate over the atoms the cap kept, not the whole input: usable as a " - "sign, not as a figure, and not comparable with another artifact's; pass " - "--all-atoms for the full sweep", + "`added` is a rate over the sample of extracted atoms the cap kept, not over every " + "atom: usable as a sign, not as a figure, and not comparable with another " + "artifact's; pass --all-atoms for the full sweep", "`frac_carrying_nothing` is not interpretable on a capped atom set (0.064 read " "as 0.743 on the study's largest artifact)", ], @@ -57,20 +60,22 @@ } -def forfeits(settings: dict, *, n_atoms: int | None = None) -> list[str]: +def forfeits(settings: dict, *, sampled: bool | None = None) -> list[str]: """The claims a run with these settings may not make. Args: settings: Resolved ``AnalyzeConfig`` fields; a stage switch that is absent is off. - n_atoms: Atoms extracted, when known. Before extraction a cap is assumed reached. + sampled: Whether the cap left atoms out (``ExtractionReport.sampled`` is set), once + extraction has run: an input with as many atoms as the cap, or fewer, keeps every + atom. ``None`` before extraction, when a cap is assumed to bind. """ cap = settings.get("max_atoms") - capped = cap is not None and (n_atoms is None or n_atoms >= cap) + capped = cap is not None and sampled is not False out = list(FORFEITS["sampled"]) if capped else [] if capped and settings.get("shorten_artifact"): out.append( - "shortening candidates are scored only against the capped atoms, i.e. the " - "opening of the artifact: a cut of the uncovered tail looks free" + f"shortening candidates are scored against a spread sample of {cap} of the " + "extracted atoms: a cut of a span with no sampled atom looks free" ) for setting, lines in FORFEITS.items(): if setting != "sampled" and not settings.get(setting): @@ -154,7 +159,8 @@ class RunSettings: ``values`` holds every :data:`SETTING_NAMES` entry, resolved; ``tier`` and ``overrides`` say how they were reached, because a ``--tier triage --max-atoms 90`` run is not a stock triage run and must not be read as one. The forfeits and warnings - follow ``values`` (and, once known, the atom count) -- never the tier's name. + follow ``values`` (and, once known, whether the cap left atoms out) -- never the tier's + name. """ values: dict[str, Any] @@ -172,18 +178,18 @@ def config_fields(self) -> dict[str, Any]: """The ``AnalyzeConfig`` keyword arguments that carry these settings.""" return {**self.values, "tier": self.tier, "tier_overrides": dict(self.overrides)} - def forfeits(self, n_atoms: int | None = None) -> list[str]: + def forfeits(self, sampled: bool | None = None) -> list[str]: """The claims a run at these settings may not make (see :func:`forfeits`).""" - return forfeits(self.values, n_atoms=n_atoms) + return forfeits(self.values, sampled=sampled) - def tier_record(self, n_atoms: int | None = None) -> dict | None: + def tier_record(self, sampled: bool | None = None) -> dict | None: """``run_config.tier``: name, forfeits, overrides; ``None`` without a tier.""" if self.tier is None: return None - return {"name": self.tier, "forfeits": self.forfeits(n_atoms), + return {"name": self.tier, "forfeits": self.forfeits(sampled), "overrides": dict(self.overrides)} - def warnings(self, n_atoms: int | None = None) -> list[str]: + def warnings(self, sampled: bool | None = None) -> list[str]: """The warnings a result at these settings carries. With a tier: its forfeits, recomputed from what the run did, and the overrides. @@ -197,7 +203,7 @@ def warnings(self, n_atoms: int | None = None) -> list[str]: if not self.values["reversals"]: out.extend(FORFEITS["reversals"]) return out - out = [f"tier `{self.tier}` forfeits: {line}" for line in self.forfeits(n_atoms)] + out = [f"tier `{self.tier}` forfeits: {line}" for line in self.forfeits(sampled)] if self.overrides: out.append( f"tier `{self.tier}` was overridden by flags " @@ -212,10 +218,12 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, The atom count is the larger of one atom per two spans and one per :data:`TOKENS_PER_ATOM` tokens (spans alone undercount: short spans, such as table - rows, yield an atom each), capped at ``max_atoms``. Without ``n_tokens`` only the - span rule applies. The forfeits assume the cap is reached, since the atom count is - a guess until extraction runs: an artifact with fewer atoms than the cap drops the - cap's forfeits. + rows, yield an atom each), capped at ``max_atoms``. Extraction reads the whole input + whatever the cap, so its calls are the span batches it asks (see + :func:`estimate_calls`). Without ``n_tokens`` only the span rule applies. The + forfeits assume the cap is reached, since the atom count is a guess until extraction + runs: an artifact with no more atoms than the cap keeps them all and drops the cap's + forfeits. """ cap = self.values["max_atoms"] expected = max(1, n_spans // 2) @@ -229,7 +237,7 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, # The full resolved settings, so the projection is the model `estimate_calls` # implements rather than a subset of it. "estimate": estimate_calls(n_atoms, dict(self.values, budgets=tuple(budgets)), - n_arms=n_arms), + n_arms=n_arms, n_spans=n_spans), "tier": self.tier_record(), "caveat": ( "forfeits assume the atom cap is reached; the run recomputes them from the " @@ -372,7 +380,8 @@ def estimate_usd( return round(total, 6) -def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2) -> dict: +def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2, + n_spans: int | None = None) -> dict: """Project the model-call bill before spending it. From the instrumented runs: every probe, reversal included, costs an answer and a grade @@ -390,6 +399,11 @@ def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2) -> dict: n_atoms: Atoms expected after the cap. settings: Resolved tier settings (``paraphrases``, ``reversals``, stage switches). n_arms: Arms in the run, including the baseline. + n_spans: Spans of the input. Extraction asks every batch of + :data:`~dokimi.novelty.atoms.BATCH_SPANS` spans whatever the cap, so it makes + ``ceil(n_spans / BATCH_SPANS)`` calls (splits of an unparseable reply aside). + Without it, one call per 12 atoms of ``n_atoms``: the rule of the measured runs, + whose extraction stopped at the cap. Returns: ``{"n_atoms", "paired_probes", "reversal_probes", "probe_calls", "stage_calls", @@ -408,8 +422,10 @@ def estimate_calls(n_atoms: int, settings: dict, *, n_arms: int = 2) -> dict: # Reversals are graded in every arm too: the baseline arm's resistance to a planted # inversion is a reported control, not a by-product. probe_calls = (paired + reversals) * n_arms * 2 + batches = (max(1, round(n_atoms / 12)) if n_spans is None + else math.ceil(n_spans / BATCH_SPANS)) stage_calls = ( - max(1, round(n_atoms / 12)) # extraction batches + batches # extraction batches + (n_atoms if settings.get("verify_faithfulness") else 0) # one per atom + (1 if settings.get("paraphrases") else 0) + (1 if settings.get("reversals") else 0) diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index 9bd471137..8f2a62bad 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -71,7 +71,7 @@ }, "sampling": { "type": "string", - "description": "How the probed subset was chosen (e.g. cap-per-artifact:120). Present so a capped run is never read as a full sweep." + "description": "How the probed subset was chosen: full, every extracted atom (also a cap the atoms meet or fall under), or cap-spread: (e.g. cap-spread:120), only when the cap left atoms out: n atoms spread over every extracted atom in document order (extraction.sampled). Runs before cap-spread wrote cap-per-artifact:, the first n atoms in document order. Present so a capped run is never read as a full sweep." }, "faithfulness": { "type": ["object", "null"], @@ -270,7 +270,7 @@ }, "conflict_coverage": { "type": "object", - "description": "How much of the atom-pair space the conflict check compared. pairs_compared counts pairs that shared an answered call; pairs_total is every pair among the extracted atoms, which is the capped sample when spans.capped_atom_set is true. pairs_compared <= pairs_total is not expressible here and is enforced by the gate. A run that left pairs_compared below pairs_total, or capped the atom set, does not clear harmful_atoms at a count of 0.", + "description": "How much of the atom-pair space the conflict check compared. pairs_compared counts pairs that shared an answered call; pairs_total is every pair among the probed atoms, which is the capped sample when spans.capped_atom_set is true. pairs_compared <= pairs_total is not expressible here and is enforced by the gate. A run that left pairs_compared below pairs_total, or capped the atom set, does not clear harmful_atoms at a count of 0.", "additionalProperties": false, "required": ["pairs_compared", "pairs_total"], "properties": { @@ -302,7 +302,7 @@ }, "capped_atom_set": { "type": "boolean", - "description": "True when the indexed atom set hit the extraction cap. The dead-token share is then a property of the sample, not of the artifact, and must not be quoted as the latter." + "description": "True when the atom cap left extracted atoms out, so the indexed atoms are a sample (extraction.sampled is set); an input with as many atoms as the cap, or fewer, keeps every atom and is false. Runs written before the spread rule (#293) set it when the indexed atoms reached the cap. The dead-token share is then a property of the sample, not of the artifact, and must not be quoted as the latter." }, "sole_carrier": {"type": "integer", "minimum": 0}, "protected": {"type": "integer", "minimum": 0}, @@ -334,11 +334,11 @@ }, "retried_batches": {"type": "integer", "minimum": 0, "description": "Batches whose reply did not parse, or had no usable atom, and were split and re-asked."}, "calls": {"type": "integer", "minimum": 0, "description": "Extraction calls made, splits included. An n-span batch that never parses costs up to 2n-1 calls."}, - "batches": {"type": "integer", "minimum": 0, "description": "Calls the same extraction would make with no splitting: batches the run reached (a capped run stops early), each counted once whatever its halves cost."}, + "batches": {"type": "integer", "minimum": 0, "description": "Calls the same extraction would make with no splitting: every batch of the input (a capped run before cap-spread stopped early and counted only the batches it reached), each counted once whatever its halves cost."}, "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."}, "reached": { "type": ["object", "null"], - "description": "How far through the input a capped atom set reached; null when the cap did not bind. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). Extraction walks the spans in document order and stops at the cap, so a capped set covers the input up to the span it stopped at, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", + "description": "Written by runs before cap-spread (atoms.sampling cap-per-artifact:): how far through the input a capped atom set reached; null when the cap did not bind. A cap-spread run extracts every span and writes null here, and says what it probed in sampled. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). On those older runs extraction walked the spans in document order and stopped at the cap, so a capped set covers the input up to the span it stopped at, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", "properties": { "last_span": {"type": "string"}, "spans": {"type": "integer", "minimum": 1}, @@ -346,6 +346,16 @@ "tokens_share": {"type": "number", "minimum": 0, "maximum": 1} }, "required": ["last_span", "spans", "of", "tokens_share"] + }, + "sampled": { + "type": ["object", "null"], + "description": "What a binding cap probed (atoms.sampling cap-spread:): atoms of the of atoms extracted from every span some call read (extraction.unread_span_ids lists the rest), at indices floor((i + 0.5) * of / atoms) for i in 0..atoms-1 of the extracted list sorted by the earliest span each atom cites (document order; on a bundle run, the files in bundle order). No seed: the same extracted list gives the same sample. null when the cap kept every atom. Absent from artifacts written before it existed.", + "properties": { + "atoms": {"type": "integer", "minimum": 1}, + "of": {"type": "integer", "minimum": 1}, + "rule": {"type": "string"} + }, + "required": ["atoms", "of", "rule"] } } }, diff --git a/dokimi/research/records/decision/D017.md b/dokimi/research/records/decision/D017.md new file mode 100644 index 000000000..1db6ab037 --- /dev/null +++ b/dokimi/research/records/decision/D017.md @@ -0,0 +1,24 @@ +--- +id: D017 +type: decision +date: '2026-11-01' +title: A capped novelty run probes atoms spread over the input +ideas: +- I005 +state: accepted +links: +- https://github.com/intel/dokimi/issues/293 +- https://github.com/intel/dokimi/issues/371 +--- + +## Rationale +A cap that takes atoms in document order measures the opening of the input, not the input. At `--tier triage` on the dpnp repository's agent guidance, the 25 capped atoms of the root `AGENTS.md` (61 spans) came from spans s0002 to s0012, about a quarter of its tokens, and those of `dpnp/tests/AGENTS.md` (32 spans) from s0003 to s0013, about a third. Two artifacts' capped rates then describe different shares of their text, and a file's rate moves when its sections are reordered. + +A capped run now extracts every span and, of the M atoms extracted, probes the n at indices `floor((i + 0.5) * M / n)` of the atoms in document order, by the earliest span each cites (on a bundle run, in file order). A per-atom stride estimates the same per-atom rate an uncapped run reports. Equal token strata would not: they would weight a sparse introduction like a dense rule list. Like any fixed stride, it can fall in step with a repeated structure: with M = 100 and n = 25 it always takes atoms 2, 6, ..., 98 (from 0), so an input of four-atom sections of the same shape, each with its new fact in the same place, can give a sampled rate far from the whole input's, and re-extracting the same atoms cannot show the gap. A run on an artifact of repeated, same-shaped sections, or whose rate decides something, passes `--all-atoms`. The stride needs no seed, so the same extracted atom list gives the same sample; a fresh extraction is new model calls and can give other atoms, so runs that must probe identical atoms reuse one `atoms.json` through `novelty probe --atoms`. Probing stays at the cap; extraction grows to every batch of the input. + +The tier profiles (the call and token figures `--estimate-only` is fitted and checked on, and the per-tier call counts in `docs/skill-novelty.md`) were measured with the document-order cap. Which atoms are probed changes only for artifacts with more atoms than a tier's cap (25 at triage, 60 at standard, 120 at study); at or below the cap every atom is probed, as before. Extraction cost changes for more runs: any run whose cap was reached before its last batch, an artifact whose atoms exactly meet the cap included, now asks the remaining batches. Re-measuring those runs, and refitting the extraction profile to a run that asks every batch, is follow-up work this decision does not do. + +A capped run still indexes spans, writes `atoms.json` and shortens from the sampled atoms only, so spans whose atoms fell outside the sample read as carrying nothing; indexing them from every extracted atom is https://github.com/intel/dokimi/issues/371. diff --git a/dokimi/scripts/growth_baseline.json b/dokimi/scripts/growth_baseline.json index b0b75b387..84deff3fd 100644 --- a/dokimi/scripts/growth_baseline.json +++ b/dokimi/scripts/growth_baseline.json @@ -49,7 +49,7 @@ "dokimi/harnesses/tasks.py::harbor_task_problem": {"complexity": 11}, "dokimi/mcp/context7.py::Context7Client.get_library_docs": {"complexity": 12}, "dokimi/novelty/analyze.py::analyze": {"complexity": 22}, - "dokimi/novelty/atoms.py::extract_atoms": {"complexity": 16}, + "dokimi/novelty/atoms.py::extract_atoms": {"complexity": 14}, "dokimi/novelty/bundle.py::load_bundle": {"complexity": 14}, "dokimi/novelty/items.py::discrimination": {"complexity": 11}, "dokimi/novelty/items.py::load_probe_cells": {"complexity": 11, "rationale": "pre-existing when the gate was introduced"}, diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index c814879bc..edacb94a3 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -610,52 +610,132 @@ def test_a_tier_writes_its_forfeits_into_the_run_it_produced(tmp_path): assert len(forfeits) == len(TIERS["triage"]["forfeits"]) - len(FORFEITS["sampled"]) assert sorted(artifact["run_config"]["tier"]["forfeits"]) == sorted( w.split("forfeits: ", 1)[1] for w in forfeits) - assert not any("rate over the atoms the cap kept" in w for w in forfeits) + assert not any("atoms the cap kept" in w for w in forfeits) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) -def test_a_capped_run_says_how_far_through_the_artifact_the_cap_reached(tmp_path): - # One extraction batch sends all five spans; the cap keeps the first atom, from s0002. - artifact = _analyze(tmp_path, max_atoms=1).artifact - # s0001 and s0002 are 15 of the body's 42 span tokens. - reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} - assert artifact["extraction"]["reached"] == reached - warning = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " - "of its tokens): rates describe the opening of the input, not all of it; pass " - "--all-atoms for the full sweep") +def test_a_capped_run_probes_a_spread_sample_and_says_so(tmp_path, monkeypatch): + # One batch, two atoms (s0002, s0003); a cap of 1 keeps the middle of its one stride, the + # second, where the document-order cap kept the first. + from dokimi.novelty import shorten + + scored: list = [] + build = shorten.build_candidates + + def recording(span_list, atom_list, **kwargs): + scored.append([atom.span_ids for atom in atom_list]) + return build(span_list, atom_list, **kwargs) + + monkeypatch.setattr(shorten, "build_candidates", recording) + result = _analyze(tmp_path, max_atoms=1) + artifact = result.artifact + assert [atom.span_ids for atom in result.atoms] == [("s0003",)] + # Shortening candidates are judged against the sampled atom only. + assert scored == [[("s0003",)]] + assert artifact["atoms"]["sampling"] == "cap-spread:1" + assert artifact["extraction"]["sampled"] == {"atoms": 1, "of": 2, "rule": "cap-spread"} + assert artifact["extraction"]["reached"] is None + warning = ("atom extraction read the whole input, and the atom cap probes 1 of the input's 2 " + "atoms, spread over all of it (a stride of 2 atoms in document order): rates are " + "estimated from that sample; pass --all-atoms to probe every atom") assert warning in artifact["warnings"] - assert not any("sampled atoms" in w for w in artifact["warnings"]) - assert ("1 atoms (cap-per-artifact:1), 1 probed. The atom cap stopped extraction at span " - "s0002 (2 of the input's 5 spans, 35% of its tokens): the atoms come from the " - "opening of the input, not all of it.") in render_novelty_markdown(artifact) + assert not any("opening" in w for w in artifact["warnings"]) + assert ("1 atoms (cap-spread:1), 1 probed. Extraction read the whole input; the atom cap " + "probed 1 of the input's 2 atoms, spread over all of it (a stride of 2 atoms in " + "document order).") in render_novelty_markdown(artifact) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + # The renderers read every key of the sample, so a partial one does not validate. + partial = dict(artifact["extraction"], sampled={"atoms": 1}) + with pytest.raises(ArtifactValidationError, match="extraction/sampled: 'of' is a required"): + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=partial)) + # A sample of no atoms has no stride to describe; `null` is the absence of a sample. + empty = dict(artifact["extraction"], sampled={"atoms": 0, "of": 2, "rule": "cap-spread"}) + with pytest.raises(ArtifactValidationError, match="extraction/sampled/atoms: 0 is less than"): + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=empty)) + + +def test_an_artifact_written_under_the_document_order_cap_still_validates(tmp_path): + artifact = _analyze(tmp_path, max_atoms=1).artifact + old = {key: value for key, value in artifact["extraction"].items() if key != "sampled"} + old["reached"] = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=old, + atoms=dict(artifact["atoms"], sampling="cap-per-artifact:1"))) # The renderers read every key of the reach, so a partial one does not validate. - partial = dict(artifact["extraction"], reached={"spans": 3}) + partial = dict(old, reached={"spans": 3}) with pytest.raises(ArtifactValidationError, match="extraction/reached: 'last_span' is a required"): validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=partial)) -def test_the_reach_is_null_only_when_the_cap_did_not_bind(tmp_path): - # A cap the last atom meets binds, and reaches that atom's span; one above it does not. +def test_a_cap_the_atoms_meet_samples_nothing(tmp_path): + # Two atoms under a cap of 2: every atom is probed, exactly as with no cap, and the label + # says so: `cap-spread:` names a sample only when the cap left atoms out (#358). artifact = _analyze(tmp_path, max_atoms=2).artifact - assert artifact["atoms"]["n"] == 2 and artifact["spans"]["capped_atom_set"] is True - assert artifact["extraction"]["reached"]["last_span"] == "s0003" - uncapped = _analyze(tmp_path / "again", max_atoms=3).artifact - assert uncapped["extraction"]["reached"] is None - assert not any("hit the cap" in w for w in uncapped["warnings"]) - assert "atom cap stopped" not in render_novelty_markdown(uncapped) - validate_artifact("novelty", dict(uncapped, schema_version="novelty.v1")) - - -@pytest.mark.parametrize("cap, header, tail", [ - (2, "s0002 (2 of the input's 8 spans, 25% of its tokens): the atoms come from the opening " - "of the input, not all of it.", "come from the first 25% of the input's tokens"), - (8, "s0008 (8 of the input's 8 spans, 100% of its tokens): the atoms reach the input's last " - "span, and the cap dropped any atoms past it in the last reply.", - "reach the input's last span, and the cap dropped any atoms past it"), -], ids=["mid-input", "last-span"]) -def test_a_capped_bundle_run_reaches_through_its_input_not_the_artifact(tmp_path, capsys, cap, - header, tail): + assert artifact["atoms"]["n"] == 2 and artifact["atoms"]["sampling"] == "full" + assert artifact["extraction"]["sampled"] is None + assert not any("atom cap probes" in w for w in artifact["warnings"]) + assert "atom cap" not in render_novelty_markdown(artifact) + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + + +def test_a_cap_above_the_atom_count_labels_the_run_full(tmp_path): + # Ten spans, one atom each, under the triage cap of 25: nothing is left out, so the run is + # a full one in its label as in `extraction.sampled` and the report header. + path = tmp_path / "ten" / "SKILL.md" + path.parent.mkdir(parents=True) + rules = "\n\n".join(f"Rule {i}: set `--opt{i}` to {i}." for i in range(10)) + path.write_text(f"---\nname: ten\ndescription: Ten rules\n---\n\n{rules}\n", encoding="utf-8") + kind, name, description, frontmatter, body = load_artifact_text(f"skill:{path}") + scripted = _scripted("informed") + + def per_span(prompt, system): + if "Extract the checkable knowledge" not in prompt: + return scripted(prompt, system) + return json.dumps([dict(_ATOMS[0], statement=f"{sid} holds.", question=f"What of {sid}?", + span_ids=[sid]) + for sid in re.findall(r"^\[(s\d+)\]", prompt, flags=re.M)]), None + + config = AnalyzeConfig(artifact_ref=f"skill:{path}", artifact_kind=kind, name=name, + description=description, frontmatter=frontmatter, concurrency=1, + max_atoms=25) + _, answerer, judge = _clients() + extractor = LLMClient(model="extractor-m", role="extractor", transport=per_span) + artifact = analyze(body, config, extractor=extractor, answerer=answerer, + judge=judge).artifact + assert artifact["atoms"]["n"] == 10 + assert artifact["atoms"]["sampling"] == "full" + assert artifact["extraction"]["sampled"] is None + assert "10 atoms (full), 10 probed." in render_novelty_markdown(artifact) + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + + +@pytest.mark.parametrize("cap, capped", [(2, False), (1, True)], ids=["cap-met", "cap-binds"]) +def test_only_a_cap_that_leaves_atoms_out_counts_as_capped(tmp_path, cap, capped): + # Two atoms at `study`: a cap they meet keeps them all, so no cap forfeit and no flag; + # one atom more than the cap is a sample, and says so (#358). + from dokimi.novelty.tiers import FORFEITS, resolve_settings + + settings = resolve_settings("study", {"max_atoms": cap}) + artifact = _analyze(tmp_path, **settings.config_fields()).artifact + assert (artifact["extraction"]["sampled"] is not None) is capped + assert artifact["spans"]["capped_atom_set"] is capped + assert artifact["atoms"]["sampling"] == (f"cap-spread:{cap}" if capped else "full") + shortening = (f"shortening candidates are scored against a spread sample of {cap} of the " + "extracted atoms: a cut of a span with no sampled atom looks free") + warned = "\n".join(artifact["warnings"]) + recorded = artifact["run_config"]["tier"]["forfeits"] + for line in [*FORFEITS["sampled"], shortening]: + assert (line in warned) is capped, line + assert (line in recorded) is capped, line + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + + +@pytest.mark.parametrize("cap, probed, by_file", [ + # 8 atoms, one per span; SKILL.md holds s0001-s0005, references/tuning.md s0006-s0008. + (2, ["s0003", "s0007"], {"SKILL.md": 1, "references/tuning.md": 1}), + (3, ["s0002", "s0005", "s0007"], {"SKILL.md": 2, "references/tuning.md": 1}), +], ids=["cap-2", "cap-3"]) +def test_a_capped_bundle_run_strides_across_its_files_in_file_order(tmp_path, capsys, cap, probed, + by_file): from dokimi.commands import novelty as novelty_cmd from dokimi.novelty.bundle import load_bundle @@ -664,7 +744,7 @@ def test_a_capped_bundle_run_reaches_through_its_input_not_the_artifact(tmp_path (root / "SKILL.md").write_text(SKILL, encoding="utf-8") (root / "references" / "tuning.md").write_text( "# Tuning\n\nSet `--threads` to the core count.\n", encoding="utf-8") - # 500 words over the 100-word file budget: on disk, in no span the reach counts. + # 500 words over the 100-word file budget: on disk, in no span. (root / "references" / "zlarge.md").write_text( " ".join(f"word{i}" for i in range(500)), encoding="utf-8") bundle = load_bundle(f"skill:{root}", include_resources=True, max_file_words=100) @@ -685,15 +765,17 @@ def per_span(prompt, system): ) _, answerer, judge = _clients() extractor = LLMClient(model="extractor-m", role="extractor", transport=per_span) - artifact = analyze(bundle.text, config, extractor=extractor, answerer=answerer, - judge=judge, artifact_bundle=bundle).artifact - assert artifact["extraction"]["reached"]["of"] == 8 - assert f"The atom cap stopped extraction at span {header}" in render_novelty_markdown(artifact) + result = analyze(bundle.text, config, extractor=extractor, answerer=answerer, + judge=judge, artifact_bundle=bundle) + artifact = result.artifact + assert [atom.span_ids[0] for atom in result.atoms] == probed + assert {row["source"]: row["n_atoms"] for row in artifact["bundle"]["by_source"]} == by_file + assert artifact["extraction"]["sampled"] == {"atoms": cap, "of": 8, "rule": "cap-spread"} novelty_cmd._print_coverage(artifact["bundle"], artifact["extraction"]) - # 6% of the artifact went into the input; the reach is a share of the input alone. + # 6% of the artifact went into the input; the sample is of the input's atoms. assert capsys.readouterr().out.startswith( - "coverage: 6% of the artifact's readable words went into the input; the capped atoms " - f"{tail}\n") + "coverage: 6% of the artifact's readable words went into the input; the cap probed " + f"{cap} of the input's 8 atoms, spread over all of it") def test_skipping_the_faithfulness_check_is_recorded_as_a_warning(tmp_path): diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 8851908e5..aed6790f4 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -7,13 +7,11 @@ import json import re -from types import SimpleNamespace import pytest -from dokimi.novelty.atoms import ExtractionError, ExtractionReport, _cap_reach, extract_atoms +from dokimi.novelty.atoms import ExtractionError, ExtractionReport, extract_atoms, spread_sample from dokimi.novelty.client import LLMClient -from dokimi.novelty.report import floored_percent from dokimi.novelty.spans import segment DOC = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 11)) @@ -198,7 +196,7 @@ def starved_first(prompt, system): assert report.as_dict()["last_error"] == report.last_error -def test_a_capped_extraction_counts_only_the_batches_it_reached(): +def test_a_capped_extraction_sends_every_batch(): """200 spans, 3 atoms each; every 25-span reply is cut and both halves parse.""" doc = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 201)) spans = segment(doc) @@ -215,12 +213,104 @@ def three_per_span_cut_above_13(prompt, system): client = LLMClient(model="x", transport=three_per_span_cut_above_13) atoms = extract_atoms(spans, client=client, cap=120, report=report) assert len(atoms) == 120 - # Two 25-span batches reached the cap: each cost one cut call and two halves. - assert (report.calls, report.batches, report.retried_batches) == (6, 2, 2) + # All eight 25-span batches were asked, though the first two held 150 atoms: each cost + # one cut call and two halves. + assert (report.calls, report.batches, report.retried_batches) == (24, 8, 8) assert ( - "atom extraction re-asked 2 unparseable batches as halves: 6 extraction calls where " - "2 would do without splitting; a higher --max-tokens avoids the extra calls" + "atom extraction re-asked 8 unparseable batches as halves: 24 extraction calls where " + "8 would do without splitting; a higher --max-tokens avoids the extra calls" ) in report.warnings(len(spans)) + # The 120 kept are spread over all 600, one per 5, from the middle of the first stride. + assert [a.span_ids[0] for a in atoms[:3]] == ["s0001", "s0003", "s0005"] + assert atoms[-1].span_ids[0] == "s0200" + + +# The issue's root AGENTS.md: 61 atoms under the triage cap of 25. +SPREAD_61_25 = [1, 3, 6, 8, 10, 13, 15, 18, 20, 23, 25, 28, 30, 32, 35, 37, 40, 42, 45, 47, + 50, 52, 54, 57, 59] + + +@pytest.mark.parametrize("n_atoms, cap, kept", [ + (61, 25, SPREAD_61_25), + (61, 1, [30]), # one atom: the middle one, not the first + (25, 25, list(range(25))), # a cap the atoms meet exactly keeps them all + (10, 25, list(range(10))), + (10, None, list(range(10))), +], ids=["61-of-25", "cap-1", "at-cap", "under-cap", "no-cap"]) +def test_a_cap_keeps_every_kth_atom_from_the_middle_of_the_first_stride(n_atoms, cap, kept): + """``floor((i + 0.5) * M / n)``: the stride rule of https://github.com/intel/dokimi/issues/293.""" + assert spread_sample([f"atom{i}" for i in range(n_atoms)], cap) == [f"atom{i}" for i in kept] + + +def _cited(*span_ids: str) -> dict: + return {"statement": f"{'+'.join(span_ids)} holds a default.", "question": f"q {span_ids}", + "answer": "a value", "kind": "proposition", "tags": [], "span_ids": list(span_ids)} + + +@pytest.mark.parametrize("reply, cap, probed", [ + # Sorted s0001 s0002 s0003; the one stride's middle is s0002, where reply order gives s0003. + ([("s0001",), ("s0003",), ("s0002",)], 1, [("s0002",)]), + ([("s0001",), ("s0003",), ("s0002",), ("s0004",)], 2, [("s0002",), ("s0004",)]), + # By the earliest span an atom cites, not its first: (s0004, s0001) sorts first. + ([("s0003",), ("s0004", "s0001"), ("s0002",)], 1, [("s0002",)]), + # A cap that does not bind keeps the reply's order: atoms.json is as before #293. + ([("s0002",), ("s0001",)], 2, [("s0002",), ("s0001",)]), + ([("s0001",), ("s0003",), ("s0002",)], None, [("s0001",), ("s0003",), ("s0002",)]), +], ids=["cap-1", "cap-2-of-4", "earliest-cited", "at-cap", "no-cap"]) +def test_a_binding_cap_strides_in_document_order_whatever_the_reply_s_order(reply, cap, probed): + def out_of_order(prompt, system): + return json.dumps([_cited(*span_ids) for span_ids in reply]), None + + atoms = extract_atoms(segment(DOC), client=LLMClient(model="x", transport=out_of_order), + cap=cap) + assert [atom.span_ids for atom in atoms] == probed + + +def test_a_cap_below_one_is_refused_before_any_call_or_change_to_the_spans(): + calls: list[str] = [] + + def counted(prompt, system): + calls.append(prompt) + return _three_per_span(prompt, system) + + spans = segment(DOC) + spans[0].unread = True # the caller's marks, which a refused cap must leave as they were + with pytest.raises(ValueError, match="the atom cap must be at least 1"): + extract_atoms(spans, client=LLMClient(model="x", transport=counted), cap=0) + assert calls == [] + assert [span.unread for span in spans] == [True] + [False] * (len(spans) - 1) + + +def test_a_binding_cap_over_a_partly_read_input_makes_no_whole_input_claim(): + """50 spans, cap 2: the first batch gives 25 atoms and the second an empty reply.""" + from dokimi.novelty.report import _capped_at, cap_warning, coverage_reach + + spans = segment("\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 51))) + second_batch = {s.id for s in spans[25:]} + + def second_empty(prompt, system): + if second_batch & set(re.findall(r"^\[(s\d+)\]", prompt, flags=re.M)): + return "", None + return json.dumps(_rows(prompt)), None + + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=second_empty), cap=2, + report=report) + assert [a.span_ids[0] for a in atoms] == ["s0007", "s0019"] # both from the opening + assert report.sampled == {"atoms": 2, "of": 25, "rule": "cap-spread"} + assert len(report.unread_span_ids) == 25 + record = report.as_dict() + sample = ("2 of the 25 atoms extracted from the spans that were read, spread over those (a " + "stride of 12.5 atoms in document order; 25 of the input's spans were not read, see " + "extraction.unread_span_ids)") + texts = [cap_warning(record), _capped_at({"extraction": record}), coverage_reach(record)[1]] + assert texts == [ + f"atom extraction could not read 25 of the input's spans, and the atom cap probes {sample}: " + "rates are estimated from that sample; pass --all-atoms to probe every atom", + f" Extraction could not read 25 of the input's spans; the atom cap probed {sample}.", + f"; the cap probed {sample}", + ] + assert not any("whole input" in text or "all of it" in text for text in texts) def _three_per_span(prompt, system): @@ -230,60 +320,47 @@ def _three_per_span(prompt, system): return json.dumps(rows), None -def test_a_capped_extraction_records_the_last_span_its_atoms_reached(tmp_path): - """60 spans, 3 atoms each, 10 per batch: the 25th atom is span 9's, mid-batch.""" +def test_a_capped_extraction_reads_every_span_and_records_the_sample(tmp_path): + """60 spans, 3 atoms each, 10 per batch: a cap of 25 keeps 25 of 180, from every part.""" from dokimi.novelty.atoms import save_atoms doc = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61)) spans = segment(doc) + calls: list[str] = [] + + def counted(prompt, system): + calls.append(prompt) + return _three_per_span(prompt, system) + report = ExtractionReport() - atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=counted), batch_spans=10, cap=25, report=report) - assert len(atoms) == 25 and report.batches == 1 - # The batch sent s0001-s0010; the cap cut its reply inside s0009's atoms. - # Nine spans of 8 tokens, of 480 tokens in all: 72 / 480. - assert [s.tokens for s in spans[:9]] == [8] * 9 and sum(s.tokens for s in spans) == 480 - assert report.reached == {"last_span": "s0009", "spans": 9, "of": 60, "tokens_share": 0.15} + assert (len(calls), report.batches, report.calls) == (6, 6, 6) + assert len(atoms) == 25 + # Every batch gives sampled atoms, the first from s0002 (atom 3) and the last from s0059 (176). + assert {(int(a.span_ids[0][1:]) - 1) // 10 for a in atoms} == set(range(6)) + assert atoms[0].span_ids[0] == "s0002" and atoms[-1].span_ids[0] == "s0059" + assert report.sampled == {"atoms": 25, "of": 180, "rule": "cap-spread"} path = tmp_path / "atoms.json" save_atoms(atoms, spans, path, cap=25, extraction=report) - assert json.loads(path.read_text())["extraction"]["reached"] == report.reached - + saved = json.loads(path.read_text())["extraction"] + assert (saved["sampled"], saved["reached"]) == (report.sampled, None) + # Selection is a pure function of the extracted list: a second run picks the same atoms. + again = extract_atoms(segment(doc), client=LLMClient(model="x", transport=_three_per_span), + batch_spans=10, cap=25) + assert [a.id for a in again] == [a.id for a in atoms] -def test_a_span_id_no_call_was_shown_does_not_move_the_reach(): - """The cap binds on the first batch's last span (s0010); one atom also cites s0055.""" - def made_up(prompt, system): - # Two atoms for each of s0001-s0005 and three for s0006-s0010: 25 in all. - rows = [dict(row, statement=f"{row['statement']} ({k})", question=f"q{k} {row['span_ids'][0]}") - for row in _rows(prompt) for k in range(2 if row["span_ids"][0] <= "s0005" else 3)] - rows[4]["span_ids"] = ["s0003", "s0055"] - return json.dumps(rows), None - spans = segment("\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61))) - report = ExtractionReport() - atoms = extract_atoms(spans, client=LLMClient(model="x", transport=made_up), - batch_spans=10, cap=25, report=report) - assert "s0055" in atoms[4].span_ids and report.batches == 1 - assert report.reached == {"last_span": "s0010", "spans": 10, "of": 60, - "tokens_share": 0.1666} - # An atom set that cites nothing it was shown reaches the last span sent. - first_ten = {span.id for span in spans[:10]} - assert _cap_reach(spans, [SimpleNamespace(span_ids=["s0055"])], 1, first_ten)["spans"] == 10 - - -@pytest.mark.parametrize("cap, reached", [ - (None, None), - (31, None), - # Met on the last span: still a capped set, and its reach is the whole artifact. - (30, {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0}), -]) -def test_an_extraction_the_cap_did_not_stop_reaches_nothing_or_the_end(cap, reached): +@pytest.mark.parametrize("cap", [None, 31, 30], ids=["no-cap", "above", "at-cap"]) +def test_a_cap_the_atoms_do_not_exceed_keeps_them_all_and_records_no_sample(cap): """No cap, a cap above the 30 atoms, and a cap the last atom meets exactly.""" spans = segment(DOC) report = ExtractionReport() atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), cap=cap, report=report) assert len(atoms) == 30 and len(spans) == 10 - assert report.reached == report.as_dict()["reached"] == reached + assert report.sampled is None + assert (report.as_dict()["sampled"], report.as_dict()["reached"]) == (None, None) def _half_lost(n: int = 50, silent: frozenset[int] = frozenset()): @@ -567,12 +644,62 @@ def transport(prompt, system): ) -@pytest.mark.parametrize("first, share, shown", [ - (29_999, 0.9999, "99%"), # one span short of the end: floored, never rounded up to 100% - (1, 0.0001, "<1%"), # a reach above zero is never 0% -]) -def test_a_cap_reach_share_is_floored_not_rounded(first, share, shown): - spans = [SimpleNamespace(id="s0001", tokens=first), SimpleNamespace(id="s0002", tokens=30_000 - first)] - reach = _cap_reach(spans, [SimpleNamespace(span_ids=["s0001"])], cap=1, sent={"s0001", "s0002"}) - assert reach == {"last_span": "s0001", "spans": 1, "of": 2, "tokens_share": share} - assert floored_percent(reach["tokens_share"]) == shown +@pytest.mark.parametrize("cap, capped", [(3, False), (2, True)], ids=["cap-met", "cap-binds"]) +def test_novelty_atoms_flags_the_atom_set_only_when_the_cap_left_atoms_out( + tmp_path, monkeypatch, capsys, cap, capped): + """Three spans, one atom each: a cap of 3 keeps every atom, a cap of 2 samples them.""" + from tests import test_novelty_cli as novelty_cli + + skill = tmp_path / "s" / "SKILL.md" + skill.parent.mkdir(parents=True) + skill.write_text("---\nname: s\ndescription: d\n---\n\n" + "\n\n".join( + f"Setting {i} defaults to {i}." for i in range(1, 4)) + "\n", encoding="utf-8") + + def transport(prompt, system=None): + ids = re.findall(r"^\[(s\d+)\]", prompt, flags=re.M) + return json.dumps([{"statement": f"{sid} holds.", "question": f"What of {sid}?", + "answer": "a value", "span_ids": [sid]} for sid in ids]), None + + real_client = novelty_cli.novelty_cmd._client + monkeypatch.setattr(novelty_cli.novelty_cmd, "_client", + lambda model, provider, role, args: novelty_cli._with_transport( + real_client(model, provider, role, args), transport)) + args = novelty_cli._parse("novelty", "atoms", "--artifact", f"skill:{skill}", "--out", + str(tmp_path / "atoms.json"), "--max-atoms", str(cap), "--no-verify") + args.func(args) + out = capsys.readouterr().out + summary = json.loads(out[out.index("{"):]) + assert summary["n_atoms_indexed"] == cap + assert summary["capped_atom_set"] is capped + + +def test_novelty_atoms_saves_the_same_spread_sample_analyze_probes(tmp_path, monkeypatch, capsys): + """Two batches of 25 spans, one atom each: a cap of 2 keeps s0013 and s0038 for probe and shorten.""" + from tests import test_novelty_cli as novelty_cli + + skill = tmp_path / "s" / "SKILL.md" + skill.parent.mkdir(parents=True) + skill.write_text("---\nname: s\ndescription: d\n---\n\n" + "\n\n".join( + f"Setting {i} defaults to {i}." for i in range(1, 51)) + "\n", encoding="utf-8") + batches: list = [] + + def transport(prompt, system=None): + ids = re.findall(r"^\[(s\d+)\]", prompt, flags=re.M) + batches.append(ids) + return json.dumps([{"statement": f"{sid} holds.", "question": f"What of {sid}?", + "answer": "a value", "span_ids": [sid]} for sid in ids]), None + + real_client = novelty_cli.novelty_cmd._client + monkeypatch.setattr(novelty_cli.novelty_cmd, "_client", + lambda model, provider, role, args: novelty_cli._with_transport( + real_client(model, provider, role, args), transport)) + out = tmp_path / "atoms.json" + args = novelty_cli._parse("novelty", "atoms", "--artifact", f"skill:{skill}", "--out", + str(out), "--max-atoms", "2", "--no-verify") + args.func(args) + payload = json.loads(out.read_text()) + assert len(batches) == 2 # the cap no longer stops extraction after the first batch + assert [atom["span_ids"] for atom in payload["atoms"]] == [["s0013"], ["s0038"]] + assert payload["extraction"]["sampled"] == {"atoms": 2, "of": 50, "rule": "cap-spread"} + assert ("WARNING atom extraction read the whole input, and the atom cap probes 2 of the " + "input's 50 atoms") in capsys.readouterr().err diff --git a/dokimi/tests/test_novelty_cli.py b/dokimi/tests/test_novelty_cli.py index f78f6351e..0df58936b 100644 --- a/dokimi/tests/test_novelty_cli.py +++ b/dokimi/tests/test_novelty_cli.py @@ -875,6 +875,17 @@ def test_a_capped_run_s_headline_does_not_say_the_whole_artifact_was_measured(ca in capsys.readouterr().out +def test_a_spread_run_s_headline_says_how_many_atoms_the_cap_probed(capsys): + sampled = {"atoms": 25, "of": 61, "rule": "cap-spread"} + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=_readable_record(resource_words_on_disk=0), + extraction={"reached": None, "sampled": sampled})) + out = capsys.readouterr().out + assert ("coverage: 100% of the artifact's readable words went into the input (entry point " + "only); the cap probed 25 of the input's 61 atoms, spread over all of it (a stride of " + "2.44 atoms in document order)\n") in out + assert "measured" not in out and "opening" not in out + + def test_a_capped_run_under_the_floor_warns_in_the_coverage_line_s_words(capsys): # 10 of the 40 readable words went into the input, and the atoms stopped short of them. reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index 99881e1e2..35e939184 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -6,7 +6,13 @@ import pytest from dokimi.gate.novelty_gate import uncleared_conflicts -from dokimi.novelty.report import _capped_at, cap_warning, coverage_reach, render_novelty_markdown +from dokimi.novelty.report import ( + _capped_at, + cap_warning, + coverage_reach, + floored_percent, + render_novelty_markdown, +) def _artifact(**overrides): @@ -72,27 +78,61 @@ def test_the_cost_line_names_replayed_calls_only_when_there_are_some(cost, line) assert line in render_novelty_markdown(_artifact(cost=cost)).splitlines() -def test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): +SAMPLED = {"atoms": 25, "of": 61, "rule": "cap-spread"} + + +def test_a_spread_run_says_how_many_atoms_it_probed_spread_over_the_whole_input(): + assert cap_warning({"sampled": SAMPLED, "unread_span_ids": []}) == ( + "atom extraction read the whole input, and the atom cap probes 25 of the input's 61 atoms, " + "spread over all of it (a stride of 2.44 atoms in document order): rates are estimated from " + "that sample; pass --all-atoms to probe every atom") + assert coverage_reach({"reached": None, "sampled": SAMPLED}) == ( + "went into the input", "; the cap probed 25 of the input's 61 atoms, spread over all of it " + "(a stride of 2.44 atoms in document order)") + header = _capped_at({"extraction": {"reached": None, "sampled": SAMPLED}}) + assert header == (" Extraction read the whole input; the atom cap probed 25 of the input's 61 " + "atoms, spread over all of it (a stride of 2.44 atoms in document order).") + assert "opening" not in header + + +def test_an_uncapped_record_says_the_words_were_measured_and_adds_no_header(): + assert coverage_reach({"reached": None, "sampled": None}) == ("were measured", "") + assert _capped_at({"extraction": {"reached": None, "sampled": None}}) == "" + + +def test_an_old_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): + # A `cap-per-artifact` artifact, written before the spread rule: `reached`, no `sampled`. end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} assert coverage_reach({"reached": end}) == ( "went into the input", "; the capped atoms reach the input's last span, and the cap dropped any atoms " "past it") assert "opening" not in _capped_at({"extraction": {"reached": end}}) - assert cap_warning(30, end) == ( - "atom extraction hit the cap of 30 at span s0010 (10 of the input's 10 spans, 100% of its tokens): " - "any atoms past the cap in its last reply were dropped; pass --all-atoms for the full sweep") -def test_a_cap_that_stopped_inside_the_input_still_says_the_rates_describe_the_opening(): +def test_an_old_cap_that_stopped_inside_the_input_still_says_the_rates_describe_the_opening(): inside = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} assert coverage_reach({"reached": inside}) == ( "went into the input", "; the capped atoms come from the first 35% of the input's tokens") assert _capped_at({"extraction": {"reached": inside}}) == ( " The atom cap stopped extraction at span s0002 (2 of the input's 5 spans, 35% of its tokens): the " "atoms come from the opening of the input, not all of it.") - assert cap_warning(1, inside) == ( - "atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% of its tokens): rates " - "describe the opening of the input, not all of it; pass --all-atoms for the full sweep") + + +def test_an_old_artifact_with_a_reach_renders_its_header_as_before(): + artifact = _artifact(atoms={"n": 1, "n_probed": 1, "sampling": "cap-per-artifact:1"}, + extraction={"reached": {"last_span": "s0002", "spans": 2, "of": 5, + "tokens_share": 0.3571}}) + assert ("1 atoms (cap-per-artifact:1), 1 probed. The atom cap stopped extraction at span " + "s0002 (2 of the input's 5 spans, 35% of its tokens): the atoms come from the " + "opening of the input, not all of it.") in render_novelty_markdown(artifact) + + +@pytest.mark.parametrize("share, shown", [ + (0.9999, "99%"), # one span short of the end: floored, never rounded up to 100% + (0.0001, "<1%"), # a reach above zero is never 0% +]) +def test_an_old_reach_share_is_floored_not_rounded(share, shown): + assert floored_percent(share) == shown def test_a_cap_met_on_the_last_span_leaves_the_zero_uncleared_without_naming_an_opening(): diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 713ab052b..ba575660a 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -27,6 +27,7 @@ import io import json import os +import re import sys from decimal import ROUND_FLOOR, Decimal from pathlib import Path @@ -136,8 +137,9 @@ def _headline_coverage(text: str, work: Path) -> dict[str, Any]: printed = [line for line in text.splitlines() if line.startswith(COVERAGE_LINES)] record = work / "out" / "novelty.json" saved = json.loads(record.read_text(encoding="utf-8")) if record.exists() else {} + extraction = saved.get("extraction") or {} return {"printed": printed, "measured_share": (saved.get("bundle") or {}).get("measured_share"), - "reached": (saved.get("extraction") or {}).get("reached")} + "capped": bool(extraction.get("reached") or extraction.get("sampled"))} def _check_headline(coverage: dict[str, Any]) -> None: @@ -151,7 +153,7 @@ def _check_headline(coverage: dict[str, Any]) -> None: pct = f"{int(Decimal(str(share)).scaleb(2).to_integral_value(ROUND_FLOOR))}%" assert printed, "a run with a record must print its coverage line" # A capped run's words went into the input; its atoms do not cover all of them. - went = "went into the input" if coverage["reached"] else "were measured" + went = "went into the input" if coverage["capped"] else "were measured" assert printed[0].startswith( f"coverage: {pct} of the artifact's readable words {went}" ), printed @@ -215,6 +217,8 @@ def scripted(model, provider, role, args): "n_atoms", "total_calls", "total_calls_if_every_pair_escalates", + # Extraction asks every batch under a cap too (#293); tests/test_novelty_tiers.py pins it. + "stage_calls", ) @@ -259,20 +263,46 @@ def _refusal_as_exit_2(step: dict[str, Any]) -> dict[str, Any]: # atoms a sample. In the warnings and on stderr alike. CAP_WARNING_ON_MAIN = ("atom extraction hit the cap of 1; rates describe the sampled atoms, " "not the whole artifact") -CAP_WARNING = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " - "of its tokens): rates describe the opening of the input, not all of it; pass " - "--all-atoms for the full sweep") +# Intended change (#293): the cap probes atoms spread over the input, and the warning says so. +CAP_WARNING = ("atom extraction read the whole input, and the atom cap probes 1 of the input's 2 " + "atoms, spread over all of it (a stride of 2 atoms in document order): rates are " + "estimated from that sample; pass --all-atoms to probe every atom") # Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier # record, the warnings and the estimate's stderr. CAP_FORFEIT_ON_MAIN = "`added` is a sampled rate over the atom cap: usable" -CAP_FORFEIT = "`added` is a rate over the atoms the cap kept, not the whole input: usable" +CAP_FORFEIT = ("`added` is a rate over the sample of extracted atoms the cap kept, not over every " + "atom: usable") +# Intended change (#293): shortening is scored against a spread sample, not the opening. The +# forfeit names the cap, which tests/test_novelty_tiers.py pins; here both sides read . +SHORTEN_FORFEIT_ON_MAIN = ("shortening candidates are scored only against the capped atoms, i.e. " + "the opening of the artifact: a cut of the uncovered tail looks free") +SHORTEN_FORFEIT = ("shortening candidates are scored against a spread sample of of the " + "extracted atoms: a cut of a span with no sampled atom looks free") + + +# Intended change (#293): a cap of 1 on two atoms probes the second (s0003, mid-stride), not the first, +# so what follows the atom (dead tokens, oracle calls, the warnings on it) is not main's. +SPREAD_RUNS = {"run/max-atoms-1", "run/triage-max-atoms-1"} + + +def _settings_only(step: dict) -> dict: + """What the flags decide in a run step: its status, run_config and tier and cap warnings.""" + return {"status": step["status"], "run_config": step.get("run_config"), + "warnings": [w for w in step.get("warnings", []) + if w.startswith("tier `") or w == CAP_WARNING]} + + +def _cap_named(steps: list) -> list: + return json.loads(re.sub(r"spread sample of \d+ of", "spread sample of of", + json.dumps(steps))) @pytest.mark.parametrize("name", list(CASES)) def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): golden = json.loads(json.dumps(json.loads(GOLDEN.read_text(encoding="utf-8"))) - .replace(CAP_FORFEIT_ON_MAIN, CAP_FORFEIT)) - steps = _capture(name, tmp_path) + .replace(CAP_FORFEIT_ON_MAIN, CAP_FORFEIT) + .replace(SHORTEN_FORFEIT_ON_MAIN, SHORTEN_FORFEIT)) + steps = _cap_named(_capture(name, tmp_path)) coverage = [step.pop("coverage") for step in steps] for step_coverage in coverage: if step_coverage is not None: @@ -292,6 +322,9 @@ def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): # Intended change: `novelty atoms` prints the cap warning `analyze` records. expected = [dict(step, stderr=f"{step['stderr']}WARNING {CAP_WARNING}\n") for step in expected] + if name in SPREAD_RUNS: + steps, expected = ([_settings_only(step) for step in side] for side in (steps, expected)) + assert any(CAP_WARNING in step["warnings"] for step in steps) assert steps == expected diff --git a/dokimi/tests/test_novelty_spans.py b/dokimi/tests/test_novelty_spans.py index ad51f4894..a4de2a034 100644 --- a/dokimi/tests/test_novelty_spans.py +++ b/dokimi/tests/test_novelty_spans.py @@ -90,6 +90,7 @@ def test_span_summary_flags_a_capped_atom_set_because_the_cap_inverts_the_metric Atom(statement=f"claim {i}", question="q", answer="a", span_ids=(spans[1].id,)) for i in range(3) ] - assert span_summary(spans, atoms, atom_cap=3)["capped_atom_set"] is True - assert span_summary(spans, atoms, atom_cap=4)["capped_atom_set"] is False + # The run says whether its cap left atoms out; three atoms alone do not say it. + assert span_summary(spans, atoms, capped=True)["capped_atom_set"] is True + assert span_summary(spans, atoms)["capped_atom_set"] is False assert span_summary(spans)["n_atoms_indexed"] is None diff --git a/dokimi/tests/test_novelty_tiers.py b/dokimi/tests/test_novelty_tiers.py index 8133eee3d..0eb51da3d 100644 --- a/dokimi/tests/test_novelty_tiers.py +++ b/dokimi/tests/test_novelty_tiers.py @@ -222,10 +222,10 @@ def test_forfeits_follow_the_resolved_settings(): sampled = FORFEITS["sampled"][0] standard = TIERS["standard"]["settings"] - # Before extraction the cap is assumed reached; an atom set under it is not sampled. + # Before extraction the cap is assumed to bind; a cap that kept every atom sampled none. assert sampled in forfeits(standard) - assert sampled not in forfeits(standard, n_atoms=15) - assert sampled in forfeits(standard, n_atoms=60) + assert sampled not in forfeits(standard, sampled=False) + assert sampled in forfeits(standard, sampled=True) # --all-atoms drops the cap's forfeit, triage included. args = _parse("novelty", "analyze", "--artifact", "skill:x", "--tier", "triage", "--all-atoms") @@ -271,16 +271,22 @@ def test_flags_that_drop_the_probe_kinds_warn_without_a_tier(tmp_path): assert not any("no paraphrase probe" in w or "no reversal probe" in w for w in full) -def test_shortening_under_a_reached_cap_says_it_saw_only_the_opening(): +def test_shortening_under_a_cap_that_left_atoms_out_says_it_saw_only_a_spread_sample(): from dokimi.novelty.tiers import forfeits on = {"max_atoms": 60, "shorten_artifact": True, "paraphrases": 1, "reversals": True, "verify_faithfulness": True, "classify_content": True, "confirm_carriers": True} - assert any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=60)) - assert not any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=10)) - assert not any("shortening candidates are scored only" in f for f in forfeits(dict(on, max_atoms=None))) - assert not any("shortening candidates are scored only" in f - for f in forfeits(dict(on, shorten_artifact=False), n_atoms=60)) + line = ("shortening candidates are scored against a spread sample of 60 of the extracted " + "atoms: a cut of a span with no sampled atom looks free") + assert line in forfeits(on, sampled=True) + # Before extraction (an estimate) the cap is assumed to leave atoms out. + assert line in forfeits(on) + assert not any("opening" in f or "tail" in f for f in forfeits(on, sampled=True)) + # A cap that kept every atom (M <= n, the atoms meeting it exactly included) sampled none. + assert not any("spread sample" in f for f in forfeits(on, sampled=False)) + assert not any("spread sample" in f for f in forfeits(dict(on, max_atoms=None))) + assert not any("spread sample" in f + for f in forfeits(dict(on, shorten_artifact=False), sampled=True)) def test_the_settings_travel_through_the_config_unchanged(): @@ -315,8 +321,10 @@ def test_the_estimate_caps_the_expected_atoms_and_records_the_tier(): study = resolve_settings("study", {}) bill = study.estimate(300, budgets=(0.5,), n_arms=3) assert (bill["spans"], bill["atom_cap"]) == (300, 120) + # Probing stops at the cap; extraction asks every batch of the 300 spans. assert bill["estimate"] == estimate_calls( - 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3) + 120, dict(TIERS["study"]["settings"], budgets=(0.5,)), n_arms=3, n_spans=300) + assert bill["estimate"]["n_atoms"] == 120 assert bill["tier"] == {"name": "study", "forfeits": TIERS["study"]["forfeits"], "overrides": {}} assert "cap is reached" in bill["caveat"] @@ -326,13 +334,23 @@ def test_the_estimate_caps_the_expected_atoms_and_records_the_tier(): assert (sweep["caveat"], sweep["tier"]) == (None, None) +@pytest.mark.parametrize(("spans", "tokens", "batches"), [(300, 19000, 12), (26, None, 2), + (25, None, 1)]) +def test_the_estimate_counts_one_extraction_call_per_span_batch(spans, tokens, batches): + """A capped triage run asks ceil(spans / 25) extraction calls, not one per 12 atoms + expected before the cap (826 at 19,000 tokens, which counted 69).""" + bill = resolve_settings("triage", {}).estimate(spans, n_tokens=tokens)["estimate"] + # Triage runs no stage but extraction and the need graph. + assert bill["stage_calls"] == bill["by_role"]["extractor"] == batches + 1 + + def test_the_warnings_follow_the_settings_and_the_atom_count(): standard = resolve_settings("standard", {"no_verify": True}) - warned = standard.warnings(n_atoms=60) + warned = standard.warnings(sampled=True) assert all(w.startswith("tier `standard`") for w in warned) assert any(FORFEITS["verify_faithfulness"][0] in w for w in warned) assert "overridden by flags (no_verify)" in warned[-1] - assert not any(FORFEITS["sampled"][0] in w for w in standard.warnings(n_atoms=10)) + assert not any(FORFEITS["sampled"][0] in w for w in standard.warnings(sampled=False)) # Without a tier only the dropped probe kinds warn; the stages warn in the analyzer. bare = resolve_settings(None, {"no_paraphrase": True, "no_classify": True}) assert bare.warnings() == FORFEITS["paraphrases"] @@ -354,7 +372,7 @@ def test_a_tier_on_the_config_and_the_older_tier_argument_record_the_same(tmp_pa judge=_clients()[2], tier={"name": "triage", "overrides": {"max_atoms": 1}}).artifact assert on_config["run_config"]["tier"] == as_argument["run_config"]["tier"] == ( - settings.tier_record(n_atoms=1)) + settings.tier_record(sampled=True)) assert on_config["warnings"] == as_argument["warnings"] assert any("tier `triage` was overridden by flags (max_atoms)" in w for w in on_config["warnings"]) @@ -412,10 +430,14 @@ def pinned_list_prices(monkeypatch): def _projection(tier, spans, tokens): - """The estimate a run at this tier on this artifact prints, and its dollar figure.""" - estimate = resolve_settings(tier, {}).estimate( - spans, n_tokens=tokens, budgets=_BUDGETS, n_arms=2 - )["estimate"] + """The estimate of this run as it was measured, and its dollar figure. + + The runs predate the spread cap (#293): extraction stopped at the cap, so its batches follow + the capped atoms here. The estimate a run prints now counts every batch (no measured run yet). + """ + settings = resolve_settings(tier, {}) + n_atoms = settings.estimate(spans, n_tokens=tokens, budgets=_BUDGETS)["estimate"]["n_atoms"] + estimate = estimate_calls(n_atoms, dict(settings.values, budgets=_BUDGETS), n_arms=2) return estimate, estimate_usd(estimate["by_role"], tokens, _MODELS, tier)