Repository navigation
novelty: spread the atom cap across the artifact instead of its opening #293
Description
Problem
novelty analyze caps the atoms it probes (25 at --tier triage, 60 at standard). Extraction takes them in document order and stops at the cap, so a capped run measures only the opening of the artifact. Triage on IntelPython/dpnp#3099's guidance files:
| file | spans | atoms from | share of tokens |
|---|---|---|---|
root AGENTS.md |
61 | s0002–s0012 | about a quarter |
dpnp/tests/AGENTS.md |
32 | s0003–s0013 | about a third |
The rules a reader most wanted judged (the leftover-reference rule, the changelog format, the CI table, ## Reviewing, the directory guides) were never measured.
A separate change makes the run say how far the cap reached. That doesn't solve the underlying problem: the capped rate describes whatever the file opens with. So two artifacts' rates mean different things when one opens with an introduction and the other with rules, and a file's rate moves when its sections are reordered.
Proposal
Extract every span, then probe max_atoms atoms spread over the file.
- Selection: stratified by span position (or by tokens), with a fixed seed.
samplingrecords the rule (cap-spread:<n>). - Cost: probing, the expensive part, stays at the cap. Extraction grows: 3 batches instead of 1 for the 61-span file. The whole triage run there was 103 calls and $0.78.
- Bundles: with
--include-resources, spread over the files as well, soreferences/is reached on a skill whoseSKILL.mdalone has 25 atoms (todaydocs/reviewing.mdsays the flag reachesreferences/only below the cap).
This changes what a tier measures, so it needs a research-register decision. The tier profiles in docs/skill-novelty-evidence.md were measured with document order.
Open questions
- Strata: by spans, by tokens, or by headings?
- Tier evidence: does it change the tier comparison in the evidence page enough to need re-measuring a subset?
- Shortening: the
shorten_artifactforfeit ("scored only against the capped atoms, i.e. the opening") changes. Do shortening candidates then cover the whole file? - Determinism and cache: the selection must be reproducible across runs and stable under the response cache.
Done when
A decision record exists, and triage (and the other capped tiers, if agreed) probe a spread sample. Tests pin the selection, the docs describe it, and the cap warning names the rule.
Activity
antonwolfy commentedon Nov 1, 2026
Proposed answers to the open questions, so a PR can start from them.
docs/skill-novelty-evidence.md).novelty/tiers.pysays candidates "are scored only against the capped atoms, i.e. the opening of the artifact: a cut of the uncovered tail looks free". With a spread sample it becomes: candidates are scored against a spread sample of n of M atoms, so cutting a span with no sampled atom looks free. Candidates then come from the whole file, but stay unjudged where no atom was sampled, so the forfeit stays, reworded. It applies at standard and study, whereshorten_artifactis on.samplingrecordscap-spread:<n>;novelty.v1takes any string there, so the schema does not change. The cap warning from fix(novelty): a capped run says how far the cap reached #299 names the rule.--include-resources. Apply the same stride over the bundle's atoms in file order, soreferences/gets its share by atom count. The sentence indocs/reviewing.mdthat the flag reachesreferences/only whenSKILL.mdhas fewer than 25 atoms changes with it.Order. This one is independent of #290–#292 and #294: it touches only novelty (#316 also edits
novelty/atoms.py). It can start now, with the decision record first, since it changes what a tier measures.fix(novelty): stride in document order; qualify partial-read samples
fix(novelty): count a run as capped only when the cap left atoms out