feat(routing): estimate-only, all-vs-all repeats and cost, description queries

dokimi/changelog.d/20261031_120000_napetrov_routing-cli-gaps.md

@@ -0,0 +1,19 @@
+<!--
+Copyright (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+### Added
+
+- **`novelty routing --estimate-only`** prints, as JSON, the calls, input and output tokens and the list-price `expected_usd` that `--model` would make, from the same selection prompt the run sends, and makes no model call. Without `--model` the bill is zero. `--all-vs-all` has no estimate yet; its stderr line still says how many calls the model arm makes.
+- **`novelty routing --all-vs-all --repeats N`** asks each query N times, as a single-artifact run does, and prints and writes the `noise` block (agreement per query, the rate of each repeat, `half_range`). The model arm's counts are then calls. `--repeats` still needs `--model`.
+- **`novelty routing --all-vs-all --model`** writes the same `cost` block as the single-artifact run, on stdout and in the result file.
+
+### Fixed
+
+- **A query that names a skill left out for `disable-model-invocation` says so.** `--all-vs-all` refused it as "not in the field" with no hint of why; the message now names each such skill and points to `--include-disabled`.
+- **`--queries-from-headings` no longer gives up on a skill with no usable heading.** Only `##` and `###` headings of two or more content words were offered, so a skill whose headings were one word each, or a `#` title alone, had no queries (24 of 144 real skills). Its description's sentences are now the fallback, still stamped as self-derived triage, and the result records `queries_source`. A skill with neither still exits non-zero with the reason.
+- **`--estimate-only` without `--model` reports no query sent** (`n_queries_sent` 0), as it makes no call.
+- **A failed model call's message says what it leaves out.** Failed calls are left out of the model arm's scoring; a query with a failed repeat is also left out of the repeat-noise rates. The partial result keeps the calls that succeeded.
+- **A description with a parenthesised abbreviation is one query.** `(e.g. …)` and `(i.e. …)` no longer split the description after the abbreviation's period.
+- **`--seed` help says what it fixes.** It seeds the card order only; it does not make the model's answers repeatable, which is what `--repeats` measures. The help no longer calls the run reproducible.

dokimi/docs/reference/cli-interface.json

@@ -8277,6 +8277,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreTrueAction",
+              "choices": null,
+              "const": true,
+              "default": false,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--estimate-only",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_HelpAction",
               "choices": null,

dokimi/docs/skill-novelty.md

@@ -618,6 +618,10 @@ A catalog that holds the same `SKILL.md` under two paths (a plugin copy of its `
 tree) counts it once: the copy nearest the catalog root stands for the card, and the
 others are named on stderr.
 
+`--estimate-only` prints, as JSON, the calls, input and output tokens and the list-price `expected_usd` the
+`--model` arm would make, and makes no model call. Without `--model` the run is lexical and the bill
+is zero. Input is the selection prompt sized at chars / 4, so it is a projection, not a count.
+
 Two numbers, reported side by side and never merged:
 
 - **Lexical** — BM25 over each candidate's `name + description`, which is exactly the text
@@ -626,7 +630,9 @@ Two numbers, reported side by side and never merged:
   a **seeded random permutation per query** (`--seed`), so a model that always picks the
   first option tends to `1/N` rather than to a figure set by how the query count lines up
   with the candidate count. It *tends* to it: at five queries the realised rate is a fixed
-  offset from `1/N`, so read small query sets as such. Each permutation and the position
+  offset from `1/N`, so read small query sets as such. `--seed` fixes that card order and
+  nothing else: the model's answers are still sampled and can differ between runs, so measure
+  that spread with `--repeats`. Each permutation and the position
   the gold card was shown at are recorded per query. **An abstention counts against
   `selection_rate`** — not picking anything is the routing failure — with
   `selection_rate_among_answered` beside it; a failed call is excluded from both and
@@ -642,7 +648,8 @@ Two numbers, reported side by side and never merged:
   fresh sample, so under `DOKIMI_RESPONSE_CACHE` a repeat is not a
   replay. Repeats need `--model`, and cost N times the calls. With repeats, `n_calls` is
   queries x repeats, `n_abstained` and `n_failed` count calls, and each `per_query` row
-  carries its `repeat` index.
+  carries its `repeat` index. In an `--all-vs-all` run each `noise` row also names its `gold`
+  skill, since two skills can list the same request.
 
 A single-artifact `--model` run also writes a `cost` block (calls, input and output tokens, cache read and
 write tokens, and `usd`, which is `null` when any call this run paid for had no known price) and prints the
@@ -651,7 +658,7 @@ the provider they may or may not be counted inside `input_tokens`. A run without
 no calls and has no `cost` block. `usd` is also `null` when a call returned no usage, and a
 call that raised is not counted. A call replayed from the response cache is counted in
 `replayed_calls` and left out of the token and `usd` totals, since this run did not pay for it;
-`llm_calls` still counts it. `--all-vs-all` writes no `cost` block.
+`llm_calls` still counts it. `--all-vs-all` writes the same `cost` block over its model arm.
 
 `recall_at_3` comes back `null` with three or fewer candidates, where every rank is `<= 3`
 and the figure would be a structural 1.0. The lexical arm also reports `n_queries_given`
@@ -674,10 +681,14 @@ is no attractor, and a tied skill's own rate is lower by the picks inside its gr
 model call is unscored, and a skill with none scored has an own rate of `null`, not 0.0. A run
 with any failed model call writes the result and exits non-zero. The saved file records the
 field rule under `field` (`include_disabled`, the `competitors` and what was `left_out`); a
-single-artifact run records it too. It takes no `--artifact`, `--queries`,
-`--queries-from-headings` or `--repeats` (one model call per query). The model arm also records the selector `model`, so two runs with
+single-artifact run records it too. It takes no `--artifact`, `--queries` or `--queries-from-headings`. `--repeats` needs `--model`
+and asks each query that many times, as a single-artifact run does: the model arm then carries a
+`noise` block (agreement per query, the rate of each repeat, `half_range`), and its counts are
+calls, one per query per repeat. The model arm also records the selector `model`, so two runs with
 different models can be told apart. It is refused with fewer than two skills, a name repeated
-in the field, a query file that names a skill outside the field, or no queries at all.
+in the field, or no queries at all. A query file that names a skill outside the field is refused;
+when that skill is one the field leaves out for `disable-model-invocation`, the message names it and
+points to `--include-disabled`.
 
 **Cost of `--model` in all-vs-all.** The model arm makes one selector call per query, across
 every skill in the field: the call count is the number of queries in `queries.json` that have a
@@ -698,7 +709,10 @@ Three properties worth knowing before quoting either:
   and its own relevance. `--queries-from-headings` exists for triage, warns, and stamps
   `queries_self_derived: true` into the result; corpus-wide those self-derived queries gave
   recall@1 0.225, and a skill's own bundled `evals/` questions gave 0.667. Headings inside
-  fenced code blocks are not offered as queries. A run with no usable query exits non-zero,
+  fenced code blocks are not offered as queries. Only `##` and `###` headings of two or more
+  content words count; a skill whose headings give none falls back to the sentences of its
+  description with two or more content words, and `queries_source` (`headings`, `description` or
+  `none`) says which was used. A run with no usable query exits non-zero,
   says why, and still writes the result with `"status": "no_queries"`. Every rate line
   prints its `n_queries` (and `triage: self-derived` when it is self-derived), and fewer
   than five scored queries draw a warning.