fix(answers,pipeline,judge,gate): Stack P fixes, step 2 of #228

dokimi/changelog.d/20261028_120000_napetrov_stack-p-step2.md

@@ -0,0 +1,61 @@
+### Added
+
+- **`benchmark run --judge-temperature`** — a new flag: it sets the judge's sampling
+  temperature and is recorded as `judge_temperature` in the eval output. Before this change the
+  flag did not exist, and `benchmark run` rejected it as an unknown argument.
+
+### Changed
+
+- **Stack P scores move (#228 step 2)** — the context arm now asks the doc source for the
+  `top_k` docs the command was given: `benchmark run` asks for 3 (`DEFAULT_RETRIEVAL_TOP_K`; it
+  has no `--top-k`), `dokimi evaluate` and `answers generate` for their `--top-k` (default 3).
+  Before, they left `max_results` at the client default of 5. The doc library id resolves
+  through the registry's `context7_id` before the client's own resolver, in the `--registry`
+  that was loaded. A question with no document (no hit, or the reranker raised) is no longer judged on
+  its `[FALLBACK: ...]` placeholder by default, so it drops out of the paired statistics; the
+  record and its `retrieval_hit` stay, its diagnosis stays `empty_retrieval`, and the run
+  summary says how many questions it left out. Pass `--judge-placeholder` to judge the
+  placeholder again (this restores only that part of the old method). The dashboard's arm
+  means and delta leave such a question out too, as `report eval` does. Its doc score counts a
+  retrieval miss as 0 only when the baseline has a score and no error touched the pair (neither
+  arm has a judge error); a miss with an error leaves the doc score alone. `report eval` and `report generate` show `—` for the arm means and delta when
+  no question is scored in both arms (the `report generate --format json` `stats` block carries
+  `null` for `avg`, `min` and `max`), and the run summary prints no means then. `benchmark run`,
+  `evaluate` and `eval score` take the same flag; `benchmark batch` has none and never judges
+  the placeholder. Scores from before this change are not comparable with new ones.
+- **Trust gate** — `retrieval_hit_rate` counts every evaluation's recorded retrieval outcome,
+  not only the ones scored in both arms, so unjudged misses still count against the rate, and a
+  judge-error row keeps the outcome of its answer. The rate is shown even when no question is
+  scored in both arms. A judge-error row of an older eval file, which has no `retrieval_hit`,
+  is left out of that count as before. The thresholds are unchanged.
+- **Zero-score check** — a judge-error row on a retrieval miss now counts as unusable in
+  `zero_score_fraction` (a judge error on a retrieval hit is still left out), so a run can fail
+  that fail-severity check where intel/main passed it.
+- **Stamped artifacts** — the answers and eval artifacts carry `method_version` (2). Artifacts
+  from before this change have no stamp and represent the legacy method (version 1); readers do
+  not yet normalize or enforce the version, so it does not yet stop a comparison of mixed methods.
+  `eval score` stamps its eval with the version of the answers file it judges, so legacy answers
+  re-judged today stay version 1. An `eval score` artifact also carries `judge_placeholder`, so two
+  files of one version that differ in it are told apart. `eval panel-score` (the JudgePanel path)
+  writes neither stamp yet, so its artifact reads as version 1 whatever answers it judged. An absent
+  or null stamp means version 1; any other value must be a positive integer, or `eval score` exits
+  2 before any judge call.
+
+### Fixed
+
+- **Stack P no longer ignores `top_k`** — `Answerer` passes `max_results=top_k` to the doc
+  source; the context arm's retrieval request is now the one the arms stack makes.
+  `answers generate --top-k` takes a positive integer, as `dokimi evaluate` does, and a config
+  `retrieval.top_k` that is not one exits 2.
+- **Library id from the registry** — `EvaluationPipeline` resolves the library id through
+  `resolve_library_id_for_product` (explicit id, then an `mcp:` source's `;id=`, then `context7_id`,
+  then the client), in the
+  registry that `benchmark run` and `benchmark batch` loaded from `--registry`. A registered
+  product without `context7_id` now fails for a client that needs a declared id.
+  The question step's topics cache (`.cache/topics`) is keyed on the doc source as well, so
+  two doc sets of one product (two `local:` dirs, say) no longer share their topics.
+
+### Removed
+
+- Dead code in the answering path: the unused `Answerer.llm` client and the unreachable
+  "no documents" branch of `_generate_with_docs`.

dokimi/docs/methodology.md

@@ -199,11 +199,11 @@ RAGAS meta-evaluation can be run after answer generation to add faithfulness and
 
 Every serious run should answer "Can we trust this run?" before interpreting deltas. The trust gate checks:
 
-- minimum evaluated question count,
-- zero-score fraction,
-- minimum `with_docs` average,
-- average delta,
-- score coefficient of variation,
+- minimum evaluated question count (both arms scored; the single `Judge` leaves no-document fallback answers unscored by default, but includes them when `--judge-placeholder` is enabled and both arms receive scores; `JudgePanel` also judges nonempty fallback answers),
+- zero-score fraction (computed when at least one question is scored in both arms; over those paired questions plus retrieval misses with an unscored context arm and either a scored baseline or a judge error, which count as unusable),
+- minimum `with_docs` average (both-arm-scored questions only),
+- average delta (both-arm-scored questions only),
+- score coefficient of variation (both-arm-scored questions only),
 - inter-rater agreement when panel data exists,
 - retrieval hit rate — the share of questions where at least one document reached the `with_docs` prompt, from the per-evaluation `retrieval_hit` the judge and panel record; eval files judged before that field existed show it as not measured, and so do files where fewer than 80% of evaluations record it (`min_retrieval_coverage`), since the rate would then describe a subset,
 - multi-run variance when `--multi-run N` is used.

dokimi/docs/reference/cli-interface.json

@@ -344,7 +344,7 @@
               "nargs": null,
               "position": null,
               "required": false,
-              "type": "int"
+              "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"
             }
           ],
           "path": "dokimi answers generate",
@@ -2299,6 +2299,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreTrueAction",
+              "choices": null,
+              "const": true,
+              "default": false,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--judge-placeholder",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_StoreAction",
               "choices": [
@@ -2323,6 +2339,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--judge-temperature",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "float"
+            },
             {
               "action": "_StoreAction",
               "choices": null,
@@ -5805,6 +5837,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreTrueAction",
+              "choices": null,
+              "const": true,
+              "default": false,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--judge-placeholder",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_StoreAction",
               "choices": [
@@ -6015,6 +6063,22 @@
           "required": false,
           "type": null
         },
+        {
+          "action": "_StoreTrueAction",
+          "choices": null,
+          "const": true,
+          "default": false,
+          "group_key": null,
+          "group_required": null,
+          "hidden": false,
+          "is_flag": true,
+          "key": "--judge-placeholder",
+          "kind": "option",
+          "nargs": 0,
+          "position": null,
+          "required": false,
+          "type": null
+        },
         {
           "action": "_StoreAction",
           "choices": [

dokimi/dokimi/commands/answers.py

@@ -85,6 +85,10 @@ def cmd_answers_generate(args: argparse.Namespace) -> None:
     _retrieval_cfg = config.get("retrieval") or {}
     if args.top_k is None:
         args.top_k = _retrieval_cfg.get("top_k", DEFAULT_RETRIEVAL_TOP_K)
+        if isinstance(args.top_k, bool) or not isinstance(args.top_k, int) or args.top_k < 1:
+            print(f"Error: retrieval.top_k in the config must be a positive integer, "
+                  f"got {args.top_k!r}", file=sys.stderr)
+            sys.exit(EXIT_USAGE)
 
     questions = _load_questions(args.questions)
     print(f"Loaded {len(questions)} questions from {args.questions}")
@@ -162,7 +166,7 @@ def register(sub, positive_int) -> None:
     gen_a_p.add_argument("--model", default=DEFAULT_ANSWER_MODEL, help="LLM model for answering")
     gen_a_p.add_argument("--provider", default=DEFAULT_ANSWER_PROVIDER, choices=ANSWER_PROVIDERS)
     gen_a_p.add_argument("--max-tokens", type=int, default=DEFAULT_MAX_TOKENS_PER_QUESTION, help="Max tokens to retrieve per question")
-    gen_a_p.add_argument("--top-k", type=int, default=None, help=f"Number of docs to retrieve before reranking (default from config/products.yaml retrieval.top_k or {DEFAULT_RETRIEVAL_TOP_K})")
+    gen_a_p.add_argument("--top-k", type=positive_int, default=None, help=f"Number of docs to retrieve before reranking (default from config/products.yaml retrieval.top_k or {DEFAULT_RETRIEVAL_TOP_K})")
     gen_a_p.add_argument("--rerank-threshold", type=float, default=0.3, help="Min relevance score (0-1) to keep docs")
     gen_a_p.add_argument("--debug-retrieval", action="store_true", help="Include retrieval metadata in output")
     gen_a_p.add_argument("--concurrency", type=positive_int, default=DEFAULT_CONCURRENCY, help="Parallel API calls (default: 5)")

dokimi/dokimi/commands/benchmark.py

@@ -27,9 +27,20 @@
 )
 from dokimi.commands.library import _load_registry
 from dokimi.exit_codes import EXIT_USAGE
+from dokimi.llm_cli import SamplingError, check_sampling
 
 
-def _run_single_library(entry, output_dir: str, model: str, provider: str, judge_model: str, judge_provider: str = DEFAULT_JUDGE_PROVIDER, doc_source_override=None, max_tokens_per_question: int = DEFAULT_MAX_TOKENS_PER_QUESTION, force_regen: bool = False, concurrency: int = DEFAULT_CONCURRENCY, questions_from=None, questions_root=None) -> dict:
+def _refuse_judge_sampling(args: argparse.Namespace) -> None:
+    """Exit 2 before any answer is paid for when a CLI judge is given a temperature (the judge would refuse later)."""
+    try:
+        check_sampling(getattr(args, "judge_provider", DEFAULT_JUDGE_PROVIDER),
+                       getattr(args, "judge_temperature", None), None)
+    except SamplingError as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        sys.exit(EXIT_USAGE)
+
+
+def _run_single_library(entry, output_dir: str, model: str, provider: str, judge_model: str, judge_provider: str = DEFAULT_JUDGE_PROVIDER, doc_source_override=None, max_tokens_per_question: int = DEFAULT_MAX_TOKENS_PER_QUESTION, force_regen: bool = False, concurrency: int = DEFAULT_CONCURRENCY, questions_from=None, questions_root=None, judge_temperature=None, judge_placeholder=False, registry=None) -> dict:
     """Run full evaluation pipeline for one ProductEntry. Returns result dict."""
     from dokimi.orchestrator import EvaluationPipeline
     from pathlib import Path as _Path
@@ -60,12 +71,15 @@ def _run_single_library(entry, output_dir: str, model: str, provider: str, judge
         provider=provider,
         judge_model=judge_model,
         judge_provider=judge_provider,
+        judge_temperature=judge_temperature,
+        judge_placeholder=judge_placeholder,
         context7_id=entry.context7_id,
         doc_source=doc_source,
         max_tokens_per_question=max_tokens_per_question,
         force_regen=force_regen,
         questions_from=questions_from,
         product_key=entry.key,
+        registry=registry,
         questions_root=_Path(questions_root) if questions_root else None,
     )
     result = pipeline.run(concurrency=concurrency)
@@ -301,6 +315,7 @@ def refuse(message: str) -> None:
 def cmd_benchmark_run(args: argparse.Namespace) -> None:
     """Run full evaluation pipeline for a single registered library."""
     import statistics as _stats
+    _refuse_judge_sampling(args)
     registry = _load_registry(args)
     try:
         entry = registry.get(args.library)
@@ -334,6 +349,9 @@ def cmd_benchmark_run(args: argparse.Namespace) -> None:
             concurrency=concurrency,
             questions_from=questions_from,
             questions_root=questions_root,
+            judge_temperature=getattr(args, "judge_temperature", None),
+            judge_placeholder=getattr(args, "judge_placeholder", False),
+            registry=registry,
         )
     else:
         print(f"\n🔁 Multi-run mode: {n_runs} evaluation passes")
@@ -356,6 +374,9 @@ def cmd_benchmark_run(args: argparse.Namespace) -> None:
                 concurrency=concurrency,
                 questions_from=qfrom,
                 questions_root=questions_root,
+                judge_temperature=getattr(args, "judge_temperature", None),
+                judge_placeholder=getattr(args, "judge_placeholder", False),
+                registry=registry,
             )
             try:
                 eval_summary = r["result"]["steps"]["evaluation"]["summary"]
@@ -375,6 +396,7 @@ def cmd_benchmark_run(args: argparse.Namespace) -> None:
 
 def cmd_benchmark_batch(args: argparse.Namespace) -> None:
     """Run evaluation pipeline for multiple registered libraries."""
+    _refuse_judge_sampling(args)
     registry = _load_registry(args)
 
     if args.all_libraries or not args.libraries:
@@ -410,6 +432,7 @@ def cmd_benchmark_batch(args: argparse.Namespace) -> None:
                 force_regen=getattr(args, "force_regen", False),
                 concurrency=getattr(args, "concurrency", DEFAULT_CONCURRENCY),
                 questions_root=getattr(args, "questions_root", None),
+                registry=registry,
             )
             results.append(r)
         except Exception as exc:
@@ -456,6 +479,10 @@ def register(sub, positive_int) -> None:
                              help=JUDGE_MODEL_HELP)
     bench_run_p.add_argument("--judge-provider", default=DEFAULT_JUDGE_PROVIDER, dest="judge_provider",
                              choices=JUDGE_PROVIDERS)
+    bench_run_p.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
+                             help="Judge sampling temperature, recorded as judge_temperature in the eval output (default: provider default)")
+    bench_run_p.add_argument("--judge-placeholder", action="store_true", dest="judge_placeholder",
+                             help="Judge the no-retrieval placeholder answers too (default: they are not scored)")
     bench_run_p.add_argument("--registry", default=None, help="Path to custom products.yaml")
     bench_run_p.add_argument("--max-tokens", type=positive_int, default=DEFAULT_MAX_TOKENS_PER_QUESTION, dest="max_tokens",
                              help="Max tokens to retrieve per question from doc source (default: 4000)")

dokimi/dokimi/commands/evaluate.py

@@ -18,6 +18,7 @@
     DEFAULT_JUDGE_PROVIDER,
     JUDGE_MODEL_FOR_PROVIDER,
     JUDGE_MODEL_HELP,
+    answers_method_version,
 )
 from dokimi.eval.judge import JUDGE_PROVIDERS
 from dokimi.eval.judge_validity import DEFAULT_MIN_ACCURACY, NoUsablePairs
@@ -198,13 +199,21 @@ def cmd_eval_score(args: argparse.Namespace) -> None:
             context="eval-score",
         )
 
+    try:
+        method_version = answers_method_version(run.meta)
+    except ValueError as exc:  # refused before any judge call is paid for
+        print(f"Error: {args.answers}: {exc}", file=sys.stderr)
+        sys.exit(EXIT_USAGE)
     print(f"Evaluating with judge: {args.judge_provider}/{args.judge_model}")
 
     # Evaluate
     judge = Judge(
         model=args.judge_model,
         provider=args.judge_provider,
         temperature=args.judge_temperature,
+        judge_placeholder=getattr(args, "judge_placeholder", False),
+        # The eval says which answering method made the answers it judges; an unstamped file is legacy.
+        answers_method_version=method_version,
         run_metadata={
             "question_model": run.meta.get("question_model", "unknown"),
             "question_provider": run.meta.get("question_provider", "unknown"),
@@ -480,6 +489,8 @@ def register(sub, positive_int) -> None:
     score_p.add_argument("--concurrency", type=positive_int, default=DEFAULT_CONCURRENCY, help="Parallel judge calls (default: 5)")
     score_p.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
                          help="Judge sampling temperature, recorded as judge_temperature in the output")
+    score_p.add_argument("--judge-placeholder", action="store_true", dest="judge_placeholder",
+                         help="Judge the no-retrieval placeholder answers too (default: they are not scored)")
     score_p.set_defaults(func=cmd_eval_score)
 
     # eval ragas

dokimi/dokimi/commands/orchestrate.py

@@ -68,6 +68,7 @@ def cmd_evaluate(args: argparse.Namespace) -> None:
             judge_model=args.judge_model,
             judge_provider=args.judge_provider,
             judge_temperature=args.judge_temperature,
+            judge_placeholder=getattr(args, "judge_placeholder", False),
             personas_count=args.personas_count,
             questions_per_topic=args.questions_per_topic,
             # Only pass top_k when explicitly set; otherwise let pipeline use config default
@@ -130,6 +131,8 @@ def register(sub, positive_int) -> None:
     eval_p.add_argument("--judge-provider", default=DEFAULT_JUDGE_PROVIDER, choices=JUDGE_PROVIDERS)
     eval_p.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
                         help="Judge sampling temperature, recorded as judge_temperature in the eval output")
+    eval_p.add_argument("--judge-placeholder", action="store_true", dest="judge_placeholder",
+                        help="Judge the no-retrieval placeholder answers too (default: they are not scored)")
     eval_p.add_argument("--personas-count", type=positive_int, default=5, help="Target number of personas")
     eval_p.add_argument("--questions-per-topic", type=positive_int, default=2, help="Questions per topic per persona")
     eval_p.add_argument("--top-k", type=positive_int, default=None, help="Docs to retrieve before reranking (default from config/products.yaml retrieval.top_k)")