feat(cli): eval group: move the evaluation commands under eval (#94 M1)

.agent-eval/Dockerfile

@@ -2,9 +2,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 # Build image for dokimi's own repository-mode evals: CI's Python, the lock, every CI tool,
-# and a warm uv cache so `make dev` inside a run works without downloading anything.
-# Pinned by digest, like the tools below, so a rerun grades in the same image. To move it:
-# `docker buildx imagetools inspect python:3.11-slim-bookworm`, then `repo tasks check`.
+# pinned by digest. To move the pin: `docker buildx imagetools inspect python:3.11-slim-bookworm`, then `eval check`.
 FROM python:3.11-slim-bookworm@sha256:0a310eeecf4e1f5a0743f9a6520c90c88d089c903ca5fd283f501e3a805f5f89
 # Debian's point releases replace these package versions, so pinning them would break the build.
 # hadolint ignore=DL3008
@@ -17,6 +15,7 @@ COPY dokimi/requirements.lock.txt dokimi/requirements-ci.lock.txt dokimi/require
 # The lock and the tools CI installs (uv among them), hash-checked so a rerun grades alike.
 RUN pip install --no-cache-dir --require-hashes -r /opt/dokimi-req/requirements.lock.txt \
        -r /opt/dokimi-req/requirements-ci.lock.txt
+# A warm uv cache, so `make dev` inside a run downloads nothing.
 ENV UV_CACHE_DIR=/opt/uv-cache UV_PYTHON_PREFERENCE=only-system UV_LINK_MODE=copy
 RUN uv venv --python 3.11 --seed /tmp/warm && uv pip install --python /tmp/warm/bin/python \
        --require-hashes -r /opt/dokimi-req/requirements.lock.txt -r /opt/dokimi-req/requirements-ci.lock.txt \

.agent-eval/README.md

@@ -42,11 +42,11 @@ Process metrics are reported and never gated. They come from `events.jsonl` and
 
 ## Running
 
-`repo tasks check` and `repo run` need Docker. Run them from `dokimi/`:
+`eval check` and `repo run` need Docker. Run them from `dokimi/`:
 
 ```bash
 # self-check, no LLM: the untouched workspace fails each gate, the reference solution passes it
-python cli.py repo tasks check .. --manifest .agent-eval/repo-eval.yaml -j 4
+python cli.py eval check .. --manifest .agent-eval/repo-eval.yaml -j 4
 
 # agents: one cell per (task, arm, model, rep)
 python cli.py repo run .. --manifest .agent-eval/repo-eval.yaml \
@@ -79,4 +79,4 @@ A task is a directory under `tasks/` with these files:
 Write the prompt the way a maintainer would ask, and leave out the repository's
 conventions: those are what the guidance is measured on.
 
-A task is ready when `repo tasks check` reports it `OK`.
+A task is ready when `eval check` reports it `OK`.

.github/workflows/ci.yml

@@ -399,9 +399,8 @@ jobs:
       - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
         with:
           python-version: '3.13'
-      # The `[ragas]` extra is not in requirements.txt or the lock; this job is
-      # where `eval ragas` runs on real packages, and DOKIMI_REQUIRE_RAGAS=1 makes
-      # tests/test_ragas_runtime.py fail rather than skip if ragas is missing.
+      # The `[ragas]` extra is not in requirements.txt or the lock; this job is where
+      # `eval score ragas` runs on real packages (DOKIMI_REQUIRE_RAGAS=1 fails, not skips).
       - name: Install newest allowed deps
         run: pip install --upgrade -r requirements.txt -r requirements-test.txt -e '.[ragas]'
       - name: Run tests

.github/workflows/repo-eval.yml

@@ -3,26 +3,8 @@
 
 name: repo-eval
 
-# The no-LLM checks of a repository's evals (dokimi/docs/repo-evaluation-contract.md),
-# for repositories to call from their own workflow:
-#
-#   validate     `repo validate`: the manifest and every task.json (seconds; a gate)
-#   scan         `repo scan`: the L0 claims of the guidance (report only)
-#   tasks-check  `repo tasks check`, one job per selected task: the untouched
-#                workspace fails the gate, the reference solution passes it
-#
-# Nothing here calls a model or reads a secret. The caller pins dokimi twice,
-# by the same commit SHA (or tag): in `uses: ...@<sha>` and in `dokimi-ref`. The
-# ref installs dokimi and, through its requirements.lock.txt, every dependency,
-# so a rerun checks with the same framework. A branch name is refused. The
-# calling job grants `contents: read`, all any job here asks for.
-#
-# Task selection (`tasks`): `changed` on a pull request is the tasks whose
-# directories it touches, plus `smoke-tasks` when it changes anything else the
-# self-check depends on: the eval directory outside its tasks (harness,
-# graders, Dockerfile; Markdown excepted), the manifest, the calling workflow
-# file, or a path under `smoke-paths`. On a push `changed` diffs against the
-# previous tip; on any other event it means `all`.
+# The no-LLM checks of a repo's evals, as a reusable workflow: no model, no secret.
+# Jobs, task selection and the pin rule: dokimi/docs/repo-evaluation-contract.md, "CI for a repository".
 
 on:
   workflow_call:
@@ -74,14 +56,14 @@ on:
         default: 4
       warm:
         description: >-
-          Pass --warm to `repo tasks check`: null and oracle run with network so a
+          Pass --warm to `eval check`: null and oracle run with network so a
           cold cache fills (an ephemeral runner starts with empty caches)
         required: false
         type: boolean
         default: false
       allow-unsupported:
         description: >-
-          Pass --allow-unsupported to `repo tasks check`: a task this runner misses the task.json
+          Pass --allow-unsupported to `eval check`: a task this runner misses the task.json
           `host` requirements of passes its job (the reason is still in the log and summary)
         required: false
         type: boolean
@@ -375,7 +357,7 @@ jobs:
           pip install --no-deps "dokimi @ git+https://github.com/$DOKIMI_REPO@$SHA#subdirectory=dokimi"
       # Builds the image from the manifest's Dockerfile (the tree without its
       # tasks), then grades the null and the oracle workspace in it.
-      - name: repo tasks check
+      - name: eval check
         env:
           MANIFEST: ${{ inputs.manifest }}
           TASK: ${{ matrix.task }}
@@ -387,7 +369,7 @@ jobs:
           flags=()
           if [ "$WARM" = true ]; then flags+=(--warm); fi
           if [ "$ALLOW_UNSUPPORTED" = true ]; then flags+=(--allow-unsupported); fi
-          dokimi repo tasks check . --manifest "$MANIFEST" --task "$TASK" --out "$RUNNER_TEMP/repo-check" "${flags[@]}"
+          dokimi eval check . --manifest "$MANIFEST" --task "$TASK" --out "$RUNNER_TEMP/repo-check" "${flags[@]}"
       # always(): a hung oracle ends in the job timeout, which cancels the job; a
       # cancelled job still runs these steps, so the run leaves a verdict.
       - name: Summary

dokimi/CONTRIBUTING.md

@@ -49,7 +49,7 @@ The package exposes a console entry point:
 dokimi --help           # equivalent to `python -m dokimi`
 ```
 
-Optional feature sets are available as extras: `ragas` (`eval ragas`, which
+Optional feature sets are available as extras: `ragas` (`eval score ragas`, which
 `dev` includes), `ocr` (Docling ingestion), `mcp`, and `embeddings`. Install
 what you need, e.g. `pip install -e ".[dev,ocr]"`.
 

dokimi/README.md

@@ -67,7 +67,7 @@ The project is pre-1.0; interfaces and artifact schemas may still change.
   `AWS_REGION` + `AWS_BEARER_TOKEN_BEDROCK` for Bedrock). `GITHUB_TOKEN` is
   optional and raises the persona-discovery rate limit.
 - Docker, only for the executable-task, harness and repository tracks.
-- Optional extras: `ragas` (`eval ragas`, `answers generate --ragas`), `ocr`
+- Optional extras: `ragas` (`eval score ragas`, `answers generate --ragas`), `ocr`
   (Docling ingestion) and `mcp` (real MCP servers).
 
 ```bash
@@ -123,10 +123,10 @@ The top-level CLI is `python cli.py` (or `dokimi` once installed):
 Context benchmark
   benchmark           Preflight and run the benchmark for registered targets
   evaluate            Run the full pipeline in one command
-  personas            Discover and approve target user personas
-  questions           Generate, analyze, refine, and panel-review questions
+  eval personas       Discover and approve target user personas
+  eval questions      Generate, analyze, refine, and review questions
   answers             Generate context-arm and baseline answers
-  eval                Score answers, run RAGAS, or use a multi-judge panel
+  eval                Build and check eval sets, score answers, calibrate judges
   arms                Compare treatment arms (docs / MCP / skills / profiles)
   subjects            Run a subject descriptor (artifact + suite) into a scorecard
 
@@ -156,7 +156,7 @@ That is the curated set; `python cli.py --help` lists every group.
 Use `--help` on any command for detailed options, for example:
 
 ```bash
-python cli.py questions --help
+python cli.py eval questions --help
 python cli.py eval --help
 python cli.py dashboard generate --help
 ```

dokimi/SECURITY.md

@@ -97,7 +97,7 @@ fix the locks can take:
     that can create files in the lock directory gets a stale-lock break to meet
     a reused inode (ext4, xfs).
 
-The locks do not contain ragas: it is the optional `[ragas]` extra (`eval ragas`,
+The locks do not contain ragas: it is the optional `[ragas]` extra (`eval score ragas`,
 `answers generate --ragas`), which installs ragas 0.3.9 and diskcache 5.6.3. Their
 advisories do not affect Dokimi:
 

dokimi/changelog.d/20261030_160000_napetrov_eval-group-m1.md

@@ -0,0 +1,41 @@
+### Changed
+
+- **The evaluation commands live under `eval` (#94 M1)** — `personas`, `questions` and the
+  evaluation leaves of `tasks`, `catalog` and `repo` are now under `dokimi eval`, and the
+  `eval` leaves are grouped into `eval score`, `eval judge`, `eval questions`, `eval personas`
+  and `eval triggers`. Every old path still answers, but it prints `moved: dokimi <new argv>`
+  to stderr and exits 2 without running anything, so a script fails loudly and names the new
+  command. `eval score` stays a live group: bare `eval score` exits 2 with a usage error, and
+  `eval score --help` lists its leaves. Move the scripts to the new paths before the next release. The arguments are
+  unchanged. `dokimi repo tasks replay`, `repo run`, `repo report` and `repo compare` stay where
+  they are.
+
+  | Before | After |
+  |---|---|
+  | `tasks inspect` | `eval inspect` |
+  | `repo tasks check` | `eval check` |
+  | `tasks list` | `eval tasks list` |
+  | `personas discover` | `eval personas discover` |
+  | `personas approve` | `eval personas validate` |
+  | `questions generate` | `eval questions generate` |
+  | `questions analyze` | `eval questions analyze` |
+  | `questions refine` | `eval questions refine` |
+  | `questions panel-review` | `eval questions review` |
+  | `questions import` | `eval import` |
+  | `catalog trigger-set` | `eval triggers generate` |
+  | `eval score` | `eval score pointwise` |
+  | `eval pairwise` | `eval score pairwise` |
+  | `eval checklist` | `eval score checklist` |
+  | `eval panel-score` | `eval score panel` |
+  | `eval grounding` | `eval score grounding` |
+  | `eval ragas` | `eval score ragas` |
+  | `eval reference` | `eval score reference` |
+  | `eval judge-validity` | `eval judge validate` |
+  | `eval judge-wording` | `eval judge wording` |
+  | `eval cross-judge` | `eval judge agreement` |
+  | `personas` (bare, or its help) | `eval personas` |
+  | `questions` (bare, or its help) | `eval questions` |
+
+  The `[free]`/`[paid]` labels in `--help` follow the commands. The `consumers` labels that
+  `tasks inspect` reports (and the `task_inspect.v1` schema enum) name the new paths. The
+  `repo tasks check` step of `repo-eval.yml` runs `dokimi eval check`; its job name is unchanged.

dokimi/data/README.md

@@ -35,7 +35,7 @@ content repository and are supplied to the framework by path —
 from `data/`.
 
 **Captured runs.** Answer runs, judge scores and static-benchmark snapshots are
-*outputs*, so none are committed. `cli.py answers generate`, `cli.py eval score` and
+*outputs*, so none are committed. `cli.py answers generate`, `cli.py eval score pointwise` and
 `cli.py run` write them to the git-ignored scratch dirs (`answers/`, `eval/`,
 `results/`, `reports/`, `baselines/`); the schema each one must satisfy is locked by
 `dokimi/schemas/*.json` and by `tests/test_artifacts.py`, not by a committed
@@ -54,7 +54,7 @@ Generated artifacts are the other case, and they do not land here. A question-ge
 run writes to the git-ignored scratch dir, and the result is evaluation content:
 
 ```bash
-python cli.py questions generate --product oneTBB \
+python cli.py eval questions generate --product oneTBB \
   --personas personas/onetbb.json \
   --output questions/onetbb.json      # git-ignored; commit it to the content repo
 ```

dokimi/data/skills/dokimi-design-evals/SKILL.md

@@ -57,7 +57,7 @@ in [references/example-scenario.md](references/example-scenario.md).
 Prefer deterministic checks: tests, file and diff assertions, exit codes,
 parsed output. Use an LLM judge only for qualities no check can capture; then
 use a written rubric, calibrate it
-(`python cli.py eval judge-validity --answers <answers.json> --sequential --strict`),
+(`python cli.py eval judge validate --answers <answers.json> --sequential --strict`),
 and keep a human review of a sample. The command calibrates only dokimi's own
 answer judge (rows with `question_id`), not a task's rubric judge; check that one
 by hand.

dokimi/data/skills/dokimi-design-evals/references/verifier-self-test.md

@@ -11,7 +11,7 @@ Repository tasks (`repo-eval.yaml` + `task.json`):
 python cli.py repo validate <source> --manifest <path/to/repo-eval.yaml> --fail-on-warning
 ```
 
-Harbor tasks: there is no validate command. `tasks list` only shows that the
+Harbor tasks: there is no validate command. `eval tasks list` only shows that the
 task was discovered (it reads `task.toml` and `instruction.md`); it never prints
 an error row, and it lists a task that has no `tests/test.sh`, `solution/` or
 `environment/`. A task without `instruction.md` aborts the listing with a
@@ -25,7 +25,7 @@ out as `verifier/` + `oracle/`, but the Harbor-backed arms (`codex`,
 the task will run in the paid arms.
 
 ```bash
-python cli.py tasks list --tasks-root <tasks-root>
+python cli.py eval tasks list --tasks-root <tasks-root>
 ```
 
 ## 2. Oracle passes, untouched workspace fails, isolation
@@ -35,7 +35,7 @@ read tasks or source; `docs/repo-evaluation-contract.md` §4.3). It exits 1 on
 any broken or unsupported task.
 
 ```bash
-python cli.py repo tasks check <source> --manifest <path/to/repo-eval.yaml> --task <id>
+python cli.py eval check <source> --manifest <path/to/repo-eval.yaml> --task <id>
 ```
 
 Harbor tasks: oracle and isolation have harnesses; expect `pass_rate: 1.0`
@@ -69,7 +69,7 @@ changing only the reference solution so it produces the mutated end state:
   `pass_rate: 0.0`.
 - Repository: write a scratch copy of `repo-eval.yaml` whose `oracle` command
   runs the real oracle and then the mutation, and pass it with
-  `--manifest-file` to `repo tasks check`. Expect the oracle check to fail.
+  `--manifest-file` to `eval check`. Expect the oracle check to fail.
   The oracle command is split into argv and run **without a shell**, so `&&`,
   pipes and redirects are passed to the program as arguments and the mutation
   never runs (the check then passes, which looks like a verifier that survived).
@@ -94,7 +94,7 @@ a correct answer (a negative task, an out-of-scope request), set
 `valid_inaction: true`: the null workspace is then expected to **pass**, and
 the "untouched fails" check does not apply. Use a mutation that does act
 wrongly instead (an unrequested edit). Do not reword the task so that
-inaction fails just to satisfy the template. Note that `repo tasks check`
+inaction fails just to satisfy the template. Note that `eval check`
 always requires the null to fail; a valid-inaction task needs a grader that
 fails on the unwanted action and its own manual null check.
 
@@ -103,14 +103,14 @@ fails on the unwanted action and its own manual null check.
 A rubric judge must tell good answers from degraded ones before its scores
 count. It costs model calls; the sequential mode stops early.
 
-`eval judge-validity` calibrates only dokimi's own answer judge: its answers
+`eval judge validate` calibrates only dokimi's own answer judge: its answers
 file needs a `question_id` and the question text per row. It stops with
 `missing question_id` (exit 2) on a task-rubric answers file, so a task's own
 rubric judge is not covered by this command. Check that judge by hand: run it on
 a known-good and a degraded answer you wrote, and have a person read the scores.
 
 ```bash
-python cli.py eval judge-validity --answers <answers.json> --sequential --strict
+python cli.py eval judge validate --answers <answers.json> --sequential --strict
 ```
 
 Then have a person review a sample of judged cells against the rubric and

dokimi/docs/README.md

@@ -66,7 +66,7 @@ checks the eval set itself: that it parses, that each task's grader fails an
 untouched workspace and passes the reference solution, and that an LLM judge
 scores degraded answers lower.
 
-![Three checks of an eval set: valid (dokimi repo validate), graded right (dokimi repo tasks check), judge checked (dokimi eval judge-validity)](assets/readme/eval-checks.png)
+![Three checks of an eval set: valid (dokimi repo validate), graded right (dokimi eval check), judge checked (dokimi eval judge validate)](assets/readme/eval-checks.png)
 
 **Where it is going:** mapping each claim in a skill to the tasks that
 exercise it, so the claims no eval tests are listed next to the score.
@@ -93,7 +93,7 @@ From the studies so far ([evidence](skill-novelty-evidence.md)):
 | `skill:` | a skill (`SKILL.md`) injected as context |
 | `profile:` | an agent persona system prompt |
 | `agent:` / `skill-agent:` | **agentic** use — the model gets a tool and *decides* whether to fetch docs / load the skill |
-| `skill-catalog:` | **selection** — every skill in a catalog is offered (a copy under a `template/` or `templates/` directory, a byte-identical mirror that loads under the same skill name, and anything under a vendored or VCS directory such as `.git`, `node_modules` or `.venv` are left out); scored on which one loads (`catalog trigger-set` builds the queries) |
+| `skill-catalog:` | **selection** — every skill in a catalog is offered (a copy under a `template/` or `templates/` directory, a byte-identical mirror that loads under the same skill name, and anything under a vendored or VCS directory such as `.git`, `node_modules` or `.venv` are left out); scored on which one loads (`eval triggers generate` builds the queries) |
 
 Two fine-print notes:
 
@@ -351,7 +351,7 @@ code.
   partly implemented)* what a repository provides to be evaluated as a whole:
   `repo-eval.yaml`, `task.json`, the `grade` entry point and `grade.json`,
   harness profiles, and the guidance reach probe; and how to run its task
-  self-check, agent runs, report and arm comparison (`repo tasks check`,
+  self-check, agent runs, report and arm comparison (`eval check`,
   `repo run`, `repo report`, `repo compare`). Tasks and graders stay in the scanned repository.
 - [repo-evaluation-mergeability.md](repo-evaluation-mergeability.md) —
   *(proposed)* grading whether a maintainer would merge a test-passing change:

dokimi/docs/adding-doc-source.md

@@ -70,7 +70,7 @@ python cli.py evaluate --product oneTBB \
   --doc-source okf:/path/to/onetbb-okf-bundle
 ```
 
-`--doc-source` is supported by `answers generate`, `questions generate`, and `evaluate`.
+`--doc-source` is supported by `answers generate`, `eval questions generate`, and `evaluate`.
 
 ---