feat(repo): dokimi:typecheck grader and repo scan --typecheck for Python fences

dokimi/changelog.d/20261031_120000_napetrov_repo-typecheck.md

@@ -0,0 +1,15 @@
+### Added
+
+- **`dokimi:typecheck` built-in grader and `repo scan --typecheck` (Python fences).** The grader type-checks the
+  task's listed files with the image's pinned mypy and passes when none has an error (`tool`, `version`, `paths`
+  in `task.json`; a mismatched version is a grader failure). `repo scan --typecheck` (with `--typecheck-tool mypy|basedpyright`) checks
+  each Python fence of the guidance against `--python-env`'s site-packages: a fence with an error is a `fail`
+  stale-API candidate, a fence that does not parse or has no interpreter is `unresolved`. The report gains a
+  `typecheck` block, `null` without the flag. See `docs/repo-evaluation-contract.md` §4.5 and §8.2.
+- A `dokimi:typecheck` task needs no `reference/` directory: with one, its files are the repair the grade
+  reads; without one, the listed files are graded as they are and the self-check runs the null step only,
+  reporting a listed path that is missing or a suppression the workspace already has (no run could pass).
+- A tool report without its diagnostic fields (for example `{}`) is a tool failure, not a clean run.
+- A fence that imports a module the environment lacks is `unresolved`, not `fail`. A fence whose only errors are
+  a statement that needs a function or loop around it (`break`, `continue`, `yield from`, `nonlocal`) is unresolved too.
+- The grader refuses a workspace module named as one the image's site-packages provides, as it does a standard-library name.

dokimi/docs/reference/cli-interface.json

@@ -12152,6 +12152,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--python-env",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_StoreAction",
               "choices": null,
@@ -12167,6 +12183,41 @@
               "position": null,
               "required": false,
               "type": null
+            },
+            {
+              "action": "_StoreTrueAction",
+              "choices": null,
+              "const": true,
+              "default": false,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--typecheck",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
+            {
+              "action": "_StoreAction",
+              "choices": [
+                "'basedpyright'",
+                "'mypy'"
+              ],
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--typecheck-tool",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": null
             }
           ],
           "path": "dokimi repo scan",

dokimi/docs/repo-evaluation-contract.md

@@ -552,7 +552,8 @@ measure the patch's text, such as a guidance line, not the tool.
 For every task, before any paid run:
 
 - the null workspace (prepared, untouched) must fail the gate
-- the workspace after `oracle` must pass the gate
+- the workspace after `oracle` must pass the gate (a `dokimi:typecheck` task with no `reference/`
+  has no oracle step: its null step alone must fail, §4.5)
 - for score-only tasks, the oracle must reach `score.max` on `score.metric`
 - the agent container must not be able to read any file under `tasks` or
   `{source_dir}` (§1.1)
@@ -601,7 +602,8 @@ repository's checks with its own without knowing their rules.
 A task whose `grader` is `dokimi:<name>` is graded by the framework, not by
 the repository's `grade` and `oracle` entry points; any other name goes to
 those entry points. Such a task is data only: its directory holds
-`task.json`, the patch and the JSON files the grader reads. The `dokimi:`
+`task.json`, the patch and the JSON files the grader reads (and, for
+`dokimi:typecheck`, an optional `reference/` directory, §4.5 below). The `dokimi:`
 prefix is the framework's: an unknown `dokimi:` name fails `tasks check` and
 the run's grade, and is never handed to the repository.
 
@@ -611,7 +613,9 @@ framework's grader code is mounted read-only at `/eval/graders` (never into
 the agent's container) and runs as
 `python3 -I -B /eval/graders/main.py grade|oracle <name> <task dir> /eval/run`,
 where `<task dir>` is the task's directory under `{eval_dir}`. It uses the
-standard library only and the image's `python3` (3.8 or newer). `-I` keeps
+standard library only and the image's `python3` (3.8 or newer), except
+`dokimi:typecheck`, which imports the image's mypy (the pinned one, on Python
+3.10 or newer). `-I` keeps
 `PYTHONPATH` from `image.env` and the workspace off `sys.path`; `-B` prevents
 writing `.pyc` files to the read-only mount. An image with no `python3` on
 `PATH` is a grader failure that says so: the grader exits 127 and its
@@ -625,7 +629,8 @@ other exit 127 is reported as a plain `grade exited 127`.
   `pass: null` with `errors`. A metric the task's `gate` names that the grader
   did not produce is `null`.
 - `oracle` writes the reference answer to `answer.txt` and leaves the
-  workspace alone.
+  workspace alone, except `dokimi:typecheck`, whose `oracle` copies `reference/`
+  over the workspace (see its paragraph below).
 
 A review task with `pass: null` is scored, not gated: give it `score`. A `gate`
 on its metrics is not applied (`success` is null), and without `score` the run
@@ -654,6 +659,28 @@ failed the task, and on a task with `score` that is a failed run with score
 |---|---|---|---|---|
 | `dokimi:review_seeds` | `manifest.json` (`manifest`); oracle answer `oracle.json` (`oracle_answer`), a list of findings | `null`; for a manifest with no seeds (a clean-diff control), `true` when no finding is `blocker` or `major` | `n_findings`, `n_severe`, `n_decoys`, `unmatched`, `recall` (`null` with no seeds) | `seeds_hit`, `decoys_flagged`, `recall_by_class` (`"hit/total"`) |
 | `dokimi:review_replay` | `gold.json` (`gold`); `task.json` `window`, default 5 | `null` (score-only) | `n_findings`, `n_severe`, `n_threads`, `recall` (= `recall_all`), `recall_addressed`, `precision_proxy` | `threads_hit`, `matches`, `recall_by_kind`, `decoys_flagged` (always `[]`) |
+| `dokimi:typecheck` | none: the task's own fields `tool` (`mypy`), `version` (the image's mypy, exactly), `paths` (workspace-relative files) | `true` when no error is reported on a listed file, every listed file exists, and nothing silences mypy, else `false` | `n_errors`, `n_missing`, `n_suppressions`, `n_files` | `errors` (first 50, `file:line: message [code]`, and each silenced spot), `missing`, `tool`, `tool_version` |
+
+`dokimi:typecheck` checks the listed files only. Its pass is "no error on a
+listed file" (an error in a module a listed file imports belongs to that module,
+not to the listed file), and a listed file that is missing fails. Silencing mypy fails too: a `# type: ignore` or `# mypy:` comment or the name `no_type_check` in a listed file, or a `.pyi` anywhere in the workspace (it would replace the module it names), or a top-level module or package named as a standard-library one or one the image's site-packages provides (`json.py`, `requests/__init__.py`, the same replacement), is a suppression, and the grade counts it in `n_suppressions`. A listed path
+reached through a symbolic link is missing, not read. A symbolic link in the workspace whose target
+lies outside it refuses the whole grade (a grader error, not a pass), listed or not, whenever mypy runs, that is
+when some listed file is present: mypy follows an import through a link, so an unlisted module could supply what a
+listed file imports. When every listed file is missing, mypy does not run, and the grade is a failure with
+`n_missing` set, not a grader error, even if a link lies outside. The exception
+is a blocking error (a syntax error, a duplicate module) that stops mypy before
+it reaches the listed files: then that error is the grade's, and the grade fails.
+`reference/` is optional. The task's `oracle` copies its `reference/` directory, when it has one, over the
+workspace, so a reference repair is what gets graded; its answer is empty. Without `reference/` the grade
+reads the listed files as they are, and the self-check (§4.3) runs the null step only: the untouched
+listed files must fail, since there is no reference repair for the oracle step to pass.
+It runs mypy in-process in the grading container, so the image's `python3`
+must import the pinned mypy: a `version` other than the installed one is a
+grader failure, not a fail. It is not "no new errors against the base": the
+grader is stdlib-only and cannot run `git` to read the base, and it does not
+start basedpyright (the L0 check, §8.2, runs mypy or basedpyright on the host, as `--typecheck-tool` selects). A repository that
+needs that comparison writes it in its own `grade` entry point.
 
 `manifest.json` is `{"seeds": [{id, class, file, lines, kw}], "decoys":
 [{id, file, lines, kw}]}`; `file` `"*"` matches any file, and `kw` is a
@@ -1507,6 +1534,55 @@ oneDAL they tracked agent cost or error over 39 modules only, which cannot
 separate structure from question design.
 Whether the index is current is not checked: `not_observed` says so.
 
+### 8.2 Python fences (`repo scan --typecheck`)
+
+`--typecheck` type-checks every fence tagged
+`python`, `py` or `python3` in the guidance files, with `--typecheck-tool mypy|basedpyright`
+(default mypy). Each fence that parses is one
+`python_fence` claim, so the report and `--fail-on-fail` treat it like any other
+claim:
+
+- `ok`: the tool reports no error in the fence.
+- `fail`: a stale-API candidate. The claim's line is the error's line in the
+  guidance file, and its `why` names the first error (`mypy: line N: message`).
+- `unresolved`: the fence is not valid Python (a placeholder such as `f(<arg>)`),
+  is too deeply nested for the parser, or no `--python-env` was given, so nothing
+  was checked. It is also unresolved when its only errors are missing context (any
+  undefined name, as from an earlier fence or an omitted import, so a removed builtin such as
+  `unicode` is unresolved too; `await`/`return`/`yield`/`yield from`, `break`/`continue` outside a loop,
+  `async with`/`async for` outside a function, `nonlocal` at module level), or a module the
+  environment does not have (see below), or when the
+  repository is older than the tool checks (mypy: Python 3.10). The reason says which.
+
+`--python-env PYTHON` names the repository's interpreter, a virtualenv with its
+pinned dependencies installed (and the repository itself, if its fences import
+it). The fences are checked against that site-packages with imports followed
+silently (`--follow-imports=silent`), so only errors in the fence count. A
+module the environment does not have is `unresolved`, not `fail`: the fence's own
+import fails, so nothing it says about that module can be checked, and a fence that
+imports an uninstalled package is not a stale-API candidate. A real error in the same fence still
+fails it. `pip install
+dokimi` does not install mypy: the tested pin is `requirements-test.txt`
+(`mypy==2.4.0`), and a missing mypy exits 2. basedpyright must be on `PATH` and
+is recorded with the version it reports, not pinned by dokimi. The report's
+`typecheck` block names the tool, its version, the interpreter (`null`: unknown) and the counts
+(`fences`, `checked`, `fail`, `unknown`, `syntax`); it is `null` without the
+flag. `--python-env` and `--typecheck-tool` need `--typecheck`. The fences are parsed with the
+grammar of the repository interpreter's version, and mypy runs with that
+`--python-version` and an explicit empty config, so the subject's and the host's
+mypy settings do not apply. basedpyright gets the same version as `pythonVersion`
+in its explicit project file, since `--pythonpath` alone sets the import paths. The tools run on the host through
+`dokimi.process`; one run covers every fence of the repository. When mypy stops on a blocking error, it runs
+again on the fences that error did not name, so one fence cannot leave the others unchecked.
+
+```bash
+python cli.py repo scan <source> --manifest <path> --typecheck --python-env <repo-venv>/bin/python
+```
+
+A guidance note under test can be scored the same way as any other content:
+feed the note as a `patch:<file>` arm (see §3) and grade the arm's workspace with
+the `dokimi:typecheck` grader above.
+
 ## 9. Running
 
 `repo tasks check` and `repo run` need docker. Both export `eval_dir` from the

dokimi/dokimi/commands/repo.py

@@ -14,6 +14,7 @@
 from typing import Any
 
 from dokimi.exit_codes import EXIT_USAGE
+from dokimi.commands.repo_scan import _print_scan_text, _typecheck_or_exit
 
 
 def _normalize_whitespace(text: str, max_len: int | None = 100) -> str:
@@ -126,11 +127,17 @@ def cmd_repo_scan(args: argparse.Namespace) -> None:
     """L0: check every mechanically checkable claim in the guidance against the same tree."""
     from dokimi.artifacts import save_artifact, stamp
     from dokimi.repos.l0 import scan_repo
+    from dokimi.repos.l0.typecheck import TypecheckError
 
     _, tree = _source_tree(args)
     manifest = _manifest_or_exit(tree, args, guidance_default=True, also_usage=(OSError,))
     static = _read_static(args.static) if args.static else None
-    payload = scan_repo(tree, manifest, static)
+    check = _typecheck_or_exit(args)
+    try:
+        payload = scan_repo(tree, manifest, static, typecheck=check)
+    except TypecheckError as exc:
+        print(f"✗ repo scan --typecheck: {exc}", file=sys.stderr)  # the error says whether the tool or the interpreter
+        sys.exit(EXIT_USAGE)
     if args.out:
         try:
             save_artifact("repo_readiness", payload, Path(args.out))
@@ -144,25 +151,7 @@ def cmd_repo_scan(args: argparse.Namespace) -> None:
     if args.json:
         print(json.dumps(stamp("repo_readiness", dict(payload)), indent=2))
     else:
-        c = payload["counts"]
-        detected = " (detected)" if payload["adapters_detected"] else ""
-        print(f"{manifest.id}: {len(payload['files'])} guidance files in {payload['repo']['tree']}; "
-              f"adapters: {', '.join(payload['adapters']) or 'none'}{detected}")
-        print(f"  claims: {c['total']} (ok {c['ok']}, unresolved {c['unresolved']}, fail {c['fail']})")
-        for warning in payload["warnings"]:
-            print(f"  warning: {warning}")
-        if payload["incomplete"]:
-            print(f"  incomplete: {'; '.join(payload['incomplete_reasons'])} (repo scan --fail-on-fail exits 1)")
-        if static_info is not None and not static_errors:
-            r = payload["by_source"]["repository"]
-            print(f"  repository static: {r['total']} claims, {r['fail']} fail; "
-                  f"{static_info['superseded']} framework claims replaced")
-        for claim in payload["claims"]:
-            if claim["status"].split("_", 1)[0] == "fail":
-                print(f"  FAIL {claim['file']}:{claim['line']} [{claim.get('kind')}] "
-                      f"{str(claim.get('claim'))[:90]} -- {claim.get('why')}")
-        if args.out:
-            print(f"wrote {args.out}")
+        _print_scan_text(manifest, payload, args.out)
     _scan_gate(payload, args.fail_on_fail)  # runs first: an invalid --static still gets the incomplete reason
     if static_errors:
         sys.exit(1)
@@ -665,6 +654,17 @@ def register(sub, positive_int) -> None:
                         "the report, its claims replacing the framework's on the same file, line and text. "
                         "A file that cannot be read or is not JSON exits 2; a schema-invalid document is "
                         "reported and makes the command exit 1")
+    scan_p.add_argument("--typecheck", action="store_true", default=False,
+                        help="Type-check each Python fence of the guidance (§8.2): one claim per fence, fail when the "
+                        "tool reports an error. The tool is mypy unless --typecheck-tool names basedpyright (on PATH). "
+                        "mypy is not installed by pip install dokimi: the tested pin is requirements-test.txt "
+                        "(mypy==2.4.0); a missing mypy exits 2. Without --python-env the fences are unknown (unresolved), "
+                        "not checked")
+    scan_p.add_argument("--typecheck-tool", default=None, dest="typecheck_tool", choices=("mypy", "basedpyright"),
+                        help="The tool --typecheck runs (default mypy). Needs --typecheck")
+    scan_p.add_argument("--python-env", default=None, dest="python_env", metavar="PYTHON",
+                        help="The repository's interpreter (a venv with its pinned dependencies installed); the fences "
+                        "are checked against its site-packages. Needs --typecheck")
     scan_p.add_argument("--out", default=None, help="Write the repo_readiness.v1 report to this file")
     scan_p.add_argument("--json", action="store_true", help="Print the full report as JSON")
     scan_p.add_argument("--fail-on-fail", action="store_true", dest="fail_on_fail",