diff --git a/dokimi/changelog.d/20261031_120000_napetrov_guidance-format-arms.md b/dokimi/changelog.d/20261031_120000_napetrov_guidance-format-arms.md new file mode 100644 index 000000000..1767c07e8 --- /dev/null +++ b/dokimi/changelog.d/20261031_120000_napetrov_guidance-format-arms.md @@ -0,0 +1,3 @@ +### Added + +- **`repo run` arms `guidance:format=xml` and `guidance:as-skill`** — the guidance files of the base, rendered as XML tags (each Markdown heading becomes a `
` element, checked to restore to the original file exactly), or moved into one skill directory (`.claude/skills/-guidance/SKILL.md`, the repo id made into an Agent Skills name). Both compare the same guidance content in another shape under the same model; `guidance:format=md` is refused as a duplicate of `guidance:raw`, and POML is not offered yet. A `guidance:as-skill` catalog arm refuses an `expected_skill` that names the generated skill, before any cell. diff --git a/dokimi/docs/benchmarking-and-comparison.md b/dokimi/docs/benchmarking-and-comparison.md index a46e330d8..e2628b276 100644 --- a/dokimi/docs/benchmarking-and-comparison.md +++ b/dokimi/docs/benchmarking-and-comparison.md @@ -301,6 +301,43 @@ both per row. Every cell also records `cache_read_tokens` and `cache_write_tokens` with the input and output tokens, and `repo report --json` gives their means. +### Guidance formats and packaging + +Two guidance arms answer "does the same content work better in another shape?" +The model stays the same, so run them per model (one `--model` each) and compare +the cells the way the skill arms are compared above. + +- `guidance:format=xml`: every guidance file of the base, rendered as XML tags. + Each Markdown heading becomes `
`, closed before + the next heading of its level or above; `` wraps the file. No other + line changes. The rendering is checked before it is written: it must restore to + the original file exactly, so a file with a line that looks like one of these + tags (or a heading the restore cannot rebuild) fails the cell's preparation + rather than measuring altered content. A heading inside a code fence is body + text. A heading line that is not `#`…`######`, one space, then the title, with no + carriage return, is body text too: a file with CRLF line endings has no sections + at all, only the wrapper. `guidance:format=md` is `guidance:raw` + and is refused as a second name for it. POML is not offered yet: its documented + components (`role`, `task`, `example`, ...) have no heading tag, so a POML arm + needs a decision about which element a heading maps to. +- `guidance:as-skill`: the guidance files are removed (as `guidance:none` removes + them) and their text goes into `.claude/skills/-guidance/SKILL.md`, each + file under a `## File: ` line with its text verbatim (a missing final newline is added); a symlink to another + guidance file (`CLAUDE.md` to `AGENTS.md`) is packaged once under that file, and a + symlink to any other file fails the cell. The frontmatter + description is one fixed sentence for every repository, so the skill is + selected on its description alone. Nested guidance files lose the directory + they applied to; the skill's name, path and sha256 are recorded in + `meta.json` under `guidance_skill`. Its usage is recorded separately in + `result.json` as `guidance_skill_loaded` (`null` if the trace is unreadable), + and is not currently aggregated by `repo report`. + +Reading the two together: `guidance:format=xml` against `guidance:raw` isolates +the markup, and `guidance:as-skill` against `guidance:raw` isolates the packaging +(always in context, or loaded on demand), with the fixed description a known +confound of the second. The arms do not rewrite the content, so a difference +comes from the format or the packaging, not from edited guidance. + ### Reading the significance tables Arms are compared **paired per question** against the one shared `baseline` diff --git a/dokimi/docs/preparing-a-repository.md b/dokimi/docs/preparing-a-repository.md index 7831f32fd..0b52048fe 100644 --- a/dokimi/docs/preparing-a-repository.md +++ b/dokimi/docs/preparing-a-repository.md @@ -6,7 +6,8 @@ SPDX-License-Identifier: Apache-2.0 # Preparing a repository for coding agents > **Status: recommendations.** §1–§7 apply today. The measurement loop in §8 runs -> today for guidance arms (`guidance:none`, `guidance:raw`, `guidance:bridge`). +> today for guidance arms (`guidance:none`, `guidance:raw`, `guidance:bridge`, +> `guidance:as-skill`, `guidance:format=xml`). > `guidance:bridge` measures the `CLAUDE.md` import (§7). The `skill:` and > `skill-catalog:` arms, which install skills, run today too > ([skill arms](benchmarking-and-comparison.md#skill-arms-in-repository-mode)). Arms diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index b8be2e30e..7ac15bbc1 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -308,7 +308,7 @@ belongs to the repository's grader and is passed through unread, except `grader` | `timeout_s`, `max_turns`, `budget_usd` | no | framework | per-task caps, overriding run defaults; each a finite number > 0 | | `host` | no | framework | `{"min_cpus": , "cpu_vendors": [], "cpu_flags": []}`, every key optional: what the hidden test needs to show the bug (a thread race that needs 3+ CPUs, an ISA path). Vendor and flag names are `/proc/cpuinfo`'s (`GenuineIntel`, `avx512f`); both match in any case, and a vendor other than `GenuineIntel`, `AuthenticAMD`, `HygonGenuine`, `CentaurHauls`, `Shanghai` is refused (`repo validate` too). A probe that read neither vendor nor flags from `/proc/cpuinfo` is a probe failure, not a miss They are checked against a probe container started like the grading one: CPUs are its affinity mask capped by its cgroup CPU quota (`docker --cpus`). On a host that misses them the task is unsupported (§4.3), and the probed facts go to `meta.json` `host` of a cell that runs, or to its `unsupported.json` `host` when it does not | | `egress` | no | framework | extra hosts (DNS names, optional `:port`; `*` stands for one whole label and the last two labels are literal, so `*.pythonhosted.org` passes and `*` or `*.org` does not) this task's agent and entry points reach through the proxy (§2), for example `["pypi.org", "files.pythonhosted.org"]` | -| `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. `result.json` `skills` records whether the agent used it and whether it used it first | +| `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. On a `guidance:as-skill` catalog arm the generated `-guidance` skill is refused too (its usage is `guidance_skill_loaded`). `result.json` `skills` records whether the agent used it and whether it used it first | | `useful` | no | framework | the definition of the run's first useful result, reported as `first_useful` in `result.json` (§4.1): `{"command": ""}` for a task whose grade runs the target tests, or `{"edit": ""}` (or a list of globs) for one that grades an edit to a target file. Omit it when no event is the useful result. A malformed value is refused by `repo validate`, `tasks check` and `repo run` | | anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` | @@ -324,7 +324,10 @@ belongs to the repository's grader and is passed through unread, except `grader` 3. `git init` and commit (no history, so fixes cannot leak) 4. apply `setup_patches` 5. apply the arm: `guidance:none`, `guidance:raw`, `guidance:bridge`, - `guidance:rev:` or `patch:` + `guidance:rev:`, `patch:`, `guidance:format=xml` (the guidance + files rendered as tagged sections, `dokimi/repos/guidance_format.py`) or + `guidance:as-skill` (the guidance files removed, their text packaged as one + skill, see [Guidance formats and packaging](benchmarking-and-comparison.md#guidance-formats-and-packaging)) 6. amend the commit 7. apply `review_patch` with `git am` under a neutral author 8. expire the reflog and prune unreachable objects: the @@ -417,7 +420,9 @@ answer was public; the author date alone cannot show it, as an amend or a rebase keeps it. For `patch:` an `arm_patch` object (the file's absolute `path` and full `sha256`, `added`: the files the patch added that neither the guidance globs nor `.mcp.json` match, and `mcp_servers`: the servers -the patch added to the `.mcp.json` at the root, read before the agent runs). `args_if_present` lists the profile's +the patch added to the `.mcp.json` at the root, read before the agent runs); for `guidance:format=` a +`guidance_links_kept` list of the guidance links that keep their raw bytes; for `guidance:as-skill` a +`guidance_skill` object (`name`, `path`, `sha256`) that is not in `skills`. `args_if_present` lists the profile's `args_if_present` files the cell's command line carries (§6): `[]` when the workspace has none of them, and always outside the `all` tool mode. @@ -534,6 +539,9 @@ the first one used; `null` when the task names none). When the trace could not be read (the harness never started), `measured` is `false` and `skill_invoked` and `skill_selected_correct` are `null`: the usage is unknown, not "did not use the skill". `repo report` leaves such cells out of its skill counts. +A `guidance:as-skill` cell has no `skills` entry for its generated skill: its +usage is `guidance_skill_loaded` (`true` when the agent called that skill or read +its `SKILL.md`, `false` when not, `null` when the trace could not be read). For a `patch:` arm with `added` files or `mcp_servers`, `result.json` has an `uptake` object: whether the agent used what the arm added. `files` and diff --git a/dokimi/dokimi/commands/repo_run.py b/dokimi/dokimi/commands/repo_run.py index 8c81f0859..1fc63c1fa 100644 --- a/dokimi/dokimi/commands/repo_run.py +++ b/dokimi/dokimi/commands/repo_run.py @@ -193,7 +193,7 @@ def cmd_repo_tasks_check(args: argparse.Namespace) -> None: ctx, _ = _context(args, out) tasks = load_tasks(ctx.content_dir, ctx.manifest, args.task) arms = [a for spec in args.arm or () for a in spec.split(",") if a] - check_arms([parse_arm(a) for a in arms], ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs) # skill paths exist, catalogs hold each expected_skill + check_arms([parse_arm(a) for a in arms], ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs, ctx.manifest.id) # skill paths exist, catalogs hold each expected_skill if arms: print(f"{len(arms)} arm(s) ok: {', '.join(arms)}", file=sys.stderr) print(f"{len(tasks)} task(s), image {ctx.image}{' (warm: with network)' if args.warm else ''}", @@ -282,7 +282,7 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock lock.enter_context(batch_lock(batch_dir)) # before _context, which rewrites _eval/content ctx, _ = _context(args, batch_dir, pin=True) tasks = load_tasks(ctx.content_dir, ctx.manifest, args.task) - check_arms(arms, ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs) # skill paths exist, catalogs hold each expected_skill + check_arms(arms, ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs, ctx.manifest.id) # skill paths exist, catalogs hold each expected_skill check_mcp_configs(ctx.git_dir, tasks, profile, args.tools) # a malformed .mcp.json fails here, not in every cell skill_pin = pin_skills(batch_dir, arms, ctx.content_dir) # a resume may not swap an arm's skills validate_tasks(tasks) # refuse malformed host requirements before containers start @@ -460,9 +460,9 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None: "result.json per cell, already finished cells skipped") _common(run_p, source_help, cache_help, positive_int) run_p.add_argument("--arm", action="append", default=None, required=True, - help="guidance:none | guidance:raw | guidance:bridge | guidance:rev:REV (the base with " - "the guidance files of revision REV of the source; the arm is named by REV's full SHA, " - "resolved once per batch) | patch:FILE (a unified diff on this host, applied with git " + help="guidance:none | guidance:raw | guidance:bridge | guidance:as-skill | " + "guidance:format=xml | guidance:rev:REV (the base with the guidance files of revision REV " + "of the source; the arm is named by REV's full SHA, resolved once per batch) | patch:FILE (a unified diff on this host, applied with git " "apply on top of guidance:raw; named patch:@, path and " "sha256 in meta.json) | skill: | skill-catalog: (skills installed in the " "workspace; a relative dir is in eval_dir; named by the directory, not the path). A " diff --git a/dokimi/dokimi/repos/arms.py b/dokimi/dokimi/repos/arms.py index ee96d526c..4405f5f3a 100644 --- a/dokimi/dokimi/repos/arms.py +++ b/dokimi/dokimi/repos/arms.py @@ -27,10 +27,13 @@ import yaml +from dokimi.skills.formats import SPEC_NAME_MAX + from dokimi import process SKILL_KINDS = ("skill", "skill-catalog") SKILLS_DIR = ".claude/skills" +GUIDANCE_SUFFIX = "-guidance" def slug(text: str) -> str: @@ -38,6 +41,13 @@ def slug(text: str) -> str: return re.sub(r"[^A-Za-z0-9._-]+", "_", text).strip("_") or "x" +def guidance_skill_name(repo_id: str) -> str: + """The name of the skill ``guidance:as-skill`` writes for a repository: the Agent Skills name rule (lowercase + letters, digits and single hyphens, at most ``SPEC_NAME_MAX``), so ``numpy.dev`` gives ``numpy-dev-guidance``.""" + words = re.sub(r"[^a-z0-9]+", "-", repo_id.lower()).strip("-")[:SPEC_NAME_MAX - len(GUIDANCE_SUFFIX)] + return f"{words.rstrip('-') or 'repo'}{GUIDANCE_SUFFIX}" + + class ArmError(ValueError): """An arm that does not parse, or whose skills cannot be found.""" @@ -181,13 +191,15 @@ def _base_skill_files(git_dir: Path, base: str) -> tuple[list[str], list[str]]: def check_arms(arms: list[Any], content_dir: Path, tasks: list[Any], git_dir: Path | None = None, - guidance_globs: tuple[str, ...] | list[str] = ()) -> None: + guidance_globs: tuple[str, ...] | list[str] = (), repo_id: str | None = None) -> None: """Find the skills of every parsed arm before anything runs; a catalog must hold each task's ``expected_skill``. Two arms that would share a name or a result directory are refused, and so is a skill the base of a task (in ``git_dir``) already ships, which preparing the workspace would refuse cell by cell. - As preparation does, a ``guidance:none`` or pinned-revision arm first drops the base's files that match - ``guidance_globs``: only what is left under the skill's directory counts as shipped. A revision arm then - writes back its revision's files that match them, so those count as shipped too. The files of a ``patch:`` + As preparation does, a ``guidance:none``, ``guidance:as-skill`` or pinned-revision arm first drops the base's + files that match ``guidance_globs``: only what is left under the skill's directory counts as shipped. A revision + arm then writes back its revision's files that match them, so those count as shipped too. A ``guidance:as-skill`` + arm writes its own skill, ``.claude/skills/-guidance``: a skill of the arm with that name, or a file the + base tracks under it, is refused here (``repo_id`` given). The files of a ``patch:`` arm's patch and of a task's ``setup_patches`` are not read: a skill directory they add is refused only in preparation.""" cells: dict[str, str] = {} # cell-name slug of an arm -> the arm spec that owns it @@ -197,13 +209,39 @@ def check_arms(arms: list[Any], content_dir: Path, tasks: list[Any], git_dir: Pa if cells.setdefault(cell, arm.spec) != arm.spec: raise ArmError(f"arms {cells[cell]!r} and {arm.spec!r} would share one result directory ({cell})") skills = resolve_skills(arm, content_dir) - drops = arm.patch is None and (arm.base == "guidance:none" or arm.rev is not None) + names = {s.name for s in skills} + generated = None # the skill preparation installs beside the catalog's; its usage is scored apart + if repo_id is not None and arm.patch is None and arm.base == "guidance:as-skill": + _check_generated_skill(arm, skills, repo_id, tasks, git_dir, shipped) + generated = guidance_skill_name(repo_id) + drops = arm.patch is None and (arm.base in ("guidance:none", "guidance:as-skill") or arm.rev is not None) rev = (arm.rev_sha or arm.rev) if arm.patch is None else None if skills and git_dir is not None: for task in tasks: _check_shipped(arm, skills, task, git_dir, shipped, (drops, rev, guidance_globs)) if any(kind == "skill-catalog" for kind, _ in arm.skills): - _check_expected_skills(arm, tasks, {s.name for s in skills}) + _check_expected_skills(arm, tasks, names, generated) + + +def _check_generated_skill(arm: Any, skills: list[Skill], repo_id: str, tasks: list[Any], git_dir: Path | None, + shipped: dict[str, tuple[list[str], list[str]]]) -> None: + """Refuse a ``guidance:as-skill`` arm whose generated skill directory is a skill of the arm, or is tracked by the + base of a task, or whose base links ``.claude`` or ``.claude/skills`` (preparation installs through both): it + would fail cell by cell, not here. A ``shipped`` cache entry holds the base's files and links.""" + target = f"{SKILLS_DIR}/{guidance_skill_name(repo_id)}" + for skill in skills: + if skill.target == target: + raise ArmError(f"arm {arm.spec!r}: its generated guidance skill {target} is also one of its skills") + if git_dir is None: + return + for task in tasks: + files, links = _cached_shipped(git_dir, task.base, shipped) + for path in files: + if path == target or path.startswith(target + "/"): + raise ArmError(f"arm {arm.spec!r}: the base of task {task.id} already has {target}") + for link in links: + raise ArmError(f"arm {arm.spec!r}: the base of task {task.id} tracks {link} as a symlink; " + f"refusing to install {target} through it") def _check_shipped(arm: Any, skills: list[Skill], task: Any, git_dir: Path, @@ -214,13 +252,12 @@ def _check_shipped(arm: Any, skills: list[Skill], task: Any, git_dir: Path, from dokimi.repos.manifest import match_glob drops, rev, guidance_globs = dropping - for commit in (task.base, rev): - if commit is not None and commit not in shipped: - shipped[commit] = _base_skill_files(git_dir, commit) - left = [p for p in shipped[task.base][0] - if not (drops and any(match_glob(g, p) for g in guidance_globs))] - written = [p for p in shipped[rev][0] if any(match_glob(g, p) for g in guidance_globs)] if rev else [] - for link in shipped[task.base][1]: # preparation refuses to install through a link, cell by cell + base_files, base_links = _cached_shipped(git_dir, task.base, shipped) + if rev is not None: + rev_files = _cached_shipped(git_dir, rev, shipped)[0] + left = [p for p in base_files if not (drops and any(match_glob(g, p) for g in guidance_globs))] + written = [p for p in rev_files if any(match_glob(g, p) for g in guidance_globs)] if rev else [] + for link in base_links: # preparation refuses to install through a link, cell by cell raise ArmError(f"arm {arm.spec!r}: the base of task {task.id} tracks {link} as a symlink; " f"refusing to install {skills[0].name} through it") for skill in skills: @@ -229,10 +266,23 @@ def _check_shipped(arm: Any, skills: list[Skill], task: Any, git_dir: Path, raise ArmError(f"arm {arm.spec!r}: {where} already has {skill.target}") -def _check_expected_skills(arm: Any, tasks: list[Any], names: set[str]) -> None: - """A catalog arm has to install the ``expected_skill`` of every task that names one.""" +def _cached_shipped(git_dir: Path, commit: str, + shipped: dict[str, tuple[list[str], list[str]]]) -> tuple[list[str], list[str]]: + """``_base_skill_files`` of a commit, one ``git ls-tree`` per commit for the whole check.""" + if commit not in shipped: + shipped[commit] = _base_skill_files(git_dir, commit) + return shipped[commit] + + +def _check_expected_skills(arm: Any, tasks: list[Any], names: set[str], generated: str | None = None) -> None: + """A catalog arm has to install the ``expected_skill`` of every task that names one. The generated guidance + skill of a ``guidance:as-skill`` arm (``generated``) is refused as one: its usage is recorded apart from the + catalog's (``guidance_skill_loaded``), so the catalog's usage would read it as never invoked.""" for task in tasks: expected = task.raw.get("expected_skill") + if expected and expected == generated: + raise ArmError(f"arm {arm.spec!r}: task {task.id}'s expected_skill {expected!r} is the generated guidance " + f"skill of the arm; name a skill of the catalog instead") if expected and expected not in names: raise ArmError(f"arm {arm.spec!r}: task {task.id}'s expected_skill {expected!r} is not installed " f"(installed: {', '.join(sorted(names))})") diff --git a/dokimi/dokimi/repos/guidance_format.py b/dokimi/dokimi/repos/guidance_format.py new file mode 100644 index 000000000..353705313 --- /dev/null +++ b/dokimi/dokimi/repos/guidance_format.py @@ -0,0 +1,241 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Guidance rendered in another format: the ``guidance:format=`` arm (contract §3, step 5). + +``xml`` wraps a guidance file in a ```` element and turns each ATX heading into a +``
`` element, closed before the next heading of its level or +above. No other line changes, and :func:`restore` undoes the wrapping exactly. :func:`to_format` +refuses a file whose rendering does not restore to it (a body line that looks like one of the +tags, a heading the restore could not rebuild), so the arm never measures altered content. +Headings inside code fences and HTML comment blocks are body text. A leading YAML front matter block +(``---`` to ``---``) stays verbatim above ````, where the harness that reads it still finds it. +""" + +from __future__ import annotations + +import re + +#: The formats of ``guidance:format=``. Markdown is ``guidance:raw``, not a format of its own. +FORMATS = ("xml",) + +_ROOT, _ROOT_CLOSE, _CLOSE = "", "", "
" +_HEADING = re.compile(r"^(#{1,6}) (\S[^\r]*)$") # canonical only: one space, no CR; anything else is body text +_FENCE = re.compile(r"^( *)(`{3,}|~{3,})(.*)$") # the indentation, a run of ``` or ~~~, and what follows it +_ITEM = re.compile(r" *(?:[-*+]|(\d{1,9})[.)])(?: {1,4}(?=\S)|(?=\s*$))") # a list marker (its number if ordered), then 1-4 spaces and content, or nothing (an empty item) +_RULE = re.compile(r" *([-*_])(?: *\1){2,} *$") # a thematic break: no list item, though ``- - -`` looks like two +_OPENER = re.compile(r" {0,3}(`{3,}|~{3,})(.*)$") # up to three spaces, then a run that opens a fence +_BLOCK = re.compile(r" *(?:#{1,6}(?: |$)|>|([-*_])( *\1){2,} *$)") # a heading, quote or rule: a line that starts a block +_QUOTE_MARK = re.compile(r" *> ?") # one ``>`` of a block quote, and the space after it +_OPEN = re.compile(r'^
$') +_COMMENT = re.compile(r" {0,3}" in line: + return None + return after[-1] if after else 0 + + +def _front_matter_end(lines: list[str]) -> int: + """The index of the line that closes a leading YAML front matter block (``---`` to ``---``), or -1 if none.""" + if lines and lines[0].rstrip("\r") == _FRONT: # a CRLF file's delimiter keeps its CR in the lines, not in the match + for index in range(1, len(lines)): + if lines[index].rstrip("\r") == _FRONT: + return index + return -1 + + +def _render_xml(text: str) -> str: + lines = text.split("\n") + end = _front_matter_end(lines) + out = lines[:end + 1] + [_ROOT] # the front matter stays verbatim above the root, where its reader finds it + fence: tuple[str, int] | None = None + items: tuple[int, ...] = () + para = empty = comment = False + comment_column = 0 # the content column of the item a comment block is in: a line indented less ends both + open_levels: list[int] = [] + for line in lines[end + 1:]: + if comment: # inside an HTML comment block: body text up to the line that closes it or leaves its item + expanded = line.expandtabs(4) + if line.strip() and len(expanded) - len(expanded.lstrip(" ")) < comment_column: + comment = False # the item ends with the block, so the line is read as any other + else: + comment = "-->" not in line + out.append(line) + continue + before, before_para = items, para + fence, items, para, empty = _advance(fence, items, line, para, empty) + heading = _HEADING.match(line) if fence is None else None # a line of a fence is body text + if heading is None: + out.append(line) + column = _comment_column(line, before, before_para, items) if fence is None else None + if column is not None: + comment, para, empty = True, False, False # a comment block is not paragraph text + comment_column = column # a lazy line is in no item + continue + level = len(heading.group(1)) + while open_levels and open_levels[-1] >= level: + out.append(_CLOSE) + open_levels.pop() + out.append(f'
') + open_levels.append(level) + out.extend([_CLOSE] * len(open_levels)) + out.append(_ROOT_CLOSE) + return "\n".join(out) + + +def _escape(title: str) -> str: + return re.sub(r'[&<>"]', lambda m: _ESCAPES[m.group()], title) + + +def _unescape(title: str) -> str: + return re.sub(r"&(?:amp|lt|gt|quot);", lambda m: _UNESCAPES[m.group()], title) diff --git a/dokimi/dokimi/repos/runner.py b/dokimi/dokimi/repos/runner.py index b44ea6578..bdc092bc3 100644 --- a/dokimi/dokimi/repos/runner.py +++ b/dokimi/dokimi/repos/runner.py @@ -176,6 +176,9 @@ def _add_arm_use(record: dict[str, Any], meta: dict[str, Any], task: Task, skill if meta.get("skills"): # without a trace the usage is unmeasured, not zero record["skills"] = skills if skills is not None else skill_unmeasured( [s["name"] for s in meta["skills"]], task.expected_skill) + if meta.get("guidance_skill"): # apart from the catalog's usage: it is not in the names the catalog's usage counts + record["guidance_skill_loaded"] = (None if events is None else + bool(skill_usage(events, [meta["guidance_skill"]["name"]], None)["skills_used"])) if any(uptake_of): # an arm that added something to use: did the agent use it (#41 D4) record["uptake"] = arm_uptake(events, *uptake_of, WORKSPACE) diff --git a/dokimi/dokimi/repos/workspace.py b/dokimi/dokimi/repos/workspace.py index 0918f3d80..e8183c36e 100644 --- a/dokimi/dokimi/repos/workspace.py +++ b/dokimi/dokimi/repos/workspace.py @@ -12,7 +12,8 @@ 3. ``git init`` and commit (no history, so an upstream fix cannot leak) 4. apply ``setup_patches`` 5. apply the arm (``guidance:none`` | ``guidance:raw`` | ``guidance:bridge`` | - ``guidance:rev:`` | ``patch:@``, see :class:`Arm`), then its + ``guidance:as-skill`` | ``guidance:format=`` | ``guidance:rev:`` | + ``patch:@``, see :class:`Arm`), then its skills into ``.claude/skills/`` (:mod:`.arms`) 6. amend the commit 7. apply ``review_patch`` with ``git am`` under a neutral author @@ -49,8 +50,9 @@ from dokimi import process -from .arms import ArmError, Skill, resolve_skills, tree_sha256 +from .arms import SKILLS_DIR, ArmError, Skill, guidance_skill_name, resolve_skills, tree_sha256 from .errors import PrepError +from .guidance_format import FORMATS, FormatError, to_format from .guidance_history import arm_rev_history from .harness_profile import HarnessProfile from .manifest import RepoManifest, _is_within, match_glob @@ -58,10 +60,14 @@ from .uptake import MCP_CONFIG, base_servers, mcp_servers from .useful_spec import Useful, parse_useful -ARMS = ("guidance:none", "guidance:raw", "guidance:bridge") # the guidance parts -REV_ARM, PATCH_ARM = "guidance:rev:", "patch:" +ARMS = ("guidance:none", "guidance:raw", "guidance:bridge", "guidance:as-skill") # the guidance parts +#: The description of the skill ``guidance:as-skill`` packages the guidance into: the same for every repository. +GUIDANCE_SKILL_DESCRIPTION = ("Repository guidance for working in this codebase, its conventions, build and test " + "commands and rules. Load it before changing its code.") # no ": " (YAML plain scalar) +REV_ARM, PATCH_ARM, FORMAT_ARM = "guidance:rev:", "patch:", "guidance:format=" #: Every form ``repo run --arm`` takes -ARM_FORMS = (*ARMS, f"{REV_ARM}", f"{PATCH_ARM}", "skill:", "skill-catalog:") +ARM_FORMS = (*ARMS, f"{REV_ARM}", f"{PATCH_ARM}", f"{FORMAT_ARM}<{'|'.join(FORMATS)}>", + "skill:", "skill-catalog:") SKILL_JOIN = "+" _SKILL_PREFIXES = ("skill:", "skill-catalog:") SNAPSHOT_AUTHOR = ("dokimi", "dokimi@example.invalid") @@ -141,13 +147,19 @@ def parse_arm(spec: str) -> Arm: def _starts_part(text: str) -> bool: - return text in ARMS or text.startswith((REV_ARM, PATCH_ARM, *_SKILL_PREFIXES)) + return text in ARMS or text.startswith((REV_ARM, PATCH_ARM, FORMAT_ARM, *_SKILL_PREFIXES)) def _parse_guidance(spec: str) -> Arm: - """A guidance part: ``guidance:none|raw|bridge``, ``guidance:rev:`` or ``patch:``.""" + """A guidance part: ``guidance:none|raw|bridge|as-skill``, ``guidance:format=``, + ``guidance:rev:`` or ``patch:``.""" if spec in ARMS: return Arm(spec) + if spec.startswith(FORMAT_ARM): + fmt = spec[len(FORMAT_ARM):] + if fmt not in FORMATS: # markdown is guidance:raw, a second name for it would be a duplicate arm + raise PrepError(f"arm {spec!r}: {fmt!r} is not a guidance format (known: {', '.join(FORMATS)})") + return Arm(spec) if spec.startswith(REV_ARM): rev = spec[len(REV_ARM):] if not _ARM_REV_RE.match(rev) or ".." in rev: @@ -392,7 +404,7 @@ def prepare_workspace(git_dir: Path, manifest: RepoManifest, task: Task, arm: st arm_files, arm_meta = _apply_arm(ws, env, git_dir, manifest, arm, bridge) # 5 except PrepError as exc: raise PrepError(f"task {task.id}: {exc}") from exc - installed = _install_skills(ws, task, skills) + installed = _install_skills(ws, task, skills) # the catalog's skills; a guidance:as-skill skill is in arm_meta _git(ws, env, "add", "-A") # 6 _git(ws, env, "commit", "-q", "--no-gpg-sign", "--no-verify", "--amend", "--allow-empty", "-m", "snapshot") _apply_review_patch(ws, task, patches) # 7 @@ -462,6 +474,10 @@ def _apply_arm(ws: Path, env: dict[str, str], git_dir: Path, manifest: RepoManif return _apply_patch(ws, env, manifest, arm) tracked = [p for p in _git(ws, env, "ls-files", "-z").split("\0") if p] matched = sorted(p for p in tracked if any(match_glob(g, p) for g in manifest.guidance_globs)) + if arm.base == "guidance:as-skill": + return _apply_as_skill(ws, manifest, matched) + if arm.base.startswith(FORMAT_ARM): + return _apply_format(ws, matched, arm.base[len(FORMAT_ARM):]) if arm.base == "guidance:none" or arm.rev_sha is not None: for rel in matched: (ws / rel).unlink() @@ -472,6 +488,78 @@ def _apply_arm(ws: Path, env: dict[str, str], git_dir: Path, manifest: RepoManif return _apply_bridge(ws, matched, bridge) +def _apply_format(ws: Path, matched: list[str], fmt: str) -> tuple[list[str], dict[str, Any]]: + """Each guidance file rewritten in ``fmt`` in place. A symlink is left alone: its target is rendered + once if it is a guidance file itself, and ``meta.json`` names the links that keep their raw bytes.""" + if not matched: + raise PrepError(f"guidance:format={fmt}: the base has no guidance file to render") + written, links = [], [] + for rel in matched: + if (ws / rel).is_symlink(): + _link_file(ws, rel, matched) # the link must reach a file that is rendered, not raw guidance + links.append(rel) + continue + try: + rendered = to_format(_guidance_text(ws, rel, f"guidance:format={fmt}"), fmt) + except FormatError as exc: + raise PrepError(f"guidance:format={fmt}: {rel}: {exc}") from exc + (ws / rel).write_bytes(rendered.encode("utf-8")) # no newline translation: the restore check holds on disk + written.append(rel) + return written, ({"guidance_links_kept": links} if links else {}) + + +def _link_file(ws: Path, rel: str, matched: list[str]) -> None: + """The regular file a guidance link of the workspace reaches, resolved as the filesystem resolves it (a linked + directory on the way counts). PrepError when the link dangles or cycles, or reaches a file outside the workspace + or outside the matched guidance files: that file would be dropped or left unrendered.""" + try: + target = (ws / rel).resolve(strict=True) + except (OSError, RuntimeError) as exc: # RuntimeError: a symlink loop, before Python 3.13 + raise PrepError(f"guidance link {rel} does not reach a file (a dangling link or a cycle): {exc}") from exc + root = ws.resolve() + if not target.is_relative_to(root) or not target.is_file(): + raise PrepError(f"guidance link {rel} reaches {target}, which is outside the workspace or not a regular file") + reached = target.relative_to(root).as_posix() + if reached not in matched: + raise PrepError(f"guidance link {rel} reaches {reached}, which is not a guidance file") + + +def _guidance_text(ws: Path, rel: str, arm: str) -> str: + """The text of a guidance file of the workspace; one that is not UTF-8 is a prep error of the cell.""" + try: + return (ws / rel).read_bytes().decode("utf-8") + except UnicodeDecodeError as exc: + raise PrepError(f"{arm}: {rel} is not UTF-8: {exc}") from exc + + +def _apply_as_skill(ws: Path, manifest: RepoManifest, matched: list[str]) -> tuple[list[str], dict[str, Any]]: + """The guidance files removed (as ``guidance:none`` removes them) and their text packaged as one skill, + ``.claude/skills/-guidance/SKILL.md``: each file under a ``## File: `` line, its text verbatim.""" + if not matched: + raise PrepError("guidance:as-skill: the base has no guidance file to package") + name = guidance_skill_name(manifest.id) + skill_dir = ws / SKILLS_DIR / name + for part in (ws / ".claude", ws / SKILLS_DIR): # a repository link would write to the host + if part.is_symlink(): + raise PrepError(f"guidance:as-skill: {part.relative_to(ws)} is a symlink in the repository") + if skill_dir.exists() or skill_dir.is_symlink(): + raise PrepError(f"guidance:as-skill: the workspace already has {SKILLS_DIR}/{name}") + parts = [] + for rel in matched: + if (ws / rel).is_symlink(): # packaged once under its target's name; a link to anything else would be lost + _link_file(ws, rel, matched) + continue + text = _guidance_text(ws, rel, "guidance:as-skill") + parts.append(f"## File: {rel}\n\n" + (text if text.endswith("\n") else text + "\n")) + for rel in matched: + (ws / rel).unlink() + skill_dir.mkdir(parents=True) + frontmatter = f"---\nname: {name}\ndescription: {GUIDANCE_SKILL_DESCRIPTION}\n---\n\n" + (skill_dir / "SKILL.md").write_bytes((frontmatter + "\n".join(parts)).encode("utf-8")) # verbatim: no newline translation + return matched, {"guidance_skill": {"name": name, "path": f"{SKILLS_DIR}/{name}", + "sha256": tree_sha256(skill_dir)}} + + def _apply_rev(ws: Path, git_dir: Path, manifest: RepoManifest, arm: Arm, removed: list[str]) -> tuple[list[str], dict[str, Any]]: """The guidance files of ``arm.rev_sha`` into ``ws``, whose own were ``removed``.""" diff --git a/dokimi/tests/test_repo_guidance_format.py b/dokimi/tests/test_repo_guidance_format.py new file mode 100644 index 000000000..c9f31afd6 --- /dev/null +++ b/dokimi/tests/test_repo_guidance_format.py @@ -0,0 +1,496 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The ``guidance:format=`` and ``guidance:as-skill`` arms (contract §3, step 5). No docker, no LLM.""" + +from __future__ import annotations + +import re +import types + +import pytest +import yaml + +from dokimi.repos import RepoSource, load_manifest, open_source +from dokimi.repos.arms import guidance_skill_name, tree_sha256 +from dokimi.repos.guidance_format import FormatError, restore, to_format +from dokimi.repos.workspace import ( + GUIDANCE_SKILL_DESCRIPTION, + PrepError, + _apply_as_skill, + _apply_format, + export_eval_dir, + load_tasks, + parse_arm, + prepare_workspace, + source_git_dir, +) +from repo_eval_fixture import make_repo, write + +RICH = """Intro line before any heading. + +# Project +Use `make check`. + +## Build & test +Run `make ` with "quotes". + +```bash +# a shell comment, not a heading +make check +``` + +### Deep detail +Text. + +## Style +- keep it short + +# Other +tail""" + + +@pytest.mark.parametrize("text, sections", [ + ("- ```\n echo hi\n ```\n# Real\n", ["Real"]), # the closer is indented to the item's content column + ("- ```\n echo hi\n# Real\n", ["Real"]), # a line less indented than the item ends the item and its fence + ("- a\n - ```\n # not\n ```\n# Real\n", ["Real"]), # a nested item + ("1. ```\n # not\n ```\n# Real\n", ["Real"]), # an ordered item + ("- ```\n# Real\n```\n# Real2\n", ["Real"]), # the item ends; the ``` at column 0 opens a fence over Real2 + ("- use ``` for code\n# Real\n", ["Real"]), # a list item that is prose, not a fence + ("- - ```\n # not\n ```\n# Real\n", ["Real"]), # two markers on one line + ("- ```\n# Real\n", ["Real"]), # five spaces after the marker: indented code, not a fence + ("- ```\n\t# not\n\t```\n# Real\n", ["Real"]), # a tab is four columns of indentation: the item's content + ("- Example:\n ```\n code\n# Real\n", ["Real"]), # a fence on a continuation line takes the item's column + ("1. Step\n\n ```sh\n x\n# Real\n", ["Real"]), # the same after a blank line, ordered item + ("- Example:\n ```\n code\n ```\n# Real\n", ["Real"]), # closed inside the item: the heading is a section + ("- Example:\n\n```\n# not\n```\n# Real\n", ["Real"]), # a column-0 fence after a blank line ends the item + ("- a\n - b\n ```\n code\n# Real\n", ["Real"]), # a fence on the outer item's line after a nested item + ("-\t```\n echo hi\n ```\n# Real\n", ["Real"]), # a tab after a bullet is four columns: the item's fence + ("1.\t```\n echo hi\n ```\n# Real\n", ["Real"]), # the same after an ordered marker + ("-\t```\n echo hi\n ```\n# Real\n", []), # 2 spaces is under the item's column 4: the item ends, as in CommonMark + ("- a\n\t```\n```\n# Real\n", []), # a tab-indented continuation fence (a pin: the sections match the spaces form) + ("- Example:\ncontinued text\n ```\n code\n# Real\n", ["Real"]), # a lazy paragraph line keeps the item open + ("- a\nlazy\n```\n# not\n```\n# Real\n", ["Real"]), # a fence at column 0 is a block start, not a lazy line + ("- a\nlazy\n# Real\n", ["Real"]), # a heading is a block start too + ("- a\n> quote\n ```\n# Real\n", []), # a block quote is a block start: the item ends, the fence opens at top level + ("- a\n---\n ```\n# x\n", []), # a rule is a block start, not a lazy line + ("- a\n# H\n ```\n# x\n", ["H"]), # a heading ends the item too: only H is a section, the fence holds x + ("- a\n\nlazy\n ```\n# x\n", []), # a blank line ends the item: the next line is not a lazy continuation + ("-\n ```\n# Real\n", ["Real"]), # an empty bullet is an item: its fence is at the content column 2 + ("1.\n ```\n# Real\n", ["Real"]), # an empty ordered item: its content column is 3 + ("-\n ```\n```\n# Real\n", []), # the column-0 fence ends the empty item; the hash line is code + ("-\n ```\n# H\n```\n# Real\n", ["H"]), # H ends the item and its fence; the fence at column 0 holds Real + ("-\n\n ```\n# Real\n", []), # an empty item ends at a blank line: the indented fence is top-level + ("- - -\n ```\n# not a heading\n ```\n", []), # a thematic break is no list item + ("* * *\n ```\n# not a heading\n ```\n", []), # the same with asterisks + ("paragraph\n2. ```\n ```\n# Real\n", []), # a numbered marker other than 1 cannot interrupt a paragraph + ("- a\n2. x\n ```\n# Real\n", []), # 2. x starts a new list (the item is not continued), so the fence is top-level + ("- a\n\n code\noutside\n ```\n# not a heading\n", []), # indented code is not paragraph text + ("- > quoted\nlazy\n ```\n# Real\n", ["Real"]), # a lazy line continues the quoted paragraph in the item + ("> quoted\nlazy\n ```\n# Real\n", []), # the same at top level: the fence opens and holds Real + ("- > # H\nlazy\n ```\n# x\n", []), # a quoted heading is no paragraph: the lazy line ends the item + ("- > code\nlazy\n ```\n# Real\n", []), # indented code in the quote is no paragraph: lazy ends the item + ("para\n- > code\nlazy\n ```\n# Real\n", []), # a new item after a paragraph: the same, no lazy continuation + ("- > a\n > more\nlazy\n ```\n# Real\n", ["Real"]), # an open quoted paragraph takes an indented line too +]) +def test_a_fence_opened_by_a_list_item_holds_body_text_only_inside_the_item(text, sections): + assert re.findall(r'title="([^"]*)"', to_format(text, "xml")) == sections + + +@pytest.mark.parametrize("text", [ + RICH, + "", + "no heading at all", + "# only\n", + "# a\n### skipped level\n# b\n", + "## start deeper\n# then shallower\n", + "# no trailing newline", + "#no-space is body\n## two spaces is body\n####### seven is body\n", + "# trailing spaces \r\nbody\r\n", + "# Title with
and \"quotes\" & \nbody\n", + "~~~\n# fenced with tildes\n~~~\n# real\n", + "---\npaths:\n - src/**\n# a yaml comment\n---\n# Rules\n", + "---\n---\n# Rules\n", + "---\nunclosed\n# Rules\n", + "\n# Real\n", +]) +def test_xml_round_trips_the_markdown_exactly(text): + assert restore(to_format(text, "xml")) == text + + +def test_xml_wraps_each_heading_in_a_section_closed_before_its_peer(): + assert to_format("# A\nbody\n## B\nmore\n# C\n", "xml") == ( + "\n" + '
\nbody\n' + '
\nmore\n
\n' + "
\n" + '
\n\n' + "
\n" + "" + ) + + +@pytest.mark.parametrize("text, sections", [ + ("\n# Real\n", ["Real"]), # a comment block spans lines: its heading is body text + ("\n# Real\n", ["Real"]), # closed on its own line: nothing after it is in the comment + (" \n# Real\n", ["Real"]), # up to three spaces of indentation + ("\n", ["Real"]), # a comment inside a paragraph line is no block: the heading is a section + ("- item\n \n", ["Real"]), # the item ends at the heading, and its comment block with it + ("- item\n\n", []), # a lazy comment line starts a block outside the item: the comment holds on + ("- a\n\n \n# Real\n", ["Real"]), # the heading inside the block is body text, the one after it is not + ("1.\n1.\n1.\n ```\n# H\n", []), # an ordered list's fence is body text + ("===\n1.\n