feat(repo): guidance:format=xml and guidance:as-skill arms

dokimi/changelog.d/20261031_120000_napetrov_guidance-format-arms.md

@@ -0,0 +1,3 @@
+### Added
+
+- **`repo run` arms `guidance:format=xml` and `guidance:as-skill`** — the guidance files of the base, rendered as XML tags (each Markdown heading becomes a `<section>` element, checked to restore to the original file exactly), or moved into one skill directory (`.claude/skills/<repo>-guidance/SKILL.md`, the repo id made into an Agent Skills name). Both compare the same guidance content in another shape under the same model; `guidance:format=md` is refused as a duplicate of `guidance:raw`, and POML is not offered yet.

dokimi/docs/benchmarking-and-comparison.md

@@ -301,6 +301,43 @@ both per row. Every cell also records `cache_read_tokens` and
 `cache_write_tokens` with the input and output tokens, and
 `repo report --json` gives their means.
 
+### Guidance formats and packaging
+
+Two guidance arms answer "does the same content work better in another shape?"
+The model stays the same, so run them per model (one `--model` each) and compare
+the cells the way the skill arms are compared above.
+
+- `guidance:format=xml`: every guidance file of the base, rendered as XML tags.
+  Each Markdown heading becomes `<section level="N" title="...">`, closed before
+  the next heading of its level or above; `<guidance>` wraps the file. No other
+  line changes. The rendering is checked before it is written: it must restore to
+  the original file exactly, so a file with a line that looks like one of these
+  tags (or a heading the restore cannot rebuild) fails the cell's preparation
+  rather than measuring altered content. A heading inside a code fence is body
+  text. A heading line that is not `#`…`######`, one space, then the title, with no
+  carriage return, is body text too: a file with CRLF line endings has no sections
+  at all, only the wrapper. `guidance:format=md` is `guidance:raw`
+  and is refused as a second name for it. POML is not offered yet: its documented
+  components (`role`, `task`, `example`, ...) have no heading tag, so a POML arm
+  needs a decision about which element a heading maps to.
+- `guidance:as-skill`: the guidance files are removed (as `guidance:none` removes
+  them) and their text goes into `.claude/skills/<repo id>-guidance/SKILL.md`, each
+  file under a `## File: <path>` line with its text verbatim; a symlink to another
+  guidance file (`CLAUDE.md` to `AGENTS.md`) is packaged once under that file, and a
+  symlink to any other file fails the cell. The frontmatter
+  description is one fixed sentence for every repository, so the skill is
+  selected on its description alone. Nested guidance files lose the directory
+  they applied to; the skill's name, path and sha256 are recorded in
+  `meta.json` under `guidance_skill`. Its usage is recorded separately in
+  `result.json` as `guidance_skill_loaded` (`null` if the trace is unreadable),
+  and is not currently aggregated by `repo report`.
+
+Reading the two together: `guidance:format=xml` against `guidance:raw` isolates
+the markup, and `guidance:as-skill` against `guidance:raw` isolates the packaging
+(always in context, or loaded on demand), with the fixed description a known
+confound of the second. The arms do not rewrite the content, so a difference
+comes from the format or the packaging, not from edited guidance.
+
 ### Reading the significance tables
 
 Arms are compared **paired per question** against the one shared `baseline`

dokimi/docs/preparing-a-repository.md

@@ -6,7 +6,8 @@ SPDX-License-Identifier: Apache-2.0
 # Preparing a repository for coding agents
 
 > **Status: recommendations.** §1–§7 apply today. The measurement loop in §8 runs
-> today for guidance arms (`guidance:none`, `guidance:raw`, `guidance:bridge`).
+> today for guidance arms (`guidance:none`, `guidance:raw`, `guidance:bridge`,
+> `guidance:as-skill`, `guidance:format=xml`).
 > `guidance:bridge` measures the `CLAUDE.md` import (§7). The `skill:` and
 > `skill-catalog:` arms, which install skills, run today too
 > ([skill arms](benchmarking-and-comparison.md#skill-arms-in-repository-mode)). Arms

dokimi/docs/repo-evaluation-contract.md

@@ -321,7 +321,10 @@ belongs to the repository's grader and is passed through unread, except `grader`
 3. `git init` and commit (no history, so fixes cannot leak)
 4. apply `setup_patches`
 5. apply the arm: `guidance:none`, `guidance:raw`, `guidance:bridge`,
-   `guidance:rev:<rev>` or `patch:<file>`
+   `guidance:rev:<rev>`, `patch:<file>`, `guidance:format=xml` (the guidance
+   files rendered as tagged sections, `dokimi/repos/guidance_format.py`) or
+   `guidance:as-skill` (the guidance files removed, their text packaged as one
+   skill, see [Guidance formats and packaging](benchmarking-and-comparison.md#guidance-formats-and-packaging))
 6. amend the commit
 7. apply `review_patch` with `git am` under a neutral author
 8. expire the reflog and prune unreachable objects: the
@@ -394,7 +397,9 @@ grader diffs the agent's changes from `start_sha`. With the arm come
 files with no counterpart at `<rev>`); for `patch:` an `arm_patch` object (the
 file's absolute `path` and full `sha256`, `added`: the files the patch added
 that neither the guidance globs nor `.mcp.json` match, and `mcp_servers`: the servers
-the patch added to the `.mcp.json` at the root, read before the agent runs). `args_if_present` lists the profile's
+the patch added to the `.mcp.json` at the root, read before the agent runs); for `guidance:format=` a
+`guidance_links_kept` list of the guidance links that keep their raw bytes; for `guidance:as-skill` a
+`guidance_skill` object (`name`, `path`, `sha256`) that is not in `skills`. `args_if_present` lists the profile's
 `args_if_present` files the cell's command line carries (§6): `[]` when the
 workspace has none of them, and always outside the `all` tool mode.
 
@@ -481,6 +486,9 @@ the first one used; `null` when the task names none). When the trace could not
 be read (the harness never started), `measured` is `false` and `skill_invoked`
 and `skill_selected_correct` are `null`: the usage is unknown, not "did not use
 the skill". `repo report` leaves such cells out of its skill counts.
+A `guidance:as-skill` cell has no `skills` object for its generated skill: its
+usage is `guidance_skill_loaded` (`true` when the agent called that skill or read
+its `SKILL.md`, `false` when not, `null` when the trace could not be read).
 
 For a `patch:` arm with `added` files or `mcp_servers`, `result.json` has an
 `uptake` object: whether the agent used what the arm added. `files` and

dokimi/dokimi/commands/repo_run.py

@@ -193,7 +193,7 @@ def cmd_repo_tasks_check(args: argparse.Namespace) -> None:
         ctx, _ = _context(args, out)
         tasks = load_tasks(ctx.content_dir, ctx.manifest, args.task)
         arms = [a for spec in args.arm or () for a in spec.split(",") if a]
-        check_arms([parse_arm(a) for a in arms], ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs)  # skill paths exist, catalogs hold each expected_skill
+        check_arms([parse_arm(a) for a in arms], ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs, ctx.manifest.id)  # skill paths exist, catalogs hold each expected_skill
         if arms:
             print(f"{len(arms)} arm(s) ok: {', '.join(arms)}", file=sys.stderr)
         print(f"{len(tasks)} task(s), image {ctx.image}{' (warm: with network)' if args.warm else ''}",
@@ -282,7 +282,7 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
         lock.enter_context(batch_lock(batch_dir))  # before _context, which rewrites _eval/content
         ctx, _ = _context(args, batch_dir, pin=True)
         tasks = load_tasks(ctx.content_dir, ctx.manifest, args.task)
-        check_arms(arms, ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs)  # skill paths exist, catalogs hold each expected_skill
+        check_arms(arms, ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs, ctx.manifest.id)  # skill paths exist, catalogs hold each expected_skill
         check_mcp_configs(ctx.git_dir, tasks, profile, args.tools)  # a malformed .mcp.json fails here, not in every cell
         skill_pin = pin_skills(batch_dir, arms, ctx.content_dir)  # a resume may not swap an arm's skills
         validate_tasks(tasks)  # refuse malformed host requirements before containers start
@@ -460,9 +460,9 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None:
         "result.json per cell, already finished cells skipped")
     _common(run_p, source_help, cache_help, positive_int)
     run_p.add_argument("--arm", action="append", default=None, required=True,
-                       help="guidance:none | guidance:raw | guidance:bridge | guidance:rev:REV (the base with "
-                       "the guidance files of revision REV of the source; the arm is named by REV's full SHA, "
-                       "resolved once per batch) | patch:FILE (a unified diff on this host, applied with git "
+                       help="guidance:none | guidance:raw | guidance:bridge | guidance:as-skill | "
+                       "guidance:format=xml | guidance:rev:REV (the base with the guidance files of revision REV "
+                       "of the source; the arm is named by REV's full SHA, resolved once per batch) | patch:FILE (a unified diff on this host, applied with git "
                        "apply on top of guidance:raw; named patch:<file name>@<sha256 prefix>, path and "
                        "sha256 in meta.json) | skill:<dir> | skill-catalog:<dir> (skills installed in the "
                        "workspace; a relative dir is in eval_dir; named by the directory, not the path). A "

dokimi/dokimi/repos/arms.py

@@ -27,17 +27,27 @@
 
 import yaml
 
+from dokimi.skills.formats import SPEC_NAME_MAX
+
 from dokimi import process
 
 SKILL_KINDS = ("skill", "skill-catalog")
 SKILLS_DIR = ".claude/skills"
+GUIDANCE_SUFFIX = "-guidance"
 
 
 def slug(text: str) -> str:
     """``text`` as a file-system-safe name: the result directory of a cell, the cell name."""
     return re.sub(r"[^A-Za-z0-9._-]+", "_", text).strip("_") or "x"
 
 
+def guidance_skill_name(repo_id: str) -> str:
+    """The name of the skill ``guidance:as-skill`` writes for a repository: the Agent Skills name rule (lowercase
+    letters, digits and single hyphens, at most ``SPEC_NAME_MAX``), so ``numpy.dev`` gives ``numpy-dev-guidance``."""
+    words = re.sub(r"[^a-z0-9]+", "-", repo_id.lower()).strip("-")[:SPEC_NAME_MAX - len(GUIDANCE_SUFFIX)]
+    return f"{words.rstrip('-') or 'repo'}{GUIDANCE_SUFFIX}"
+
+
 class ArmError(ValueError):
     """An arm that does not parse, or whose skills cannot be found."""
 
@@ -181,13 +191,15 @@ def _base_skill_files(git_dir: Path, base: str) -> tuple[list[str], list[str]]:
 
 
 def check_arms(arms: list[Any], content_dir: Path, tasks: list[Any], git_dir: Path | None = None,
-               guidance_globs: tuple[str, ...] | list[str] = ()) -> None:
+               guidance_globs: tuple[str, ...] | list[str] = (), repo_id: str | None = None) -> None:
     """Find the skills of every parsed arm before anything runs; a catalog must hold each task's
     ``expected_skill``. Two arms that would share a name or a result directory are refused, and so is a skill
     the base of a task (in ``git_dir``) already ships, which preparing the workspace would refuse cell by cell.
-    As preparation does, a ``guidance:none`` or pinned-revision arm first drops the base's files that match
-    ``guidance_globs``: only what is left under the skill's directory counts as shipped. A revision arm then
-    writes back its revision's files that match them, so those count as shipped too. The files of a ``patch:``
+    As preparation does, a ``guidance:none``, ``guidance:as-skill`` or pinned-revision arm first drops the base's
+    files that match ``guidance_globs``: only what is left under the skill's directory counts as shipped. A revision
+    arm then writes back its revision's files that match them, so those count as shipped too. A ``guidance:as-skill``
+    arm writes its own skill, ``.claude/skills/<repo_id>-guidance``: a skill of the arm with that name, or a file the
+    base tracks under it, is refused here (``repo_id`` given). The files of a ``patch:``
     arm's patch and of a task's ``setup_patches`` are not read: a skill directory they add is refused only in
     preparation."""
     cells: dict[str, str] = {}  # cell-name slug of an arm -> the arm spec that owns it
@@ -197,13 +209,38 @@ def check_arms(arms: list[Any], content_dir: Path, tasks: list[Any], git_dir: Pa
         if cells.setdefault(cell, arm.spec) != arm.spec:
             raise ArmError(f"arms {cells[cell]!r} and {arm.spec!r} would share one result directory ({cell})")
         skills = resolve_skills(arm, content_dir)
-        drops = arm.patch is None and (arm.base == "guidance:none" or arm.rev is not None)
+        names = {s.name for s in skills}
+        if repo_id is not None and arm.patch is None and arm.base == "guidance:as-skill":
+            _check_generated_skill(arm, skills, repo_id, tasks, git_dir, shipped)
+            names.add(guidance_skill_name(repo_id))  # the skill preparation installs beside the catalog's
+        drops = arm.patch is None and (arm.base in ("guidance:none", "guidance:as-skill") or arm.rev is not None)
         rev = (arm.rev_sha or arm.rev) if arm.patch is None else None
         if skills and git_dir is not None:
             for task in tasks:
                 _check_shipped(arm, skills, task, git_dir, shipped, (drops, rev, guidance_globs))
         if any(kind == "skill-catalog" for kind, _ in arm.skills):
-            _check_expected_skills(arm, tasks, {s.name for s in skills})
+            _check_expected_skills(arm, tasks, names)
+
+
+def _check_generated_skill(arm: Any, skills: list[Skill], repo_id: str, tasks: list[Any], git_dir: Path | None,
+                           shipped: dict[str, tuple[list[str], list[str]]]) -> None:
+    """Refuse a ``guidance:as-skill`` arm whose generated skill directory is a skill of the arm, or is tracked by the
+    base of a task, or whose base links ``.claude`` or ``.claude/skills`` (preparation installs through both): it
+    would fail cell by cell, not here. A ``shipped`` cache entry holds the base's files and links."""
+    target = f"{SKILLS_DIR}/{guidance_skill_name(repo_id)}"
+    for skill in skills:
+        if skill.target == target:
+            raise ArmError(f"arm {arm.spec!r}: its generated guidance skill {target} is also one of its skills")
+    if git_dir is None:
+        return
+    for task in tasks:
+        files, links = _cached_shipped(git_dir, task.base, shipped)
+        for path in files:
+            if path == target or path.startswith(target + "/"):
+                raise ArmError(f"arm {arm.spec!r}: the base of task {task.id} already has {target}")
+        for link in links:
+            raise ArmError(f"arm {arm.spec!r}: the base of task {task.id} tracks {link} as a symlink; "
+                           f"refusing to install {target} through it")
 
 
 def _check_shipped(arm: Any, skills: list[Skill], task: Any, git_dir: Path,
@@ -214,13 +251,12 @@ def _check_shipped(arm: Any, skills: list[Skill], task: Any, git_dir: Path,
     from dokimi.repos.manifest import match_glob
 
     drops, rev, guidance_globs = dropping
-    for commit in (task.base, rev):
-        if commit is not None and commit not in shipped:
-            shipped[commit] = _base_skill_files(git_dir, commit)
-    left = [p for p in shipped[task.base][0]
-            if not (drops and any(match_glob(g, p) for g in guidance_globs))]
-    written = [p for p in shipped[rev][0] if any(match_glob(g, p) for g in guidance_globs)] if rev else []
-    for link in shipped[task.base][1]:  # preparation refuses to install through a link, cell by cell
+    base_files, base_links = _cached_shipped(git_dir, task.base, shipped)
+    if rev is not None:
+        rev_files = _cached_shipped(git_dir, rev, shipped)[0]
+    left = [p for p in base_files if not (drops and any(match_glob(g, p) for g in guidance_globs))]
+    written = [p for p in rev_files if any(match_glob(g, p) for g in guidance_globs)] if rev else []
+    for link in base_links:  # preparation refuses to install through a link, cell by cell
         raise ArmError(f"arm {arm.spec!r}: the base of task {task.id} tracks {link} as a symlink; "
                        f"refusing to install {skills[0].name} through it")
     for skill in skills:
@@ -229,6 +265,14 @@ def _check_shipped(arm: Any, skills: list[Skill], task: Any, git_dir: Path,
                 raise ArmError(f"arm {arm.spec!r}: {where} already has {skill.target}")
 
 
+def _cached_shipped(git_dir: Path, commit: str,
+                    shipped: dict[str, tuple[list[str], list[str]]]) -> tuple[list[str], list[str]]:
+    """``_base_skill_files`` of a commit, one ``git ls-tree`` per commit for the whole check."""
+    if commit not in shipped:
+        shipped[commit] = _base_skill_files(git_dir, commit)
+    return shipped[commit]
+
+
 def _check_expected_skills(arm: Any, tasks: list[Any], names: set[str]) -> None:
     """A catalog arm has to install the ``expected_skill`` of every task that names one."""
     for task in tasks: