feat(subjects): distribution_digest, the digest a well-known skill index carries by napetrov · Pull Request #343 · intel/dokimi

napetrov · 2026-11-01T07:19:27Z

Summary

subject_scorecard.v2 gains subject.distribution_digest (and per member): sha256: of the SKILL.md bytes, the value a .well-known/agent-skills/index.json entry of type skill-md carries (agentskills#254). A file ref or a directory holding only SKILL.md gets the digest; a skill with resources (archive bytes depend on packing) or a non-local ref gets null. ref_digest is unchanged.

Needed so a published evidence document can be matched to the skill a client installed (see the stacked exporter PR).

Evidence

tests/test_subjects_digest.py; full suite 12002 passed locally.

Merge danger

Low: additive nullable schema field.

🤖 Generated with Claude Code

exai-bot · 2026-11-01T07:23:20Z

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

Review details

Commit reviewed: 3f98c22 against merge base b4a0a8c
Coverage: 4 files in 1 groups
Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
Usage: 91k tokens (42k from cache), 15s
Tool calls: 7 (7 file_read)
Scorecard distribution digest implementation, schema, tests, and changelog: 4 files

exai-bot · 2026-11-01T07:23:21Z

+    assert scorecard["subject"]["distribution_digest"] == "sha256:" + hashlib.sha256(b"body").hexdigest()
+    validate_artifact("subject_scorecard", scorecard)


Nit: This descriptor has no members, so deleting the new per-member distribution_digest field from build_subject_scorecard would leave all distribution-digest tests passing. Schema validation would not catch it either: members has no item constraints. Add a bundle scorecard test with a standalone SKILL.md member and a resource-bearing or non-local member, asserting their digest and null values respectively.

Fixed in a5c5500. The member test now uses standalone/ and bundled/ (ref.md) plus a context7 ref, and expects [sha256 of member body, None, None], so each member is checked on its own.
Full suite at a5c5500: 12025 passed.

Copilot

🟡 Changes recommended

File references omit the distribution digest for accompanying resources, allowing evaluation evidence to be associated with a standalone skill installation that lacks those resources.

1 open issue

Include directory refs when computing skill distribution digests · New

What changed in this PR

1 new finding.

This PR adds a distribution digest to subject scorecards so published evaluation evidence can be matched to an installed skill.

Changes:

Adds nullable distribution_digest values for subjects and bundle members without changing ref_digest.
Hashes eligible local SKILL.md bytes and extends the scorecard schema.
Adds digest tests and a changelog entry.

File	Description
`dokimi/tests/test_subjects_digest.py`	Tests digest eligibility, hashing, and scorecard integration.
`dokimi/dokimi/subjects/scorecard.py`	Computes distribution digests for subjects and members.
`dokimi/dokimi/schemas/subject_scorecard.v2.json`	Allows the nullable subject digest field.
`dokimi/changelog.d/20261031_120000_napetrov_distribution-digest.md`	Documents the new field and intended matching behavior.

_{Review effort: Balanced}

napetrov · 2026-11-01T18:03:37Z

+    if path.is_dir():
+        # Any other entry makes it an archive; one stray file is enough, so stop at the second.
+        if [p.name for _, p in zip(range(2), path.iterdir())] != ["SKILL.md"]:
+            return None
+        path = path / "SKILL.md"
+    if path.name != "SKILL.md" or not path.is_file():
+        return None


Fixed in a5c5500. A SKILL.md file ref now digests only when its directory holds nothing else; a sibling resource gives null. Directory refs unchanged.
Red then green test added; mutation check caught the old behaviour.

…dex carries subject_scorecard.v2 gains subject.distribution_digest: sha256 of the SKILL.md bytes for a file ref or a directory holding only SKILL.md, the value a .well-known/agent-skills index.json entry of type skill-md carries (agentskills PR intel#254). An archive's bytes depend on how it was packed, so a skill with resources gets null, as does a non-local ref. ref_digest is unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

exai-bot · 2026-11-01T07:43:48Z

+    """
+    if not ref:
+        return None
+    path = Path(ref.removeprefix("local:"))


Must-fix: Non-local references are interpreted as local paths rather than explicitly rejected. On POSIX, create base/context7:/org/lib/SKILL.md as the directory's only entry, then call distribution_digest("context7:/org/lib", base=base): this returns a SHA-256 digest instead of None. Descriptor validation does not reject this reference, so both subject and member scorecards can incorrectly identify remote content using unrelated local bytes. Reject non-local reference schemes before filesystem lookup (while preserving Windows drive paths), and add a collision regression test; the current remote-reference assertion only covers a nonexistent path.

Answered, no new code. The scheme guard from de58607 stays in a5c5500, and the member test checks a context7: ref returns None.
Full suite at a5c5500: 12025 passed.

codecov-commenter · 2026-11-01T07:52:44Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.17%. Comparing base (b4a0a8c) to head (3f98c22).
⚠️ Report is 6 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #343      +/-   ##
==========================================
+ Coverage   94.05%   94.17%   +0.12%     
==========================================
  Files         297      304       +7     
  Lines       38018    38678     +660     
  Branches     6424     6553     +129     
==========================================
+ Hits        35756    36426     +670     
+ Misses       1452     1442      -10     
  Partials      810      810

Files with missing lines	Coverage Δ
dokimi/dokimi/subjects/scorecard.py	`91.86% <100.00%> (+5.09%)`	⬆️

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

antonwolfy

A file ref gets a digest the skill's own directory form refuses, and that is worth fixing before merge. Two nits inline.

What I ran, on a Linux review host at 099c2b8:

make ci-local: 30 passed, 0 failed, 7 skipped; changed-line coverage passes
reproductions for the code findings, with the LLM transport mocked
mutation runs on tests/test_subjects_digest.py: deleting the per-member line, and cutting the entry check to one entry, both still pass the file.

Checked, no issues

Schema: the new property is optional and nullable on an open object, so schema_compat passes and artifacts written earlier still validate; no version bump is needed.
Changelog: the fragment follows the layout of the existing ones and its claims match the code.
Commit: it carries a Signed-off-by.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy

I pushed a commit per nit, up to 1dbde6a, which fix the two nits from my review:

de58607: dokimi/dokimi/subjects/scorecard.py:156: distribution_digest returns None for a ref with a scheme other than local:, and a test creates the colliding directory.
1dbde6a: dokimi/tests/test_subjects_digest.py:62: a bundle scorecard test asserts a digest for a standalone SKILL.md member and None for a non-local member.

make ci-local at 1dbde6a: 30 passed, 0 failed, 7 skipped.

Not approving yet. Still open: the risk from my review.

A SKILL.md named by its file ref was hashed even when its directory held resources, while the directory ref got None. The index serves such a skill as an archive under either spelling, so both now get None. The member test covers a standalone member, a bundled one with resources, and a non-local ref. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

exai-bot · 2026-11-01T18:06:43Z

+    colliding = tmp_path / "context7:" / "org" / "lib"
+    colliding.mkdir(parents=True)


Must-fix: this fixture creates a directory named context7:, which is invalid on Windows. colliding.mkdir(parents=True) therefore raises before the scheme guard is tested, causing the test suite to fail on Windows even though the implementation preserves drive-letter paths. Mark this POSIX-only collision test with a Windows skip, or mock the filesystem lookup to test rejection without creating an invalid filename.

Fixed in 08cd011: the colon-named collision test is skipped on win32, since a colon is not valid in a Windows directory name. The rest of the file still runs on every platform. Local: test_subjects_digest.py 4 passed; full suite 12025 passed, 8 skipped.

A colon is not allowed in a Windows directory name, so the fixture raised before the scheme guard ran and the Windows smoke path failed. The collision case stays on POSIX; the rest of the file runs everywhere. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

antonwolfy

LGTM. A SKILL.md file ref with a sibling resource now gets null, like its directory ref, for the subject and for bundle members. The scheme guard stays, and the member test checks each member on its own. One nit inline.

What I ran, on a Linux review host at 08cd011:

make ci-local: 29 passed, 1 failed, 7 skipped. The failure is test_the_regex_limit_is_shared_by_every_pattern_of_one_check, a wall-clock comparison in tests/test_reference_checks.py, a file this PR does not touch. It passes when the file runs alone (70 passed). Changed-line coverage passes.
mutation runs on distribution_digest and its two call sites: 20 mutants. Dropping either call site, passing the subject's ref for a member, dropping the scheme guard, removeprefix("local:"), the base / join and the path.parent step each fail a test. The surviving mutant that changes a digest value is in the inline comment.
probes of odd refs (trailing slash, .., local:/abs, drive-letter paths, symlinked SKILL.md, a directory named SKILL.md, empty file): they agree with the docstring.

Checked, no issues

Schema: the new property is optional and nullable on an open object, so no version bump is needed.
Commits: every non-merge commit carries a Signed-off-by.
Changelog: it matches the code, including the file-ref wording.

Bot points I checked that no longer apply at this head (3)

Copilot's point on the file ref getting a digest despite sibling resources does not apply: a5c5500 returns None for it, for the subject and for members, and the test pins both.
exai-bot's point on non-local refs reading a colliding local directory does not apply: distribution_digest returns None for a scheme: ref other than local:, and a test creates the colliding directory.
exai-bot's point on the colon-named fixture failing on Windows does not apply: 08cd011 skips that test on win32.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy

LGTM. I pushed a commit per nit, up to 3f98c22, which fix the nit from my review:

3f98c22: dokimi/tests/test_subjects_digest.py:21: the fixture SKILL.md and the expected digest now use CRLF bytes, so a text-decoding read changes the digest and fails the test.

make ci-local at 3f98c22: 30 passed, 0 failed, 7 skipped.

Copilot AI balanced review requested due to automatic review settings November 1, 2026 07:19

napetrov requested a review from a team as a code owner November 1, 2026 07:19

This was referenced Nov 1, 2026

feat(subjects): subjects evidence renders a scorecard as skill_evidence.v1 #344

Merged

CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94

Open

Copilot started reviewing on behalf of napetrov November 1, 2026 07:19

View session

exai-bot reviewed Nov 1, 2026

View reviewed changes

Copilot AI reviewed Nov 1, 2026

View reviewed changes

napetrov force-pushed the feat/distribution-digest branch from a2bd867 to 099c2b8 Compare November 1, 2026 07:39

exai-bot reviewed Nov 1, 2026

View reviewed changes

antonwolfy reviewed Nov 1, 2026

View reviewed changes

Comment thread dokimi/dokimi/subjects/scorecard.py Resolved

Comment thread dokimi/dokimi/subjects/scorecard.py Resolved

Comment thread dokimi/dokimi/subjects/scorecard.py Resolved

antonwolfy added 2 commits November 1, 2026 14:22

Give a ref with a non-local scheme no distribution digest

de58607

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Test the distribution digest of each bundle member

1dbde6a

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy reviewed Nov 1, 2026

View reviewed changes

exai-bot reviewed Nov 1, 2026

View reviewed changes

antonwolfy reviewed Nov 1, 2026

View reviewed changes

Comment thread dokimi/tests/test_subjects_digest.py Resolved

Use CRLF bytes in the digest fixture to pin raw-byte hashing

3f98c22

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy approved these changes Nov 1, 2026

View reviewed changes

antonwolfy merged commit 575e59e into intel:main Nov 1, 2026
17 checks passed

		assert scorecard["subject"]["distribution_digest"] == "sha256:" + hashlib.sha256(b"body").hexdigest()
		validate_artifact("subject_scorecard", scorecard)

		colliding = tmp_path / "context7:" / "org" / "lib"
		colliding.mkdir(parents=True)

Conversation

napetrov commented Nov 1, 2026

Summary

Evidence

Merge danger

Uh oh!

exai-bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

exai-bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

napetrov Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

exai-bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

exai-bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

exai-bot commented Nov 1, 2026 •

edited

Loading

codecov-commenter commented Nov 1, 2026 •

edited

Loading