Repository navigation
test(catalog): skill-repo shapes through catalog scan and routing - #327
Conversation
Thirteen invented repo layouts, one per shape seen in public skill catalogs (mirrors, same-name skills, disable-model-invocation, template and vendored directories, zero skills, plugins nesting, spec errors, escaping links, no headings). Each runs catalog scan and novelty routing --catalog with a stub selector: no crash, and the expected status. Bugs still open on main are strict xfails naming the fix branch that will flip them. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #327 +/- ##
==========================================
+ Coverage 94.05% 94.06% +0.01%
==========================================
Files 297 297
Lines 38018 38018
Branches 6424 6424
==========================================
+ Hits 35756 35762 +6
+ Misses 1453 1449 -4
+ Partials 809 807 -2 🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The thirteen shapes are well chosen, the stub selector really replaces the model client, and all six strict xfails fail on the bug they name. Three nits inline.
What I ran, on a Linux review host at 793bc74:
make ci-local: 30 passed, 0 failed, 7 skipped; 12000 passed, 59 skipped, 6 xfailed; changed-line coverage step passed.- The six xfails re-run with
--runxfail: each fails on the stated dimension (routing exit 2 "different content; dedupe or rename", zero-skill scan exit 0, over-long / mismatched / title-case skill kept in the field). - Mutations of the code under test: dropping routing dedupe,
duplicate_of, disabled-skill exclusion, or the no-queries refusal each fails a test. Flipping the mirror sort key inexternal/validate.pypasses the whole file (see inline).
Checked, no issues
- Stub selector:
novelty_cmd._clientis looked up as a module global, so the monkeypatch takes effect; no real client is built. - Shapes:
template_diris excluded by directory name, the.venvhalf ofvendored_diris a real guard, and strict scan exits 1 for the three spec-error shapes. - Rules: SPDX headers, writes only under
tmp_path, growth ratchet, DCO sign-off on the one commit.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
There was a problem hiding this comment.
🟡 Changes recommended
A critical test assertion fails on Windows because serialized paths use backslashes while the expected suffix uses forward slashes, even when mirror detection is correct.
1 open issue
What changed in this PR
1 new finding.
This PR adds regression coverage for catalog scanning and novelty routing across synthetic skill-repository layouts, without changing package behavior.
Changes:
- Defines 13 repository shapes covering mirrors, invalid metadata, excluded directories, and edge cases.
- Exercises scan modes and lexical/stub-model routing, with explicit exit-code and competitor expectations.
- Tracks six known failing shapes with strict expected failures.
| File | Description |
|---|---|
dokimi/tests/test_skill_repo_shapes.py |
Adds command sweeps, expected outcomes, and focused regression assertions. |
dokimi/tests/skill_repo_shapes.py |
Defines synthetic repository layouts and shared probe fixtures. |
Review effort: Balanced
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. I pushed a commit per nit, up to f71a36c, which fix the three nits from my review:
e667713:dokimi/changelog.d/20261031_120000_napetrov_skill-repo-shapes.md: deleted the changelog fragment.30906cf:dokimi/tests/test_skill_repo_shapes.py:191: after the count, assert the duplicate row's path is theplugins/kit/skills/widget-polishcopy and itsduplicate_ofisskills/widget-polish.07c1b3c:dokimi/tests/test_skill_repo_shapes.py:171: the strict xfail marks now carryraises=AssertionError.
The review bot then flagged lines of those commits; fixed as well:
f71a36c:dokimi/tests/test_skill_repo_shapes.py:193: the mirror path is now compared as aPathagainst the expected path underroot, not as a forward-slash suffix.
make ci-local at f71a36c: 30 passed, 0 failed, 7 skipped.

Summary
One invented repo per shape, each run through
catalog scanandnovelty routing --catalog:Thirteen shapes in
tests/skill_repo_shapes.py: mirrors (identical and drifted), same name in two directories,disable-model-invocation,template/, vendored directories, a zero-skill repo,plugins/*/skillsnesting, an over-long description, a name that is not the directory name, a title-case name, links that leave the skill directory, and a skill with no headings.tests/test_skill_repo_shapes.pychecks two things. A sweep runs both commands, lexical and with a stub selector, over every shape and asserts that nothing raises. The expectation table asserts the exit codes and the routing field each shape should get.Evidence
Shapes that fail on
intel/mainare strict xfails. Each fails on the bug itself, checked with--runxfail:drifted_mirror,same_name_distinct_dirs: routing exits 2 withcatalog has widget-polish at … and … with different content; dedupe or rename. Markedfixed by fix/rescan-catalog.zero_skill:catalog scanexits 0 with no skills and no note. Markedfixed by fix/rescan-catalog.description_too_long,name_not_dir,title_case_name: the skill that fails the spec check still appears in the routing field. Markedfixed by fix/rescan-routing.Not xfailed: the all-vs-all disabled-skill case.
intel/mainalready names--include-disabledin that refusal (tests/test_novelty_routing_all_vs_all.pycovers the message), so it is a plain regression test here.Local checks on
793bc74f:ruff check .: passedmypy dokimi:Success: no issues found in 303 source filespytest -n auto -q:12032 passed, 27 skipped, 6 xfailed in 66.20spr-reviewlocal, two passes on793bc74f): noblocking:finding. The three plain-pass rows are the intended xfails; the duplicate-keying difference inreview_skillsvs routing is pre-existing and left for a follow-up.Merge Danger
Door: two-way. Tests and a changelog fragment only; a revert removes them and nothing else.
Blast Radius: none. No package code, data or CLI behaviour changes. When the
fix/rescan-catalogandfix/rescan-routingfixes land, their six strict xfails must be removed in the same PR. Otherwise they fail as XPASS.Checklist
pytestpassesruff,mypy, and schema checks pass (ruff and mypy run; schema check not run, no schema changed)AGENTS.md.github/.licenserc.yaml)🤖 Generated with Claude Code