Skip to content

Add rocprofiler-pc-sampling skill - #272

Merged
danielholanda merged 3 commits into
mainfrom
pc-sampling
Oct 19, 2026
Merged

danielholanda merged 3 commits into
mainfrom
pc-sampling

Conversation

@samcrater-amd

Copy link
Copy Markdown
Collaborator

federates a skill from rocm-systems' rocprofiler-sdk project that tells agents how to operate rocprofv3's pc sampling functionality and how to give the user a useful analysis of the results

@danielholanda danielholanda added run_behavioral Run behavioral tests on PR enable_mi_ci labels Oct 16, 2026
@danielholanda danielholanda changed the title add rocprofiler-pc-sampling skill Add rocprofiler-pc-sampling skill Oct 16, 2026
@danielholanda

Copy link
Copy Markdown
Collaborator

Status

Samuel is iterating over the tests to make sure they all pass on our MI300 runner.

@samcrater-amd
samcrater-amd deployed to behavioral-instinct October 16, 2026 22:27 — with GitHub Actions Active
@johnl-amd

Copy link
Copy Markdown
Collaborator

Couple of things to sort before this is good to go.

1. The evals fixes only exist here, not upstream

The catalog re-vendors the whole skill folder from rocm-systems every night, so anything edited on this side gets replaced. Two things in this PR currently exist only here:

  • evals/resources/demo_program.hip.cpp and cuda_demo.cu, added in dd59fd5
  • the prompts in evals.json, which were edited here to point at those files

Upstream evals/ is just evals.json, and its prompts name no file:

upstream:  "use pc sampling to profile my HIP application."
this PR:   "use pc sampling to profile my HIP application. my application is
            demo_program.hip.cpp in the skill's evals/resources/ folder"

So after the first nightly import the demo programs are gone and the prompts revert to naming no application, which leaves behavioral with nothing to profile. The green run on this PR is grading content that won't survive the merge. docs/evals.md calls this out for the dataset specifically: an edit made in the catalog is overwritten by the next import.

evals/machine.yml is fine to leave here, that one is the documented exception in CONTRIBUTING.md. It's only the fixtures and the dataset that need to move.

Fix is to land the two demo files and the updated prompts in projects/rocprofiler-sdk/skills/pc-sampling/evals/ in rocm-systems, then re-run uv run .github/scripts/federate_skills.py --only rocprofiler-pc-sampling and commit what it writes.

Worth checking one thing first: rocprofiler-sdk-formatting.yml matches projects/rocprofiler-sdk/** and only excludes doc extensions, so a .hip.cpp and a .cu dropped in there will hit clang-format, and CodeQL sweeps the tree too. If that's why they ended up over here, adding an exclude for the eval fixtures is probably cleaner than keeping them split.

2. Nothing in rocm-systems runs this skill's checks

I looked at what ran on ROCm/rocm-systems#10915, the PR that added the skill upstream. 65 checks, all rocprofiler-sdk build, test, CodeQL and formatting. The Python ones are flake8 and CodeQL, which read the files but never execute them, so tests/test_analyze_pc_sampling.py has never actually run. No structural checks either.

The template is already in that repo though. hrr-skills.yml does exactly this for projects/hrr/skills:

  • unit-tests: compileall, then discovers every test_*.py and runs one pytest per skill, failing if it finds none
  • script-syntax: bash -n over the shell scripts
  • structural: real skillscope, pinned, --repo . structural

Structural needs no API key and no GPU, so that leg should be straightforward. Swapping the paths filter and the find roots from projects/hrr/skills to projects/rocprofiler-sdk/skills covers pc-sampling plus thread-trace, rocprofv3 and rocprofv3-doctor sitting next to it, which I'm guessing are queued behind this one.

Everything else looks good: branch rather than a fork, branch: develop is correct since there's no main, vendored copy is in the PR, dataset minimums are met, skill-card is complete, links resolve.

@danielholanda

Copy link
Copy Markdown
Collaborator

Thanks @johnl-amd, good catches. @samcrater-amd promised to address both by end of week, so I think we're good to merge this one.

Tracking here so they don't get lost:

@danielholanda
danielholanda merged commit 99297ba into main Oct 19, 2026
12 checks passed

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enable_mi_ci run_behavioral Run behavioral tests on PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants