Skip to content

ci: add coverage report to tests workflow - #119

Open
adityamulik wants to merge 3 commits into
GenAI-Security-Project:mainfrom
adityamulik:feat/add-test-coverage
Open

adityamulik wants to merge 3 commits into
GenAI-Security-Project:mainfrom
adityamulik:feat/add-test-coverage

Conversation

@adityamulik

@adityamulik adityamulik commented Oct 26, 2026 •

Copy link
Copy Markdown

Add test coverage reporting and a CI coverage gate

Sets up coverage reporting for the test suite and surfaces it in CI:

  • Coverage in CI — the Tests workflow now runs pytest with term/xml/html coverage and publishes a coverage table to the Actions job summary; reports are uploaded as an artifact.
  • PR comments — adds py-cov-action with a companion workflow_run workflow so a coverage comment is posted/updated on PRs, including fork PRs.
  • Coverage gate — fails the build when total coverage drops below 60% (currently ~61%).
  • Enables relative_files so coverage paths resolve across the two workflows.

Aditya Mulik and others added 3 commits October 26, 2026 16:42
Run pytest with term-missing/xml/html coverage, publish a markdown
coverage table to the Actions job summary, and upload coverage.xml
and htmlcov/ as an artifact. Adds pip caching to speed up installs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add py-cov-action to the Tests workflow and a companion
"Post coverage comment" workflow triggered via workflow_run, so
coverage is posted as a PR comment even for fork-originated PRs
(whose pull_request runs get a read-only token). Set
coverage relative_files = true so paths resolve across the two
workflows' checkouts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add --cov-fail-under=60 to the pytest run so the Tests check fails
on regressions below the threshold. Current coverage is ~61%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@adityamulik

Copy link
Copy Markdown
Author

++ @e2hln / @eaglei15 to review

@drk-ram drk-ram left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all three changed files at f42b64887e10a3a0047e440d6bcdb48b4ccdcbb8. Approving the coverage integration.

The 60% gate is enforced by pytest's exit status, independently of whether a comment can be posted. XML/HTML artifacts and the summary run even after a test failure. The fork path leaves test execution in pull_request and uses a separate workflow_run reporter without checking out or executing the contributor's source. relative_files = true is consistent with coverage data consumed across workflow contexts.

Evidence:

  • The exact-head Tests run passed all 182 tests and reported 61% coverage; its log also confirms creation of the deferred-comment artifact.
  • Locally, python -m pytest tests -o addopts= -q --tb=short -k "not test_generate_uses_aiofiles_async and not TestFixtureEndToEnd" --cov=src --cov-report=term --cov-report=xml --cov-report=html --cov-fail-under=60 passed: 178 tests, 4 deselected, 61.20% coverage, with both report formats generated.
  • python -m coverage report --format=markdown passed.

Local scope: three tensor-fixture tests failed in an earlier broader run because this environment lacks torch; those three and one pre-existing test that hardcodes a POSIX output path were excluded from the passing subset. The complete-suite result above comes from the published exact-head run, not that subset. End-to-end posting by the new privileged workflow remains unverified until the workflow is present on the default branch.

Non-blocking hardening: pin the new third-party action to an audited commit rather than a moving v3 tag. Also consider narrowing the reporter's contents grant: editing a PR comment needs pull-requests: write, not repository-content write. Keep any content-write permission required for coverage-data branch publication confined to the path that actually performs that operation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants