Skip to content

ci(security): verify scan results and audit release dependencies - #618

Draft
yeongseon wants to merge 13 commits into
mainfrom
ci/security-gates-evidence
Draft

yeongseon wants to merge 13 commits into
mainfrom
ci/security-gates-evidence

Conversation

@yeongseon

@yeongseon yeongseon commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

Make security scan failures visible and block publishing when the candidate wheel has unresolved runtime dependency advisories.

Changes

  • Verify Bandit's source-file inventory and require valid scan results. Missing files, unapproved suppressions, tool crashes or incomplete reports cannot pass.
  • Maintain reviewed, expiring exceptions for existing low findings without changing application source code.
  • Audit both the base and otel dependency sets of the exact candidate wheel on Python 3.11–3.14 before publishing.
  • Keep development dependency scans reporting-only; preserve Semgrep pins, CodeQL, SBOM, OIDC publishing and current release dependencies.
  • Save redacted security reports with CI artifacts and job summaries; avoid copying sensitive contents.

Validation

  • New security tests: 79 passed. Full Python 3.11 suite passed at 96.26% coverage; Ruff, strict mypy, workflow checks and package build passed.
  • Bandit evidence: 38 files, 5,058 lines, 29 reviewed findings and no blockers.
  • Isolated wheel audits: 4 base and 6 otel packages, no advisories.
  • Not run: GitHub Actions execution of the new release job on the full matrix or real Azure. Other Azure configuration observations remain follow-up review findings, not established vulnerabilities.

Related Issues

Refs #615

This PR contains the security/release checks; the optional secure logging implementation is #617.

yeongseon and others added 8 commits October 23, 2026 01:39
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…udits

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…daries

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@yeongseon

Copy link
Copy Markdown
Owner Author

Merge-order note. PR #616 adds tools/verification*.py. This PR's Bandit inventory scans tools/, so after both land the security job would report those files' findings (B404, B405, B314, B603 and similar) as blocking unless they have exact reviewed policy entries. I confirmed this on the openapi pair by merging the two branches locally: 23 blocking findings, status FAIL. Merge #616 first, then rebase this PR and add the exact-location exceptions (reason, owner, expiry) for those files; pyproject.toml (typecheck line) and .github/workflows/security.yml will also need a trivial manual merge. Nothing else overlaps.

@yeongseon

yeongseon commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner Author

Applied sibling findings C1–C3. C1 a53aad3 verifies wheel METADATA against installed identity, freezes every third-party package excluding only the verified candidate, audits --requirement/--no-deps/--disable-pip, and compares returned inventory. Offline clean/advisory/skip/mismatch/exclusion regressions pass. Real unpublished logging_audit_probe_973e009226004d79a9ff83e35c9128c9==99.0.0 wheel installed into venv --without-pip via pip --python; audit reached its only dependency idna==3.11 and correctly FAILed on genuine PYSEC-2026-215 (not candidate skip). Real logging wheel base and otel audits PASS (3 and 5 third parties). C2 f872571 records action-only Semgrep success as INCOMPLETE; Make tolerates exactly exit 3 while pinned action stays blocking; collectors/uploads always run. C3 02f0e1b introduces separate Makefile security entry points and routes workflows through them; cac7cbd adds strict inventory typing, 02ad6b6 updates docs. Full suite coverage 96.26%, Ruff, strict project/security mypy, workflow lints, expanded Bandit, sdist/wheel build passed. The Bandit subprocess exception was re-reviewed at its moved exact line, not broadened. PR #616 edits maintenance/check recipes: preserve both blocks when reconciling; do not replace the Makefile wholesale.

@yeongseon

Copy link
Copy Markdown
Owner Author

Post-push CI snapshot: Analyze/CodeQL, Bandit, Semgrep, PR title, changes, branch naming, stale checks, Python 3.12 tests and both real-host smokes PASS. Quality, Python 3.11/3.13/3.14 tests, minimum dependencies, compatibility build and docs were still pending at the final observation; no failure observed. Not claiming full CI completion. Local verification and throwaway-wheel proof are recorded above.

@yeongseon
yeongseon marked this pull request as draft October 24, 2026 02:13
@yeongseon
yeongseon requested a balanced review from Copilot October 24, 2026 10:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two security regression tests pass without exercising the suppression enforcement and audit execution-error handling they are intended to protect.

2 open issues
What changed in this PR

This PR implements the security-automation portion of #615, complementing the runtime redaction work in #617. It strengthens scan evidence and adds a dependency audit gate before package publication.

Changes:

  • Expand Bandit coverage with exact inventory checks and reviewed, expiring exceptions.
  • Gate releases on candidate-wheel dependency audits across Python 3.11–3.14 and base/OTel installations.
  • Add redacted evidence, regression tests, and security-policy documentation.
File Description
tools/​security_report.py Define evidence records, statuses, and report output.
tools/​security_exceptions.py Validate owned, time-limited exceptions.
tools/​security_evidence.py Enforce Bandit inventory and record Semgrep outcomes.
tools/​dependency_audit_policy.py Classify dependency audits and enforce release policy.
tools/​candidate_inventory.py Verify candidate identity and installed dependencies.
tools/​__init__.py Make security tooling importable as a package.
tests/​test_security_workflows.py Check workflow gates, permissions, and evidence uploads.
tests/​test_security_tool_execution.py Cover scanner execution and inventory handling.
tests/​test_security_report.py Test evidence validation and summary output.
tests/​test_security_evidence.py Test static scan classification and exceptions.
tests/​test_dependency_audit_policy.py Test advisory blocking and exception rules.
tests/​test_candidate_audit.py Test candidate-specific audit inventory.
security/​dependency-audit-policy.json Establish the runtime advisory blocking policy.
security/​bandit-policy.json Record reviewed static findings and suppressions.
SECURITY.md Document gates, exceptions, evidence, and production boundaries.
pyproject.toml Add strict security-tool type checking.
Makefile Expose shared security commands.
examples/​README.md Clarify example safety and deployment limitations.
AGENTS.md Link contributor guidance to security controls.
.gitignore Exclude generated security evidence.
.github/​workflows/​security.yml Add evidence collection, uploads, and type checking.
.github/​workflows/​publish-pypi.yml Require the runtime dependency audit matrix before publishing.
.github/​workflows/​dependency-audit.yml Report isolated runtime, OTel, and development audits.

Review effort: Balanced

Comment on lines +43 to +44
(tmp_path / "src").mkdir()
(tmp_path / "src/x.py").write_text(f"value = 1 {marker}\n")

monkeypatch.setattr(subprocess, "run", fail)
assert security_evidence.run_bandit().status == Status.TOOL_ERROR
assert dependency_audit_policy.run_audit(tmp_path, tmp_path).status == Status.INCOMPLETE
@yeongseon yeongseon changed the title ci(security): enforce fail-closed scan evidence and a runtime dependency release gate ci(security): verify scan results and audit release dependencies Oct 26, 2026
@yeongseon

Copy link
Copy Markdown
Owner Author

Do not merge this older draft. Current main already contains focused replacements #629 (319c551: fail empty/missing/unparsable Bandit scope and scan shipped Python) and #630 (d3a254f: candidate-wheel base/otel runtime dependency advisory gate with expiring exceptions). Both verified by git log and git branch --contains on origin/main. These supersede those portions, not every inventory/evidence requirement in this draft; remaining gaps stay under #615. Deployment log exposure has a separate small fix #637 / #625. Leaving draft and branch intact; closing is for the maintainer.

@yeongseon

Copy link
Copy Markdown
Owner Author

Superseded draft; not merging wholesale. Focused replacements #629 and #630 merged the Bandit scope/report gate and candidate-wheel dependency audit. Secure runtime redaction is being completed in #617 under the existing eight-keyword setup ceiling. #613 SDK wheel/host compatibility remains a separate unmet item; this draft does not certify it. Leaving draft closure to the maintainer.

@yeongseon

Copy link
Copy Markdown
Owner Author

Final supersession update: #629/#630 delivered the focused scan/advisory gates, #617 delivered the eight-keyword-compliant secure preset, and #645 completed safe scanner evidence with all final-head required checks passing. #615 is now closed through merged #645. No remaining #615 implementation belongs in this draft. Do not merge this superseded draft; closure remains the maintainer’s decision. Live deployment-log proof is separately open in #625.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants