Skip to content

ci(security): enforce fail-closed scan evidence and a runtime dependency release gate - #817

Merged
yeongseon merged 17 commits into
mainfrom
ci/security-gates-evidence
Oct 24, 2026
Merged

yeongseon merged 17 commits into
mainfrom
ci/security-gates-evidence

Conversation

@yeongseon

@yeongseon yeongseon commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

Second half of #812 (security automation and release evidence). Scan results can no longer pass vacuously, runtime-dependency advisories block a release, and the pinned Swagger UI CDN assets are verified against their SRI hashes. The runtime key-forwarding half is in #823 (merged).

Changes

  • Bandit scope and fail-closed evidence. tools/security_evidence.py scans src, examples, demo, scripts and tools with an explicit inventory. Zero files scanned, tool errors, a crash or missing JSON, and unlisted suppressions each fail with a status distinct from findings (PASS|FAIL|TOOL_ERROR|INCOMPLETE). Thirteen reviewed LOW findings are recorded as exact id/file/line exceptions with owner and expiry (2027-01-19); inline #nosec is not allowed.
  • Semgrep. The existing pinned action is unchanged; an evidence step records its outcome (if: always()).
  • Runtime dependency policy (blocking). security/dependency-audit-policy.json (all known runtime advisories block, unknown severity blocks, exceptions need a reason, owner and at most 90 days, expired exceptions fail, stale ones are reported, no exceptions seeded). New runtime-dependency-audit job in publish-pypi.yml audits the exact candidate wheel from the build artifact in an isolated venv on Python 3.11 to 3.14, and publish now needs it. TOOL_ERROR and INCOMPLETE also block; workflow_dispatch on the tag is the recovery path. dependency-audit.yml reuses the same helper and stays reporting-only for runtime-plus-development findings.
  • Browser CDN/SRI evidence. Offline tests validate the pinned SRI format, version embedding and rendered integrity/crossorigin. tools/browser_integrity.py downloads the pinned assets and compares sha384 in the release gate (mismatch is FAIL, network failure is TOOL_ERROR).
  • Evidence. Redacted security-results/** JSON plus a Job Summary, uploaded with if: always(); only IDs, paths, lines and severities are recorded.
  • Docs. SECURITY.md documents what blocks merge, release and what only reports, the exception process, and local-only versus production defaults; example/demo READMEs label anonymous/local-only configs. AGENTS.md gets a one-line pointer.
  • Unchanged: PyPI OIDC trusted publishing, exact-wheel reuse, azure-e2e chain, existing job names, all action SHA pins, permissions: contents: read on every new job.

Type of Change

  • Chore (maintenance, dependencies, CI, etc.)
  • Documentation update

Checklist

  • My code follows the project's code style
  • I have run make check-all and all checks pass (including tests and at least 95% coverage)
  • I have added tests for new functionality (if applicable)
  • I have updated documentation (if applicable)
  • My changes do not introduce new warnings

Verification (Python 3.11): full suite passes at 95.11% coverage; 94 new security tests (97.28% helper coverage); ruff clean; strict mypy clean on the 5 new tool modules; the three workflow lints exit 0. Real runs: Bandit evidence over 48 files / 8,759 LOC (13 reviewed LOW, zero errors); pip-audit==2.10.1 against a venv containing only the built wheel: 10 packages, zero advisories; live CDN check: both assets match the pinned SRI.

Not verified: behaviour inside GitHub Actions (the new release job has only run locally) and anything on real Azure. The Bandit/mypy tooling lines may need a trivial reconciliation with #813 (pyproject.toml typecheck line, tools/__init__.py).

Deliberately unchanged (code/config review findings, not demonstrated vulnerabilities): infra/main.bicep still uses shared-key storage/deployment authentication, public networking, and lacks explicit Function App/SCM minimum-TLS settings. Changing these needs real-Azure validation of the e2e certification flow, so they are left for a follow-up.

Related Issues

Closes #812

#812 acceptance mapping

  • Protected-docs alternatives, credential/browser negative tests, retained CSP/SRI/XSS defenses and route compatibility: merged feat(docs-routes): forward protected Swagger UI keys by header instead of embedding them #823.
  • Scan scope, fail-closed evidence, severity-aware runtime advisory policy with owned/expiring exceptions, preserved release/supply-chain gates, production-vs-local guidance: this PR.
  • Infrastructure findings (shared-key deployment/storage auth, public reachability, minimum-TLS settings) were reviewed and are documented as certification-infrastructure recommendations, not demonstrated exploits. They are deferred, not implemented.

@yeongseon

Copy link
Copy Markdown
Owner Author

Merge-order note. PR #813 adds tools/verification*.py. This PR's Bandit inventory scans tools/, so after both land the security job would report those files' findings (B404, B405, B314, B603 and similar) as blocking unless they have exact reviewed policy entries. I confirmed this on the openapi pair by merging the two branches locally: 23 blocking findings, status FAIL. Merge #813 first, then rebase this PR and add the exact-location exceptions (reason, owner, expiry) for those files; pyproject.toml (typecheck line) and .github/workflows/security.yml will also need a trivial manual merge. Nothing else overlaps.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The dependency audit rejects the unpublished candidate itself, blocking first-time publication even when its third-party dependencies are clean.

1 open issue
What changed in this PR

This PR strengthens security evidence and release checks for azure-functions-openapi, addressing the security-automation portion of #812 separately from the protected-docs changes.

Changes:

  • Expands Bandit coverage and adds classified, redacted scan evidence with expiring exceptions.
  • Adds candidate-wheel dependency auditing and Swagger UI CDN integrity verification as release gates.
  • Adds regression tests and documents enforcement, recovery, and local-only example defaults.
File Description
tools/​security_report.py Defines evidence statuses and JSON/Markdown reporting.
tools/​security_exceptions.py Validates narrow, owned, expiring exceptions.
tools/​security_evidence.py Collects Bandit inventory and Semgrep outcome evidence.
tools/​dependency_audit_policy.py Runs and classifies runtime dependency audits.
tools/​browser_integrity.py Verifies pinned CDN asset hashes.
tests/​test_security_workflows.py Checks workflow gates, scope, and artifact retention.
tests/​test_security_tool_execution.py Covers tool execution and failure handling.
tests/​test_security_policy_boundaries.py Tests input validation and exception boundaries.
tests/​test_security_evidence.py Tests scan classification and redacted reporting.
tests/​test_dependency_audit_policy.py Tests advisory blocking and exception handling.
tests/​test_browser_integrity_evidence.py Tests rendered integrity attributes and hash classification.
security/​dependency-audit-policy.json Establishes blocking runtime advisory policy.
security/​bandit-policy.json Records reviewed static-analysis exceptions.
SECURITY.md Documents controls, evidence, recovery, and deployment boundaries.
pyproject.toml Adds type checking for security helpers.
examples/​README.md Clarifies local-only authentication defaults.
docs/​contributing.md Documents expanded CI scan scope.
demo/​README.md Clarifies preview tooling and synthetic-input requirements.
AGENTS.md Documents the new release gate.
.gitignore Excludes generated security evidence.
.github/​workflows/​security.yml Adds expanded scanning and retained evidence.
.github/​workflows/​publish-pypi.yml Gates publishing on dependency and CDN checks.
.github/​workflows/​dependency-audit.yml Reuses helpers for scheduled advisory reporting.

Review effort: Balanced

Comment thread tools/dependency_audit_policy.py Outdated
@yeongseon

Copy link
Copy Markdown
Owner Author

C2/C3 follow-up: 7914927 records successful action-only Semgrep evidence as INCOMPLETE (explicit action outcome only); 38a1a31 tolerates exactly collector exit 3 while preserving the pinned action gate and tool-error failures. Bandit/Semgrep collectors run always. Shared Make targets cover expanded scan/inventory, type checks, runtime blocking audit, development reporting audit, and browser/SRI; workflows and SECURITY.md use them. make security now includes the expanded inventory scan. Tests: test_semgrep_make_collector_tolerates_only_incomplete and test_local_security_includes_expanded_inventory_scan. CI is green at 9350af8. Integration note with #813: retain its verification wrapper/check-commands and maintenance interpreter; replace only its security recipe with security: security-bandit plus the separate new target block. Retain both security-types and verification mypy paths in pyproject; retain setup from #813 and collector/always changes here in security.yml. No action pins, gate names, matrix, permissions, or browser/SRI gate were weakened. Release-only workflow execution and Azure certification were not dispatched.

Bandit 1.9.4 prints a "Working..." progress bar to stdout when it scans more
than 50 files at INFO log level, so the evidence collector could not parse the
JSON that follows and reported TOOL_ERROR. The declared scope is 49 files
today, so two new Python files would have turned the gate red.

Pass -q to keep stdout pure JSON, and add a regression test that runs the real
scanner through run_bandit() on either side of the 50-file threshold.
@yeongseon

Copy link
Copy Markdown
Owner Author

R3 authorization for head a53df03603faccf3df46226fb49392ea5d63c2c1.

The maintainer delegated the merge decision to an Oracle review. The Oracle (AI) evaluated this exact head through code and workflow inspection and independent probes (real Bandit red/green at 49 and 51 files, real pip-audit with an unpublished candidate, live candidate-wheel SRI checks on 3.11/3.14, 107 targeted tests) and returned GO after one NO-GO round that is fixed in this head. This is advisory AI evidence, not line-by-line human review or human approval. The Copilot thread on unpublished candidates is addressed and resolved. Release PRs are not merged by this change.

@yeongseon
yeongseon merged commit 4855d52 into main Oct 24, 2026
37 checks passed
@yeongseon
yeongseon deleted the ci/security-gates-evidence branch October 24, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(docs-routes): harden Swagger UI key handling and release security gates

2 participants