Skip to content

test(security): verify validation error handling and release checks - #569

Draft
yeongseon wants to merge 20 commits into
mainfrom
test/validation-trust-boundaries-and-security-evidence
Draft

yeongseon wants to merge 20 commits into
mainfrom
test/validation-trust-boundaries-and-security-evidence

Conversation

@yeongseon

@yeongseon yeongseon commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

Strengthen validation security tests and release checks. Testing did not identify a default 5xx response leak, but it found inconsistent handling of deeply nested JSON across Python versions.

Changes

  • Test error disclosure across sync/async adapters, SDK requests, custom formatters, backend extensions, malformed payloads, and expose_internal_errors.
  • Apply a 512-level JSON nesting limit before parsing. Excessive nesting now returns a sanitized HTTP 400 on Python 3.11–3.14 instead of varying by interpreter. Ordinary model-validation failures remain HTTP 422.
  • Check pinned schema integrity without modifying the original file; add settings-redaction regression tests.
  • Add fail-closed Bandit evidence and expiring exceptions, plus a release dependency audit of the exact wheel across Python 3.11–3.14.
  • Document the boundary between client responses and server-side diagnostics in SECURITY.md. Existing logging behavior remains unchanged; log redaction or handler-exception containment would require separate work.

Validation

  • Local Python 3.11 full suite passed at 97.39% coverage. Ruff, strict mypy, workflow lints, strict MkDocs, package build, and wheel schema verification passed.
  • Intentional mutation tests correctly detected unsafe formatter and digest-comparison changes.
  • Bandit evidence covered 35 files and 3,956 lines of code: 33 reviewed low findings and no blocking findings. Isolated wheel dependency audit found zero advisories among nine packages.
  • Not run: GitHub Actions execution, SDK/worker matrix, or real Azure checks. Remaining Azure configuration concerns are review findings, not confirmed vulnerabilities.

Related Issues

Closes #567

yeongseon and others added 14 commits October 23, 2026 00:39
Reject bodies beyond 512 container levels before parsing so CPython scanner differences cannot change a malformed-input 400 into a model-type 422. Preserve ordinary model validation and ignore brackets within JSON strings.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@yeongseon

Copy link
Copy Markdown
Owner Author

Merge-order note. PR #568 adds tools/verification*.py. This PR's Bandit inventory scans tools/, so after both land the security job would report those files' findings (B404, B405, B314, B603 and similar) as blocking unless they have exact reviewed policy entries. I confirmed this on the openapi pair by merging the two branches locally: 23 blocking findings, status FAIL. Merge #568 first, then rebase this PR and add the exact-location exceptions (reason, owner, expiry) for those files; pyproject.toml (typecheck line) and .github/workflows/security.yml will also need a trivial manual merge. Nothing else overlaps.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The JSON pre-scan permits quadratic CPU consumption, and the required dependency audit blocks unpublished release candidates.

2 open issues
What changed in this PR

Strengthens security verification for the Azure Functions validation library, covering HTTP trust boundaries, package integrity, and release-time dependency checks for #567.

Changes:

  • Adds security regression tests and a portable 512-level JSON nesting limit.
  • Introduces redacted scanner evidence, expiring exceptions, and a blocking runtime dependency audit.
  • Documents production boundaries and extends SDK/worker compatibility coverage.
File Description
tools/​security_report.py Defines structured, allow-listed security reports.
tools/​security_exceptions.py Validates exact, time-limited exceptions.
tools/​security_evidence.py Collects Bandit inventory and Semgrep execution evidence.
tools/​dependency_audit_policy.py Classifies dependency audits and enforces release policy.
tests/​test_security_workflows.py Checks audit gating, permissions, and evidence uploads.
tests/​test_security_runtime.py Tests response sanitation across extension points.
tests/​test_security_inputs.py Tests hostile input, limits, and nesting boundaries.
tests/​test_security_evidence.py Tests scanner failures, exceptions, and redacted evidence.
tests/​test_security_auth.py Verifies host-owned authentication remains unchanged.
tests/​test_print_safe_app_settings.py Adds credential-shape redaction coverage.
tests/​test_dependency_audit_policy.py Tests advisory blocking and exception handling.
tests/​test_check_wheel_schemas.py Verifies tampering fails without modifying artifacts.
src/​azure_functions_validation/​decorator.py Clarifies the production risk of error exposure.
src/​azure_functions_validation/​adapter.py Routes body parsing through the depth-limited helper.
src/​azure_functions_validation/​_json_body.py Adds nesting checks and normalized parsing errors.
security/​dependency-audit-policy.json Sets the runtime advisory blocking policy.
security/​bandit-policy.json Records exact, expiring static-analysis exceptions.
SECURITY.md Documents trust boundaries, controls, and recovery.
pyproject.toml Adds explicit type checking for security tools.
examples/​README.md Distinguishes demonstrations from production configurations.
docs/​configuration.md Documents nesting limits and formatter safety.
AGENTS.md Links contributor guidance to security policy.
.gitignore Excludes generated security evidence.
.github/​workflows/​worker-nightly.yml Runs security tests against worker previews.
.github/​workflows/​security.yml Adds expanded scanning and evidence uploads.
.github/​workflows/​publish-pypi.yml Requires candidate dependency audits before publishing.
.github/​workflows/​dependency-audit.yml Reuses shared tooling for advisory reports.
.github/​workflows/​ci-test.yml Adds security tests to SDK 2.x compatibility checks.

Review effort: Balanced

Comment thread src/azure_functions_validation/_json_body.py
Comment thread tools/dependency_audit_policy.py
@yeongseon
yeongseon marked this pull request as draft October 24, 2026 02:13
@yeongseon yeongseon changed the title test(security): prove validation trust boundaries and gate releases on runtime advisories test(security): verify validation error handling and release checks Oct 26, 2026
@yeongseon

Copy link
Copy Markdown
Owner Author

Do not merge this draft wholesale. The depth fix is already on main via #583; fail-closed expanded Bandit landed via #587. Fresh focused #591 addresses #585 (oversized JSON integer 500) and draft #592 addresses the supported Semgrep CLI for #579. These supersede the corresponding portions, not the entire security tracker: exact-candidate runtime advisory gating and remaining disclosure/auth proofs still need focused current-main work. #567 stays open until all acceptance criteria/children are done. Leaving draft closure to the maintainer; no changes made to its branch or dirty checkout.

@yeongseon

Copy link
Copy Markdown
Owner Author

This draft is superseded by focused fixes on current main: #583 bounds hostile JSON depth, #591 sanitizes oversized integers, #587 makes expanded Bandit scanning fail closed, and #592 replaces deprecated Semgrep (still awaiting GitHub seeded-finding proof). #567 still needs exact-candidate runtime dependency advisory gating across Python 3.11–3.14, fail-closed tool-error behavior, and focused regression evidence for the remaining error-disclosure/auth contracts. This draft’s broader exception/reporting framework is explicitly outside #567 scope. Leaving this superseded draft open for the maintainer; it will not be merged.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Security interpreter selection breaks supported setups, and the audit CLI regression test bypasses the failure-propagation path it intends to verify.

2 open issues
2 resolved since last review

Review effort: Balanced

Comment thread Makefile
grep -E '^\.PHONY: ' Makefile | cut -d ':' -f2 | xargs -n1 echo " - make"

# Security entry points use the selected CI interpreter without provisioning Hatch.
SECURITY_PYTHON ?= $(if $(CI),python,hatch run python)
Comment on lines +218 to +225
args = ["audit", "--site-packages", str(tmp_path), "--output", str(tmp_path)]
if development:
args.append("--development")
monkeypatch.setattr(sys, "argv", args)
monkeypatch.setattr(
dependency_audit_policy,
"run_audit",
lambda path, policy: Check("pip-audit", (), Status.INCOMPLETE, None),
@yeongseon

Copy link
Copy Markdown
Owner Author

The remaining #567 scope is now completed by merged #608: candidate runtime advisory matrix and focused response/auth proofs, on top of prior hostile-input, Bandit and supported-Semgrep fixes. #567 is closed. This superseded draft remains open for the maintainer as instructed and must not merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(security): bound hostile input and gate releases on runtime advisories

2 participants