ci: unify verification commands and test reports by yeongseon · Pull Request #813 · yeongseon/azure-functions-openapi-python

yeongseon · 2026-10-22T14:09:42Z

Summary

Use consistent verification commands locally and in CI, with structured test reports for troubleshooting.

Changes

Add canonical Makefile commands and keep check-all as an alias for verify.
Move test orchestration into Python helpers under tools/ and add regression tests.
Generate redacted reports under gitignored test-results/, with explicit pass, fail, incomplete, and not-applicable states.
Run lint, typing, security, offline tests, Functions host/Azurite integration, and package checks through the shared commands.
Reuse the same reporting approach in CI without changing required job names, release gates, or publication artifacts.
Avoid repeated Hatch bootstrap and preserve the existing deep-clean command.

Validation

Locally passed: make doctor, make check, make test, make lint-workflows, make perf, make build, make docs, make integration, make verify, make report, and make clean.
Tests: 1,437 passed, 95.11% coverage on Python 3.11.15. All six local verification lanes passed.
Unconfigured make e2e-azure correctly returned INCOMPLETE.
Remote CI matrix, live Azure certification, and non-dry-run bootstrap commands were not verified locally.

Related Issues

Closes #811

… step accurately

Copilot

🟡 Changes recommended

Credential redaction can expose secrets and corrupt saved evidence, and verification reliability issues remain unresolved.

5 open issues

Redaction corrupts quoted secrets and serialized JSON · New
Quality evidence upload has no producer · New
Read-only doctor command provisions Hatch environment · New
Tests write synthetic failures to CI job summary · New
Startup retry loop misses TimeoutError · New

What changed in this PR

Standardizes local and CI verification for the OpenAPI library, with shared runners and saved evidence for issue #811.

Changes:

Defines canonical Makefile commands and separates offline tests, managed integration, and Azure certification.
Adds verification helpers for process cleanup, reporting, redaction, and wheel smoke checks.
Updates CI artifact uploads, regression tests, and contributor documentation.

File	Description
`tools/verification.py`	Orchestrates verification lanes and aggregate results.
`tools/verification_wheel.py`	Adds installed-wheel smoke verification.
`tools/verification_process.py`	Manages subprocess timeouts, cleanup, and logs.
`tools/verification_host.py`	Runs managed Functions-host and Azurite checks.
`tools/verification_evidence.py`	Collects, redacts, saves, and renders evidence.
`tools/verification_environment.py`	Adds prerequisite diagnostics and artifact cleanup.
`tools/__init__.py`	Establishes the tools package.
`tests/test_verification_evidence.py`	Tests evidence classification, serialization, and redaction.
`tests/test_verification_commands.py`	Tests command contracts and process lifecycle.
`tests/test_validate_http_shorthand_conflict.py`	Adds a typed helper for optional validator imports.
`tests/test_ci_workflow_hardening.py`	Checks shared runners and always-run uploads.
`pyproject.toml`	Routes tests through the runner and expands type checking.
`Makefile`	Defines canonical commands and delegates orchestration.
`docs/testing.md`	Explains test lanes and CI evidence.
`docs/development.md`	Documents commands and evidence layout.
`docs/contributing.md`	Updates contributor validation guidance.
`docs/agent-playbook.md`	Updates automated-development verification guidance.
`CONTRIBUTING.md`	Documents setup, verification, and evidence contracts.
`AGENTS.md`	Updates canonical commands and the local merge gate.
`.gitignore`	Excludes generated verification evidence.
`.github/workflows/publish-pypi.yml`	Uses the shared library-test runner and uploads evidence.
`.github/workflows/e2e-azure.yml`	Integrates Azure verification evidence and JUnit paths.
`.github/workflows/ci-test.yml`	Adopts shared runners and evidence uploads across CI lanes.

_{Review effort: Balanced}

yeongseon · 2026-10-23T00:44:50Z

+    text = re.sub(r"(?im)(authorization[\"']?\s*[:=]\s*)[^\r\n<\"]+", r"\1[REDACTED]", text)
+    text = re.sub(r"(https?://)[^/@\s]+:[^/@\s]+@", r"\1[REDACTED]@", text)
+    text = re.sub(
+        r"(?i)(?<![\w-])((?:[\w-]*(?:token|password|secret|connectionstring)|AccountKey|"
+        r"SharedAccessSignature|AzureWebJobsStorage|x-functions-key|code|sig)"
+        r"[\"']?\s*[:=]\s*[\"']?)[^\s;&<\"']+",
+        r"\1[REDACTED]",


Verified already fixed by a94d522; additional regression coverage in 7f64309. Existing parametrized tests cover quoted Authorization/Proxy-Authorization, whitespace passwords, delimiters, escaped quotes, connection-string keys, XML validity, JSON serialization, binary preservation and symlink safety. test_overlapping_escaped_environment_secrets_are_replaced_longest_first additionally proves longest-first replacement of overlapping JSON-escaped environment secrets and nested tuple/string-leaf serialization. Full suite passes at 95.11% coverage; CI green at b1cc3ed. No thread resolved.

yeongseon · 2026-10-23T00:44:52Z

+check: format-check typecheck security lint-workflows
+	@$(RUN) -m ruff check src tests tools/verification*.py
+	@$(RUN) scripts/lint_mermaid.py
+	@$(RUN) scripts/check_screenshots.py --strict


Fixed in 926eba6. make check now enters the shared evidence runner before invoking make check-commands, so all prerequisites run inside the recorded check lane. summary.json, redacted logs and Job Summary survive prerequisite failure and the original nonzero result propagates. test_check_saves_prerequisite_evidence covers both success and a failing prerequisite containing a credential canary (failed before the fix). CI green at b1cc3ed, including the quality job.

yeongseon · 2026-10-23T00:44:53Z

-	@echo "Azure Function Core Tools version:" && func --version || echo "func not found"
-	@echo "Pre-commit hook installed:"
-	@if [ -f .git/hooks/pre-commit ]; then echo yes; else echo no; fi
+	@$(RUN) -m tools.verification doctor


Fixed in 926eba6 for doctor, report and clean. Maintenance resolves an existing interpreter read-only (active environment, .venv, hatch env find, then compatible system Python) without hatch run/create/sync. Missing required modules produce development environment not created: run make install. test_maintenance_never_bootstraps_hatch executes all three commands in fresh checkout mirrors with CI unset, empty HATCH_DATA_DIR and a pip-free environment; it asserts no provisioning and no pyvenv.cfg under Hatch data. These cases failed before the fix. CI green at b1cc3ed.

yeongseon · 2026-10-23T00:44:54Z

+    assert process.poll() is not None
+
+
+def test_verify_runs_every_required_lane_even_after_failure(


Fixed in 7555b1d with an autouse fixture redirecting GITHUB_STEP_SUMMARY for every verification test module. Regression test_verification_tests_leave_outer_summary_untouched in 926eba6 launches the deliberately failing/missing-prerequisite tests in a subprocess with an outer summary sentinel and proves it remains byte-for-byte unchanged. Tests that explicitly inspect summaries still use their own temporary destination. CI green at b1cc3ed.

yeongseon · 2026-10-23T00:44:56Z

+            with urlopen(url, timeout=1) as response:
+                if response.status < 500:
+                    return
+        except URLError:


Fixed in 926eba6. The bounded readiness loop retries OSError (including TimeoutError, ConnectionError and URL errors) and http.client.HTTPException rather than aborting startup. The other readiness loop already catches OSError. test_readiness_retries_transient_failure covers each transient failure followed by success; all four failed before the fix. Both real-host CI jobs pass at b1cc3ed.

…dit job

…g evidence

…sioning

yeongseon · 2026-10-23T00:45:15Z

All five inline findings replied with commit and test evidence; overview adds no other findings. CI green at b1cc3ed. Integration note for #817: preserve this branch’s check wrapper/check-commands and maintenance interpreter; add its separate security Make target block and security: security-bandit dependency, plus both tool typecheck sets. security.yml needs this branch’s selected-interpreter setup and #817’s always-run evidence collectors. No threads resolved and no PR merged.

yeongseon · 2026-10-26T09:13:40Z

Superseded by #843 (CI test evidence retention) and #844 (integration, wheel-smoke and verify targets), which implement the remaining #811 items minimally on current main. This draft now conflicts with main in 7 files and adds tooling the issue lists as out of scope. Leaving it open for the maintainer to close.

yeongseon added 10 commits October 22, 2026 23:06

ci: add versioned redacted verification evidence

55e6589

ci: manage fail-closed verification lanes and process lifecycles

a00930f

ci: standardize canonical development and verification commands

31aee7a

ci: share verification runners across interpreter and host lanes

f9452d1

ci: retain standardized evidence through release certification gates

fa09202

test: type optional validation decorators without stale suppressions

4fc1b3b

docs: define canonical verification commands and evidence meanings

df7a319

docs: align developer command tables with full local verification

d6668fa

docs: distinguish offline host and Azure verification lanes

f0502ee

ci: keep clean-all as the deep maintenance clean and name the quality…

4168e55

… step accurately

yeongseon requested a balanced review from Copilot October 22, 2026 14:24

Copilot started reviewing on behalf of yeongseon October 22, 2026 14:25

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

yeongseon mentioned this pull request Oct 22, 2026

ci(security): enforce fail-closed scan evidence and a runtime dependency release gate #817

Merged

7 tasks done

yeongseon added 7 commits October 23, 2026 07:10

ci: parse generated test reports without semgrep's unsafe xml parse rule

6f1e8ca

ci(security): install the selected interpreter's tools before the ban…

bc9e2bd

…dit job

ci: redact quoted and connection-string credentials without corruptin…

a94d522

…g evidence

test(evidence): cover overlapping escaped credentials

7f64309

test(evidence): isolate synthetic job summaries

7555b1d

ci(verification): retain quality evidence and avoid maintenance provi…

926eba6

…sioning

style(scripts): sort screenshot checker imports

b1cc3ed

yeongseon marked this pull request as draft October 24, 2026 02:13

yeongseon mentioned this pull request Oct 24, 2026

ci: align command contracts and close real-host and lane coverage gaps #811

Closed

5 tasks

yeongseon changed the title ~~ci: standardize canonical verification commands and evidence~~ ci: unify verification commands and test reports Oct 25, 2026

Navigation Menu

Repository navigation

ci: unify verification commands and test reports - #813

ci: unify verification commands and test reports#813
yeongseon wants to merge 17 commits into
mainfrom
ci/standardize-verification-evidence

yeongseon commented Oct 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

yeongseon Oct 23, 2026

Uh oh!

yeongseon Oct 23, 2026

Uh oh!

yeongseon Oct 23, 2026

Uh oh!

yeongseon Oct 23, 2026

Uh oh!

yeongseon Oct 23, 2026

Uh oh!

yeongseon commented Oct 23, 2026

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		assert process.poll() is not None


		def test_verify_runs_every_required_lane_even_after_failure(

Conversation

yeongseon commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Related Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

yeongseon Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

yeongseon Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

yeongseon Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

yeongseon Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

yeongseon Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

yeongseon commented Oct 23, 2026

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yeongseon commented Oct 22, 2026 •

edited

Loading