ci: unify logging verification commands and reports by yeongseon · Pull Request #616 · yeongseon/azure-functions-logging-python

yeongseon · 2026-10-22T16:04:42Z

Summary

Standardize verification commands and redacted reports for the logging library and its OpenTelemetry/Functions host integrations.

Changes

Add canonical Makefile commands, keeping check-all equivalent to verify.
Report offline tests, OpenTelemetry instrumentation, Functions host/Azurite integration, and Azure E2E separately.
Distinguish local success from actual Application Insights ingestion and deployment verification; missing cloud assertions produce INCOMPLETE.
Redact connection strings, instrumentation keys, tokens, and SAS values in saved and displayed test evidence.
Reuse verification helpers in the CI interpreter matrix while preserving required release and security checks.
Add Mermaid render lint to make check, requiring mmdc and a discoverable Chrome binary.

Validation

Python 3.11.15: make doctor, make check, make test, workflow lint, integration, verify, build, docs, report and clean passed.
Tests: 894 passed, four optional-dependency skips, 96.26% coverage.
Host/Azurite integration: 77 passed. All five local verification lanes passed.
Not run: remote CI matrix, real Azure deployment, and Application Insights ingestion. Unconfigured E2E returned INCOMPLETE.

Related Issues

Closes #614

…dit job

Mock Python module discovery for the Mermaid prerequisite test and prove each required module still fails closed independently. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>

…ance

yeongseon · 2026-10-23T01:05:20Z

Applied sibling review findings B1–B5. B1 cbc18fb ports a94d522 while preserving App Insights InstrumentationKey/ingestion/live endpoint and storage masking; quoted/escaped/whitespace credentials, JSON leaves, binaries and external symlinks covered by test_verification_review_regressions. B2/B3 7d95d22 resolves an existing interpreter without Hatch synchronization, tests fresh HATCH_DATA_DIR, and captures the complete check prerequisite graph with redacted log/summary and failure exit preservation. B4 e2f6b74 isolates synthetic summaries. B5 97cf1e4 retries readiness transport/timeouts (timeout then success regression). Clean venv matrix 3.11–3.14 tests/OTel, minimum azure-functions==1.21.0 full suite, quality/build/wheel/docs, real host integrations 3.11/3.12 passed. Coverage 96.26%; strict mypy, Ruff and all three workflow lints passed. CI all green including ci-required, minimum dependencies, host smokes, wheel install/rejection, docs, security and CodeQL. No inline/overview reviews existed. Security PR #618 changes the Makefile separately; combine its security block with this branch’s maintenance/check changes rather than taking either whole file.

Copilot

🟡 Changes recommended

Confirmed credential leaks in saved JSON, XML, and HTML evidence must be resolved before CI uploads these artifacts.

5 open issues

What changed in this PR

This PR standardizes local and CI verification for the Azure Functions logging library, adding shared runners and saved evidence for unit, integration, and cloud checks.

Changes:

Defines canonical Makefile commands and selected-interpreter CI execution.
Adds lane summaries, credential redaction, managed host processes, and stricter certification checks.
Adds regression tests and updates contributor documentation and workflows.

File	Description
`tools/verification.py`	Orchestrates verification lanes and aggregate results.
`tools/verification_wheel.py`	Checks installed-wheel imports and formatting.
`tools/verification_process.py`	Manages subprocess cleanup and captured output.
`tools/verification_host.py`	Runs managed Functions host and Azurite checks.
`tools/verification_evidence.py`	Collects, sanitizes, saves, and renders evidence.
`tools/verification_environment.py`	Checks prerequisites and cleans generated artifacts.
`tools/verification_check.py`	Wraps quality checks for evidence capture.
`tools/maintenance_python.sh`	Selects an existing maintenance interpreter.
`tools/__init__.py`	Marks verification tooling as a Python package.
`tests/test_verification_review_regressions.py`	Covers redaction, readiness, and failure evidence.
`tests/test_verification_evidence.py`	Tests evidence classification and runner behavior.
`tests/test_verification_commands.py`	Tests public command contracts.
`tests/test_verification_ci.py`	Checks workflow and interpreter contracts.
`tests/conftest.py`	Isolates verification tests from CI job summaries.
`pyproject.toml`	Routes Hatch commands through shared verification.
`Makefile`	Defines canonical and focused verification targets.
`docs/testing.md`	Explains offline, integration, and certification lanes.
`docs/development.md`	Documents setup, commands, and evidence semantics.
`docs/contributing.md`	Updates contributor validation guidance.
`docs/agent-playbook.md`	Updates agent-facing verification guidance.
`CONTRIBUTING.md`	Defines command and evidence contracts.
`AGENTS.md`	Updates validation rules and command references.
`.gitignore`	Excludes generated verification evidence.
`.github/workflows/security.yml`	Uses selected-interpreter dependency setup.
`.github/workflows/publish-pypi.yml`	Captures library evidence and pins the publisher action.
`.github/workflows/otel-nightly.yml`	Uses the OTel runner and uploads evidence.
`.github/workflows/e2e-host-matrix.yml`	Replaces shell orchestration with managed host checks.
`.github/workflows/e2e-azure.yml`	Uses certification-aware execution and evidence uploads.
`.github/workflows/ci-test.yml`	Adopts shared runners across CI verification jobs.

_{Review effort: Balanced}

+_SECRET_ENV_NAME = re.compile(
+    r"(?i)secret|token|password|connection|string|accountkey|functions.key|api.?key|sas|appinsights"
+)
+_TEXT_SUFFIXES = frozenset({".log", ".txt", ".xml", ".json", ".md"})


+    if isinstance(value, dict):
+        return {redact(str(key)): redact_json(item) for key, item in value.items()}


+            return dump_json(json.loads(text))
+        except ValueError:
+            return redact(text)
+    return redact(text)


+def verify() -> int:
+    directory = Path("test-results/verify")
+    if directory.exists():
+        shutil.rmtree(directory)
+    outcomes = []


+                wait_ready(host, base + ("/health" if matrix else "/smoke"))
+                text = assert_routes(base, matrix, directory)


yeongseon · 2026-10-26T11:40:21Z

Do not merge this superseded broad draft. Main already includes #628 (86f964a: fail-on-skip certification/nightly hook and YAML tests) and #631 (f1956b7: tooling bootstrap/check/doctor contract), verified with git log and git branch --contains. These implement the narrower approach recorded in #614. Residual local-only wording and negative-run acceptance evidence are recorded on #614; do not add the old shared report framework or mandatory Mermaid/browser tooling to resolve them. Leaving this draft open and its branch untouched for the maintainer to close.

yeongseon · 2026-10-26T13:57:53Z

Superseded draft; not merging wholesale. Focused #628 and #631 merged strict skip policies and lean Makefile/tool checks. #614 still requires an explicit local-only e2e command contract plus recorded negative-run evidence and verification against current main. #613 SDK wheel/host range certification is not established by this draft. Remaining changes will use focused PRs; leaving draft closure to the maintainer.

yeongseon · 2026-10-26T15:19:06Z

Final supersession update: #628/#631 and merged #640 complete #614, now closed. SDK floor/latest candidate-wheel and real local-host certification landed separately in #643 and closed #613. No remaining acceptance from either issue requires this broad verification draft. Do not merge the superseded draft; closure remains the maintainer’s decision.

yeongseon added 8 commits October 23, 2026 01:02

ci: add fail-closed verification evidence and managed host lanes

94619ce

build: standardize local verification command contracts

67ce9f3

ci: reuse canonical verification on selected matrix interpreters

6e5ea10

ci: record separate OTel instrumentation evidence

a86f3ec

ci: manage host matrix lifecycle and correlation evidence

fab53c0

ci: require observed App Insights assertions for certification

8ee29d0

ci: preserve release evidence and pin the OIDC publisher

38dd4b3

docs: define verification commands and certification boundaries

84e0009

yeongseon mentioned this pull request Oct 22, 2026

ci(security): verify scan results and audit release dependencies #618

Draft

yeongseon and others added 7 commits October 23, 2026 07:10

ci: parse generated test reports without semgrep's unsafe xml parse rule

492b307

ci(security): install the selected interpreter's tools before the ban…

b09ad6d

…dit job

test(verification): isolate synthetic job summaries

e2f6b74

ci(evidence): redact whole credentials and preserve structured assets

cbc18fb

ci(verification): capture quality gates without bootstrapping mainten…

7d95d22

…ance

ci(host): retry transient readiness transport errors

97cf1e4

yeongseon marked this pull request as draft October 24, 2026 02:13

yeongseon mentioned this pull request Oct 24, 2026

ci: fail certification and upstream-canary lanes that skip tests, and trim the command contract #614

Closed

6 tasks

yeongseon requested a balanced review from Copilot October 24, 2026 10:04

Copilot started reviewing on behalf of yeongseon October 24, 2026 10:05

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

yeongseon mentioned this pull request Oct 24, 2026

ci(release): block publishing on known runtime dependency advisories #630

Merged

1 of 3 tasks

yeongseon changed the title ~~ci: standardize canonical verification commands and evidence~~ ci: unify logging verification commands and reports Oct 26, 2026

Navigation Menu

Repository navigation

ci: unify logging verification commands and reports - #616

ci: unify logging verification commands and reports#616
yeongseon wants to merge 15 commits into
mainfrom
ci/standardize-verification-evidence

yeongseon commented Oct 22, 2026 •

edited

Loading

Uh oh!

yeongseon commented Oct 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if isinstance(value, dict):
		return {redact(str(key)): redact_json(item) for key, item in value.items()}

		wait_ready(host, base + ("/health" if matrix else "/smoke"))
		text = assert_routes(base, matrix, directory)

Conversation

yeongseon commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Related Issues

Uh oh!

yeongseon commented Oct 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

yeongseon commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yeongseon commented Oct 22, 2026 •

edited

Loading