Repository navigation
Conversation
|
Applied B1–B5 in 2865a7a, 8b0cdfb and 59cc649. Regression-first credential/symlink/binary/timeout tests; make check now records check/summary.json and redacted logs even on prerequisite failure; maintenance avoids hatch run; synthetic summaries isolated. Full suite: 97.04% coverage, Ruff and strict mypy/workflow lints pass. Clean venv mirrors: Python 3.12/3.13/3.14 full tests + worker lanes; Python 3.11 quality/schema-hash/build/docs; minimum dependencies and SDK spike pass; real func+Azurite host on 3.11/3.12 pass. Makefile overlap with security PR is confined to separate target blocks. LSP tool cannot inspect this sibling checkout (request cwd restriction); strict mypy passed instead. |
yeongseon
changed the title
|
Do not merge this draft as the implementation of #566. Current main already independently landed #586 (SDK 2.x fail-on-skip guard), #587 (expanded fail-closed Bandit), and #588 (idempotent bootstrap/typecheck/doctor fixes). The remaining #566 work should be small PRs from current main, without the custom lane summary/status/report framework excluded by the issue. This draft is not a substitute for the remaining interpreter-correct Makefile commands, JUnit/coverage always-upload, and fail-closed host/Azure lanes. Leaving closure to the maintainer. |
|
This draft is superseded by focused work on current main: #586 added fail-closed spike guards, #587 expanded and validated Bandit scope, #588 made bootstrap/check/doctor thin and fail-closed, and #601 adds always-uploaded JUnit/coverage evidence. Remaining #566 acceptance includes canonical Makefile execution in each CI matrix cell, local test-results output, worker/host/Azure prerequisite guards and retained host logs. The custom reporting/orchestration framework in this draft is out of scope of #566. Leaving this superseded draft open for the maintainer; it will not be merged. |
There was a problem hiding this comment.
🟡 Changes recommended
Credential-redaction gaps can expose secrets in uploaded reports, and local diagnostics can inspect the wrong Python environment.
3 open issues
What changed in this PR
This PR standardizes local and CI validation for the library, introducing shared verification helpers and saved test evidence.
Changes:
- Defines canonical Makefile commands and makes
check-allan alias ofverify. - Adds fail-closed verification for tests, schema integrity, worker compatibility, host integration, and installed wheels.
- Adds CI artifact uploads, regression tests, and contributor documentation.
| File | Description |
|---|---|
| tools/verification.py | Coordinates verification lanes and result summaries. |
| tools/verification_wheel.py | Checks freshly built wheel imports and schemas. |
| tools/verification_process.py | Manages subprocess cleanup and redacted logs. |
| tools/verification_host.py | Runs managed Functions host and Azurite smoke checks. |
| tools/verification_evidence.py | Collects, redacts, and publishes verification evidence. |
| tools/verification_environment.py | Checks prerequisites and cleans generated artifacts. |
| tools/verification_contracts.py | Checks schema hashes without rewriting pins. |
| tools/__init__.py | Makes verification helpers importable as a package. |
| tests/test_verification_review.py | Tests redaction, readiness, and maintenance behavior. |
| tests/test_verification_evidence.py | Tests evidence classification and runner behavior. |
| tests/test_verification_commands.py | Tests command contracts and failure evidence. |
| tests/test_verification_ci.py | Checks workflow and Makefile integration contracts. |
| tests/conftest.py | Isolates job summaries during verification tests. |
| pyproject.toml | Includes helpers in quality checks and updates runners. |
| Makefile | Defines canonical validation and maintenance commands. |
| docs/testing.md | Explains offline tests and integration lanes. |
| docs/METADATA_SPEC.md | Updates schema-check command documentation. |
| docs/development.md | Documents commands and evidence semantics. |
| docs/contributing.md | Updates the required local verification guidance. |
| docs/agent-playbook.md | Aligns agent guidance with canonical commands. |
| CONTRIBUTING.md | Documents setup, validation, and saved evidence. |
| AGENTS.md | Updates repository validation guidance. |
| .gitignore | Excludes generated test evidence. |
| .github/workflows/worker-nightly.yml | Uses the worker runner and uploads evidence. |
| .github/workflows/security.yml | Installs dependencies on the selected interpreter. |
| .github/workflows/publish-pypi.yml | Uses the shared test runner and uploads results. |
| .github/workflows/e2e-azure.yml | Uses the Azure runner and uploads its reports. |
| .github/workflows/ci-test.yml | Integrates shared verification and artifact uploads. |
Review effort: Balanced
| _SECRET_ENV_NAME = re.compile( | ||
| r"(?i)secret|token|password|connection|string|accountkey|functions.key|api.?key|sas" | ||
| ) | ||
| _TEXT_SUFFIXES = frozenset({".log", ".txt", ".xml", ".json", ".md"}) |
| if isinstance(value, dict): | ||
| return {redact(str(key)): redact_json(item) for key, item in value.items()} |
| else | ||
| RUN := $(PYTHON) | ||
| endif | ||
| MAINTENANCE_PYTHON ?= $(shell for p in "$$VIRTUAL_ENV/bin/python" "$(VENV_DIR)/bin/python" python3.11 python3 python; do if "$$p" -c 'import sys; sys.exit(not (3,11) <= sys.version_info[:2] < (3,15))' 2>/dev/null; then command -v "$$p"; break; fi; done) |
Summary
Standardize local and CI validation and generate redacted test reports, including worker-signature and schema checks.
Changes
check-allas an alias forverify.Validation
make doctor,make check,make test, workflow lint, build, docs, integration,make verify, report and clean passed on Python 3.11.15.make verifyreported 804 passes.INCOMPLETE.Related Issues
Closes #566