Repository navigation
test(host-smoke): request every registered docs route on the real Functions host - #829
Conversation
…marker Request every route the fixture registers through register_openapi_routes (JSON, YAML and docs; anonymous and protected) on a running host and assert 200 plus a body marker. The real-host tests are now selected by a host_smoke marker instead of a HOST_SMOKE_BASE_URL skipif, so they fail rather than skip when no host is given, and pyproject addopts deselect them from the offline suite (0 new skips). Refs #824 Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…host The host-smoke job only proved that the Functions host indexes the fixture and answered three hand-written routes, so a helper handler that fails when invoked passed CI. Install pytest, pytest-cov and requests into the isolated smoke venv and run the host_smoke tests against the live host before it is stopped. The pytest status is kept until the host log has been printed and scanned, and the step still fails with pytest's own exit code (5 for a wrong marker selector). A workflow hardening test keeps HOST_SMOKE_BASE_URL, the -m host_smoke selector and the status propagation wired, so the real-host tests cannot go dead again. Refs #824 Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
|
R3 authorization for head The maintainer delegated the merge decision to an Oracle review. The Oracle (AI) reviewed this change at head |
Summary
The
host-smokejob proved that the Functions host indexestests/host_smoke/function_app.py, but it only ever requested three hand-written routes. The six routes the fixture registers throughregister_openapi_routes(JSON, YAML and docs; anonymous and protected) were never requested, so a helper handler that fails when invoked passed CI, and the only real-host test (test_real_host_protected_docs_never_echoes_key) was skipped on every lane because nothing setsHOST_SMOKE_BASE_URL. This PR requests every registered route on the real host, asserting 200 plus a body marker, and makes that test impossible to skip silently.Changes
tests/host_smoke/test_fixture_routes.py: new parametrized real-host test over the 6 registered routes (registered/*,protected/*; JSON, YAML, docs). The two real-host tests carry ahost_smokemarker instead of theHOST_SMOKE_BASE_URLskipif; selected without that variable they fail (KeyError) instead of skipping.pyproject.toml: registers thehost_smokemarker and addsnot host_smoketoaddopts, so the offline suite deselects the real-host tests likee2e/perf/browser(0 skips, coverage unchanged)..github/workflows/ci-test.yml(host-smokejob only; job and step names, matrix, required contexts and the final host-log scan are unchanged): installspytest pytest-cov requestsinto the isolated smoke venv (pytest-covis needed to parse the--covoptions inaddopts), and runsHOST_SMOKE_BASE_URL=http://127.0.0.1:7071 tests/host_smoke/.venv/bin/python -m pytest tests/host_smoke -m host_smoke --no-cov -rsagainst the live host before it is killed. The pytest status is kept until the host log has been printed and scanned, so a failing run still shows the host's own exception, and the step then exits with pytest's exit code.tests/test_ci_workflow_hardening.py: YAML-parses thehost-smokejob and assertsHOST_SMOKE_BASE_URL=,-m host_smokeand the status propagation stay wired, so the real-host tests cannot go dead again.Net diff: 4 files, +53/-5. No Makefile,
tools/,src/,examples/,scripts/ordemo/change.Verification (everything below was executed locally; Python 3.11 unless stated,
azure-functions1.25.0, Core Tools 4.8.0, Azurite 3.35.0)Offline suite (
pytest tests/, the 3.11 x latest coverage cell,azure-functions-validationinstalled):origin/main4d3a528)1534 passed, 1 skipped, 15 deselected, coverage 95.14%; the skip istests/host_smoke/test_fixture_routes.py:1101535 passed, 22 deselected, 0 skipped, coverage 95.14%1504 passed, 31 skipped; after1505 passed, 30 skipped(the 30 are the existing companion-dependent skips, none fromtests/host_smoke)make check-all(Hatch env, py3.11): exit 0 (lint-workflows, ruff,ruff format --check, mypy strict onsrc tests,make security-types, tests1505 passed, 30 skipped, 22 deselectedat 95.14%, Bandit).Real Functions host: the modified
Exercise OpenAPI routes on a real Functions hoststep script was run from the workflow YAML (bash -e) with only the substitutions a shared box forces: localazuriteon free ports in a private--locationinstead ofnpx --yes azurite,func start --port <free>with every7071replaced, and the trailingwait "$host_pid"replaced by a SIGKILL of that run's own host because the localfuncignores SIGTERM. The smoke venv was built by the unchanged install step lines (minus the globalnpm install).origin/mainstep7 passed, 5 deselected(6 runs on 3.11, 1 on 3.12)openapi_yamlhandler raises when invoked (host still indexes the app)2 failed, 5 passed,assert 500 == 200forregistered/openapi.yamlandprotected/openapi.yaml, host log printed,::error::The Functions host reported indexing or invocation errors(same on 3.12)openapi_jsonreturns an explicit 500 (host logsSucceeded, so only the new tests can see it)2 failed, 5 passed,::error::The real-host route tests failed (pytest exit 1)-m host_smok)pytest exit 5)Outside the step:
pytest tests/host_smoke -m host_smokewithHOST_SMOKE_BASE_URLunset gives7 failed(KeyError: 'HOST_SMOKE_BASE_URL', exit 1, no skips); pointing it at a closed port gives7 failed(ConnectionError, exit 1);--collect-only -m host_smokeselects exactly 7 tests.Guard test: passes on this branch; fails against
origin/main's workflow and against copies of this workflow with the-m host_smokeselector, theHOST_SMOKE_BASE_URL=variable or theexit "$pytest_status"propagation removed.Bandit evidence (#817):
make securityisPASS {"files": 49, "loc": 8902, "errors": 0, "blocking": 0, "stale_exceptions": 0}with the 13 line-pinned LOW findings still excepted; nothing undersrc examples demo scripts toolschanged, so thesecurity/bandit-policy.jsonpins are unaffected.Not verified locally: the GitHub-hosted run itself (
npm install --global azure-functions-core-tools@4,npx --yes azurite,ubuntu-latest); see this PR's checks.Type of Change
Checklist
make check-alland all checks pass (including tests and at least 95% coverage)Related Issues
Closes #824
Refs #811
Review Evidence
host-smokejob steps inci-test.ymlchange; nopermissions:, token, secrets, release/publishing, Swagger UI, CLI-import or policy-file change