Skip to content

test(host-smoke): request every registered docs route on the real Functions host - #829

Merged
yeongseon merged 3 commits into
mainfrom
test/host-smoke-requests-every-route
Oct 24, 2026
Merged

yeongseon merged 3 commits into
mainfrom
test/host-smoke-requests-every-route

Conversation

@yeongseon

Copy link
Copy Markdown
Owner

Summary

The host-smoke job proved that the Functions host indexes tests/host_smoke/function_app.py, but it only ever requested three hand-written routes. The six routes the fixture registers through register_openapi_routes (JSON, YAML and docs; anonymous and protected) were never requested, so a helper handler that fails when invoked passed CI, and the only real-host test (test_real_host_protected_docs_never_echoes_key) was skipped on every lane because nothing sets HOST_SMOKE_BASE_URL. This PR requests every registered route on the real host, asserting 200 plus a body marker, and makes that test impossible to skip silently.

Changes

  • tests/host_smoke/test_fixture_routes.py: new parametrized real-host test over the 6 registered routes (registered/*, protected/*; JSON, YAML, docs). The two real-host tests carry a host_smoke marker instead of the HOST_SMOKE_BASE_URL skipif; selected without that variable they fail (KeyError) instead of skipping.
  • pyproject.toml: registers the host_smoke marker and adds not host_smoke to addopts, so the offline suite deselects the real-host tests like e2e/perf/browser (0 skips, coverage unchanged).
  • .github/workflows/ci-test.yml (host-smoke job only; job and step names, matrix, required contexts and the final host-log scan are unchanged): installs pytest pytest-cov requests into the isolated smoke venv (pytest-cov is needed to parse the --cov options in addopts), and runs HOST_SMOKE_BASE_URL=http://127.0.0.1:7071 tests/host_smoke/.venv/bin/python -m pytest tests/host_smoke -m host_smoke --no-cov -rs against the live host before it is killed. The pytest status is kept until the host log has been printed and scanned, so a failing run still shows the host's own exception, and the step then exits with pytest's exit code.
  • tests/test_ci_workflow_hardening.py: YAML-parses the host-smoke job and asserts HOST_SMOKE_BASE_URL=, -m host_smoke and the status propagation stay wired, so the real-host tests cannot go dead again.

Net diff: 4 files, +53/-5. No Makefile, tools/, src/, examples/, scripts/ or demo/ change.

Verification (everything below was executed locally; Python 3.11 unless stated, azure-functions 1.25.0, Core Tools 4.8.0, Azurite 3.35.0)

Offline suite (pytest tests/, the 3.11 x latest coverage cell, azure-functions-validation installed):

result
before (origin/main 4d3a528) 1534 passed, 1 skipped, 15 deselected, coverage 95.14%; the skip is tests/host_smoke/test_fixture_routes.py:110
after (this branch) 1535 passed, 22 deselected, 0 skipped, coverage 95.14%
dev-only env, no companion before 1504 passed, 31 skipped; after 1505 passed, 30 skipped (the 30 are the existing companion-dependent skips, none from tests/host_smoke)

make check-all (Hatch env, py3.11): exit 0 (lint-workflows, ruff, ruff format --check, mypy strict on src tests, make security-types, tests 1505 passed, 30 skipped, 22 deselected at 95.14%, Bandit).

Real Functions host: the modified Exercise OpenAPI routes on a real Functions host step script was run from the workflow YAML (bash -e) with only the substitutions a shared box forces: local azurite on free ports in a private --location instead of npx --yes azurite, func start --port <free> with every 7071 replaced, and the trailing wait "$host_pid" replaced by a SIGKILL of that run's own host because the local func ignores SIGTERM. The smoke venv was built by the unchanged install step lines (minus the global npm install).

scenario origin/main step this PR's step
healthy host CI green today exit 0, 7 passed, 5 deselected (6 runs on 3.11, 1 on 3.12)
openapi_yaml handler raises when invoked (host still indexes the app) exit 0 (false green) exit 1, 2 failed, 5 passed, assert 500 == 200 for registered/openapi.yaml and protected/openapi.yaml, host log printed, ::error::The Functions host reported indexing or invocation errors (same on 3.12)
openapi_json returns an explicit 500 (host logs Succeeded, so only the new tests can see it) not run exit 1, 2 failed, 5 passed, ::error::The real-host route tests failed (pytest exit 1)
wrong marker selector (-m host_smok) n/a exit 5 (pytest exit 5)

Outside the step: pytest tests/host_smoke -m host_smoke with HOST_SMOKE_BASE_URL unset gives 7 failed (KeyError: 'HOST_SMOKE_BASE_URL', exit 1, no skips); pointing it at a closed port gives 7 failed (ConnectionError, exit 1); --collect-only -m host_smoke selects exactly 7 tests.

Guard test: passes on this branch; fails against origin/main's workflow and against copies of this workflow with the -m host_smoke selector, the HOST_SMOKE_BASE_URL= variable or the exit "$pytest_status" propagation removed.

Bandit evidence (#817): make security is PASS {"files": 49, "loc": 8902, "errors": 0, "blocking": 0, "stale_exceptions": 0} with the 13 line-pinned LOW findings still excepted; nothing under src examples demo scripts tools changed, so the security/bandit-policy.json pins are unaffected.

Not verified locally: the GitHub-hosted run itself (npm install --global azure-functions-core-tools@4, npx --yes azurite, ubuntu-latest); see this PR's checks.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • Chore (maintenance, dependencies, CI, etc.)

Checklist

  • My code follows the project's code style
  • I have run make check-all and all checks pass (including tests and at least 95% coverage)
  • I have added tests for new functionality (if applicable)
  • I have updated documentation (if applicable) - not applicable: test and CI wiring only, no documented command or user-facing behavior changes
  • My changes do not introduce new warnings (the suite reports the same 5 warnings before and after)

Related Issues

Closes #824
Refs #811

Review Evidence

  • Review source and reviewed commit SHA: none yet; author verification only, at head 6432b2e
  • Material findings and disposition (fixed / acknowledged-wont-fix / deferred-with-issue), or "none": none
  • Unresolved blockers, or "none": none
  • R3 (workflow permissions, release/publishing, Swagger UI credential/CSP handling, CLI import trust, review-policy files): no - only the host-smoke job steps in ci-test.yml change; no permissions:, token, secrets, release/publishing, Swagger UI, CLI-import or policy-file change

yeongseon and others added 3 commits October 24, 2026 14:02
…marker

Request every route the fixture registers through register_openapi_routes (JSON, YAML and docs; anonymous and protected) on a running host and assert 200 plus a body marker. The real-host tests are now selected by a host_smoke marker instead of a HOST_SMOKE_BASE_URL skipif, so they fail rather than skip when no host is given, and pyproject addopts deselect them from the offline suite (0 new skips).

Refs #824

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…host

The host-smoke job only proved that the Functions host indexes the fixture and answered three hand-written routes, so a helper handler that fails when invoked passed CI. Install pytest, pytest-cov and requests into the isolated smoke venv and run the host_smoke tests against the live host before it is stopped. The pytest status is kept until the host log has been printed and scanned, and the step still fails with pytest's own exit code (5 for a wrong marker selector).

A workflow hardening test keeps HOST_SMOKE_BASE_URL, the -m host_smoke selector and the status propagation wired, so the real-host tests cannot go dead again.

Refs #824

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@yeongseon

Copy link
Copy Markdown
Owner Author

R3 authorization for head 48fa342e89a5de513ede7d315035db3911881a21.

The maintainer delegated the merge decision to an Oracle review. The Oracle (AI) reviewed this change at head 6432b2ea78c019494e787d0e29d2cfa70f059368 through code inspection and independent probes and returned GO (healthy host 7 passed; handler exception and HTTP 500 both exit 1; missing env fails not skips; wrong marker exit 5; 95.14% coverage; Bandit pins unchanged). Changes since that head are merges of main only; required checks were re-run on this exact head and are green. This is advisory AI evidence, not line-by-line human review or human approval. Closes #824. The reviewer verified that re-introducing the v0.30.0/0.30.1 _req bug now fails the host smoke; candidate-wheel and Azure route checks stay out of scope.

@yeongseon
yeongseon merged commit 9127b17 into main Oct 24, 2026
28 checks passed
@yeongseon
yeongseon deleted the test/host-smoke-requests-every-route branch October 24, 2026 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(host-smoke): request every route the smoke fixture registers

1 participant