Runtime annotation introspection raises in tests/workflow_contracts

[leynos]

tests/workflow_contracts annotates several functions with names imported only under typing.TYPE_CHECKING. Introspecting those annotations at runtime raises NameError. This records the position reached on #728 and #729, where reviewers asked for a change on both, so the next reviewer does not reopen it from scratch.

The position

Four facts, each measured rather than assumed.

The annotations do not fail at import. PYTHON_BASELINE is 3.14, where PEP 649 defers annotation evaluation. Both affected modules import, and the workflow-contract suite that imports them passes:

$ python3 -c "import whole_run_ordering, nextest_budgets"
both modules import cleanly under 3.14.3

The future import is forbidden here. The house lint refuses it by name:

tests/workflow_contracts/whole_run_ordering.py:13:0: C9112: Remove
'from __future__ import annotations' on a 3.14+ baseline
(redundant-future-annotations)

No module in this repository uses it, which is consistent with the rule rather than an oversight. Applying the reviewers' suggested fix reds make lint-python.

The placement is ruff's own requirement. typing-only-standard-library-import fires on a runtime import used only in annotations, so moving the import back out trades one gate failure for another.

The introspection failure is real, and older than either pull request. It is true on main today for cabc.Iterable[CoverageLane] in whole_run_ordering, from well before this work:

>>> whole_run_ordering.watchdog_required_for.__annotations__
NameError: name 'fractions' is not defined
>>> typing.get_type_hints(whole_run_ordering.watchdog_required_for)
NameError: name 'fractions' is not defined

Nothing in the repository introspects them. ty reads them statically and passes.

What would not fix it

The obvious candidate is extending lint-workflow-scripts, which loads every module under .github/scripts and whose recipe comment says that "only loading them catches a definition-time failure such as an annotation naming a TYPE_CHECKING-only import", to walk tests/ as well.

Measured on a minimal module with exactly this shape, it would not catch it:

--- runpy.run_path, the loader gate's own mechanism ---
loaded with no error
--- accessing __annotations__ ---
NameError: name 'fractions' is not defined
--- typing.get_type_hints ---
NameError: name 'fractions' is not defined

The loader's mechanism is what PEP 649 made insufficient. Worth noting separately: that makes the recipe comment stale for the 3.14 baseline. Loading a module no longer catches this class, so the gate is still useful for other definition-time failures but not for the one it names.

What would

A check that resolves the annotations rather than merely loading the module: walk each module's public functions and call typing.get_type_hints on each, which is what the two runs above show to be the discriminating operation. That is a different gate from the loader, not an extension of it, and it would fail on main today until the existing annotations are addressed, so adopting it means deciding what to do about those at the same time.

Three ways to address them, none obviously right, which is why this is an issue rather than a change:

1. Import the names at runtime and suppress typing-only-standard-library-import where they are annotation-only. Costs a suppression per module and a small import cost.
2. Quote the annotations. Keeps the lint clean and makes get_type_hints work, at the cost of the quoting being invisible to a reader as a deliberate choice.
3. Decide that runtime introspection is not a use these modules support, and write that down instead of adding a gate. They are workflow contracts read by ty and by pytest, neither of which introspects.

Not urgent

Nothing reads these annotations at runtime, every gate passes, and both #728 and #729 are unaffected in behaviour. This is a latent sharp edge in a helper plus a stale comment, recorded so the position is citable.

Raised out of the review rounds on #728 and #729, which each declined the change on the grounds above.

[leynos]

@CodeRabbit plan

[coderabbitai]

Coding Plan

Summary

Treat the ticket as a record-of-decision task, not an annotation change, because every gate passes and no reader introspects the annotations.

Adopt Option 3 in a new ADR-034: declare runtime annotation introspection unsupported for the affected tests/workflow_contracts modules, with a measurable revisit gate.

Correct the stale lint-workflow-scripts comment so it no longer claims that loading catches TYPE_CHECKING-only annotation faults under PEP 649.

Cross-reference the decision from the index, the developers' guide, the incident docstring, the affected modules, and the changelog, so the position is citable.

---

File-level changes and follow-up prompts

New file

• docs/adr-034-runtime-annotation-introspection-in-workflow-contracts.md — the decision record.

Modified files

• Makefile — correct the lint-workflow-scripts recipe comment.
• docs/contents.md — add the ADR-034 index entry.
• docs/developers-guide.md — add an ADR-034 cross-reference in the baseline/lint chain narrative.
• tests/workflow_contracts/python_shell_interpreter_test.py — update the module docstring.
• tests/workflow_contracts/whole_run_ordering.py — add the module-level non-support note.
• tests/workflow_contracts/nextest_budgets.py — add the module-level non-support note.
• CHANGELOG.md — add the entry.

Follow-up agent prompt

Author docs/adr-034-runtime-annotation-introspection-in-workflow-contracts.md.
Follow the ADR template and section order in docs/documentation-style-guide.md.
Set Status to Accepted with today's date. Record the four measured facts from
issue `#728/`#729: PEP 649 defers evaluation under the 3.14 baseline; C9112
forbids `from __future__ import annotations`; TC003 forbids runtime
annotation-only imports; `typing.get_type_hints` and `__annotations__` raise
NameError and predate both pull requests. Add an Options considered table for
the three options and the discarded loader-gate extension. Choose Option 3:
runtime annotation introspection is not a supported use of these modules. Add a
Revisit gate stating the reopening condition. Cite `#728` and `#729`.
Then correct the lint-workflow-scripts comment in the Makefile, update the
python_shell_interpreter_test.py docstring, add the module-level note to
whole_run_ordering.py and nextest_budgets.py, index the ADR in
docs/contents.md, cross-link it in docs/developers-guide.md, and add a
CHANGELOG.md entry.
Do not change any annotations, imports, or the lint-workflow-scripts recipe body.
Run `make fmt`, `make markdownlint`, and `make nixie` for the Markdown changes.
Run `make lint-python` to confirm the Makefile and module edits keep every gate green.

Design Choices

Design Choice 1: Which remediation option does the decision record adopt?

Options Considered:

1. Import the names at runtime and suppress TC003 per module. Adds a suppression and a small import cost to each module.
2. Quote the annotations. Keeps the lint clean and makes get_type_hints succeed, but the quoting reads as an accident rather than a deliberate choice.
3. Decide that runtime annotation introspection is not a supported use of these modules, and record that decision instead of adding a gate.

Chosen Option: 3

Rationale: The ticket presents three options and states that none is obviously right, which is why it is filed as an issue rather than a change. The ticket framing ("This is an issue rather than a change", "Not urgent", "recorded so the position is citable") points to recording the decision rather than mutating the annotations. ty and pytest read these modules and neither introspects annotations at runtime, so recording the non-support decision matches the measured facts and avoids per-module churn that a future reader could not distinguish from an accident.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Record the decision

This phase captures the reviewed position as a durable, citable decision record, so a later reviewer does not reopen the question from scratch. It documents the four measured facts, the three options, the chosen option, and a revisit gate.

Task 1: Author the decision record

Write a new ADR that records the position and adopts Option 3.

• Create docs/adr-034-runtime-annotation-introspection-in-workflow-contracts.md, following the ADR template and section order in docs/documentation-style-guide.md.
• Set ## Status to Accepted with the date, and add ## Date.
• In ## Context and problem statement, state the four measured facts from the ticket: the annotations import cleanly under 3.14 (PEP 649 defers evaluation); C9112 forbids from __future__ import annotations; TC003 forbids moving the imports to runtime; runtime introspection (__annotations__, typing.get_type_hints) raises NameError and predates both pull requests.
• Add an ## Options considered section with a comparison table covering the three options from the ticket, including the discarded loader-gate extension and why runpy.run_path does not catch this class under PEP 649.
• Add a ## Decision outcome section that names Option 3: runtime annotation introspection is not a supported use of whole_run_ordering.py, nextest_budgets.py, and the sibling contract modules that share the shape.
• Add a ## Revisit gate section that states the measurable condition to reopen: any code or gate that must call typing.get_type_hints or read __annotations__ on these modules at runtime.
• Add a ## Consequences section noting that the TYPE_CHECKING pattern stays, that ty and pytest are unaffected, and that no new gate is added.
• Add a ## References section citing #728 and #729 and cross-linking tests/workflow_contracts/python_shell_interpreter_test.py.

Task 2: Index and cross-link the decision record

Make the new ADR discoverable through the standard documentation index.

• Update docs/contents.md under the ## Decision records heading to add one entry for ADR-034 with a short, audience-focused description.
• Add a cross-reference to ADR-034 in docs/developers-guide.md where the PYTHON_BASELINE, Ruff target-version, Pylint py-version, and df12 C9112/R9112 chain is documented.

🤖 Prompt for AI agents

Implement Phase 1: record the reviewed decision as a durable ADR and make it
discoverable.

- Create
`docs/adr-034-runtime-annotation-introspection-in-workflow-contracts.md`. Follow
the ADR template and section order set out in
`docs/documentation-style-guide.md`.
- Set `## Status` to `Accepted` with the date, and add a `## Date` section.
- In `## Context and problem statement`, state all four measured facts: the
annotations import cleanly under Python 3.14 because PEP 649 defers evaluation;
`C9112` forbids `from __future__ import annotations`; `TC003` forbids moving the
imports to runtime; runtime introspection via `__annotations__` or
`typing.get_type_hints` raises `NameError` and predates both pull requests `#728`
and `#729`.
- Add `## Options considered` with a comparison table covering all three options
from the ticket, and explain why the discarded loader-gate extension using
`runpy.run_path` does not catch this class of fault under PEP 649.
- Add `## Decision outcome` naming Option 3 as the adopted choice: runtime
annotation introspection is not a supported use of `whole_run_ordering.py`,
`nextest_budgets.py`, and sibling contract modules that share the shape.
- Add `## Revisit gate` stating the measurable reopening condition: any code or
gate that must call `typing.get_type_hints` or read `__annotations__` on these
modules at runtime.
- Add `## Consequences` noting that the `TYPE_CHECKING` pattern stays unchanged,
that `ty` and pytest are unaffected, and that no new gate is introduced.
- Add `## References` citing `#728` and `#729`, and cross-link
`tests/workflow_contracts/python_shell_interpreter_test.py`.
- Update `docs/contents.md` under `## Decision records` to add one entry for
ADR-034 with a short, audience-focused description.
- Add a cross-reference to ADR-034 in `docs/developers-guide.md` at the point
where the `PYTHON_BASELINE`, Ruff `target-version`, Pylint `py-version`, and
df12 `C9112`/`R9112` chain is documented.

Phase 2: Correct the stale gate comment and make the decision discoverable in code

This phase aligns the code and its comments with the recorded decision. It fixes the misleading lint-workflow-scripts comment and points readers of the affected modules to the ADR.

Task 1: Correct the `lint-workflow-scripts` recipe comment

Remove the stale claim that loading a module catches a TYPE_CHECKING-only annotation fault.

• Edit the recipe comment block of lint-workflow-scripts in Makefile (the comment lines that begin "Ruff and ty read the modules; only loading them catches a definition-time failure...").
• State that under the 3.14 baseline PEP 649 defers annotation evaluation, so loading a module no longer catches an annotation that names a TYPE_CHECKING-only import.
• State that the gate remains useful for other definition-time failures.
• Cross-reference ADR-034 in the comment.
• Do not change the recipe body; keep the runpy.run_path loop unchanged.

Task 2: Update the incident narrative docstring

Align the incident narrative with the recorded decision.

• Update the module docstring of tests/workflow_contracts/python_shell_interpreter_test.py.
• Note that under the 3.14 baseline the annotations no longer raise at load time, that runtime introspection still raises, and that ADR-034 records the decision not to support runtime introspection for these modules.
• Do not change the assertions or the interpreter-baseline invariants.

Task 3: Make the decision visible at the affected modules

Record the non-support decision at the point of the code so a reader does not treat the pattern as an oversight.

• Add a short module-level note to tests/workflow_contracts/whole_run_ordering.py and tests/workflow_contracts/nextest_budgets.py.
• State that these modules are read by ty and by pytest, that they do not support runtime annotation introspection, and that ADR-034 records the reason.
• Keep the if typ.TYPE_CHECKING: import blocks and the existing annotations unchanged.

Task 4: Record the change in the changelog

Record the decision and the comment correction for release history.

• Add an entry to CHANGELOG.md referencing ADR-034 and the corrected lint-workflow-scripts comment.

🤖 Prompt for AI agents

Implement Phase 2: align code comments and docstrings with the ADR-034 decision,
without changing any annotations, imports, or the lint-workflow-scripts recipe
body.

- In `Makefile`, edit the recipe comment block of `lint-workflow-scripts` (the
comment lines beginning "Ruff and ty read the modules; only loading them catches
a definition-time failure..."). State that under the 3.14 baseline, PEP 649
defers annotation evaluation, so loading a module no longer catches an
annotation that names a `TYPE_CHECKING`-only import. State that the gate remains
useful for other definition-time failures. Cross-reference ADR-034. Keep the
`runpy.run_path` loop unchanged.
- Update the module docstring of
`tests/workflow_contracts/python_shell_interpreter_test.py`. Note that under the
3.14 baseline the annotations no longer raise at load time, that runtime
introspection still raises, and that ADR-034 records the decision not to support
runtime introspection for these modules. Do not change the assertions or the
interpreter-baseline invariants.
- Add a short module-level note to
`tests/workflow_contracts/whole_run_ordering.py` and
`tests/workflow_contracts/nextest_budgets.py`. State that these modules are read
by `ty` and by pytest, that they do not support runtime annotation
introspection, and that ADR-034 records the reason. Keep the `if
typ.TYPE_CHECKING:` import blocks and existing annotations unchanged.
- Add an entry to `CHANGELOG.md` referencing ADR-034 and the corrected
`lint-workflow-scripts` comment.

Research

tests/workflow_contracts/ holds pure-logic helper modules and their pytest contracts. whole_run_ordering.py and nextest_budgets.py import fractions, collections.abc as cabc, and CoverageLane only under if typ.TYPE_CHECKING:, then name those symbols directly in function signatures.

The repository pins PYTHON_BASELINE = 3.14 in the Makefile, target-version = "py314" for Ruff, and py-version = "3.14" for Pylint. The external df12-python-lints plugin enforces C9112 (redundant-future-annotations), which forbids from __future__ import annotations on this baseline. Ruff's TC family enforces typing-only-standard-library-import (TC003), which forbids annotation-only imports at runtime.

The Makefile lint-workflow-scripts target loads every .github/scripts/*.py module through runpy.run_path, with a comment claiming this catches annotations that name TYPE_CHECKING-only imports; under PEP 649 that claim no longer holds.

Decision records live in docs/, follow the template in docs/documentation-style-guide.md, and are indexed in docs/contents.md. The highest existing ADR number is 033.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: record the reviewed decision as a durable ADR and make it
discoverable.

- Create
`docs/adr-034-runtime-annotation-introspection-in-workflow-contracts.md`. Follow
the ADR template and section order set out in
`docs/documentation-style-guide.md`.
- Set `## Status` to `Accepted` with the date, and add a `## Date` section.
- In `## Context and problem statement`, state all four measured facts: the
annotations import cleanly under Python 3.14 because PEP 649 defers evaluation;
`C9112` forbids `from __future__ import annotations`; `TC003` forbids moving the
imports to runtime; runtime introspection via `__annotations__` or
`typing.get_type_hints` raises `NameError` and predates both pull requests `#728`
and `#729`.
- Add `## Options considered` with a comparison table covering all three options
from the ticket, and explain why the discarded loader-gate extension using
`runpy.run_path` does not catch this class of fault under PEP 649.
- Add `## Decision outcome` naming Option 3 as the adopted choice: runtime
annotation introspection is not a supported use of `whole_run_ordering.py`,
`nextest_budgets.py`, and sibling contract modules that share the shape.
- Add `## Revisit gate` stating the measurable reopening condition: any code or
gate that must call `typing.get_type_hints` or read `__annotations__` on these
modules at runtime.
- Add `## Consequences` noting that the `TYPE_CHECKING` pattern stays unchanged,
that `ty` and pytest are unaffected, and that no new gate is introduced.
- Add `## References` citing `#728` and `#729`, and cross-link
`tests/workflow_contracts/python_shell_interpreter_test.py`.
- Update `docs/contents.md` under `## Decision records` to add one entry for
ADR-034 with a short, audience-focused description.
- Add a cross-reference to ADR-034 in `docs/developers-guide.md` at the point
where the `PYTHON_BASELINE`, Ruff `target-version`, Pylint `py-version`, and
df12 `C9112`/`R9112` chain is documented.
===============================================================================

Task: 2

Implement Phase 2: align code comments and docstrings with the ADR-034 decision,
without changing any annotations, imports, or the lint-workflow-scripts recipe
body.

- In `Makefile`, edit the recipe comment block of `lint-workflow-scripts` (the
comment lines beginning "Ruff and ty read the modules; only loading them catches
a definition-time failure..."). State that under the 3.14 baseline, PEP 649
defers annotation evaluation, so loading a module no longer catches an
annotation that names a `TYPE_CHECKING`-only import. State that the gate remains
useful for other definition-time failures. Cross-reference ADR-034. Keep the
`runpy.run_path` loop unchanged.
- Update the module docstring of
`tests/workflow_contracts/python_shell_interpreter_test.py`. Note that under the
3.14 baseline the annotations no longer raise at load time, that runtime
introspection still raises, and that ADR-034 records the decision not to support
runtime introspection for these modules. Do not change the assertions or the
interpreter-baseline invariants.
- Add a short module-level note to
`tests/workflow_contracts/whole_run_ordering.py` and
`tests/workflow_contracts/nextest_budgets.py`. State that these modules are read
by `ty` and by pytest, that they do not support runtime annotation
introspection, and that ADR-034 records the reason. Keep the `if
typ.TYPE_CHECKING:` import blocks and existing annotations unchanged.
- Add an entry to `CHANGELOG.md` referencing ADR-034 and the corrected
`lint-workflow-scripts` comment.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---