[masonc08]

What did you change, and why?

CUJ2's Claude and Codex tests required models to copy a random ID, making successful inference fail the test when models changed that text. They now ask for a greeting and require a completed parent turn with a nonempty answer, while retaining prompt correlation and provider, model, and HTTP checks.

The existing direct Anthropic MPS test also validates a complete assistant Messages response. File, skill, MCP, and subagent output checks remain intact.

How do you know it works?

• All 197 focused evidence and integration-contract tests passed, including negative checks for empty, incomplete, unrelated, and child-session answers and preserved file-value requirements.
• All three existing CUJ2 cases collect successfully.
• Ruff lint/format and Ty passed.
• Initial full CI passed; checks are rerunning for the review updates.

[github-actions]

Polly AI Review

Blocking issues

None found.

Security vulnerabilities

None identified.

Non-blocking notes

• tests/test_cuj_evidence.py:150: can we retain one successful greeting per agent? “Hello!” and “Hi there.” exercise the same acceptance path.

Approach

The approach is sound. Reusing SessionEvidence removes parallel transcript readers while preserving prompt, parent-session, and turn correlation. Request-model and HTTP checks remain separate from answer-content checks.

Summary

Diff size (computed): 7 files; +82 / -53 lines (135 changed text lines); 0 binary files.

Category	Files	Added	Deleted	Share of changed text lines
Tests and test support	7	82	53	100.0%
Documentation	0	0	0	0.0%
Dependencies and lockfiles	0	0	0	0.0%
Implementation / other	0	0	0	0.0%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

The PR removes fixed-text reproduction from inference smoke-test success criteria. It updates shared evidence helpers, switches CUJ2 to those helpers, strengthens direct Anthropic Messages validation, and updates coverage documentation. No production code or dependencies change.

The size is proportional to the outcome. Replacing scenario-specific helpers avoids unnecessary scaffolding; no materially simpler implementation is apparent.

Tests

Test-by-test assessment

The offline cases exercise acceptance and rejection boundaries at the unit layer. Existing coverage also addresses prior-session exclusion, duplicate prompts, incorrect Codex turns, and HTTP/model/schema failures. No test functions are removed, and no concrete missing regression case was established.

Test / case	Behavior protected	Layer	Needed?	Action / rationale
tests/test_cuj_evidence.py::test_cuj_inference_requires_nonempty_final_answer — both agents, successful greetings	Nonempty answers accepted without an expected value	Unit	consolidate	One successful greeting per agent provides the same coverage.
tests/test_cuj_evidence.py::test_cuj_inference_requires_nonempty_final_answer — both agents, empty and whitespace answers	Empty or whitespace-only answers rejected	Unit	keep	Exercises empty content and whitespace normalization directly.
tests/test_cuj_evidence.py::test_cuj_file_task_still_requires_hidden_value — both agents	File tasks still require their expected value	Unit	keep	Guards against unintentionally relaxing other task types.
tests/test_cuj_evidence.py::test_cuj_evidence_rejects_false_positives — no_prompt, file/inference, both agents	Prompt correlation remains mandatory	Unit	keep	Extends the existing rejection check to optional-value mode.
tests/test_cuj_evidence.py::test_cuj_evidence_rejects_false_positives — incomplete, file/inference, both agents	Missing final evidence rejected	Unit	keep	Protects completion handling after removing the text-match requirement.
tests/test_cuj_evidence.py::test_cuj_evidence_rejects_false_positives — child, file/inference, both agents	Child-session evidence cannot satisfy the parent task	Unit	keep	Tests the session boundary without another live journey.
tests/test_cuj_evidence.py::test_cuj_evidence_rejects_false_positives — unsupported, file/inference, both agents	Expected model belongs to the supplied supported set	Unit	keep	Preserves model-membership checks within the existing compact matrix.
tests/test_cuj_evidence.py::test_cuj_evidence_codex_rejects_wrong_turn — file/inference	Completion belongs to the target turn	Unit	keep	Protects turn correlation independently of answer content.
tests/e2e_cuj/helpers/evidence.py — ClaudeCujHelper._completed_turn, CodexCujHelper._completed_turn, and completed_turn documentation	Optional expected value with mandatory nonempty answer	Shared test helper	keep	Minimal shared extension; correlation and expected-value checks remain covered.
tests/e2e_cuj/test_cuj2_mps_mcp.py — _agent_prompt	Greeting request retains its correlation marker	Scenario helper	keep	Separates request identification from model-generated answer content.
tests/e2e_cuj/test_cuj2_mps_mcp.py — removed _parent_transcripts, _task_complete, _assert_task_evidence	Replaces marker-based transcript scanning	Scenario helpers	remove	Superseded by shared evidence handling; removing answer-marker matching is intentional.
tests/e2e_cuj/test_cuj2_mps_mcp.py::TestCuj2CodexInference::test_cuj_codex_inference	Real Codex parent completion and MPS request/response evidence	E2E	keep	Verifies the agent/process/provider boundary that offline evidence tests cannot establish.
tests/e2e_cuj/test_cuj2_mps_mcp.py::TestCuj2ClaudeInference::test_cuj_claude_inference	Real Claude parent answer and MPS request/response evidence	E2E	keep	Covers the distinct Claude process and protocol boundary.
tests/test_e2e.py::TestAnthropicNonRelayMps::test_direct_inference_routes_to_haiku	Complete assistant Messages response from the expected model	Live HTTP integration	keep	Strengthens an existing direct-MPS case without adding another live scenario.
tests/README.md — documentation edits	Updated coverage description	Documentation	keep	Records the revised inference and direct-Messages contracts.
tests/e2e_cuj/README.md — documentation edits	CUJ2 success criteria	Documentation	keep	Explains request correlation separately from answer content.
tests/integration/README.md — documentation edits	Integration evidence contract	Documentation	keep	Documents preserved expected-value requirements for other task types.

Execution was not independently performed. The PR reports 199 focused tests passing, three CUJ2 cases collecting, and Ruff/Ty passing; live validation remains pending CI.

To verify offline, run uv run pytest tests/test_cuj_evidence.py tests/test_integration_contract.py. For live verification, inspect the CUJ2 CI artifacts for completed, prompt-matched parent answers and paired successful MPS responses.

Scope

All changes support the stated inference-validation outcome. Strengthening the existing direct Anthropic response check serves the same goal. No unrelated changes or unresolved scope questions were found.

---

_{Automated review by Polly · workflow run}