Skip to content

Correct four release claims and fix two defects - #303

Merged
chris-colinsky merged 2 commits into
mainfrom
fix/release-review-claims-and-two-defects
Sep 26, 2026
Merged

chris-colinsky merged 2 commits into
mainfrom
fix/release-review-claims-and-two-defects

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Acts on spec's release review of v0.16.0..HEAD (coord release-v0.17.0/62), which found three things blocking the tag. This closes the first and takes the two code fixes spec recommended doing "while it is cheap". Blockers 2 and 3 are deliberately deferred, with reasoning at the bottom.

None of the four claims was a regression. Each was a statement about the release that was not true.

Four claims

The CHANGELOG was dated 2026-07-20. 67 days stale, and unchanged across ~70 commits including the one titled "Bring the CHANGELOG current before tagging". Now 2026-09-25, and RELEASING.md already requires re-confirming it on tag day.

The CHANGELOG contradicted itself about 0124. Line 40 said not-yet, line 61 said partial, the manifest said partial. The summary bullet predated the adoption that landed in HEAD.

0085 was implemented with half of it absent. Nothing writes enclosing_fan_out_lineage; compiled.py:372 says so in its own comment. Verified: zero write sites in src/. The consequence is narrower than "broken" and wider than "an optimization is missing":

  • §10.11's no-mis-skip floor holds, because an unmatched record re-runs rather than skipping.
  • §10.11.1's exactly-once guarantee does not extend to nested fan-outs, because a completed inner instance re-runs on resume and its side effects execute twice.

Fixture 076 passes against a record the test seeds by hand, not one the engine wrote. Now partial. The old note defended the gap by citing "§66 / §76", which do not exist in 0085; its actual Out of scope section lists three items and the write side is not among them.

0124's note named two unmet arms; there are three, and a fourth is retracted. Now listed worst first:

  1. The resolver still keys on span openness, which §5.5 forbids outright, and no fixture can reach that check because the conformance adapter attaches instance or branch middleware where §5.1 specifies node middleware.
  2. No subgraph span is synthesized on either observer, so an orphan inside a plain subgraph parents under the invocation span rather than the subgraph span.
  3. The detached arms stay unmirrored on both observers.

The retracted one was a callable parallel branch on Langfuse. §8.4.8 makes that branch's single observation its per-branch dispatch observation, so there is no leaf-versus-dispatch race to lose, and the parent is stable with and without a yield (reproduced both ways). It reached a shipped manifest note from a review finding that was never reproduced. The residual callable-branch gap is real but narrower: that observation omits parallel_branches_parent_node_name, which §8.4.2 requires. Tracked, not fixed here.

Two defects, both public API

StructuredOutputInvalid used the wrong field names. It carried raw_content and failure_description where llm-provider §7 names output_content and error_message, and §7.1 says the reask builder receives those two. So a caller writing the documented builder hit AttributeError, openai.py:803 converted it to raise exc from reask_error, and the call failed on the first invalid output exactly as if no builder had been supplied. Reask did not work as documented.

Renamed across 62 references in 7 files. This is a pre-1.0 break for a caller reading either attribute. The conformance harness carried an alias bridging the two spellings, with a comment saying it would stay correct if the implementation ever renamed to match; it has, so the alias resolved to itself and is removed.

A per-attempt retry override replaced the extras container. §7.1: "undeclared extras (§6) merge by the same per-key rule." The analogue of a declared field is an extras KEY rather than the container, so a key the override sets replaces that key and a key it does not mention inherits the base's. The implementation replaced the whole container and logged the base keys it dropped, so an override adjusting one vendor knob silently withdrew every other knob the caller had set for that attempt. Nothing is dropped now, so the warning goes with it.

test_per_attempt_override_with_extras_replaces_and_warns asserted the defect. It is inverted and renamed, and now asserts the inherited keys, which is the half that can fail: the override's own key arrives under either reading, so asserting only that would pass against the old behaviour.

Evidence

Mutation Result
extras reverted to replace-the-container killed (the inverted test)
harness carries-lookup returns None killed (conformance 022, 023)

2301 passing, ruff and pyright clean, manifest consistent.

Deferred, and why

Blocker 2 (the resolver's openness check). Pre-existing: orphan parentage depended on scheduling before 0124 too. 0124 already ships partial, and the note now names this as the primary unmet arm rather than a footnote. The fix also needs the adapter attachment changed before any fixture can hold it, which makes it two pieces of work rather than one.

Blocker 3 (fixtures that run and assert nothing, and extending the 0120 vocabulary check one level down into expected: / resume: / nodes:). Spec calls this the highest-leverage item in the review and I agree. It is also not a regression, it is substantial, and the last three review rounds on #302 showed what happens when work of this size is done under tag pressure. It wants its own pass.

Both are tracked locally with spec's measurements attached.

Also noted, not fixed

scripts/check_conformance_manifest.py validates that every accepted proposal has an entry and that status is one of four strings. Nothing compares a status to the code, which is how 0085 shipped implemented over a stub and passed cleanly. That is the same green-is-the-null-result failure AGENTS.md warns about for fixtures, one level up. Worth a guard; not one to build under tag pressure.

From spec's release review. None of the four claims was a regression;
each was a statement about the release that was not true.

The CHANGELOG was dated 2026-07-20, 67 days stale and unchanged across
the commit titled "Bring the CHANGELOG current before tagging". It also
said 0124 was `not-yet` twenty-one lines before saying it was `partial`.

0085 was `implemented` with half of it absent: nothing writes
`enclosing_fan_out_lineage`, as compiled.py's own comment says, so
section 10.11's no-mis-skip floor holds but section 10.11.1's
exactly-once guarantee does not extend to nested fan-outs. A completed
inner instance re-runs and its side effects execute twice. Fixture 076
passes against a record the test seeds by hand. Now `partial`, and the
note states what is met rather than citing two sections of 0085 that do
not exist.

0124's note named two unmet arms. It now names three, worst first: the
resolver still keys on span openness, which section 5.5 forbids and
which no fixture can reach because the adapter attaches instance or
branch middleware where section 5.1 specifies node middleware; no
subgraph span is synthesized on either observer; and the detached arms
stay unmirrored. A fourth arm, a callable branch on Langfuse, is
RETRACTED: section 8.4.8 makes that branch's single observation its
dispatch observation, so there is no race to lose, and the parent is
stable with and without a yield. It came from a review finding that was
never reproduced before being written into a shipped note.

Two defects, both public API.

`StructuredOutputInvalid` carried `raw_content` and
`failure_description` where llm-provider section 7 names
`output_content` and `error_message`, which section 7.1 says the reask
builder receives. A caller writing the documented builder hit
AttributeError, the loop converted it to `raise exc from reask_error`,
and the call failed on the first invalid output as if no builder had
been supplied. Renamed; a pre-1.0 break for anyone reading the two
attributes. The conformance harness alias bridging the spellings is
removed as dead.

A per-attempt retry override replaced the extras container where
section 7.1 says undeclared extras merge by the same per-key rule. An
override adjusting one vendor knob silently withdrew every other knob
the caller had set for that attempt. The test asserting the wrong
behaviour is inverted, and now asserts the inherited keys, which is the
half that can fail.
Copilot AI lite review requested due to automatic review settings September 26, 2026 01:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved changelog, manifest, documentation, and test-helper inconsistencies remain.

Review effort: Lite
Findings: 3 Low severity

Open (3)
What changed in this PR

Corrects v0.17.0 release claims and fixes structured-output error naming and retry extras merging.

Changes:

  • Aligns release metadata and proposal statuses.
  • Renames structured-output error fields to the documented API.
  • Merges retry extras per key and updates tests and documentation.
File Summary
tests/​unit/​test_structured_output.py Updates validation assertions for renamed fields.
tests/​unit/​test_observability_otel.py Updates reask assertions; simplify the obsolete identity alias map. Nit (1 vote).
tests/​unit/​test_llm_provider.py Strengthens retry extras regression coverage.
tests/​conformance/​harness/​wire.py Removes legacy aliasing; update the carry assertion docstring. Nit (4 votes).
src/​openarmature/​llm/​retry.py Updates retry terminology; document per-key extras merging. Nit (1 vote).
src/​openarmature/​llm/​providers/​openai.py Applies renamed fields and per-key extras merging; update stale docstrings. Nit (3 votes).
src/​openarmature/​llm/​errors.py Renames public exception fields.
docs/​concepts/​llms.md Updates API examples; correct grammar and describe per-key merging. Nits (4 and 1 votes).
conformance.toml Corrects proposal statuses; update the stale alias note. Nit (1 vote).
CHANGELOG.md Updates release claims; resolve contradictory status, reask, and extras-merging notes. Nits (1 vote each).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/concepts/llms.md Outdated
Comment thread src/openarmature/llm/providers/openai.py
Comment thread tests/conformance/harness/wire.py
All three from review, all three documentation left asserting what the
code no longer does.

The field rename was a mechanical string replacement, which turned "a
failure_description" into "a error_message" in published concept docs.
Swept the renamed files for any other article broken the same way; this
was the only one.

Two docstrings still said a non-empty extras container replaces
wholesale: `_config_for_attempt`'s and `LlmRetryConfig`'s. The second is
the damaging one, since LlmRetryConfig is public API and that text is
what a caller reads before writing an override, so it stated the
opposite of what the call now does. The inline comment beside the code
was updated and these were not.

`assert_error_carries`'s docstring documented three attributes that no
longer resolve, now that the alias table is gone and the carries key
resolves straight to an attribute. Two of the three were broken by the
rename; `raw_response_content` was already stale, retired from the
fixtures by 0098 and surviving only in that text.
@chris-colinsky
chris-colinsky merged commit 19e99ba into main Sep 26, 2026
6 checks passed
@chris-colinsky
chris-colinsky deleted the fix/release-review-claims-and-two-defects branch September 26, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants