Repository navigation
Add reask and extras examples, and ship the manifest an agent is told to read - #304
Merged
Merged
Conversation
Two v0.17.0 features had no example. Both demonstrate behaviour verified against a running provider rather than read off the spec. structured-output-reask extracts one mission record per lunar-landing report and corrects the model when it answers in the wrong shape. It could not have been written before the field rename in the previous commit: a builder reading the documented `output_content` and `error_message` raised AttributeError, which the retry loop turned into the original failure. A `MODE=off` arm omits the builder so the contrast is visible. provider-extras reaches two OpenAI request fields openarmature does not model, and shows the three things extras refuses. The first draft used `seed` as the vendor knob; `seed` is managed and would have rejected, so the example would have shipped demonstrating a failure as the feature. Probing the arms first caught it. Two nuances are documented rather than demonstrated: a matching extras value is a no-op rather than an error, and a sampling field left unset is not managed on that call, so managed means produced-by-this-call rather than nameable. The examples smoke test enumerated its demos with nothing checking the list against the directory, so a new example was covered by nothing until someone remembered to add it. It now derives the set from disk and compares. `tests/unit/test_structured_output.py` gains a test that writes the reask builder the documented way. Every existing test read the implementation's own spelling or went through the conformance harness, which aliased the two names, so none of them could fail on the rename bug. A consistent revert across the three source files reproduces the original AttributeError inside the builder.
Both from an agent adopting the package, who found the manifest missing from the wheel and asked whether the duplicated pattern files were deliberate. The bundled AGENTS.md tells an agent to read conformance.toml for per-proposal implementation status, which is the only artifact answering "is this real in the version I have installed" for a behaviour with no importable name. The file was not in the wheel, and the pointer said "at the repo root", so the instruction resolved for someone in a clone and for nobody who installed from PyPI. That asymmetry is why it survived: the sdist had it all along, so working from a clone or an sdist build never hits it. It is force-included at openarmature/conformance.toml now, and the pointer names the importlib.resources path. Verified through the whole chain, including a wheel built from the sdist. It matters more this release than last: 0085 and 0124 both ship partial, and their notes are where which-half-is-missing lives. Force-included rather than copied into src/ the way AGENTS.md is, because a second committed 128K file would appear in every diff that touches the manifest. The cost is that the guard checks config shape rather than a built artifact. The pattern duplication is deliberate and the generator says why: the standalone files back openarmature.patterns.list() / get(name) and use a different transform so each resolves its own cross-references. AGENTS.md now says so, since the alternative is every reader asking. Separately, found while checking the sdist: hatch packages what is on disk and does not read .git/info/exclude, so _tasks/ was being collected for publication. Those are internal working notes. Checked against PyPI: no published release contains them, so this is preventive. The pinned spec submodule is excluded in the same change, 1124 files in 0.16.0 and most of the tarball, taking it from 3.4MB to 1.4MB. tests/, examples/ and docs/ stay in, and the guard asserts that so a later tidy-up has to change the reasoning with it.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved moderate issues remain in cleanup handling, source and artifact packaging behavior, and artifact-level test coverage.
Review effort: Lite
Findings: 1
Open (2)
What changed in this PR
Adds adopter-facing reask and provider-extras examples, updates agent guidance, and improves wheel/sdist packaging.
Changes:
- Adds two runnable examples with documentation and regression coverage.
- Ships
conformance.tomlin wheels and updates generated agent guidance. - Excludes private notes and the spec submodule from sdists.
| File | Summary |
|---|---|
tests/unit/test_structured_output.py |
Tests documented reask fields and recovery. |
tests/test_smoke.py |
Guards manifest packaging and sdist configuration. |
tests/test_examples_smoke.py |
Ensures all examples are registered. |
src/openarmature/AGENTS.md |
Updates manifest and pattern guidance. |
scripts/build_agents_md.py |
Updates generated documentation. |
pyproject.toml |
Configures distribution inclusion and exclusions. |
examples/structured-output-reask/main.py |
Adds structured-output recovery example. |
examples/README.md |
Documents the new examples. |
examples/provider-extras/main.py |
Adds provider extras demonstration. |
CHANGELOG.md |
Records release changes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Two from review, both mine. The manifest pointer named only the packaged path, which resolves for an installed package and not in a clone: force-include is build-time, so nothing lands under src/. Verified by running it, `is_file()` is False here. That is the same defect the change set out to fix, with the polarity flipped, since the original said "at the repo root" and was unresolvable from a venv. It now names both and says which is which. The guard asserted the pointer was NOT repo-relative, which stopped being the defect once the clone path became correct to mention. It now asserts both paths are named, and catches either one going missing. The provider-extras run block asked for an API key the demo never reads, and then told the reader to drop a `--stub` flag that does not exist. The example parses no arguments and reads no environment variables at all, so both sentences described a program that was not there. Rewritten to say it needs no credentials, why a stub is the right thing for a demo about the outbound request body, and the one real edit that points it at a live endpoint. Swept the sibling example: every env var its run block names is actually read, so the defect was local to this one.
The demo turned on a model sending "1,340 kg" where the schema wanted a number. A capable model does not make that mistake, and an endpoint that enforces the schema on the wire cannot: run against a local vllm with guided decoding and MODE=off succeeded identically to the reask arm, so the feature the example exists for never fired. An output ceiling below what a complete record needs truncates the reply every time, on any model, and arrives at the schema boundary as the same structured_output_invalid. Recovery then needs two different things, which is the more useful lesson: the corrective message says what went wrong, and a per-attempt override raises the ceiling so the retry has somewhere to put the answer. A third mode supplies the correction without the raised ceiling, so the difference is visible rather than asserted. The builder branches on finish_reason rather than guessing from the raw text, since that is the signal the framework already reports. Also stop the cleanup path constructing a provider in order to close one, which masked whatever failure kept the run from building it.
Reading llm-provider 7.1's per-key merge correctly introduced a regression that no test caught. A base extras key is the section 6 escape hatch: unmanaged where it was written, because the base leaves the declared field unset so the mapping emits nothing of that name. Merging carried it onto a retry whose override does set the field, where it is managed, so section 8.1 rejected the call before it went out. Measured against a transport returning 503 on every call with max_attempts=3: base extras temperature plus an override declaring temperature ran one wire call and surfaced provider_invalid_request, where the replaced-container rule ran three and surfaced the transient the retry existed to ride out. The original error was not even chained, because the failure happens in the body build outside the attempt's try. A field the override declares now supersedes an inherited base extras key of the same name. The merge must not manufacture an invalid config out of two valid ones, and the override's field is the later and more specific instruction. A key the override names alongside its own declared field of that name still collides, since that config contradicts itself in a single object. Also correct claims that were wrong or went stale, several of them in the commit that was fixing other wrong claims: - The CHANGELOG said 0085 extends exactly-once to nested fan-outs while the manifest said it does not. Nothing writes the lineage onto a crash record, so a completed inner instance re-runs and its side effects happen twice. The manifest note asserted both readings itself. - The manifest called the old "66 / 76" citation nonexistent. Those were line numbers written as section marks, and they point at the write-side out-of-scope paragraph and the parallel-branches deferral. The reference was real; only the notation was wrong. - docs/concepts/llms.md and the 0122 CHANGELOG entry still documented wholesale replacement and a warning that no longer exists. An override may set an extras key to null, which sends null rather than withdrawing the key, so the previous claim that withdrawal is inexpressible overstated it. - The 0095 entry still named raw_content and failure_description. - The 0098 note still described the alias map that the rename deleted. A reask builder that raises now logs which of the two failed, since the chained cause reads as a bad model rather than a broken builder. That also gives the module's orphaned logger a consumer.
The hand-maintained orientation file had drifted far enough to send an agent to the wrong places. Verified each claim against the source rather than against the file. - It listed src/openarmature/middleware/, which does not exist. The middleware package is under graph/. - It claimed a filesystem checkpoint backend. The checkpoint backends are in-memory and SQLite; filesystem is a prompt backend, which is the likely source of the mix-up. - prompts/ and retrieval/ were absent entirely, along with every provider in them. - Langfuse appeared nowhere, neither the observer nor the prompt backend. - It said three places hold the spec version and that the third is enforced by convention. There are four, conformance.toml's spec_pin being the one that drifted silently before it was covered, and the submodule is checked by reading the spec changelog at the pinned commit. - test_smoke.py was described as version sync, which is now about half of what it guards. Also note that the bundled AGENTS.md and _patterns/ are generated, since the two files sit one directory apart and only one is safe to edit.
The demo drives a token ceiling because that failure is guaranteed on any endpoint. Nothing said so, and the surrounding prose implied the wrong-shaped answer had been dropped for being unrealistic. It is the opposite: it is the failure an adopter is most likely to meet, and they meet it precisely when their serving stack is not yet configured to prevent it. Whether it fires at all is a property of the endpoint rather than of the calling code. An endpoint that enforces the schema during decoding cannot produce it; one that ignores response_format, or sits behind a proxy that drops the field, or serves a weaker model, produces it routinely. So the demo drives the failure it can guarantee and documents the one it cannot, and the builder handles both through the exception they share. Not having to know in advance which kind of endpoint you are pointed at is the value, and that is now what the example says. The concepts page gains the same distinction plus the finish_reason tell that separates the two causes, and stops naming an escalating temperature as the only interesting per-attempt override: a truncated reply needs the ceiling raised, which a correction alone cannot fix.
A base extras key is the section 6 escape hatch, unmanaged where it was written because the base leaves the declared field unset. Merging carried it onto a retry whose override did set the field, where it is managed, so section 8.1 rejected the call before it went out and the retry died on attempt 1. The fix for that was to let the override's declared field supersede the inherited key. Spec ruled against it and is right. Section 6 says a mapping must not silently drop a conflicting extras value, and moving the discard upstream of the mapping does not change what happens to the caller's value. The argument that settles it is narrower: under the supersede, a config carrying temperature as both a declared field and an extras key was rejected when a caller wrote it and accepted when a merge assembled it. One shape, two meanings, decided by history rather than content. A merged config could then not be logged, replayed or handed to another call and behave the same way, and a second implementation would have to track provenance through the merge to stay conformant. That is the defect class proposal 0124 exists to prevent, one level down. So the collision stands. What the caller does instead is keep both values in the channel the base used: a base reaching for extras gets an override that does the same, which merges per key and sends. The ergonomic half of the report is real and is fixed differently. The collision surfaces on attempt 1, after attempt 0 has already gone out on the base alone, so a broken retry config hides until the first retry. It now reports that the attempt's config merges a base with a per-attempt override, that the two values come from different channels, and which channel to change. Detecting it before attempt 0 needs every attempt's body built eagerly, so that is recorded as a follow-up rather than built.
StructuredOutputInvalid.error_message named a single violation, because JSON Schema validation raises on the first one it finds. A five-field record that broke four ways reported one of them. The field's documented consumer is a reask builder, so a partial list makes the correction iterative: the model fixes the named field, the next attempt surfaces the next one, and a call spends one round trip per wrong field. Measured against a real model behind an endpoint that accepts response_format and ignores it, extracting a five-field record took four to five attempts and did not recover at all inside the canonical max_attempts of 3. Enumerating brings the same extraction to two. Both schema forms were affected. The class path runs JSON Schema validation before Pydantic so it inherits the dict path's strictness rather than Pydantic's coercion, so it raised from the same place. Every violation is now enumerated, one per line, sorted so the same output and schema produce the same string and a caller can diff corrections across attempts. Order itself is unspecified. A parse failure stays a single violation, since no schema check can run against output that is not well formed. The field remains a human-readable string and response_schema remains a separate attribute; composing the two into a correction is the caller's job, because the implementation authors no prompt text of its own. The exception stays unbounded, because a caller needs every violation to compose one correction. An observer's emitted copy is already bounded by its payload byte cap, so a wide schema lengthens what a caller reads and not what a trace carries. The reask example sends response_schema alongside the objection, which is the caller-side half: the enumeration says what went wrong, the schema says what right looks like, and a model that invented field names was never shown the contract.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Retry merging contradicts the stated fix, schema-error enumeration mishandles older drafts, and the sdist cannot run its retained conformance tests.
Review effort: Balanced
Findings: 1
Open (3)
Resolved since last review (2)
Two defects found in review, both in work from this PR. The sdist kept tests/ so a downstream packager could validate a build from source, and excluded the pinned spec submodule to cut the tarball from 3.4MB to 1.4MB. The conformance drivers read their fixtures out of that submodule, so the suite shipped without its inputs: building the sdist and running it gives 48 failures across nine files, on an empty corpus. The exclusion now keeps the conformance directories and drops the proposals and prose around them, which takes it to 1 failure at 2.0MB. That one is unrelated and predates all of this: the AGENTS.md drift check runs the generator, which shells out to git, and an unpacked tarball is not a repository. The guard that should have caught it asserted the exclude list's text, which stayed true while its effect changed. It now builds the artifact and looks inside: fixtures present, proposals and private notes absent. Three mutations on the exclude patterns each kill it on their own assertion. Second, the schema-violation enumeration hardcoded the 2020-12 validator while the rest of the codebase selects one from the schema's own $schema. A Draft 7 schema using the tuple form of `items` raises under 2020-12, and the fallback then returned the single error the enumeration exists to replace, so error_message silently reverted for every caller declaring an older draft. The broad catch that was meant to stop enumeration introducing a new failure mode is what hid it. Every test written for the enumeration omitted $schema, which defaults to the latest draft, so nothing exercised the disagreement. A Draft 7 case now covers it, including a violation inside the tuple-form array that a 2020-12 validator cannot reach.
Two follow-ups from the review, both about a check that reports success while doing nothing. The AGENTS.md drift test ran the generator, which asks git for the spec submodule's pinned tag so a bundle cannot ship built off an untagged commit. An unpacked sdist has no repository, so the test failed there for a reason unrelated to drift, and a packager validating a build from source saw a red suite. It now skips where there is genuinely no repository. That skip is keyed on the absence of a repository rather than on the generator failing, because the wider condition would let a broken generator read as a clean CI run. A skip condition cannot be caught by the test it guards, so a separate check asserts the condition is false inside the repository, cross-checking the filesystem and PATH against a helper that shells out to git. Widening the condition fails that check instead of going unnoticed. The schema-violation enumeration falls back to the single violation it was handed when the validator cannot process the schema, so a structured_output_invalid is never replaced by something the caller cannot handle. The fallback returned a well-formed message, which is indistinguishable from a schema that genuinely broke one way, and the hardcoded draft fixed in the previous commit reverted the field for every caller on an older draft with nothing to show for it. It now says so, and names the draft the schema declared. The test drives that by making the validator lookup raise, because the point is that we do not know which schemas reach the path. The log record is the only observable: the returned message is valid either way.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
The module already imports logging at the top, so the function-local import did nothing. Flagged by both code scanners. The file's other function-local imports are all from openarmature, kept local so collection does not pull the package in at import time. That does not apply to a stdlib module already imported at module scope.
Reporting every schema violation rather than the first was not in this PR's scope. It came out of testing the reask example, became a spec question, and got built because spec said the field's contents were undefined either way and moving first risked nothing. In two days it produced four defects. A hardcoded validator draft that silently reverted the field for every older-draft schema. A false claim that the emitted copy was bounded. A reachability change in the retry path. And, found by review rather than by test, a regression: the walk stopped at the top level, so every anyOf and oneOf reported "is not valid under any of the given schemas" where the single error it replaced had named the failing leaf. Pydantic renders every Optional and every union that way, so the field got worse for the shape callers are most likely to have, and the correction a reask builder sends named no field at all. That one is cheap to fix and not the reason to stop. The reason is that every test written for the enumeration used a flat type/required schema, which is the same single-shape blind spot that let the draft bug through one commit earlier. The field flows onto events, into both observers, into a transcript fed back to a model, and into conformance substring assertions. Two wrong guesses about which shape breaks it is not a basis for a third. Deferring costs nothing: there is no published spec text yet, and the convergence problem it solved is already solved here by the example's builder sending response_schema, which is caller-side and has no blast radius. It returns next cycle with the proposal, able to descend into nested context errors from the start.
All six are in the five commits this PR added over the last two days. The drift check's skip condition asked git for a repository, and repository discovery ASCENDS. An sdist unpacked anywhere beneath a checkout, which is the ordinary case for a packaging feedstock or a vendored copy, answered for its parent: the condition read as queryable, the generator ran, and it hard-failed on a submodule that was not there. It now asks whether the spec submodule itself resolves to the submodule, and the helper takes the tree root so the condition is testable against a synthetic tree. The companion check asserted the helper rather than the resolved marker, so widening the decorator left it green; it now reads the marker, and a second test nests a synthetic sdist inside a real repository so the ascent is reproduced rather than reasoned about. The collision hint inferred "the values came from different channels" from "a merge happened". An override that declares a field and names it in its own extras collides on a merged attempt too, and that caller was sent to inspect a base config they never wrote, to match a channel that did not exist. _config_for_attempt now reports which extras keys it inherited, and the hint is chosen per colliding key: the base's channel when the key was inherited, remove-one-of-your-two-entries when it was not. The sdist guard shelled out to uv with no executable check, no timeout and no way to tell a provisioning failure from a packaging one, so it errored rather than skipped on exactly the machine the commit exists to serve. Its fixture assertion also needed only one matching yaml anywhere, so a glob narrowed to one capability left six drivers collecting nothing while it passed; it now asserts per capability, reading the list from the module that owns it, anchored on the submodule path. A test comment still described the reverted supersede as current behaviour and named a mutation that no longer exists, in the one place this repo's rules grant an exemption for exactly that note. Last, the reject now has a test recording what it costs: build_body runs outside the attempt's try, so a pre-send raise on a retry emits no attempt event and the terminal event's request_extras describe a body with no collision in it. The gap is older than this PR and stays deferred, but it is reachable for a valid pair of configs now, so it is pinned where someone will see it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two adopter-facing pieces, neither touching library behaviour. Was stacked on #303 and is now rebased onto main.
Two v0.17.0 features had no example
structured-output-reask/Extract one structured mission record per lunar-landing report, under an output-token ceiling too tight for a complete record.
The first version of this example could not demonstrate itself on a well-configured endpoint. It turned on a model sending
"1,340 kg"where the schema wanted a number. Run against a local vllm serving Qwen3-VL-30B,MODE=offsucceeded identically to the reask arm, because the provider sendsresponse_format: json_schemaand the endpoint enforces it during decoding, soStructuredOutputInvalidcannot fire. Only a hand-written stub made the arm visible.That is a fact about the endpoint, not about the feature. An endpoint configured that way is the end state of knowing what to configure, and the guard exists for everything before it: an endpoint that ignores
response_format, a proxy that drops the field, a weaker model behind either. The wrong-shaped answer is the failure an adopter is most likely to meet, and they meet it precisely when they have not yet learned the serving-side lesson. Framing it as something a good model does not do gets the audience backwards.What it does mean is that the wrong-shaped answer cannot be the demo's driver, because whether it fires is not ours to decide. An output ceiling below what a complete record needs truncates the reply every time, on any model and any endpoint, and reaches the schema boundary as the same
structured_output_invalid. So the demo drives the failure it can guarantee, documents the one it cannot, and the builder handles both through the one exception they share. That last part is the actual value: you do not have to know in advance which kind of endpoint you are pointed at.reasksupplies the corrective message. Only the caller knows how to talk to their model about their schema.per_attempt_overrideraises the ceiling. A correction cannot help a reply that gets cut off in the same place.Three postures, measured against the live endpoint:
MODEoffnocapnocapis the arm that carries the lesson. The model is told exactly what went wrong and still has nowhere to put the answer.This example could not have been written before #303. The builder reads
exc.output_contentandexc.error_message, the names llm-provider §7.1 says a builder receives and which the implementation did not have until that PR renamed them. It also branches onexc.finish_reason == "length"rather than inspecting the raw text, since that is the signal the framework already documents as the truncation tell.provider-extras/Reach two OpenAI request fields openarmature does not model, and show the three things
extrasrefuses.service_tierfor the cheaper lane andlogit_biasto suppress a retired severity label; neither is portable, so neither has a first-class field.The first draft used
seedas the vendor knob.seedis managed and would have rejected, so the example would have shipped demonstrating a failure as if it were the feature. Probing the arms first caught that, and turned up two nuances now documented rather than demonstrated: a matching extras value is a no-op rather than an error, and a sampling field left unset is not managed on that call. Managed means "produced by this call", not "nameable".It runs with no account, against a stub transport, because the point is the shape of the outbound request:
Two test gaps found while adding them
The smoke test's
DEMOSlist had nothing checking it against the directory. A new example was covered by nothing until someone remembered to register it, which is the same fail-open shape as_DRIVER_EXPECTED_KEYS.test_every_example_directory_is_listednow derives the set from disk and compares.The rename had no test that would have caught the original bug. Every existing test read the implementation's own spelling or went through the conformance harness, which aliased the two names, so none of them could fail on it. Added a test that writes the builder the documented way. A consistent revert across the three source files reproduces
AttributeError: 'StructuredOutputInvalid' object has no attribute 'output_content'raised inside the builder, which is the original defect rather than an approximation.The
conformance.tomlmanifest was not in the wheelReported by an agent adopting the package, and confirmed from the wheel's own RECORD.
The bundled
AGENTS.mdtells an agent to read the manifest for per-proposal implementation status. That is the only artifact that answers "is this real in the version I have installed" for a behaviour with no importable name. The file was absent from the distribution, and the pointer said "at the repo root", so the instruction resolved for someone working in a clone and for nobody who ranpip install openarmature.That asymmetry is why it survived so long: the sdist had the file all along, so working from a clone or an sdist build never hits it. Only a wheel install does, and that is not how any of us work.
It is force-included at
openarmature/conformance.toml, beside the other bundled docs, reachable viaimportlib.resources.files("openarmature"). The pointer now names that path and says to check the manifest before planning against an accepted proposal. Verified through the whole chain including a wheel built from the sdist.This matters more in this release than the last:
0085and0124both shippartial, and their notes are where which-half-is-missing is recorded.Force-included rather than copied into
src/the wayAGENTS.mdis handled, because a second committed 128K file would appear in every diff that touches the manifest. The cost is that the guard checks config shape rather than a built artifact.And the patterns duplication is deliberate
The same report asked whether the seven
_patterns/*.mdfiles shipping alongside their inline copies inAGENTS.mdwas intentional. It is, and the generator says why:build_patterns_data()produces the standalone files foropenarmature.patterns.list()/get(name)using a different transform so each resolves its own cross-references.AGENTS.mdnow states that, since the alternative is every reader asking.The sdist was carrying our private notes
Found while checking whether the manifest reached the sdist.
Hatch packages what is on disk and does not read
.git/info/exclude, so_tasks/was being collected into the tarball for publication. Those are internal working notes.Checked against PyPI: no published release contains them. This is preventive rather than a leak to clean up.
The pinned
openarmature-specsubmodule is excluded in the same change: 1124 files in 0.16.0 and most of the tarball, reconstructible from thespec_versionpin. The sdist goes 3.4MB to 1.4MB.tests/,examples/anddocs/stay in deliberately, and the guard asserts that, so a later tidy-up has to change the stated reasoning rather than quietly dropping them.Evidence
DEMOSAttributeErrorforce-includeentry droppedAGENTS.mdpointer back to repo-relative_tasks/sdist exclude dropped2308 passing, ruff and pyright clean, manifest consistent.
Three findings from an adversarial pass over #303, fixed here
#303 is merged, so these ride along rather than reopening it. Each was verified locally before being written down.
A merged extras key kills the retry it rides on
#303 made the per-attempt override merge
extrasper key, which is the right reading of §7.1. It also broke a config that previously worked.A base
extraskey is the §6 escape hatch: unmanaged where it was written, because the base leaves the declared field unset so the mapping emits nothing of that name. Merging carries it onto a retry whose override does set the field, where it is managed, so §8.1 rejects the call pre-send.Measured both ways against a transport returning 503 on every call,
max_attempts=3:The retry dies on attempt 1 and the original transient is discarded, not even chained, because the body build sits outside the attempt's
try.This PR first fixed that with a supersede, and then reverted it. The reverted version dropped an inherited extras key when the override declared a field of the same name. Spec ruled against it, and the third of their three reasons is the one that decides it:
extrasescape hatch keeps the override in the same channel,RuntimeConfig(extras={"temperature": 0.3}), which merges per key and sends on every attempt. The original config switched channels mid-stream for one wire key and §8.1 caught exactly that.RuntimeConfig(temperature=0.3, extras={"temperature": 0.9})is rejected when a caller writes it and accepted when a merge assembles it. One shape, two meanings, decided by history rather than content. A merged config could not then be logged, replayed or handed to another call and behave the same way, and a second implementation would have to track provenance through the merge to stay conformant. That is the defect class proposal 0124 exists to prevent, one level down.So the collision stands and the reject is correct. What changed instead is the diagnosis. The collision surfaces on attempt 1, after attempt 0 has already gone out on the base alone, so a broken retry config hides until the first retry. The error now reports that the attempt's config merges a base with a per-attempt override, that the two values come from different channels, and which channel to change. Detecting it before attempt 0 needs every attempt's body built eagerly, so that is recorded as a follow-up rather than built.
Claims that were wrong, some of them in the PR that was fixing wrong claims
docs/concepts/llms.mdand the 0122 CHANGELOG entry still documented wholesale replacement and a warning this release deleted. An override may set an extras key toNone, which sends JSONnullrather than withdrawing the key, so the claim that withdrawal is inexpressible overstated it.raw_contentandfailure_description.error_messagenamed one violation where the output broke four waysSeparate from the extras work and also spec-ruled. JSON Schema validation raises on the first failure, so a five-field record that broke four ways reported one of them, and a reask builder quoting the field cleared one problem per round trip.
Measured against a real model behind an endpoint that accepts
response_formatand ignores it: four to five attempts to extract a five-field record, and no recovery at all inside the canonicalmax_attempts=3. Enumerating brings the same extraction to two.Both schema forms were affected. The class path runs JSON Schema validation before Pydantic, deliberately, so it inherits the dict path's strictness rather than Pydantic's coercion, and raised from the same place.
Every violation is now enumerated, one per line, sorted so the same output and schema give the same string. A parse failure stays a single violation, since no schema check runs against output that is not well formed. The field is still a human-readable string and
response_schemais still a separate attribute; composing them into a correction is the caller's job, because the implementation authors no prompt text of its own. The exception stays unbounded because a caller needs every violation, while an observer's emitted copy is already bounded by its payload byte cap, so a wide schema lengthens what a caller reads and not what a trace carries.Smaller ones
A reask builder that raises now logs which of the two failed. The cause was already chained, but the surfaced error reads as a bad model rather than a broken builder, and a caller who never inspects
__cause__debugs the wrong thing. That also gives the module's orphaned logger a consumer. A dead identity alias left in the otel test driver is gone, the same dead alias the rename removed fromwire.py. A change-history comment block went, per the project's comment rules.The root AGENTS.md had drifted far enough to misdirect
Flagged by the release review and verified claim by claim against the tree, not against the file.
src/openarmature/middleware/graph/prompts/,retrieval/conformance.toml'sspec_pinis the one that drifted silentlytest_smoke.py— version sync"The bundled
src/openarmature/AGENTS.mdis byte-identical to its generator's output and was not the problem. It now says so, since the two files sit one directory apart and only one is safe to edit.Evidence for the new work
error_messageformatter regressed to first-violation-onlyThe extras cases are measured end to end: the collision rejecting on attempt 1 with attempt 0 already sent, the same-channel override retrying cleanly, per-key merge preserved, an unrelated base key still inheriting, and a plain escalating schedule unchanged.
A sort-order mutation on the enumeration survived, correctly: order is deliberately unspecified, so no order-changing mutation can kill a test that does not pin one. That run also exposed a dead assertion of mine comparing a value to itself, which is gone.
2310 passing, 497 skipped. ruff, pyright and the manifest check clean. Both examples rerun against the live endpoint.