Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions CHANGELOG.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions conformance.toml
Original file line number Diff line number Diff line change
Expand Up @@ -901,9 +901,9 @@ note = "Nested-fan-out span lineage. graph-engine §6: fan_out_index_chain / bra
# Spec v0.80.0 (proposal 0085). Nested-fan-out checkpoint resume
# lineage (pipeline-utilities §10.11 enclosing_fan_out_lineage).
[proposals."0085"]
status = "implemented"
status = "partial"
since = "0.17.0"
note = "Nested-fan-out checkpoint lineage + no-mis-skip invariant (pipeline-utilities §10.11 / §10.7 / §10.2). SAVE-side in-memory keying shipped in #194 (v0.16.0): a fan-out instance's checkpoint tracking key carries the enclosing fan-out instance lineage in the in-memory dict + projection / lookup / cleanup, so concurrent outer instances no longer collide live. The RESUME consume-side now ships (v0.17.0): FanOutProgress gains an optional enclosing_fan_out_lineage (a sequence of {namespace, fan_out_node_name, fan_out_index}, the new EnclosingFanOutInstance record type), and _restore_fan_out_progress_state keys the tracking dict by it (projected to the flat fan_out_index tuple the re-entry key uses). This realizes the §10.11 no-mis-skip invariant + §10.11.1 exactly-once for nested fan-outs via the existing keyed re-entry: a lineage-bearing entry positively matches per outer instance (correct skip); an empty/legacy lineage keys to () and never matches a non-empty re-entering lineage (re-run, the safe floor per §10.7). Backward-compatible: flat records (empty lineage) resume identically, and the SQLite json serializers round-trip the new field. pipeline-utilities fixture 076 (both cases: lineage-matched skip + legacy full-re-run safety floor) runs in test_checkpoint.py via a seeded-record resume path; the invariants are verified against the inner-leaf source values that actually re-ran (final state alone cannot tell a correct skip from a full re-run). KNOWN follow-up (out of scope per §66 / §76): the crash-PRODUCED write side (_project_fan_out_progress emitting the rich lineage on a real crash record, so a real nested-fan-out crash resumes at the correct-skip rather than the safe re-run floor) needs lineage-qualified crash boundaries and is tracked; parallel-branches cross-nesting is a deferred dimension."
note = "Nested-fan-out checkpoint lineage + no-mis-skip invariant (pipeline-utilities §10.11 / §10.7 / §10.2). SAVE-side in-memory keying shipped in #194 (v0.16.0): a fan-out instance's checkpoint tracking key carries the enclosing fan-out instance lineage in the in-memory dict + projection / lookup / cleanup, so concurrent outer instances no longer collide live. The RESUME consume-side now ships (v0.17.0): FanOutProgress gains an optional enclosing_fan_out_lineage (a sequence of {namespace, fan_out_node_name, fan_out_index}, the new EnclosingFanOutInstance record type), and _restore_fan_out_progress_state keys the tracking dict by it (projected to the flat fan_out_index tuple the re-entry key uses). This realizes the §10.11 no-mis-skip invariant + §10.11.1 exactly-once for nested fan-outs via the existing keyed re-entry: a lineage-bearing entry positively matches per outer instance (correct skip); an empty/legacy lineage keys to () and never matches a non-empty re-entering lineage (re-run, the safe floor per §10.7). Backward-compatible: flat records (empty lineage) resume identically, and the SQLite json serializers round-trip the new field. pipeline-utilities fixture 076 (both cases: lineage-matched skip + legacy full-re-run safety floor) runs in test_checkpoint.py via a seeded-record resume path; the invariants are verified against the inner-leaf source values that actually re-ran (final state alone cannot tell a correct skip from a full re-run). partial for exactly this reason, and the citation that used to defend it as out of scope (a bogus reference to §66 / §76) does not exist in 0085, whose own Out of scope section lists three items and does not include the write side. What IS met: §10.11's no-mis-skip floor, because an unmatched record re-runs rather than skipping. What is NOT met: §10.11.1's exactly-once guarantee does not extend to nested fan-outs, since a COMPLETED inner instance re-runs on resume and its side effects execute twice. Fixture 076 passes against a record the test seeds by hand, not one the engine wrote. THE GAP: the crash-PRODUCED write side (_project_fan_out_progress emitting the rich lineage on a real crash record, so a real nested-fan-out crash resumes at the correct-skip rather than the safe re-run floor) needs lineage-qualified crash boundaries and is tracked; parallel-branches cross-nesting is a deferred dimension."

# Spec v0.79.0 (proposal 0086). Service-wide default cache_ttl_seconds
# on PromptManager (prompt-management §6). Implemented since 0.17.0;
Expand Down Expand Up @@ -1200,7 +1200,7 @@ note = "conformance-adapter §5.4 *Subgraph declaration placement*: a declaratio
[proposals."0124"]
status = "partial"
since = "0.17.0"
note = "An orphan provider span's parent is resolved structurally: the enclosing wrapper is determined by the call's position in the graph rather than by which spans an observer has materialized, and \u00a76's synthesis trigger moves from the first inner started event to the first event that needs the span. Both observers do this; PR #277 shipped the OTel half, and the Langfuse half lands with this adoption (it had no call-site synthesis at all). partial because two arms of the structural-parent guarantee are unmet, both deliberately. A wrapper-issued call inside a DETACHED wrapper still resolves by drain order on both observers: the detached openers assume a trace already exists, which holds from the node path and not from the synthesis path, so those arms are unmirrored in both synthesizers (the open half of issue #279). And a callable parallel branch on Langfuse still resolves an orphan's parent by event order, because it is stored as a leaf where OTel renders it as the dispatch span; rendering it as the dispatch was attempted and reverted, since it loses the section 8.4.2 error mapping and does not remove the ordering dependence. Tracked, and it needs a spec answer first. The failure-isolation marker resolves through the same synthesizing path as the five provider handlers rather than a bare enclosing-wrapper walk. The await_event_delivery directive is honored at all five blocks across fixtures 133 / 134 / 152 / 153, as a barrier over delivery of this call's provider event rather than the scheduler yield PR #277 implemented, which \u00a75.1 calls insufficient. The barrier verifies it bound to the live invocation before trusting the drain summary, because drain_events_for returns a clean summary for an unmatched invocation_id, and records its outcome for assertion after the run rather than raising inside middleware a collect-policy fan-out would absorb. Adopting it also required the hand-built 133 / 134 drivers to honor the fixtures' declared phase: pre, which they ignored in favor of a hardcoded post, where the node body has already synthesized the dispatch span and no ordering is left for the barrier to pin."
note = "An orphan provider span's parent is resolved structurally: the enclosing wrapper is determined by the call's position in the graph rather than by which spans an observer has materialized, and \u00a76's synthesis trigger moves from the first inner started event to the first event that needs the span. Both observers do this; PR #277 shipped the OTel half, and the Langfuse half lands with this adoption (it had no call-site synthesis at all). partial for three unmet arms of the structural-parent guarantee, listed worst first. (1) THE RESOLVER STILL KEYS ON SPAN OPENNESS: it returns the calling node's span when that span happens to be open, which section 5.5's MUST NOT forbids, and because the OTel observer publishes node spans synchronously a wrapper-issued call whose provider event drains later becomes the node's child. The parent is a function of scheduling, reachable from ordinary node middleware. No fixture can catch it: the adapter attaches instance or branch middleware where section 5.1 says node middleware, so no case reaches the openness check. (2) NO SUBGRAPH SPAN IS SYNTHESIZED, on both observers, so an orphan inside a plain subgraph parents under the invocation span on OTel and at the Trace root on Langfuse; section 5.5 names the subgraph span alongside the other two. (3) THE DETACHED ARMS are unmirrored on both observers, so a call inside a detached wrapper resolves by drain order and differs in trace id (the open half of issue #279). A previously-listed fourth arm, a callable parallel branch on Langfuse, IS RETRACTED: section 8.4.8 makes the callable branch's single observation its per-branch dispatch observation, so there is no leaf-versus-dispatch race to lose, and the parent is stable with and without a yield, reproduced both ways. The residual callable-branch gap is narrower and separate: that observation omits parallel_branches_parent_node_name, which section 8.4.2 requires. The failure-isolation marker resolves through the same synthesizing path as the five provider handlers rather than a bare enclosing-wrapper walk. The await_event_delivery directive is honored at all five blocks across fixtures 133 / 134 / 152 / 153, as a barrier over delivery of this call's provider event rather than the scheduler yield PR #277 implemented, which \u00a75.1 calls insufficient. The barrier verifies it bound to the live invocation before trusting the drain summary, because drain_events_for returns a clean summary for an unmatched invocation_id, and records its outcome for assertion after the run rather than raising inside middleware a collect-policy fan-out would absorb. Adopting it also required the hand-built 133 / 134 drivers to honor the fixtures' declared phase: pre, which they ignored in favor of a hardcoded post, where the node body has already synthesized the dispatch span and no ordering is left for the barrier to pin."

[proposals."0125"]
status = "not-yet"
Expand Down
12 changes: 6 additions & 6 deletions docs/concepts/llms.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,13 +166,13 @@ response = await provider.complete(
response_schema=schema,
retry=LlmRetryConfig(
max_attempts=3,
reask=lambda err: f"That output was invalid: {err.failure_description}. Return corrected JSON.",
reask=lambda err: f"That output was invalid: {err.error_message}. Return corrected JSON.",
),
)
```

The builder receives the raised `StructuredOutputInvalid` (its
`raw_content` is the model's invalid output, `failure_description` the
`output_content` is the model's invalid output, `error_message` the
reason) and returns the correction text. On each invalid attempt the loop
appends the model's raw output as an `assistant` message and your
correction as a `user` message to a working transcript that accumulates
Expand Down Expand Up @@ -701,8 +701,8 @@ can handle them.
`StructuredOutputInvalid` is the new one and worth a note. It fires
when a model returns content that fails to parse as JSON, or parses
but fails to validate against the supplied schema. The exception
carries the requested `response_schema`, the `raw_content` the model
produced, and a `failure_description`. It is non-transient by default
carries the requested `response_schema`, the `output_content` the model
produced, and an `error_message`. It is non-transient by default
because a model that emits non-conforming output on a given prompt
usually emits the same non-conforming output on retry. Useful retry
strategies for this case involve changing the prompt or doubling
Expand All @@ -722,8 +722,8 @@ async def classify_with_diagnostics(state):
log.warning(
"schema-validation failure on classify",
extra={
"raw_content": exc.raw_content,
"failure": exc.failure_description,
"output_content": exc.output_content,
"failure": exc.error_message,
},
)
raise
Expand Down
16 changes: 8 additions & 8 deletions src/openarmature/llm/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -199,8 +199,8 @@ class StructuredOutputInvalid(LlmProviderError):

Attributes:
response_schema: The JSON Schema requested.
raw_content: The raw response content the model produced.
failure_description: A description of the parse or validation
output_content: The raw response content the model produced.
error_message: A description of the parse or validation
failure.
finish_reason: The normalized finish reason of the response that
failed validation (``"length"`` signals a truncation, the key
Expand All @@ -219,8 +219,8 @@ class StructuredOutputInvalid(LlmProviderError):

category = STRUCTURED_OUTPUT_INVALID
response_schema: dict[str, Any]
raw_content: str
failure_description: str
output_content: str
error_message: str
finish_reason: str | None
usage: Usage | None
response_id: str | None
Expand All @@ -230,17 +230,17 @@ def __init__(
self,
*args: Any,
response_schema: dict[str, Any],
raw_content: str,
failure_description: str,
output_content: str,
error_message: str,
finish_reason: str | None = None,
usage: Usage | None = None,
response_id: str | None = None,
response_model: str | None = None,
) -> None:
super().__init__(*args)
self.response_schema = response_schema
self.raw_content = raw_content
self.failure_description = failure_description
self.output_content = output_content
self.error_message = error_message
self.finish_reason = finish_reason
self.usage = usage
self.response_id = response_id
Expand Down
Loading
Loading