From f5af18a255ecbd59191c031673f9d83a05a2169a Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 30 Aug 2026 02:13:17 +0200 Subject: [PATCH 1/9] prepare_inputs: all three selectors, signature from the input-form descriptor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `prepare_inputs` took the method closure as inline `files` only, and read the target pipe's signature from the explicit inputs template. Both halves change. It now names the method three ways — inline `files`, a `method_ref` address resolved by the runner, or a stored `method_id` resolved by the platform — exactly one per call, all server-resolved with nothing expanded client-side. Empty is absent, and none or several raises `InputPreparationError` before any request leaves the process. The signature now comes from one `POST /v1/validate` asking for `views: ["input_form"]`, and the walk is discriminated on each descriptor node's declared kind instead of on the shape of a value. The template marked a file position by rendering a `{"url": …}` dict, which is a side effect of a field being *named* `url` rather than of its concept — so an optional nested file field was never enumerated and its local path travelled to the runner as a literal string, and a text field merely named `url` was read from disk and uploaded. Both are gone. A `Dynamic` input is `kind: "unknown"` and is no longer entered, which is the one deliberate behaviour flip: such a caller uploads with `upload_file` first. `build_inputs` and its models are deleted — the wrapper existed only to be the signature source, and nothing calls `/v1/build/*` from this SDK now. The shared crate envelope it also held (`MthdsFileItem`, `CrateRequestBase`, `CrateInvalidReport`) moves to `crate_models.py`, beside the routes that still use it. `prepare_inputs` also learns the explicit `{concept, content}` input envelope, a pre-existing parity gap: the JS SDK has accepted it since an earlier release, so the two would not have been identical after this fix. Mirrors `pipelex-sdk-js` 0.17.0 (PR #42, bea4632); design of record is that repo's wip/prepare-inputs-selectors/design.md. Verified against api-dev.pipelex.com (pipelex-hosted 0.11.1): inline files defaulting via main_pipe, a method_ref with an explicit pipe_ref, the manifest-only main_pipe refusal, the envelope round-trip, the bare-pipe_ref refusal, and one real bytes upload rewritten to pipelex-storage://. Closes L-260829-8a25d5 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HVMDoPnhuLyufT1ZoMmDTH --- CHANGELOG.md | 21 ++ docs/architecture.md | 10 +- docs/input-preparation.md | 91 ++++- pipelex_sdk/build_models.py | 171 --------- pipelex_sdk/client.py | 54 ++- pipelex_sdk/crate_models.py | 91 ++++- pipelex_sdk/prepare_inputs.py | 371 +++++++++++++++++--- pipelex_sdk/product_models.py | 4 +- pipelex_sdk/validation_models.py | 9 + pyproject.toml | 2 +- tests/unit/test_build_inputs.py | 191 ---------- tests/unit/test_crate_routes.py | 3 +- tests/unit/test_prepare_inputs.py | 467 +++++++++++++++++++++---- tests/unit/test_validation_contract.py | 19 + wip/prepare-inputs-selectors/plan.md | 40 +++ 15 files changed, 1002 insertions(+), 542 deletions(-) delete mode 100644 pipelex_sdk/build_models.py delete mode 100644 tests/unit/test_build_inputs.py create mode 100644 wip/prepare-inputs-selectors/plan.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 764994e..8e0219e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,26 @@ # Changelog +## [Unreleased] + +### Added + +- **`prepare_inputs` takes the method three ways.** Beside inline `files`, it accepts a `method_ref` address (resolved by the runner) or a stored `method_id` (resolved by the hosted platform) — exactly one per call, all three server-resolved, nothing expanded client-side. Empty is absent (`files=[]`, a blank `method_ref` / `method_id`), and none or several raises `InputPreparationError` naming the three forms before any request leaves the process. A method addressed by URL that declares a file input now has an input-preparation path; previously it had none, even though the request beneath accepted the address. +- **`PipelexValidationReport.default_pipe_ref`** — the qualified `pipe_ref` a caller gets by omitting the pipe selector, or `null` when the closure declares none or several. Optional and read leniently: a runner that predates the field sends nothing, and `prepare_inputs` falls back to the opaque `bundle_blueprint.main_pipe`. + +### Changed + +- **Breaking: `prepare_inputs` reads its signature from the input-form descriptor, not the inputs template.** It composes one `POST /v1/validate` with `views: ["input_form"]` and `allow_signatures=True`, and walks the standard's `InputForm` artifact — `document` / `image` mark a file position, `object` recurses through `fields`, `list` through `item`, everything else passes through. Source-compatible for every caller passing `files`; the SDK no longer calls `/v1/build/inputs` at runtime. A valid report carrying no descriptor is an error naming the pipelex-api floor, never a silent degrade to "no uploads". +- **Breaking: `build_inputs` and its models are removed.** `client.build_inputs`, `BuildInputsRequest`, `BuildInputsValidReport`, `BuildInputsResponse`, `BuildInputsResponseAdapter` and `InputsTemplateFormat` are gone — the route wrapper existed only to be the signature source `prepare_inputs` read, and nothing calls it now. This is the Python SDK's step of the workspace program retiring `/v1/build/*`; a caller that still needs a fill-in template projects one from the descriptor. +- **Breaking: the shared crate envelope moved to `pipelex_sdk.crate_models`.** `MthdsFileItem`, `CrateRequestBase` and `CrateInvalidReport` now live beside the routes that use them (`/v1/resolve`, `/v1/codegen`) and `pipelex_sdk/build_models.py` is deleted — a module named for the build routes could not go on holding the envelope after they left. The models themselves are unchanged; update the import path. +- **Breaking: a canonical file dict nested inside a `Dynamic` input is no longer uploaded.** Such an input is `kind: "unknown"` in the descriptor — the standard's escape hatch — and the walk does not enter it. Uploading on the strength of a `url` key is the value-shape guess this change removes; a caller with a Dynamic input uploads with `upload_file` first and passes the storage URI, which `docs/input-preparation.md` has always prescribed. +- **`prepare_inputs` accepts the explicit `{concept, content}` input envelope**, not only compact values, closing a parity gap with the JS SDK. An agent that fills an explicit template — the shape the hosted console and MCP hand out — can now hand it straight back; previously every file-bearing envelope position raised `InputPreparationError: Unsupported value at a file input … got dict`. The envelope's `content` is interpreted identically and preserved on output, so the concept annotation rides through to the run. +- **The documentation is rewritten around the descriptor.** `docs/input-preparation.md` now describes the three call shapes, the signature call, pipe selection and its manifest-only `main_pipe` gap, the envelope, and why the template was the wrong signature source; `docs/architecture.md` follows the removal. + +### Security + +- **An optional nested file field is now uploaded.** The required-only inputs template never rendered one, so its file position was invisible and the caller's local path travelled to the runner as a literal string. The descriptor states `required: false` and the walk enters it. +- **A text field merely *named* `url` is no longer read from disk.** The template marked a file position by rendering a `url`-bearing dict — a side effect of the field's *name*, not of its concept — so a path-shaped text value was uploaded. `kind: "text"` ends that. + ## [v0.8.0] - 2026-08-29 ### Added diff --git a/docs/architecture.md b/docs/architecture.md index 3f8231c..4beea1e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -175,9 +175,9 @@ The types are imported and used, never re-exported from `pipelex_sdk`. Re-export `resolve` and `codegen` (`pipelex_sdk/crate_models.py` + the two client methods) are the second crate-family surface, mirroring the JS SDK's v0.12.0 routes: `POST /v1/resolve` emits the normalized library crate (the MTHDS Library Crate Format — typed as opaque transport, `dict[str, Any]`, because the crate schema is owned by the standard and a restatement here would be free to drift), and `POST /v1/codegen` projects that crate through the `kind` × `target` axes into stamped typed artifacts plus their `codegen.lock` (write both verbatim and the offline `pipelex codegen check` passes on the tree). Both follow the build routes' 200-verdict discipline, sharing `CrateInvalidReport`, and ride `_request_product`, so a no-verdict condition raises the typed `ApiResponseError`. -Their closure selector is the strict three-way XOR — inline `files`, an address-form `method_ref` (server-resolved; the registry form keeps its `501`), or the hosted `method_id` pass-through — enforced at request construction (`CrateToolingRequest._exactly_one_selector`) as well as by the server. `BuildInputsRequest` shares the `CrateRequestBase` envelope (`files` XOR `method_ref`) but takes NO `method_id`: the `/v1/build/*` projections are deliberately excluded from the hosted tooling selector, and a `method_id` handed to it raises a teaching error naming the migration (fetch the method with `get_method` and pass its source as `files`) rather than letting pydantic silently ignore the key. +Their closure selector is the strict three-way XOR — inline `files`, an address-form `method_ref` (server-resolved; the registry form keeps its `501`), or the hosted `method_id` pass-through — enforced at request construction (`CrateToolingRequest._exactly_one_selector`) as well as by the server. The shared envelope itself (`MthdsFileItem`, `CrateRequestBase`, `CrateInvalidReport`) lives in `crate_models.py` beside the routes that use it; it sat in a `build_models.py` module until `prepare_inputs` moved onto the input-form descriptor and the `/v1/build/inputs` wrapper was removed with it. -**A `method_ref` closure gets a fetch-sized budget.** Resolving an address can make the server clone a repository before it answers, and the server-side clone timeout runs well past the 30s management budget on a cold cache — an abort there would report a healthy, still-cloning server as unreachable. So a `method_ref`-carrying `build_inputs` / `resolve` / `codegen` uses an internal 3-minute budget (`_METHOD_REF_FETCH_TIMEOUT_SECONDS`, threaded through `_request_product`'s `request_timeout` override); it is internal (no new caller-facing parameter) and inert behind the hosted gateway's own cap. Mirrors the JS SDK's fetch budget; the run routes and `validate` need none because they already ride the 20-min blocking ceiling. +**A `method_ref` closure gets a fetch-sized budget.** Resolving an address can make the server clone a repository before it answers, and the server-side clone timeout runs well past the 30s management budget on a cold cache — an abort there would report a healthy, still-cloning server as unreachable. So a `method_ref`-carrying `resolve` / `codegen` uses an internal 3-minute budget (`_METHOD_REF_FETCH_TIMEOUT_SECONDS`, threaded through `_request_product`'s `request_timeout` override); it is internal (no new caller-facing parameter) and inert behind the hosted gateway's own cap. Mirrors the JS SDK's fetch budget; the run routes and `validate` need none because they already ride the 20-min blocking ceiling. ## Pipelex product surface (hosted management routes) @@ -215,7 +215,7 @@ The wire models are snake_case Pydantic v2. Response models are extension-open ( ## Out of scope -- The remaining `/v1/build/*` authoring helpers — `build_output`, `build_runner`, `concept`, `pipe_spec`. `build_inputs` shipped in 0.5.0 and is no longer deferred. +- The `/v1/build/*` helpers — `build_output`, `build_runner`, `concept`, `pipe_spec`. `build_inputs` shipped in 0.5.0 and was removed again once `prepare_inputs`, its only caller, moved onto `validate` + the input-form descriptor: this SDK no longer touches `/v1/build/*`, which the workspace is retiring (`wip/build-retirement/`). - Organization *switch* (a WorkOS session operation, not a `/v1` route). - A `~/.pipelex/config` file reader (env-only for now, matching the JS SDK). - A synchronous client facade. @@ -229,12 +229,12 @@ The wire models are snake_case Pydantic v2. Response models are extension-open ( This SDK is a port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`) and tracks it closely, but it is **not surface-complete, and this section is where the gaps are named.** The Checkpoint-5 parity audit walked the JS `src/client.ts`, `src/index.ts` (the public barrel), and `docs/architecture.md`, plus a field-by-field sweep of `runs.ts` / `product-models.ts` / `models.ts`; it concluded surface-completeness, and that conclusion went stale as the JS SDK grew. The honest list of what has no Python counterpart today: - **Tooling routes** — `lint`, `format` (`resolve` and `codegen` shipped in 0.8.0 with the method selectors and are **not** gaps). -- **Authoring helpers** — `build_output`, `build_runner`, `concept`, `pipe_spec` (`build_inputs` shipped in 0.5.0 and is **not** a gap). +- **Authoring helpers** — `build_output`, `build_runner`, `concept`, `pipe_spec`. JS still exports its `buildInputs` wrapper; Python's was **deleted** rather than left unused, because `prepare_inputs` was its only caller. A deliberate divergence, not a gap: the JS wrappers retire together as their own step of the same program. - **Offline helpers** — `run_codegen_check` (the codegen drift check) and `get_method_closure` (client-side sugar that parses the polymorphic `mthds` source into a run-ready closure — in the JS SDK it is the documented migration target for the deleted by-id expansion legs; this SDK never had such legs, so the utility stays deferred rather than required). Each stays deferred rather than silently missing. Everything else — the protocol routes, the durable lifecycle, the whole product surface, and the errors — does have a Python equivalent. -**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), `build_inputs`, the crate routes (`resolve`, `codegen`), the input-preparation surface (`upload_file` / `prepare_inputs`), and `health`. The method selectors (`method_ref` / `method_id`) match the JS v0.16.0 surface across the run and tooling methods, with one signature-shape divergence: JS `validate` takes a `ValidateMethodSelector` object in place of its first argument, while Python takes `method_ref=` / `method_id=` keyword parameters — same wire, same XOR, idiomatic per language. +**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), the crate routes (`resolve`, `codegen`), the input-preparation surface (`upload_file` / `prepare_inputs`, taking all three method selectors and reading its signature from the input-form descriptor), and `health`. The method selectors (`method_ref` / `method_id`) match the JS v0.16.0 surface across the run and tooling methods, with one signature-shape divergence: JS `validate` takes a `ValidateMethodSelector` object in place of its first argument, while Python takes `method_ref=` / `method_id=` keyword parameters — same wire, same XOR, idiomatic per language. **Models** — field-for-field across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. Two divergences worth naming: the page envelopes keep the wire's snake_case `next_cursor`, where the JS mirror renamed it `nextCursor` for its own consumers; and the method-files catalog converter (`parse_method_files` / `serialize_method_files`) lives in this package, where the JS pair lives in `mthds-js` because `pipelex-mcp` consumes the same format and wanted one owner. There is no second Python consumer, and the catalog serialization is a Pipelex product concern rather than an MTHDS protocol one, so this SDK is a proper home for it. If `mthds-python` ever grows an owner for the format, this SDK adopts it then. diff --git a/docs/input-preparation.md b/docs/input-preparation.md index 6d65eed..34cc21f 100644 --- a/docs/input-preparation.md +++ b/docs/input-preparation.md @@ -1,8 +1,8 @@ # Input preparation (`upload_file` / `prepare_inputs`) -> **Status: implemented** (`pipelex_sdk/upload.py`, `pipelex_sdk/prepare_inputs.py`, `pipelex_sdk/build_models.py`). This document records the contract (design source: `wip/upload/README.md` in the workspace, tracked in `TODOS.md`). `upload_file` and `prepare_inputs` are the Python counterpart of `@pipelex/sdk`'s `uploadFile` / `prepareInputs`, built on the raw `upload()` wire call. This work also added the `build_inputs` route (the signature source), which the Python SDK previously lacked. +> **Status: implemented** (`pipelex_sdk/upload.py`, `pipelex_sdk/prepare_inputs.py`). `upload_file` and `prepare_inputs` are the Python counterpart of `@pipelex/sdk`'s `uploadFile` / `prepareInputs`, built on the raw `upload()` wire call. The design of record for the current shape is `pipelex-sdk-js/wip/prepare-inputs-selectors/design.md` in the sibling repo; the two SDKs are kept semantically identical. > -> **Current scope.** `prepare_inputs` takes the method closure as inline `files` (the signature source). Two pieces are deliberately deferred and additive (they do not change this contract): resolving a closure from a catalog `method_id`, and the opt-in ingest of `http(s)` URLs into storage — for now an `http(s)` URL at a file position always passes through unchanged. Kept in parity with `@pipelex/sdk`. +> **Current scope.** `prepare_inputs` names the method three ways — inline `files`, a `method_ref` address, or a stored `method_id` — and reads the target pipe's signature from the standard's input-form descriptor. One piece is deliberately deferred and additive (it does not change this contract): the opt-in ingest of `http(s)` URLs into storage — for now an `http(s)` URL at a file position always passes through unchanged. ## Why this exists @@ -14,8 +14,6 @@ Preparation is **explicit and separate from running.** `execute` / `start` never This is the Python side of one cross-language contract. The behavior matrix, pass-through rules, `Dynamic` handling, dedup, and failure categories are identical to the JS SDK; only the accepted source types differ per language. The two SDKs must agree semantically. See the JS counterpart's `docs/input-preparation.md` for the mirror. -The Python SDK previously had **no `/v1/build/*` coverage** — this change added the `build_inputs` counterpart `prepare_inputs` needs to resolve the declared signature (the JS SDK already exposes `buildInputs`). - ## The two operations ### `upload_file` — single-asset convenience @@ -40,29 +38,92 @@ The MIME type and size are known client-side, so the record is assembled without ### `prepare_inputs` — signature-driven input preparation ``` -prepare_inputs(method_ref, pipe, inputs) → PreparedInputs +client.prepare_inputs(files=…, pipe_ref=…, inputs=…) → PreparedInputs +client.prepare_inputs(method_ref=…, pipe_ref=…, inputs=…) → PreparedInputs +client.prepare_inputs(method_id=…, pipe_ref=…, inputs=…) → PreparedInputs ``` -Takes the **method reference** (bundle files or catalog `method_id`) plus the target **pipe**, resolves the pipe's declared input signature, interprets the caller's compact `inputs` top-down against that signature, uploads the file-bearing values, and returns `PreparedInputs`: +Takes the **method** as exactly one of three selectors, the optional target **pipe**, and the caller's `inputs`; resolves the pipe's declared input signature; interprets the inputs top-down against it; uploads the file-bearing values; and returns `PreparedInputs`. Per input, the caller may submit **either** the compact value **or** the explicit `{concept, content}` envelope — see "[Compact or explicit-envelope inputs](#compact-or-explicit-envelope-inputs)" below: - `inputs` — a **copy** of the caller's inputs with each asset reference replaced by the canonical content shape carrying `pipelex-storage://` in its `url` field (see "Rewritten-input shape" below). Copy-on-write: the caller's original object is never mutated. -- `uploads` — one upload record per prepared asset (the `upload_file` record shape), exposing `uri` so callers can log which source became which reference without reverse-engineering the rewritten object. +- `uploads` — one `UploadRecord` per prepared asset, exposing `uri` so callers can log which source became which reference without reverse-engineering the rewritten object. The prepared `inputs` are passed to the existing run lifecycle unchanged. +#### The three selectors + +Exactly one per call. **Empty is absent** — `files=[]`, `method_ref=""`, `method_id=" "` — mirroring the run options' rule, so an empty selector may sit beside a real one without tripping the exclusivity check. None or several raises `InputPreparationError` naming the three forms, before any request leaves the process. + +| Selector | What it is | Who resolves it | +| --- | --- | --- | +| `files` | the inline MTHDS closure (`MthdsFileItem` entries: `content` plus an optional `source` label) | nobody — inline | +| `method_ref` | a published method's address, `github.com//[/][@]` | the runner, server-side (pipelex-api >= 0.21.0 fetches the repository at the tag) | +| `method_id` | a stored method's catalog id (`mt_…`) | the hosted platform, which injects the stored source before the runner sees the request | + +Nothing is expanded client-side: `method_id` here is a **pass-through**, the same rule every other id-taking operation in this SDK follows. + +#### Where the signature comes from + +One `POST /v1/validate` per call, whatever the selector, asking for the **input-form descriptor**: + +```python +await client.validate(, True, …, views=[VALIDATION_VIEW_INPUT_FORM]) +``` + +`allow_signatures=True` is deliberate. Preparation needs a pipe's *declared* inputs, and a bundle mid-authoring with an unresolved signature somewhere else must not be refused inputs for a pipe whose inputs are declared — whether the bundle runs is the run's verdict, not preparation's. An `is_valid: false` verdict still means the closure does not load, which is a preparation failure. + +A `method_ref` makes the server clone a repository first; `validate` needs no special budget for it, because the route already defaults to the 20-minute execute ceiling. + +**A valid report that carries no descriptor is an error, never a silent "no uploads".** The descriptor rides `views: ["input_form"]` on pipelex-api >= 0.18.0; pointed at an older runner, `prepare_inputs` says so rather than returning inputs whose local paths would travel to the runner verbatim. + +#### Pipe selection + +`validate` has no pipe selector — its report describes every pipe, keyed by qualified `pipe_ref` — so the helper picks one, in this order: + +1. **`pipe_ref` when given.** Qualified-only: `domain.pipe_code`. A bare code, or a ref the method does not declare, is an `InputPreparationError` listing the qualified refs — one step to fix. The helper never grows a searched `pipe_code`: search is a run-route affordance, and the descriptor is keyed by qualified refs. +2. **The report's typed resolved default** (`default_pipe_ref`), once the runner serves it: the ref a caller gets by omitting the selector, manifest-aware for a fetched package. Read when present; a server that predates it sends nothing. +3. **The bundle's declared `main_pipe`**, read defensively from the opaque `bundle_blueprint` and qualified by its `domain`. +4. **The single pipe**, when the method declares exactly one. +5. Otherwise an `InputPreparationError` naming the candidates and asking for `pipe_ref`. + +> **The manifest-only `main_pipe` gap.** A published package may name its entry pipe in `METHODS.toml` alone — `github.com/Pipelex/methods/documents` and `.../image_generation` do — and the validate report never carries a manifest. Until step 2's field ships, such a package needs an explicit `pipe_ref`; the error lists the candidates, so the fix is one line. + +## Compact or explicit-envelope inputs + +Each input may be submitted in **either** of two shapes, and preparation treats them equivalently: + +- **Compact** — the bare value: a source string / `bytes` / `Path` / canonical `{"url": …}` content (e.g. `photo="…/p.png"`). +- **Explicit envelope** — the `{"concept", "content"}` shape (e.g. `photo={"concept": "native.Image", "content": {"url": "…"}}`). This is the template shape the hosted console and MCP hand agents to fill, so an agent that fills a template can hand it straight back. + +When a value is an envelope (a dict whose keys are **exactly** `concept` and `content`, matching the runtime's `_is_explicit` in `input_shaper.py`), preparation unwraps `content`, interprets it exactly as the compact value would be, and **re-wraps** the result — so the concept annotation rides through to the run. The envelope's `content` may itself be a scalar, canonical file content, a list, or a structured object nesting file fields; the same top-down walk applies underneath. + ## Signature-driven asset identification -The SDK **must not** guess that every string resembling a path is an asset — that would make ordinary text inputs environment-dependent and could upload unintended files. Interpretation comes from the method's **declared signature**, never from a value's shape alone. This mirrors the runtime's own top-down interpretation (`pipelex/pipelex/core/memory/input_shaper.py`, `InputShaper`) combined with the file-reference resolution of `pipelex/pipelex/pipeline/input_normalizer.py`, so local and hosted execution read the same compact inputs the same way. +The SDK **must not** guess that every string resembling a path is an asset — that would make ordinary text inputs environment-dependent and could upload unintended files. Interpretation comes from the method's **declared signature**, never from a value's shape alone. This mirrors the runtime's own top-down interpretation (`pipelex`'s `InputShaper`) combined with the file-reference resolution of `input_normalizer`, so local and hosted execution read the same compact inputs the same way. + +The signature is the **input-form descriptor** (`InputForm` from `mthds.protocol.input_form`), and the walk is discriminated on each node's declared `kind`: + +| Node kind | What the walk does | +| --- | --- | +| `document`, `image` | a **file position**, whatever the value's shape — resolved per the pass-through rules below | +| `object` | walks the declared `fields` by name against a dict value; keys the descriptor does not name are copied through untouched | +| `list` | walks `item` against each element of a list value | +| `text`, `prose`, `date`, `number`, `boolean`, `enum`, `unknown` | passes through at any depth | + +An **optional** field (`required: false`) is walked when the caller supplies it. A caller value whose shape disagrees with the node — a scalar at an `object`, a non-list at a `list` — passes through for the run to reject; preparation never second-guesses the signature. + +`unknown` is the standard's escape hatch for a `Dynamic` or `Composite` input, and it is **not** entered: the signature declares no file there. A caller with such an input uploads with `upload_file` first and passes the resulting `pipelex-storage://` URI. + +### Why the descriptor, and why not the inputs template + +Earlier releases read the signature from the explicit inputs template (`POST /v1/build/inputs`), which marked a file position by rendering a `{"url": …}` dict. That is a side effect of a field being **named** `url`, not of its concept being an Image or a Document, and two positions were misread as a result: -The declared signature is resolved via the explicit inputs template (`build_inputs` with `explicit=True`), which carries concept identity, canonical content shape, and multiplicity per input. +- an **optional nested file field** was never rendered by the required-only template, so its position was invisible and the caller's local path travelled to the runner as a literal string; +- a **text field merely named `url`** was read from disk and uploaded. -Interpretation per declared input: +The descriptor states the resolved kind at every depth and includes optional fields, so both are gone. It is also the standard's own artifact, derived from authored facts rather than from a rendered shape, and `/v1/validate` resolves all three method selectors server-side — which is what made the uniform selector surface possible at no server cost. -- A bare string, `Path`, or `bytes` value at an **Image/Document-declared** input is a **file reference**: local paths, data URLs, and bytes are uploaded and rewritten to `pipelex-storage://` URIs; HTTP(S) URLs and existing `pipelex-storage://` URIs pass through unchanged. -- The **identical** bare string at a **Text-declared** input is text and is never touched. -- **Canonical image/document content structures** are recognized by their URL-bearing fields wherever they appear, including nested in structured objects and lists — exactly as the runtime normalizer walks them. The refining case matters: a concept refining `Image`/`PDF` is classified by the **canonical content shape**, not by the concept ref alone. -- Inputs declared **`Dynamic`** are not path-interpreted (the signature genuinely cannot guide them); they accept canonical content structures or already-prepared references only. -- A repeated reference to the **same source** within one preparation is uploaded once and rewritten consistently (within-preparation dedup by source identity). +**Known limit.** A class-backed concept (`structure = "SomeClass"`) whose reflection cannot map a field annotation collapses to `kind: "unknown"` in the descriptor, so a file field beneath one is invisible to this walk. That is a fidelity bug in the runtime's `build_input_form`, tracked separately; pass such a value as an already-uploaded storage URI until it is fixed. ### Pass-through rules diff --git a/pipelex_sdk/build_models.py b/pipelex_sdk/build_models.py deleted file mode 100644 index 2e32765..0000000 --- a/pipelex_sdk/build_models.py +++ /dev/null @@ -1,171 +0,0 @@ -"""Wire models for the `/v1/build/inputs` route — the signature source `prepare_inputs` -reads to resolve a pipe's declared inputs. - -The Python SDK had no `/v1/build/*` coverage; `prepare_inputs` needs the explicit -inputs template, so this adds the `build_inputs` counterpart of `pipelex-sdk-js`'s -`buildInputs` (only this route — the other build projections are not needed here). -The crate routes (`/v1/resolve`, `/v1/codegen`) share this module's `CrateRequestBase` -envelope and `CrateInvalidReport` arm through `crate_models.py`. -A produced verdict is a `200` discriminated on `is_valid`; a no-verdict condition -(unknown `pipe_ref`, auth, server fault) throws `ApiResponseError`. -""" - -from __future__ import annotations - -from typing import Annotated, Any, Literal, Self, TypeAlias - -from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, field_validator, model_validator - -from pipelex_sdk.validation_models import ValidationErrorItem - -InputsTemplateFormat = Literal["json", "toml"] - - -class MthdsFileItem(BaseModel): - """One MTHDS file in a build closure. `source` is an optional provenance label the - server threads onto diagnostics raised from this file. - """ - - content: str - source: str | None = None - - -class CrateRequestBase(BaseModel): - """The closure selector every crate-family route shares — `/v1/resolve`, - `/v1/codegen`, and `/v1/build/*` (mirror of the server's `MthdsFilesRequest` and - of `pipelex-sdk-js`'s `CrateRequestBase`). - - Supply the closure EITHER as inline `files` OR as a `method_ref` — never both, and - never neither. An **address-form** `method_ref` - (`github.com//[/][@]`) is resolved by the server - (pipelex-api >= 0.21.0): the repository is fetched at the tag, the package is - located by manifest identity, and its `.mthds` files feed the closure with their - real relative paths as per-file sources. The **registry form** (any non-address - reference) stays reserved and answers `501` until a method registry exists. - - An EMPTY selector is normalized to absent before the exclusivity check — `files=[]` - selects no closure and `method_ref=""` (or whitespace-only) no address, the same - empty-as-absent rule the run routes apply — so an unusable value never counts as the - sole selector and never reaches the wire. - - The subclasses own the exclusivity validator, because the crate routes add a third - selector (the hosted `method_id`) that the build projections deliberately refuse. - """ - - files: list[MthdsFileItem] | None = None - method_ref: str | None = None - - @field_validator("files") - @classmethod - def _empty_files_are_absent(cls, value: list[MthdsFileItem] | None) -> list[MthdsFileItem] | None: - # `files=[]` is not a closure — normalize to absent so the XOR counts real selectors only. - return value or None - - @field_validator("method_ref") - @classmethod - def _blank_method_ref_is_absent(cls, value: str | None) -> str | None: - # A blank address selects nothing — same empty-as-absent rule as the run routes' - # `_normalized_selector` boundary. A real value is passed through untouched. - if value is None or not value.strip(): - return None - return value - - -class BuildInputsRequest(CrateRequestBase): - """Request for `POST /v1/build/inputs`. The closure is inline `files` XOR a - `method_ref` address; there is NO by-id form — the `/v1/build/*` projections are - deliberately excluded from the hosted tooling selector (`method_id` covers - `validate` / `resolve` / `codegen` only), so a stored method is expanded first - (fetch it with `get_method` and pass its source as `files`). - """ - - pipe_ref: str | None = None - format: InputsTemplateFormat = "json" - explicit: bool = False - - @model_validator(mode="before") - @classmethod - def _refuse_method_id(cls, data: Any) -> Any: - # A teaching error beats pydantic's default extra="ignore" silently dropping the - # key: a caller migrating from the by-id habit must learn the build routes have - # no by-id form (mirrors the JS `method_id: never` pin + runtime guard). - raw: Any = data - if isinstance(data, dict) and "method_id" in data: - msg = ( - "build_inputs takes no method_id — the /v1/build/* projections are excluded from the " - "hosted tooling selector (it covers validate/resolve/codegen only). Expand the stored " - "method first: fetch it with get_method and pass its MTHDS source as files." - ) - raise ValueError(msg) - return raw - - @model_validator(mode="after") - def _exactly_one_closure_selector(self) -> Self: - # Mirrors the server's own XOR so an illegal shape fails at construction, before - # anything hits the wire. - if (self.files is None) == (self.method_ref is None): - msg = "provide exactly one of `files` or `method_ref`" - raise ValueError(msg) - return self - - -class BuildInputsValidReport(BaseModel): - """The `is_valid: true` arm. The template rides `inputs` (json) or `inputs_toml` (toml).""" - - model_config = ConfigDict(extra="allow") - - is_valid: Literal[True] - pipe_ref: str - requested_pipe_ref: str | None = None - message: str - format: InputsTemplateFormat - explicit: bool - inputs: dict[str, Any] | None = None - inputs_toml: str | None = None - - @model_validator(mode="after") - def _template_matches_format(self) -> Self: - # Honor the adapter's malformed-200 guarantee for the template shape too: a valid - # verdict must carry the template field its `format` selects (and not the other). - # Without this, an `is_valid: true` body missing both templates would parse as a valid - # report and only fail one layer down in `prepare_inputs`. - match self.format: - case "json": - if self.inputs is None: - msg = "inputs is required when format is 'json'" - raise ValueError(msg) - if self.inputs_toml is not None: - msg = "inputs_toml must be absent when format is 'json'" - raise ValueError(msg) - case "toml": - if self.inputs_toml is None: - msg = "inputs_toml is required when format is 'toml'" - raise ValueError(msg) - if self.inputs is not None: - msg = "inputs must be absent when format is 'toml'" - raise ValueError(msg) - return self - - -class CrateInvalidReport(BaseModel): - """The `is_valid: false` arm shared by the build routes — an unresolvable closure is a - produced verdict on a `200`, never a thrown error. Branch on `is_valid`, not transport. - """ - - model_config = ConfigDict(extra="allow") - - is_valid: Literal[False] - validation_errors: list[ValidationErrorItem] - message: str - - -BuildInputsResponse: TypeAlias = Annotated[ - BuildInputsValidReport | CrateInvalidReport, - Field(discriminator="is_valid"), -] - -# The single parse path for a 200 `/build/inputs` body — discriminated on `is_valid`, built once at -# import (TypeAdapter construction is expensive), mirroring `PipelexValidationResultAdapter`. A -# malformed 200 (or an empty body) raises a clean `pydantic.ValidationError` rather than being -# mistaken for a valid verdict. -BuildInputsResponseAdapter: TypeAdapter[BuildInputsResponse] = TypeAdapter(BuildInputsResponse) # pylint: disable=invalid-name diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index e601af9..1f51018 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -32,11 +32,11 @@ from pydantic_core import to_json from typing_extensions import override -from pipelex_sdk.build_models import BuildInputsRequest, BuildInputsResponse, BuildInputsResponseAdapter, MthdsFileItem from pipelex_sdk.crate_models import ( CodegenRequest, CodegenResponse, CodegenResponseAdapter, + MthdsFileItem, ResolveRequest, ResolveResponse, ResolveResponseAdapter, @@ -173,7 +173,7 @@ # `method_ref` resolution can make the server CLONE a repository before it answers, and the # server-side clone timeout runs well past the 30s management budget on a cold cache — an # abort there would report a healthy, still-cloning server as unreachable. So a -# `method_ref`-carrying crate/build request gets this internal fetch-sized budget instead of +# `method_ref`-carrying crate request gets this internal fetch-sized budget instead of # `_POLL_REQUEST_TIMEOUT_SECONDS` (no new caller-facing parameter, and inert behind the # hosted gateway's own cap). The run routes and `validate` need no such override: they # already ride `request_timeout_seconds` (the 20-min blocking-execute ceiling), which clears @@ -1117,25 +1117,6 @@ async def upload(self, upload_input: UploadInput) -> UploadedFile: body = upload_input.model_dump(mode="json", exclude_none=True) return UploadedFile.model_validate(await self._request_product("POST", "upload", body=body)) - async def build_inputs(self, request: BuildInputsRequest) -> BuildInputsResponse: - """Project a pipe's declared inputs as a fill-in template — `POST /v1/build/inputs`. - - The closure is inline `files` XOR a `method_ref` address (server-resolved, - pipelex-api >= 0.21.0; the registry form stays a `501`). There is NO by-id form: the - `/v1/build/*` projections take no `method_id` (the hosted tooling selector covers - `validate`/`resolve`/`codegen` only) — expand a stored method's source into `files` - yourself (fetch it with `get_method`). - - Returns a 200 verdict: branch on `is_valid` before reading the arm — an unresolvable - closure comes back as `is_valid: false` with `validation_errors`, not a thrown error. - A no-verdict condition (unknown `pipe_ref`, a selector-resolution failure, auth, server - fault) raises `ApiResponseError`. This is the signature source `prepare_inputs` reads - (with `explicit=True`). - """ - body = request.model_dump(mode="json", exclude_none=True) - raw = await self._request_product("POST", "build/inputs", body=body, request_timeout=_crate_request_timeout_seconds(request.method_ref)) - return BuildInputsResponseAdapter.validate_python(raw) - # ── Crate extensions (Pipelex API — `/v1/resolve`, `/v1/codegen`) ───── # # The second crate-family surface, mirroring the JS SDK: `/v1/resolve` emits the @@ -1203,7 +1184,9 @@ async def upload_file( async def prepare_inputs( self, *, - files: list[MthdsFileItem], + files: list[MthdsFileItem] | None = None, + method_ref: str | None = None, + method_id: str | None = None, pipe_ref: str | None = None, inputs: dict[str, Any], ) -> PreparedInputs: @@ -1211,10 +1194,25 @@ async def prepare_inputs( assets, and return copy-on-write rewritten inputs (canonical content carrying `pipelex-storage://` in `url`) plus one upload record per prepared asset. HTTP(S) URLs and existing `pipelex-storage://` URIs pass through unchanged; all failures are raised - before any run is created. The caller supplies the method closure as inline `files`. - See `docs/input-preparation.md`. + before any run is created. + + The method is named exactly one of three ways — inline `files`, a `method_ref` address + (runner-resolved) or a stored `method_id` (platform-resolved) — all server-resolved, + with nothing expanded client-side. An empty selector is treated as absent. The + signature comes from one `POST /v1/validate` asking for the `input_form` view, so the + walk is guided by each input's DECLARED kind rather than by the shape of its value. + + `pipe_ref` is qualified-only (`domain.pipe_code`); omit it to default. See + `docs/input-preparation.md`. """ - return await _prepare_inputs_impl(self, files=files, pipe_ref=pipe_ref, inputs=inputs) + return await _prepare_inputs_impl( + self, + files=files, + method_ref=method_ref, + method_id=method_id, + pipe_ref=pipe_ref, + inputs=inputs, + ) async def list_runs( self, @@ -1420,9 +1418,9 @@ def _assert_method_ref_pairs_with_nothing(*, mthds_contents: list[str] | None, m def _crate_request_timeout_seconds(method_ref: str | None) -> float: - """The request budget for a call carrying a crate closure (the crate routes and the build - projections alike): the management default, unless the closure is a `method_ref` the - server may have to fetch first — see `_METHOD_REF_FETCH_TIMEOUT_SECONDS`. + """The request budget for a call carrying a crate closure (`/v1/resolve`, `/v1/codegen`): + the management default, unless the closure is a `method_ref` the server may have to fetch + first — see `_METHOD_REF_FETCH_TIMEOUT_SECONDS`. """ return _METHOD_REF_FETCH_TIMEOUT_SECONDS if method_ref else _POLL_REQUEST_TIMEOUT_SECONDS diff --git a/pipelex_sdk/crate_models.py b/pipelex_sdk/crate_models.py index 5048e7e..167ab4c 100644 --- a/pipelex_sdk/crate_models.py +++ b/pipelex_sdk/crate_models.py @@ -1,13 +1,18 @@ -"""Wire models for the crate routes — `POST /v1/resolve` and `POST /v1/codegen`. - -The second crate-family surface, mirroring `pipelex-sdk-js`: `/v1/resolve` emits the -normalized library crate, `/v1/codegen` projects that crate into stamped typed artifacts -plus their lock. Both are Pipelex API extensions (NOT MTHDS Protocol routes) over the -standard-owned artifact, so their wire fields stay brand-neutral. Same envelope and same -verdict discipline as the build routes: a produced verdict is a `200` discriminated on -`is_valid`, with `CrateInvalidReport` (from `build_models`) as the shared invalid arm; a -no-verdict condition (a malformed selector, a selector-resolution failure, auth, a server -fault) raises `ApiResponseError`. +"""Wire models for the crate routes — `POST /v1/resolve` and `POST /v1/codegen` — and the +shared crate envelope they are built on. + +The envelope lives here because these are the routes that still use it. `MthdsFileItem`, +`CrateRequestBase` and `CrateInvalidReport` used to sit in a `build_models` module beside the +`/v1/build/inputs` wire models; those went when `prepare_inputs` moved its signature source to +the input-form descriptor and this SDK stopped calling `/v1/build/*` (workspace campaign +`wip/build-retirement/`). Nothing about the envelope changed in the move. + +`/v1/resolve` emits the normalized library crate, `/v1/codegen` projects that crate into stamped +typed artifacts plus their lock. Both are Pipelex API extensions (NOT MTHDS Protocol routes) over +the standard-owned artifact, so their wire fields stay brand-neutral. A produced verdict is a +`200` discriminated on `is_valid`, with `CrateInvalidReport` as the shared invalid arm; a +no-verdict condition (a malformed selector, a selector-resolution failure, auth, a server fault) +raises `ApiResponseError`. The closure arrives in exactly one of three forms — the tooling routes' strict three-way XOR: inline `files`, an address-form `method_ref` (server-resolved, pipelex-api >= 0.21.0; @@ -23,7 +28,69 @@ from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, field_validator, model_validator -from pipelex_sdk.build_models import CrateInvalidReport, CrateRequestBase +from pipelex_sdk.validation_models import ValidationErrorItem + + +class MthdsFileItem(BaseModel): + """One MTHDS file in a crate closure. `source` is an optional provenance label the + server threads onto diagnostics raised from this file. + """ + + content: str + source: str | None = None + + +class CrateRequestBase(BaseModel): + """The closure selector every crate-family route shares — `/v1/resolve` and + `/v1/codegen` (mirror of the server's `MthdsFilesRequest` and of + `pipelex-sdk-js`'s `CrateRequestBase`). + + Supply the closure EITHER as inline `files` OR as a `method_ref` — never both, and + never neither. An **address-form** `method_ref` + (`github.com//[/][@]`) is resolved by the server + (pipelex-api >= 0.21.0): the repository is fetched at the tag, the package is + located by manifest identity, and its `.mthds` files feed the closure with their + real relative paths as per-file sources. The **registry form** (any non-address + reference) stays reserved and answers `501` until a method registry exists. + + An EMPTY selector is normalized to absent before the exclusivity check — `files=[]` + selects no closure and `method_ref=""` (or whitespace-only) no address, the same + empty-as-absent rule the run routes apply — so an unusable value never counts as the + sole selector and never reaches the wire. + + The subclass owns the exclusivity validator, because the crate routes add a third + selector (the hosted `method_id`) this base does not know about. + """ + + files: list[MthdsFileItem] | None = None + method_ref: str | None = None + + @field_validator("files") + @classmethod + def _empty_files_are_absent(cls, value: list[MthdsFileItem] | None) -> list[MthdsFileItem] | None: + # `files=[]` is not a closure — normalize to absent so the XOR counts real selectors only. + return value or None + + @field_validator("method_ref") + @classmethod + def _blank_method_ref_is_absent(cls, value: str | None) -> str | None: + # A blank address selects nothing — same empty-as-absent rule as the run routes' + # `_normalized_selector` boundary. A real value is passed through untouched. + if value is None or not value.strip(): + return None + return value + + +class CrateInvalidReport(BaseModel): + """The `is_valid: false` arm shared by the crate routes — an unresolvable closure is a + produced verdict on a `200`, never a thrown error. Branch on `is_valid`, not transport. + """ + + model_config = ConfigDict(extra="allow") + + is_valid: Literal[False] + validation_errors: list[ValidationErrorItem] + message: str class CrateToolingRequest(CrateRequestBase): @@ -95,7 +162,7 @@ class ResolveValidReport(BaseModel): ] # The single parse path for a 200 `/resolve` body — discriminated on `is_valid`, built once -# at import (TypeAdapter construction is expensive), mirroring `BuildInputsResponseAdapter`. +# at import (TypeAdapter construction is expensive), mirroring `PipelexValidationResultAdapter`. ResolveResponseAdapter: TypeAdapter[ResolveResponse] = TypeAdapter(ResolveResponse) # pylint: disable=invalid-name diff --git a/pipelex_sdk/prepare_inputs.py b/pipelex_sdk/prepare_inputs.py index 7857ad8..0b75b64 100644 --- a/pipelex_sdk/prepare_inputs.py +++ b/pipelex_sdk/prepare_inputs.py @@ -1,16 +1,21 @@ -"""`prepare_inputs` — signature-driven input preparation. Resolves the target pipe's -declared inputs via the explicit inputs template, interprets the caller's compact inputs -top-down against it, uploads the file-bearing values, and returns rewritten inputs -(canonical content carrying `pipelex-storage://` in `url`) plus one upload record per -prepared asset. Python counterpart of `pipelex-sdk-js`'s `prepareInputs`. - -The classification mirrors the runtime: `pipelex`'s `input_normalizer` walks -Image/Document contents (recognized by their `url`-bearing shape, incl. nested in -structured content) and `resolve_uri` decides upload vs pass-through. The declared -signature comes from the explicit template (`build_inputs`, `explicit=True`), whose -canonical content shape is the classifier — the file signal is a value that is a dict -containing a `url` key. See the shared behavior matrix (`wip/upload/behavior-matrix.md`) -and `docs/input-preparation.md`. +"""`prepare_inputs` — signature-driven input preparation. Names the method three ways, +resolves the target pipe's declared inputs from the standard's input-form descriptor, +interprets the caller's inputs top-down against it, uploads the file-bearing values, and +returns rewritten inputs (canonical content carrying `pipelex-storage://` in `url`) plus one +upload record per prepared asset. Python counterpart of `pipelex-sdk-js`'s `prepareInputs`. + +The signature comes from ONE `POST /v1/validate` asking for `views: ["input_form"]`, and the +walk is discriminated on each descriptor node's declared `kind` — never on the shape of a +value. That is the whole point: the previous source, the explicit inputs template, marked a +file position by rendering a `{"url": …}` dict, which is a side effect of a field being NAMED +`url` rather than of its concept being an Image or a Document. Two positions were misread as a +result — an OPTIONAL nested file field, which the required-only template never rendered, was +left un-uploaded and its local path travelled to the runner as a literal string; and a text +field merely named `url` was read from disk and uploaded. The descriptor states the resolved +kind at every depth and includes optional fields, so both are gone. + +See `docs/input-preparation.md`, and the design of record in +`pipelex-sdk-js/wip/prepare-inputs-selectors/design.md`. """ from __future__ import annotations @@ -22,13 +27,29 @@ from typing import TYPE_CHECKING, Any, Protocol, cast from urllib.parse import unquote_to_bytes +from mthds.protocol.input_form import ( + BooleanItem, + DateItem, + DocumentItem, + EnumItem, + ImageItem, + InputForm, + InputFormItem, + ListItem, + NumberItem, + ObjectItem, + ProseItem, + TextItem, + UnknownItem, +) from pydantic import BaseModel -from pipelex_sdk.build_models import BuildInputsRequest, BuildInputsResponse, CrateInvalidReport, MthdsFileItem from pipelex_sdk.errors import InputPreparationError from pipelex_sdk.upload import UploadRecord, UploadSource, upload_file +from pipelex_sdk.validation_models import VALIDATION_VIEW_INPUT_FORM, PipelexInvalidReport, PipelexValidationReport, PipelexValidationResult if TYPE_CHECKING: + from pipelex_sdk.crate_models import MthdsFileItem from pipelex_sdk.product_models import UploadedFile, UploadInput PIPELEX_STORAGE_SCHEME = "pipelex-storage://" @@ -48,11 +69,24 @@ class PreparedInputs(BaseModel): class _PrepareClient(Protocol): - """The client surface `prepare_inputs` needs: raw `upload` plus the `build_inputs` signature source.""" + """The client surface `prepare_inputs` needs: raw `upload` plus `validate` as the + signature source. Typed as `PipelexAPIClient.validate`'s own signature so the client + satisfies it structurally. + """ async def upload(self, upload_input: UploadInput) -> UploadedFile: ... - async def build_inputs(self, request: BuildInputsRequest) -> BuildInputsResponse: ... + async def validate( + self, + mthds_contents: list[str] | None = None, + allow_signatures: bool = False, + mthds_sources: list[str] | None = None, + render: list[str] | None = None, + views: list[str] | None = None, + *, + method_ref: str | None = None, + method_id: str | None = None, + ) -> PipelexValidationResult: ... class _PrepareContext: @@ -65,11 +99,39 @@ def __init__(self, client: _PrepareClient) -> None: self.dedup: dict[UploadSource, str] = {} +def _non_empty_string(value: object) -> str | None: + """A trimmed non-empty string, or `None` — the "empty is absent" rule. + + Deliberately local rather than reusing `client.py`'s `_normalized_selector`: that helper + is private to the client boundary and raises `PipelineRequestError`, where every failure + of this module owes an `InputPreparationError`. + """ + if not isinstance(value, str): + return None + trimmed = value.strip() + return trimmed or None + + def _is_file_content(node: Any) -> bool: - """A canonical Image/Document content is a dict carrying a `url` key.""" + """A canonical Image/Document content is a dict carrying a `url` key. + + A value-shape helper only, consulted at a position the DESCRIPTOR already declared a + file. It is no longer a classifier: reading it as one is the defect this module removed. + """ return isinstance(node, dict) and "url" in node +def _is_explicit_envelope(value: Any) -> bool: + """The explicit `{concept, content}` input envelope — keys EXACTLY `concept` and `content`. + + Matches the runtime's `_is_explicit` (`input_shaper.py`), so an agent that filled an + explicit template can hand it straight back. Anything else is a compact value. + """ + if not isinstance(value, dict): + return False + return set(cast("dict[str, Any]", value)) == {"concept", "content"} + + def _decode_data_url(data_url: str) -> tuple[bytes, str]: """Decode a `data:` URL into bytes plus its MIME type. @@ -140,7 +202,7 @@ async def _resolve_source(ctx: _PrepareContext, source: Any) -> str: async def _resolve_file_position(ctx: _PrepareContext, caller_value: Any) -> Any: """Resolve a value known to sit at a file position into canonical content with a rewritten `url`.""" - if isinstance(caller_value, dict) and "url" in caller_value: + if _is_file_content(caller_value): content = cast("dict[str, Any]", caller_value) resolved = await _resolve_source(ctx, content["url"]) return {**content, "url": resolved} @@ -148,61 +210,258 @@ async def _resolve_file_position(ctx: _PrepareContext, caller_value: Any) -> Any return {"url": resolved} -async def _resolve_node(ctx: _PrepareContext, template_node: Any, caller_value: Any) -> Any: - """Template-guided walk: a template node that is canonical file content marks a file position.""" - if _is_file_content(template_node): - return await _resolve_file_position(ctx, caller_value) - if isinstance(template_node, list) and template_node: - element_template = cast("list[Any]", template_node)[0] - if isinstance(caller_value, list): - items = cast("list[Any]", caller_value) - return [await _resolve_node(ctx, element_template, item) for item in items] - return caller_value # shape mismatch — leave it for the run to reject - if isinstance(template_node, dict) and isinstance(caller_value, dict): - template_dict = cast("dict[str, Any]", template_node) - caller_dict = cast("dict[str, Any]", caller_value) - result: dict[str, Any] = dict(caller_dict) - for key in template_dict: - if key in caller_dict: - result[key] = await _resolve_node(ctx, template_dict[key], caller_dict[key]) - return result - return caller_value # scalar (text/number/…) or shape mismatch — pass through +async def _resolve_node(ctx: _PrepareContext, node: InputFormItem, caller_value: Any) -> Any: + """Descriptor-guided walk, discriminated on the node's declared kind. + + - `document` / `image` — a file position, whatever the value's shape; + - `object` — walk the declared `fields` by name; keys the descriptor does not name are + copied through untouched. An OPTIONAL field is walked when present, which is what + makes an optional nested file reachable at all; + - `list` — walk `item` against each element; + - every other kind — pass through at any depth. `unknown` is the standard's escape hatch + for a `Dynamic` / `Composite` input and is deliberately NOT entered: the signature + declares no file there, and uploading on the strength of a `url` key is the value-shape + guess this walk removes. Such a caller uploads with `upload_file` first and passes the + storage URI. + + A caller value whose shape disagrees with the node (a scalar at an `object`, a non-list at + a `list`) passes through for the run to reject — preparation never second-guesses the + signature. The match is over the item classes rather than over `kind`, because each + per-kind `*Field` derives from its `*Item`: one set of patterns covers both the named + layer (top level, `object.fields`) and the nameless one (`list.item`), and it narrows the + node for the type checker where matching on `node.kind` would not. + """ + match node: + case DocumentItem() | ImageItem(): + return await _resolve_file_position(ctx, caller_value) + case ObjectItem(): + if not isinstance(caller_value, dict): + return caller_value + caller_dict = cast("dict[str, Any]", caller_value) + result: dict[str, Any] = dict(caller_dict) + for field in node.fields: + if field.name in caller_dict: + result[field.name] = await _resolve_node(ctx, field, caller_dict[field.name]) + return result + case ListItem(): + if not isinstance(caller_value, list): + return caller_value + elements = cast("list[Any]", caller_value) + return [await _resolve_node(ctx, node.item, element) for element in elements] + case TextItem() | ProseItem() | DateItem() | NumberItem() | BooleanItem() | EnumItem() | UnknownItem(): + return caller_value + + +def _resolve_selector( + *, + files: list[MthdsFileItem] | None, + method_ref: str | None, + method_id: str | None, +) -> tuple[list[MthdsFileItem] | None, str | None, str | None]: + """Normalize the three selectors and check that exactly one remains. + + Empty is absent — `files=[]`, `method_ref=""`, `method_id=" "` — mirroring the run + options' rule and the `CrateRequestBase` normalizers, so an empty selector may sit beside + a real one without tripping the XOR. The check lives here because this module is what + composes the `validate` call, and it runs BEFORE any request. + """ + selected_files = files or None + selected_method_ref = _non_empty_string(method_ref) + selected_method_id = _non_empty_string(method_id) + + given: list[str] = [] + if selected_files is not None: + given.append("`files`") + if selected_method_ref is not None: + given.append("`method_ref`") + if selected_method_id is not None: + given.append("`method_id`") + + if not given: + msg = ( + "Cannot prepare inputs: no method selector. Supply exactly one of `files` (an inline MTHDS " + "closure), `method_ref` (a published method's address) or `method_id` (a stored method's " + "catalog id)." + ) + raise InputPreparationError(msg) + if len(given) > 1: + msg = ( + f"Cannot prepare inputs: {' and '.join(given)} were both given. Supply exactly one method " + "selector — `files`, `method_ref` or `method_id`." + ) + raise InputPreparationError(msg) + return selected_files, selected_method_ref, selected_method_id + + +async def _fetch_signature( + client: _PrepareClient, + *, + files: list[MthdsFileItem] | None, + method_ref: str | None, + method_id: str | None, +) -> PipelexValidationReport: + """Ask `validate` for the signature, whatever the selector, and hand back the valid report. + + `allow_signatures=True` on purpose: preparation needs a pipe's DECLARED inputs, and a + bundle mid-authoring with an unresolved signature elsewhere must not be refused inputs for + a pipe whose inputs are declared — whether the bundle runs is the run's verdict, not + preparation's. An `is_valid: false` arm still means the closure does not load, which IS a + preparation failure. + + No timeout override for a `method_ref`: `validate` already rides the 20-minute blocking + ceiling, and the internal 3-minute fetch budget exists to RAISE the ~30s poll-ceiling + routes, not to lower this one. + """ + views = [VALIDATION_VIEW_INPUT_FORM] + result: PipelexValidationResult + if files is not None: + contents = [file_item.content for file_item in files] + # `validate_files`' rule: label every content once any file names a source, so the + # server never sees a length-mismatched `mthds_sources` array. + sources: list[str] | None + if any(file_item.source is not None for file_item in files): + sources = [file_item.source or f"inline://file-{index + 1}.mthds" for index, file_item in enumerate(files)] + else: + sources = None + result = await client.validate(contents, True, sources, None, views) + else: + result = await client.validate(None, True, None, None, views, method_ref=method_ref, method_id=method_id) + + if isinstance(result, PipelexInvalidReport): + first = result.validation_errors[0].message if result.validation_errors else result.message + msg = f"Cannot prepare inputs: the method signature did not resolve — {first}" + raise InputPreparationError(msg) + return result + + +def _blueprint_main_pipe_ref(blueprint: dict[str, Any]) -> str | None: + """The bundle blueprint's declared `main_pipe`, qualified by its `domain` when authored bare. + + Every read is defensive: `bundle_blueprint` is carried opaquely by this SDK on purpose — + its schema is the runtime's, not ours — so a shape that does not match falls through + rather than raising. + """ + main_pipe = _non_empty_string(blueprint.get("main_pipe")) + if main_pipe is None: + return None + if "." in main_pipe: + return main_pipe + domain = _non_empty_string(blueprint.get("domain")) + return f"{domain}.{main_pipe}" if domain is not None else None + + +def _select_pipe_ref(report: PipelexValidationReport, input_form: InputForm, requested: str | None) -> str: + """Pick the pipe whose descriptor guides the walk. + + `validate` has no pipe selector — its report describes every pipe, keyed by qualified + `pipe_ref` — so the choice is made here, in the order `docs/input-preparation.md` + documents: an explicit qualified `pipe_ref`, then the report's typed resolved default, + then the bundle's declared `main_pipe`, then the single pipe, else an error naming the + candidates. + """ + refs = list(input_form) + candidates = ", ".join(refs) if refs else "(none — the closure declares no pipes)" + + if requested is not None: + if "." not in requested: + msg = ( + "Cannot prepare inputs: `pipe_ref` must be qualified (`domain.pipe_code`), got the bare " + f'"{requested}". The method declares: {candidates}.' + ) + raise InputPreparationError(msg) + if requested not in input_form: + msg = f'Cannot prepare inputs: the method declares no pipe "{requested}". It declares: {candidates}.' + raise InputPreparationError(msg) + return requested + + # The typed resolved default, when the runner serves it (manifest-aware for a `method_ref` + # package, which is why it outranks the blueprint read below). + typed_default = _non_empty_string(report.default_pipe_ref) + if typed_default is not None and typed_default in input_form: + return typed_default + + blueprint_default = _blueprint_main_pipe_ref(report.bundle_blueprint) + if blueprint_default is not None and blueprint_default in input_form: + return blueprint_default + + if len(refs) == 1: + return refs[0] + + msg = f"Cannot prepare inputs: the method declares no single default pipe, so `pipe_ref` is required. It declares: {candidates}." + raise InputPreparationError(msg) async def prepare_inputs( client: _PrepareClient, *, - files: list[MthdsFileItem], + files: list[MthdsFileItem] | None = None, + method_ref: str | None = None, + method_id: str | None = None, pipe_ref: str | None = None, inputs: dict[str, Any], ) -> PreparedInputs: """Prepare a pipe's inputs: upload local/byte/data-URL assets at the signature's file-bearing positions and return copy-on-write rewritten inputs plus upload records. - HTTP(S) URLs and existing `pipelex-storage://` URIs pass through unchanged. All failures - are raised before any run is created. The declared signature is resolved from the inline - `files` closure; a closure that does not resolve raises `InputPreparationError`. No-verdict - conditions from the signature route (unknown `pipe_ref`, auth, server fault) surface as the - build route's `ApiResponseError`. + Args: + client: The client supplying `upload` and `validate`. + files: The method closure inline. Exactly one of `files` / `method_ref` / `method_id`. + method_ref: A published method's address — + `github.com//[/][@]` — resolved by the runner. + method_id: A stored method's hosted catalog id (`mt_…`), resolved by the platform. + A pure pass-through: nothing is expanded client-side. + pipe_ref: The target pipe as a QUALIFIED `domain.pipe_code`. Omit it to default — + see "Pipe selection" in `docs/input-preparation.md`. A bare `pipe_code` is + refused: the descriptor is keyed by qualified refs, and search is a run-route + affordance this helper deliberately does not grow. + inputs: The caller's inputs (variable name → value), compact or explicit-envelope + per input. + + Returns: + `PreparedInputs` — a copy of `inputs` with each file-bearing value rewritten to + canonical content carrying `pipelex-storage://` in `url`, plus one `UploadRecord` + per uploaded asset. + + Raises: + InputPreparationError: No selector or several; the closure did not resolve; the + report carries no descriptor; the pipe could not be selected; or a value at a + file position is unusable. HTTP(S) URLs and existing `pipelex-storage://` URIs + pass through unchanged, and every failure is raised BEFORE any run is created. + ApiResponseError: A no-verdict condition from `/v1/validate` — a malformed selector, + an unknown or foreign-org `method_id` (`404`), a stored method with no source, a + fetch failure at the address. """ - report = await client.build_inputs(BuildInputsRequest(files=files, pipe_ref=pipe_ref, format="json", explicit=True)) - if isinstance(report, CrateInvalidReport): - first = report.validation_errors[0].message if report.validation_errors else report.message - msg = f"Cannot prepare inputs: the method signature did not resolve — {first}" + selected_files, selected_method_ref, selected_method_id = _resolve_selector(files=files, method_ref=method_ref, method_id=method_id) + report = await _fetch_signature(client, files=selected_files, method_ref=selected_method_ref, method_id=selected_method_id) + + input_form = report.input_form + if input_form is None: + # Never a silent degrade to "no uploads": without the descriptor there is no + # signature to prepare against. + msg = ( + "Cannot prepare inputs: the validate report carries no `input_form` descriptor — the signature " + 'preparation reads. The descriptor rides `views: ["input_form"]` on pipelex-api >= 0.18.0; ' + "point the client at a runner that serves it." + ) raise InputPreparationError(msg) - if report.format != "json" or report.inputs is None: - msg = f'Cannot prepare inputs: expected a JSON inputs template, got "{report.format}".' - raise InputPreparationError(msg) - template = report.inputs + + selected_pipe_ref = _select_pipe_ref(report, input_form, _non_empty_string(pipe_ref)) + declared = {field.name: field for field in input_form[selected_pipe_ref].fields} ctx = _PrepareContext(client) rewritten = dict(inputs) for name, caller_value in inputs.items(): - entry = template.get(name) - if not isinstance(entry, dict) or "content" not in entry: - # Not a declared input (or an unexpected envelope) — pass through untouched. + field = declared.get(name) + if field is None: + # Not a declared input — pass through untouched. continue - content = cast("dict[str, Any]", entry)["content"] - rewritten[name] = await _resolve_node(ctx, content, caller_value) + if _is_explicit_envelope(caller_value): + # Unwrap, walk the content against the same node, re-wrap: the concept annotation + # rides through to the run, which accepts the envelope as an input. + envelope = cast("dict[str, Any]", caller_value) + walked = await _resolve_node(ctx, field, envelope["content"]) + rewritten[name] = {**envelope, "content": walked} + else: + rewritten[name] = await _resolve_node(ctx, field, caller_value) return PreparedInputs(inputs=rewritten, uploads=ctx.uploads) diff --git a/pipelex_sdk/product_models.py b/pipelex_sdk/product_models.py index 287a5ba..cbd4339 100644 --- a/pipelex_sdk/product_models.py +++ b/pipelex_sdk/product_models.py @@ -58,8 +58,8 @@ class MethodFile(BaseModel): This is the shape the hosted platform persists for a method's custom PipeFunc Python: a JSON `[{name, content}]` array in one wire string. It is deliberately distinct from two neighbours that look similar and are not: `MthdsFile` (`client.py`) is the *validate* - input, content plus an optional provenance URI; `MthdsFileItem` (`build_models.py`) is - the *build* closure entry. Three shapes for three surfaces — do not merge them. + input, content plus an optional provenance URI; `MthdsFileItem` (`crate_models.py`) is + the *crate* closure entry. Three shapes for three surfaces — do not merge them. """ model_config = ConfigDict(extra="allow") diff --git a/pipelex_sdk/validation_models.py b/pipelex_sdk/validation_models.py index 96c275f..d57925f 100644 --- a/pipelex_sdk/validation_models.py +++ b/pipelex_sdk/validation_models.py @@ -299,6 +299,15 @@ class PipelexValidationReport(ValidationReport): """The parsed bundle, carried opaquely: no published package declares its shape, so a type here could only be a copy free to drift from the runtime that emits it.""" + default_pipe_ref: str | None = None + """The qualified `pipe_ref` a caller gets by omitting the pipe selector, or `None` when the + closure declares none or several. + + Manifest-aware for a fetched package, which is what makes it outrank a `bundle_blueprint` read: + a published package may name its entry pipe in `METHODS.toml` alone, and the blueprint never + carries a manifest. Optional and read leniently — a runner that predates the field simply sends + nothing, so a consumer falls back (`prepare_inputs` reads the blueprint's `main_pipe` next).""" + pipe_io_contracts: PipeIOContracts = Field(default_factory=dict) """The per-pipe I/O contracts, typed by importing the standard's own client models. diff --git a/pyproject.toml b/pyproject.toml index 23f7b6a..59ea0da 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -337,7 +337,7 @@ convention = "google" "implicit-namespace-package", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) "private-member-access", # Unit tests legitimately probe private transport/error helpers (e.g. _request_product, _request_json) "import-private-name", # Unit tests legitimately import private module helpers under test (e.g. _parse_error_body) - "unused-method-argument", # Test-double methods match a Protocol signature; an unused param (e.g. a fake build_inputs ignoring `request`) is intentional + "unused-method-argument", # Test-double methods match a Protocol signature; an unused param (e.g. a fake validate ignoring `render`) is intentional "float-equality-comparison", # Tests assert exact float literals that round-trip exactly; `pytest.approx` would only add noise ] "examples/**/*.py" = [ diff --git a/tests/unit/test_build_inputs.py b/tests/unit/test_build_inputs.py deleted file mode 100644 index c262a00..0000000 --- a/tests/unit/test_build_inputs.py +++ /dev/null @@ -1,191 +0,0 @@ -"""The `build_inputs` route — the signature source `prepare_inputs` reads. Pins the verb + -path + body, the 200-verdict discipline (branch on `is_valid`), and the no-verdict throw. - -Ports the relevant slice of `pipelex-sdk-js/tests/build-routes.test.ts` for `/v1/build/inputs`. -`_send` is mocked; a produced verdict is a 200 discriminated on `is_valid`, a no-verdict -condition throws `ApiResponseError`. -""" - -import asyncio -import json - -import httpx -import pytest -from pydantic import ValidationError -from pytest_mock import MockerFixture, MockType - -from pipelex_sdk.build_models import BuildInputsRequest, BuildInputsResponseAdapter, BuildInputsValidReport, CrateInvalidReport, MthdsFileItem -from pipelex_sdk.client import PipelexAPIClient -from pipelex_sdk.errors import ApiResponseError - -_BASE_URL = "http://localhost:8081" - - -def _response(status_code: int, *, json_body: object | None = None) -> httpx.Response: - request = httpx.Request("POST", f"{_BASE_URL}/x") - if json_body is not None: - return httpx.Response(status_code, json=json_body, request=request) - return httpx.Response(status_code, request=request) - - -class TestBuildInputs: - def _client(self) -> PipelexAPIClient: - return PipelexAPIClient(api_key="test-token", base_url=_BASE_URL) - - def _mock_send(self, mocker: MockerFixture, client: PipelexAPIClient, response: httpx.Response) -> MockType: - return mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=response)) - - def test_posts_files_and_flags_to_build_inputs(self, mocker: MockerFixture) -> None: - client = self._client() - valid = { - "is_valid": True, - "pipe_ref": "demo.main", - "message": "ok", - "format": "json", - "explicit": True, - "inputs": {"photo": {"concept": "demo.Photo", "content": {"url": "https://mock/p.png"}}}, - } - send = self._mock_send(mocker, client, _response(200, json_body=valid)) - - report = asyncio.run( - client.build_inputs(BuildInputsRequest(files=[MthdsFileItem(content='domain = "demo"', source="b.mthds")], format="json", explicit=True)) - ) - - call = send.call_args - assert call.args[0] == "POST" - assert call.args[1] == f"{_BASE_URL}/v1/build/inputs" - body = json.loads(call.kwargs["content"]) - assert body == {"files": [{"content": 'domain = "demo"', "source": "b.mthds"}], "format": "json", "explicit": True} - assert isinstance(report, BuildInputsValidReport) - assert report.pipe_ref == "demo.main" - assert report.inputs is not None - - def test_invalid_closure_is_a_200_verdict(self, mocker: MockerFixture) -> None: - client = self._client() - invalid = { - "is_valid": False, - "message": "closure did not validate", - "validation_errors": [{"category": "blueprint_validation", "message": "unknown pipe type"}], - } - self._mock_send(mocker, client, _response(200, json_body=invalid)) - - report = asyncio.run(client.build_inputs(BuildInputsRequest(files=[MthdsFileItem(content="x")]))) - - assert isinstance(report, CrateInvalidReport) - assert report.validation_errors[0].message == "unknown pipe type" - - @pytest.mark.parametrize( - "body", - [ - # format=json but carrying no template at all — the flagged malformed-200 shape. - {"is_valid": True, "pipe_ref": "demo.main", "message": "ok", "format": "json", "explicit": True}, - # format=json but carrying the toml template (mismatched/opposite shape). - {"is_valid": True, "pipe_ref": "demo.main", "message": "ok", "format": "json", "explicit": True, "inputs_toml": "x = 1"}, - # format=toml but carrying no toml template. - {"is_valid": True, "pipe_ref": "demo.main", "message": "ok", "format": "toml", "explicit": True}, - ], - ) - def test_valid_report_without_matching_template_is_rejected(self, body: dict[str, object]) -> None: - # A valid verdict must carry the template its `format` selects — the adapter's - # malformed-200 guarantee, now honored for the template shape too. - with pytest.raises(ValidationError): - BuildInputsResponseAdapter.validate_python(body) - - def test_valid_toml_report_is_accepted(self) -> None: - body = {"is_valid": True, "pipe_ref": "demo.main", "message": "ok", "format": "toml", "explicit": True, "inputs_toml": "photo = 1"} - report = BuildInputsResponseAdapter.validate_python(body) - assert isinstance(report, BuildInputsValidReport) - assert report.inputs_toml == "photo = 1" - - def test_no_verdict_422_raises_api_response_error(self, mocker: MockerFixture) -> None: - client = self._client() - problem = {"detail": "Unknown pipe_ref", "error_type": "PipeNotFound"} - self._mock_send(mocker, client, _response(422, json_body=problem)) - - with pytest.raises(ApiResponseError) as exc_info: - asyncio.run(client.build_inputs(BuildInputsRequest(files=[MthdsFileItem(content="x")], pipe_ref="demo.nope"))) - assert exc_info.value.status == 422 - - # ── the closure selector (files XOR method_ref, NO method_id) ──── - - def test_method_ref_closure_rides_the_body_with_the_fetch_budget(self, mocker: MockerFixture) -> None: - """An address closure may make the server clone before answering, so the request gets - the fetch-sized budget instead of the 30s management one. - """ - client = self._client() - valid: dict[str, object] = { - "is_valid": True, - "pipe_ref": "documents.summarize", - "message": "ok", - "format": "json", - "explicit": False, - "inputs": {}, - } - send = self._mock_send(mocker, client, _response(200, json_body=valid)) - - report = asyncio.run(client.build_inputs(BuildInputsRequest(method_ref="github.com/Pipelex/methods/documents@v0.1.0"))) - - call = send.call_args - body = json.loads(call.kwargs["content"]) - assert body == {"method_ref": "github.com/Pipelex/methods/documents@v0.1.0", "format": "json", "explicit": False} - assert call.kwargs["request_timeout"] == 180.0 - assert isinstance(report, BuildInputsValidReport) - - def test_inline_files_keep_the_management_budget(self, mocker: MockerFixture) -> None: - client = self._client() - valid: dict[str, object] = {"is_valid": True, "pipe_ref": "demo.main", "message": "ok", "format": "json", "explicit": False, "inputs": {}} - send = self._mock_send(mocker, client, _response(200, json_body=valid)) - - asyncio.run(client.build_inputs(BuildInputsRequest(files=[MthdsFileItem(content="x")]))) - - assert send.call_args.kwargs["request_timeout"] == 30.0 - - @pytest.mark.parametrize( - "kwargs", - [ - {}, - {"files": [MthdsFileItem(content="x")], "method_ref": "github.com/x/y@v1"}, - ], - ) - def test_request_construction_enforces_files_xor_method_ref(self, kwargs: dict[str, object]) -> None: - with pytest.raises(ValidationError, match="exactly one"): - BuildInputsRequest.model_validate(kwargs) - - @pytest.mark.parametrize( - "kwargs", - [ - {"files": []}, - {"method_ref": ""}, - {"method_ref": " "}, - {"files": [], "method_ref": " "}, - ], - ) - def test_empty_closure_selectors_are_absent_and_fail_the_xor(self, kwargs: dict[str, object]) -> None: - """`files=[]` and a blank `method_ref` select nothing — normalized to absent before the - XOR, so an unusable value is refused at construction instead of reaching the wire. - """ - with pytest.raises(ValidationError, match="exactly one"): - BuildInputsRequest.model_validate(kwargs) - - def test_blank_method_ref_beside_files_is_simply_absent(self, mocker: MockerFixture) -> None: - """Same rule as the run boundary: an empty selector beside a real one is absent, not a - conflict — the closure is `files`, the empty key is not sent, the budget stays 30s. - """ - request = BuildInputsRequest(files=[MthdsFileItem(content="x")], method_ref="") - assert request.method_ref is None - - client = self._client() - valid: dict[str, object] = {"is_valid": True, "pipe_ref": "demo.main", "message": "ok", "format": "json", "explicit": False, "inputs": {}} - send = self._mock_send(mocker, client, _response(200, json_body=valid)) - asyncio.run(client.build_inputs(request)) - - body = json.loads(send.call_args.kwargs["content"]) - assert "method_ref" not in body - assert send.call_args.kwargs["request_timeout"] == 30.0 - - def test_method_id_is_refused_with_a_teaching_error(self) -> None: - """The `/v1/build/*` projections take no `method_id` — a teaching error beats pydantic - silently ignoring the unknown key for a caller migrating off the by-id habit. - """ - with pytest.raises(ValidationError, match="build_inputs takes no method_id"): - BuildInputsRequest.model_validate({"method_id": "mt_1"}) diff --git a/tests/unit/test_crate_routes.py b/tests/unit/test_crate_routes.py index 6d0e88e..9f620b9 100644 --- a/tests/unit/test_crate_routes.py +++ b/tests/unit/test_crate_routes.py @@ -14,9 +14,8 @@ from pydantic import ValidationError from pytest_mock import MockerFixture, MockType -from pipelex_sdk.build_models import CrateInvalidReport, MthdsFileItem from pipelex_sdk.client import PipelexAPIClient -from pipelex_sdk.crate_models import CodegenRequest, CodegenValidReport, ResolveRequest, ResolveValidReport +from pipelex_sdk.crate_models import CodegenRequest, CodegenValidReport, CrateInvalidReport, MthdsFileItem, ResolveRequest, ResolveValidReport from pipelex_sdk.errors import ApiResponseError _BASE_URL = "http://localhost:8081" diff --git a/tests/unit/test_prepare_inputs.py b/tests/unit/test_prepare_inputs.py index 18c79a8..6ba5a93 100644 --- a/tests/unit/test_prepare_inputs.py +++ b/tests/unit/test_prepare_inputs.py @@ -1,50 +1,109 @@ -"""`prepare_inputs` — signature-driven input preparation. Cases derive from the shared -behavior matrix (`wip/upload/behavior-matrix.md`): file-bearing positions are found from the -explicit template's canonical content shape (a `{"url": …}` dict), assets are uploaded and -rewritten to `pipelex-storage://` in `url`, http(s)/storage references pass through, dedup -keys on source identity, and the call is copy-on-write. - -Ports `pipelex-sdk-js/tests/prepare-inputs.test.ts`. The fake client returns a canned explicit -template from `build_inputs` and a counting `upload`; one wiring test drives the real client. +"""`prepare_inputs` — signature-driven input preparation over the input-form descriptor. + +Cases derive from the shared behavior matrix (`wip/upload/behavior-matrix.md`) and port +`pipelex-sdk-js/tests/prepare-inputs.test.ts`: file-bearing positions come from the DESCRIPTOR's +declared kind (`document` / `image`), assets are uploaded and rewritten to `pipelex-storage://` +in `url`, http(s)/storage references pass through, dedup keys on source identity, and the call +is copy-on-write. + +The fake client returns a canned `PipelexValidationReport` from `validate` and records the call, +so the request shape is asserted and not just the outcome; one wiring test drives the real client. """ import asyncio import base64 +import json from pathlib import Path from typing import Any import httpx import pytest +from mthds.protocol.input_form import ( + DocumentField, + DocumentItem, + ImageField, + InputFormField, + ListField, + ObjectField, + PipeInputFormDescriptor, + TextField, + UnknownField, +) +from mthds.protocol.pipe_io_contracts import PresenceMarker from pytest_mock import MockerFixture -from pipelex_sdk.build_models import BuildInputsRequest, BuildInputsResponse, BuildInputsValidReport, CrateInvalidReport, MthdsFileItem from pipelex_sdk.client import PipelexAPIClient +from pipelex_sdk.crate_models import MthdsFileItem from pipelex_sdk.errors import ApiResponseError, InputPreparationError, RejectedAssetError from pipelex_sdk.prepare_inputs import prepare_inputs from pipelex_sdk.product_models import UploadedFile, UploadInput +from pipelex_sdk.validation_models import PipelexInvalidReport, PipelexValidationReport, PipelexValidationResult _BASE_URL = "http://localhost:8081" _FILES = [MthdsFileItem(content='domain = "demo"')] +_PIPE_REF = "demo.main" + + +def _required(**kwargs: Any) -> dict[str, Any]: + """The pipe-slot facts every TOP-LEVEL field must state (`required` restates `presence`).""" + return {"required": True, "presence": PresenceMarker.PLAIN, "gating": True, **kwargs} + + +def _optional(**kwargs: Any) -> dict[str, Any]: + """An optional slot: `required: false`, `presence: optional`, and it never gates.""" + return {"required": False, "presence": PresenceMarker.OPTIONAL, "gating": False, **kwargs} + + +def _form(*fields: InputFormField, pipe_ref: str = _PIPE_REF) -> dict[str, PipeInputFormDescriptor]: + return {pipe_ref: PipeInputFormDescriptor(fields=list(fields))} -def _entry(concept: str, content: Any) -> dict[str, Any]: - return {"concept": concept, "content": content} +def _report( + input_form: dict[str, PipeInputFormDescriptor] | None, + *, + bundle_blueprint: dict[str, Any] | None = None, + default_pipe_ref: str | None = None, +) -> PipelexValidationReport: + return PipelexValidationReport( + is_valid=True, + bundle_blueprint=bundle_blueprint if bundle_blueprint is not None else {}, + default_pipe_ref=default_pipe_ref, + input_form=input_form, + ) class _FakePrepareClient: - """Fake client: `build_inputs` returns the given envelope template; `upload` counts calls.""" + """Fake client: `validate` returns the given report and records the call; `upload` counts calls.""" - def __init__(self, template: dict[str, Any], *, report: BuildInputsResponse | None = None, upload_error: Exception | None = None) -> None: - self._template = template - self._report = report + def __init__(self, result: PipelexValidationResult, *, upload_error: Exception | None = None) -> None: + self._result = result self._upload_error = upload_error self.upload_calls: list[UploadInput] = [] + self.validate_calls: list[dict[str, Any]] = [] self._counter = 0 - async def build_inputs(self, request: BuildInputsRequest) -> BuildInputsResponse: - if self._report is not None: - return self._report - return BuildInputsValidReport(is_valid=True, pipe_ref="demo.main", message="ok", format="json", explicit=True, inputs=self._template) + async def validate( + self, + mthds_contents: list[str] | None = None, + allow_signatures: bool = False, + mthds_sources: list[str] | None = None, + render: list[str] | None = None, + views: list[str] | None = None, + *, + method_ref: str | None = None, + method_id: str | None = None, + ) -> PipelexValidationResult: + self.validate_calls.append( + { + "mthds_contents": mthds_contents, + "allow_signatures": allow_signatures, + "mthds_sources": mthds_sources, + "views": views, + "method_ref": method_ref, + "method_id": method_id, + } + ) + return self._result async def upload(self, upload_input: UploadInput) -> UploadedFile: if self._upload_error is not None: @@ -54,9 +113,168 @@ async def upload(self, upload_input: UploadInput) -> UploadedFile: return UploadedFile(uri=f"pipelex-storage://user/assets/{self._counter}.bin", filename=upload_input.filename) +def _image_client(name: str = "photo", **upload_error: Any) -> _FakePrepareClient: + return _FakePrepareClient(_report(_form(ImageField(name=name, **_required()))), **upload_error) + + class TestPrepareInputs: + # ── The signature call ──────────────────────────────────────────────── + + def test_asks_validate_for_the_input_form_view(self) -> None: + client = _image_client() + + asyncio.run(prepare_inputs(client, files=_FILES, inputs={})) + + call = client.validate_calls[0] + assert call["views"] == ["input_form"] + assert call["allow_signatures"] is True + assert call["mthds_contents"] == ['domain = "demo"'] + # No file names a source, so none is synthesized — the server never sees a + # length-mismatched `mthds_sources` array. + assert call["mthds_sources"] is None + + def test_labels_every_content_once_any_file_names_a_source(self) -> None: + client = _image_client() + files = [MthdsFileItem(content="a"), MthdsFileItem(content="b", source="b.mthds")] + + asyncio.run(prepare_inputs(client, files=files, inputs={})) + + assert client.validate_calls[0]["mthds_sources"] == ["inline://file-1.mthds", "b.mthds"] + + def test_method_ref_is_a_server_side_pass_through(self) -> None: + client = _image_client() + + asyncio.run(prepare_inputs(client, method_ref="github.com/Pipelex/methods/documents", inputs={})) + + call = client.validate_calls[0] + assert call["method_ref"] == "github.com/Pipelex/methods/documents" + assert call["mthds_contents"] is None + assert call["views"] == ["input_form"] + + def test_method_id_is_a_server_side_pass_through(self) -> None: + client = _image_client() + + asyncio.run(prepare_inputs(client, method_id="mt_abc123", inputs={})) + + call = client.validate_calls[0] + assert call["method_id"] == "mt_abc123" + assert call["mthds_contents"] is None + + # ── The three selectors ─────────────────────────────────────────────── + + def test_no_selector_is_refused_before_any_request(self) -> None: + client = _image_client() + + with pytest.raises(InputPreparationError, match="no method selector"): + asyncio.run(prepare_inputs(client, inputs={"photo": bytes([1])})) + assert client.validate_calls == [] + assert client.upload_calls == [] + + @pytest.mark.parametrize( + "kwargs", + [ + {"files": _FILES, "method_ref": "github.com/o/r"}, + {"files": _FILES, "method_id": "mt_1"}, + {"method_ref": "github.com/o/r", "method_id": "mt_1"}, + ], + ) + def test_several_selectors_are_refused_before_any_request(self, kwargs: dict[str, Any]) -> None: + client = _image_client() + + with pytest.raises(InputPreparationError, match="exactly one method selector"): + asyncio.run(prepare_inputs(client, inputs={}, **kwargs)) + assert client.validate_calls == [] + + def test_empty_selectors_are_absent_beside_a_real_one(self) -> None: + # `files=[]` and a blank `method_id` select nothing, so they may sit beside a real + # `method_ref` without tripping the XOR — the run options' empty-as-absent rule. + client = _image_client() + + asyncio.run(prepare_inputs(client, files=[], method_ref="github.com/o/r", method_id=" ", inputs={})) + + assert client.validate_calls[0]["method_ref"] == "github.com/o/r" + + def test_only_empty_selectors_is_no_selector(self) -> None: + client = _image_client() + + with pytest.raises(InputPreparationError, match="no method selector"): + asyncio.run(prepare_inputs(client, files=[], method_ref="", inputs={})) + + # ── Pipe selection ──────────────────────────────────────────────────── + + def test_uses_the_single_declared_pipe_when_no_ref_is_given(self) -> None: + client = _image_client() + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": bytes([1])})) + + assert prepared.inputs == {"photo": {"url": "pipelex-storage://user/assets/1.bin"}} + + def test_typed_default_pipe_ref_outranks_the_blueprint(self) -> None: + input_form = { + "demo.first": PipeInputFormDescriptor(fields=[TextField(name="photo", **_required())]), + "demo.second": PipeInputFormDescriptor(fields=[ImageField(name="photo", **_required())]), + } + client = _FakePrepareClient(_report(input_form, bundle_blueprint={"domain": "demo", "main_pipe": "first"}, default_pipe_ref="demo.second")) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": bytes([1])})) + + # `demo.second` declares `photo` as an image; `demo.first` declares it as text. + assert prepared.inputs == {"photo": {"url": "pipelex-storage://user/assets/1.bin"}} + + def test_falls_back_to_the_blueprint_main_pipe_qualified_by_its_domain(self) -> None: + input_form = { + "demo.first": PipeInputFormDescriptor(fields=[ImageField(name="photo", **_required())]), + "demo.second": PipeInputFormDescriptor(fields=[TextField(name="photo", **_required())]), + } + client = _FakePrepareClient(_report(input_form, bundle_blueprint={"domain": "demo", "main_pipe": "first"})) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": bytes([1])})) + + assert prepared.inputs == {"photo": {"url": "pipelex-storage://user/assets/1.bin"}} + + def test_explicit_pipe_ref_wins(self) -> None: + input_form = { + "demo.first": PipeInputFormDescriptor(fields=[TextField(name="photo", **_required())]), + "demo.second": PipeInputFormDescriptor(fields=[ImageField(name="photo", **_required())]), + } + client = _FakePrepareClient(_report(input_form, default_pipe_ref="demo.first")) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, pipe_ref="demo.second", inputs={"photo": bytes([1])})) + + assert prepared.inputs == {"photo": {"url": "pipelex-storage://user/assets/1.bin"}} + + def test_bare_pipe_ref_is_refused_naming_the_qualified_candidates(self) -> None: + client = _image_client() + + with pytest.raises(InputPreparationError, match="must be qualified") as exc_info: + asyncio.run(prepare_inputs(client, files=_FILES, pipe_ref="main", inputs={})) + assert _PIPE_REF in str(exc_info.value) + + def test_unknown_pipe_ref_is_refused_naming_the_candidates(self) -> None: + client = _image_client() + + with pytest.raises(InputPreparationError, match="declares no pipe") as exc_info: + asyncio.run(prepare_inputs(client, files=_FILES, pipe_ref="demo.absent", inputs={})) + assert _PIPE_REF in str(exc_info.value) + + def test_several_pipes_and_no_default_is_an_honest_refusal(self) -> None: + # The manifest-only `main_pipe` gap: a fetched package may name its entry pipe in + # METHODS.toml alone, which the report never carries. The error lists the candidates + # so the caller's fix is one line. + input_form = { + "demo.first": PipeInputFormDescriptor(fields=[]), + "demo.second": PipeInputFormDescriptor(fields=[]), + } + client = _FakePrepareClient(_report(input_form)) + + with pytest.raises(InputPreparationError, match="no single default pipe") as exc_info: + asyncio.run(prepare_inputs(client, method_ref="github.com/Pipelex/methods/documents", inputs={})) + assert "demo.first, demo.second" in str(exc_info.value) + + # ── The descriptor-guided walk ──────────────────────────────────────── + def test_uploads_top_level_image_bytes(self) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": bytes([1, 2, 3])})) @@ -65,7 +283,7 @@ def test_uploads_top_level_image_bytes(self) -> None: assert prepared.uploads[0].uri == "pipelex-storage://user/assets/1.bin" def test_passes_http_url_through(self) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": "https://example.com/real.png"})) @@ -74,7 +292,7 @@ def test_passes_http_url_through(self) -> None: assert client.upload_calls == [] def test_passes_existing_storage_uri_through(self) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": "pipelex-storage://user/assets/already.png"})) @@ -82,7 +300,7 @@ def test_passes_existing_storage_uri_through(self) -> None: assert prepared.uploads == [] def test_decodes_and_uploads_data_url(self) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": "data:image/png;base64,AQIDBA=="})) @@ -100,7 +318,7 @@ def test_decodes_and_uploads_data_url(self) -> None: def test_malformed_base64_data_url_raises_typed_error(self, data_url: str) -> None: # A malformed base64 data URL must surface as the typed `InputPreparationError` # (never a raw binascii.Error), and must never upload silently-corrupted bytes. - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() with pytest.raises(InputPreparationError): asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": data_url})) @@ -109,14 +327,14 @@ def test_malformed_base64_data_url_raises_typed_error(self, data_url: str) -> No def test_percent_encoded_binary_data_url_keeps_exact_bytes(self) -> None: # A non-base64 data URL carrying percent-encoded binary must upload its exact bytes; # decoding as UTF-8 text first would corrupt any byte >= 0x80 (e.g. %FF). - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": "data:application/octet-stream,%00%ff%01"})) assert base64.b64decode(client.upload_calls[0].data) == bytes([0x00, 0xFF, 0x01]) - def test_uploads_each_element_of_declared_multiple(self) -> None: - client = _FakePrepareClient({"exhibits": _entry("demo.Exhibit", [{"url": "https://mock/d.pdf"}])}) + def test_uploads_each_element_of_a_declared_list(self) -> None: + client = _FakePrepareClient(_report(_form(ListField(name="exhibits", item=DocumentItem(required=True), **_required())))) prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"exhibits": [bytes([1]), bytes([2])]})) @@ -124,41 +342,43 @@ def test_uploads_each_element_of_declared_multiple(self) -> None: assert len(prepared.uploads) == 2 def test_leaves_text_input_untouched(self) -> None: - client = _FakePrepareClient({"question": _entry("demo.Question", {"text": "text_value"})}) + client = _FakePrepareClient(_report(_form(TextField(name="question", **_required())))) prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"question": "notes/summary.txt"})) assert prepared.inputs == {"question": "notes/summary.txt"} assert client.upload_calls == [] - def test_uploads_only_nested_image_of_structured_input(self) -> None: - client = _FakePrepareClient( - {"dossier": _entry("demo.Dossier", {"title": "title_value", "cover": {"url": "https://mock/c.png", "mime_type": "image/png"}})} + def test_uploads_only_the_nested_image_of_a_structured_input(self) -> None: + dossier = ObjectField( + name="dossier", + fields=[TextField(name="title", required=True), ImageField(name="cover", required=True)], + **_required(), ) + client = _FakePrepareClient(_report(_form(dossier))) prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"dossier": {"title": "Q3 report", "cover": bytes([7, 7])}})) assert prepared.inputs == {"dossier": {"title": "Q3 report", "cover": {"url": "pipelex-storage://user/assets/1.bin"}}} assert len(prepared.uploads) == 1 - def test_does_not_path_interpret_bare_string_at_dynamic_input(self) -> None: - client = _FakePrepareClient({"freeform": _entry("native.Anything", {"whatever": "value"})}) + def test_preserves_sibling_keys_of_canonical_file_content(self) -> None: + client = _image_client() - prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"freeform": "resembles/a/path"})) + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": {"url": bytes([1]), "mime_type": "image/png"}})) - assert prepared.inputs == {"freeform": "resembles/a/path"} - assert client.upload_calls == [] + assert prepared.inputs == {"photo": {"url": "pipelex-storage://user/assets/1.bin", "mime_type": "image/png"}} - def test_uploads_canonical_image_nested_in_dynamic(self) -> None: - client = _FakePrepareClient({"data": _entry("native.Composite", {"text": "t", "images": [{"url": "https://mock/i.png"}]})}) + def test_copies_through_object_keys_the_descriptor_does_not_name(self) -> None: + dossier = ObjectField(name="dossier", fields=[ImageField(name="cover", required=True)], **_required()) + client = _FakePrepareClient(_report(_form(dossier))) - prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"data": {"text": "hi", "images": [bytes([5])]}})) + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"dossier": {"cover": bytes([1]), "note": "kept"}})) - assert prepared.inputs == {"data": {"text": "hi", "images": [{"url": "pipelex-storage://user/assets/1.bin"}]}} - assert len(prepared.uploads) == 1 + assert prepared.inputs["dossier"]["note"] == "kept" def test_dedups_by_source_identity(self) -> None: - client = _FakePrepareClient({"exhibits": _entry("demo.Exhibit", [{"url": "https://mock/d.pdf"}])}) + client = _FakePrepareClient(_report(_form(ListField(name="exhibits", item=DocumentItem(required=True), **_required())))) shared = bytes([9, 9, 9]) prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"exhibits": [shared, shared]})) @@ -168,7 +388,7 @@ def test_dedups_by_source_identity(self) -> None: assert exhibits[0]["url"] == exhibits[1]["url"] def test_is_copy_on_write(self) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() original = {"photo": bytes([1, 2, 3])} asyncio.run(prepare_inputs(client, files=_FILES, inputs=original)) @@ -176,14 +396,14 @@ def test_is_copy_on_write(self) -> None: assert original["photo"] == bytes([1, 2, 3]) def test_passes_through_undeclared_input(self) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": "https://example.com/p.png", "stray": "left alone"})) assert prepared.inputs["stray"] == "left alone" def test_uploads_real_local_path(self, tmp_path: Path) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() path = tmp_path / "shot.png" path.write_bytes(bytes([1, 2, 3, 4])) @@ -192,8 +412,110 @@ def test_uploads_real_local_path(self, tmp_path: Path) -> None: assert prepared.inputs == {"photo": {"url": "pipelex-storage://user/assets/1.bin"}} assert client.upload_calls[0].content_type == "image/png" + def test_a_shape_mismatch_passes_through_for_the_run_to_reject(self) -> None: + # A scalar where the descriptor declares an object: preparation never second-guesses + # the signature, so the value rides through and the run answers for it. + dossier = ObjectField(name="dossier", fields=[ImageField(name="cover", required=True)], **_required()) + client = _FakePrepareClient(_report(_form(dossier))) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"dossier": "not an object"})) + + assert prepared.inputs == {"dossier": "not an object"} + assert client.upload_calls == [] + + # ── The two misclassifications of L-260826-ddd843 ───────────────────── + + def test_uploads_an_optional_top_level_file_field_when_supplied(self) -> None: + client = _FakePrepareClient(_report(_form(DocumentField(name="appendix", **_optional())))) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"appendix": bytes([3])})) + + assert prepared.inputs == {"appendix": {"url": "pipelex-storage://user/assets/1.bin"}} + + def test_uploads_an_optional_nested_file_field(self) -> None: + # First edge: the required-only inputs template never rendered an optional nested + # file field, so its position was invisible and the caller's local path travelled to + # the runner as a literal string. The descriptor states `required: false` and the + # walk enters it. + dossier = ObjectField( + name="dossier", + fields=[TextField(name="title", required=True), ImageField(name="cover", required=False)], + **_required(), + ) + client = _FakePrepareClient(_report(_form(dossier))) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"dossier": {"title": "t", "cover": bytes([7])}})) + + assert prepared.inputs["dossier"]["cover"] == {"url": "pipelex-storage://user/assets/1.bin"} + + def test_does_not_read_a_text_field_merely_named_url_from_disk(self) -> None: + # Second edge: the template marked a file position by rendering a `url`-bearing dict — + # a side effect of the field's NAME, not of its concept — so a path-shaped text value + # was uploaded. `kind: "text"` ends that. + client = _FakePrepareClient(_report(_form(TextField(name="url", **_required())))) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"url": "notes/summary.txt"})) + + assert prepared.inputs == {"url": "notes/summary.txt"} + assert client.upload_calls == [] + + def test_does_not_enter_a_dynamic_input(self) -> None: + # A `Dynamic` / `Composite` input is `kind: "unknown"` — the standard's escape hatch — + # and the walk does not enter it, so a canonical file dict nested inside is NOT + # uploaded. Uploading on the strength of a `url` key is the value-shape guess this + # walk removes; such a caller uses `upload_file` first and passes the storage URI. + client = _FakePrepareClient(_report(_form(UnknownField(name="data", **_required())))) + nested = {"text": "hi", "images": [{"url": "https://mock/i.png"}]} + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"data": nested})) + + assert prepared.inputs == {"data": nested} + assert client.upload_calls == [] + + def test_does_not_path_interpret_a_bare_string_at_a_dynamic_input(self) -> None: + client = _FakePrepareClient(_report(_form(UnknownField(name="freeform", **_required())))) + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"freeform": "resembles/a/path"})) + + assert prepared.inputs == {"freeform": "resembles/a/path"} + assert client.upload_calls == [] + + # ── The explicit `{concept, content}` envelope ──────────────────────── + + def test_unwraps_and_rewraps_the_explicit_envelope(self) -> None: + client = _image_client() + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": {"concept": "native.Image", "content": bytes([1])}})) + + # The concept annotation rides through to the run; only `content` is rewritten. + assert prepared.inputs == {"photo": {"concept": "native.Image", "content": {"url": "pipelex-storage://user/assets/1.bin"}}} + + def test_walks_inside_an_envelope_carrying_a_structured_content(self) -> None: + dossier = ObjectField( + name="dossier", + fields=[TextField(name="title", required=True), ImageField(name="cover", required=True)], + **_required(), + ) + client = _FakePrepareClient(_report(_form(dossier))) + envelope = {"concept": "demo.Dossier", "content": {"title": "t", "cover": bytes([7])}} + + prepared = asyncio.run(prepare_inputs(client, files=_FILES, inputs={"dossier": envelope})) + + assert prepared.inputs["dossier"]["concept"] == "demo.Dossier" + assert prepared.inputs["dossier"]["content"]["cover"] == {"url": "pipelex-storage://user/assets/1.bin"} + + def test_a_dict_that_is_not_exactly_concept_and_content_is_not_an_envelope(self) -> None: + # The envelope test matches the runtime's `_is_explicit`: keys EXACTLY `concept` and + # `content`. A third key means it is ordinary content, not an envelope. + client = _image_client() + + with pytest.raises(InputPreparationError, match="Unsupported value at a file input"): + asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": {"concept": "x", "content": bytes([1]), "extra": 1}})) + + # ── Failures, all raised before any run exists ──────────────────────── + def test_raises_for_unrecognized_value_at_file_position(self) -> None: - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}) + client = _image_client() # A plain object that is neither a canonical {url} content nor bytes — a realistic # caller typo — must surface as a typed error, not pass through unresolved. @@ -201,40 +523,48 @@ def test_raises_for_unrecognized_value_at_file_position(self) -> None: asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": {"mimeType": "image/png", "bytes": [1, 2, 3]}})) assert client.upload_calls == [] - def test_raises_when_signature_does_not_resolve(self) -> None: - report = CrateInvalidReport(is_valid=False, message="closure did not validate", validation_errors=[]) - client = _FakePrepareClient({}, report=report) + def test_raises_when_the_signature_does_not_resolve(self) -> None: + invalid = PipelexInvalidReport(is_valid=False, message="closure did not validate", validation_errors=[]) + client = _FakePrepareClient(invalid) - with pytest.raises(InputPreparationError): + with pytest.raises(InputPreparationError, match="the method signature did not resolve"): + asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": bytes([1])})) + + def test_a_report_without_the_descriptor_is_an_error_not_a_silent_no_op(self) -> None: + # Never a silent degrade to "no uploads": without the descriptor there is no signature + # to prepare against, and the caller's local path would travel to the runner verbatim. + client = _FakePrepareClient(_report(None)) + + with pytest.raises(InputPreparationError, match="carries no `input_form` descriptor"): asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": bytes([1])})) + assert client.upload_calls == [] def test_surfaces_rejected_asset_before_returning(self) -> None: error = ApiResponseError( "HTTP 413", api_url=f"{_BASE_URL}/v1/upload", status=413, status_text="Payload Too Large", response_body="", server_message="too big" ) - client = _FakePrepareClient({"photo": _entry("demo.Photo", {"url": "https://mock/p.png"})}, upload_error=error) + client = _image_client(upload_error=error) with pytest.raises(RejectedAssetError): asyncio.run(prepare_inputs(client, files=_FILES, inputs={"photo": bytes([1])})) + # ── Wiring ──────────────────────────────────────────────────────────── + def test_wires_through_the_real_client(self, mocker: MockerFixture) -> None: client = PipelexAPIClient(api_key="test-token", base_url=_BASE_URL) - build_body = { + validate_body = { "is_valid": True, - "pipe_ref": "demo.main", - "message": "ok", - "format": "json", - "explicit": True, - "inputs": {"photo": {"concept": "demo.Photo", "content": {"url": "https://mock/p.png"}}}, + "bundle_blueprint": {}, + "input_form": {_PIPE_REF: {"fields": [{"kind": "image", "name": "photo", "required": True, "presence": "plain", "gating": True}]}}, } upload_body = {"uri": "pipelex-storage://user/assets/1.bin", "filename": "upload.bin"} request = httpx.Request("POST", f"{_BASE_URL}/x") - mocker.patch.object( + send = mocker.patch.object( client, "_send", mocker.AsyncMock( side_effect=[ - httpx.Response(200, json=build_body, request=request), + httpx.Response(200, json=validate_body, request=request), httpx.Response(200, json=upload_body, request=request), ] ), @@ -244,3 +574,22 @@ def test_wires_through_the_real_client(self, mocker: MockerFixture) -> None: assert prepared.inputs == {"photo": {"url": "pipelex-storage://user/assets/1.bin"}} assert len(prepared.uploads) == 1 + assert send.await_args_list[0].args[1] == f"{_BASE_URL}/v1/validate" + + def test_wires_a_method_ref_through_the_real_client(self, mocker: MockerFixture) -> None: + client = PipelexAPIClient(api_key="test-token", base_url=_BASE_URL) + validate_body = { + "is_valid": True, + "bundle_blueprint": {}, + "input_form": {_PIPE_REF: {"fields": [{"kind": "text", "name": "question", "required": True, "presence": "plain", "gating": True}]}}, + } + request = httpx.Request("POST", f"{_BASE_URL}/x") + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=httpx.Response(200, json=validate_body, request=request))) + + asyncio.run(client.prepare_inputs(method_ref="github.com/o/r", inputs={"question": "hi"})) + + body = json.loads(send.await_args_list[0].kwargs["content"]) + assert body["method_ref"] == "github.com/o/r" + assert body["views"] == ["input_form"] + assert body["allow_signatures"] is True + assert "mthds_contents" not in body diff --git a/tests/unit/test_validation_contract.py b/tests/unit/test_validation_contract.py index b009a00..8d98e90 100644 --- a/tests/unit/test_validation_contract.py +++ b/tests/unit/test_validation_contract.py @@ -378,6 +378,25 @@ def test_valid_arm_carries_warnings_liftable_pipes_and_input_form(self) -> None: # Keyed exactly like `pipe_io_contracts` — the same `pipe_ref` set addresses both artifacts. assert set(report.input_form) == set(report.pipe_io_contracts) + def test_default_pipe_ref_is_absent_by_default(self) -> None: + """A runner that predates the field simply sends nothing — the report still parses.""" + report = _parse(VALID_BODY_WITH_VIEWS) + assert isinstance(report, PipelexValidationReport) + assert report.default_pipe_ref is None + + def test_default_pipe_ref_reads_as_the_qualified_ref_when_served(self) -> None: + body = {**VALID_BODY_WITH_VIEWS, "default_pipe_ref": "legal_contracts.summarize"} + report = _parse(body) + assert isinstance(report, PipelexValidationReport) + assert report.default_pipe_ref == "legal_contracts.summarize" + + def test_default_pipe_ref_reads_an_explicit_null(self) -> None: + """`null` is how the runner says the closure declares no single default — not a parse failure.""" + body = {**VALID_BODY_WITH_VIEWS, "default_pipe_ref": None} + report = _parse(body) + assert isinstance(report, PipelexValidationReport) + assert report.default_pipe_ref is None + def test_input_form_reads_as_the_standards_models(self) -> None: """The descriptor is typed by import: nodes narrow on `kind`, and the recursion is typed through.""" report = _parse(VALID_BODY_WITH_VIEWS) diff --git a/wip/prepare-inputs-selectors/plan.md b/wip/prepare-inputs-selectors/plan.md new file mode 100644 index 0000000..c6e601a --- /dev/null +++ b/wip/prepare-inputs-selectors/plan.md @@ -0,0 +1,40 @@ +--- +status: draft +item: L-260829-8a25d5 +--- + +# `prepare_inputs`: three selectors, signature from the input-form descriptor + +The Python half of the workspace campaign retiring `/v1/build/*` (epic `L-260829-848001`, `wip/build-retirement/` at the workspace root). + +## The design of record is the JS one + +This repo writes no second design. `pipelex-sdk-js/wip/prepare-inputs-selectors/design.md` holds the investigation, Louis's ruling of 2026-08-29, the surface, the walk, the pipe-selection ladder, the error wordings, the alternatives rejected and the known limits — and it names this item's mandate explicitly: `prepare_inputs` lands the same surface, and because `build_inputs` and `BuildInputsRequest` exist here only to back it, this item deletes them. + +The JS twin (`L-260829-300c50`) landed as `pipelex-sdk-js` PR #42 (`bea4632`) and is the reference implementation. Divergence from it is a bug unless recorded below. + +## What this repo did + +- `prepare_inputs(client, *, files=None, method_ref=None, method_id=None, pipe_ref=None, inputs)` — keyword parameters rather than JS's `never`-pinned discriminated union, matching how `validate` already takes its selectors here. Empty-as-absent and the exactly-one check run before any request, raising `InputPreparationError`. +- One `validate(..., allow_signatures=True, views=["input_form"])` per call; the walk is a `match` over the descriptor's item classes rather than over `kind`, because each `*Field` derives from its `*Item` — one set of patterns covers the named layer (top level, `object.fields`) and the nameless one (`list.item`), and it narrows for pyright where matching on `node.kind` would not. +- `PipelexValidationReport.default_pipe_ref` added ahead of the server (`L-260829-0208c7`), as JS did. +- `build_inputs` and the `BuildInputs*` models deleted. `build_models.py` deleted with them: the three survivors it also held — `MthdsFileItem`, `CrateRequestBase`, `CrateInvalidReport` — moved to `crate_models.py`, beside the routes that still use them. +- The explicit `{concept, content}` envelope is now accepted. This was a **pre-existing parity gap**, not part of the item's letter: JS gained it in an earlier release and Python never did, so the two SDKs would not have been identical after the fix. Ruled in scope with the user on 2026-08-30. + +## Decisions taken here + +| Decision | Why | +|---|---| +| Keyword selectors, not a request model | The repo's own `validate` idiom; `architecture.md` already records the JS-vs-Python signature-shape divergence as idiomatic per language. | +| `build_inputs` deleted, where JS kept `buildInputs` | JS has a wrapper family (`buildOutput`, `buildRunner`, `concept`, `pipeSpec`) retiring together under `L-260829-eefc3f`. Python only ever had this one, added in 0.5.0 solely to back `prepare_inputs`. | +| `build_models.py` folded into `crate_models.py` | A module named for the build routes cannot go on owning the crate envelope after those routes leave. | +| A local `_non_empty_string`, not `client._normalized_selector` | That helper is private to the client boundary and raises `PipelineRequestError`; every failure of this module owes an `InputPreparationError`. | +| No fetch budget on the signature call | `validate` already rides the 20-minute ceiling; the 3-minute budget exists to *raise* the ~30s poll-ceiling routes. JS implemented this and reverted it — do not re-add. | + +## What this supersedes + +`wip/pr-11-review-notes.md` recorded a nested-file limitation of the old template walk: a top-level `url` key caused an early return, so a sibling file field went un-uploaded, and the note explained that shape refinement was ambiguous because the walk dropped the envelope's `concept`. The descriptor walk removes that class of problem structurally — position and kind are stated, never inferred — so the note is history, not open work. + +## Release + +None from this item directly. The change lands on `dev` and records its warrant under `## [Unreleased]`; `/release` cuts the version. `L-260826-ddd843` (the two misclassifications) closes only when **both** SDKs have shipped a release carrying the fix — the JS half was still unreleased when this landed. From 1d66144b401a809830e71adcf1fbb4cca53f1911 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 7 Sep 2026 09:05:02 +0200 Subject: [PATCH 2/9] docs: catch up with what the mthds 0.13.0 bump left stale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v0.9.0 release moved the model and the pin but updated no prose, so merging it made three statements in `docs/architecture.md` false: - the `views` list was described as having one token, where `VALIDATION_VIEW_OUTPUT_FORM` now sits beside it; - the report was said to add "three typed fields", enumerated — now also `output_form` and this branch's `default_pipe_ref` (the count goes, per the workspace rule against hardcoding them); - the pipe I/O contracts claimed an output carries "no schema … the payload a run produces is the run's own result". `mthds` 0.13.0 makes `json_schema` required on `PipeOutputContract`, reversing exactly that reasoning: an output's schema is the concept's content model, declared and knowable before any run. `PipelexValidationReport`'s own docstring carried the same clause and is corrected with it. `_body_with_contracts` in the validation-contract tests was weakened by the same bump. Its output block stated no `json_schema`, so under 0.13.0 every body it built failed to parse — and its two `test_artifact_drift_fails_the_parse` cases were passing on that, not on the input drift each one names. Stating the schema restores what they test; verified by parsing a body from the helper with a clean input contract, which now succeeds. Two additions rather than corrections: the removed `build_inputs` gets a real migration target, since `mthds` 0.12.0 shipped `mthds.protocol.inputs_template` and its own `InputsTemplateFormat` — the template is relocated to the standard's package and projected client-side, not lost — and `output_form` gets its own bullet in the typed-by-import section. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DzVXcNRcVz8DUvAzmEuxPD --- CHANGELOG.md | 4 ++-- docs/architecture.md | 15 ++++++++------- docs/input-preparation.md | 2 ++ pipelex_sdk/validation_models.py | 9 ++++++--- tests/unit/test_validation_contract.py | 16 ++++++++++++++-- 5 files changed, 32 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a08871..53187f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,11 +10,11 @@ ### Changed - **Breaking: `prepare_inputs` reads its signature from the input-form descriptor, not the inputs template.** It composes one `POST /v1/validate` with `views: ["input_form"]` and `allow_signatures=True`, and walks the standard's `InputForm` artifact — `document` / `image` mark a file position, `object` recurses through `fields`, `list` through `item`, everything else passes through. Source-compatible for every caller passing `files`; the SDK no longer calls `/v1/build/inputs` at runtime. A valid report carrying no descriptor is an error naming the pipelex-api floor, never a silent degrade to "no uploads". -- **Breaking: `build_inputs` and its models are removed.** `client.build_inputs`, `BuildInputsRequest`, `BuildInputsValidReport`, `BuildInputsResponse`, `BuildInputsResponseAdapter` and `InputsTemplateFormat` are gone — the route wrapper existed only to be the signature source `prepare_inputs` read, and nothing calls it now. This is the Python SDK's step of the workspace program retiring `/v1/build/*`; a caller that still needs a fill-in template projects one from the descriptor. +- **Breaking: `build_inputs` and its models are removed.** `client.build_inputs`, `BuildInputsRequest`, `BuildInputsValidReport`, `BuildInputsResponse`, `BuildInputsResponseAdapter` and `InputsTemplateFormat` are gone — the route wrapper existed only to be the signature source `prepare_inputs` read, and nothing calls it now. This is the Python SDK's step of the workspace program retiring `/v1/build/*`; a caller that still needs a fill-in template projects one from the descriptor with `mthds.protocol.inputs_template` (`render_inputs_template` / `project_inputs_template`, in both the compact and explicit shapes, as JSON or TOML), which is also where `InputsTemplateFormat` now lives. The wrapper is not a capability lost but one relocated to the standard's own package — and projected client-side, so a method reached by `method_ref` or `method_id` gets a template with no server round-trip at all. - **Breaking: the shared crate envelope moved to `pipelex_sdk.crate_models`.** `MthdsFileItem`, `CrateRequestBase` and `CrateInvalidReport` now live beside the routes that use them (`/v1/resolve`, `/v1/codegen`) and `pipelex_sdk/build_models.py` is deleted — a module named for the build routes could not go on holding the envelope after they left. The models themselves are unchanged; update the import path. - **Breaking: a canonical file dict nested inside a `Dynamic` input is no longer uploaded.** Such an input is `kind: "unknown"` in the descriptor — the standard's escape hatch — and the walk does not enter it. Uploading on the strength of a `url` key is the value-shape guess this change removes; a caller with a Dynamic input uploads with `upload_file` first and passes the storage URI, which `docs/input-preparation.md` has always prescribed. - **`prepare_inputs` accepts the explicit `{concept, content}` input envelope**, not only compact values, closing a parity gap with the JS SDK. An agent that fills an explicit template — the shape the hosted console and MCP hand out — can now hand it straight back; previously every file-bearing envelope position raised `InputPreparationError: Unsupported value at a file input … got dict`. The envelope's `content` is interpreted identically and preserved on output, so the concept annotation rides through to the run. -- **The documentation is rewritten around the descriptor.** `docs/input-preparation.md` now describes the three call shapes, the signature call, pipe selection and its manifest-only `main_pipe` gap, the envelope, and why the template was the wrong signature source; `docs/architecture.md` follows the removal. +- **The documentation is rewritten around the descriptor.** `docs/input-preparation.md` now describes the three call shapes, the signature call, pipe selection and its manifest-only `main_pipe` gap, the envelope, and why the template was the wrong signature source; `docs/architecture.md` follows the removal. It also catches up with v0.9.0, which shipped `output_form` and the `mthds` 0.13.0 bump without touching the docs: the views list is no longer described as having one token, the report's typed fields now include `output_form` and `default_pipe_ref`, and the pipe I/O contracts no longer claim an output carries no schema — 0.13.0 made `json_schema` required there, reversing the reasoning the doc still quoted. ### Security diff --git a/docs/architecture.md b/docs/architecture.md index 4beea1e..02be059 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -31,7 +31,7 @@ MTHDS is the brand of the open standard (the language, the protocol). Pipelex is The Pipelex narrowing of the `/v1/validate` verdict union is one such implementation envelope and lives here (`pipelex_sdk.validation_models`): `PipelexValidationReport` / `PipelexInvalidReport` / the `PipelexValidationResult` union, plus the supporting `ValidationErrorItem` / `ValidationErrorCategory` / `ValidatedPipeEntry` / `DryRunStatus` / `LiftablePipeEntry` / `SuggestedFix` / the `FixOp` variants / `FixOpKind` / `FixSafety`. They narrow the neutral `ValidationReport` / `InvalidValidationReport` / `ValidationResult` bases that `mthds` keeps (in `mthds.protocol.models`). The report/union types carry the `Pipelex` prefix; the supporting types stay neutrally named — branding the envelope, not the fields inside it. The brand-neutral `Dict*` wire concretes (`DictRunResultExecute` and friends) stay in `mthds` — they are a shared wire contract the `pipelex` runtime itself builds on — and this SDK reuses them by inheritance rather than redefining them (a deliberate divergence from `pipelex-sdk-js`, which duplicates both the `Dict*` and the `Pipelex*` types in its own `models.ts`). -The same boundary decides two members *inside* the Pipelex envelope. The input-form descriptor and the pipe I/O contracts are MTHDS artifacts — the standard's own recommended extension fields of the validate report, each with a normative page — so this SDK types them by importing `mthds.protocol.input_form` and `mthds.protocol.pipe_io_contracts` rather than declaring them, and does not re-export them under its own name. A Pipelex-branded envelope may carry a neutral artifact; it may not adopt it. See "Typed by import" below. +The same boundary decides several members *inside* the Pipelex envelope. The input- and output-form descriptors and the pipe I/O contracts are MTHDS artifacts — the standard's own recommended extension fields of the validate report, each with a normative page — so this SDK types them by importing `mthds.protocol.input_form`, `mthds.protocol.output_form` and `mthds.protocol.pipe_io_contracts` rather than declaring them, and does not re-export them under its own name. A Pipelex-branded envelope may carry a neutral artifact; it may not adopt it. See "Typed by import" below. ## Credentials & configuration @@ -143,25 +143,26 @@ The protocol `validate` is **overridden** (not inherited) to add the Pipelex-API - **Markdown render is always injected.** `validate(...)` adds `"markdown"` to the `render` list (de-duplicated, caller tokens first) so both a valid `PipelexValidationReport` and a produced `PipelexInvalidReport` carry `rendered_markdown`. Unknown render tokens are server-side lenient-ignored. - **`mthds_sources`** is a named parameter (parallel to `mthds_contents`) threaded onto each diagnostic's `source`; sent only when provided. -- **`views`** is the opt-in for the server's structured views, on both `validate` and `validate_files`. `input_form` — named by the `VALIDATION_VIEW_INPUT_FORM` constant — is the only token today. Unlike `render`, the list travels **verbatim**: nothing is injected, nothing is de-duplicated, and an explicit `[]` is sent as `[]`, because the server resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so client-side normalization would only hide what the caller asked for. Left at `None` the key is not sent at all, which is what keeps the default response byte-identical for the consumers that discard views (hook pipelines, CI gates, agent loops). The constant is a constant rather than a closed enum on purpose: the request boundary is deliberately open, so a stale token never fails a call. +- **`views`** is the opt-in for the server's structured views, on both `validate` and `validate_files`. The tokens are named by constants — `VALIDATION_VIEW_INPUT_FORM` for `input_form` and `VALIDATION_VIEW_OUTPUT_FORM` for `output_form`, the descriptor of what the pipe produces. Unlike `render`, the list travels **verbatim**: nothing is injected, nothing is de-duplicated, and an explicit `[]` is sent as `[]`, because the server resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so client-side normalization would only hide what the caller asked for. Left at `None` the key is not sent at all, which is what keeps the default response byte-identical for the consumers that discard views (hook pipelines, CI gates, agent loops). The constant is a constant rather than a closed enum on purpose: the request boundary is deliberately open, so a stale token never fails a call. - **`validate_files(files, …)`** takes `MthdsFile(content, uri?)` records. When any file carries a URI, every content gets a parallel source label — the named file's URI, or a deterministic `inline://file-N.mthds` for an unnamed sibling — so the server never sees a length-mismatched `mthds_sources`. The override reuses the inherited base transport seam `_post_validate` (which builds the body — passing `render` / `mthds_sources` through the protocol's `extra` extension passthrough — sends the request, and raises on a no-verdict non-2xx), then parses the raw 200 body into this SDK's own `PipelexValidationResult` via `PipelexValidationResultAdapter`. Body-building and transport stay shared with the base; only the Pipelex presentation/sources concerns and the branded narrowing live here. The validation models (`PipelexValidationResult` = `PipelexValidationReport | PipelexInvalidReport`, with `rendered_markdown`) are **owned by this SDK** (`pipelex_sdk.validation_models`); they narrow `mthds`'s neutral verdict bases, completing the brand boundary (the resolved follow-up #9). The base `MthdsAPIClient.validate()` returns the neutral `ValidationResult` instead. **Checkpoint-5 decision (validate error regime):** the delegation keeps the inherited `httpx.HTTPStatusError` regime on a *no-verdict* non-2xx, where the JS `validate` raises `ApiResponseError`. Kept as-is (deferred parity), because in both SDKs `validate`'s error regime matches the *other* protocol routes of that SDK — JS routes all raise `ApiResponseError`, Python protocol routes all inherit `httpx.HTTPStatusError` (decision #5). Making Python's `validate` alone raise `ApiResponseError` would make it inconsistent with `execute`/`start`/`models`/`version`, which is worse than the JS divergence. The verdict itself (valid/invalid) is always a 200 either way — only the no-verdict failure *presentation* differs. -**What the report carries.** A valid `PipelexValidationReport` adds three typed fields beyond the protocol base. `warnings: list[ValidationErrorItem]` are advisory lints on a bundle that is nonetheless valid — the same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid` (this is where the `hint_*` error types ride). `liftable_pipes: list[LiftablePipeEntry]` inventories the pipes the runtime may skip when an optional slot resolves absent. `input_form: InputForm | None` carries the per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`, and is present only when the request named the `input_form` view. The two lists default empty and `input_form` defaults `None`, so a body from an older runner still parses; the empty default is also what a clean bundle yields, so no caller can tell the two apart. `PipelexInvalidReport` gains none of them: `warnings` and `input_form` derive from a crate that was never assembled. +**What the report carries.** A valid `PipelexValidationReport` adds typed fields beyond the protocol base. `warnings: list[ValidationErrorItem]` are advisory lints on a bundle that is nonetheless valid — the same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid` (this is where the `hint_*` error types ride). `liftable_pipes: list[LiftablePipeEntry]` inventories the pipes the runtime may skip when an optional slot resolves absent. `input_form: InputForm | None` and `output_form: OutputForm | None` carry the per-pipe input- and output-form descriptors, keyed exactly like `pipe_io_contracts`, each present only when the request named that view. `default_pipe_ref: str | None` is the qualified `pipe_ref` a caller gets by omitting the pipe selector, `None` when the closure declares none or several — manifest-aware for a fetched package, which is what makes it outrank a `bundle_blueprint` read; `docs/input-preparation.md` walks the fallback ladder `prepare_inputs` runs when a runner predates it. The lists default empty and the optionals default `None`, so a body from an older runner still parses; an empty list is also what a clean bundle yields, so no caller can tell the two apart. On the view fields the same default says something stronger: an opt-in view's absence means the request did not ask for it, never that the method has nothing to describe. `PipelexInvalidReport` gains none of them — they all derive from a crate that was never assembled. `ValidationErrorItem` gains `missing_pipe_code` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None` — a deterministic repair proposal with a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops`: a list discriminated on `kind` over the closed `FixOpKind` vocabulary (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), narrowed with an exhaustive `match op: case SetKeyOp(): …`. The ops are **reader** models here: `extra="allow"`, no `frozen`, none of the runtime's wildcard-refusing validators, because this SDK only reads fixes where the runtime plans them. A `kind` this SDK does not know fails the whole verdict parse, deliberately and consistently with `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break. -### Typed by import: the descriptor and the pipe I/O contracts +### Typed by import: the descriptors and the pipe I/O contracts -Two members of the valid arm are the **standard's** artifacts rather than Pipelex's, and this SDK narrows them by importing the standard's own client models instead of restating their shape: +Several members of the valid arm are the **standard's** artifacts rather than Pipelex's, and this SDK narrows them by importing the standard's own client models instead of restating their shape: -- **`pipe_io_contracts: PipeIOContracts`** — `dict[pipe_ref, PipeIOContract]` from `mthds.protocol.pipe_io_contracts`. An input slot reads as typed members: `concept_ref`, a three-valued `presence` (`PresenceMarker`, so an authored `!` is not flattened into a boolean), a `multiplicity` (`IOMultiplicity`), the `item_count` that is non-null exactly on the fixed arm, and the slot's `json_schema`. The output side is deliberately asymmetric — a two-valued `optional` and no schema — because `!` is rejected on an output and the payload a run produces is the run's own result. +- **`pipe_io_contracts: PipeIOContracts`** — `dict[pipe_ref, PipeIOContract]` from `mthds.protocol.pipe_io_contracts`. An input slot reads as typed members: `concept_ref`, a three-valued `presence` (`PresenceMarker`, so an authored `!` is not flattened into a boolean), a `multiplicity` (`IOMultiplicity`), the `item_count` that is non-null exactly on the fixed arm, and the slot's `json_schema`. The output side stays asymmetric in exactly one place: a two-valued `optional`, because `!` is a use-site assertion about an input and is rejected on an output. It is no longer asymmetric on the schema — since `mthds` 0.13.0 an output carries a required `json_schema` too, and the two answer different questions. An input's describes what a caller **sends**, so a plural slot's is a bare array; an output's describes what **comes back**, which is the concept's content model, so a plural output's is that model's list envelope. Both are declared facts, knowable before any run happens. - **`input_form: InputForm | None`** — `dict[pipe_ref, PipeInputFormDescriptor]` from `mthds.protocol.input_form`. A descriptor's `fields` are the recursive `InputFormField` union discriminated on `kind`; narrow a node with `match node: case ListField(): …` or an `isinstance` check, importing the per-kind models from `mthds.protocol.input_form`. An `object` node recurses through `fields`, a `list` node through `item` — and the item changes layer. Since `mthds` v0.10.0 the union is split by whether a node names itself: a top-level field is the named union (`TextField`, `DocumentField`, …, each requiring `name: str`), a `ListField.item` is the nameless one (`TextItem`, `DocumentItem`, …), which refuses a `name` at the parse. Narrow a list's item to `DocumentItem`, never `DocumentField`; because each `*Field` derives from its `*Item`, the item layer is the only safe narrowing target in that position. +- **`output_form: OutputForm | None`** — `dict[pipe_ref, PipeOutputFormDescriptor]` from `mthds.protocol.output_form`, the twin of the above on the other side of the pipe. It carries a single `field` rather than a `fields` list, since a pipe has exactly one output, and reuses the same `InputFormField` node union verbatim rather than declaring a second one that could drift — so a renderer walks it with the patterns it already has. It states no `presence` and no `gating`: those are facts of a slot a caller fills, and a result is not one; plurality is stated by wrapping the field in a `list` node. Read it together with that pipe's `output.json_schema` off `pipe_io_contracts` — the descriptor says what the result IS, the schema names the property its payload arrives under, and a consumer holding one but not the other is back to inferring the other from the value. -**Why import rather than declare.** These artifacts belong to MTHDS: they describe a method's inputs, which is a language-level fact, and any engine derives them from a resolved library with no Pipelex API in the loop. They were carried opaquely until now for a reason that has since expired — when that call was made, no published Python package declared them, so "type it here" could only mean "copy it here", and a copy is free to drift from the runtime that emits it. Since `mthds` 0.9.0 the standard's own client declares both, so typing them means importing them: one declaration per language, nothing to drift from. The principle the opaque ruling was protecting — this SDK is transport and does not own these types — is what an import preserves and a restatement would have broken. +**Why import rather than declare.** These artifacts belong to MTHDS: they describe what a method takes and what it produces, which is a language-level fact, and any engine derives them from a resolved library with no Pipelex API in the loop. They were carried opaquely until now for a reason that has since expired — when that call was made, no published Python package declared them, so "type it here" could only mean "copy it here", and a copy is free to drift from the runtime that emits it. Since `mthds` 0.9.0 the standard's own client declares them, so typing them means importing them: one declaration per language, nothing to drift from. The principle the opaque ruling was protecting — this SDK is transport and does not own these types — is what an import preserves and a restatement would have broken. The types are imported and used, never re-exported from `pipelex_sdk`. Re-exporting would put this package's name on a vocabulary it does not own and hand consumers a second import path to drift against; import them from `mthds.protocol` directly. diff --git a/docs/input-preparation.md b/docs/input-preparation.md index 34cc21f..2d4bc87 100644 --- a/docs/input-preparation.md +++ b/docs/input-preparation.md @@ -123,6 +123,8 @@ Earlier releases read the signature from the explicit inputs template (`POST /v1 The descriptor states the resolved kind at every depth and includes optional fields, so both are gone. It is also the standard's own artifact, derived from authored facts rather than from a rendered shape, and `/v1/validate` resolves all three method selectors server-side — which is what made the uniform selector surface possible at no server cost. +**If you actually wanted the template.** `prepare_inputs` no longer needs one, and this SDK's `build_inputs` wrapper went with it, but the template itself did not disappear — `mthds.protocol.inputs_template` projects one from the same descriptor, client-side: `render_inputs_template(descriptor=…, explicit=…, output_format=…)` for the JSON or TOML text, `project_inputs_template(descriptor=…, explicit=…)` for the dict. Ask `validate` for the `input_form` view, hand the pipe's descriptor to either, and the round-trip the removed route used to cost is gone too — which is what makes a template available for a method named only by `method_ref` or `method_id`. + **Known limit.** A class-backed concept (`structure = "SomeClass"`) whose reflection cannot map a field annotation collapses to `kind: "unknown"` in the descriptor, so a file field beneath one is invisible to this walk. That is a fidelity bug in the runtime's `build_input_form`, tracked separately; pass such a value as an already-uploaded storage URI until it is fixed. ### Pass-through rules diff --git a/pipelex_sdk/validation_models.py b/pipelex_sdk/validation_models.py index 97327fb..a5bcf1a 100644 --- a/pipelex_sdk/validation_models.py +++ b/pipelex_sdk/validation_models.py @@ -322,9 +322,12 @@ class PipelexValidationReport(ValidationReport): declared input slot reads as typed members — `concept_ref`, a three-valued `presence` (`PresenceMarker`), a `multiplicity` (`IOMultiplicity`), the `item_count` that is non-null exactly on the fixed arm, and its `json_schema` — and the output side reads its own asymmetric shape - (a two-valued `optional`, because `!` is rejected on an output). The artifact belongs to the - standard, so it is imported rather than restated: one declaration per language is what makes - drift impossible, which is precisely what keeping it opaque used to buy. + (a two-valued `optional`, because `!` is rejected on an output). Its `json_schema` is required + too since `mthds` 0.13.0, but states the concept's CONTENT MODEL rather than a caller's + argument: where a plural input's schema is a bare array, a plural output's is that model's list + envelope. The artifact belongs to the standard, so it is imported rather than restated: one + declaration per language is what makes drift impossible, which is precisely what keeping it + opaque used to buy. Contracts are **closed** shapes: a member this `mthds` version does not define is version drift and fails the parse. That closure is scoped to the artifact — the report around it stays diff --git a/tests/unit/test_validation_contract.py b/tests/unit/test_validation_contract.py index d0df57e..55624b4 100644 --- a/tests/unit/test_validation_contract.py +++ b/tests/unit/test_validation_contract.py @@ -231,13 +231,25 @@ def _body_with_contracts(input_contract: dict[str, Any]) -> dict[str, Any]: - """A valid body whose one pipe declares exactly `input_contract` as its single input slot.""" + """A valid body whose one pipe declares exactly `input_contract` as its single input slot. + + The output block states a `json_schema` for the same reason `VALID_BODY` does — required on + the contract since the output side gained a payload schema. It matters more here: every body + this helper builds feeds a test asserting the parse FAILS, so an incomplete output would make + each of them fail on the output rather than on the input drift the test names. + """ return { **VALID_BODY, "pipe_io_contracts": { "legal_contracts.summarize": { "inputs": {"contract": input_contract}, - "output": {"concept_ref": "legal_contracts.Summary", "multiplicity": "single", "item_count": None, "optional": False}, + "output": { + "concept_ref": "legal_contracts.Summary", + "multiplicity": "single", + "item_count": None, + "optional": False, + "json_schema": {}, + }, } }, } From a5ae0c449383b7bccb1b4838e24e879b756b5446 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 7 Sep 2026 09:11:08 +0200 Subject: [PATCH 3/9] fix: refuse a non-string method or pipe selector instead of reading it as absent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_non_empty_string` coerced anything that was not a string to `None`, and it served two callers that want opposite answers for that case. Reading the opaque `bundle_blueprint` — whose schema is the runtime's, not ours — a non-string really is an absent value to fall through on. Reading a CALLER's selector it is a mistake, and calling it absent made the exactly-one check unsound: `method_ref=123` beside a real `files` passed the check and prepared against a method the caller never named, `method_id=123` alone reported that no selector was given at all, and a non-string `pipe_ref` was quietly absorbed by the pipe defaulting. Split the two rules rather than raising inside the shared helper, which would have made the defensive blueprint reads throw on a shape they exist to tolerate. `_caller_selector` refuses a non-string with an `InputPreparationError` naming the argument and the type it got; `_non_empty_string` keeps the lenient contract for payload reads. `pipe_ref` is normalized beside the method selectors now, so both refusals land on the same pre-request boundary. Also corrects `prepare_inputs`'s `Raises:` section, which named `ApiResponseError` for a no-verdict `/v1/validate` failure. `validate` is 200-diagnostic and stays on the inherited `httpx.HTTPStatusError` regime, so a caller following the docstring would have missed exactly the fetch failures and 404s it listed. Advances L-260829-8a25d5 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DzVXcNRcVz8DUvAzmEuxPD --- CHANGELOG.md | 2 +- docs/input-preparation.md | 4 ++- pipelex_sdk/prepare_inputs.py | 53 +++++++++++++++++++++------- tests/unit/test_prepare_inputs.py | 33 ++++++++++++++++- wip/prepare-inputs-selectors/plan.md | 12 ++++++- 5 files changed, 88 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 53187f9..f78a51a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- **`prepare_inputs` takes the method three ways.** Beside inline `files`, it accepts a `method_ref` address (resolved by the runner) or a stored `method_id` (resolved by the hosted platform) — exactly one per call, all three server-resolved, nothing expanded client-side. Empty is absent (`files=[]`, a blank `method_ref` / `method_id`), and none or several raises `InputPreparationError` naming the three forms before any request leaves the process. A method addressed by URL that declares a file input now has an input-preparation path; previously it had none, even though the request beneath accepted the address. +- **`prepare_inputs` takes the method three ways.** Beside inline `files`, it accepts a `method_ref` address (resolved by the runner) or a stored `method_id` (resolved by the hosted platform) — exactly one per call, all three server-resolved, nothing expanded client-side. Empty is absent (`files=[]`, a blank `method_ref` / `method_id`) but the wrong type is not: none, several, or a non-string selector raises `InputPreparationError` before any request leaves the process — reading a mistyped selector as absent would let the exclusivity check pass and prepare against the wrong method. A non-string `pipe_ref` is refused on the same boundary, rather than being absorbed by the pipe defaulting. A method addressed by URL that declares a file input now has an input-preparation path; previously it had none, even though the request beneath accepted the address. - **`PipelexValidationReport.default_pipe_ref`** — the qualified `pipe_ref` a caller gets by omitting the pipe selector, or `null` when the closure declares none or several. Optional and read leniently: a runner that predates the field sends nothing, and `prepare_inputs` falls back to the opaque `bundle_blueprint.main_pipe`. ### Changed diff --git a/docs/input-preparation.md b/docs/input-preparation.md index 2d4bc87..43929c4 100644 --- a/docs/input-preparation.md +++ b/docs/input-preparation.md @@ -54,6 +54,8 @@ The prepared `inputs` are passed to the existing run lifecycle unchanged. Exactly one per call. **Empty is absent** — `files=[]`, `method_ref=""`, `method_id=" "` — mirroring the run options' rule, so an empty selector may sit beside a real one without tripping the exclusivity check. None or several raises `InputPreparationError` naming the three forms, before any request leaves the process. +**Empty is absent; the wrong type is not.** A non-string `method_ref` / `method_id` / `pipe_ref` raises `InputPreparationError` naming the argument and the type it got, on that same pre-request boundary. Reading it as absent instead is what would make the exclusivity check unsound — `method_ref=123` beside a real `files` would pass the check and silently prepare against the wrong method — and would let a mistyped `pipe_ref` be absorbed by the defaulting below. + | Selector | What it is | Who resolves it | | --- | --- | --- | | `files` | the inline MTHDS closure (`MthdsFileItem` entries: `content` plus an optional `source` label) | nobody — inline | @@ -80,7 +82,7 @@ A `method_ref` makes the server clone a repository first; `validate` needs no sp `validate` has no pipe selector — its report describes every pipe, keyed by qualified `pipe_ref` — so the helper picks one, in this order: -1. **`pipe_ref` when given.** Qualified-only: `domain.pipe_code`. A bare code, or a ref the method does not declare, is an `InputPreparationError` listing the qualified refs — one step to fix. The helper never grows a searched `pipe_code`: search is a run-route affordance, and the descriptor is keyed by qualified refs. +1. **`pipe_ref` when given.** Qualified-only: `domain.pipe_code`. A bare code, a non-string, or a ref the method does not declare, is an `InputPreparationError` listing the qualified refs — one step to fix. The helper never grows a searched `pipe_code`: search is a run-route affordance, and the descriptor is keyed by qualified refs. 2. **The report's typed resolved default** (`default_pipe_ref`), once the runner serves it: the ref a caller gets by omitting the selector, manifest-aware for a fetched package. Read when present; a server that predates it sends nothing. 3. **The bundle's declared `main_pipe`**, read defensively from the opaque `bundle_blueprint` and qualified by its `domain`. 4. **The single pipe**, when the method declares exactly one. diff --git a/pipelex_sdk/prepare_inputs.py b/pipelex_sdk/prepare_inputs.py index 0b75b64..ec79e5b 100644 --- a/pipelex_sdk/prepare_inputs.py +++ b/pipelex_sdk/prepare_inputs.py @@ -102,6 +102,11 @@ def __init__(self, client: _PrepareClient) -> None: def _non_empty_string(value: object) -> str | None: """A trimmed non-empty string, or `None` — the "empty is absent" rule. + Lenient on purpose, because what it reads is OPAQUE server payload — `bundle_blueprint`, + whose schema is the runtime's, not ours — where a shape that does not match is genuinely + an absent value to fall through on. A CALLER-supplied selector goes through + `_caller_selector` instead, which refuses a non-string rather than reading it as absent. + Deliberately local rather than reusing `client.py`'s `_normalized_selector`: that helper is private to the client boundary and raises `PipelineRequestError`, where every failure of this module owes an `InputPreparationError`. @@ -112,6 +117,22 @@ def _non_empty_string(value: object) -> str | None: return trimmed or None +def _caller_selector(value: object, *, argument: str) -> str | None: + """A caller-supplied selector, trimmed — `None` when absent, refused when not a string. + + The "empty is absent" rule of `_non_empty_string`, plus the boundary check that helper + must not make. Coercing a non-string to `None` here would read `method_ref=123` as an + absent selector and let it fall through to another one — defeating the exactly-one check + this whole surface rests on — and would let a non-string `pipe_ref` silently take the + default pipe instead of the one the caller named. Both are caller mistakes, and a caller + mistake owes an `InputPreparationError` raised before any request. + """ + if value is None or isinstance(value, str): + return _non_empty_string(value) + msg = f"Cannot prepare inputs: `{argument}` must be a string, got {type(value).__name__}." + raise InputPreparationError(msg) + + def _is_file_content(node: Any) -> bool: """A canonical Image/Document content is a dict carrying a `url` key. @@ -262,12 +283,14 @@ def _resolve_selector( Empty is absent — `files=[]`, `method_ref=""`, `method_id=" "` — mirroring the run options' rule and the `CrateRequestBase` normalizers, so an empty selector may sit beside - a real one without tripping the XOR. The check lives here because this module is what - composes the `validate` call, and it runs BEFORE any request. + a real one without tripping the XOR. A non-string `method_ref` / `method_id` is NOT absent + but refused, so a mistyped selector cannot slip past the XOR as a silent `None`. The check + lives here because this module is what composes the `validate` call, and it runs BEFORE + any request. """ selected_files = files or None - selected_method_ref = _non_empty_string(method_ref) - selected_method_id = _non_empty_string(method_id) + selected_method_ref = _caller_selector(method_ref, argument="method_ref") + selected_method_id = _caller_selector(method_id, argument="method_id") given: list[str] = [] if selected_files is not None: @@ -423,15 +446,21 @@ async def prepare_inputs( per uploaded asset. Raises: - InputPreparationError: No selector or several; the closure did not resolve; the - report carries no descriptor; the pipe could not be selected; or a value at a - file position is unusable. HTTP(S) URLs and existing `pipelex-storage://` URIs - pass through unchanged, and every failure is raised BEFORE any run is created. - ApiResponseError: A no-verdict condition from `/v1/validate` — a malformed selector, - an unknown or foreign-org `method_id` (`404`), a stored method with no source, a - fetch failure at the address. + InputPreparationError: No selector or several; a selector that is not a string; the + closure did not resolve; the report carries no descriptor; the pipe could not be + selected; or a value at a file position is unusable. HTTP(S) URLs and existing + `pipelex-storage://` URIs pass through unchanged, and every failure is raised + BEFORE any run is created. + httpx.HTTPStatusError: A no-verdict condition from `/v1/validate` — a malformed + selector, an unknown or foreign-org `method_id` (`404`), a stored method with no + source, a fetch failure at the address. `validate` is 200-diagnostic and stays on + the inherited protocol error regime, so a no-verdict failure arrives as the raw + status error rather than the product routes' `ApiResponseError`. """ selected_files, selected_method_ref, selected_method_id = _resolve_selector(files=files, method_ref=method_ref, method_id=method_id) + # Normalized here rather than at its use below, so a mistyped `pipe_ref` is refused on the + # same pre-request boundary as a mistyped selector — before the `validate` round-trip. + requested_pipe_ref = _caller_selector(pipe_ref, argument="pipe_ref") report = await _fetch_signature(client, files=selected_files, method_ref=selected_method_ref, method_id=selected_method_id) input_form = report.input_form @@ -445,7 +474,7 @@ async def prepare_inputs( ) raise InputPreparationError(msg) - selected_pipe_ref = _select_pipe_ref(report, input_form, _non_empty_string(pipe_ref)) + selected_pipe_ref = _select_pipe_ref(report, input_form, requested_pipe_ref) declared = {field.name: field for field in input_form[selected_pipe_ref].fields} ctx = _PrepareContext(client) diff --git a/tests/unit/test_prepare_inputs.py b/tests/unit/test_prepare_inputs.py index 6ba5a93..69fcb4f 100644 --- a/tests/unit/test_prepare_inputs.py +++ b/tests/unit/test_prepare_inputs.py @@ -14,7 +14,7 @@ import base64 import json from pathlib import Path -from typing import Any +from typing import Any, cast import httpx import pytest @@ -200,6 +200,37 @@ def test_only_empty_selectors_is_no_selector(self) -> None: with pytest.raises(InputPreparationError, match="no method selector"): asyncio.run(prepare_inputs(client, files=[], method_ref="", inputs={})) + @pytest.mark.parametrize( + ("kwargs", "argument", "type_name"), + [ + ({"method_ref": 123}, "method_ref", "int"), + ({"method_id": ["mt_1"]}, "method_id", "list"), + ({"method_ref": True}, "method_ref", "bool"), + ], + ) + def test_a_non_string_selector_is_refused_rather_than_read_as_absent(self, kwargs: dict[str, Any], argument: str, type_name: str) -> None: + # Empty is absent, but a WRONG TYPE is not: coercing it to `None` would let the XOR + # pass on `files` alone and prepare against a method the caller did not name. + client = _image_client() + + with pytest.raises(InputPreparationError) as exc_info: + asyncio.run(prepare_inputs(client, files=_FILES, inputs={}, **kwargs)) + + assert str(exc_info.value) == f"Cannot prepare inputs: `{argument}` must be a string, got {type_name}." + assert client.validate_calls == [] + assert client.upload_calls == [] + + def test_a_non_string_pipe_ref_is_refused_rather_than_silently_defaulted(self) -> None: + # Read as absent, it would be absorbed by the single-declared-pipe default: the pipe + # the caller named would vanish without a word. Refused on the pre-request boundary. + client = _image_client() + + with pytest.raises(InputPreparationError) as exc_info: + asyncio.run(prepare_inputs(client, files=_FILES, pipe_ref=cast("str", 123), inputs={})) + + assert str(exc_info.value) == "Cannot prepare inputs: `pipe_ref` must be a string, got int." + assert client.validate_calls == [] + # ── Pipe selection ──────────────────────────────────────────────────── def test_uses_the_single_declared_pipe_when_no_ref_is_given(self) -> None: diff --git a/wip/prepare-inputs-selectors/plan.md b/wip/prepare-inputs-selectors/plan.md index c6e601a..e54abe2 100644 --- a/wip/prepare-inputs-selectors/plan.md +++ b/wip/prepare-inputs-selectors/plan.md @@ -1,5 +1,5 @@ --- -status: draft +status: active item: L-260829-8a25d5 --- @@ -29,12 +29,22 @@ The JS twin (`L-260829-300c50`) landed as `pipelex-sdk-js` PR #42 (`bea4632`) an | `build_inputs` deleted, where JS kept `buildInputs` | JS has a wrapper family (`buildOutput`, `buildRunner`, `concept`, `pipeSpec`) retiring together under `L-260829-eefc3f`. Python only ever had this one, added in 0.5.0 solely to back `prepare_inputs`. | | `build_models.py` folded into `crate_models.py` | A module named for the build routes cannot go on owning the crate envelope after those routes leave. | | A local `_non_empty_string`, not `client._normalized_selector` | That helper is private to the client boundary and raises `PipelineRequestError`; every failure of this module owes an `InputPreparationError`. | +| Two helpers, not one: `_caller_selector` beside `_non_empty_string` | Review round 1. The single lenient helper read both a CALLER's selector and the OPAQUE `bundle_blueprint`, and those want opposite answers for a non-string: absent for the payload whose schema is the runtime's, refused for the argument. Raising inside the shared helper — the suggested fix — would have made the defensive blueprint reads throw on a shape they exist to tolerate. | | No fetch budget on the signature call | `validate` already rides the 20-minute ceiling; the 3-minute budget exists to *raise* the ~30s poll-ceiling routes. JS implemented this and reverted it — do not re-add. | ## What this supersedes `wip/pr-11-review-notes.md` recorded a nested-file limitation of the old template walk: a top-level `url` key caused an early return, so a sibling file field went un-uploaded, and the note explained that shape refinement was ambiguous because the walk dropped the envelope's `concept`. The descriptor walk removes that class of problem structurally — position and kind are stated, never inferred — so the note is history, not open work. +## Review + +Round 1 (2026-09-07) confirmed one defect in two threads and one wrong docstring, both fixed on the branch: + +- **A non-string selector was read as absent.** `_non_empty_string` coerced any non-string to `None`, so `method_ref=123` beside a real `files` passed the exactly-one check and prepared against a method the caller never named, and a non-string `pipe_ref` was absorbed by the pipe defaulting. Split into `_caller_selector` (refuses, per the decision row above) and the unchanged lenient reader, with `pipe_ref` normalization hoisted so both refusals land on the same pre-request boundary. +- **The `Raises:` section named the wrong exception.** `validate` is 200-diagnostic and stays on the inherited `httpx.HTTPStatusError` regime, not the product routes' `ApiResponseError`, so a caller following the docstring would have missed exactly the no-verdict failures it listed. + +Nothing else was raised. + ## Release None from this item directly. The change lands on `dev` and records its warrant under `## [Unreleased]`; `/release` cuts the version. `L-260826-ddd843` (the two misclassifications) closes only when **both** SDKs have shipped a release carrying the fix — the JS half was still unreleased when this landed. From 7253954381be1f8622e52b36f60f02a3ff7bbd99 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 7 Sep 2026 09:15:14 +0200 Subject: [PATCH 4/9] docs: land the prepare-inputs-selectors tracker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #21 merged to `dev` as `7b1892f` and `L-260829-8a25d5` closed with it, so the campaign is over and its plan says so: `status: landed`, plus the landing record the merge is what makes knowable — the SHA, what closed, what was only advanced and why, and the one thing left for a person. Advances L-260830-f5b65e Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DzVXcNRcVz8DUvAzmEuxPD --- wip/prepare-inputs-selectors/plan.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/wip/prepare-inputs-selectors/plan.md b/wip/prepare-inputs-selectors/plan.md index e54abe2..a8a6228 100644 --- a/wip/prepare-inputs-selectors/plan.md +++ b/wip/prepare-inputs-selectors/plan.md @@ -1,5 +1,5 @@ --- -status: active +status: landed item: L-260829-8a25d5 --- @@ -45,6 +45,14 @@ Round 1 (2026-09-07) confirmed one defect in two threads and one wrong docstring Nothing else was raised. +## Landing + +PR #21 merged to `dev` as `7b1892f`, closing `L-260829-8a25d5` (kind `fixed`). CI was green across every lint and test job on the reviewed commit; `make agent-check` and `make agent-test` both pass. The merge has not reached `main`, which is the deliberate part — see Release below. + +`L-260826-ddd843` was advanced, not closed: the Python half of the two misclassifications is fixed on `dev`, and that item's own bar is a shipped release from **both** SDKs, which neither has cut. + +The one thing this landing leaves open for a person: `L-260830-f5b65e` re-points `prepare_inputs` onto `POST /v1/input-form` before the Python release, and it waits on the pipelex-api route `L-260830-352005`. + ## Release None from this item directly. The change lands on `dev` and records its warrant under `## [Unreleased]`; `/release` cuts the version. `L-260826-ddd843` (the two misclassifications) closes only when **both** SDKs have shipped a release carrying the fix — the JS half was still unreleased when this landed. From dc8a21f5b7f3ae914834303924f05082481d8d91 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 7 Sep 2026 10:53:04 +0200 Subject: [PATCH 5/9] chore: move the exact mthds pin to 0.14.0 so pipelex and pipelex-sdk co-install (#25) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both packages pin `mthds` exactly, so they resolve together only when they name the same version. `pipelex` moved to `mthds==0.14.0` on its `dev` branch, which left this SDK at `0.13.0` and made the pair unresolvable from source; on PyPI the break arrives with the next `pipelex` release. Nothing in this client needed adapting. The release's substantive change is `MTHDS_STANDARD_VERSION` going from 1.0.0 to 2.0.0, which this SDK never reads — it stamps no crates and ships no manifest — and the new `is_mthds_version_satisfied` helper and the `parse_constraint` whitespace fix land in `mthds.package.manifest.schema`, which nothing here imports. `PROTOCOL_VERSION` stays at 0.6.0, so the routes and the wire contract are untouched. The inherited seam is unchanged: every protected member this client extends is still present, the four overrides still have base counterparts, and the ruff `runtime-evaluated-base-classes` dotted paths all still resolve. Claude-Session: https://claude.ai/code/session_01DuEKzDiMDWrABBibqwdFTt Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 1 + pyproject.toml | 2 +- uv.lock | 8 ++++---- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f78a51a..e0c392b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ - **Breaking: a canonical file dict nested inside a `Dynamic` input is no longer uploaded.** Such an input is `kind: "unknown"` in the descriptor — the standard's escape hatch — and the walk does not enter it. Uploading on the strength of a `url` key is the value-shape guess this change removes; a caller with a Dynamic input uploads with `upload_file` first and passes the storage URI, which `docs/input-preparation.md` has always prescribed. - **`prepare_inputs` accepts the explicit `{concept, content}` input envelope**, not only compact values, closing a parity gap with the JS SDK. An agent that fills an explicit template — the shape the hosted console and MCP hand out — can now hand it straight back; previously every file-bearing envelope position raised `InputPreparationError: Unsupported value at a file input … got dict`. The envelope's `content` is interpreted identically and preserved on output, so the concept annotation rides through to the run. - **The documentation is rewritten around the descriptor.** `docs/input-preparation.md` now describes the three call shapes, the signature call, pipe selection and its manifest-only `main_pipe` gap, the envelope, and why the template was the wrong signature source; `docs/architecture.md` follows the removal. It also catches up with v0.9.0, which shipped `output_form` and the `mthds` 0.13.0 bump without touching the docs: the views list is no longer described as having one token, the report's typed fields now include `output_form` and `default_pipe_ref`, and the pipe I/O contracts no longer claim an output carries no schema — 0.13.0 made `json_schema` required there, reversing the reasoning the doc still quoted. +- **Requires `mthds` 0.14.0 (breaking).** The pin moves from 0.13.0. Nothing in this client changed with it: the release's substantive cut is `MTHDS_STANDARD_VERSION` going from `1.0.0` to `2.0.0`, which this SDK never reads — it stamps no crates and ships no manifest — and the new `is_mthds_version_satisfied` helper and the `parse_constraint` whitespace fix sit in `mthds.package.manifest.schema`, which nothing here imports. `PROTOCOL_VERSION` stays at `0.6.0`, so the routes and the wire contract are untouched. The move still matters to anyone installing this package: `pipelex` pins `mthds` exactly too and has already moved to 0.14.0, so two exact pins on different versions made the pair unresolvable — this is what lets `pipelex` and `pipelex-sdk` co-install again. ### Security diff --git a/pyproject.toml b/pyproject.toml index c72f56f..220359a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ classifiers = [ ] dependencies = [ - "mthds==0.13.0", + "mthds==0.14.0", "pydantic>=2.10.6,<3.0.0", "typing-extensions>=4.0.0", "httpx>=0.23.0,<1.0.0", diff --git a/uv.lock b/uv.lock index 1a38edf..2a51741 100644 --- a/uv.lock +++ b/uv.lock @@ -212,7 +212,7 @@ wheels = [ [[package]] name = "mthds" -version = "0.13.0" +version = "0.14.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "httpx" }, @@ -221,9 +221,9 @@ dependencies = [ { name = "tomlkit" }, { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/e4/b6/6c67b86693d01d473444523db0aeadc1afdffaa1748a264dfab9c1c77b1c/mthds-0.13.0.tar.gz", hash = "sha256:ef59ffa94902da72371390cc39d8c0d85dfd2a524dbca2a8909a0a1ba6255607", size = 216142, upload-time = "2026-09-02T15:09:15.937Z" } +sdist = { url = "https://files.pythonhosted.org/packages/92/59/4ca9539571a2030f9427aeddb01bc334910953ebdfa4ab7db2051a6b8ae7/mthds-0.14.0.tar.gz", hash = "sha256:d2b4a9cd064004dfd5b802bb71c9894601fba9e598426f96e4bede16d52c3900", size = 221186, upload-time = "2026-09-06T21:44:54.585Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/89/26/f73e625b54cfba2e09a29df5e7173e508c25d3f7ce106be69fae6065d76c/mthds-0.13.0-py3-none-any.whl", hash = "sha256:606fef7978bbc8a5b249882c0b9377fed37123f618f8183f29e89b71a4eadac0", size = 85993, upload-time = "2026-09-02T15:09:14.334Z" }, + { url = "https://files.pythonhosted.org/packages/28/0b/32908eeed33396c5aafd4bcb2a1adc38d0ab8d85c2fde3af0bf4c2f73745/mthds-0.14.0-py3-none-any.whl", hash = "sha256:59a706205b6e6df47345caac01038588d21ab9e9c246afc765ee50b5f27f05a8", size = 87192, upload-time = "2026-09-06T21:44:53.022Z" }, ] [[package]] @@ -326,7 +326,7 @@ dev = [ [package.metadata] requires-dist = [ { name = "httpx", specifier = ">=0.23.0,<1.0.0" }, - { name = "mthds", specifier = "==0.13.0" }, + { name = "mthds", specifier = "==0.14.0" }, { name = "mypy", marker = "extra == 'dev'", specifier = "==1.19.1" }, { name = "pydantic", specifier = ">=2.10.6,<3.0.0" }, { name = "pylint", marker = "extra == 'dev'", specifier = "==4.0.4" }, From d05ddb526e215727522ccae0d857d4e054269713 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 8 Sep 2026 00:12:04 +0200 Subject: [PATCH 6/9] docs: cut the /release skill down to pipelex-sdk-python's specifics (#26) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The skill now names the workspace release play and declares only what is this repo's own: what the push to main publishes through publish.yml and how the landing verifies it, the pyproject.toml version with the uv.lock that make li regenerates, the agent-check and agent-test gates including the version-sync test that must run again after the lock, the files the release commit carries by name, the workflows that gate a release pull request, and the particulars — the refused pre-release form, the lightweight tags, the SHA-pinned Sigstore action, the exact mthds pin, and the absence of automatic release follow-ups. Dropped the procedure the play already carries once for every repo: the git status pre-flight that offered to fold uncommitted changes or unpushed commits into the release, the release branch created in place from the current HEAD, the numbered restatement of the changelog, bump, commit and pull request steps, and the post-merge reminder that the merge is what publishes. Claude-Session: https://claude.ai/code/session_016F72qvy4QBZHe7XX24QmcT Co-authored-by: Claude Opus 5 --- .claude/skills/release/SKILL.md | 158 ++++++++++---------------------- 1 file changed, 50 insertions(+), 108 deletions(-) diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 6e6f873..e4c467b 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -1,133 +1,75 @@ --- name: release description: > - Automates the pipelex-sdk-python release workflow: bumps the version in pyproject.toml, finalizes the CHANGELOG.md Unreleased section, runs quality checks, regenerates uv.lock, creates a release/vX.Y.Z branch, commits, pushes, and opens a PR to main. Use when user says "release", "cut a release", "bump version", "prepare a release", "make a release", "ship it", "create release branch", or any variation of shipping a new version of the pipelex-sdk Python package. The user can optionally provide changelog content inline when invoking the skill (e.g. "/release Added the storage routes"), which will be used as the changelog entry for this version. + Cut a release of pipelex-sdk-python, the Python client for the Pipelex hosted + API published to PyPI as pipelex-sdk: the release/vX.Y.Z worktree, the + pyproject.toml bump and the uv.lock that follows, the changelog entry, the + gates, one commit, and a pull request to main. Use when the user says + "release", "cut a release", "bump version", "prepare a release", "make a + release", "ship it", "create release branch", "promote dev to main", "tag a + version", or any variation of shipping a new version of the pipelex-sdk Python + package. Changelog content passed inline ("/release Added the storage routes") + becomes the entry. This is releasing pipelex-sdk itself, not moving the exact + mthds dependency pin, which is the bump-mthds skill. The merge is landed by + /ledger-land, never by this skill. --- -# pipelex-sdk-python Release Workflow +# Releasing pipelex-sdk-python -This skill handles the full release cycle for the `pipelex-sdk` Python package (import package `pipelex_sdk`, the `pipelex-sdk-python` repo). A release is a `release/vX.Y.Z` branch that PRs into `main`; merging to `main` triggers `publish.yml`, which builds the wheel, publishes it to PyPI as `pipelex-sdk` via Trusted Publishing (OIDC, no token), and creates a Sigstore-signed GitHub release from the changelog notes. +The procedure is the workspace release play, [`docs/releasing.md`](../../../../docs/releasing.md) at the workspace root — read it first, then run it with what follows. The repo key is `pipelex-sdk-python`, the base is `dev`, and the pull request targets `main`: `guard-branches.yml`'s `gate-main` refuses any head branch but `release/vX.Y.Z` into `main`, so there is no other way in. The release worktree is `_pipelex-sdk-python--release`, made with `wt add pipelex-sdk-python release --branch release/vX.Y.Z`. The repo declares neither `.worktree.toml` nor `.worktreeinclude`, so `wt` resolves the base from `origin/dev` and provisions with the Makefile's `install` target, which is what creates the `.venv` every gate below runs out of. The distribution is `pipelex-sdk` and the import package is `pipelex_sdk`; the version number is the distribution's. -## Files touched +## What ships -- **`pyproject.toml`** — the `version` field (line 3, under `[project]`) -- **`CHANGELOG.md`** — add `## [vX.Y.Z] - YYYY-MM-DD` entry (convert the `## [Unreleased]` section if present) -- **`uv.lock`** — regenerated via `make li` (lock + install) +The merge to `main` publishes through `publish.yml`, which fires on the push (`on: push: branches: [main]`) rather than on the pull request closing, so its run sits under `main` and its head SHA is the merge commit. Its jobs run in sequence: -## Workflow +- **build** — `python3 -m build` produces the sdist and the wheel and uploads them as a workflow artifact. Every later job downloads that artifact rather than rebuilding. +- **publish-to-pypi** — trusted publishing (OIDC, no token) through `pypa/gh-action-pypi-publish` into the `pypi` environment, pinned to . It sets no `skip-existing`, so a push to `main` that did not bump the version **fails at the upload** — there is nothing else standing between an unbumped push and this job, since `version-check.yml` runs on pull requests only. +- **github-release** — reads the version back out of `pyproject.toml`, slices the changelog section for that version out of `CHANGELOG.md`, signs the dists with Sigstore, creates the `vX.Y.Z` GitHub Release with those notes, and uploads `dist/**` to it. `gh release create` is passed `--generate-notes` alongside `--notes`, so the published body is the changelog slice with GitHub's own generated "What's Changed" section appended under it: a Release carrying more than the entry you wrote is the workflow behaving normally, not a slice that went wrong. Signing happens **before** the Release is created and is not `continue-on-error`, so a signing outage leaves the version on PyPI with no Release and no tag; re-running that run's failed jobs is the way back, because the job downloads the stored build rather than making a new one. When no `## [vX.Y.Z] - ` heading is found the extraction step warns, sets the notes empty and exits, and the Release then ships with the placeholder line `Release vX.Y.Z` where the changelog section should have been, instead of failing — `changelog-check.yml` on the pull request is the only thing that keeps that from happening. -### 1. Pre-flight checks +The landing verifies the publish — the run, the registry's answer, the tag: -- Read the current version from `pyproject.toml`. -- Read `CHANGELOG.md` to understand the current state (this repo keeps a `## [Unreleased]` section at the top). -- Run `git status` and `git log origin/main..HEAD` to assess the working tree: - - If there are **uncommitted changes** (staged or unstaged), warn the user and ask whether to commit them as part of the release, stash them, or abort. - - If there are **unpushed commits** on the current branch, list them so the user is aware — these will be included in the release branch. - -### 2. Determine the bump type - -Ask the user which kind of version bump they want — **patch**, **minor**, or **major** — unless they already specified it. Show the current version and what the new version would be for each option so the choice is concrete. - -While the package is pre-1.0 (`0.y.z`), treat the `0.MINOR.PATCH` segments the way the project has been using them: a breaking change bumps the minor, a backward-compatible feature or fix bumps the patch. If the changelog for this release contains a `### Breaking Changes` section (or otherwise describes a breaking change), steer the user toward at least a minor bump — this matches the repo's "pre-1.0 breaking changes → minor version bump" rule. - -### 3. Run quality checks - -Run `make agent-check`. This is the gate — if it fails, stop and report the errors so they can be fixed before retrying. Do not proceed past this step on failure. - -### 4. Ensure we're on the right branch - -The release branch must be named `release/vX.Y.Z` where X.Y.Z is the **new** version. The CI guards in this repo are strict about this: - -- `guard-branches.yml` (`gate-main`) rejects any source branch other than `release/vX.Y.Z` merging into `main`. -- `version-check.yml` rejects a mismatch between the branch name and the `pyproject.toml` version. - -Both guards match the **exact** regex `release/v[0-9]+\.[0-9]+\.[0-9]+` (strict three-segment semver, no suffix). All file modifications (changelog, version bump, lock) must happen on this branch. - -- If already on `release/vX.Y.Z` matching the new version, stay on it. -- If on `dev`, `main`, or any other branch, create and switch to `release/vX.Y.Z` from the current HEAD. -- If on a `release/` branch for a **different** version, warn the user and ask how to proceed. - -### 5. Finalize the changelog - -Add a new version entry for the release. This repo uses the workspace-wide `## [vX.Y.Z]` header convention (the changelog and publish workflows key off it). - -1. If there is an `## [Unreleased]` section, **convert it**: remove the `## [Unreleased]` heading (and any blank lines that immediately follow it) and replace it with the new `## [vX.Y.Z] - YYYY-MM-DD` heading. Any content that was under `[Unreleased]` becomes the content of the new version. -2. If there is no `[Unreleased]` section, insert the new version heading directly after the `# Changelog` intro block. -3. **Never recreate an `[Unreleased]` heading.** After a release the changelog should contain only concrete version entries — the next change adds a fresh `## [Unreleased]` section organically when someone starts the next cycle. -4. If the user provided changelog content when invoking the skill (e.g. `/release Added the storage routes`), **merge** that content with any existing `[Unreleased]` content (do not discard either source). Format the combined content under the appropriate headings — this repo uses `### Breaking Changes`, `### Added`, `### Changed`, `### Fixed`, `### Removed` — inferring headings from the content when possible. -5. If the release has no changelog content yet (neither from an `[Unreleased]` section nor from inline user input), ask the user what to include before proceeding. -6. The result should look like: - -```markdown -# Changelog - -All notable changes to `pipelex-sdk` are documented here. ... - -## [vX.Y.Z] - YYYY-MM-DD - -### Changed -- ... - -## [vPREVIOUS] - PREVIOUS-DATE -... +```bash +gh run list --workflow=publish.yml --branch main --limit 3 --json conclusion,headSha,url # the run whose headSha is the merge SHA: success +curl -s https://pypi.org/pypi/pipelex-sdk/json | jq -r .info.version # the registry's answer: X.Y.Z +git fetch --tags --prune origin && git tag --list vX.Y.Z # the tag ``` -### 6. Bump the version in pyproject.toml +`gh release view vX.Y.Z` confirms the Release and its notes. The registry can equally be read with `pip index versions pipelex-sdk` where a `pip` is on the PATH — the worktree's uv-made venv has none. Never create the tag by hand ahead of the merge: `gh release create` is what makes it. -Edit `pyproject.toml` line 3 (`version = "..."` under `[project]`) to the new version string. Only change the version field — don't touch anything else. +## Version files and the lock -### 7. Lock dependencies +- **`pyproject.toml`** — the `[project]` table's `version`, the one and only place the number is written. Keep it the file's **first** `version = ` line and the only line beginning with `version`: `changelog-check.yml` and `publish.yml` both read it with `grep -m 1 'version = '`, `version-check.yml` with `grep '^version'`, and `tests/unit/test_version.py` with `^version\s*=\s*"([^"]+)"` under `MULTILINE`. +- **`uv.lock`** — regenerated by `make li` (`lock` then `install`, that is `uv lock` followed by `uv sync --all-extras`), run after the bump so the lockfile records the new number and the installed distribution metadata is refreshed with it. `package-check.yml` runs `uv lock --locked` on every pull request and fails on a stale lock, so this step is not optional. If it fails, stop and report it rather than committing a stale lock. +- **Also stamped:** nothing. `pipelex_sdk/version.py` derives `__version__` from the installed distribution metadata through `importlib.metadata`, so there is no literal to move, and the README carries no version badge. -Run `make li` to regenerate `uv.lock` and reinstall. This ensures the lockfile reflects the new version in `pyproject.toml`. The `package-check.yml` CI job runs `uv lock --locked` and fails the PR if `uv.lock` is out of sync, so this step is not optional. If it fails, stop and report the error. +## Gates -### 8. Commit and push +Run in the worktree, in this order, before the commit: -Stage all release-related changes. This includes at minimum `pyproject.toml`, `CHANGELOG.md`, and `uv.lock`, plus any other files the user chose to include in step 1 (e.g. previously uncommitted work that belongs in this release). +1. **`make agent-check`** — `fix-unused-imports`, ruff format, ruff lint with `--fix`, pyright, mypy. **It rewrites files**, so whatever it touched joins the release commit. Red blocks the release: fix the code, never loosen the target. `lint-check.yml` runs the read-only twins of the same tools (`merge-check-ruff-format`, `merge-check-ruff-lint`, `merge-check-pyright`, `merge-check-mypy`) across every supported Python on the pull request, so a red here is a red pull request there. +2. **`make agent-test`** — the pytest suite, quiet unless it fails; `tests-check.yml` runs the same suite as `make gha-tests` (`--exitfirst --quiet`) across the same matrix. **Run this gate again after the bump and `make li`**: `tests/unit/test_version.py` compares `__version__`, read from the installed distribution metadata, against the `[project].version` it re-reads from `pyproject.toml`, so a bump whose `uv sync` did not land leaves the installed metadata stale and that test red. The test skips itself when the distribution is not installed at all, which is why the run after the lock, rather than the one before the bump, is what actually proves the stamp. -Commit with the message: +`make check` is the wider local target — it adds `pylint`, `check-unused-imports` and the `cleanderived` pass — and it is not part of the release: no workflow runs pylint, and nothing in CI will fail over it. -``` -Release vX.Y.Z -``` +## The release commit -Push the branch to origin with `-u` to set up tracking. +`pyproject.toml`, `CHANGELOG.md`, `uv.lock`, and each file `make agent-check` rewrote — staged by name. -### 9. Open a PR +## CI on the release pull request -Create a pull request targeting `main` with: +- **`guard-branches.yml`** — `gate-main` refuses any head into `main` that does not match `^release\/v[0-9]+\.[0-9]+\.[0-9]+$`, which is what makes the release branch name the only way in. The same workflow's `gate-release` governs pull requests into `dev`, `release/*` and `pre-release/*`, and `protect-workflows` refuses a workflow-file edit from an author whose resolved repository permission is not write or admin. +- **`version-check.yml`** (pull requests into `main`) — the `pyproject.toml` version must equal the version in the release branch name. It compares against the branch alone and never against the version already on `main`, so a bump that goes backwards is not caught here. +- **`changelog-check.yml`** (pull requests into `main`) — `CHANGELOG.md` must carry a `## [v] - ` heading for the version in `pyproject.toml`. It asserts nothing about `[Unreleased]`: a leftover heading passes CI and ships a wrong changelog, so removing it is this skill's job, not CI's. +- **`package-check.yml`** (`uv-lock-check`, every pull request) — `uv lock --locked` must leave `uv.lock` unchanged. +- **`lint-check.yml`** (every pull request) — the read-only merge checks on every supported Python; the aggregator job `Lint (all versions)` is the single required status. +- **`tests-check.yml`** (every pull request) — `make gha-tests` on the same matrix, aggregated by `Tests (all)`, with superseded runs on the same head cancelled. +- **`cla.yml`** — the CLA assistant against the `Pipelex/cla-signatures` registry, pointed at this repo's own `CLA.md`, with maintainers allowlisted; an external first-time author is prompted to sign before the pull request can merge. -- **Title:** `Release vX.Y.Z` -- **Body:** Include: - - The changelog entries for this version (copied from CHANGELOG.md) - - A note about the version bump from old to new - -Use this format for the PR body: - -```markdown -## Release vX.Y.Z - -Bumps version from `A.B.C` to `X.Y.Z`. - -### Changelog - - -``` +## Particulars -Report the PR URL back to the user, and remind them that **merging the PR into `main` is what publishes** — `publish.yml` builds the wheel, pushes it to PyPI as `pipelex-sdk` (Trusted Publishing), and cuts the Sigstore-signed GitHub release automatically. Nothing publishes until the PR is merged. - -## Important details - -- The version follows semver: `MAJOR.MINOR.PATCH`. -- Always confirm the bump type with the user before making changes. -- If `make agent-check` fails, the release is blocked — help the user fix the issues rather than skipping the checks. -- The CI gates a `release/vX.Y.Z` → `main` PR with: - - `version-check.yml` — the `pyproject.toml` version must match the `release/vX.Y.Z` branch name. - - `changelog-check.yml` — `CHANGELOG.md` must contain a `## [vX.Y.Z] -` entry for the new version. - - `package-check.yml` — `uv.lock` must be in sync with `pyproject.toml` (`uv lock --locked`). - - `tests-check.yml` — the test matrix must pass on every supported Python version (3.11 through 3.14). - - `lint-check.yml` — ruff format, ruff lint, pyright, and mypy merge checks across the same Python matrix (the same gates as `make agent-check`). - - `guard-branches.yml` — only `release/vX.Y.Z` branches may target `main`. - - `cla.yml` — the PR author must have signed the Pipelex CLA (maintainers are allow-listed; an external first-time author will be prompted to sign before the PR can merge). -- All checks must pass for the PR to be mergeable, so getting the changelog, version, and lockfile right is critical. -- **Pre-release versions are not supported through this flow.** Unlike `mthds-python`, this repo's `guard-branches.yml` (`gate-main`) and `version-check.yml` both match the exact regex `release/v[0-9]+\.[0-9]+\.[0-9]+` — a PEP 440 suffix (`a`/`b`/`rc`, e.g. `0.2.0rc1`) on a `release/v0.2.0rc1` branch would be **rejected** by the branch guard even though `publish.yml` can detect pre-releases. Stick to strict three-segment versions for the `release/vX.Y.Z` → `main` flow; raise it with the user if they ask for a pre-release. -- Today's date for the changelog entry: use the current date in `YYYY-MM-DD` format. +- **No pre-release form.** `gate-main` and `version-check.yml` both anchor `release/v[0-9]+\.[0-9]+\.[0-9]+` at both ends, so a PEP 440 suffix (`release/v0.10.0rc1`) is refused at the branch guard. The version check does not merely skip such a head either: the `exit 0` on its non-release path ends that step alone, the comparison that follows then runs with an empty branch version, and the mismatch fails the job. `publish.yml` does carry pre-release detection — it flags the Release `--prerelease` when the version ends in an `a`, `b` or `rc` group — but nothing can reach it through this flow. Ship a plain `X.Y.Z`, and raise it with the user if they ask for a pre-release. +- **The changelog heading carries the `v`** — `## [vX.Y.Z] - YYYY-MM-DD`, which is exactly what `changelog-check.yml` greps for and what `publish.yml` slices the Release notes out of. The repo keeps an `## [Unreleased]` section at the top between releases; it is folded into the new entry and none is left behind, and the next change re-creates one. +- **The tags are lightweight**, created as a side effect of `gh release create` rather than by `git tag -a`, and the job passes no `--verify-tag`. Always pass `--tags` when reading them: bare `git describe` finds no annotated tag here and dies. +- **The Sigstore action is pinned to a commit SHA on purpose.** The enterprise Actions allowlist keys on the exact SHA rather than a tag, so moving that action to another version needs an enterprise admin to allowlist the new SHA first, or the `github-release` job fails before it runs. That is a change to make deliberately and outside a release, never on the release branch. The context is `docs/ci-cd.md`, "Required org/repo configuration". +- **`mthds` is pinned exactly, and the pin is not this skill's to move.** `[project].dependencies` carries `mthds==X.Y.Z`; moving it is the `bump-mthds` skill, and a release ships whatever pin already landed on `dev`. The pin must name a version published to PyPI, and `pipelex` pins `mthds` exactly too — two exact pins on different versions make `pipelex` and `pipelex-sdk` unresolvable together, and the changelog records a pin move made precisely to restore that co-installability. Check where the pin stands before recommending the bump, and say so in the entry when it moved. +- **Nothing is armed automatically.** `ledger.toml` declares no `release_followups` for this repo, so the follow-ups a release arms — a floor to raise in a consumer, a doc that quotes the version — are yours to file alongside the release item. From 917790e0f2b4482ef4e967bb119e4dcd3eb538b5 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:11:20 +0200 Subject: [PATCH 7/9] =?UTF-8?q?feature/Codegen-tree-writer=20=C2=B7=20L-26?= =?UTF-8?q?0907-202383=20=E2=80=94=20regenerate=20without=20the=20pipelex?= =?UTF-8?q?=20runtime=20(#28)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `write_codegen_tree`, which writes a valid `codegen()` response to disk byte for byte, so a Python project can regenerate its typed tree with nothing installed but `pipelex-sdk`. It follows `pipelex`'s own `write_stamped_projection` discipline: validate every path first, refuse an unowned file, write only what changed, prune stamped artifacts the previous lock tracked, and write the lock last. It refuses a little earlier than `pipelex` where the response arrives as two independent fields: the response's lock must parse and track exactly its artifacts, and the paths about to be pruned are checked before the first write rather than after. The lock and stamp primitives are public modules so the offline drift check can build on them, and a side-by-side run diffed the resulting trees against `pipelex`'s writer. Closes L-260907-202383 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) --- CHANGELOG.md | 1 + README.md | 22 +- docs/architecture.md | 22 +- pipelex_sdk/client.py | 2 +- pipelex_sdk/codegen_lock.py | 217 +++++++++++++++++ pipelex_sdk/codegen_stamp.py | 42 ++++ pipelex_sdk/codegen_writer.py | 226 +++++++++++++++++ pipelex_sdk/errors.py | 28 +++ tests/unit/test_codegen_lock.py | 82 +++++++ tests/unit/test_codegen_stamp.py | 34 +++ tests/unit/test_codegen_writer.py | 391 ++++++++++++++++++++++++++++++ 11 files changed, 1062 insertions(+), 5 deletions(-) create mode 100644 pipelex_sdk/codegen_lock.py create mode 100644 pipelex_sdk/codegen_stamp.py create mode 100644 pipelex_sdk/codegen_writer.py create mode 100644 tests/unit/test_codegen_lock.py create mode 100644 tests/unit/test_codegen_stamp.py create mode 100644 tests/unit/test_codegen_writer.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e0c392b..1e7e6dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Added +- **`write_codegen_tree`, the verbatim codegen tree writer**: `pipelex_sdk.codegen_writer.write_codegen_tree(report, output_dir=…)` writes a valid `codegen()` response to disk byte for byte — every artifact at its `path`, the lock as `codegen.lock` — so a Python project regenerates a tree identical to a local `pipelex codegen types` run without installing `pipelex`. Like that command it validates every path before writing, raises `CodegenError` rather than overwrite a file codegen does not own, rewrites only what changed, and prunes stamped artifacts the previous lock tracked; unlike it, it also checks the paths it is about to prune and requires the response's lock to track exactly its artifacts before the first write; the lock format and path rules it reads are public in `pipelex_sdk.codegen_lock` and `pipelex_sdk.codegen_stamp`. - **`prepare_inputs` takes the method three ways.** Beside inline `files`, it accepts a `method_ref` address (resolved by the runner) or a stored `method_id` (resolved by the hosted platform) — exactly one per call, all three server-resolved, nothing expanded client-side. Empty is absent (`files=[]`, a blank `method_ref` / `method_id`) but the wrong type is not: none, several, or a non-string selector raises `InputPreparationError` before any request leaves the process — reading a mistyped selector as absent would let the exclusivity check pass and prepare against the wrong method. A non-string `pipe_ref` is refused on the same boundary, rather than being absorbed by the pipe defaulting. A method addressed by URL that declares a file input now has an input-preparation path; previously it had none, even though the request beneath accepted the address. - **`PipelexValidationReport.default_pipe_ref`** — the qualified `pipe_ref` a caller gets by omitting the pipe selector, or `null` when the closure declares none or several. Optional and read leniently: a runner that predates the field sends nothing, and `prepare_inputs` falls back to the opaque `bundle_blueprint.main_pipe`. diff --git a/README.md b/README.md index 6bdfcc1..5521494 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,25 @@ report = await client.validate(method_ref="github.com/Pipelex/methods/documents@ report = await client.validate(method_id="mt_123") ``` +### Generate typed code into your project + +`codegen()` projects a method into stamped typed artifacts plus their `codegen.lock`, and `write_codegen_tree` writes that response to disk verbatim, so the tree is byte-identical to a local `pipelex codegen types` run and no `pipelex` install is needed: + +```python +from pathlib import Path + +from pipelex_sdk.codegen_writer import write_codegen_tree +from pipelex_sdk.crate_models import CodegenRequest, CodegenValidReport + +report = await client.codegen(CodegenRequest(method_ref="github.com/Pipelex/methods/documents@v0.1.0", target="python-pydantic")) +if not isinstance(report, CodegenValidReport): + raise SystemExit(report.message) +written = write_codegen_tree(report, output_dir=Path("src/generated/documents")) +print(written.written, written.removed) +``` + +It never overwrites a file codegen does not own, rewrites only what changed, and prunes stamped artifacts that dropped out of the set. Commit the tree; do not run a formatter over it. + ### Long runs: start + poll explicitly Behind the hosted gateway, a synchronous `execute()` is cut off at ~30s and surfaces a `PipelineExecuteTimeoutError` pointing here. For long methods, drive the durable lifecycle yourself — the run survives client disconnects and is resumable by `pipeline_run_id`: @@ -103,7 +122,8 @@ There is no barrel import — package `__init__.py` files stay empty. Import eac - **Run lifecycle types** — `from pipelex_sdk.runs import RunStatus, RunPublic, RunRead, RunResults, RunResultState, WaitForResultOptions, PollInfo` - **Product wire models** — `from pipelex_sdk.product_models import UserProfile, MethodData, MethodWriteInput, Membership, MembershipsResponse, SubscriptionResponse, PlanView, InvoiceView, OnboardingSubmission, UploadInput, UploadedFile, PipelineRun, ...` - **Validation verdict types** — `from pipelex_sdk.validation_models import PipelexValidationResult, PipelexValidationReport, PipelexInvalidReport, ValidationErrorItem, SuggestedFix, VALIDATION_VIEW_INPUT_FORM, ...` -- **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, PagingNotTerminatingError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError, ...` +- **Codegen tree** — `from pipelex_sdk.codegen_writer import write_codegen_tree, CodegenTreeWriteReport`, with the format primitives in `pipelex_sdk.codegen_lock` (`CodegenLock`, `parse_lock`, `load_lock`, `validate_artifact_path`, ...) and `pipelex_sdk.codegen_stamp` +- **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, PagingNotTerminatingError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError, CodegenError, CodegenLockError, ...` - **Version** — `from pipelex_sdk.version import __version__` - **Protocol surface** (the MTHDS standard's wire types) comes from the `mthds` dependency — e.g. `from mthds.protocol.exceptions import PipelineRequestError`, `from mthds.protocol.models import ValidationResult` (the neutral verdict union that `PipelexValidationResult` narrows). - **Input-form descriptors and pipe I/O contracts** come from `mthds` too, because they are the standard's artifacts and this SDK only carries them: `from mthds.protocol.input_form import InputForm, InputFormField, ListField, TextField, ...` and `from mthds.protocol.pipe_io_contracts import PipeIOContracts, PipeInputContract, PresenceMarker, IOMultiplicity, ...`. `PipelexValidationReport.input_form` and `.pipe_io_contracts` are typed with them, so a node narrows on its `kind` and a slot's presence and multiplicity read as enums — but `pipelex_sdk` does not re-export the vocabulary, and importing it from here is the one supported path. diff --git a/docs/architecture.md b/docs/architecture.md index 02be059..78fd905 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -180,6 +180,22 @@ Their closure selector is the strict three-way XOR — inline `files`, an addres **A `method_ref` closure gets a fetch-sized budget.** Resolving an address can make the server clone a repository before it answers, and the server-side clone timeout runs well past the 30s management budget on a cold cache — an abort there would report a healthy, still-cloning server as unreachable. So a `method_ref`-carrying `resolve` / `codegen` uses an internal 3-minute budget (`_METHOD_REF_FETCH_TIMEOUT_SECONDS`, threaded through `_request_product`'s `request_timeout` override); it is internal (no new caller-facing parameter) and inert behind the hosted gateway's own cap. Mirrors the JS SDK's fetch budget; the run routes and `validate` need none because they already ride the 20-min blocking ceiling. +## Codegen tree writer (`write_codegen_tree`) + +`codegen()` returns a projection; `write_codegen_tree(report, output_dir=…)` (`pipelex_sdk/codegen_writer.py`) puts it on disk, so a Python project regenerates its typed tree with nothing installed but this SDK. It owns **bytes to disk and nothing else**: which methods to generate, which directory each tree goes into, and anything a project writes beside the tree stay project policy, with the caller. + +**Verbatim is the contract.** Every `artifacts[]` entry is written at its `path` and the `lock` as `lock_filename`, as UTF-8 bytes with no newline translation, no reformatting and no re-serialized lock. That fidelity is what makes the tree identical to a local `pipelex codegen types` run, and therefore what lets the offline check pass on it; a writer that "tidied" an artifact would produce a tree the check reports as hand-edited. The discipline is `pipelex`'s own `write_stamped_projection` (`pipelex/codegen/emission.py`), applied to a projection the server already stamped, and a tree written here was diffed byte for byte against that function's output over a generation and a pruning regeneration: + +- **Validate before the first byte.** A `lock_filename` other than `codegen.lock`, an unsafe or duplicate artifact path, a lock this SDK cannot read or that does not track exactly the response's artifacts, a symbolic link or a regular file on the way to a destination, a destination that is not a regular file, or a previously tracked path about to be pruned that sits behind a symbolic link or is no longer a regular file raises `CodegenError` and writes nothing. The lock and prune checks go further than `pipelex`: its writer builds the lock from the files it writes, so the two cannot disagree, and it resolves prune targets only after writing. Here the lock arrives as a separate field, and it is what the next run trusts for ownership and pruning. A previously tracked path whose directory has since become a regular file is skipped, as `pipelex` skips it: no file can be there to prune, and refusing it would block every rerun, because the refusal also keeps the new lock from replacing the old one. +- **Never overwrite a file codegen does not own.** An existing file at an artifact's path is replaced only when it is already identical, when the previous lock tracked it, or when it opens with a stamp. Opening with a stamp means the begin line here, where `pipelex` also requires the rest of the header to parse, so a file whose stamp header was edited is overwritten by this writer and refused by `pipelex`. +- **Write only what changed**, so regenerating over a current tree touches no mtime. +- **Prune what dropped out.** A file the previous lock tracked, absent from the new set and still stamped, is deleted; a file whose stamp was removed by hand, or that no lock ever tracked, is left for the offline check to report. +- **Write the lock last.** A previous lock that cannot be read (malformed, not UTF-8, an unknown `lock_version`) is replaced and prunes nothing; a previous lock tracking an unsafe path is refused, because that is a containment violation rather than corrupt state. + +The format primitives sit in two modules of their own, mirroring `pipelex`'s layout so the offline check can build on them: `pipelex_sdk/codegen_lock.py` (the lock model and parser, the version gate, the artifact path rules and the symlink-refusing output resolution) and `pipelex_sdk/codegen_stamp.py` (the stampable suffixes, their comment syntax, and the begin-line predicate that decides ownership). Nothing in either builds a stamp or encodes a lock; the server does both. + +**Synchronous on purpose.** It is local file I/O with no network leg, so it is a plain function called after `await client.codegen(...)`, like `pipelex`'s writer, rather than an `async def` that would only wrap blocking calls. + ## Pipelex product surface (hosted management routes) The hosted catalog/account routes the webapp drives (`pipelex_sdk/product_models.py` + the client's product methods). Every route rides the same `{base}/v1/*` surface, `Authorization: Bearer`, org-from-JWT contract as the protocol routes, and goes through `_request_product`, which maps a non-2xx `problem+json` to a typed `ApiResponseError` — **consumers branch on `.code`, never the HTTP status**. @@ -231,13 +247,13 @@ This SDK is a port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`) and tra - **Tooling routes** — `lint`, `format` (`resolve` and `codegen` shipped in 0.8.0 with the method selectors and are **not** gaps). - **Authoring helpers** — `build_output`, `build_runner`, `concept`, `pipe_spec`. JS still exports its `buildInputs` wrapper; Python's was **deleted** rather than left unused, because `prepare_inputs` was its only caller. A deliberate divergence, not a gap: the JS wrappers retire together as their own step of the same program. -- **Offline helpers** — `run_codegen_check` (the codegen drift check) and `get_method_closure` (client-side sugar that parses the polymorphic `mthds` source into a run-ready closure — in the JS SDK it is the documented migration target for the deleted by-id expansion legs; this SDK never had such legs, so the utility stays deferred rather than required). +- **Offline helpers** — `run_codegen_check` (the codegen drift check: `write_codegen_tree` puts a tree on disk, but nothing in this SDK verifies one yet) and `get_method_closure` (client-side sugar that parses the polymorphic `mthds` source into a run-ready closure — in the JS SDK it is the documented migration target for the deleted by-id expansion legs; this SDK never had such legs, so the utility stays deferred rather than required). Each stays deferred rather than silently missing. Everything else — the protocol routes, the durable lifecycle, the whole product surface, and the errors — does have a Python equivalent. -**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), the crate routes (`resolve`, `codegen`), the input-preparation surface (`upload_file` / `prepare_inputs`, taking all three method selectors and reading its signature from the input-form descriptor), and `health`. The method selectors (`method_ref` / `method_id`) match the JS v0.16.0 surface across the run and tooling methods, with one signature-shape divergence: JS `validate` takes a `ValidateMethodSelector` object in place of its first argument, while Python takes `method_ref=` / `method_id=` keyword parameters — same wire, same XOR, idiomatic per language. +**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), the crate routes (`resolve`, `codegen`) with the verbatim `write_codegen_tree`, the input-preparation surface (`upload_file` / `prepare_inputs`, taking all three method selectors and reading its signature from the input-form descriptor), and `health`. The method selectors (`method_ref` / `method_id`) match the JS v0.16.0 surface across the run and tooling methods, with one signature-shape divergence: JS `validate` takes a `ValidateMethodSelector` object in place of its first argument, while Python takes `method_ref=` / `method_id=` keyword parameters — same wire, same XOR, idiomatic per language. -**Models** — field-for-field across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. Two divergences worth naming: the page envelopes keep the wire's snake_case `next_cursor`, where the JS mirror renamed it `nextCursor` for its own consumers; and the method-files catalog converter (`parse_method_files` / `serialize_method_files`) lives in this package, where the JS pair lives in `mthds-js` because `pipelex-mcp` consumes the same format and wanted one owner. There is no second Python consumer, and the catalog serialization is a Pipelex product concern rather than an MTHDS protocol one, so this SDK is a proper home for it. If `mthds-python` ever grows an owner for the format, this SDK adopts it then. +**Models** — field-for-field across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. One addition runs ahead of the JS SDK: `write_codegen_tree` has no `@pipelex/sdk` counterpart, because the JS writer lives inside `pipelex-starter-js`'s own harness, mixed in with project policy; the byte-fidelity contract is small and load-bearing enough to belong to the SDK, where every consumer shares one correct implementation. Two divergences worth naming: the page envelopes keep the wire's snake_case `next_cursor`, where the JS mirror renamed it `nextCursor` for its own consumers; and the method-files catalog converter (`parse_method_files` / `serialize_method_files`) lives in this package, where the JS pair lives in `mthds-js` because `pipelex-mcp` consumes the same format and wanted one owner. There is no second Python consumer, and the catalog serialization is a Pipelex product concern rather than an MTHDS protocol one, so this SDK is a proper home for it. If `mthds-python` ever grows an owner for the format, this SDK adopts it then. **Errors** — `ApiResponseError`, `ApiUnreachableError`, `PipelineExecuteTimeoutError`, `RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError`, `PagingNotTerminatingError` are owned here; `RunStillRunningError` is re-exported from `mthds`. `ClientAuthenticationError` is **not** ported: it is a dormant export in the JS barrel (defined and exported but never raised by the client), and in Python it already lives in `mthds.runners.api.exceptions` — importable directly if ever needed, with no barrel here to re-export it through. diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index 1f51018..1395ec8 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -1155,7 +1155,7 @@ async def codegen(self, request: CodegenRequest) -> CodegenResponse: consumers, `python-structures`, `ts-zod`) — and returns the artifact set plus its `codegen.lock`. Write both verbatim and the tree is byte-identical to a local `pipelex codegen types` run, so the offline `pipelex codegen check` passes on it; - the SDK deliberately does not write files for you. + `pipelex_sdk.codegen_writer.write_codegen_tree` does exactly that. Same 200-verdict discipline and same three-form closure selector as `resolve`. A no-verdict condition (an unknown `kind`/`target`, a `pipe_ref` on the diff --git a/pipelex_sdk/codegen_lock.py b/pipelex_sdk/codegen_lock.py new file mode 100644 index 0000000..4716b0b --- /dev/null +++ b/pipelex_sdk/codegen_lock.py @@ -0,0 +1,217 @@ +"""`codegen.lock` and the artifact path rules: the set-level half of a generated codegen tree. + +Mirrors `pipelex/codegen/lock.py`. A stamp lets a lone file testify about itself; the lock records the +generated artifact *set* (each artifact's path and body hash, plus the crate fingerprint and engine +version the set was generated against), so the one drift a stamp cannot see, a deleted concept whose +stale file lingers, is still caught. A tree writer reads the *previous* lock to learn which files it +tracked and may prune; the offline check reads the lock to find a missing or an orphaned artifact. + +The lock text is never produced here. `/v1/codegen` returns it already encoded and it is written +verbatim, because re-serializing it would change the bytes a check compares. + +## Path rules + +A `/v1/codegen` response names every artifact's path, and a writer that joined an unvetted path onto +its output root could be routed out of the tree, by a `..` component, an absolute or drive-prefixed +path, or a symbolic link, to a place no stamp guards and no check looks. The rules below are the +reference's, so a path this SDK refuses is one `pipelex codegen types` refuses too. + +## Lock versions + +The lock is a cross-language interchange format read with `extra="forbid"`, and it carries +`lock_version` so that a format change is named rather than reported as an opaque shape error. A reader +refuses a version it does not know *before* validating the key set, and says which side to upgrade. A +lock with no `lock_version` key is version 1 by definition, since the field arrived with version 1. +""" + +import os +import tomllib +from collections.abc import Iterable +from pathlib import Path, PurePosixPath, PureWindowsPath +from typing import Any, NoReturn +from unicodedata import category + +from pydantic import BaseModel, ConfigDict, Field, ValidationError + +from pipelex_sdk._pydantic_utils import empty_list_factory_of +from pipelex_sdk.codegen_stamp import STAMPABLE_SUFFIXES +from pipelex_sdk.errors import CodegenError, CodegenLockError + +CODEGEN_LOCK_FILENAME = "codegen.lock" +"""The one filename a codegen lock is written as and read from.""" + +CODEGEN_LOCK_VERSION = 1 +"""The lock format version this SDK reads, mirroring pipelex's `CODEGEN_LOCK_VERSION`.""" + + +class CodegenLockEntry(BaseModel): + """One tracked artifact: its path relative to the lock, and the hash of its body below the stamp.""" + + model_config = ConfigDict(frozen=True, extra="forbid") + + path: str + content_hash: str + + +class CodegenLock(BaseModel): + """The generated artifact set of one output root, keyed to the crate and engine it was built against.""" + + model_config = ConfigDict(frozen=True, extra="forbid") + + lock_version: int = 1 + crate_fingerprint: str + engine_version: str + artifacts: list[CodegenLockEntry] = Field(default_factory=empty_list_factory_of(CodegenLockEntry)) + + def paths(self) -> set[str]: + """The set of tracked artifact paths, relative to the lock.""" + return set(validate_artifact_paths(entry.path for entry in self.artifacts)) + + +def parse_lock(content: str) -> CodegenLock: + """Parse the text of a `codegen.lock`. + + Raises `CodegenLockError` for malformed TOML, a shape the format does not define, or a `lock_version` + this SDK cannot read. Raises a plain `CodegenError` for an unsafe or duplicate artifact path, which is + a containment violation rather than corrupt state. + """ + try: + data = tomllib.loads(content) + except tomllib.TOMLDecodeError as exc: + msg = f"Malformed codegen lock: {exc}" + raise CodegenLockError(msg) from exc + _reject_unknown_lock_version(data) + try: + lock = CodegenLock.model_validate(data) + except ValidationError as exc: + msg = f"Malformed codegen lock: {exc}" + raise CodegenLockError(msg) from exc + validate_artifact_paths(entry.path for entry in lock.artifacts) + return lock + + +def load_lock(lock_path: Path) -> CodegenLock | None: + """Read and parse the lock at `lock_path`, or return `None` when no file is there. + + A lock that exists but cannot be read, or whose bytes are not UTF-8, is a `CodegenLockError` like any + other malformed lock. + """ + if not lock_path.is_file(): + return None + try: + content = lock_path.read_bytes().decode("utf-8") + except (OSError, UnicodeDecodeError) as exc: + msg = f"Unreadable codegen lock at '{lock_path}': {exc}" + raise CodegenLockError(msg) from exc + try: + return parse_lock(content) + except CodegenLockError as exc: + msg = f"Codegen lock at '{lock_path}': {exc}" + raise CodegenLockError(msg) from exc + + +def validate_artifact_path(path: str) -> Path: + """Validate one artifact path, from a response or a lock, and return its relative filesystem form.""" + if not path: + _raise_path_error(path, reason="path is empty") + if "\\" in path: + _raise_path_error(path, reason="backslashes are not allowed; use forward slashes") + if any(category(character).startswith("C") for character in path): + _raise_path_error(path, reason="control characters are not allowed") + + posix_path = PurePosixPath(path) + windows_path = PureWindowsPath(path) + if posix_path.is_absolute() or windows_path.is_absolute() or windows_path.drive or windows_path.root: + _raise_path_error(path, reason="absolute paths and drive prefixes are not allowed") + + parts = path.split("/") + if any(part in {"", ".", ".."} for part in parts): + _raise_path_error(path, reason="empty, '.', and '..' path components are not allowed") + + relative_path = Path(*parts) + if relative_path.suffix not in STAMPABLE_SUFFIXES: + expected = ", ".join(sorted(STAMPABLE_SUFFIXES)) + _raise_path_error(path, reason=f"unsupported artifact suffix (expected one of: {expected})") + return relative_path + + +def validate_artifact_paths(paths: Iterable[str]) -> dict[str, Path]: + """Validate a collection of artifact paths and reject a duplicate.""" + validated: dict[str, Path] = {} + for path in paths: + if path in validated: + _raise_path_error(path, reason="duplicate artifact path") + validated[path] = validate_artifact_path(path) + return validated + + +def resolve_artifact_path(*, root: Path, artifact_path: str) -> Path: + """Resolve a validated artifact path beneath `root` without following any symbolic link.""" + return resolve_output_path(root=root, relative_path=validate_artifact_path(artifact_path)) + + +def resolve_output_path(*, root: Path, relative_path: Path) -> Path: + """Resolve one output file beneath `root`, refusing a symbolic link anywhere on the way to it. + + The root itself must not be a symbolic link and, when it exists, must be a directory. Every component + below it must not be a symbolic link, every one on the way to the destination that already exists must + be a directory, the resolved destination must stay inside the root, and a destination that already + exists must be a regular file. The directory check is what lets a writer refuse a tree before its first + write: without it, a regular file where an artifact needs a parent directory is only discovered when + creating that directory fails, after the artifacts before it were written. + """ + if relative_path.is_absolute() or not relative_path.parts or any(part in {"", ".", ".."} for part in relative_path.parts): + _raise_path_error(str(relative_path), reason="internal output path must be a canonical relative path") + + requested_root = Path(os.path.normpath(root.absolute())) + if requested_root.is_symlink(): + _raise_path_error(str(requested_root), reason="output root must not be a symbolic link") + normalized_root = requested_root.resolve(strict=False) + if normalized_root.exists() and not normalized_root.is_dir(): + _raise_path_error(str(normalized_root), reason="output root exists but is not a directory") + + destination = normalized_root / relative_path + _reject_unsafe_components(root=normalized_root, relative_path=relative_path) + if not destination.resolve(strict=False).is_relative_to(normalized_root): + _raise_path_error(str(destination), reason=f"resolved path escapes output root '{normalized_root}'") + if destination.exists() and not destination.is_file(): + _raise_path_error(str(destination), reason="output destination exists but is not a regular file") + return destination + + +def _reject_unsafe_components(*, root: Path, relative_path: Path) -> None: + current = root + for depth, part in enumerate(relative_path.parts, start=1): + current /= part + if current.is_symlink(): + _raise_path_error(str(root / relative_path), reason=f"symbolic link component is not allowed: '{current}'") + if depth < len(relative_path.parts) and current.exists() and not current.is_dir(): + _raise_path_error(str(root / relative_path), reason=f"a component on the way to it is not a directory: '{current}'") + + +def _raise_path_error(path: str, *, reason: str) -> NoReturn: + msg = f"Unsafe codegen artifact path '{path}': {reason}." + raise CodegenError(msg) + + +def _reject_unknown_lock_version(data: dict[str, Any]) -> None: + """Refuse a lock whose format version this SDK cannot read, before its key set is validated. + + The ordering is the point: `extra="forbid"` would otherwise reject a future lock as a shape error over + a key its writer was entitled to add, instead of naming the version and saying which side to upgrade. + """ + # No key at all means the lock predates the field, which is version 1 by definition. It still goes + # through the comparison, or the day the constant moves every such lock would skip the gate. + raw_version = data.get("lock_version", 1) + # `bool` is an `int` subclass and `True == 1`, so a boolean would otherwise read as version 1. + is_version_number = isinstance(raw_version, int) and not isinstance(raw_version, bool) + if is_version_number and raw_version == CODEGEN_LOCK_VERSION: + return + + reason: str + if is_version_number and raw_version > CODEGEN_LOCK_VERSION: + reason = f"it declares lock_version {raw_version}, so upgrade pipelex-sdk to a build that reads it" + else: + reason = f"lock_version {raw_version!r} is not a known codegen lock format version" + msg = f"Unsupported codegen lock: this SDK reads lock_version {CODEGEN_LOCK_VERSION}, but {reason}." + raise CodegenLockError(msg) diff --git a/pipelex_sdk/codegen_stamp.py b/pipelex_sdk/codegen_stamp.py new file mode 100644 index 0000000..7a28a73 --- /dev/null +++ b/pipelex_sdk/codegen_stamp.py @@ -0,0 +1,42 @@ +"""The codegen stamp header, as far as a tree writer needs it. + +Mirrors `pipelex/codegen/stamp.py`, the reference for the stamp grammar. Every generated artifact opens +with a fenced comment block (`# >>> pipelex-codegen-stamp >>>` … `# <<< pipelex-codegen-stamp <<<` in +Python, the same fence behind `//` in TypeScript) recording the crate fingerprint, the engine version, +the projection and a hash of the body below the fence. + +A writer needs two facts from that grammar and nothing more. The stampable suffixes are the only file +types that can ever be an artifact, which makes them part of the path rules in +`pipelex_sdk.codegen_lock`. And the begin-line predicate decides whether a file already on disk belongs +to codegen: the writer overwrites or prunes a file only when it does, and the offline check calls a +stamped file the lock does not track an orphan by that same predicate, so the two agree on ownership by +construction. + +Nothing here builds or rewrites a stamp. The server stamps, and the SDK keeps the bytes it was sent. +""" + +from pathlib import PurePosixPath + +from pipelex_sdk.errors import CodegenError + +_BEGIN_MARKER = ">>> pipelex-codegen-stamp >>>" + +_COMMENT_PREFIX_BY_SUFFIX = {".py": "#", ".ts": "//"} + +STAMPABLE_SUFFIXES = frozenset(_COMMENT_PREFIX_BY_SUFFIX) +"""The file suffixes codegen stamps, mirroring pipelex's `STAMPABLE_SUFFIXES`.""" + + +def comment_prefix_for(artifact_path: str) -> str: + """The line-comment prefix an artifact is stamped in, by suffix (`.py` → `#`, `.ts` → `//`).""" + prefix = _COMMENT_PREFIX_BY_SUFFIX.get(PurePosixPath(artifact_path).suffix) + if prefix is None: + expected = ", ".join(sorted(STAMPABLE_SUFFIXES)) + msg = f"No codegen stamp syntax for '{artifact_path}': its file type is not stampable (expected one of: {expected})." + raise CodegenError(msg) + return prefix + + +def has_stamp(content: str, *, comment_prefix: str) -> bool: + """Whether `content` opens with a codegen stamp block in the given comment syntax.""" + return content.startswith(f"{comment_prefix} {_BEGIN_MARKER}") diff --git a/pipelex_sdk/codegen_writer.py b/pipelex_sdk/codegen_writer.py new file mode 100644 index 0000000..a1264d4 --- /dev/null +++ b/pipelex_sdk/codegen_writer.py @@ -0,0 +1,226 @@ +"""Write a `/v1/codegen` response to disk verbatim: the tree a local `pipelex codegen types` run writes. + +`client.codegen()` returns stamped artifacts and their `codegen.lock`; `write_codegen_tree` is the last +step, and it puts those bytes on disk and does nothing else. Every artifact is written exactly as the +server sent it, at its declared `path`, and the lock is written as `lock_filename`: no reformatting, no +re-serialized lock, no newline translation. That byte-for-byte fidelity is what makes the tree identical +to a local `pipelex codegen types` run, and therefore what lets the offline codegen check pass on it. +Reformatting an artifact or rebuilding the lock breaks that trust chain. + +The discipline is `pipelex`'s own `write_stamped_projection` (`pipelex/codegen/emission.py`), applied to +a projection the server has already stamped: + +- **Validate before writing.** A `lock_filename` other than `codegen.lock`, an unsafe or duplicate + artifact path, a lock that cannot be read or does not track exactly the artifacts, a symbolic link or + a regular file on the way to a destination, a destination that is not a regular file, or a previously + tracked path about to be pruned that sits behind a symbolic link or is no longer a regular file refuses + the whole tree before the first byte is written. The lock and prune checks go further than `pipelex`, + which builds its lock from the files it writes, so the two cannot disagree, and which resolves prune + targets only after writing. A previously tracked path whose directory has become a regular file is + skipped, as `pipelex` skips it: no file can be there to prune, and refusing it would block every rerun. +- **Never overwrite a file codegen does not own.** A file already at an artifact's path is replaced only + when its content is already identical, when the previous lock tracked it, or when it carries a stamp. +- **Write only what changed.** An already-current file is left alone, so regenerating over a current + tree is a true no-op: no mtime churn and clean diffs. +- **Prune what dropped out.** A file the previous lock tracked, that the new set no longer contains and + that still carries its stamp, is deleted, so a removed concept never lingers as a stale artifact. A + file whose stamp was removed by hand is left alone, and so is any file the lock never tracked. +- **Write the lock last.** + +Everything else is project policy and stays with the caller: which methods to generate, where each tree +goes, and whatever a project writes beside the tree. The function takes one response and one directory. +""" + +from pathlib import Path + +from pydantic import BaseModel, ConfigDict, Field + +from pipelex_sdk.codegen_lock import ( + CODEGEN_LOCK_FILENAME, + load_lock, + parse_lock, + resolve_artifact_path, + resolve_output_path, + validate_artifact_path, + validate_artifact_paths, +) +from pipelex_sdk.codegen_stamp import comment_prefix_for, has_stamp +from pipelex_sdk.crate_models import CodegenValidReport +from pipelex_sdk.errors import CodegenError, CodegenLockError + + +class CodegenTreeWriteReport(BaseModel): + """What one tree write did to the output directory. Every path is relative to that directory.""" + + model_config = ConfigDict(frozen=True) + + written: list[str] = Field(default_factory=list) + """Artifacts whose content changed and were written, in response order.""" + + unchanged: list[str] = Field(default_factory=list) + """Artifacts that were already current and were left untouched, in response order.""" + + removed: list[str] = Field(default_factory=list) + """Stamped artifacts the previous lock tracked that dropped out of the set and were deleted, sorted.""" + + lock_written: bool + """Whether `codegen.lock` changed and was written.""" + + +def write_codegen_tree(report: CodegenValidReport, *, output_dir: Path) -> CodegenTreeWriteReport: + """Write a valid `/v1/codegen` report into `output_dir`, verbatim. + + Branch on `is_valid` first: only the valid arm carries a tree. Raises `CodegenError` before anything + is written when the report, its lock or the directory is unsafe, and when a file codegen does not own + sits at an artifact's path. The report's lock must parse and track exactly its artifacts, because it is + what the next run trusts for ownership and pruning. A previous lock that cannot be read is replaced, + and nothing is pruned on its account; a previous lock tracking an unsafe path is refused. + """ + if report.lock_filename != CODEGEN_LOCK_FILENAME: + msg = ( + f"Refusing to write a codegen tree whose lock is named '{report.lock_filename}': this SDK writes and reads " + f"'{CODEGEN_LOCK_FILENAME}' only, so a lock under another name would guard nothing. Upgrade pipelex-sdk." + ) + raise CodegenError(msg) + validate_artifact_paths(artifact.path for artifact in report.artifacts) + _validate_response_lock(report) + + lock_path = resolve_output_path(root=output_dir, relative_path=Path(CODEGEN_LOCK_FILENAME)) + output_root = lock_path.parent + previous_paths = _previous_tracked_paths(lock_path) + + destinations: dict[str, Path] = {} + for artifact in report.artifacts: + destinations[artifact.path] = resolve_artifact_path(root=output_root, artifact_path=artifact.path) + # Resolved now rather than while pruning: a tracked path that has since become a directory or moved + # behind a symbolic link must refuse the tree before the first write, not after the artifacts changed. + stale_destinations: dict[str, Path] = {} + for stale_path in sorted(previous_paths - set(destinations)): + stale_destination = _resolve_prune_target(root=output_root, artifact_path=stale_path) + if stale_destination is not None: + stale_destinations[stale_path] = stale_destination + _preflight_destinations(report=report, destinations=destinations, previous_paths=previous_paths) + + written: list[str] = [] + unchanged: list[str] = [] + for artifact in report.artifacts: + if _write_if_changed(path=destinations[artifact.path], content=artifact.content): + written.append(artifact.path) + else: + unchanged.append(artifact.path) + + removed = _prune_delisted(stale_destinations=stale_destinations) + lock_written = _write_if_changed(path=lock_path, content=report.lock) + + return CodegenTreeWriteReport(written=written, unchanged=unchanged, removed=removed, lock_written=lock_written) + + +def _validate_response_lock(report: CodegenValidReport) -> None: + """Refuse a report whose lock cannot be read or does not track exactly its artifacts, before any write. + + `pipelex` builds its lock from the files it is writing, so the two agree by construction. Here they + arrive as two independent fields, and the lock written now is what the next run trusts: a path it + tracks may be overwritten without a stamp and pruned. A lock tracking a path the report never wrote + would hand that ownership to a file codegen never produced, a lock this SDK cannot parse would switch + pruning off on the next run, and one tracking an unsafe path would make every later write refuse. + """ + try: + lock = parse_lock(report.lock) + except CodegenLockError as exc: + msg = f"Refusing to write a codegen tree whose lock this SDK cannot read: {exc}" + raise CodegenError(msg) from exc + artifact_paths = {artifact.path for artifact in report.artifacts} + lock_paths = lock.paths() + if lock_paths != artifact_paths: + msg = ( + "Refusing to write a codegen tree whose lock does not track exactly its artifacts: " + f"untracked artifacts {sorted(artifact_paths - lock_paths)}, tracked paths with no artifact {sorted(lock_paths - artifact_paths)}." + ) + raise CodegenError(msg) + + +def _previous_tracked_paths(lock_path: Path) -> set[str]: + try: + lock = load_lock(lock_path) + except CodegenLockError: + # The new response is authoritative and replaces a corrupt or unreadable prior lock. Without a + # trustworthy artifact set there is nothing safe to prune, so the write proceeds with none. An + # unsafe tracked path is a plain `CodegenError` instead, and propagates: it is a containment + # violation, and recovering from it would weaken the boundary. + return set() + return lock.paths() if lock is not None else set() + + +def _resolve_prune_target(*, root: Path, artifact_path: str) -> Path | None: + """Resolve a path the previous lock tracked for pruning, or return `None` when no file can be there. + + A regular file where one of the path's directories used to be means there is nothing to prune, and + `pipelex` skips it. Refusing it instead would block every later run, because the refusal also keeps the + new lock from replacing the one that tracks the path. A symbolic link met on the way first still refuses. + """ + current = root + for part in validate_artifact_path(artifact_path).parts[:-1]: + current /= part + if current.is_symlink(): + break + if current.exists() and not current.is_dir(): + return None + return resolve_artifact_path(root=root, artifact_path=artifact_path) + + +def _preflight_destinations(*, report: CodegenValidReport, destinations: dict[str, Path], previous_paths: set[str]) -> None: + """Refuse to replace a file codegen does not own, before any artifact is written.""" + for artifact in report.artifacts: + destination = destinations[artifact.path] + existing = _read_bytes_or_none(destination) + if existing is None or existing == artifact.content.encode("utf-8"): + continue + if artifact.path in previous_paths or _is_stamped(existing, artifact_path=artifact.path): + continue + msg = f"Refusing to overwrite unowned file '{destination}'. Move it or choose a different codegen output directory." + raise CodegenError(msg) + + +def _prune_delisted(*, stale_destinations: dict[str, Path]) -> list[str]: + """Delete files the previous lock tracked and the new set dropped, but only those still carrying a stamp. + + `stale_destinations` maps each such path, in sorted order, to the destination the preflight resolved. + """ + removed: list[str] = [] + for relative_path, stale_path in stale_destinations.items(): + existing = _read_bytes_or_none(stale_path) + if existing is None: + continue + if _is_stamped(existing, artifact_path=relative_path): + stale_path.unlink() + removed.append(relative_path) + return removed + + +def _write_if_changed(*, path: Path, content: str) -> bool: + """Write `content` to `path` as UTF-8 bytes, only when they differ from what is there. Returns whether it wrote. + + Bytes rather than text on purpose: text mode would translate every line feed into the platform's line + separator, and the same response written on Windows and on Linux would then be two different trees. + """ + encoded = content.encode("utf-8") + if _read_bytes_or_none(path) == encoded: + return False + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(encoded) + return True + + +def _read_bytes_or_none(path: Path) -> bytes | None: + if not path.is_file(): + return None + return path.read_bytes() + + +def _is_stamped(content: bytes, *, artifact_path: str) -> bool: + try: + text = content.decode("utf-8") + except UnicodeDecodeError: + # A stamp is UTF-8 text, so bytes that do not decode cannot open with one. + return False + return has_stamp(text, comment_prefix=comment_prefix_for(artifact_path)) diff --git a/pipelex_sdk/errors.py b/pipelex_sdk/errors.py index 375f31a..7182922 100644 --- a/pipelex_sdk/errors.py +++ b/pipelex_sdk/errors.py @@ -20,6 +20,10 @@ HANDOFF Phase 2, and removed from `mthds-python` in Phase 6). `RunStillRunningError` stays in `mthds` — it belongs to the protocol `execute()` 202-degrade path, not the lifecycle — and is re-exported here so consumers have a single import home. + +The codegen tree errors (`CodegenError`, `CodegenLockError`) are not request errors at all: they +are raised by `pipelex_sdk.codegen_writer` and `pipelex_sdk.codegen_lock` over bytes and a directory, +so they derive from `Exception` rather than from the protocol base. """ from __future__ import annotations @@ -229,3 +233,27 @@ def __init__(self, message: str, status: int) -> None: class UploadTransportError(InputPreparationError): """A network or server fault reaching the upload route (unreachable host, `5xx`).""" + + +class CodegenError(Exception): + """A codegen tree this SDK refuses to write or read. + + Raised before the first byte is written when a `/v1/codegen` response, or the directory it is + headed for, is unsafe: a `lock_filename` other than `codegen.lock`, an artifact path that could + leave the output root or name a file type codegen never emits (absolute or drive-prefixed, a `..` + or empty component, a backslash, a control character, an unstampable suffix, a duplicate), a lock + that cannot be read or does not track exactly the artifacts, a symbolic link or a regular file on + the way to a destination, a symbolic link on the way to a previously tracked path about to be pruned, + a destination or such a path that is not a regular file, or a file already at an artifact's path that + codegen does not own. + """ + + +class CodegenLockError(CodegenError): + """A `codegen.lock` that cannot be read: malformed TOML, a shape the format does not define, + bytes that are not UTF-8, or a `lock_version` this SDK does not know. + + An unsafe artifact path inside an otherwise well-formed lock is deliberately NOT this error but a + plain `CodegenError`: it is a containment violation, not corrupt state a writer may recover from + by replacing the lock. + """ diff --git a/tests/unit/test_codegen_lock.py b/tests/unit/test_codegen_lock.py new file mode 100644 index 0000000..66df659 --- /dev/null +++ b/tests/unit/test_codegen_lock.py @@ -0,0 +1,82 @@ +"""`codegen.lock` parsing and the artifact path rules, mirroring `pipelex/codegen/lock.py`. + +The lock is a cross-language interchange format: `pipelex` writes it and this SDK reads it. These tests pin the +version gate (named before the key set is validated), the closed key set, and the split between a malformed lock, +which is a `CodegenLockError`, and an unsafe path inside a lock, which is a plain `CodegenError`. +""" + +from pathlib import Path + +import pytest + +from pipelex_sdk.codegen_lock import CODEGEN_LOCK_VERSION, load_lock, parse_lock, validate_artifact_path +from pipelex_sdk.errors import CodegenError, CodegenLockError + +# Real `pipelex.codegen.lock.encode_lock` output, header comment included. +_PIPELEX_LOCK = ( + "# codegen.lock — generated artifact set (Pipelex codegen). Do not edit by hand.\n\n" + "lock_version = 1\n" + 'crate_fingerprint = "abc"\n' + 'engine_version = "0.55.0"\n\n' + '[[artifacts]]\npath = "models.py"\ncontent_hash = "111"\n\n' + '[[artifacts]]\npath = "nested/extra.ts"\ncontent_hash = "222"\n' +) +_LOCK_WITHOUT_VERSION = 'crate_fingerprint = "abc"\nengine_version = "0.55.0"\n' + + +class TestCodegenLock: + def test_parses_a_pipelex_lock(self) -> None: + lock = parse_lock(_PIPELEX_LOCK) + + assert lock.lock_version == CODEGEN_LOCK_VERSION + assert lock.crate_fingerprint == "abc" + assert lock.engine_version == "0.55.0" + assert [(entry.path, entry.content_hash) for entry in lock.artifacts] == [("models.py", "111"), ("nested/extra.ts", "222")] + assert lock.paths() == {"models.py", "nested/extra.ts"} + + def test_a_lock_without_lock_version_is_version_one(self) -> None: + lock = parse_lock(_LOCK_WITHOUT_VERSION) + + assert lock.lock_version == 1 + assert lock.artifacts == [] + + def test_a_newer_lock_version_names_the_upgrade_before_the_key_set_is_checked(self) -> None: + future = "lock_version = 2\nsome_future_key = true\n" + _LOCK_WITHOUT_VERSION + + with pytest.raises(CodegenLockError, match="declares lock_version 2, so upgrade pipelex-sdk"): + parse_lock(future) + + @pytest.mark.parametrize("bad_version", ["true", '"1"', "0"]) + def test_a_version_that_is_not_a_known_number_is_refused(self, bad_version: str) -> None: + with pytest.raises(CodegenLockError, match="is not a known codegen lock format version"): + parse_lock(f"lock_version = {bad_version}\n" + _LOCK_WITHOUT_VERSION) + + def test_an_unknown_key_is_a_malformed_lock(self) -> None: + with pytest.raises(CodegenLockError, match="Malformed codegen lock"): + parse_lock(_LOCK_WITHOUT_VERSION + "extra = 1\n") + + def test_malformed_toml_is_a_malformed_lock(self) -> None: + with pytest.raises(CodegenLockError, match="Malformed codegen lock"): + parse_lock("this is [not toml") + + @pytest.mark.parametrize("tracked", [["../escape.py"], ["models.py", "models.py"]]) + def test_an_unsafe_or_duplicate_tracked_path_is_a_containment_error_not_a_lock_error(self, tracked: list[str]) -> None: + entries = "".join(f'\n[[artifacts]]\npath = "{path}"\ncontent_hash = "0"\n' for path in tracked) + + with pytest.raises(CodegenError, match="Unsafe codegen artifact path") as exc_info: + parse_lock(_LOCK_WITHOUT_VERSION + entries) + + assert not isinstance(exc_info.value, CodegenLockError) + + def test_load_lock_returns_none_when_there_is_no_lock(self, tmp_path: Path) -> None: + assert load_lock(tmp_path / "codegen.lock") is None + + def test_load_lock_names_the_path_of_a_malformed_lock(self, tmp_path: Path) -> None: + lock_path = tmp_path / "codegen.lock" + lock_path.write_bytes(b"\xff\xfe not utf-8") + + with pytest.raises(CodegenLockError, match="Unreadable codegen lock at"): + load_lock(lock_path) + + def test_validate_artifact_path_returns_the_relative_filesystem_form(self) -> None: + assert validate_artifact_path("nested/deeper/models.ts") == Path("nested", "deeper", "models.ts") diff --git a/tests/unit/test_codegen_stamp.py b/tests/unit/test_codegen_stamp.py new file mode 100644 index 0000000..11acb02 --- /dev/null +++ b/tests/unit/test_codegen_stamp.py @@ -0,0 +1,34 @@ +"""The stamp facts a tree writer depends on: which file types are stamped, in which comment syntax, and the +begin-line predicate that decides whether a file on disk belongs to codegen (mirroring `pipelex/codegen/stamp.py`). +""" + +import pytest + +from pipelex_sdk.codegen_stamp import STAMPABLE_SUFFIXES, comment_prefix_for, has_stamp +from pipelex_sdk.errors import CodegenError + + +class TestCodegenStamp: + def test_the_stampable_suffixes_are_python_and_typescript(self) -> None: + assert frozenset({".py", ".ts"}) == STAMPABLE_SUFFIXES + + @pytest.mark.parametrize(("artifact_path", "expected_prefix"), [("models.py", "#"), ("nested/schemas.ts", "//")]) + def test_comment_prefix_follows_the_suffix(self, artifact_path: str, expected_prefix: str) -> None: + assert comment_prefix_for(artifact_path) == expected_prefix + + def test_an_unstampable_file_type_has_no_comment_prefix(self) -> None: + with pytest.raises(CodegenError, match="not stampable"): + comment_prefix_for("sources.json") + + @pytest.mark.parametrize( + ("content", "comment_prefix", "expected"), + [ + ("# >>> pipelex-codegen-stamp >>>\n# <<< pipelex-codegen-stamp <<<\nbody\n", "#", True), + ("// >>> pipelex-codegen-stamp >>>\n// <<< pipelex-codegen-stamp <<<\nbody\n", "//", True), + ("// >>> pipelex-codegen-stamp >>>\nbody\n", "#", False), + ("\n# >>> pipelex-codegen-stamp >>>\nbody\n", "#", False), + ("handwritten = True\n", "#", False), + ], + ) + def test_has_stamp_reads_only_the_opening_line(self, content: str, comment_prefix: str, expected: bool) -> None: + assert has_stamp(content, comment_prefix=comment_prefix) is expected diff --git a/tests/unit/test_codegen_writer.py b/tests/unit/test_codegen_writer.py new file mode 100644 index 0000000..9040c1c --- /dev/null +++ b/tests/unit/test_codegen_writer.py @@ -0,0 +1,391 @@ +"""`write_codegen_tree`: a `/v1/codegen` response on disk, byte for byte. + +The pipelex fixtures below are real engine output, produced by `pipelex.codegen.emission.build_stamped_projection`, +the pure half of the `write_stamped_projection` a local `pipelex codegen types` run calls. A tree these tests +accept is therefore the tree that run writes. The behaviour pinned here is that function's: validate everything +before writing, never overwrite a file codegen does not own, write only what changed, prune what the previous lock +tracked and the new set dropped, and write the lock last. +""" + +import os +from pathlib import Path + +import pytest + +from pipelex_sdk.codegen_writer import write_codegen_tree +from pipelex_sdk.crate_models import CodegenValidReport, GeneratedArtifact +from pipelex_sdk.errors import CodegenError, CodegenLockError + +_FINGERPRINT = "f" * 64 + +_PIPELEX_MODELS_PY = ( + "# >>> pipelex-codegen-stamp >>>\n" + f"# crate_fingerprint: {_FINGERPRINT}\n" + "# engine_version: 0.55.0\n" + "# projection: types / python-pydantic\n" + "# options: {}\n" + "# content_hash: d3ae42f924ea654dc34df7a2199ef4c42833b0189758af122f8a2bddcbb1f358\n" + "# <<< pipelex-codegen-stamp <<<\n" + "from pydantic import BaseModel\n\n\nclass Invoice(BaseModel):\n total: float\n" +) +_PIPELEX_EXTRA_PY = ( + "# >>> pipelex-codegen-stamp >>>\n" + f"# crate_fingerprint: {_FINGERPRINT}\n" + "# engine_version: 0.55.0\n" + "# projection: types / python-pydantic\n" + "# options: {}\n" + "# content_hash: bb97e71874d3a97080723f8984bfb17af8bf40d80f22f1d7d2223784cb3e8a2e\n" + "# <<< pipelex-codegen-stamp <<<\n" + "X = 'é'\n" +) +_PIPELEX_LOCK = ( + "# codegen.lock — generated artifact set (Pipelex codegen). Do not edit by hand.\n\n" + "lock_version = 1\n" + f'crate_fingerprint = "{_FINGERPRINT}"\n' + 'engine_version = "0.55.0"\n\n' + '[[artifacts]]\npath = "models.py"\ncontent_hash = "d3ae42f924ea654dc34df7a2199ef4c42833b0189758af122f8a2bddcbb1f358"\n\n' + '[[artifacts]]\npath = "nested/extra.py"\ncontent_hash = "bb97e71874d3a97080723f8984bfb17af8bf40d80f22f1d7d2223784cb3e8a2e"\n' +) + + +def _stamped(body: str) -> str: + """A Python artifact behind a stamp fence. The recorded hash is a placeholder: the writer never reads it.""" + return f"# >>> pipelex-codegen-stamp >>>\n# content_hash: {'0' * 64}\n# <<< pipelex-codegen-stamp <<<\n{body}" + + +def _lock_tracking(*paths: str) -> str: + entries = "".join(f'\n[[artifacts]]\npath = "{path}"\ncontent_hash = "{"0" * 64}"\n' for path in paths) + return f'lock_version = 1\ncrate_fingerprint = "{_FINGERPRINT}"\nengine_version = "0.55.0"\n{entries}' + + +def _report(artifacts: dict[str, str], *, lock: str | None = None, lock_filename: str = "codegen.lock") -> CodegenValidReport: + return CodegenValidReport( + is_valid=True, + kind="types", + target="python-pydantic", + crate_fingerprint=_FINGERPRINT, + engine_version="0.55.0", + artifacts=[GeneratedArtifact(path=path, content=content) for path, content in artifacts.items()], + lock=lock if lock is not None else _lock_tracking(*artifacts), + lock_filename=lock_filename, + message="ok", + ) + + +def _pipelex_report() -> CodegenValidReport: + return _report({"models.py": _PIPELEX_MODELS_PY, "nested/extra.py": _PIPELEX_EXTRA_PY}, lock=_PIPELEX_LOCK) + + +class TestCodegenWriter: + # ── Writing verbatim ───────────────────────────────────────────── + + def test_writes_every_artifact_at_its_path_and_the_lock_as_lock_filename_byte_for_byte(self, tmp_path: Path) -> None: + output_dir = tmp_path / "generated" + + result = write_codegen_tree(_pipelex_report(), output_dir=output_dir) + + assert (output_dir / "models.py").read_bytes() == _PIPELEX_MODELS_PY.encode("utf-8") + assert (output_dir / "nested" / "extra.py").read_bytes() == _PIPELEX_EXTRA_PY.encode("utf-8") + assert (output_dir / "codegen.lock").read_bytes() == _PIPELEX_LOCK.encode("utf-8") + assert result.written == ["models.py", "nested/extra.py"] + assert result.unchanged == [] + assert result.removed == [] + assert result.lock_written is True + + def test_writes_nothing_else_into_the_output_directory(self, tmp_path: Path) -> None: + write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + + on_disk = sorted(path.relative_to(tmp_path).as_posix() for path in tmp_path.rglob("*") if path.is_file()) + assert on_disk == ["codegen.lock", "models.py", "nested/extra.py"] + + def test_keeps_line_endings_and_non_ascii_bytes_exactly_as_sent(self, tmp_path: Path) -> None: + """Text mode would rewrite each line feed as the platform separator; the tree must not depend on the platform.""" + content = _stamped("crlf = 'a'\r\nlf = 'é'\n") + + write_codegen_tree(_report({"models.py": content}), output_dir=tmp_path) + + assert (tmp_path / "models.py").read_bytes() == content.encode("utf-8") + + def test_rewriting_a_current_tree_touches_nothing(self, tmp_path: Path) -> None: + write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + tracked = [tmp_path / "models.py", tmp_path / "nested" / "extra.py", tmp_path / "codegen.lock"] + for path in tracked: + os.utime(path, ns=(1_000_000_000, 1_000_000_000)) + + result = write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + + assert result.written == [] + assert result.unchanged == ["models.py", "nested/extra.py"] + assert result.lock_written is False + assert [path.stat().st_mtime_ns for path in tracked] == [1_000_000_000] * len(tracked) + + def test_rewrites_only_the_artifact_that_changed(self, tmp_path: Path) -> None: + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), "other.py": _stamped("B = 1\n")}), output_dir=tmp_path) + + result = write_codegen_tree(_report({"models.py": _stamped("A = 2\n"), "other.py": _stamped("B = 1\n")}), output_dir=tmp_path) + + assert result.written == ["models.py"] + assert result.unchanged == ["other.py"] + assert (tmp_path / "models.py").read_text(encoding="utf-8") == _stamped("A = 2\n") + + # ── Pruning ────────────────────────────────────────────────────── + + def test_prunes_a_delisted_artifact_that_still_carries_its_stamp(self, tmp_path: Path) -> None: + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), "gone/old.py": _stamped("OLD = 1\n")}), output_dir=tmp_path) + + result = write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + + assert result.removed == ["gone/old.py"] + assert not (tmp_path / "gone" / "old.py").exists() + assert (tmp_path / "codegen.lock").read_text(encoding="utf-8") == _lock_tracking("models.py") + + def test_keeps_a_delisted_artifact_whose_stamp_was_removed_by_hand(self, tmp_path: Path) -> None: + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), "old.py": _stamped("OLD = 1\n")}), output_dir=tmp_path) + (tmp_path / "old.py").write_text("OLD = 1 # adopted by hand\n", encoding="utf-8") + + result = write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + + assert result.removed == [] + assert (tmp_path / "old.py").read_text(encoding="utf-8") == "OLD = 1 # adopted by hand\n" + + def test_leaves_a_stamped_file_the_previous_lock_never_tracked(self, tmp_path: Path) -> None: + """Only the previous lock's set is pruned, as pipelex does; the offline check is what reports such a file.""" + write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + (tmp_path / "copied.py").write_text(_stamped("COPIED = 1\n"), encoding="utf-8") + + result = write_codegen_tree(_report({"models.py": _stamped("A = 2\n")}), output_dir=tmp_path) + + assert result.removed == [] + assert (tmp_path / "copied.py").exists() + + def test_skips_a_tracked_artifact_that_is_already_gone(self, tmp_path: Path) -> None: + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), "old.py": _stamped("OLD = 1\n")}), output_dir=tmp_path) + (tmp_path / "old.py").unlink() + + result = write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + + assert result.removed == [] + + def test_a_corrupt_previous_lock_is_replaced_and_prunes_nothing(self, tmp_path: Path) -> None: + (tmp_path / "codegen.lock").write_text("this is [not toml", encoding="utf-8") + (tmp_path / "old.py").write_text(_stamped("OLD = 1\n"), encoding="utf-8") + + result = write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + + assert result.removed == [] + assert result.lock_written is True + assert (tmp_path / "old.py").exists() + assert (tmp_path / "codegen.lock").read_text(encoding="utf-8") == _lock_tracking("models.py") + + def test_a_previous_lock_of_an_unknown_version_is_replaced(self, tmp_path: Path) -> None: + (tmp_path / "codegen.lock").write_text(_lock_tracking("old.py").replace("lock_version = 1", "lock_version = 2"), encoding="utf-8") + + result = write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + + assert result.lock_written is True + assert (tmp_path / "codegen.lock").read_text(encoding="utf-8") == _lock_tracking("models.py") + + def test_refuses_a_previous_lock_that_tracks_an_unsafe_path(self, tmp_path: Path) -> None: + output_dir = tmp_path / "generated" + output_dir.mkdir() + (tmp_path / "escape.py").write_text(_stamped("VICTIM = 1\n"), encoding="utf-8") + (output_dir / "codegen.lock").write_text(_lock_tracking("../escape.py"), encoding="utf-8") + + with pytest.raises(CodegenError, match="Unsafe codegen artifact path") as exc_info: + write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=output_dir) + + assert not isinstance(exc_info.value, CodegenLockError) + assert (tmp_path / "escape.py").exists() + assert not (output_dir / "models.py").exists() + + def test_refuses_a_tracked_path_that_became_a_directory_before_writing_anything(self, tmp_path: Path) -> None: + """Prune targets are resolved in the preflight: refusing one while pruning would leave rewritten artifacts beside the old lock.""" + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), "old.py": _stamped("OLD = 1\n")}), output_dir=tmp_path) + previous_lock = (tmp_path / "codegen.lock").read_bytes() + (tmp_path / "old.py").unlink() + (tmp_path / "old.py").mkdir() + + with pytest.raises(CodegenError, match="not a regular file"): + write_codegen_tree(_report({"models.py": _stamped("A = 2\n")}), output_dir=tmp_path) + + assert (tmp_path / "models.py").read_text(encoding="utf-8") == _stamped("A = 1\n") + assert (tmp_path / "codegen.lock").read_bytes() == previous_lock + + def test_refuses_a_tracked_path_behind_a_symlink_before_writing_anything(self, tmp_path: Path) -> None: + output_dir = tmp_path / "generated" + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), "sub/old.py": _stamped("OLD = 1\n")}), output_dir=output_dir) + elsewhere = tmp_path / "elsewhere" + (output_dir / "sub").rename(elsewhere) + (output_dir / "sub").symlink_to(elsewhere, target_is_directory=True) + + with pytest.raises(CodegenError, match="symbolic link component is not allowed"): + write_codegen_tree(_report({"models.py": _stamped("A = 2\n")}), output_dir=output_dir) + + assert (elsewhere / "old.py").exists() + assert (output_dir / "models.py").read_text(encoding="utf-8") == _stamped("A = 1\n") + + def test_skips_a_tracked_path_whose_directory_became_a_regular_file(self, tmp_path: Path) -> None: + """No file can sit under a regular file, so there is nothing to prune: `pipelex` skips it, and refusing would block every rerun.""" + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), "sub/old.py": _stamped("OLD = 1\n")}), output_dir=tmp_path) + (tmp_path / "sub" / "old.py").unlink() + (tmp_path / "sub").rmdir() + (tmp_path / "sub").write_text("a hand-written file named sub\n", encoding="utf-8") + + result = write_codegen_tree(_report({"models.py": _stamped("A = 2\n")}), output_dir=tmp_path) + + assert result.written == ["models.py"] + assert result.removed == [] + assert (tmp_path / "sub").read_text(encoding="utf-8") == "a hand-written file named sub\n" + assert (tmp_path / "codegen.lock").read_text(encoding="utf-8") == _lock_tracking("models.py") + + # ── Ownership ──────────────────────────────────────────────────── + + def test_refuses_to_overwrite_an_unowned_file_and_writes_nothing(self, tmp_path: Path) -> None: + (tmp_path / "nested").mkdir() + (tmp_path / "nested" / "extra.py").write_text("handwritten = True\n", encoding="utf-8") + + with pytest.raises(CodegenError, match="Refusing to overwrite unowned file"): + write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + + assert (tmp_path / "nested" / "extra.py").read_text(encoding="utf-8") == "handwritten = True\n" + assert not (tmp_path / "models.py").exists() + assert not (tmp_path / "codegen.lock").exists() + + def test_an_unowned_file_already_identical_to_the_artifact_is_accepted(self, tmp_path: Path) -> None: + (tmp_path / "models.py").write_bytes(_PIPELEX_MODELS_PY.encode("utf-8")) + + result = write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + + assert result.unchanged == ["models.py"] + + def test_overwrites_a_stamped_file_the_previous_lock_never_tracked(self, tmp_path: Path) -> None: + """Ownership reads the stamp's begin line only. `pipelex` also requires the header to parse, and refuses this file (L-260912-bb83ca).""" + (tmp_path / "models.py").write_text(_stamped("STALE = 1\n"), encoding="utf-8") + + result = write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + + assert "models.py" in result.written + assert (tmp_path / "models.py").read_bytes() == _PIPELEX_MODELS_PY.encode("utf-8") + + def test_overwrites_an_unstamped_file_the_previous_lock_tracked(self, tmp_path: Path) -> None: + write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + (tmp_path / "models.py").write_text("A = 1 # stamp stripped by a formatter\n", encoding="utf-8") + + result = write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}), output_dir=tmp_path) + + assert result.written == ["models.py"] + assert (tmp_path / "models.py").read_text(encoding="utf-8") == _stamped("A = 1\n") + + # ── Refusals before the first byte ─────────────────────────────── + + def test_refuses_a_lock_filename_other_than_codegen_lock(self, tmp_path: Path) -> None: + output_dir = tmp_path / "generated" + + with pytest.raises(CodegenError, match=r"whose lock is named 'types\.lock'"): + write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}, lock_filename="types.lock"), output_dir=output_dir) + + assert not output_dir.exists() + + @pytest.mark.parametrize( + "unsafe_path", + ["", "../escape.py", "/absolute.py", "C:/drive.py", "nested\\windows.py", "nested//empty.py", "./dot.py", "notes.txt", "control\x00.py"], + ) + def test_refuses_an_unsafe_artifact_path_and_writes_nothing(self, tmp_path: Path, unsafe_path: str) -> None: + output_dir = tmp_path / "generated" + + with pytest.raises(CodegenError, match="Unsafe codegen artifact path"): + write_codegen_tree(_report({"models.py": _stamped("A = 1\n"), unsafe_path: _stamped("X = 1\n")}, lock=""), output_dir=output_dir) + + assert not output_dir.exists() + + @pytest.mark.parametrize( + ("artifact_paths", "lock_paths"), + [(["models.py"], ["models.py", "hand.py"]), (["models.py", "other.py"], ["models.py"])], + ) + def test_refuses_a_response_whose_lock_does_not_track_exactly_its_artifacts( + self, tmp_path: Path, artifact_paths: list[str], lock_paths: list[str] + ) -> None: + """The written lock is what the next run trusts: a path it tracks may be overwritten without a stamp, and pruned.""" + output_dir = tmp_path / "generated" + report = _report({path: _stamped("X = 1\n") for path in artifact_paths}, lock=_lock_tracking(*lock_paths)) + + with pytest.raises(CodegenError, match="does not track exactly its artifacts"): + write_codegen_tree(report, output_dir=output_dir) + + assert not output_dir.exists() + + @pytest.mark.parametrize("lock", ["this is [not toml", _lock_tracking("models.py").replace("lock_version = 1", "lock_version = 2")]) + def test_refuses_a_response_lock_this_sdk_cannot_read(self, tmp_path: Path, lock: str) -> None: + """Unlike a corrupt previous lock, which is replaced, a response lock that cannot be read is refused: writing it would switch pruning off.""" + output_dir = tmp_path / "generated" + + with pytest.raises(CodegenError, match="whose lock this SDK cannot read") as exc_info: + write_codegen_tree(_report({"models.py": _stamped("A = 1\n")}, lock=lock), output_dir=output_dir) + + assert not isinstance(exc_info.value, CodegenLockError) + assert not output_dir.exists() + + def test_refuses_a_response_lock_that_tracks_an_unsafe_path(self, tmp_path: Path) -> None: + output_dir = tmp_path / "generated" + report = _report({"models.py": _stamped("A = 1\n")}, lock=_lock_tracking("models.py", "../escape.py")) + + with pytest.raises(CodegenError, match=r"Unsafe codegen artifact path '\.\./escape\.py'"): + write_codegen_tree(report, output_dir=output_dir) + + assert not output_dir.exists() + + def test_refuses_a_duplicate_artifact_path(self, tmp_path: Path) -> None: + report = _report({"models.py": _stamped("A = 1\n")}) + duplicated = report.model_copy(update={"artifacts": [*report.artifacts, GeneratedArtifact(path="models.py", content=_stamped("A = 2\n"))]}) + + with pytest.raises(CodegenError, match="duplicate artifact path"): + write_codegen_tree(duplicated, output_dir=tmp_path / "generated") + + assert not (tmp_path / "generated").exists() + + def test_refuses_a_symlinked_output_root(self, tmp_path: Path) -> None: + real_root = tmp_path / "real" + real_root.mkdir() + (tmp_path / "link").symlink_to(real_root, target_is_directory=True) + + with pytest.raises(CodegenError, match="output root must not be a symbolic link"): + write_codegen_tree(_pipelex_report(), output_dir=tmp_path / "link") + + assert list(real_root.iterdir()) == [] + + def test_refuses_a_symlink_on_the_way_to_a_destination(self, tmp_path: Path) -> None: + output_dir = tmp_path / "generated" + output_dir.mkdir() + elsewhere = tmp_path / "elsewhere" + elsewhere.mkdir() + (output_dir / "nested").symlink_to(elsewhere, target_is_directory=True) + + with pytest.raises(CodegenError, match="symbolic link component is not allowed"): + write_codegen_tree(_pipelex_report(), output_dir=output_dir) + + assert list(elsewhere.iterdir()) == [] + assert not (output_dir / "models.py").exists() + + def test_refuses_a_file_where_an_artifact_needs_a_directory_before_writing_anything(self, tmp_path: Path) -> None: + (tmp_path / "nested").write_text("not a directory\n", encoding="utf-8") + + with pytest.raises(CodegenError, match="a component on the way to it is not a directory"): + write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + + assert not (tmp_path / "models.py").exists() + assert not (tmp_path / "codegen.lock").exists() + + def test_refuses_a_destination_that_is_a_directory(self, tmp_path: Path) -> None: + (tmp_path / "models.py").mkdir() + + with pytest.raises(CodegenError, match="not a regular file"): + write_codegen_tree(_pipelex_report(), output_dir=tmp_path) + + assert not (tmp_path / "codegen.lock").exists() + + def test_refuses_an_output_root_that_is_a_file(self, tmp_path: Path) -> None: + output_file = tmp_path / "generated" + output_file.write_text("", encoding="utf-8") + + with pytest.raises(CodegenError, match="output root exists but is not a directory"): + write_codegen_tree(_pipelex_report(), output_dir=output_file) From 9c220fa360816f2edc0b033c33c7e4821d7ba985 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 02:10:27 +0200 Subject: [PATCH 8/9] =?UTF-8?q?feature/Codegen-drift-check=20=C2=B7=20L-26?= =?UTF-8?q?0830-4e43cd=20=E2=80=94=20CI=20gate=20without=20the=20pipelex?= =?UTF-8?q?=20runtime=20(#29)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `run_codegen_check(root=…)` verifies a generated codegen tree against its `codegen.lock` by hashing alone — no engine boot, no network, no API key and no dependency on `pipelex`, which is the point: until now the only offline gate a Python project could use was the `pipelex` CLI, the whole runtime, exactly the dependency a hosted-API consumer took this SDK to avoid. It is the counterpart of `write_codegen_tree` and takes the directory that writer wrote into, and it mirrors `pipelex`'s own `run_codegen_check` down to the drift detail sentences so a consumer reads the same report from either. The stamp module gains the reader it needs, `CodegenLock` gains `hash_by_path()`, and the lock is now read in text mode so universal-newline translation folds a CRLF away as pipelex's reader does. Two divergences from the reference are deliberate, and both are named in the README and in `docs/architecture.md`. A relaxation inherited from `@pipelex/sdk`: a well-formed projection line whose axes are outside this SDK's vocabulary is accepted rather than called a hand edit, because an SDK copy of that vocabulary can lag the emitter and would otherwise redden a correctly generated tree. And a tightening of this reader's own, found by review: a Python artifact declaring a PEP 263 source encoding is refused, because that declaration chooses the codec CPython decodes the file with and so makes the stamp header's comment-prefix gate unsound — a tree can otherwise read as current to both readers while importing it executes a statement hidden in the unhashed header. Closes L-260830-4e43cd 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) --- .gitattributes | 5 + CHANGELOG.md | 2 + README.md | 20 +- docs/architecture.md | 26 +- pipelex_sdk/codegen_check.py | 302 +++++++++++ pipelex_sdk/codegen_lock.py | 53 +- pipelex_sdk/codegen_stamp.py | 191 ++++++- pipelex_sdk/errors.py | 11 +- .../unit/data/real_codegen_tree/codegen.lock | 9 + .../unit/data/real_codegen_tree/models.py.txt | 127 +++++ tests/unit/test_codegen_check.py | 497 ++++++++++++++++++ tests/unit/test_codegen_lock.py | 17 + tests/unit/test_codegen_stamp.py | 120 ++++- 13 files changed, 1344 insertions(+), 36 deletions(-) create mode 100644 .gitattributes create mode 100644 pipelex_sdk/codegen_check.py create mode 100644 tests/unit/data/real_codegen_tree/codegen.lock create mode 100644 tests/unit/data/real_codegen_tree/models.py.txt create mode 100644 tests/unit/test_codegen_check.py diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..9ab6726 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,5 @@ +# The codegen check's fixture tree is genuine `pipelex codegen types` output, committed byte for byte so +# the suite re-proves the verdict over real engine bytes. Its content hashes cover those bytes exactly, so +# a checkout that rewrote its line endings (`core.autocrlf=true`) would hand the suite a different tree +# than the one the stamp and the lock were computed over. Pin it. +tests/unit/data/real_codegen_tree/** -text diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e7e6dd..b3b2639 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ ### Added +- **`run_codegen_check`, the offline codegen drift check**: `pipelex_sdk.codegen_check.run_codegen_check(root=…)` verifies a generated tree against its `codegen.lock` by hashing alone — no engine boot, no network, no API key, and no dependency on `pipelex`. That last point is the reason it exists: until now the only offline gate a Python project could use was the `pipelex` CLI, the whole runtime, which is exactly the dependency a hosted-API consumer took this SDK to avoid, so the same integration advice left a TypeScript project with a CI drift gate and a Python one without. It is the counterpart of `write_codegen_tree` and takes the directory that writer wrote into. The verdict rides a structured `CodegenCheckReport` — `lock_found`, `is_current`, and a `drifts` list whose categories are `missing`, `modified`, `hand-edited` and `orphan`, at most one per locked artifact, locked drifts first in ascending path order and then orphans — with the lock header's `crate_fingerprint` and `engine_version` surfaced so a caller can ask the one question the offline check cannot, whether the tree still matches what the method resolves to. A lock that cannot be read, or a tree whose paths are not safe and canonical, raises `CodegenLockError`: the absence of a verdict rather than a drift. It is a mirror of `pipelex`'s own `run_codegen_check` down to the drift sentences, so a consumer reads the same report from either, and the agreement was established by running both over scenario trees derived from real `pipelex codegen types` output, one drift class at a time. Two divergences are deliberate and documented: a relaxation inherited from `@pipelex/sdk`, which accepts a well-formed projection line whose axes are outside this SDK's vocabulary rather than report a tree generated by a newer engine as entirely hand-edited; and a tightening of this reader's own, which refuses a Python artifact declaring a PEP 263 source encoding, because that declaration chooses the codec CPython decodes the file with and so makes the header's comment-prefix gate unsound — a tree can otherwise read as current while importing it executes a statement hidden in the unhashed header. Unlike the JS export, which is pure because it must stay browser-bundleable, this one owns the tree walk and the decoding, so none of that SDK's caller obligations carry over; an unreadable file or directory under the root is therefore a `CodegenLockError` rather than the reference's bare `PermissionError`, leaving a CI caller one class to catch. +- **`pipelex_sdk.codegen_stamp` gains the stamp reader the check needs** — `parse_stamped`, `compute_content_hash`, `is_stampable_artifact_path` and the `ParsedStamp` model, beside the ownership predicate and stampable suffixes it already carried — and `CodegenLock` gains `hash_by_path()`. A `codegen.lock` is now read in text mode, so universal-newline translation folds a CRLF away before the TOML parser sees it, as `pipelex`'s reader does; the writer still compares raw bytes, deliberately, because translation there would make one response two different trees across platforms. - **`write_codegen_tree`, the verbatim codegen tree writer**: `pipelex_sdk.codegen_writer.write_codegen_tree(report, output_dir=…)` writes a valid `codegen()` response to disk byte for byte — every artifact at its `path`, the lock as `codegen.lock` — so a Python project regenerates a tree identical to a local `pipelex codegen types` run without installing `pipelex`. Like that command it validates every path before writing, raises `CodegenError` rather than overwrite a file codegen does not own, rewrites only what changed, and prunes stamped artifacts the previous lock tracked; unlike it, it also checks the paths it is about to prune and requires the response's lock to track exactly its artifacts before the first write; the lock format and path rules it reads are public in `pipelex_sdk.codegen_lock` and `pipelex_sdk.codegen_stamp`. - **`prepare_inputs` takes the method three ways.** Beside inline `files`, it accepts a `method_ref` address (resolved by the runner) or a stored `method_id` (resolved by the hosted platform) — exactly one per call, all three server-resolved, nothing expanded client-side. Empty is absent (`files=[]`, a blank `method_ref` / `method_id`) but the wrong type is not: none, several, or a non-string selector raises `InputPreparationError` before any request leaves the process — reading a mistyped selector as absent would let the exclusivity check pass and prepare against the wrong method. A non-string `pipe_ref` is refused on the same boundary, rather than being absorbed by the pipe defaulting. A method addressed by URL that declares a file input now has an input-preparation path; previously it had none, even though the request beneath accepted the address. - **`PipelexValidationReport.default_pipe_ref`** — the qualified `pipe_ref` a caller gets by omitting the pipe selector, or `null` when the closure declares none or several. Optional and read leniently: a runner that predates the field sends nothing, and `prepare_inputs` falls back to the opaque `bundle_blueprint.main_pipe`. diff --git a/README.md b/README.md index 5521494..e57a2b0 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,24 @@ print(written.written, written.removed) It never overwrites a file codegen does not own, rewrites only what changed, and prunes stamped artifacts that dropped out of the set. Commit the tree; do not run a formatter over it. +### Gate a committed tree in CI, with no key and no `pipelex` + +`run_codegen_check` is the writer's counterpart: pure hashing over the tree and its lock, so it boots no engine, reaches no network and needs no API key. Point it at the directory you generated into: + +```python +from pathlib import Path + +from pipelex_sdk.codegen_check import run_codegen_check + +report = run_codegen_check(root=Path("src/generated/documents")) +if not report.is_current: + for drift in report.drifts: + print(f"{drift.path}: {drift.category} — {drift.detail}") + raise SystemExit(1) +``` + +The drift categories are `pipelex codegen check`'s, and so are the sentences: an artifact edited below its stamp is `hand-edited`, one off the locked hash is `modified`, one the lock tracks and disk has lost is `missing`, and a stamped file the lock does not track is an `orphan` — the stale-artifact class a per-file stamp cannot catch alone. The two readers reach the same verdict over the same bytes apart from two deliberate divergences, both documented in `docs/architecture.md`: this one accepts a projection line whose axes are outside its own vocabulary, where the CLI calls such a tree hand-edited, and it refuses a Python artifact that declares a PEP 263 source encoding, where the CLI calls that one current. Regeneration stays a developer action, because it needs the engine; the check is the CI action, because it needs only hashes, so an upstream template improvement never reddens your pipeline. Whether the tree still matches what the *method* resolves to is a separate question the engine alone can answer — compare `report.crate_fingerprint` against a live `codegen()` response to close it. + ### Long runs: start + poll explicitly Behind the hosted gateway, a synchronous `execute()` is cut off at ~30s and surfaces a `PipelineExecuteTimeoutError` pointing here. For long methods, drive the durable lifecycle yourself — the run survives client disconnects and is resumable by `pipeline_run_id`: @@ -122,7 +140,7 @@ There is no barrel import — package `__init__.py` files stay empty. Import eac - **Run lifecycle types** — `from pipelex_sdk.runs import RunStatus, RunPublic, RunRead, RunResults, RunResultState, WaitForResultOptions, PollInfo` - **Product wire models** — `from pipelex_sdk.product_models import UserProfile, MethodData, MethodWriteInput, Membership, MembershipsResponse, SubscriptionResponse, PlanView, InvoiceView, OnboardingSubmission, UploadInput, UploadedFile, PipelineRun, ...` - **Validation verdict types** — `from pipelex_sdk.validation_models import PipelexValidationResult, PipelexValidationReport, PipelexInvalidReport, ValidationErrorItem, SuggestedFix, VALIDATION_VIEW_INPUT_FORM, ...` -- **Codegen tree** — `from pipelex_sdk.codegen_writer import write_codegen_tree, CodegenTreeWriteReport`, with the format primitives in `pipelex_sdk.codegen_lock` (`CodegenLock`, `parse_lock`, `load_lock`, `validate_artifact_path`, ...) and `pipelex_sdk.codegen_stamp` +- **Codegen tree** — `from pipelex_sdk.codegen_writer import write_codegen_tree, CodegenTreeWriteReport` to write one, `from pipelex_sdk.codegen_check import run_codegen_check, CodegenCheckReport, CodegenDrift, DriftCategory` to verify one, with the format primitives in `pipelex_sdk.codegen_lock` (`CodegenLock`, `parse_lock`, `load_lock`, `validate_artifact_path`, ...) and `pipelex_sdk.codegen_stamp` (`STAMPABLE_SUFFIXES`, `is_stampable_artifact_path`, `compute_content_hash`, `parse_stamped`, ...) - **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, PagingNotTerminatingError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError, CodegenError, CodegenLockError, ...` - **Version** — `from pipelex_sdk.version import __version__` - **Protocol surface** (the MTHDS standard's wire types) comes from the `mthds` dependency — e.g. `from mthds.protocol.exceptions import PipelineRequestError`, `from mthds.protocol.models import ValidationResult` (the neutral verdict union that `PipelexValidationResult` narrows). diff --git a/docs/architecture.md b/docs/architecture.md index 78fd905..2768ed8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -192,10 +192,28 @@ Their closure selector is the strict three-way XOR — inline `files`, an addres - **Prune what dropped out.** A file the previous lock tracked, absent from the new set and still stamped, is deleted; a file whose stamp was removed by hand, or that no lock ever tracked, is left for the offline check to report. - **Write the lock last.** A previous lock that cannot be read (malformed, not UTF-8, an unknown `lock_version`) is replaced and prunes nothing; a previous lock tracking an unsafe path is refused, because that is a containment violation rather than corrupt state. -The format primitives sit in two modules of their own, mirroring `pipelex`'s layout so the offline check can build on them: `pipelex_sdk/codegen_lock.py` (the lock model and parser, the version gate, the artifact path rules and the symlink-refusing output resolution) and `pipelex_sdk/codegen_stamp.py` (the stampable suffixes, their comment syntax, and the begin-line predicate that decides ownership). Nothing in either builds a stamp or encodes a lock; the server does both. +The format primitives sit in two modules of their own, mirroring `pipelex`'s layout so the offline check can build on them: `pipelex_sdk/codegen_lock.py` (the lock model and parser, the version gate, the artifact path rules and the symlink-refusing output resolution) and `pipelex_sdk/codegen_stamp.py` (the stampable suffixes, their comment syntax, the ownership predicate, and the stamp parser and content hash the check needs). Nothing in either builds a stamp or encodes a lock; the server does both. **Synchronous on purpose.** It is local file I/O with no network leg, so it is a plain function called after `await client.codegen(...)`, like `pipelex`'s writer, rather than an `async def` that would only wrap blocking calls. +## Offline codegen drift check (`run_codegen_check`) + +`run_codegen_check(root=…)` (`pipelex_sdk/codegen_check.py`) is the writer's counterpart and the reason a Python consumer of the hosted API needs no `pipelex` install at all: pure hashing over a generated tree and its `codegen.lock`, so it boots no engine, opens no socket and reads no API key. Before it existed the only offline gate was the `pipelex` CLI — the whole runtime, which is precisely the dependency a hosted-API consumer took this SDK to avoid, and `pipelex-starter-python` had been pointing a `PIPELEX ?=` variable at an install it does not depend on for exactly that reason. `@pipelex/sdk` has carried the TypeScript half since v0.12.0; this closes the asymmetry. + +**What it proves, and what it cannot.** It proves the tree and its lock still agree: nothing edited, nothing missing, nothing lingering. Whether the tree still matches what the *method* resolves to is a second question that needs the engine, so the check never asks it — the report surfaces `crate_fingerprint` and `engine_version` from the lock header so a caller can close that gap against a live `codegen()` response. The split is the point: regeneration is a **developer** action because it needs the engine, the check is the **CI** action because it needs only hashes, so an upstream template improvement never reddens a consumer's pipeline. + +**Four drift categories, one per artifact at most.** A locked artifact absent on disk is `missing`; one whose body is off the locked hash while its stamp still agrees is `modified`; one whose stamp is gone, unparseable or self-inconsistent is `hand-edited`; a stamped file the lock does not track is an `orphan` — the stale-artifact class a per-file stamp cannot catch alone, and the whole reason a lock sits beside the stamps. `hand-edited` outranks `modified`, so a hand edit trips both conditions and is reported once. The order is contractual: locked drifts first in ascending path order, then orphans in that same order. A missing lock is `lock_found: False` rather than a drift — there is nothing to check — and `is_current` still refuses to vouch for such a tree. A lock that cannot be read, or a tree whose paths are not safe and canonical, raises `CodegenLockError`: the absence of a verdict, never a drift. + +**A mirror of `pipelex.codegen.check.run_codegen_check`, verbatim down to the drift sentences**, because the codegen spec pins the algorithm as pure hashing precisely so every client reaches the same verdict over the same bytes; a consumer moving between `pipelex codegen check` and this function must read the same report. Artifacts are read in text mode, so universal-newline translation folds a CRLF away before anything is hashed — not a convenience but what keeps the two in agreement, since `pipelex`'s reader does the same and a Windows-generated tree would otherwise read as entirely hand-edited here and current there. The orphan scan prunes the same vendor, VCS and cache directories and skips symbolic links for the same reason. + +**Two deliberate divergences, one in each direction.** The **relaxation** is inherited from the TypeScript port: `parse_stamped` requires the projection line to be present and well-formed but does not match its `kind` / `target` against this SDK's vocabulary. `pipelex` validates them against its own enums, which cannot lag its own emitter; an SDK copy can — `CodegenKind` here is a single-member `Literal` — and rejecting an unknown-but-valid future axis would report every artifact of a correctly generated tree as hand-edited. For today's vocabulary the two readers are identical on that axis. + +The **tightening** is this reader's own: a Python artifact declaring a PEP 263 source encoding is refused as hand-edited, where `pipelex` and `@pipelex/sdk` both call it current. Such a declaration makes the comment-prefix gate unsound, because CPython chooses the file's codec from it before tokenizing, so under an escape-decoding codec a header value can become a line break followed by an executable statement — in the one region of the file no hash covers. It is the same hole the `splitlines` rule closes for U+2028, reached through another door, and closing it rejects nothing a real generated tree contains because the emitter never writes a `coding:` line. Diverging by catching a tampered tree the reference passes is the safe direction for a CI gate; the gap in the other two implementations is filed on the ledger. + +Beyond those two the verdicts match, including the drift sentences. One difference is not a verdict at all: where the reference lets a `PermissionError` out of an unreadable file or directory, this one raises `CodegenLockError`, so a CI caller has a single class to catch. Neither produces a verdict for that state. + +**What the committed fixture proves, and what it does not.** `tests/unit/data/real_codegen_tree/` holds a genuine `pipelex codegen types` tree — engine 0.57.0 over the cookbook's `documents` method, byte for byte, the artifact suffixed `.txt` only so this repo's linters leave it alone — and the suite re-proves on every run that *this* implementation reads real engine bytes as current, flips to `hand-edited` with the exact sentence when a line is appended, and is unmoved by a CRLF checkout. Its hashes are self-proving: recompute SHA-256 over the body below the fence and it equals both the stamp's recorded value and the lock's. What the suite cannot re-prove is agreement with `pipelex`, because this package must not depend on it — that is the point of the module. Cross-implementation parity was established once, by an out-of-tree harness that ran both implementations as subprocesses in separate virtualenvs over scenario trees derived from real engine output, one drift class at a time. Re-proving it per run needs a home that may depend on both, which is the workspace's `conformance/` suite rather than this repo. + ## Pipelex product surface (hosted management routes) The hosted catalog/account routes the webapp drives (`pipelex_sdk/product_models.py` + the client's product methods). Every route rides the same `{base}/v1/*` surface, `Authorization: Bearer`, org-from-JWT contract as the protocol routes, and goes through `_request_product`, which maps a non-2xx `problem+json` to a typed `ApiResponseError` — **consumers branch on `.code`, never the HTTP status**. @@ -247,13 +265,13 @@ This SDK is a port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`) and tra - **Tooling routes** — `lint`, `format` (`resolve` and `codegen` shipped in 0.8.0 with the method selectors and are **not** gaps). - **Authoring helpers** — `build_output`, `build_runner`, `concept`, `pipe_spec`. JS still exports its `buildInputs` wrapper; Python's was **deleted** rather than left unused, because `prepare_inputs` was its only caller. A deliberate divergence, not a gap: the JS wrappers retire together as their own step of the same program. -- **Offline helpers** — `run_codegen_check` (the codegen drift check: `write_codegen_tree` puts a tree on disk, but nothing in this SDK verifies one yet) and `get_method_closure` (client-side sugar that parses the polymorphic `mthds` source into a run-ready closure — in the JS SDK it is the documented migration target for the deleted by-id expansion legs; this SDK never had such legs, so the utility stays deferred rather than required). +- **Offline helpers** — `get_method_closure` (client-side sugar that parses the polymorphic `mthds` source into a run-ready closure — in the JS SDK it is the documented migration target for the deleted by-id expansion legs; this SDK never had such legs, so the utility stays deferred rather than required). `run_codegen_check` was the other entry here and is no longer a gap: it shipped as `pipelex_sdk.codegen_check`, with a filesystem-shaped signature where the JS export is pure. Each stays deferred rather than silently missing. Everything else — the protocol routes, the durable lifecycle, the whole product surface, and the errors — does have a Python equivalent. -**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), the crate routes (`resolve`, `codegen`) with the verbatim `write_codegen_tree`, the input-preparation surface (`upload_file` / `prepare_inputs`, taking all three method selectors and reading its signature from the input-form descriptor), and `health`. The method selectors (`method_ref` / `method_id`) match the JS v0.16.0 surface across the run and tooling methods, with one signature-shape divergence: JS `validate` takes a `ValidateMethodSelector` object in place of its first argument, while Python takes `method_ref=` / `method_id=` keyword parameters — same wire, same XOR, idiomatic per language. +**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), the crate routes (`resolve`, `codegen`) with the verbatim `write_codegen_tree` and the offline `run_codegen_check`, the input-preparation surface (`upload_file` / `prepare_inputs`, taking all three method selectors and reading its signature from the input-form descriptor), and `health`. The method selectors (`method_ref` / `method_id`) match the JS v0.16.0 surface across the run and tooling methods, with one signature-shape divergence: JS `validate` takes a `ValidateMethodSelector` object in place of its first argument, while Python takes `method_ref=` / `method_id=` keyword parameters — same wire, same XOR, idiomatic per language. -**Models** — field-for-field across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. One addition runs ahead of the JS SDK: `write_codegen_tree` has no `@pipelex/sdk` counterpart, because the JS writer lives inside `pipelex-starter-js`'s own harness, mixed in with project policy; the byte-fidelity contract is small and load-bearing enough to belong to the SDK, where every consumer shares one correct implementation. Two divergences worth naming: the page envelopes keep the wire's snake_case `next_cursor`, where the JS mirror renamed it `nextCursor` for its own consumers; and the method-files catalog converter (`parse_method_files` / `serialize_method_files`) lives in this package, where the JS pair lives in `mthds-js` because `pipelex-mcp` consumes the same format and wanted one owner. There is no second Python consumer, and the catalog serialization is a Pipelex product concern rather than an MTHDS protocol one, so this SDK is a proper home for it. If `mthds-python` ever grows an owner for the format, this SDK adopts it then. +**Models** — field-for-field across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. One addition runs ahead of the JS SDK: `write_codegen_tree` has no `@pipelex/sdk` counterpart, because the JS writer lives inside `pipelex-starter-js`'s own harness, mixed in with project policy; the byte-fidelity contract is small and load-bearing enough to belong to the SDK, where every consumer shares one correct implementation. The drift check goes the other way and takes a different shape on purpose: `@pipelex/sdk`'s `runCodegenCheck` is **pure** — the caller walks its own tree and hands in the text — because that module must stay free of Node builtins for a browser bundle, and its doc consequently loads the caller with obligations (walk the whole tree, do not reformat, decode strictly) whose every breach yields a wrong verdict rather than an error. Python has no such constraint, this package already does filesystem work, and `pipelex`'s own surface takes a root — so `run_codegen_check(root=…)` takes one too. Every caller obligation becomes the library's, the verdict is provable against `pipelex` by calling both with the same directory, and it composes with `write_codegen_tree(report, output_dir=…)` as the same path in and out. Two divergences worth naming: the page envelopes keep the wire's snake_case `next_cursor`, where the JS mirror renamed it `nextCursor` for its own consumers; and the method-files catalog converter (`parse_method_files` / `serialize_method_files`) lives in this package, where the JS pair lives in `mthds-js` because `pipelex-mcp` consumes the same format and wanted one owner. There is no second Python consumer, and the catalog serialization is a Pipelex product concern rather than an MTHDS protocol one, so this SDK is a proper home for it. If `mthds-python` ever grows an owner for the format, this SDK adopts it then. **Errors** — `ApiResponseError`, `ApiUnreachableError`, `PipelineExecuteTimeoutError`, `RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError`, `PagingNotTerminatingError` are owned here; `RunStillRunningError` is re-exported from `mthds`. `ClientAuthenticationError` is **not** ported: it is a dormant export in the JS barrel (defined and exported but never raised by the client), and in Python it already lives in `mthds.runners.api.exceptions` — importable directly if ever needed, with no barrel here to re-export it through. diff --git a/pipelex_sdk/codegen_check.py b/pipelex_sdk/codegen_check.py new file mode 100644 index 0000000..d7ae9e1 --- /dev/null +++ b/pipelex_sdk/codegen_check.py @@ -0,0 +1,302 @@ +"""The offline codegen drift check: pure hashing over a tree and its lock — no engine, no network, no key. + +`client.codegen()` returns stamped artifacts plus a `codegen.lock`, and `write_codegen_tree` puts them on +disk; a project that commits that tree needs a CI gate over it. This is that gate, and it is the reason a +Python consumer of the hosted API needs no `pipelex` install to have one: before this existed, the only +offline gate available was the `pipelex` CLI — the whole runtime, which is exactly the dependency a +hosted-API consumer took this SDK to avoid. `@pipelex/sdk` has carried the TypeScript half since v0.12.0; +this is its counterpart. + +**What it proves, and what it cannot.** It proves the tree and its lock still agree with each other: no +artifact edited, none missing, none lingering. Whether the tree still matches what the *method* resolves +to is a second question, and answering it needs the engine — so the check never asks it. A caller closes +that gap by comparing `CodegenCheckReport.crate_fingerprint` against a live `codegen()` response. This +split is the point: regeneration is a **dev action** (it needs the engine), the check is the **CI action** +(it needs only hashes), so an upstream template improvement never reddens a consumer's CI. + +**A mirror, deliberately.** The algorithm is `pipelex.codegen.check.run_codegen_check`, and the spec pins +it as pure hashing precisely so that every client — the CLI, an SDK, a short CI script — reaches the same +verdict over the same bytes. The drift `detail` sentences are kept verbatim for the same reason: a +consumer moving between `pipelex codegen check` and this function reads the same report. + +Two divergences are documented, both in `pipelex_sdk.codegen_stamp`: a **relaxation**, which does not +match the projection axes against this SDK's vocabulary, and a **tightening**, which refuses a Python +artifact that declares a PEP 263 source encoding because such a declaration makes the header's +comment-prefix gate unsound. A third difference is not a divergence of verdict: where the reference lets +a `PermissionError` out of an unreadable file or directory, this module raises `CodegenLockError`, so a +CI caller has one class to catch. Neither reaches a verdict for that state. + +The algorithm, per the codegen spec's "Offline check algorithm": + +1. For each artifact the lock tracks, locate the file and recompute the hash of the body below its stamp; + absent is a `missing` drift, a mismatch is a `modified` drift. +2. A locked file whose stamp is gone, unparseable or self-inconsistent is `hand-edited`. +3. A stamped file the lock does not track is an `orphan` — the stale-artifact class a per-file stamp + cannot catch on its own, which is the whole reason the lock exists beside the stamps. +""" + +from collections.abc import Iterator +from enum import StrEnum +from pathlib import Path + +from pydantic import BaseModel, ConfigDict, Field, computed_field + +from pipelex_sdk._pydantic_utils import empty_list_factory_of +from pipelex_sdk.codegen_lock import CODEGEN_LOCK_FILENAME, CodegenLock, load_lock, resolve_artifact_path, resolve_output_path +from pipelex_sdk.codegen_stamp import comment_prefix_for, compute_content_hash, has_stamp, is_stampable_artifact_path, parse_stamped +from pipelex_sdk.errors import CodegenError, CodegenLockError + +_SKIP_DIRS = frozenset( + {".git", ".hg", ".svn", ".venv", "venv", "node_modules", "__pycache__", "dist", "build", ".next", ".mypy_cache", ".pytest_cache", ".ruff_cache"} +) +"""Vendor / VCS / cache directories the orphan scan never descends into, mirroring pipelex's list. + +They never hold generated output, so pruning them keeps a check run at a project root from reading — and +choking on — unrelated files beneath them. +""" + +_MISSING_DETAIL = "Locked artifact is absent on disk." +_NO_STAMP_DETAIL = "Stamp header is missing or unparseable." +_STAMP_MISMATCH_DETAIL = "Body was edited below the stamp (stamp hash no longer matches)." +_MODIFIED_DETAIL = "Body no longer matches the locked hash — regenerate." +_ORPHAN_DETAIL = "Stamped generated file not tracked by the lock — stale; remove or regenerate." +_NOT_UTF8_DETAIL = "File is not valid UTF-8 — not generated output." + + +class DriftCategory(StrEnum): + """The kind of drift found for one artifact — the canonical wire values, shared with pipelex.""" + + MISSING = "missing" + """Tracked by the lock, absent on disk (a deleted-artifact drift).""" + + MODIFIED = "modified" + """Present, and its stamp is self-consistent, but its body hash no longer matches the locked hash.""" + + HAND_EDITED = "hand-edited" + """Present, but its stamp is missing, unparseable or self-inconsistent (edited below the stamp).""" + + ORPHAN = "orphan" + """A stamped generated file on disk that the lock does not track (a stale lingering artifact).""" + + +class CodegenDrift(BaseModel): + """One drifting artifact: its path relative to the lock, the drift category, and a human detail.""" + + model_config = ConfigDict(frozen=True) + + path: str + category: DriftCategory + detail: str + + +class CodegenCheckReport(BaseModel): + """The structured verdict of one offline check over one output root.""" + + model_config = ConfigDict(frozen=True) + + lock_found: bool + """Whether a `codegen.lock` was there at all. No lock is not a drift — it is nothing to check.""" + + drifts: list[CodegenDrift] = Field(default_factory=empty_list_factory_of(CodegenDrift)) + """Every drift found: locked-artifact drifts first in ascending path order, then orphans in that order.""" + + crate_fingerprint: str | None = None + """The lock header's crate fingerprint, or `None` when no lock was found. + + Surfaced so a caller can compare a committed tree against a live `codegen()` response's + `crate_fingerprint` — the engine-needing comparison this check deliberately never makes itself. + + It is the lock header's value as read, **not** cross-checked against the fingerprint each artifact's + own stamp records, and the check does not compare those stamps to each other either. So a tree whose + artifacts were generated against different crates — the shape a merge taking one artifact and its lock + entry from each side produces — is reported current, with this field naming only the crate the lock + header claims. `pipelex codegen check` has the same blind spot, so closing it is a spec question + rather than this reader's to answer alone. + """ + + engine_version: str | None = None + """The lock header's `pipelex` engine version, or `None` when no lock was found — same purpose, and + the same caveat: it is read from the header and never checked against the artifacts' own stamps.""" + + @computed_field # type: ignore[prop-decorator] + @property + def is_current(self) -> bool: + """Whether the generated tree is in sync: a lock was found and no drift was detected. + + A computed field rather than a bare property, so it survives `model_dump()` and + `model_dump_json()`. A CI gate that serializes the report — the natural shape for one — would + otherwise lose the verdict silently while every other field came through. The report is a local + return value and not a wire contract, so carrying it costs no parity with the reference. + """ + return self.lock_found and not self.drifts + + +def run_codegen_check(*, root: Path) -> CodegenCheckReport: + """Run the offline drift check over `root`, the directory holding `codegen.lock`. + + Pass the same directory `write_codegen_tree` wrote into; the two are counterparts, and a tree that + writer produced from a `codegen()` response **into a directory codegen owns** is current by + construction. The qualifier is the writer's own: a stamped file it never tracked is deliberately left + alone "for the offline check to report", so a stray artifact sharing the output root is an `orphan` + here even on a tree the writer just wrote. + + The drift order is part of the contract, so a second implementation can mirror it exactly: every + locked-artifact drift comes first, in ascending order of the full relative path compared as a plain + string, then every orphan, ordered by that same rule. A locked path yields **at most one** drift, and + `hand-edited` outranks `modified` when a file is both self-inconsistent and off the locked hash. + + Artifacts are read in text mode, so Python's universal-newline translation folds a CRLF and a lone CR + into an LF before anything is hashed. That is not a convenience but what keeps this check in agreement + with `pipelex codegen check`, whose reader does the same: a tree generated on Windows, or checked out + under `core.autocrlf=true`, is current to both readers rather than hand-edited to one. + + Raises `CodegenLockError` for a no-verdict condition — a lock that is malformed, unreadable or of a + `lock_version` this SDK does not know, a tree whose paths are not safe and canonical, or a file or + directory under the root that the process cannot read. A drift is a verdict and rides the report; this + is the absence of one, and it is one class so a CI caller has one thing to catch. The reference lets a + `PermissionError` out of the equivalent paths instead; the verdict is the same in both — there is none. + """ + try: + lock_path = resolve_output_path(root=root, relative_path=Path(CODEGEN_LOCK_FILENAME)) + safe_root = lock_path.parent + lock = load_lock(lock_path) + if lock is None: + return CodegenCheckReport(lock_found=False) + + drifts: list[CodegenDrift] = [] + drifts += _check_locked_artifacts(root=safe_root, lock=lock) + drifts += _find_orphans(root=safe_root, lock=lock) + return CodegenCheckReport( + lock_found=True, + drifts=drifts, + crate_fingerprint=lock.crate_fingerprint, + engine_version=lock.engine_version, + ) + except CodegenLockError: + # Already precise and actionable — re-wrapping it as an unsafe tree would bury the one thing the + # reader needs to know. `CodegenLockError` subclasses `CodegenError`, so this clause comes first. + raise + except CodegenError as exc: + msg = f"Unsafe codegen artifact tree at '{root}': {exc}" + raise CodegenLockError(msg) from exc + + +def _check_locked_artifacts(*, root: Path, lock: CodegenLock) -> list[CodegenDrift]: + drifts: list[CodegenDrift] = [] + for path, locked_hash in sorted(lock.hash_by_path().items()): + file_path = _locate_locked_artifact(root=root, path=path) + if file_path is None: + drifts.append(CodegenDrift(path=path, category=DriftCategory.MISSING, detail=_MISSING_DETAIL)) + continue + drift = _check_present_artifact(path=path, file_path=file_path, locked_hash=locked_hash) + if drift is not None: + drifts.append(drift) + return drifts + + +def _locate_locked_artifact(*, root: Path, path: str) -> Path | None: + """The file at a locked artifact's path, or `None` when nothing is there and it is a `missing` drift. + + `require_directory_components=False` because this is a reader: a regular file where the artifact needs a + parent directory is a writer's refusal and a reader's `missing` drift, which is what `pipelex codegen + check` reports for it. Symbolic links and escapes are still refused. + + Resolving the path and stat-ing it both reach the filesystem, and both raise `EACCES` for an artifact + under a directory the process cannot search. That is the same no-verdict condition the artifact read and + the orphan walk already wrap, and the contract is a single catchable class, so a `PermissionError` must + not escape from this leg either. An unsafe path is a different matter: it raises `CodegenError`, which is + not an `OSError` and so passes through this guard to the caller that wraps it as an unsafe tree. + """ + try: + file_path = resolve_artifact_path(root=root, artifact_path=path, require_directory_components=False) + return file_path if file_path.is_file() else None + except OSError as exc: + msg = f"Unreadable path under the codegen output root at '{root / path}': {exc}" + raise CodegenLockError(msg) from exc + + +def _check_present_artifact(*, path: str, file_path: Path, locked_hash: str) -> CodegenDrift | None: + """The drift for one locked artifact that is present, or `None` when it is current. + + The precedence is the point: returning early makes `hand-edited` outrank `modified`, so a hand edit — + which trips both the stamp check and the lock check — is reported once, as the hand edit. + """ + text = _read_text_or_none(file_path) + if text is None: + return CodegenDrift(path=path, category=DriftCategory.HAND_EDITED, detail=_NOT_UTF8_DETAIL) + # Never raises: a path the lock tracks was suffix-validated when the lock was parsed. + parsed = parse_stamped(text, comment_prefix=comment_prefix_for(path)) + if parsed is None: + return CodegenDrift(path=path, category=DriftCategory.HAND_EDITED, detail=_NO_STAMP_DETAIL) + body_hash = compute_content_hash(parsed.body) + if parsed.content_hash != body_hash: + return CodegenDrift(path=path, category=DriftCategory.HAND_EDITED, detail=_STAMP_MISMATCH_DETAIL) + if body_hash != locked_hash: + return CodegenDrift(path=path, category=DriftCategory.MODIFIED, detail=_MODIFIED_DETAIL) + return None + + +def _find_orphans(*, root: Path, lock: CodegenLock) -> list[CodegenDrift]: + tracked = lock.paths() + orphans: list[CodegenDrift] = [] + for file_path in _iter_stampable_files(directory=root): + relative = file_path.relative_to(root).as_posix() + if relative in tracked: + continue + text = _read_text_or_none(file_path) + if text is not None and has_stamp(text, comment_prefix=comment_prefix_for(relative)): + orphans.append(CodegenDrift(path=relative, category=DriftCategory.ORPHAN, detail=_ORPHAN_DETAIL)) + return sorted(orphans, key=lambda drift: drift.path) + + +def _iter_stampable_files(*, directory: Path) -> Iterator[Path]: + """Yield stampable files under `directory`, pre-order and deterministic, pruning vendor / VCS dirs. + + Symbolic links are skipped rather than refused, as in the reference: the orphan scan reads whatever + happens to share the output root, and a link parked beside the tree is not the tree's fault. A link at + an artifact's own path is a different matter and `resolve_artifact_path` still refuses it. + + A directory the process cannot list is a no-verdict condition rather than an empty directory: skipping + it would hide exactly the stale artifact the orphan scan exists to find, so it is reported as one. + """ + try: + entries = sorted(directory.iterdir()) + except OSError as exc: + msg = f"Unreadable directory under the codegen output root at '{directory}': {exc}" + raise CodegenLockError(msg) from exc + for entry in entries: + try: + # `is_symlink` first and inside the same guard, because it is what keeps a link out of the walk: + # asking `is_dir` about a symlink loop raises where this short-circuit never looks. Stat can fail + # for its own reasons too — a path longer than the platform allows, a component turned + # unsearchable — so the classification is guarded rather than assumed to answer. + if entry.is_symlink(): + continue + is_directory = entry.is_dir() + except OSError as exc: + msg = f"Unreadable entry under the codegen output root at '{entry}': {exc}" + raise CodegenLockError(msg) from exc + if is_directory: + if entry.name not in _SKIP_DIRS: + yield from _iter_stampable_files(directory=entry) + elif is_stampable_artifact_path(entry.name): + yield entry + + +def _read_text_or_none(path: Path) -> str | None: + """Read UTF-8 text, or `None` when the bytes are not UTF-8 — in which case it is not generated output. + + Text mode on purpose: see `run_codegen_check` on universal-newline translation. + + Bytes that are not UTF-8 are a verdict — the file cannot be generated output — but a file that cannot + be *read at all* is the absence of one, and is raised rather than guessed at. Returning `None` for it + would report an unreadable artifact as hand-edited, and a file vanishing between the walk and this read + would report a stale artifact as absent: both are wrong verdicts where no verdict is available. + """ + try: + return path.read_text(encoding="utf-8") + except UnicodeDecodeError: + return None + except OSError as exc: + msg = f"Unreadable file under the codegen output root at '{path}': {exc}" + raise CodegenLockError(msg) from exc diff --git a/pipelex_sdk/codegen_lock.py b/pipelex_sdk/codegen_lock.py index 4716b0b..b4ead74 100644 --- a/pipelex_sdk/codegen_lock.py +++ b/pipelex_sdk/codegen_lock.py @@ -67,6 +67,11 @@ def paths(self) -> set[str]: """The set of tracked artifact paths, relative to the lock.""" return set(validate_artifact_paths(entry.path for entry in self.artifacts)) + def hash_by_path(self) -> dict[str, str]: + """Map each tracked artifact path to the locked hash of its body — what the offline check compares.""" + validate_artifact_paths(entry.path for entry in self.artifacts) + return {entry.path: entry.content_hash for entry in self.artifacts} + def parse_lock(content: str) -> CodegenLock: """Parse the text of a `codegen.lock`. @@ -99,7 +104,12 @@ def load_lock(lock_path: Path) -> CodegenLock | None: if not lock_path.is_file(): return None try: - content = lock_path.read_bytes().decode("utf-8") + # Text mode, so Python's universal-newline translation folds `\r\n` and a lone `\r` into `\n` before + # the TOML parser sees them. That is what `pipelex`'s reader does (`load_text_from_path` is + # `read_text(encoding="utf-8")`), so a lock checked out with CRLF reads identically on both sides. + # The writer reads *bytes* for its write-if-changed comparison, deliberately and for the opposite + # reason: there, translation would make the same response two different trees across platforms. + content = lock_path.read_text(encoding="utf-8") except (OSError, UnicodeDecodeError) as exc: msg = f"Unreadable codegen lock at '{lock_path}': {exc}" raise CodegenLockError(msg) from exc @@ -145,20 +155,33 @@ def validate_artifact_paths(paths: Iterable[str]) -> dict[str, Path]: return validated -def resolve_artifact_path(*, root: Path, artifact_path: str) -> Path: - """Resolve a validated artifact path beneath `root` without following any symbolic link.""" - return resolve_output_path(root=root, relative_path=validate_artifact_path(artifact_path)) +def resolve_artifact_path(*, root: Path, artifact_path: str, require_directory_components: bool = True) -> Path: + """Resolve a validated artifact path beneath `root` without following any symbolic link. + + See `resolve_output_path` on `require_directory_components`: a writer wants it, a reader does not. + """ + return resolve_output_path( + root=root, + relative_path=validate_artifact_path(artifact_path), + require_directory_components=require_directory_components, + ) -def resolve_output_path(*, root: Path, relative_path: Path) -> Path: +def resolve_output_path(*, root: Path, relative_path: Path, require_directory_components: bool = True) -> Path: """Resolve one output file beneath `root`, refusing a symbolic link anywhere on the way to it. The root itself must not be a symbolic link and, when it exists, must be a directory. Every component - below it must not be a symbolic link, every one on the way to the destination that already exists must - be a directory, the resolved destination must stay inside the root, and a destination that already - exists must be a regular file. The directory check is what lets a writer refuse a tree before its first - write: without it, a regular file where an artifact needs a parent directory is only discovered when - creating that directory fails, after the artifacts before it were written. + below it must not be a symbolic link, the resolved destination must stay inside the root, and a + destination that already exists must be a regular file. + + `require_directory_components` additionally refuses a component on the way to the destination that + exists but is not a directory, and it is **a writer's guard, not a reader's**. It is what lets a writer + refuse a tree before its first write: without it, a regular file where an artifact needs a parent + directory is only discovered when creating that directory fails, after the artifacts before it were + written. A reader has no such stake and must pass `False`, because to a reader that state is not a + violation at all — it simply means no file can be at the artifact's path, which is an ordinary + `missing` drift and a verdict. `pipelex`'s reader has no equivalent guard, so keeping it on would make + this SDK raise where the reference reports a drift. """ if relative_path.is_absolute() or not relative_path.parts or any(part in {"", ".", ".."} for part in relative_path.parts): _raise_path_error(str(relative_path), reason="internal output path must be a canonical relative path") @@ -171,7 +194,11 @@ def resolve_output_path(*, root: Path, relative_path: Path) -> Path: _raise_path_error(str(normalized_root), reason="output root exists but is not a directory") destination = normalized_root / relative_path - _reject_unsafe_components(root=normalized_root, relative_path=relative_path) + _reject_unsafe_components( + root=normalized_root, + relative_path=relative_path, + require_directory_components=require_directory_components, + ) if not destination.resolve(strict=False).is_relative_to(normalized_root): _raise_path_error(str(destination), reason=f"resolved path escapes output root '{normalized_root}'") if destination.exists() and not destination.is_file(): @@ -179,13 +206,13 @@ def resolve_output_path(*, root: Path, relative_path: Path) -> Path: return destination -def _reject_unsafe_components(*, root: Path, relative_path: Path) -> None: +def _reject_unsafe_components(*, root: Path, relative_path: Path, require_directory_components: bool) -> None: current = root for depth, part in enumerate(relative_path.parts, start=1): current /= part if current.is_symlink(): _raise_path_error(str(root / relative_path), reason=f"symbolic link component is not allowed: '{current}'") - if depth < len(relative_path.parts) and current.exists() and not current.is_dir(): + if require_directory_components and depth < len(relative_path.parts) and current.exists() and not current.is_dir(): _raise_path_error(str(root / relative_path), reason=f"a component on the way to it is not a directory: '{current}'") diff --git a/pipelex_sdk/codegen_stamp.py b/pipelex_sdk/codegen_stamp.py index 7a28a73..562629b 100644 --- a/pipelex_sdk/codegen_stamp.py +++ b/pipelex_sdk/codegen_stamp.py @@ -1,30 +1,86 @@ -"""The codegen stamp header, as far as a tree writer needs it. +"""The codegen stamp header: the grammar that lets a lone generated file testify about itself. -Mirrors `pipelex/codegen/stamp.py`, the reference for the stamp grammar. Every generated artifact opens -with a fenced comment block (`# >>> pipelex-codegen-stamp >>>` … `# <<< pipelex-codegen-stamp <<<` in -Python, the same fence behind `//` in TypeScript) recording the crate fingerprint, the engine version, -the projection and a hash of the body below the fence. +Mirrors `pipelex/codegen/stamp.py`, the reference for that grammar. Every generated artifact opens with a +fenced comment block (`# >>> pipelex-codegen-stamp >>>` … `# <<< pipelex-codegen-stamp <<<` in Python, the +same fence behind `//` in TypeScript) recording the crate fingerprint, the engine version, the projection +and a hash of the body below the fence. Two readers in this SDK live off it. -A writer needs two facts from that grammar and nothing more. The stampable suffixes are the only file -types that can ever be an artifact, which makes them part of the path rules in -`pipelex_sdk.codegen_lock`. And the begin-line predicate decides whether a file already on disk belongs -to codegen: the writer overwrites or prunes a file only when it does, and the offline check calls a -stamped file the lock does not track an orphan by that same predicate, so the two agree on ownership by -construction. +The **writer** needs two facts and nothing more. The stampable suffixes are the only file types that can +ever be an artifact, which makes them part of the path rules in `pipelex_sdk.codegen_lock`. And +`has_stamp`, the begin-line predicate, decides whether a file already on disk belongs to codegen: the +writer overwrites or prunes a file only when it does. + +The **offline check** needs the rest: `parse_stamped` splits a file into the hash its stamp recorded and +the body that hash covers, and `compute_content_hash` recomputes it. That is the whole of the drift +verdict for one file — no engine, no network, no key. + +Two deliberate divergences from the reference, one in each direction. + +The **relaxation** is inherited from `@pipelex/sdk`'s port: the projection line must be *present and +well-formed*, but its `kind` / `target` values are not checked against this SDK's vocabulary. `pipelex` +validates them against its own enums, which cannot lag its own emitter; an SDK copy can, and rejecting +an unknown-but-valid future `kind` would report every artifact in the tree as hand-edited. For today's +vocabulary the two readers are identical. + +The **tightening** is this reader's own: a Python artifact declaring a PEP 263 source encoding is +refused, because such a declaration makes the comment-prefix gate below unsound — CPython decodes the +file before tokenizing it, so a header line can carry an escape that becomes executable code the hashes +never cover. `pipelex` and `@pipelex/sdk` both miss it today. It is the only case where this reader +reports a drift the reference calls current, and it rejects nothing a real generated tree contains: +`_declares_a_python_source_encoding` carries the reasoning. Nothing here builds or rewrites a stamp. The server stamps, and the SDK keeps the bytes it was sent. """ +import hashlib +import json +import re from pathlib import PurePosixPath +from typing import NoReturn + +from pydantic import BaseModel, ConfigDict from pipelex_sdk.errors import CodegenError _BEGIN_MARKER = ">>> pipelex-codegen-stamp >>>" +_END_MARKER = "<<< pipelex-codegen-stamp <<<" + +_PYTHON_COMMENT_PREFIX = "#" + +_PEP_263_CODING_DECLARATION = re.compile(r"^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)") +"""PEP 263's own source-encoding pattern, which CPython honours on a file's first two lines only.""" _COMMENT_PREFIX_BY_SUFFIX = {".py": "#", ".ts": "//"} STAMPABLE_SUFFIXES = frozenset(_COMMENT_PREFIX_BY_SUFFIX) -"""The file suffixes codegen stamps, mirroring pipelex's `STAMPABLE_SUFFIXES`.""" +"""The file suffixes codegen stamps, mirroring pipelex's `STAMPABLE_SUFFIXES`. + +A file whose suffix is not here can never be an artifact and can never be an orphan, so the orphan scan +skips it rather than refusing it — which is what lets a project park a sidecar such as `sources.json` +beside the lock. `is_stampable_artifact_path` is the predicate over it, and the one the scan itself calls. +""" + + +class ParsedStamp(BaseModel): + """A stamp read back off a file: the hash it recorded, and the body text that hash covers.""" + + model_config = ConfigDict(frozen=True) + + content_hash: str + """The `content_hash` field as the stamp spells it — recomputed from `body` by the offline check.""" + + body: str + """Everything below the end-marker line, byte-exact, so rehashing it reproduces the recorded value.""" + + +def is_stampable_artifact_path(artifact_path: str) -> bool: + """Whether `artifact_path` names a file type codegen stamps, and therefore one the check considers. + + The offline check's own orphan scan filters on this, so a caller reasoning about a path — deciding + whether a file beside a tree could be an artifact at all — reaches the same answer by construction + rather than by keeping a second copy of the rule in step. + """ + return PurePosixPath(artifact_path).suffix in STAMPABLE_SUFFIXES def comment_prefix_for(artifact_path: str) -> str: @@ -40,3 +96,114 @@ def comment_prefix_for(artifact_path: str) -> str: def has_stamp(content: str, *, comment_prefix: str) -> bool: """Whether `content` opens with a codegen stamp block in the given comment syntax.""" return content.startswith(f"{comment_prefix} {_BEGIN_MARKER}") + + +def compute_content_hash(body: str) -> str: + """The canonical content hash of a generated body: lowercase SHA-256 hex over its UTF-8 bytes.""" + return hashlib.sha256(body.encode("utf-8")).hexdigest() + + +def parse_stamped(content: str, *, comment_prefix: str) -> ParsedStamp | None: + """Split a stamped file into the hash its stamp recorded and the body below the fence, or `None`. + + `None` means there is no stamp this reader will trust — a missing or unterminated fence, a header + line that is not a comment, a malformed projection, or options that are not a JSON object — and the + check reports such a file as hand-edited. The body is everything after the end-marker line, + byte-exact, so recomputing its hash reproduces the value the stamp recorded. + """ + begin_line = f"{comment_prefix} {_BEGIN_MARKER}" + end_line = f"{comment_prefix} {_END_MARKER}" + if not content.startswith(begin_line): + return None + end_index = content.find(f"\n{end_line}\n") + if end_index == -1: + return None + header_region = content[len(begin_line) + 1 : end_index] + body = content[end_index + len(end_line) + 2 :] + + # Every line the emitter ever writes inside the fence carries the comment prefix, so anything else in + # there was injected by hand — and it would otherwise verify as pristine, since the hash covers only + # the body below the fence. An executable line hiding inside a "DO NOT EDIT" block is not a valid stamp. + # + # `splitlines` is the deliberate split, as in the reference: it also breaks on U+2028, U+2029 and + # U+0085, and the first two terminate a `//` comment in ECMAScript. Split on `"\n"` alone, a `.ts` + # header carrying a raw U+2028 followed by a statement is one prefixed line to this gate and two lines + # to the JavaScript engine — so the check would report the file current while it executes the injected + # code, since the header itself is not hashed. + if any(not line.startswith(comment_prefix) for line in header_region.splitlines()): + return None + if _declares_a_python_source_encoding(content, comment_prefix=comment_prefix): + return None + + fields = _parse_fields(header_region, comment_prefix=comment_prefix) + projection = fields.get("projection") + if projection is None or not _is_well_formed_projection(projection): + return None + if not _is_json_object(fields.get("options", "{}")): + return None + return ParsedStamp(content_hash=fields.get("content_hash", ""), body=body) + + +def _declares_a_python_source_encoding(content: str, *, comment_prefix: str) -> bool: + """Whether a Python artifact declares a PEP 263 source encoding, which makes the prefix gate a lie. + + The comment-prefix gate above proves every header line *begins* with a comment marker in the bytes on + disk. It does not prove those lines are inert, because CPython decodes the file before it tokenizes it, + and PEP 263 lets the file itself choose the codec from a comment on either of its first two lines. Under + `raw_unicode_escape` or `unicode_escape` a header line carrying a literal six-character u000a escape + becomes two lines once decoded, the second of them executable — and the body below the fence, which is + the only part the hashes cover, is untouched. Both hashes therefore still agree and the tree reads as + current while importing the artifact runs the injected statement. + + It is the same hole the `splitlines` rule above closes for U+2028, reached through a different door, so + it is closed the same way: the emitter never writes a `coding:` line, so refusing one rejects nothing a + correctly generated tree contains. That makes this a *tightening* against `pipelex`, which shares the + gap (as does `@pipelex/sdk`) — the one divergence in this module that reports a drift the reference + calls current, and the safe direction for a CI gate to diverge in. Only Python has such a mechanism; + `.ts` needs no counterpart, since every line terminator ECMAScript honours is one `splitlines` breaks on. + """ + if comment_prefix != _PYTHON_COMMENT_PREFIX: + return False + # PEP 263 reads the first two physical lines and no further, so a `coding:` line below them is inert and + # is left to the ordinary field parsing rather than reported as a drift it is not. + return any(_PEP_263_CODING_DECLARATION.match(line) for line in content.split("\n")[:2]) + + +def _parse_fields(header_region: str, *, comment_prefix: str) -> dict[str, str]: + fields: dict[str, str] = {} + # The same split rule as the gate in `parse_stamped`, and it has to stay the same one: a narrower + # split here would rejoin a line the gate had already split, so a field value would swallow the + # injected text the gate exists to catch. + for raw_line in header_region.splitlines(): + # `parse_stamped` has already rejected any line without the prefix, so stripping it is unconditional. + stripped = raw_line[len(comment_prefix) :].strip() + key, separator, value = stripped.partition(":") + if separator: + fields[key.strip()] = value.strip() + return fields + + +def _is_well_formed_projection(projection: str) -> bool: + """` / `, with an optional trailing ` / ` for a per-pipe projection. + + Shape only: see the module docstring on why the axes' values are not matched against this SDK's + vocabulary. + """ + parts = [segment.strip() for segment in projection.split("/")] + return len(parts) >= 2 and parts[0] != "" and parts[1] != "" + + +def _is_json_object(options_raw: str) -> bool: + try: + # The stamp header is a cross-language interchange format, so a stamp only Python can read is not + # a valid stamp: `parse_constant` turns `NaN` / `Infinity` / `-Infinity` — which Python's `json` + # accepts and conformant parsers refuse — into the `ValueError` below, as the reference does. + loaded = json.loads(options_raw, parse_constant=_reject_json_constant) + except ValueError: # JSONDecodeError is a subclass, so this one clause covers malformed JSON too + return False + return isinstance(loaded, dict) + + +def _reject_json_constant(value: str) -> NoReturn: + msg = f"Non-standard JSON constant in stamp options: {value}" + raise ValueError(msg) diff --git a/pipelex_sdk/errors.py b/pipelex_sdk/errors.py index 7182922..b61fd04 100644 --- a/pipelex_sdk/errors.py +++ b/pipelex_sdk/errors.py @@ -21,9 +21,10 @@ stays in `mthds` — it belongs to the protocol `execute()` 202-degrade path, not the lifecycle — and is re-exported here so consumers have a single import home. -The codegen tree errors (`CodegenError`, `CodegenLockError`) are not request errors at all: they -are raised by `pipelex_sdk.codegen_writer` and `pipelex_sdk.codegen_lock` over bytes and a directory, -so they derive from `Exception` rather than from the protocol base. +The codegen tree errors (`CodegenError`, `CodegenLockError`) are not request errors at all: they are +raised by `pipelex_sdk.codegen_writer`, `pipelex_sdk.codegen_check`, `pipelex_sdk.codegen_lock` and +`pipelex_sdk.codegen_stamp` over +bytes and a directory, so they derive from `Exception` rather than from the protocol base. """ from __future__ import annotations @@ -253,6 +254,10 @@ class CodegenLockError(CodegenError): """A `codegen.lock` that cannot be read: malformed TOML, a shape the format does not define, bytes that are not UTF-8, or a `lock_version` this SDK does not know. + It is also the offline check's one no-verdict class, raised where that check can reach no verdict at + all rather than find a drift — including a file or directory under the output root the process cannot + read, so a CI caller has a single thing to catch. + An unsafe artifact path inside an otherwise well-formed lock is deliberately NOT this error but a plain `CodegenError`: it is a containment violation, not corrupt state a writer may recover from by replacing the lock. diff --git a/tests/unit/data/real_codegen_tree/codegen.lock b/tests/unit/data/real_codegen_tree/codegen.lock new file mode 100644 index 0000000..855b0e8 --- /dev/null +++ b/tests/unit/data/real_codegen_tree/codegen.lock @@ -0,0 +1,9 @@ +# codegen.lock — generated artifact set (Pipelex codegen). Do not edit by hand. + +lock_version = 1 +crate_fingerprint = "38d02d151de391f760bcaa1bf1c376cd617a598964f13db2fcfab0930e1322d1" +engine_version = "0.57.0" + +[[artifacts]] +path = "models.py" +content_hash = "851d1b768c089a94be893f66ba5325d6dc27f5c73124ea353c4f5486a97954f2" diff --git a/tests/unit/data/real_codegen_tree/models.py.txt b/tests/unit/data/real_codegen_tree/models.py.txt new file mode 100644 index 0000000..d67731d --- /dev/null +++ b/tests/unit/data/real_codegen_tree/models.py.txt @@ -0,0 +1,127 @@ +# >>> pipelex-codegen-stamp >>> +# crate_fingerprint: 38d02d151de391f760bcaa1bf1c376cd617a598964f13db2fcfab0930e1322d1 +# engine_version: 0.57.0 +# projection: types / python-pydantic +# options: {} +# content_hash: 851d1b768c089a94be893f66ba5325d6dc27f5c73124ea353c4f5486a97954f2 +# <<< pipelex-codegen-stamp <<< +# --------------------------------------------------------------------------- +# AUTOGENERATED by Pipelex codegen — DO NOT EDIT. +# +# This file is a projection of a normalized MTHDS library crate. It is +# regenerated from the method; any hand edit here is overwritten. +# +# To customize a generated type, do NOT edit this file. Create a sibling +# module and subclass — subclasses survive regeneration: +# +# # my_types_ext.py +# from .structures import Report +# +# class MyReport(Report): +# ... +# +# projection: types / python-pydantic +# --------------------------------------------------------------------------- +from __future__ import annotations + +from pydantic import BaseModel, Field + + +class Document(BaseModel): + """A document""" + + url: str = Field( + ..., + description="The document URL: a storage URI, an HTTP(S) URL, or a base64 data URL", + ) + public_url: str | None = Field( + default=None, + description="The public HTTPS URL of the document", + ) + mime_type: str | None = Field( + default=None, + description="The MIME type of the document", + ) + filename: str | None = Field( + default=None, + description="The original filename of the document", + ) + title: str | None = Field( + default=None, + description="The title of the document or source", + ) + snippet: str | None = Field( + default=None, + description="A text snippet or excerpt from the document", + ) + + +class Image(BaseModel): + """An image""" + + url: str = Field( + ..., + description="The image URL: a storage URI, an HTTP(S) URL, or a base64 data URL", + ) + public_url: str | None = Field( + default=None, + description="The public URL of the image", + ) + source_prompt: str | None = Field( + default=None, + description="The source prompt of the image", + ) + source_negative_prompt: str | None = Field( + default=None, + description="The source negative prompt of the image", + ) + caption: str | None = Field(default=None, description="The caption of the image") + mime_type: str | None = Field( + default=None, + description="The MIME type of the image", + ) + width: int | None = Field( + default=None, + description="The width of the image, in pixels", + ) + height: int | None = Field( + default=None, + description="The height of the image, in pixels", + ) + filename: str | None = Field( + default=None, + description="The original filename of the image", + ) + + +class Page(BaseModel): + """The content of a page of a document, comprising text and linked images and an optional page view image""" + + text_and_images: TextAndImages = Field( + ..., + description="The text and images content extracted from the page", + ) + page_view: Image | None = Field( + default=None, + description="The screenshot of the page", + ) + + +class Text(BaseModel): + """A text""" + + text: str = Field(..., description="The text") + + +class TextAndImages(BaseModel): + """A text and an image""" + + text: Text | None = Field(default=None, description="A text content") + images: list[Image] | None = Field( + default=None, + description="A list of images that were extracted from the text", + ) + raw_html: str | None = Field( + default=None, + description="The raw HTML of the fetched page, if requested", + ) diff --git a/tests/unit/test_codegen_check.py b/tests/unit/test_codegen_check.py new file mode 100644 index 0000000..a91ae4f --- /dev/null +++ b/tests/unit/test_codegen_check.py @@ -0,0 +1,497 @@ +"""`run_codegen_check`: the offline drift verdict over a generated tree and its lock. + +The tree under `data/real_codegen_tree/` is genuine engine output — `pipelex codegen types --target +python-pydantic` over the cookbook's `documents` method, engine 0.57.0, copied in byte for byte (the +artifact carries a `.txt` suffix only so this repo's linters leave it alone; every test renames it back to +`models.py`, which is the path the lock tracks, and `.gitattributes` pins its line endings). + +What that fixture re-proves on every run is that **this** implementation reads real engine bytes as +current, and what it cannot re-prove is agreement with `pipelex`, because this package must not depend on +it — which is the whole point of the module. Its hashes are self-proving, so the fixture cannot rot +unnoticed: recompute SHA-256 over the body below the fence and it equals both the stamp's recorded value +and the lock's. Cross-implementation parity was established once, by an out-of-tree harness running both +implementations in separate virtualenvs; `docs/architecture.md` records what that covered and why +re-proving it per run belongs to the workspace's `conformance/` suite rather than here. + +The smaller fixtures below are the same grammar at a size a reader can hold: a real stamp fence whose +`content_hash` is the true SHA-256 of the body under it, and a real lock tracking the same hashes. +""" + +import hashlib +import json +from pathlib import Path + +import pytest +from pytest_mock import MockerFixture + +from pipelex_sdk.codegen_check import CodegenCheckReport, DriftCategory, run_codegen_check +from pipelex_sdk.errors import CodegenError, CodegenLockError + +_REAL_TREE = Path(__file__).parent / "data" / "real_codegen_tree" +_FINGERPRINT = "f" * 64 +_ENGINE_VERSION = "0.57.0" + + +def _hash(body: str) -> str: + return hashlib.sha256(body.encode("utf-8")).hexdigest() + + +def _stamped(body: str, *, comment_prefix: str = "#", recorded_hash: str | None = None, projection: str = "types / python-pydantic") -> str: + """A stamped artifact in pipelex's exact fence grammar, recording the true hash of `body` by default.""" + return ( + f"{comment_prefix} >>> pipelex-codegen-stamp >>>\n" + f"{comment_prefix} crate_fingerprint: {_FINGERPRINT}\n" + f"{comment_prefix} engine_version: {_ENGINE_VERSION}\n" + f"{comment_prefix} projection: {projection}\n" + f"{comment_prefix} options: {{}}\n" + f"{comment_prefix} content_hash: {recorded_hash if recorded_hash is not None else _hash(body)}\n" + f"{comment_prefix} <<< pipelex-codegen-stamp <<<\n" + f"{body}" + ) + + +def _lock(*entries: tuple[str, str]) -> str: + """A `codegen.lock` in the encoding `pipelex` writes, tracking `(path, body)` pairs by their true hash.""" + artifacts = "".join(f'\n[[artifacts]]\npath = "{path}"\ncontent_hash = "{_hash(body)}"\n' for path, body in entries) + return ( + "# codegen.lock — generated artifact set (Pipelex codegen). Do not edit by hand.\n\n" + f'lock_version = 1\ncrate_fingerprint = "{_FINGERPRINT}"\nengine_version = "{_ENGINE_VERSION}"\n{artifacts}' + ) + + +def _write(root: Path, relative: str, content: str) -> Path: + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + return path + + +def _tree(root: Path, *entries: tuple[str, str]) -> None: + """Write a current tree: each `(path, body)` stamped, and a lock tracking exactly those bodies.""" + for relative, body in entries: + comment_prefix = "#" if relative.endswith(".py") else "//" + _write(root, relative, _stamped(body, comment_prefix=comment_prefix)) + _write(root, "codegen.lock", _lock(*entries)) + + +def _real_tree(root: Path) -> Path: + """Materialize the genuine `pipelex codegen types` tree, artifact back under the path the lock tracks. + + The copy folds any CRLF back to LF. `.gitattributes` pins the fixture to LF, so in a correct checkout + this is a no-op — but a clone made before that pin, or with `core.autocrlf=true`, holds CRLF bytes, and + then `test_a_real_tree_checked_out_with_crlf_is_still_current` would double every carriage return and + fail over its own fixture rather than over the code. The materialized tree is the engine's bytes either + way. + """ + (root / "codegen.lock").write_bytes(_to_lf((_REAL_TREE / "codegen.lock").read_bytes())) + (root / "models.py").write_bytes(_to_lf((_REAL_TREE / "models.py.txt").read_bytes())) + return root / "models.py" + + +def _to_lf(raw: bytes) -> bytes: + return raw.replace(b"\r\n", b"\n").replace(b"\r", b"\n") + + +def _categories(report: CodegenCheckReport) -> list[tuple[str, str]]: + return [(drift.path, drift.category) for drift in report.drifts] + + +class TestRealGeneratedTree: + """The verdict over bytes a real `pipelex codegen types` run wrote.""" + + def test_a_real_generated_tree_is_current(self, tmp_path: Path) -> None: + _real_tree(tmp_path) + report = run_codegen_check(root=tmp_path) + assert report.is_current + assert report.lock_found + assert report.drifts == [] + + def test_the_report_surfaces_the_locks_crate_fingerprint_and_engine_version(self, tmp_path: Path) -> None: + _real_tree(tmp_path) + report = run_codegen_check(root=tmp_path) + # The caller compares these against a live `codegen()` response to close the question the offline + # check cannot ask: whether the tree still matches what the method resolves to. + assert report.crate_fingerprint == "38d02d151de391f760bcaa1bf1c376cd617a598964f13db2fcfab0930e1322d1" + assert report.engine_version == "0.57.0" + + def test_one_appended_line_in_a_real_artifact_is_a_hand_edit(self, tmp_path: Path) -> None: + artifact = _real_tree(tmp_path) + artifact.write_text(artifact.read_text(encoding="utf-8") + "\nSNUCK_IN = True\n", encoding="utf-8") + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("models.py", "hand-edited")] + assert report.drifts[0].detail == "Body was edited below the stamp (stamp hash no longer matches)." + + def test_a_real_tree_checked_out_with_crlf_is_still_current(self, tmp_path: Path) -> None: + artifact = _real_tree(tmp_path) + artifact.write_bytes(artifact.read_bytes().replace(b"\n", b"\r\n")) + # Universal-newline translation, as pipelex's reader applies it: a Windows checkout is not a drift. + assert run_codegen_check(root=tmp_path).is_current + + +class TestTheReportItself: + """What a caller reads off the report, including what survives serializing it.""" + + def test_the_verdict_survives_model_dump(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + report = run_codegen_check(root=tmp_path) + # A CI gate that serializes the report is the natural shape for one, and a bare property would drop + # the verdict out of it silently while every other field came through. + assert report.model_dump()["is_current"] is True + assert json.loads(report.model_dump_json())["is_current"] is True + + def test_the_verdict_survives_model_dump_when_the_tree_has_drifted(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + (tmp_path / "models.py").unlink() + report = run_codegen_check(root=tmp_path) + assert report.model_dump()["is_current"] is False + assert json.loads(report.model_dump_json())["drifts"][0]["category"] == "missing" + + def test_the_reported_fingerprint_is_the_locks_header_and_is_not_cross_checked(self, tmp_path: Path) -> None: + older, newer = "a" * 64, "b" * 64 + body_a, body_b = "A = 1\n", "B = 1\n" + _write(tmp_path, "a.py", _stamped(body_a).replace(_FINGERPRINT, older)) + _write(tmp_path, "b.py", _stamped(body_b).replace(_FINGERPRINT, newer)) + _write(tmp_path, "codegen.lock", _lock(("a.py", body_a), ("b.py", body_b)).replace(_FINGERPRINT, newer)) + report = run_codegen_check(root=tmp_path) + # Two artifacts generated against different crates, each body matching its own stamp and the lock. + # The check is pure hashing, so it reports current and surfaces only what the lock header claims — + # it never compares the artifacts' own stamped fingerprints with the header or with each other. + # `pipelex codegen check` answers identically; the blind spot is the algorithm's, not this port's. + assert report.is_current + assert report.crate_fingerprint == newer + + +class TestCurrentTree: + def test_a_multi_artifact_nested_tree_is_current(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n"), ("nested/deep/schemas.ts", "export const A = 1;\n")) + assert run_codegen_check(root=tmp_path).is_current + + def test_a_non_stampable_sidecar_beside_the_lock_is_ignored(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "sources.json", '{"method": "documents"}\n') + assert run_codegen_check(root=tmp_path).is_current + + def test_an_unstamped_hand_authored_sibling_is_ignored(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "helper.py", "HELPER = 2\n") + assert run_codegen_check(root=tmp_path).is_current + + def test_a_tree_with_no_lock_is_not_checked_and_is_not_current(self, tmp_path: Path) -> None: + _write(tmp_path, "models.py", _stamped("A = 1\n")) + report = run_codegen_check(root=tmp_path) + # No lock is nothing to check rather than a drift — but `is_current` still refuses to vouch for it. + assert report.lock_found is False + assert report.drifts == [] + assert report.is_current is False + assert report.crate_fingerprint is None + assert report.engine_version is None + + def test_a_lock_tracking_nothing_over_an_empty_tree_is_current(self, tmp_path: Path) -> None: + _write(tmp_path, "codegen.lock", _lock()) + assert run_codegen_check(root=tmp_path).is_current + + +class TestDriftCategories: + def test_a_locked_artifact_absent_on_disk_is_missing(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + (tmp_path / "models.py").unlink() + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("models.py", "missing")] + assert report.drifts[0].detail == "Locked artifact is absent on disk." + + def test_a_body_off_the_locked_hash_whose_stamp_still_agrees_is_modified(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + # Restamped, so the file is self-consistent and only the lock disagrees — which is what a stale + # tree regenerated from a newer crate looks like. + _write(tmp_path, "models.py", _stamped("A = 2\n")) + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("models.py", "modified")] + assert report.drifts[0].detail == "Body no longer matches the locked hash — regenerate." + + def test_a_body_edited_below_an_untouched_stamp_is_hand_edited(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "models.py", _stamped("A = 1\nEDIT = True\n", recorded_hash=_hash("A = 1\n"))) + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("models.py", "hand-edited")] + assert report.drifts[0].detail == "Body was edited below the stamp (stamp hash no longer matches)." + + def test_a_locked_artifact_whose_stamp_was_stripped_is_hand_edited(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "models.py", "A = 1\n") + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("models.py", "hand-edited")] + assert report.drifts[0].detail == "Stamp header is missing or unparseable." + + def test_a_locked_artifact_that_is_not_utf8_is_hand_edited(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + (tmp_path / "models.py").write_bytes(b"# >>> pipelex-codegen-stamp >>>\n\xff\xfe\n") + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("models.py", "hand-edited")] + assert report.drifts[0].detail == "File is not valid UTF-8 — not generated output." + + def test_a_stamped_file_the_lock_does_not_track_is_an_orphan(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "stale.py", _stamped("STALE = 1\n")) + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("stale.py", "orphan")] + assert report.drifts[0].detail == "Stamped generated file not tracked by the lock — stale; remove or regenerate." + + def test_an_orphan_is_found_at_any_depth(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "a/b/c/stale.ts", _stamped("export const STALE = 1;\n", comment_prefix="//")) + assert _categories(run_codegen_check(root=tmp_path)) == [("a/b/c/stale.ts", "orphan")] + + @pytest.mark.parametrize("skipped_dir", ["node_modules", ".venv", "__pycache__", ".git", "dist"]) + def test_an_orphan_inside_a_vendor_or_cache_directory_is_not_scanned(self, tmp_path: Path, skipped_dir: str) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, f"{skipped_dir}/stale.py", _stamped("STALE = 1\n")) + assert run_codegen_check(root=tmp_path).is_current + + def test_a_symlinked_orphan_is_skipped_but_its_target_is_still_found(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + target = _write(tmp_path, "stale.py", _stamped("STALE = 1\n")) + (tmp_path / "linked.py").symlink_to(target) + # The scan reads whatever shares the output root; a link parked beside the tree is not the tree's. + assert _categories(run_codegen_check(root=tmp_path)) == [("stale.py", "orphan")] + + +class TestStampParsing: + """What `parse_stamped` refuses, every refusal reported as a hand edit.""" + + def test_an_uncommented_line_inside_the_fence_is_refused(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + stamped = _stamped("A = 1\n").replace("# <<< pipelex-codegen-stamp <<<", "import os\n# <<< pipelex-codegen-stamp <<<") + _write(tmp_path, "models.py", stamped) + # The hash covers only the body BELOW the fence, so an executable line hiding inside a "DO NOT + # EDIT" block would otherwise verify as pristine. + assert _categories(run_codegen_check(root=tmp_path)) == [("models.py", "hand-edited")] + + def test_a_line_separator_hiding_a_statement_in_a_field_is_refused(self, tmp_path: Path) -> None: + _tree(tmp_path, ("nested.ts", "export const A = 1;\n")) + stamped = _stamped("export const A = 1;\n", comment_prefix="//").replace("// options: {}", "// options: {}\u2028import os") + _write(tmp_path, "nested.ts", stamped) + # U+2028 terminates a `//` comment in ECMAScript, so splitting on "\n" alone would read this as one + # commented line while the JavaScript engine reads two and runs the second. + assert _categories(run_codegen_check(root=tmp_path)) == [("nested.ts", "hand-edited")] + + def test_an_unterminated_fence_is_refused(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "models.py", "# >>> pipelex-codegen-stamp >>>\n# content_hash: x\nA = 1\n") + assert _categories(run_codegen_check(root=tmp_path)) == [("models.py", "hand-edited")] + + def test_a_byte_order_mark_above_the_fence_is_refused(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + (tmp_path / "models.py").write_bytes(b"\xef\xbb\xbf" + (tmp_path / "models.py").read_bytes()) + assert _categories(run_codegen_check(root=tmp_path)) == [("models.py", "hand-edited")] + + @pytest.mark.parametrize("options", ["not-json", "[1, 2]", "null", '{"a": NaN}', '{"a": Infinity}']) + def test_options_that_are_not_a_json_object_are_refused(self, tmp_path: Path, options: str) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "models.py", _stamped("A = 1\n").replace("# options: {}", f"# options: {options}")) + # `NaN` / `Infinity` are the interesting pair: Python's `json` accepts them and conformant parsers + # do not, so a stamp only Python could read is not a valid stamp. + assert _categories(run_codegen_check(root=tmp_path)) == [("models.py", "hand-edited")] + + @pytest.mark.parametrize("projection", ["onlyonepart", " / python-pydantic", "types / ", ""]) + def test_a_malformed_projection_line_is_refused(self, tmp_path: Path, projection: str) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "models.py", _stamped("A = 1\n", projection=projection)) + assert _categories(run_codegen_check(root=tmp_path)) == [("models.py", "hand-edited")] + + def test_a_missing_projection_line_is_refused(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "models.py", _stamped("A = 1\n").replace("# projection: types / python-pydantic\n", "")) + assert _categories(run_codegen_check(root=tmp_path)) == [("models.py", "hand-edited")] + + @pytest.mark.parametrize("projection", ["futurekind / python-pydantic", "types / future-target", "types / python-pydantic / domain.some_pipe"]) + def test_projection_axes_outside_this_sdks_vocabulary_are_accepted(self, tmp_path: Path, projection: str) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "models.py", _stamped("A = 1\n", projection=projection)) + # The one documented relaxation against `pipelex`, inherited from `@pipelex/sdk`: the axes must be + # present and well-formed, not members of a vocabulary an SDK copy is free to lag. Validating them + # would report every artifact of a tree generated by a newer engine as hand-edited. + assert run_codegen_check(root=tmp_path).is_current + + +class TestDriftOrdering: + def test_locked_drifts_come_first_in_path_order_then_orphans_in_path_order(self, tmp_path: Path) -> None: + _tree(tmp_path, ("zz.py", "Z = 1\n"), ("aa.py", "A = 1\n"), ("mm/nested.py", "M = 1\n")) + (tmp_path / "zz.py").unlink() + (tmp_path / "aa.py").unlink() + (tmp_path / "mm" / "nested.py").unlink() + _write(tmp_path, "zzz-orphan.py", _stamped("O = 1\n")) + _write(tmp_path, "aaa-orphan.py", _stamped("O = 1\n")) + assert _categories(run_codegen_check(root=tmp_path)) == [ + ("aa.py", "missing"), + ("mm/nested.py", "missing"), + ("zz.py", "missing"), + ("aaa-orphan.py", "orphan"), + ("zzz-orphan.py", "orphan"), + ] + + def test_a_file_that_is_both_hand_edited_and_off_the_locked_hash_drifts_once_as_the_hand_edit(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + # Body changed AND the stamp left recording the old hash: both conditions trip, one drift is reported. + _write(tmp_path, "models.py", _stamped("A = 99\n", recorded_hash=_hash("A = 1\n"))) + report = run_codegen_check(root=tmp_path) + assert _categories(report) == [("models.py", "hand-edited")] + assert report.drifts[0].category is DriftCategory.HAND_EDITED + + +class TestNoVerdict: + """A lock or a tree the check cannot reason about at all — an error, never a drift.""" + + def test_a_malformed_lock_raises(self, tmp_path: Path) -> None: + _write(tmp_path, "codegen.lock", "lock_version = = 1\n") + with pytest.raises(CodegenLockError, match="Malformed codegen lock"): + run_codegen_check(root=tmp_path) + + def test_a_future_lock_version_names_the_side_to_upgrade(self, tmp_path: Path) -> None: + _write(tmp_path, "codegen.lock", _lock().replace("lock_version = 1", "lock_version = 2")) + with pytest.raises(CodegenLockError, match="upgrade pipelex-sdk"): + run_codegen_check(root=tmp_path) + + def test_an_unknown_lock_key_raises(self, tmp_path: Path) -> None: + _write(tmp_path, "codegen.lock", _lock() + "\nfuture_key = 3\n") + with pytest.raises(CodegenLockError, match="Malformed codegen lock"): + run_codegen_check(root=tmp_path) + + @pytest.mark.parametrize("tracked_path", ["../escape.py", "/etc/passwd.py", "models.txt", "./models.py", "nested/../models.py"]) + def test_a_lock_tracking_an_unsafe_path_raises_rather_than_drifting(self, tmp_path: Path, tracked_path: str) -> None: + _write(tmp_path, "codegen.lock", _lock(("models.py", "A = 1\n")).replace('path = "models.py"', f'path = "{tracked_path}"')) + # `CodegenLockError` subclasses `CodegenError`, and the path refusal is wrapped as the former so a + # consumer has one no-verdict class to catch. + with pytest.raises(CodegenLockError, match="Unsafe codegen artifact"): + run_codegen_check(root=tmp_path) + + def test_a_symbolic_link_at_a_locked_artifacts_path_raises(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + elsewhere = _write(tmp_path, "elsewhere.txt", (tmp_path / "models.py").read_text(encoding="utf-8")) + (tmp_path / "models.py").unlink() + (tmp_path / "models.py").symlink_to(elsewhere) + with pytest.raises(CodegenLockError, match="symbolic link component is not allowed"): + run_codegen_check(root=tmp_path) + + def test_an_output_root_that_is_a_regular_file_raises(self, tmp_path: Path) -> None: + root = _write(tmp_path, "not-a-directory", "") + with pytest.raises(CodegenLockError, match="exists but is not a directory"): + run_codegen_check(root=root) + + def test_an_unreadable_locked_artifact_is_a_no_verdict_error_not_a_drift(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + (tmp_path / "models.py").chmod(0o000) + try: + # Bytes that are not UTF-8 are a verdict — not generated output — but bytes that cannot be read + # are the absence of one. Reporting this as hand-edited would be a wrong verdict, and letting the + # `PermissionError` out would break the one class a CI caller has to catch. + with pytest.raises(CodegenLockError, match="Unreadable file under the codegen output root"): + run_codegen_check(root=tmp_path) + finally: + (tmp_path / "models.py").chmod(0o644) + + def test_a_locked_artifact_that_cannot_be_located_is_a_no_verdict_error(self, tmp_path: Path, mocker: MockerFixture) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + # Locating a locked artifact is a third filesystem leg beside reading one and walking the tree, and it + # failed the same way: resolving the path stats every component and `is_file` stats the destination, so + # an `EACCES` or an `ENAMETOOLONG` escaped as a bare `OSError` while the other two legs were wrapped — + # a hole in the single class a CI caller catches, on the path every run takes first. + # + # Injected at the call rather than reproduced with `chmod`, for the same reason the walk's twin below + # is: the real trigger is interpreter-dependent. Python 3.11 to 3.13 propagate `EACCES` out of + # `is_file()` for an artifact under an unsearchable directory; 3.14 swallows it and answers `False`, so + # on that interpreter the same tree reaches no verdict by a different leg. The guard is what is under + # test, not the platform's errno. + real_is_file = Path.is_file + + def failing_is_file(self: Path) -> bool: + if self.name == "models.py": + raise OSError(63, "File name too long") + return real_is_file(self) + + mocker.patch.object(Path, "is_file", failing_is_file) + with pytest.raises(CodegenLockError, match="Unreadable path under the codegen output root"): + run_codegen_check(root=tmp_path) + + def test_an_entry_the_walk_cannot_stat_is_a_no_verdict_error(self, tmp_path: Path, mocker: MockerFixture) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + # `iterdir` is not the only syscall in the walk: classifying an entry stats it, and that fails on a + # path longer than the platform allows — a real 1400-deep tree does exactly this on macOS — or under + # a component that has become unsearchable. The depth reproduction is platform-dependent, so the + # failure is injected at the same call, for the walked entry alone rather than for every `is_dir`. + real_is_dir = Path.is_dir + + def failing_is_dir(self: Path) -> bool: + if self.name == "models.py": + raise OSError(63, "File name too long") + return real_is_dir(self) + + mocker.patch.object(Path, "is_dir", failing_is_dir) + with pytest.raises(CodegenLockError, match="Unreadable entry under the codegen output root"): + run_codegen_check(root=tmp_path) + + def test_an_unreadable_directory_under_the_root_is_a_no_verdict_error(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + _write(tmp_path, "sub/stale.py", _stamped("STALE = 1\n")) + (tmp_path / "sub").chmod(0o000) + try: + # Treating an unlistable directory as an empty one would hide exactly the stale artifact the + # orphan scan exists to find. + with pytest.raises(CodegenLockError, match="Unreadable directory under the codegen output root"): + run_codegen_check(root=tmp_path) + finally: + (tmp_path / "sub").chmod(0o755) + + +class TestParityWithTheReference: + """States where a reader and a writer must answer differently, and where the two readers must agree.""" + + def test_a_locked_artifact_whose_parent_is_a_regular_file_is_missing_not_an_error(self, tmp_path: Path) -> None: + _write(tmp_path, "codegen.lock", _lock(("nested/models.py", "A = 1\n"))) + _write(tmp_path, "nested", "I am a file where a directory should be.\n") + report = run_codegen_check(root=tmp_path) + # `pipelex codegen check` reports `missing` here, so this reader must too. Refusing the tree is a + # *writer's* guard — it must not start writing into a blocked path — and the check inherited it by + # sharing the writer's resolver. To a reader the state says only that no file can be at that path. + assert _categories(report) == [("nested/models.py", "missing")] + assert report.drifts[0].detail == "Locked artifact is absent on disk." + + def test_a_symbolic_link_component_is_still_refused_on_the_read_path(self, tmp_path: Path) -> None: + _write(tmp_path, "codegen.lock", _lock(("nested/models.py", "A = 1\n"))) + (tmp_path / "elsewhere").mkdir() + (tmp_path / "nested").symlink_to(tmp_path / "elsewhere") + # Relaxing the directory guard must not relax containment: a link on the way to an artifact still + # routes the read out of the tree, and is still refused. + with pytest.raises(CodegenLockError, match="symbolic link component is not allowed"): + run_codegen_check(root=tmp_path) + + def test_a_python_artifact_declaring_a_source_encoding_is_hand_edited(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + # Every header line still starts with `#`, and the body below the fence is untouched, so both the + # stamp hash and the locked hash still agree. But PEP 263 lets line 2 choose the codec CPython + # decodes the file with, and under `raw_unicode_escape` a header value carrying the six literal + # characters of a backslash-u-000a escape becomes a newline — so the text after it is a statement + # that runs on import, in a region no hash covers. Without the gate this tree reads as current + # while executing injected code. + injected = "# note: " + chr(92) + "u000aINJECTED = True" + stamped = ( + _stamped("A = 1\n") + .replace( + "# >>> pipelex-codegen-stamp >>>\n", + "# >>> pipelex-codegen-stamp >>>\n# coding: raw_unicode_escape\n", + ) + .replace("# options: {}\n", f"# options: {{}}\n{injected}\n") + ) + _write(tmp_path, "models.py", stamped) + assert _categories(run_codegen_check(root=tmp_path)) == [("models.py", "hand-edited")] + + def test_a_coding_declaration_below_pep_263s_two_line_window_is_not_a_drift(self, tmp_path: Path) -> None: + _tree(tmp_path, ("models.py", "A = 1\n")) + # CPython reads the declaration on the first two lines and no further, so one below them changes + # nothing about how the file decodes. Reporting it would be a drift the state does not justify. + _write(tmp_path, "models.py", _stamped("A = 1\n").replace("# options: {}\n", "# options: {}\n# coding: utf-8\n")) + assert run_codegen_check(root=tmp_path).is_current + + def test_every_no_verdict_condition_is_catchable_as_a_codegen_error(self, tmp_path: Path) -> None: + _write(tmp_path, "codegen.lock", "lock_version = = 1\n") + with pytest.raises(CodegenError): + run_codegen_check(root=tmp_path) diff --git a/tests/unit/test_codegen_lock.py b/tests/unit/test_codegen_lock.py index 66df659..90d7def 100644 --- a/tests/unit/test_codegen_lock.py +++ b/tests/unit/test_codegen_lock.py @@ -78,5 +78,22 @@ def test_load_lock_names_the_path_of_a_malformed_lock(self, tmp_path: Path) -> N with pytest.raises(CodegenLockError, match="Unreadable codegen lock at"): load_lock(lock_path) + @pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"]) + def test_load_lock_reads_every_line_ending_dialect(self, tmp_path: Path, line_ending: str) -> None: + lock_path = tmp_path / "codegen.lock" + lock_path.write_bytes(_PIPELEX_LOCK.replace("\n", line_ending).encode("utf-8")) + + # The reader is deliberately in text mode, so universal-newline translation folds all three into LF + # before `tomllib` sees them. A lone CR is the case that distinguishes the two modes: `tomllib` + # already accepts CRLF, but reading the bytes raw made a lone-CR lock a `CodegenLockError` while + # `pipelex`'s reader parsed it. That divergence is what text mode closes — and it is not only the + # check's concern, because a previous lock the writer cannot parse silently switches pruning off, + # leaving a delisted artifact behind as exactly the stale file the lock exists to catch. + lock = load_lock(lock_path) + + assert lock is not None + assert lock.paths() == {"models.py", "nested/extra.ts"} + assert lock.hash_by_path() == {"models.py": "111", "nested/extra.ts": "222"} + def test_validate_artifact_path_returns_the_relative_filesystem_form(self) -> None: assert validate_artifact_path("nested/deeper/models.ts") == Path("nested", "deeper", "models.ts") diff --git a/tests/unit/test_codegen_stamp.py b/tests/unit/test_codegen_stamp.py index 11acb02..2f763cd 100644 --- a/tests/unit/test_codegen_stamp.py +++ b/tests/unit/test_codegen_stamp.py @@ -1,12 +1,37 @@ -"""The stamp facts a tree writer depends on: which file types are stamped, in which comment syntax, and the -begin-line predicate that decides whether a file on disk belongs to codegen (mirroring `pipelex/codegen/stamp.py`). +"""The stamp grammar both codegen readers depend on, mirroring `pipelex/codegen/stamp.py`: which file types are +stamped, in which comment syntax, the begin-line predicate that decides whether a file on disk belongs to +codegen, and the parser plus content hash that turn one stamped file into a drift verdict. """ +import hashlib + import pytest -from pipelex_sdk.codegen_stamp import STAMPABLE_SUFFIXES, comment_prefix_for, has_stamp +from pipelex_sdk.codegen_stamp import ( + STAMPABLE_SUFFIXES, + comment_prefix_for, + compute_content_hash, + has_stamp, + is_stampable_artifact_path, + parse_stamped, +) from pipelex_sdk.errors import CodegenError +_BODY = "from pydantic import BaseModel\n" + + +def _stamped(*, comment_prefix: str = "#", body: str = _BODY, content_hash: str | None = None) -> str: + return ( + f"{comment_prefix} >>> pipelex-codegen-stamp >>>\n" + f"{comment_prefix} crate_fingerprint: {'f' * 64}\n" + f"{comment_prefix} engine_version: 0.57.0\n" + f"{comment_prefix} projection: types / python-pydantic\n" + f"{comment_prefix} options: {{}}\n" + f"{comment_prefix} content_hash: {content_hash if content_hash is not None else compute_content_hash(body)}\n" + f"{comment_prefix} <<< pipelex-codegen-stamp <<<\n" + f"{body}" + ) + class TestCodegenStamp: def test_the_stampable_suffixes_are_python_and_typescript(self) -> None: @@ -32,3 +57,92 @@ def test_an_unstampable_file_type_has_no_comment_prefix(self) -> None: ) def test_has_stamp_reads_only_the_opening_line(self, content: str, comment_prefix: str, expected: bool) -> None: assert has_stamp(content, comment_prefix=comment_prefix) is expected + + @pytest.mark.parametrize( + ("artifact_path", "expected"), + [ + ("models.py", True), + ("nested/deep/schemas.ts", True), + ("sources.json", False), + ("codegen.lock", False), + ("README", False), + (".py", False), + ("archive.tar.py", True), + ], + ) + def test_the_stampable_predicate_follows_the_suffix(self, artifact_path: str, expected: bool) -> None: + # Public so a caller's tree walk filters exactly as the check does; a suffixless name and a dotfile + # both have no suffix, so neither can ever be an artifact. + assert is_stampable_artifact_path(artifact_path) is expected + + def test_the_content_hash_is_lowercase_sha256_hex_over_utf8_bytes(self) -> None: + body = "X = 'é'\n" + assert compute_content_hash(body) == hashlib.sha256(body.encode("utf-8")).hexdigest() + + def test_parse_stamped_returns_the_recorded_hash_and_the_body_byte_exactly(self) -> None: + parsed = parse_stamped(_stamped(), comment_prefix="#") + assert parsed is not None + assert parsed.body == _BODY + assert parsed.content_hash == compute_content_hash(_BODY) + + def test_parse_stamped_does_not_verify_the_hash_it_reports(self) -> None: + parsed = parse_stamped(_stamped(content_hash="0" * 64), comment_prefix="#") + # Splitting and verifying are separate steps: the caller compares, which is what lets the check tell + # a self-inconsistent stamp apart from a body that merely drifted off the lock. + assert parsed is not None + assert parsed.content_hash == "0" * 64 + + def test_parse_stamped_keeps_an_empty_body_empty(self) -> None: + parsed = parse_stamped(_stamped(body=""), comment_prefix="#") + assert parsed is not None + assert parsed.body == "" + + def test_parse_stamped_reads_the_typescript_fence(self) -> None: + parsed = parse_stamped(_stamped(comment_prefix="//", body="export const A = 1;\n"), comment_prefix="//") + assert parsed is not None + assert parsed.body == "export const A = 1;\n" + + def test_parse_stamped_refuses_a_fence_in_the_wrong_comment_syntax(self) -> None: + assert parse_stamped(_stamped(comment_prefix="//"), comment_prefix="#") is None + + @pytest.mark.parametrize( + "content", + [ + "handwritten = True\n", + "\n# >>> pipelex-codegen-stamp >>>\n# <<< pipelex-codegen-stamp <<<\nbody\n", + "# >>> pipelex-codegen-stamp >>>\n# projection: types / python-pydantic\nbody\n", + ], + ) + def test_parse_stamped_refuses_a_missing_or_unterminated_fence(self, content: str) -> None: + assert parse_stamped(content, comment_prefix="#") is None + + @pytest.mark.parametrize("declaration", ["# coding: raw_unicode_escape", "# -*- coding: unicode_escape -*-", "#coding=utf-8"]) + def test_parse_stamped_refuses_a_python_artifact_that_declares_a_source_encoding(self, declaration: str) -> None: + stamped = _stamped().replace( + "# >>> pipelex-codegen-stamp >>>\n", + f"# >>> pipelex-codegen-stamp >>>\n{declaration}\n", + ) + # The prefix gate proves every header line opens with a comment marker in the bytes on disk. A PEP 263 + # declaration makes that proof worthless, because CPython decodes the file before tokenizing it and the + # declaration chooses the codec: under an escape-decoding codec a header value can carry characters that + # become a line break plus a statement, in the one region no hash covers. All three spellings here are + # ones CPython honours, and the emitter writes none of them. + assert parse_stamped(stamped, comment_prefix="#") is None + + def test_parse_stamped_accepts_a_coding_declaration_below_the_first_two_lines(self) -> None: + stamped = _stamped().replace("# options: {}\n", "# options: {}\n# coding: raw_unicode_escape\n") + # CPython honours the declaration on the first two lines only, so one below them decodes nothing + # differently. Refusing it would report a drift the state does not justify. + assert parse_stamped(stamped, comment_prefix="#") is not None + + def test_parse_stamped_does_not_apply_the_python_encoding_rule_to_typescript(self) -> None: + body = "export const A = 1;\n" + stamped = _stamped(comment_prefix="//", body=body).replace( + "// >>> pipelex-codegen-stamp >>>\n", + "// >>> pipelex-codegen-stamp >>>\n// coding: raw_unicode_escape\n", + ) + # TypeScript has no source-encoding declaration, so the line is an ordinary header field there. Every + # line terminator ECMAScript honours is one `splitlines` already breaks on, which is the `.ts` half. + parsed = parse_stamped(stamped, comment_prefix="//") + assert parsed is not None + assert parsed.body == body From f3401ee5e5b876bbc4b076eb08d98d95ee0c4fbd Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 02:30:09 +0200 Subject: [PATCH 9/9] Release v0.10.0 Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 +- pyproject.toml | 2 +- uv.lock | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b3b2639..ac62b17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [v0.10.0] - 2026-09-13 ### Added diff --git a/pyproject.toml b/pyproject.toml index 220359a..9e6fc3b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "pipelex-sdk" -version = "0.9.0" +version = "0.10.0" description = "The Python client for the Pipelex hosted API — the MTHDS Protocol surface plus the durable run lifecycle and the Pipelex product surface, built on the `mthds` protocol base." authors = [{ name = "Evotis S.A.S.", email = "oss@pipelex.com" }] maintainers = [{ name = "Pipelex staff", email = "oss@pipelex.com" }] diff --git a/uv.lock b/uv.lock index 2a51741..d0a1725 100644 --- a/uv.lock +++ b/uv.lock @@ -303,7 +303,7 @@ wheels = [ [[package]] name = "pipelex-sdk" -version = "0.9.0" +version = "0.10.0" source = { editable = "." } dependencies = [ { name = "httpx" },