Skip to content

fix: template chat and tasks, honest prompt optimization, readable reports, working Studio - #7

Merged
piotrlaczkowski merged 16 commits into
mainfrom
pp/fervent-faraday-4n9ii2
Sep 30, 2026
Merged

fix: template chat and tasks, honest prompt optimization, readable reports, working Studio#7
piotrlaczkowski merged 16 commits into
mainfrom
pp/fervent-faraday-4n9ii2

Conversation

@piotrlaczkowski

@piotrlaczkowski piotrlaczkowski commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why

  • A project scaffolded with agentomatic init NAME --template basic|full returned 400 when you chatted with it or ran it as a task, and Studio couldn't start a new thread.
  • Prompt optimization had several connected problems:
    • OptimizeMetricAdapter had no evaluate.
    • Data augmentation added nothing.
    • Nothing stopped a prompt from overfitting the validation set.
    • The training report showed numbers but no prompts.
  • The example examples/prompt_optimization/train.py didn't run.
  • Several Studio views were broken with template agents.

What changed

Threads, tasks and templates

  • A fresh project now keeps chat threads in a bounded in-memory store. AGENTOMATIC_EPHEMERAL_THREADS=0 turns this off. A configured but broken DB still fails loudly rather than falling back.
  • Studio runs now load the thread's history and save each turn.
  • Synchronous nodes run in a worker thread, so they no longer block the event loop.
  • The basic/full templates now send the conversation history to the model.

Metrics

  • Both metric protocols now work in every slot:
    • class-agent metrics use score(example, prediction);
    • optimize metrics use async evaluate(...);
    • as_optimize_metric / as_agent_metric convert between them.
  • Judges, composite metrics and multi-judge panels report failures as failures instead of inventing a score.
  • Judge scores on other scales are normalised to 0–1.
  • A metric that reads example metadata now scores the same inside a composite as it does in fit().

Anti-overfitting

  • The optimizer learns from train and selects on validation. A win must pass a paired-bootstrap significance test.
  • The holdout split can only veto a candidate: for a regression, a widened gap, or a gain that doesn't carry over.
  • The test split is never passed to the fitter.
  • Every candidate records its decision and the reason.

Augmentation

  • It now actually adds rows, seeded from individual examples.
  • Label-preserving strategies keep the seed's answer and metadata.
  • Only the train split grows, and near-copies of eval questions are dropped.
  • Truncated model replies are salvaged; augment_stats records what happened.

Reports

  • HolySheet's Accordion never rendered, so prompts were hidden; everything is now shown inline.
  • With several epochs the report compared the final prompt with itself. Epochs are now merged, from the original prompt to the final one.
  • New sections:
    • verdict;
    • test scores before and after;
    • the initial-to-final prompt diff;
    • every candidate with its reason;
    • per-example answers before and after.
  • A JSON copy of the report is written next to the HTML.
  • agent.fit() now exposes History.fit_results.

Examples and docs

  • examples/prompt_optimization has a working train.py plus four new scripts, all commented step by step, with an end-to-end test for each.
  • A new top-level Optimization docs section covers mechanics, metrics and judges, generalization, data, the high- and low-level APIs, reports and the examples.

Studio (server side)

  • The Debug view now shows state and history for class agents.
  • agents/<name>/pipeline.yaml is discovered.
  • A failed pipeline, or an endpoint whose upstreams all failed, now marks its task failed; a partial pipeline still succeeds.
  • A condition that can't be evaluated respects the pipeline's on_error: continue.
  • The pipeline template is valid.
  • /feedback/export keeps the query, response and correction. The SQL columns are added automatically.
  • Connections are looked up by the registered agent name, not the URL slug.
  • The Studio UI is updated to 0.4.1, the same bundle as the published studio-ui-0.4.1.tar.gz built from UnicoLab/agentomatic-studio#1.

CI and dependencies

  • uv.lock upgrades anyio, PyJWT, urllib3 and virtualenv past known advisories, so pip-audit (the CI audit job and the release gate) passes. PyJWT 2.14+ fetches JWKS through its own opener, so the JWT end-to-end test now stubs OpenerDirector.open.
  • pre-commit no longer runs its fixers on the synced Studio bundle.
  • sync-studio.yml no longer fails when the automated / studio-ui labels are missing. It had pushed the v0.4.0 and v0.4.1 branches but never opened their PRs.

Verification

Run locally:

  • ruff check and format;
  • mypy;
  • mkdocs build --strict;
  • pip-audit;
  • uv build, twine check and scripts/verify_wheel_studio.py;
  • the full pytest suite: 2946 passed. One test fails only as root: test_save_read_only_storage_returns_actionable_conflict, because root can write to a read-only directory.

The Studio views were checked in Chromium against a freshly scaffolded project.

The examples and optimization tests run against a deterministic fake OpenAI-compatible server (tests/fake_openai_server.py). They haven't been run against a real model.

🤖 Generated with Claude Code

https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX

Generated by Claude Code

claude added 16 commits September 30, 2026 18:14
@claude
fix(studio,tasks): make template agents chat, thread and run as tasks…
8cc6ac0 
… out of the box

A freshly scaffolded project (`agentomatic new` + `init --template basic|full`)
could not chat from Studio:

- No store is configured by default, so every thread route answered
  `400 Thread storage not configured`. Studio's "New Chat" creates a thread
  before the first message, so no conversation could ever start. The platform
  now falls back to a bounded in-memory store (1000 threads, LRU-evicted) when
  no store / MEMORY connection / database URL is configured
  (`AGENTOMATIC_EPHEMERAL_THREADS=0` disables it). A configured-but-broken
  database still refuses the fallback.
- Studio runs never loaded thread history nor saved the turn, so a chatbot
  answered every Studio message as if it were the first and reopened threads
  were empty. Studio runs now use the same memory handling as `/chat`.
- The basic/full templates sent the model one string and ignored
  `state.messages`; they now send a real system message plus the turns.
- Sync graph nodes (every template's blocking `llm.invoke`) ran inline on the
  event loop, so an async task submission only returned once the model had
  answered and polling/SSE/health checks stalled meanwhile. Sync nodes now run
  in a worker thread; `report_stage_sync` forwards progress back to the loop.
- The per-request prompt override was an instance attribute, so concurrent
  runs (and concurrently scored optimizer candidates) could read or clear each
  other's prompt. It is now context-local.
- `build_invoke_state` no longer mutates the caller's payload (a task's stored
  `input.metadata` was rewritten by the run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
fix(optimize): accept every metric protocol everywhere; hold the test…
868755f 
… split out of selection

- `PromptFitterBridge(metric=OptimizeMetricAdapter(judge))` crashed mid-fit with
  "'OptimizeMetricAdapter' object has no attribute 'evaluate'" — logged as a
  warning, so fit() silently continued without optimizing. The two metric
  protocols (sync `score(example, prediction)` / async `evaluate(query,
  response, expected, context)`) are now bridged in both directions:
  `as_optimize_metric` / `ScoreMetricAdapter` and `as_agent_metric`, applied
  by compile/evaluate, MetricLoss/resolve_loss, agents.WeightedMetric,
  optimize.WeightedMetric/CompositeMetric, resolve_metrics, PromptFitterBridge
  (fails fast at construction) and PromptFitter.fit.
- Class-agent metrics scored inside the fitter now see the example's metadata
  and the agent's structured output (scoring context), so a deterministic
  metric scores the same during selection as in agent.evaluate().
- OptimizeMetricAdapter no longer fabricates a neutral 0.5 when the judge
  fails: it scores 0.0, logs, and counts `failures`.
- PromptFitterBridge folded the `test` split into the pool candidates are
  selected on; test is now passed to the fitter as its held-out set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
feat(optimize): an anti-overfitting mechanism for prompt optimization
a39b603 
Prompt optimization could accept prompts that only fit the data they were
selected on. It now separates data roles and gates acceptance:

- The optimizer learns from TRAIN only: failure clustering, judge feedback,
  few-shot demos and expected-tips come from a train reflection sample
  (`reflect_on="train"`, `reflection_size`), never from the validation
  examples candidates are selected on (their gold answers used to be pasted
  into the rewrite briefing). A holdout carved from train leaves the pool.
- Candidates are SELECTED on validation, and only with a paired-bootstrap
  confidence that they beat the incumbent (`min_confidence=0.8`) — judge noise
  no longer passes as an improvement.
- A held-out slice can only VETO, measured against the incumbent: no
  regression (`holdout_tolerance`), no widened validation/held-out gap
  (`max_generalization_gap`), and a validation gain must transfer
  (`min_transfer_ratio=0.25`) — the signature of memorised answers.
- `test` is never passed to the fitter; an optional `split="holdout"` is the
  gate (else a slice of validation). AgentDataset gains `.holdout` and warns
  about unknown split labels.
- Failed points (agent errors, judge failures) count as 0.0 instead of being
  dropped, so a candidate that breaks on hard inputs cannot win.
- `max_trials` is a budget of distinct candidates (duplicates of an already
  scored config are skipped); rounds round up; promotion compares against the
  incumbent on the same minibatch; ROUND_END fires on every round and a stop
  requested in one fit no longer ends the next.
- Every trial records its prompt, decision and reason.
- Rewrites are no longer corrupted: text replies keep fenced blocks (only the
  first fence used to be returned) and `extract_prompt_text` splits on a
  `---` line only.
- Concurrent candidate evaluation no longer swaps shared agent attributes for
  BaseGraphAgent (the race leaked prompts/temperature).
- PromptFitterBridge selects on the compiled loss/metric by default, keeps
  every epoch's result (`agent._fit_results`), applies a config only when it
  improved, bakes few-shot demos into the served prompt, and routes judge
  calls to `llm_base_url` from construction (the epoch-0 baseline hit the
  default endpoint). The expected-tips candidate no longer injects another
  project's domain text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
fix(optimize): judge scores and metric aggregation mean what they say
950baa3 
- Judge replies are normalised instead of clamped: a 0–10 or 0–100 score
  (`7`, `85`, "8/10", "80%") used to read as a perfect 1.0. LocalJudgeMetric
  accepts `score`/`overall`/`rating` as well as `overall_score`, falls back to
  the mean of the dimension scores, parses each dimension on its own (one
  nested dimension used to discard a valid overall score), keeps string
  fields whole, and flags failure explicitly rather than by feedback text.
  LLMJudgeMetric/GEvalMetric accept `overall_score`; judges take their own
  `base_url`/`api_key`.
- CompositeMetric counts a failed component as 0 at its full weight (a failed
  judge used to *raise* the composite) and reports failure when half the
  weight failed; negative weights are rejected (every metric is
  higher-is-better); its sync `score()` shares the one adapter.
- MultiJudgePanel keeps each weight with its own judge when another fails,
  guards a zero total, implements median/min/max (unknown names raise), and
  exposes a name and dimensions.
- DeepEval bias/toxicity/hallucination are inverted to higher-is-better
  (maximising them favoured toxic answers); a missing DeepEval score is a
  failed evaluation. LatencyMetric reads the measured call duration; a
  CustomMetric returning NaN is a failure, other values are clamped.
- local_judges no longer mutate the caller's CompositeMetric each epoch.
- agents.EarlyStopping follows Keras patience, implements restore_best and
  infers mode from the `loss` suffix; fit() rejects optimize-side callbacks
  with a clear message; fit_agent monitors `loss` when there is no validation
  split. Crashed examples no longer count as passed. Metric names must be
  unique. evaluate() reports metric failures. One judge call per example no
  matter how many wrappers share the judge.
- build_default_metrics selects candidates on the same blend the loss
  reports (the fit metric had no judge). run_sync reuses one background loop
  inside a running loop instead of leaking a new loop per call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
fix(optimize): data augmentation actually adds usable, leak-free rows
7a46ae7 
`prepare_dataset(augment=True)` often added nothing, and what it added was
mislabelled:

- One LLM call per strategy asked for dozens of examples under
  max_tokens=2000; the reply was cut off and the parser (clean JSON array
  only) returned zero rows without a warning. Generation is now seed-anchored:
  one seed + one strategy per call, a few variations each, looping until the
  target is met (bounded call budget). Replies are parsed whatever the
  wrapping — fences, prose, `{"examples": [...]}`, a single object, JSONL —
  and complete objects are salvaged from a truncated reply. Truncation
  (`finish_reason=length`) is logged.
- Synthetic rows mirror their seed: the seed's input is copied with only the
  question replaced; label-preserving strategies (paraphrase, perturbation,
  …) keep the seed's expected_output and metadata (`must_include` etc.);
  others get the generated answer in the seed's schema or are dropped. The
  fabricated `{"content", "next_action": "Follow up with stakeholder."}`
  label is gone.
- Only train rows seed augmentation and only train grows; generated
  questions that duplicate — or nearly duplicate — a validation, holdout or
  test question are discarded.
- The endpoint is passed per call (no process-wide LLMCaller.configure);
  `dataset.metadata["augment_stats"]` reports requested/calls/parsed/
  duplicates/near-duplicates/added per strategy, a shortfall is warned
  (`strict=True` raises), and `augmented` reflects rows actually added.
- Persist-only never overwrites the seed file; `prefer_augmented` also finds
  `all_augmented.jsonl`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
feat(optimize): fit reports show what changed and whether it helped
735f9c5 
The training report had numbers but no prompts. Causes and fixes:

* Every prompt panel was a HolySheet Accordion, which the bundled viewer
  never renders, so prompts, few-shot demos and per-example details were
  hidden. Panels are now flattened into always-visible blocks.
* With epochs > 1 only the last epoch's PromptFitResult was reported. Its
  baseline was a prompt an earlier epoch had already improved, so the
  report compared the final prompt with itself and said "no improvement".
  merge_fit_results() spans the original prompt to the final one.
  agent.fit() now returns History.fit_results (one result per epoch), and
  generate_fit_report() accepts that History, a list of results, or the
  agent.
* New sections at the top of the report:
  - verdict: validation, held-out gate and test, before and after;
  - test scoreboard;
  - initial-to-final prompt diff, with every accepted change in order;
  - every candidate with its decision and reason;
  - per-example answers and scores, before and after;
  - data, augmentation and fitter settings.
  A JSON sidecar carries the same data.
* Diffs wrap long single-line prompts, so they show the sentences that
  changed.
* The eval report table lists whatever metrics were compiled, not a
  fixed judge/f1/keywords set.
* The deployment rollout rendered as "—" because it read __dict__ on a
  slots dataclass.
* run_train (train_and_report) now scores the held-out split before and
  after fit (TrainConfig.evaluate_baseline). It applies and reports the
  merged result.
* Augmentation rotates strategies call by call, so a short run still
  mixes them.
* Fixed three mypy errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
feat(examples): working prompt-optimization examples, low to high level
7d04c78 
examples/prompt_optimization now holds a self-contained SupportAgent, a
26-row dataset with train/validation/holdout/test splits, and five
scripts. Every script is commented step by step and lists each method's
options, and each runs end to end in tests against a deterministic fake
OpenAI-compatible server:

* metrics_and_judges.py: every metric family, LLM judges
  (single, per-dimension, panel), composite objectives, adapters and
  losses, scored on three fixed answers.
* data_augmentation.py: prepare_dataset strategies and options, leak
  protection, augment_stats.
* anti_overfitting.py: the acceptance rules on hand-made numbers, then
  a fit where a prompt memorizing the validation answers is vetoed and
  a general one is kept.
* low_level_prompt_fitter.py: PromptFitter with every knob, a
  before/after test evaluation, and a report.
* train.py: the Keras-style compile/fit loop (the script from the
  report), now runnable with augmentation and a meaningful report.

Library fixes found while doing this:

* A metadata-reading class-agent metric inside a CompositeMetric scored
  0 in agent.evaluate() and compile(metrics=...), but correctly inside
  the fitter. OptimizeMetricAdapter now exposes the example as the
  current scoring run.
* Exported paired_improvement_confidence, GeneralizationCheck,
  EvalResult and coerce_judge_score.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
docs(optimization): a dedicated section covering the whole mechanism
dcc74af 
Prompt optimization was documented on one page that mixed the CLI, two
Python APIs and legacy paths. It did not say which metrics to use, how
metrics, the objective and the loss relate, or how overfitting is
prevented. It now has its own top-level section:

* Overview: entry points and a complete run.
* How it works: the fit loop step by step, how candidates reach the
  agent, the optimizers, and epochs.
* Metrics, judges & loss: the three slots, both metric families,
  every metric with its options, LLM-as-judge and panels, composite
  objectives, losses, and how to choose.
* Generalization & overfitting: data roles, each acceptance rule with
  its knob and default, worked examples, and practical guidance.
* Datasets & augmentation: the JSONL fields, every prepare_dataset
  option and strategy, and the leak protections.
* Class agents: compile, fit, evaluate, PromptFitterBridge,
  callbacks, History, and train_and_report flags.
* Low level: every PromptFitter option, the search space, callbacks,
  and PromptFitterResult.
* Reports: what each section shows and how to read it.
* CLI & legacy API: moved from the old page; --test-dataset is the
  holdout gate.
* Examples: mirrors examples/prompt_optimization.

The old page links to the new pages. The agent primer and the template
train.py staged block now use the before/after report API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
fix(studio): chat, debug, tasks, pipelines and feedback work with tem…
8b45d7d 
…plate agents

A browser run of every Studio view against a freshly scaffolded project
found these defects. Studio bundle (prebuilt; patched and re-hashed, since
assets are served immutable):

* Chat never sent the typed message: a string input was dropped from the
  run request unless the input object had a `query`, so every agent
  answered an empty question.
* Integer message ids from the SQL and memory stores crashed the chat
  view (`id.includes`); ids are now normalised to strings.
* The Task Board could not load a pipeline's input form: the contract
  lookup only matched literal OpenAPI paths, not
  `/api/v1/pipelines/{name}/run`.
* From 1536 to about 1800 px, the agent picker overlapped the header
  navigation. The nav now shrinks and scrolls.

Server:

* The Debug view's state and time-travel tabs were empty for class
  agents. The graph-agent adapter now records the state after every node
  as a checkpoint, persisted when a store is configured.
* `agents/<name>/pipeline.yaml` was never discovered. The extraction
  template ships one. The loader scanned the agents folder as a project
  root.
* A failed pipeline, or an endpoint whose upstreams all failed, showed as
  a succeeded task. Tasks now fail with TargetFailedError and keep the
  target's report as the result. A partial pipeline still succeeds.
* A condition that could not be evaluated ignored the pipeline's
  `on_error: continue`.
* The pipeline template used `input:`/`output:` (ignored keys) and an
  invalid `ctx.steps.…` condition that always raised.
* `/feedback/export` was empty whenever a store was configured, which is
  now the default: stores dropped the query, response and correction.
  SQL gains the columns additively.
* The connections middleware keyed scopes by URL slug instead of the
  registered agent name.

Also updates the template train.py contract test for the before/after
report block introduced in the docs commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
chore: sync uv.lock with the 1.14.0 project version
9b88a71 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
chore(studio): ship the Studio UI 0.4.1 build from source
b5b60d1 
Replaces the hand-patched bundle with a build of agentomatic-studio 0.4.1
(`VITE_BASE_URL=/studio/ui/ npm run build`). That release carries the
same fixes in source:

* a typed chat message is sent as `query`;
* integer message ids are normalised to strings;
* pipeline input forms resolve through templated routes;
* the header tab bar no longer runs under the agent picker.

The bundle test now checks the source implementation of the
templated-route lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
chore(studio): rebuild the Studio UI 0.4.1 bundle with patched depend…
1428e2d 
…encies

Matches agentomatic-studio 0.4.1 after its lockfile audit fix (updated
js-yaml and browserslist change the entry chunk and CSS hashes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
ci: keep pre-commit fixers off the synced Studio bundle
54ce4a0 
The PR gate runs pre-commit on changed files. `trailing-whitespace`
rewrote minified Studio chunks, which failed the gate. It would also have
made the shipped files differ from the agentomatic-studio release tarball
that sync-studio.yml installs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
fix(deps): upgrade locked anyio, PyJWT, urllib3 and virtualenv past a…
20eb8ae 
…dvisories

`pip-audit` (the CI audit job and the release gate) failed on the locked
set with 21 known vulnerabilities:

* anyio 4.13.0 → 4.14.2 / 4.15.1
* PyJWT 2.13.0 → 2.15.1
* urllib3 2.7.0 → 2.8.0
* virtualenv 21.4.3 → 21.14.1 (python-discovery follows)

PyJWT >= 2.14 fetches a JWKS through its own `build_opener(...)`, which
refuses redirects, instead of `urllib.request.urlopen`. The JWT
end-to-end test therefore stubs `OpenerDirector.open`, which both paths
use. Production code is unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
chore: ignore ModelCheckpoint's default output directory
921a97d 
optimization_results/checkpoints/ is where ModelCheckpoint() writes by
default; test runs and local fits leave files there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@claude
ci(sync-studio): do not fail the sync when PR labels are missing
69e1636 
`gh pr create --label automated,studio-ui` exits 1 when a label does not
exist on the repository, so the v0.4.0 and v0.4.1 syncs pushed their
branch but never opened a PR. Create the PR first and add the labels
best-effort.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gEMypXBosTqaEh3QFVXCX
@piotrlaczkowski
piotrlaczkowski merged commit 4eda703 into main Sep 30, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@piotrlaczkowski @claude