diff --git a/.github/workflows/sync-studio.yml b/.github/workflows/sync-studio.yml index e73b626..b18d8a6 100644 --- a/.github/workflows/sync-studio.yml +++ b/.github/workflows/sync-studio.yml @@ -165,8 +165,10 @@ jobs: --- *This PR was automatically created by the [sync-studio workflow](https://github.com/UnicoLab/agentomatic/actions/workflows/sync-studio.yml).*" \ - --label "automated,studio-ui" \ --base main echo "✅ Pull request created successfully" + # Labels are best-effort: a missing label must not fail the sync. + gh pr edit "${BRANCH}" --add-label "automated,studio-ui" \ + || echo "::warning::Could not add labels (create 'automated' and 'studio-ui')" fi diff --git a/.gitignore b/.gitignore index b599411..7ec53c4 100644 --- a/.gitignore +++ b/.gitignore @@ -48,3 +48,6 @@ site/ # Poetry (legacy) poetry.lock inspiration/ +examples/prompt_optimization/out/ +# ModelCheckpoint default output (runtime artefacts, never committed) +optimization_results/checkpoints/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 0346635..c91567d 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,3 +1,8 @@ +# The Studio UI bundle is a build artifact synced from agentomatic-studio +# releases (sync-studio.yml). Whitespace fixers would rewrite its minified +# chunks, so the shipped files would no longer match the release tarball. +exclude: ^src/agentomatic/studio/static/ + repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v5.0.0 diff --git a/CLAUDE.md b/CLAUDE.md index a557184..257582f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,6 +145,7 @@ into the image/compose that drive the same `main.py`. | `AGENTOMATIC_ENABLE_CONTROL_PLANE` / `AGENTOMATIC_CONTROL_TOKEN` | Control plane | | `AGENTOMATIC_ENABLE_RATE_LIMIT` | Rate limiting | | `AGENTOMATIC_LOGS_HISTORY` | Persist per-agent invoke/chat/stream history (default off) | +| `AGENTOMATIC_EPHEMERAL_THREADS` | With no store/DB configured, keep chat threads in a bounded in-memory store (default on; `0` disables — thread routes then answer 400) | | `AGENTOMATIC_ALLOW_LOGSLLM_ANALYSIS` | Enable LLM analysis over those logs (default off) | | `AGENTOMATIC_INGESTION_ROOT` | Confine ingestion source/output paths to this dir (default: cwd) | | `AGENTOMATIC_TASK_EVENT_LOG` | Task-event replay backend: `memory` (default), `none` to disable, or a registered provider | @@ -235,6 +236,12 @@ register_vector_store_adapter("my_db", MyDbStore) # upsert/query/delete `"skipped: "`). - Model defaults resolve provider-agnostically from `AGENTOMATIC_TASK_MODEL` / `LLM__MODEL` before any local fallback. +- Data roles: **train** is what the optimizer learns from, **validation** selects + candidates, **holdout** only vetoes (no transfer / regression / widened gap), + **test** is never passed to the fitter — `agent.evaluate(data.test)` before and + after `fit()`. `generate_fit_report(history, baseline_eval=..., final_eval=...)` + shows the prompt diff, every candidate's decision and reason, and per-example + answers. Guide: `docs/optimization/`; runnable: `examples/prompt_optimization/`. ## Project conventions (match these when editing the framework) diff --git a/README.md b/README.md index 0b88415..e728818 100644 --- a/README.md +++ b/README.md @@ -764,7 +764,7 @@ Failure cluster 2: agentomatic run # 2. Prepare a labelled JSONL evaluation dataset (query + expected_answer). -# See docs/guide/optimization.md for the format and metric guidance. +# See docs/optimization/ for the format, metrics and anti-overfitting guidance. # 3. Optimize a running agent with the current fitter CLI. agentomatic optimize scope_agent \ diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 54422fa..89a2745 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -671,7 +671,7 @@ Now that your first agent is running, explore these resources: | **[Deep Agent Integration](../guide/deep-agents.md)** | Register and debug Deep Agent workflows with full Studio support | | **[Chat Interface](../guide/debug-ui.md)** | Chainlit-based conversational testing | | **[Prompt Management](../guide/prompts.md)** | Template versioning, hot-reload, and A/B testing | -| **[Prompt Optimization](../guide/optimization.md)** | Auto-tune prompts with DSPy-inspired optimization | +| **[Prompt Optimization](../optimization/index.md)** | Auto-tune prompts with DSPy-inspired optimization | | **[Storage Backends](../guide/storage.md)** | Configure PostgreSQL, SQLite, or custom adapters | | **[Middleware](../guide/middleware.md)** | Auth, rate limiting, metrics, and custom middleware | | **[Configuration](../guide/configuration.md)** | Platform settings, CORS, environment variables | diff --git a/docs/guide/class-agents.md b/docs/guide/class-agents.md index 0593db6..c6b0058 100644 --- a/docs/guide/class-agents.md +++ b/docs/guide/class-agents.md @@ -1163,7 +1163,7 @@ agents/my_agent/ Class scaffolds emit flat scripts (full abstraction): `TrainCliSettings` / `EvalCliSettings` → - [`train_and_report`](optimization.md) / [`evaluate_and_report`](optimization.md). + [`train_and_report`](../optimization/class-agents.md#one-call-train_and_report) / [`evaluate_and_report`](../optimization/reports.md#evaluation-only-reports). `train.py` also includes a **commented staged** path (`compile_agent` → `fit_agent` → `evaluate_agent`) for full control — same primitives under the hood. @@ -1205,7 +1205,7 @@ agents/my_agent/ ``` For the current in-process fitter API and explicit search spaces, see - [Prompt Optimization](optimization.md#modern-python-api-promptfitter). + [Low level: PromptFitter](../optimization/prompt-fitter.md). ??? example "Generated `dataset.jsonl`" diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 082d835..47fbc80 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -189,6 +189,7 @@ directory, host, port, and stack values passed to `agentomatic run`. | `AGENTOMATIC_ENABLE_CONTROL_PLANE` / `AGENTOMATIC_CONTROL_TOKEN` | control plane | `true` / empty in generated `main.py`; `false` in CLI fallback | Mount control-plane routes and protect mutations | | `AGENTOMATIC_ENABLE_RATE_LIMIT` / `AGENTOMATIC_RATE_LIMIT_TRUST_PROXY_HEADERS` | rate limiting | `false` / `false` | Enable the limiter; trust forwarded client IPs only behind a trusted proxy | | `AGENTOMATIC_LOGS_HISTORY` / `DATABASE_URL` | invocation history / storage | `false` / empty | Persist logs and threads using an async SQLAlchemy URL | +| `AGENTOMATIC_EPHEMERAL_THREADS` | thread-store fallback | `true` | With no `store=`, MEMORY connection or database URL, keep conversation threads in a bounded in-memory store (1000 threads, lost on restart) so Studio chat and `/chat` history work out of the box. `0` disables it — thread routes then answer `400 Thread storage not configured` | | `AGENTOMATIC_ARTIFACT_ROOT` | `artifact_root` | `.local/artifacts` | Versioned plugin/model artifact bundles | | `AGENTOMATIC_PLUGIN_AUTORELOAD` | `plugin_autoreload` | `false` | Watch the promoted artifact version and reload plugins safely | | `AGENTOMATIC_PLUGIN_AUTORELOAD_INTERVAL` | `plugin_autoreload_interval` | `5` | Seconds between artifact-pointer checks | diff --git a/docs/guide/optimization.md b/docs/guide/optimization.md index 6ccb9cc..7681316 100644 --- a/docs/guide/optimization.md +++ b/docs/guide/optimization.md @@ -1,361 +1,20 @@ # Prompt optimization -Agentomatic evaluates an agent against a labelled dataset, searches for a -better prompt or runtime configuration, validates the candidate on held-out -examples, and only then offers to write a new prompt version. It does not -alter live traffic, route versions, or promote deployments. Treat a generated -version as a candidate: review it, run your deployment verification, then -release it through your normal delivery process. - -![Prompt Optimization Flow](../assets/optimization_flow.png) - -## Choose one entry point - -There are three supported entry points. They share the same dataset format but -serve different integration needs. Pick one for a run; they are alternatives, -not steps to chain together. - -| Use case | Supported entry point | What it talks to | -|---|---|---| -| Optimize a deployed agent from a terminal | `agentomatic optimize` | The running Agentomatic API (`--host`) | -| Optimize an in-process agent with full control over metrics and search space | `PromptFitter` | A local agent instance or the API | -| Keep an existing integration working | `PromptOptimizer` / `--mode prompt_only` | The running Agentomatic API | - -For new work, use the CLI with a fitter mode or use `PromptFitter` directly. -`PromptOptimizer` remains supported for backwards compatibility, but has a -smaller search surface. - -## Before you start - -Install the optimization dependencies and create a dataset with a stable, -measurable expectation for each request: - -```bash -pip install "agentomatic[optimize]" -``` - -```jsonl -{"query":"What is the refund period?","expected_answer":"30 days"} -{"query":"How do I reset my password?","expected_answer":"reset, password"} -``` - -`query` is required. `expected_answer`, `context` (a list of strings), and -`metadata` are optional. Use `exact_match` when the expected answer should -match exactly; use `contains` with comma-separated required phrases when a -response can vary in wording. For generative quality, use a calibrated judge -metric and a held-out test dataset. - -Start the target platform before running the API-backed paths: - -```bash -agentomatic run --port 8000 -``` - -## Recommended: CLI fitter modes - -The CLI has two families. `prompt_only` is the legacy default; every other -mode uses the modern `PromptFitter` path. - -```bash -# Prompt + configuration rewrite against the running agent. -agentomatic optimize support_bot \ - --dataset eval.jsonl \ - --val-dataset validation.jsonl \ - --test-dataset holdout.jsonl \ - --mode rewrite \ - --metrics contains \ - --llm omlx/Qwen3.5-9B-MLX-4bit \ - --rewrite-llm omlx/Qwen3.5-9B-MLX-4bit \ - --host http://127.0.0.1:8000 \ - --max-trials 20 \ - --apply -``` - -`--apply` writes a prompt version only when the result passes the fitter's -improvement and generalization guards. Omit it to inspect the report first. -The command writes reports and trial artifacts under `.optimize/` by default. - -### Mode reference - -| CLI mode | Fitter optimizer | Best for | -|---|---|---| -| `rewrite` | `RewriteOptimizer` | Improving system instructions from failure patterns | -| `gepa_like` | `GEPALikeOptimizer` | Targeted edits once evaluator feedback is useful | -| `mipro_like` | `MIPROLikeOptimizer` | Exploring prompt and few-shot combinations | -| `few_shot` | `FewShotBootstrapOptimizer` | Selecting strong demonstrations from labelled data | -| `param_search` | `ParamSearchOptimizer` | Searching model, RAG, or tool parameter grids | -| `apo` | `APOOptimizer` | Trace-aware critique and edit search | - -`--mode` accepts exactly the values in the first column plus `prompt_only`. -The fitter API also accepts the aliases `mipro`, `gepa`, and -`few_shot_bootstrap`; prefer the CLI spellings above in new scripts. - -### Parameter-only search - -Use a search space to make the allowed changes explicit. This avoids an -optimization run silently tuning parameters you did not intend to change. - -```yaml -# search-space.yaml -optimize_system_prompt: false -optimize_few_shot: false -optimize_model_params: true -search_method: tpe -model_param_space: - temperature: [0.0, 0.2, 0.5] - top_p: [0.8, 0.95] -rag_param_space: - top_k: [3, 5, 8] -``` - -```bash -agentomatic optimize support_bot \ - --dataset eval.jsonl \ - --mode param_search \ - --search-space search-space.yaml \ - --search-method tpe \ - --no-optimize-prompt \ - --param temperature=0.0,0.2,0.5 \ - --max-trials 18 -``` - -`--param` adds or replaces a model parameter grid entry. `--search-method` -accepts `grid`, `random`, or `tpe`. For `apo`, `--node-match` scopes trace -critique to matching graph-node or subagent names; `--n-runners` sets fitter -evaluation concurrency. - -Run `agentomatic optimize --help` for the complete, version-specific CLI -contract. Do not use undocumented commands such as `agentomatic eval`, -`agentomatic route`, or `agentomatic promote`: they are not Agentomatic CLI -commands. +Prompt optimization now has its own section: + +| Topic | Page | +| --- | --- | +| Overview and choosing an entry point | [Optimization overview](../optimization/index.md) | +| The fit loop step by step | [How it works](../optimization/how-it-works.md) | +| Metrics, LLM judges, composite objectives, losses | [Metrics, judges & loss](../optimization/metrics.md) | +| Data roles and the anti-overfitting rules | [Generalization & overfitting](../optimization/generalization.md) | +| Dataset format and augmentation | [Datasets & augmentation](../optimization/data.md) | +| `compile` / `fit` / `evaluate`, `train_and_report` | [Class agents](../optimization/class-agents.md) | +| `PromptFitter` and every option | [Low level: PromptFitter](../optimization/prompt-fitter.md) | +| Reading the before/after report | [Reports](../optimization/reports.md) | +| `agentomatic optimize` and `PromptOptimizer` | [CLI & legacy API](../optimization/cli.md) | +| Runnable scripts | [Examples](../optimization/examples.md) | ## Modern Python API: `PromptFitter` -Use `PromptFitter` when your application already owns the agent instance or -needs custom metrics, callbacks, and search spaces. The example below uses a -local agent and an oMLX-compatible endpoint for all optimizer model calls. - -```python -from agentomatic.optimize import ( - Dataset, - ExactMatchMetric, - PromptFitter, - PromptSearchSpace, -) - -# Import and construct your project's BaseGraphAgent subclass. -from agents.support_bot.agent import SupportBotAgent - -dataset = Dataset.from_list( - [ - {"query": "What is the refund period?", "expected_answer": "30 days"}, - {"query": "How do I reset my password?", "expected_answer": "reset password"}, - {"query": "Where is my order?", "expected_answer": "tracking"}, - {"query": "Can I change an address?", "expected_answer": "before shipping"}, - ] -) -trainset, valset = dataset.split(ratio=0.75) - -search_space = PromptSearchSpace( - optimize_system_prompt=True, - optimize_few_shot=False, - optimize_model_params=True, - model_param_space={"temperature": [0.0, 0.2]}, -) - -fitter = PromptFitter( - agent="support_bot", - local_agent=SupportBotAgent(), - task_model="omlx/Qwen3.5-9B-MLX-4bit", - rewrite_model="omlx/Qwen3.5-9B-MLX-4bit", - llm_base_url="http://127.0.0.1:8000/v1", - llm_api_key="local", - optimizer="rewrite", - search_space=search_space, - max_trials=12, -) - -result = await fitter.fit(trainset, valset, ExactMatchMetric()) -print(result.summary()) - -# Writes prompts.json only when the acceptance and generalization checks pass. -result.apply(version="v2_fit", agent_dir="agents/support_bot") -``` - -Use a realistic validation set. If you do not supply `testset` to `fit`, the -fitter reserves a holdout slice automatically when possible. Pass a separate -`testset` when you have one: - -```python -result = await fitter.fit(trainset, valset, ExactMatchMetric(), testset=holdout) -``` - -### Metrics - -Metrics are async and are evaluated for each candidate. The core options are: - -```python -from agentomatic.optimize import ( - CompositeMetric, - ContainsMetric, - ExactMatchMetric, - LLMJudgeMetric, -) - -deterministic = CompositeMetric([ExactMatchMetric(), ContainsMetric()]) -judge = LLMJudgeMetric( - criteria="The answer must be accurate, concise, and safe.", - model="omlx/Qwen3.5-9B-MLX-4bit", -) -``` - -Start with deterministic metrics where a response contract permits them. -Judge metrics are useful for open-ended quality, but require a reachable model -and a rubric that is stable enough to compare candidates fairly. - -### Result and apply safeguards - -`PromptFitResult` records the baseline and best scores, trial history, -suggested parameter changes, failure clusters, and any holdout score: - -```python -print(result.best_prompt) -print(result.best_params) -print(result.absolute_improvement) -print(result.holdout_score) -print(result.generalization_gap) -print(result.to_dict()) -``` - -`result.apply(...)` refuses a non-improving candidate by default and can reject -a candidate with an excessive generalization gap. Review the report and use -`force=True` only when an operator has deliberately accepted that risk. - -## Compatibility API: `PromptOptimizer` - -`PromptOptimizer` and CLI `--mode prompt_only` remain available for existing -API-backed integrations. It supports three legacy strategies only: -`iterative_rewrite`, `few_shot`, and `chain_of_thought`. - -```python -from agentomatic.optimize import Dataset, PromptOptimizer - -optimizer = PromptOptimizer( - agent="support_bot", - metrics=["contains"], - strategy="iterative_rewrite", - llm="omlx/Qwen3.5-9B-MLX-4bit", - api_base="http://127.0.0.1:8000", -) - -dataset = Dataset.from_jsonl("eval.jsonl") -result = await optimizer.optimize( - dataset=dataset, - max_iterations=10, - target_score=0.9, -) -print(result.report()) -``` - -Equivalent CLI: - -```bash -agentomatic optimize support_bot \ - --dataset eval.jsonl \ - --mode prompt_only \ - --strategy iterative_rewrite \ - --host http://127.0.0.1:8000 -``` - -Do not use old names such as `mipro`, `ensemble`, or -`bootstrap_randomsearch` with `--strategy`; they are not legacy CLI strategy -values. Use the fitter modes above instead. - -## Advanced extensions - -These are extensions to `PromptFitter`, not additional quick-start paths. - -### Callbacks and reports - -Use callbacks when a long-running fit needs an operator-visible stopping or -checkpoint policy. `PromptFitter` accepts them directly: - -```python -from agentomatic.optimize import EarlyStopping, ModelCheckpoint, PromptFitter - -fitter = PromptFitter( - agent="support_bot", - optimizer="gepa_like", - callbacks=[ - EarlyStopping(monitor="score", patience=3, min_delta=0.01), - ModelCheckpoint(save_dir="optimization_results/support_bot"), - ], -) -``` - -The fitter generates a report by default. For a serializable result from any -supported path, create a standalone HTML report with: - -```python -from agentomatic.optimize import generate_html_report - -report_path = generate_html_report(result, output_path="reports/support-fit.html") -``` - -### Synthetic seed data - -`DataSynthesizer` is a Python API for creating a seed dataset; it is not an -`agentomatic dataset` CLI command. Always review generated examples and keep a -separate human-verified validation set. - -```python -from agentomatic.optimize import DataSynthesizer - -synthesizer = DataSynthesizer(model="omlx/Qwen3.5-9B-MLX-4bit") -dataset = await synthesizer.generate( - description="A support assistant that explains order status and refunds.", - n_samples=40, - categories=["orders", "refunds"], -) -dataset.to_jsonl("seed-eval.jsonl") -``` - -### Scaffolded training scripts - -Class-agent project templates may include `train.py` and `eval.py` helpers. -They use `TrainCliSettings`, `train_and_report`, and the same fitter -primitives described here; they are project scripts, not global -`agentomatic train` or `agentomatic eval` commands. Use the generated script's -`--help` output as its contract, and keep agent-specific metrics and schema -requirements in that project. - -## Local oMLX verification - -The live optimization tests exercise the actual OpenAI-compatible provider -path. Point them at a real local model rather than demo mode: - -```bash -export OMLX_BASE_URL=http://127.0.0.1:8000/v1 -export OMLX_API_KEY=local -export AGENTOMATIC_LIVE_MODEL=omlx/Qwen3.5-9B-MLX-4bit - -uv run pytest tests/test_live_omlx_optimize.py \ - tests/test_live_omlx_keras_optimize.py \ - -q --override-ini='addopts=' -``` - -Those tests are intentionally skipped when no reachable model endpoint is -configured. The standard test suite still verifies deterministic optimizer -logic, but a production rollout should run the live suite with a real model and -your own representative evaluation dataset. - -## Production checklist - -1. Keep a labelled validation set and a separate held-out test set. -2. Use a deterministic metric or a calibrated judge before trusting a score. -3. Run without `--apply` first and review the report, prompt diff, and failure - clusters. -4. Verify the saved version in a staging deployment with the same auth, - connections, tools, and model configuration as production. -5. Run the [deployment verifier](verifying-a-deployment.md) before promoting - the application through your normal release process. +Moved to [Low level: PromptFitter](../optimization/prompt-fitter.md). diff --git a/docs/guide/platform-features.md b/docs/guide/platform-features.md index ea961d1..4ac99bf 100644 --- a/docs/guide/platform-features.md +++ b/docs/guide/platform-features.md @@ -487,13 +487,22 @@ By default the chain advances only on configured triggers (`timeout`, ## 🧠 12. Conversation Memory & Session Management -Agentomatic provides **automatic conversation memory** for all deployed agents. When a thread store is configured, every `/chat` and `/invoke` call automatically: +Agentomatic provides **automatic conversation memory** for all deployed agents. When a thread store is configured, every `/chat`, `/invoke` and Studio chat run automatically: 1. **Loads prior conversation history** into the agent's `messages` state 2. **Invokes the agent** with that state 3. **Persists** both user and assistant messages to the store 4. **Summarises** older messages when the conversation grows long +!!! note "No store configured? Threads still work" + A freshly scaffolded project configures no database. Instead of failing + every thread route with `400 Thread storage not configured` (which made + Studio's *New Chat* unusable), the platform falls back to a bounded + in-memory store — capped at 1000 threads, oldest evicted first, lost on + restart — and logs a warning. Configure `store=SQLAlchemyStore(...)`, a + `MEMORY` connection, or `DATABASE_URL` / `AGENTOMATIC_DB_URL` for durable + history; set `AGENTOMATIC_EPHEMERAL_THREADS=0` to disable the fallback. + !!! warning "Your agent has to *read* `messages` — loading it is not enough" The platform fills `state["messages"]` and reports `history_loaded`. Whether the model ever sees those turns is the agent's decision. An @@ -1225,7 +1234,7 @@ Each log entry includes `resource_type`, `resource_name` (and BC `recommendations`. Offline train can also audit retrain runs with `TrainConfig.persist_fit_store` / `fit_store_url` → `AGENTOMATIC_FIT_STORE_URL` / `DATABASE_URL` (`OptimizationRunStore`). -See [Prompt Optimization](optimization.md). +See [Prompt Optimization](../optimization/index.md). !!! note "Current gaps" - **Async tasks are not recorded** in invocation logs. Synchronous diff --git a/docs/guide/prompts.md b/docs/guide/prompts.md index ea736b1..13e181a 100644 --- a/docs/guide/prompts.md +++ b/docs/guide/prompts.md @@ -157,6 +157,6 @@ async def generate_response(state: dict[str, Any]) -> dict[str, Any]: | Topic | Link | |-------|------| | Agent structure & discovery | [Agent Structure](agent-structure.md) | -| Prompt optimization | [Optimization](optimization.md) | +| Prompt optimization | [Optimization](../optimization/index.md) | | A/B prompt routing | [Platform Features](platform-features.md) | | Class-based agent integration | [Class-Based Agents](class-agents.md) | diff --git a/docs/guide/tasks.md b/docs/guide/tasks.md index beb10d7..7df5f1b 100644 --- a/docs/guide/tasks.md +++ b/docs/guide/tasks.md @@ -387,6 +387,13 @@ platform = AgentPlatform.from_folder( | `duration_ms` | `float \| None` | Wall-clock duration | | `callback_url` | `str \| None` | Completion webhook | +A task is `failed` when its target raises **or reports failure**: a pipeline +whose status is `failed`, or an endpoint none of whose upstream calls +succeeded. The target's own report is kept in `result` (every step's status +and error, every upstream response) and summarised in `error`. A pipeline that +finished with status `partial` (some steps failed under `on_error: continue`) +is a completed run, so its task `succeeded`; check `result.status`. + ## Persistence & durability The default [`InMemoryTaskStore`][store] keeps records in a bounded, TTL-aware diff --git a/docs/guide/telemetry.md b/docs/guide/telemetry.md index f98a284..b64f4d5 100644 --- a/docs/guide/telemetry.md +++ b/docs/guide/telemetry.md @@ -142,6 +142,13 @@ When enabled, Agentomatic mounts these three endpoints for **every registered ag 2. `GET /api/v1/{agent}/feedback` — List historical feedback entries. 3. `GET /api/v1/{agent}/feedback/export` — Export logs as a JSONL dataset. +Each exported line is `{"query", "expected_answer", "metadata"}`, where +`expected_answer` is the submitted `correction` (or the rated `response`), so +the export can seed an [optimization dataset](../optimization/data.md). The +thread store keeps `query`, `response` and `correction` with every feedback +row (older SQL databases gain these columns automatically at startup); only +rows with a `query` are exported. + ### Submitting Feedback (Client Example) ```python diff --git a/docs/guide/testing.md b/docs/guide/testing.md index 747d8cf..84c208f 100644 --- a/docs/guide/testing.md +++ b/docs/guide/testing.md @@ -830,7 +830,7 @@ tests/ | [Class-Based Agents](class-agents.md) | Full reference for `BaseGraphAgent` | | [Agent Structure](agent-structure.md) | Folder layout and conventions | | [Configuration](configuration.md) | `agent.yaml` and environment config | -| [Optimization](optimization.md) | `compile()` / `fit()` / `evaluate()` lifecycle | +| [Optimization](../optimization/class-agents.md) | `compile()` / `fit()` / `evaluate()` lifecycle | | [Cookbook](cookbook.md) | Practical recipes and patterns | | [Studio Debugging](debug-ui.md) | Visual debugging with Agentomatic Studio | diff --git a/docs/index.md b/docs/index.md index f8d2e51..1497ffb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -147,7 +147,7 @@ It works with **any agent framework** — LangGraph, LangChain, Deep Agent, or r DSPy-inspired prompt fitting with **5 optimizer strategies** (GEPA, MIPRO, rewrite, few-shot bootstrap, param search). Train against a local LLM — no cloud keys, no HTTP server — using the `compile → fit → evaluate` ML lifecycle. - [:octicons-arrow-right-24: Optimization Guide](guide/optimization.md) + [:octicons-arrow-right-24: Optimization Guide](optimization/index.md) @@ -493,7 +493,7 @@ app = platform.build() Register and debug Deep Agent workflows with full Studio support. -- :material-tune-vertical:{ .lg .middle } **[Prompt Optimization](guide/optimization.md)** +- :material-tune-vertical:{ .lg .middle } **[Prompt Optimization](optimization/index.md)** --- diff --git a/docs/optimization/class-agents.md b/docs/optimization/class-agents.md new file mode 100644 index 0000000..62788be --- /dev/null +++ b/docs/optimization/class-agents.md @@ -0,0 +1,202 @@ +# Class agents: compile, fit, evaluate + +`BaseGraphAgent` has a Keras-style optimization API. You describe the data, +what "good" means and how to search; `fit()` runs the +[optimization loop](how-it-works.md) once per epoch and keeps the best prompt. + +Runnable: [`examples/prompt_optimization/train.py`](examples.md) — every step +below, with every option listed in comments. + +## Requirements on the agent + +* Nodes read the prompt with `self.resolve_system_prompt(default=...)` (see + [How it works](how-it-works.md#the-agent-must-read-its-prompt-through-the-framework)). +* `input_to_state(data)` builds the state from the example's `input` + (the question arrives as `data["current_query"]`). +* `state_to_output(state)` returns a dict with the answer under `response` + (or your structured keys — metrics read them). + +## Step by step + +```python +from agentomatic.agents import ( + CallableMetric, EarlyStopping, EpochDiffCallback, MetricLoss, + PromptFitterBridge, WeightedMetric, +) +from agentomatic.optimize import ( + LocalJudgeMetric, PromptSearchSpace, generate_fit_report, load_data, + merge_fit_results, prepare_dataset, +) + +# 1. data — train / validation / holdout / test (see Datasets) +data = load_data("agents/support/datasets/all.jsonl") +data, _ = prepare_dataset(data, augment=True, n_examples=60, + model="omlx/Qwen3.5-9B-MLX-4bit", + llm_base_url="http://127.0.0.1:8000/v1") + +# 2. metrics — reported every epoch (see Metrics, judges & loss) +judge = LocalJudgeMetric(name="judge", model="omlx/Qwen3.5-9B-MLX-4bit", + criteria="Correct, grounded, complete and concise.", + dimensions=["correctness", "completeness"]) +facts = CallableMetric("facts", must_include_score) +quality = WeightedMetric([("judge", judge, 0.6), ("facts", facts, 0.4)], name="quality") + +# 3. loss — what fit() records as loss / val_loss +loss = MetricLoss(quality) + +# 4. optimizer — how candidates are proposed and selected +optimizer = PromptFitterBridge( + agent_name=agent.agent_name, + task_model="omlx/Qwen3.5-9B-MLX-4bit", + rewrite_model="omlx/Qwen3.5-9B-MLX-4bit", + llm_base_url="http://127.0.0.1:8000/v1", + llm_api_key="local", + optimizer="gepa_like", + metric=quality, + max_trials=8, + search_space=PromptSearchSpace(optimize_system_prompt=True, optimize_few_shot=False, + optimize_model_params=False), + min_confidence=0.8, # any PromptFitter keyword is passed through +) + +# 5. compile +agent.compile(data, metrics=[judge, facts, quality], optimizer=optimizer, loss=loss) + +# 6. fit — the test split is never used; measure it before and after +before = agent.evaluate(data.test) +history = agent.fit( + data, + epochs=2, + callbacks=[EpochDiffCallback(epochs=2), + EarlyStopping(monitor="val_loss", mode="min", patience=1)], +) +after = agent.evaluate(data.test) + +# 7. report, and keep the prompt if it earned it +generate_fit_report(history, output_path="reports/support.html", + baseline_eval=before, final_eval=after, eval_dataset=data.test) +merge_fit_results(history.fit_results).apply(version="v2_fit", agent_dir="agents/support") +``` + +## `compile(dataset=None, metrics=None, optimizer=None, loss=None)` + +| Argument | Meaning | +| --- | --- | +| `dataset` | Default data for `fit()`/`evaluate()`. | +| `metrics` | Reported metrics — either [family](metrics.md#two-metric-families). Names must be unique and not `loss`. | +| `optimizer` | A `PromptFitterBridge` (or any object with `optimize(agent, dataset, metrics) -> dict`). | +| `loss` | A loss, or a metric (wrapped as `MetricLoss`). | + +## `PromptFitterBridge(...)` + +The bridge runs one `PromptFitter.fit` per epoch on the agent instance, starting +from the agent's current prompt. + +| Option | Default | Meaning | +| --- | --- | --- | +| `agent_name` | `""` | Agent name (reports, `prompts.json`). | +| `task_model` | `"ollama/qwen2.5:7b"` | Model for failure clustering and the optimizer's analysis calls. (The agent itself runs on its own LLM.) | +| `rewrite_model` | `"openai/gpt-4.1"` | Model that writes the candidate prompts. | +| `llm_base_url`, `llm_api_key` | — | OpenAI-compatible endpoint for optimizer and judge calls (also set as the process default at construction, so epoch-0 judge scoring uses it). | +| `optimizer` | `"rewrite"` | See [optimizers](how-it-works.md#optimizers-how-candidates-are-proposed). | +| `metric` | compiled loss's metric, else first compiled metric | The objective candidates are selected on. | +| `max_trials` | `8` | Distinct candidates per epoch. | +| `**kwargs` | | Passed to `PromptFitter`: `search_space`, `patience`, `min_absolute_improvement`, `min_confidence`, `max_generalization_gap`, `holdout_tolerance`, `min_transfer_ratio`, `holdout_fraction`, `reflect_on`, `reflection_size`, `concurrency`, `callbacks`, `auto_report`, `experiment_dir`, … (see [PromptFitter](prompt-fitter.md#constructor-options)). | + +How the bridge splits your `AgentDataset`: `train` → what the optimizer +learns from; `validation` → selection; `holdout` → the veto gate (or a slice of +validation); `test` → dropped. Without a `validation` split, 20 % of train is +carved off with a seeded shuffle. + +After each epoch: `agent._last_fit_result` is that epoch's `PromptFitResult`, +`agent._last_optimize_status` is `"ok"` or `"skipped: "` (a failed +optimizer never crashes `fit()`; check this when nothing changed), and the +prompt is kept only if the epoch improved it. + +## `fit(dataset=None, *, epochs=1, verbose=1, callbacks=None, validation_data=None, ...)` + +| Argument | Meaning | +| --- | --- | +| `epochs` | Optimization passes; each starts from the previous best. | +| `verbose` | `0` silent, `1` one line per epoch. | +| `callbacks` | `agentomatic.agents` callbacks (below). | +| `validation_data` | Examples for `val_*` logs (default: the validation split). | +| `search_space`, `optimize_mode`, `optimize_prompt`, `optimize_params`, `optimize_few_shot`, `model_param_space`, `max_trials`, `**optimize_kwargs` | Per-call overrides for the bridge. Given without a compiled optimizer, `fit()` builds a `PromptFitterBridge` from them. | + +Before the first epoch, `fit()` records a baseline row (epoch −1), so the +curve starts at the true starting loss. + +Returns a `History`: + +| Attribute | Contents | +| --- | --- | +| `history.history` | `{"loss": [...], "val_loss": [...], "judge": [...], "val_judge": [...], ...}` — baseline first | +| `history.epoch` | Epoch indices (`-1` = baseline) | +| `history.fit_results` | One `PromptFitResult` per optimized epoch | +| `history.params` | Epochs, optimizer, metric names, split sizes | +| `history.best("val_loss", mode="min")`, `history.final("loss")`, `history.summary()` | Helpers | + +### Callbacks for `fit()` + +| Callback | Options | Does | +| --- | --- | --- | +| `EarlyStopping` | `monitor="loss"`, `mode="auto"` (min for `*loss`), `patience=0`, `min_delta=0.0`, `restore_best=False` | Stops when `monitor` stops improving; `restore_best` restores the best epoch's prompt/config | +| `EpochDiffCallback` | `epochs=1`, `prompt_key="system_prompt"` | Prints loss and a unified prompt diff after each epoch; `per_epoch` keeps the records | +| your own `Callback` subclass | `on_train_begin/end`, `on_epoch_begin/end(epoch, logs)` | Anything; set `self.agent.stop_training = True` to stop | + +These are different from the `agentomatic.optimize` callbacks +(`EarlyStopping`, `ModelCheckpoint`, …), which control the fitter's inner +trial loop — pass those to the bridge as `callbacks=[...]`. Passing one to +`fit()` raises a `TypeError` that says so. + +## `evaluate(dataset, metrics=None)` + +Runs the agent on each example and scores it with `metrics` (default: the +compiled ones). Returns an `EvaluationReport`: `scores` (mean per metric), +`example_results` (per example: `prediction`, `scores`, `error`, +`duration_ms`, judge rationale in `metadata`) and `metadata` +(`metric_failures`, `transform_errors`). Pass two reports (before and after +`fit()`) to [`generate_fit_report`](reports.md) for a per-example comparison. + +## Saving + +`agent.save(path)` writes the compiled config (best prompt, params, few-shot +examples) and `fit_history.json`; `Agent.load(path)` / `agent.load_compiled(path)` +restore them. `PromptFitResult.apply(version=..., agent_dir=...)` instead writes a +new version to the agent's `prompts.json`. + +## One call: `train_and_report` + +Scaffolded projects (`agentomatic init NAME --template basic|full|…`) ship +`agents/NAME/train.py`, which wraps the steps above in `train_and_report` and +reads the model endpoint from the project's stack: + +```bash +python agents/NAME/train.py --epochs 2 --trials 12 --augment --n-examples 40 +AGENTOMATIC_STACK=gemini python agents/NAME/train.py --optimizer gepa_like --apply +``` + +| Flag / env (`AGENTOMATIC_…`) | Default | Meaning | +| --- | --- | --- | +| `--stack` | `local` | Stack whose `default` / `rewrite` LLM profiles are used | +| `--epochs` | `2` | Epochs | +| `--trials` | `12` | Candidates per epoch | +| `--patience` | `2` | Fitter patience | +| `--optimizer` | `rewrite` | `rewrite`, `gepa_like`, `mipro_like`, `few_shot_bootstrap`, `param_search` | +| `--dataset` | `datasets/all.jsonl` | Dataset path | +| `--augment`, `--n-examples`, `--persist` | off | Augmentation (see [Datasets](data.md#augmentation)) | +| `--apply`, `--apply-as` | off, `v2_fit` | Write the prompt to `prompts.json` if it improved | +| `--min-improvement` | `0.001` | Minimum validation lift | +| `--persist-fit-store` | off | Also store retrain artefacts in `AGENTOMATIC_FIT_STORE_URL` / `DATABASE_URL` | + +In Python, `train_and_report(agent, config=TrainConfig(...))` accepts the same +settings plus `required_keys`, `judge_criteria`, `judge_dimensions`, +`judge_weight`, `sequential`, `concurrency`, `optimize_model_params`, +`augment_strategies`, `augment_model` and `evaluate_baseline` (default on: +score the test split before *and* after, for the report). It returns a +`TrainResult` with `history`, `fit_result` (merged across epochs), +`fit_results`, `baseline_eval_scores`, `eval_scores`, `report_path`, +`optimize_status`, `applied_version` and `dataset_sizes`. + +The staged primitives it uses are public too: `build_default_metrics`, +`compile_agent`, `fit_agent`, `evaluate_agent`. diff --git a/docs/optimization/cli.md b/docs/optimization/cli.md new file mode 100644 index 0000000..e6d5a9d --- /dev/null +++ b/docs/optimization/cli.md @@ -0,0 +1,187 @@ +# CLI & legacy API + +`agentomatic optimize` optimizes an agent that is **already running** behind +the Agentomatic API, from a terminal. It uses the same `PromptFitter` engine, +datasets and acceptance rules as the Python APIs ([How it works](how-it-works.md)). +For in-process agents and full control, prefer +[class agents](class-agents.md) or [`PromptFitter`](prompt-fitter.md). + +## Before you start + +Install the optimization dependencies and create a dataset with a stable, +measurable expectation for each request: + +```bash +pip install "agentomatic[optimize]" +``` + +```jsonl +{"query":"What is the refund period?","expected_answer":"30 days"} +{"query":"How do I reset my password?","expected_answer":"reset, password"} +``` + +`query` is required. `expected_answer`, `context` (a list of strings), and +`metadata` are optional. Use `exact_match` when the expected answer should +match exactly; use `contains` with comma-separated required phrases when a +response can vary in wording. For generative quality, use a calibrated judge +metric and a held-out test dataset. + +Start the target platform before running the API-backed paths: + +```bash +agentomatic run --port 8000 +``` + +## Recommended: CLI fitter modes + +The CLI has two families. `prompt_only` is the legacy default; every other +mode uses the modern `PromptFitter` path. + +```bash +# Prompt + configuration rewrite against the running agent. +agentomatic optimize support_bot \ + --dataset eval.jsonl \ + --val-dataset validation.jsonl \ + --test-dataset holdout.jsonl \ + --mode rewrite \ + --metrics contains \ + --llm omlx/Qwen3.5-9B-MLX-4bit \ + --rewrite-llm omlx/Qwen3.5-9B-MLX-4bit \ + --host http://127.0.0.1:8000 \ + --max-trials 20 \ + --apply +``` + +`--dataset` is what the optimizer learns from, `--val-dataset` what candidates +are selected on, and `--test-dataset` the fitter's **holdout gate** — it can +only veto candidates ([Generalization](generalization.md#data-roles)). Keep a +further test set outside the command and evaluate it afterwards. + +`--apply` writes a prompt version only when the result passes the fitter's +improvement and generalization guards. Omit it to inspect the report first. +The command writes reports and trial artifacts under `.optimize/` by default. + +### Mode reference + +| CLI mode | Fitter optimizer | Best for | +|---|---|---| +| `rewrite` | `RewriteOptimizer` | Improving system instructions from failure patterns | +| `gepa_like` | `GEPALikeOptimizer` | Targeted edits once evaluator feedback is useful | +| `mipro_like` | `MIPROLikeOptimizer` | Exploring prompt and few-shot combinations | +| `few_shot` | `FewShotBootstrapOptimizer` | Selecting strong demonstrations from labelled data | +| `param_search` | `ParamSearchOptimizer` | Searching model, RAG, or tool parameter grids | +| `apo` | `APOOptimizer` | Trace-aware critique and edit search | + +`--mode` accepts exactly the values in the first column plus `prompt_only`. +The fitter API also accepts the aliases `mipro`, `gepa`, and +`few_shot_bootstrap`; prefer the CLI spellings above in new scripts. + +### Parameter-only search + +Use a search space to make the allowed changes explicit. This avoids an +optimization run silently tuning parameters you did not intend to change. + +```yaml +# search-space.yaml +optimize_system_prompt: false +optimize_few_shot: false +optimize_model_params: true +search_method: tpe +model_param_space: + temperature: [0.0, 0.2, 0.5] + top_p: [0.8, 0.95] +rag_param_space: + top_k: [3, 5, 8] +``` + +```bash +agentomatic optimize support_bot \ + --dataset eval.jsonl \ + --mode param_search \ + --search-space search-space.yaml \ + --search-method tpe \ + --no-optimize-prompt \ + --param temperature=0.0,0.2,0.5 \ + --max-trials 18 +``` + +`--param` adds or replaces a model parameter grid entry. `--search-method` +accepts `grid`, `random`, or `tpe`. For `apo`, `--node-match` scopes trace +critique to matching graph-node or subagent names; `--n-runners` sets fitter +evaluation concurrency. + +Run `agentomatic optimize --help` for the complete, version-specific CLI +contract. Do not use undocumented commands such as `agentomatic eval`, +`agentomatic route`, or `agentomatic promote`: they are not Agentomatic CLI +commands. + +## Compatibility API: `PromptOptimizer` + +`PromptOptimizer` and CLI `--mode prompt_only` remain available for existing +API-backed integrations. It supports three legacy strategies only: +`iterative_rewrite`, `few_shot`, and `chain_of_thought`. + +```python +from agentomatic.optimize import Dataset, PromptOptimizer + +optimizer = PromptOptimizer( + agent="support_bot", + metrics=["contains"], + strategy="iterative_rewrite", + llm="omlx/Qwen3.5-9B-MLX-4bit", + api_base="http://127.0.0.1:8000", +) + +dataset = Dataset.from_jsonl("eval.jsonl") +result = await optimizer.optimize( + dataset=dataset, + max_iterations=10, + target_score=0.9, +) +print(result.report()) +``` + +Equivalent CLI: + +```bash +agentomatic optimize support_bot \ + --dataset eval.jsonl \ + --mode prompt_only \ + --strategy iterative_rewrite \ + --host http://127.0.0.1:8000 +``` + +Do not use old names such as `mipro`, `ensemble`, or +`bootstrap_randomsearch` with `--strategy`; they are not legacy CLI strategy +values. Use the fitter modes above instead. + +## Local oMLX verification + +The live optimization tests exercise the actual OpenAI-compatible provider +path. Point them at a real local model rather than demo mode: + +```bash +export OMLX_BASE_URL=http://127.0.0.1:8000/v1 +export OMLX_API_KEY=local +export AGENTOMATIC_LIVE_MODEL=omlx/Qwen3.5-9B-MLX-4bit + +uv run pytest tests/test_live_omlx_optimize.py \ + tests/test_live_omlx_keras_optimize.py \ + -q --override-ini='addopts=' +``` + +Those tests are intentionally skipped when no reachable model endpoint is +configured. The standard test suite still verifies deterministic optimizer +logic, but a production rollout should run the live suite with a real model and +your own representative evaluation dataset. + +## Production checklist + +1. Keep a labelled validation set and a separate held-out test set. +2. Use a deterministic metric or a calibrated judge before trusting a score. +3. Run without `--apply` first and review the report, prompt diff, and failure + clusters. +4. Verify the saved version in a staging deployment with the same auth, + connections, tools, and model configuration as production. +5. Run the [deployment verifier](../guide/verifying-a-deployment.md) before promoting + the application through your normal release process. diff --git a/docs/optimization/data.md b/docs/optimization/data.md new file mode 100644 index 0000000..d87020a --- /dev/null +++ b/docs/optimization/data.md @@ -0,0 +1,149 @@ +# Datasets & augmentation + +## The dataset file + +Class-agent datasets are JSONL, one `AgentExample` per line (the format every +template's `datasets/all.jsonl` uses): + +```json +{"id": "refund_2", + "split": "validation", + "input": {"current_query": "I bought an annual plan last week. Can I still get my money back?"}, + "expected_output": {"response": "Yes. Annual plans can be refunded within 30 days of purchase."}, + "metadata": {"must_include": ["30 days"], "topic": "refund"}, + "rubric": {"correctness": "States the 30-day window for annual plans."}, + "tags": ["billing"]} +``` + +| Field | Required | Meaning | +| --- | --- | --- | +| `id` | recommended | Stable id; generated (`example_0003`) when missing. Reports join before/after results on it. | +| `split` | recommended | `train`, `validation`, `holdout` or `test` (default `train`). See [data roles](generalization.md#data-roles). | +| `input` | yes | What the agent receives — the same dict `input_to_state()` gets. The question is read from `question`, `query`, `current_query` or `request`. `input.context` becomes the judge's retrieval context. | +| `expected_output` | for scoring | The reference answer (`{"response": "..."}`, or your structured output). Judges receive it as the reference. | +| `metadata` | no | Anything metrics need: required facts, topic, difficulty. Class-agent metrics read it (`example.metadata`). | +| `rubric` | no | Per-dimension criteria, appended to the judge's reference. | +| `tags` | no | Free-form labels. | + +```python +from agentomatic.optimize import load_data + +data = load_data("agents/support/datasets/all.jsonl") # AgentDataset +data.train, data.validation, data.holdout, data.test # lists of AgentExample +``` + +`PromptFitter` works on the simpler `agentomatic.optimize.Dataset` of +`DataPoint(query, expected_answer, context, metadata)`. Convert, or build one +directly: + +```python +from agentomatic.optimize import Dataset + +valset = Dataset(points=[e.to_datapoint() for e in data.validation]) # keeps metadata +Dataset.from_list([{"query": "Refund window?", "expected_answer": "30 days"}]) +Dataset.from_jsonl("eval.jsonl") # {"query", "expected_answer", "context", "metadata"} +trainset, valset = dataset.split(ratio=0.8) +``` + +## Writing good examples + +* **One fact per expectation you care about.** An expected answer is what the + judge compares against; state the facts a correct answer must contain, not + a vague description. +* **Put checkable facts in `metadata`** (`must_include`, allowed values) so a + deterministic metric can score them. +* **Cover the question space in every split.** Validation and holdout should + ask about the same topics as train, in different words. +* **Size.** Validation decides; it needs enough examples for a real + improvement to be [significant](generalization.md#the-acceptance-rules) — + 20+ if you can. Seed rows in scaffolded templates are placeholders: replace + them before trusting a score. + +## Augmentation + +`prepare_dataset(augment=True)` asks an LLM for variations of your **train** +rows and adds the ones that are safe to learn from. + +```python +from agentomatic.optimize import prepare_dataset + +data, written = prepare_dataset( + data, + augment=True, + n_examples=60, # target TOTAL size + strategies=["paraphrase", "formality_shift", "expansion"], + model="omlx/Qwen3.5-9B-MLX-4bit", + llm_base_url="http://127.0.0.1:8000/v1", + llm_api_key="local", + per_call=4, + persist=True, + persist_path="agents/support/datasets/all.augmented.jsonl", +) +print(data.metadata["augment_stats"]) +``` + +### Options + +| Option | Default | Meaning | +| --- | --- | --- | +| `augment` | `False` | Run augmentation (`False`: only persist / pass through). | +| `n_examples` | 3 × seed size | Target **total** size; only train grows. | +| `strategies` | `["paraphrase"]` | See the table below. Strategies rotate call by call, so a short run still mixes them. | +| `model` | `"openai/gpt-4o-mini"` | Augmentation LLM spec (`omlx/…`, `openai/…`, `ollama/…`, …). | +| `llm_base_url`, `llm_api_key` | process default | Endpoint for these calls only — no process-wide side effect. | +| `per_call` | `4` | Variations requested per call. Small batches finish within small local models' output limits; a truncated reply is salvaged, not dropped. | +| `max_tokens` | `4096` | Reply budget per call. | +| `strict` | `False` | Raise `RuntimeError` when fewer rows than requested were added (otherwise a warning). | +| `persist`, `persist_path`, `seed_path` | | Write the result as JSONL for review. Default path `.augmented.jsonl` (or `.prepared.jsonl` without augmentation); the seed file is never overwritten. | + +### Strategies + +| Strategy | Kind | Produces | +| --- | --- | --- | +| `paraphrase` | label-preserving | Same question, different words | +| `perturbation` | label-preserving | Small wording changes, typos, word order | +| `add_noise` | label-preserving | Irrelevant detail around the question | +| `simplify` | label-preserving | Shorter, plainer phrasing | +| `formality_shift` | label-preserving | More or less formal register | +| `expansion` | new question | Related questions on the same topic | +| `complicate` | new question | Multi-part or harder versions | +| `adversarial` | new question | Tricky or misleading phrasings | +| `edge_case` | new question | Boundary conditions | + +**Label-preserving** rows keep the seed's `expected_output`, `metadata` and +input keys, so every metric scores them exactly like the seed. **New +question** rows get an LLM-written answer in the seed's schema — review them +before trusting scores computed on them. + +### What keeps augmented data safe + +* Only **train** grows; validation, holdout and test are untouched, and + without a train split nothing is generated. +* Generated questions that duplicate an existing one, or are ≥ 90 % similar + to any validation / holdout / test question, are dropped (a paraphrase of a + test question in train would inflate every score measured on it). +* Each new row records `metadata.source = "augment"`, its `strategy` and its + `parent_id`. +* `data.metadata["augment_stats"]` records `requested`, `calls`, `parsed`, + `duplicates`, `near_duplicates`, `empty_calls`, `schema_mismatch`, `added` + and `by_strategy`; the [report](reports.md) shows them. + +Runnable: [`data_augmentation.py`](examples.md), or +`python examples/prompt_optimization/train.py --augment`. + +## Generating a seed dataset from scratch + +`DataSynthesizer` can draft a first dataset from a description. Always review +it and keep a human-written validation set. + +```python +from agentomatic.optimize import DataSynthesizer + +synth = DataSynthesizer(model="omlx/Qwen3.5-9B-MLX-4bit", base_url="http://127.0.0.1:8000/v1") +dataset = await synth.generate( + description="A support assistant that explains refunds and account settings.", + n_samples=40, + categories=["refunds", "security"], +) +dataset.to_jsonl("seed.jsonl") +``` diff --git a/docs/optimization/examples.md b/docs/optimization/examples.md new file mode 100644 index 0000000..4c86a5c --- /dev/null +++ b/docs/optimization/examples.md @@ -0,0 +1,62 @@ +# Examples + +[`examples/prompt_optimization`](https://github.com/UnicoLab/agentomatic/tree/main/examples/prompt_optimization) +contains runnable scripts, from scoring a few answers to a full training run. +Each one is commented step by step and lists every option of the methods it +uses. All of them optimize the same small `SupportAgent` (it answers Acme +Cloud customer questions from a policy knowledge base) on a 26-row dataset +split into train / validation / holdout / test. + +| Script | Level | Covers | Docs | +| --- | --- | --- | --- | +| `metrics_and_judges.py` | building blocks | Every metric family, LLM judges (single, per-dimension, panel), composite objectives, adapters, losses — scored on three fixed answers | [Metrics](metrics.md) | +| `data_augmentation.py` | building blocks | `prepare_dataset` strategies and options, leak protection, `augment_stats` | [Datasets](data.md) | +| `anti_overfitting.py` | mechanics | The acceptance rules on hand-made numbers; a fit in which a prompt memorising the validation answers is vetoed and a general one kept | [Generalization](generalization.md) | +| `low_level_prompt_fitter.py` | low level | `PromptFitter` with every knob, inner-loop callbacks, before/after test evaluation, report | [PromptFitter](prompt-fitter.md) | +| `train.py` | high level | `load_data` → `prepare_dataset` → metrics → loss → `PromptFitterBridge` → `compile` → `fit` → test evaluation → `generate_fit_report` | [Class agents](class-agents.md) | + +## Running them + +Start an OpenAI-compatible model server (oMLX shown; llama.cpp, vLLM, +LM Studio or Ollama's `/v1` work the same way), then: + +```bash +omlx serve --model Qwen3.5-9B-MLX-4bit # http://127.0.0.1:8000/v1 + +python examples/prompt_optimization/metrics_and_judges.py +python examples/prompt_optimization/data_augmentation.py --n-examples 40 +python examples/prompt_optimization/anti_overfitting.py +python examples/prompt_optimization/low_level_prompt_fitter.py --trials 8 +python examples/prompt_optimization/train.py --epochs 2 --trials 6 --augment +``` + +Every script takes `--base-url`, `--api-key`, `--model`, `--judge-model`, +`--rewrite-model`, `--out-dir` and `--log-level` (or `OMLX_BASE_URL`, +`OMLX_API_KEY`, `AGENTOMATIC_LOCAL_MODEL`, …). Outputs land in +`examples/prompt_optimization/out/`. + +What `train.py` prints at the end (with a real model the numbers differ): + +```text +Step 10/10 — results + validation (selection): 0.000 → 0.875 + test judge 0.000 → 0.840 + test facts 0.000 → 1.000 + test quality 0.000 → 0.904 + prompt diff: +--- initial ++++ final +@@ -1 +1 @@ +-You are a support assistant for Acme Cloud. Help the customer. ++You are a precise Acme Cloud support assistant. Quote the relevant policy snippet verbatim, then answer in one short sentence. Never invent policy. + report: examples/prompt_optimization/out/train_report.html +``` + +## Tested + +`tests/test_prompt_optimization_examples.py` runs every script as a +subprocess against a deterministic fake OpenAI-compatible server +(`tests/fake_openai_server.py`) and checks the behaviour each one teaches: +the prompt improves on validation *and* test, augmentation grows only train, +the memorising prompt is rejected, and the report contains the prompts, +candidates and per-example results. diff --git a/docs/optimization/generalization.md b/docs/optimization/generalization.md new file mode 100644 index 0000000..73d55e9 --- /dev/null +++ b/docs/optimization/generalization.md @@ -0,0 +1,138 @@ +# Generalization & overfitting + +An optimizer rewarded only for its validation score will find prompts that +are good *at the validation set*: prompts that paste validation answers in, +that exploit one judge's quirks, or that win by luck on six examples. None of +that survives contact with real users. Agentomatic prevents it with two +things: **separate data roles** and **acceptance rules** every candidate must +pass. + +Runnable demo: [`anti_overfitting.py`](examples.md) applies the rules to +hand-made numbers, then runs a fit in which a prompt that memorises the +validation answers is vetoed and a general prompt is kept. + +## Data roles + +| Split | Role | Can it change which prompt wins? | +| --- | --- | --- | +| `train` | What the optimizer **learns from**: failure clusters, judge feedback, few-shot demos. Augmentation grows only this split. | Indirectly (it shapes proposals) | +| `validation` | What candidates are **scored and selected on**. | Yes | +| `holdout` | A **veto-only gate**: a candidate that wins on validation but does not carry over here is rejected. It never picks a winner. | Can only say no | +| `test` | **Never passed to the fitter.** Evaluate it before and after `fit()`. | No | + +* Label rows with `"split": "train" | "validation" | "holdout" | "test"` + (see [Datasets](data.md)). Unknown split names trigger a warning. +* No `holdout` rows? The fitter reserves `holdout_fraction` (default 20 %, at + least one example) of validation as the gate. If validation is too small to + split, the slice comes from train instead and is removed from train, so the + gate stays unseen by the optimizer. +* No `validation` rows? The class-agent bridge carves 20 % of train (seeded + shuffle); the optimizer then still never selects on what it learned from. +* With `PromptFitter` directly, `fit(trainset, valset, metric, testset=gate)`: + the `testset` argument **is the holdout gate**. Keep your real test split + out of the fitter and evaluate it yourself. + +Why keep the test split out? Once a split has been used to accept or reject +candidates, the winner was chosen partly *because* it did well there; its +score on that split is optimistic. Only a split that played no part in the +choice gives an unbiased estimate. + +## The acceptance rules + +A candidate replaces the incumbent (the best config so far) only if **all** +of these hold. Every rejected candidate records which rule failed in its +`reason`. + +| Rule | Knob (default) | Rejects | +| --- | --- | --- | +| **Minimum lift**: validation score − incumbent ≥ threshold | `min_absolute_improvement` (`0.001`; the examples use `0.01`) | Changes too small to matter | +| **Significance**: paired bootstrap over the validation examples, P(candidate > incumbent) ≥ threshold | `min_confidence` (`0.8`) | Wins that are likely judge noise or luck on a few examples | +| **No holdout regression**: holdout − incumbent's holdout ≥ −tolerance | `holdout_tolerance` (`0.02`) | Prompts that get better on validation and worse elsewhere | +| **Bounded gap growth**: (validation − holdout) − (incumbent's validation − holdout) ≤ max(gap, ½ × validation lift) | `max_generalization_gap` (`0.15`) | Prompts whose gain is concentrated on the validation examples | +| **Transfer**: holdout lift ≥ ratio × validation lift | `min_transfer_ratio` (`0.25`) | Memorisation: large validation gain, (almost) none held out. Skipped when the holdout is already ≥ 0.99. | + +Before a candidate reaches these rules it must also beat the incumbent on a +validation **minibatch of the same examples** (promotion), and be a config +that was not already scored (duplicates are skipped). + +The rules are relative to the **incumbent**, not to absolute values: a +dataset whose baseline already scores differently on validation and holdout is +not penalised for that difference, only for widening it. + +### Worked examples + +Incumbent: validation 0.50, holdout 0.50. Defaults as above. + +| Candidate | Validation | Holdout | Decision | Why | +| --- | --- | --- | --- | --- | +| General improvement | 0.80 | 0.75 | accept | lift 0.30, gap grows 0.05 | +| Memorised validation answers | 1.00 | 0.50 | reject | gap grows 0.50 > 0.25; holdout lift 0 < 25 % of 0.50 | +| Gain does not transfer | 0.62 | 0.52 | reject | holdout +0.02 < 25 % of +0.12 | +| Holdout regression | 0.52 | 0.45 | reject | holdout −0.05 < −0.02 | +| Gap blows up | 0.95 | 0.60 | reject | gap grows 0.35 > 0.225 | + +Significance, six validation examples, incumbent per-example scores +`[0, 0, 1, 1, 0, 1]`: + +| Candidate | Per-example | P(better) | Decision | +| --- | --- | --- | --- | +| One example flipped | `[1, 0, 1, 1, 0, 1]` | 0.67 | reject (not significant) | +| Every failure fixed | `[1, 1, 1, 1, 1, 1]` | 0.98 | accept | + +Both checks are public functions if you want to apply them yourself: + +```python +from agentomatic.optimize import check_generalization, paired_improvement_confidence + +check = check_generalization( + fit_score=0.62, holdout_score=0.52, # candidate + baseline_fit=0.50, baseline_holdout=0.50, # incumbent + max_gap=0.15, holdout_tolerance=0.02, min_transfer_ratio=0.25, +) +check.ok, check.reason # False, "Does not transfer: validation improved +0.1200 but …" + +paired_improvement_confidence([1, 0, 1, 1, 0, 1], [0, 0, 1, 1, 0, 1]) # 0.67 +``` + +## Learning only from train + +The optimizer's inputs — the examples it analyses, the failure clusters, the +judge feedback, the few-shot demos it may add — come from **train** +(`reflect_on="train"`, `reflection_size=8`). An optimizer that read the +validation failures would write prompts that fix exactly those examples, and +validation would stop measuring anything. `reflect_on="validation"` restores +that behaviour for comparison only. + +## After the fit: `apply()` and the test split + +```python +before = agent.evaluate(data.test) # or evaluate before fit() +history = agent.fit(data, epochs=2) +after = agent.evaluate(data.test) +result = merge_fit_results(history.fit_results) + +result.apply(version="v2_fit", agent_dir="agents/support") +``` + +`apply()` writes `prompts.json` only when the result improved +(`min_improvement`) and, when a holdout score exists, its generalization gap +is within `max_generalization_gap`. `force=True` overrides both — do it only +after reading the report. + +## Practical guidance + +* **Size.** Six validation examples can only distinguish large improvements + (one flipped example is not significant at 0.8). Aim for 20+ validation and + 10+ holdout examples; augment **train**, never validation or holdout. +* **Diversity.** Validation and holdout should cover the same kinds of + questions, phrased differently. A holdout made of one topic cannot tell + memorisation from a topic-specific gain. +* **Deterministic judges.** `temperature=0.0` for every judge; add a + deterministic metric to the objective. +* **Read the reasons.** A run where every candidate is `not significant` + means the validation set is too small or the judge too noisy; a run full of + `does not transfer` means the optimizer is fitting the validation set — add + data before loosening the rules. +* **Loosening the rules** (`min_confidence=0.6`, `min_transfer_ratio=0`, + larger `max_generalization_gap`) makes more candidates pass and more of + them overfit. Measure the test split before and after if you do. diff --git a/docs/optimization/how-it-works.md b/docs/optimization/how-it-works.md new file mode 100644 index 0000000..dbd1b93 --- /dev/null +++ b/docs/optimization/how-it-works.md @@ -0,0 +1,125 @@ +# How optimization works + +This page follows one optimization run from start to finish. Every entry point +(`train_and_report`, `agent.fit()`, `PromptFitter`, `agentomatic optimize`) +runs this loop. + +## The agent must read its prompt through the framework + +Candidates are scored by running your real agent with a different prompt. The +prompt reaches the agent through +`BaseGraphAgent.resolve_system_prompt(...)`, which resolves, first match wins: + +1. the per-request `system_prompt_override` — how each **candidate** is injected; +2. `compiled_config["system_prompt"]` — what `fit()` keeps as the best prompt; +3. the agent's `system_prompt` attribute — the **baseline**; +4. the active version in `prompts.json`; +5. the `default=` you pass. + +```python +class SupportAgent(BaseGraphAgent[SupportState]): + system_prompt = "You are a support assistant for Acme Cloud." # baseline + + def answer(self, state: SupportState) -> SupportState: + prompt = self.resolve_system_prompt(default=self.system_prompt) # ← required + reply = self.llm.invoke([{"role": "system", "content": prompt}, ...]) + ... +``` + +A node that hard-codes its prompt string cannot be optimized: every candidate +would produce identical answers. Overrides are per request (a `ContextVar`), +so candidates evaluated concurrently on one agent instance never see each +other's prompt. + +## One `fit`, step by step + +```mermaid +flowchart TD + B[1 · Score the baseline prompt
on validation and on the holdout gate] --> R + R[2 · Reflect on TRAIN failures
cluster them, collect judge feedback] --> P + P[3 · Propose up to 4 candidates
rewrite · GEPA · MIPRO · few-shot · params · APO] --> DUP{Same config
already scored?} + DUP -- yes --> SKIP[skip: duplicate] + DUP -- no --> MB[4 · Minibatch screen
candidate vs incumbent on the same examples] + MB -- not better --> NP[not promoted] + MB -- better --> FV[5 · Full validation score] + FV --> ACC{6 · Acceptance rules} + ACC -- all pass --> INC[accepted → new incumbent] + ACC -- any fails --> REJ[rejected, with the reason] + INC --> STOP{7 · Budget spent, patience
exhausted or callback stop?} + REJ --> STOP + NP --> STOP + SKIP --> STOP + STOP -- no --> R + STOP -- yes --> RES[PromptFitResult] +``` + +1. **Baseline.** The starting prompt is scored on the validation set (the + fitter's `valset`) and, if there is one, on the holdout gate. These are the + first incumbent's scores. +2. **Reflection.** Up to `reflection_size` (default 8) **train** examples are + run and scored; examples scoring below 0.5 are clustered into failure + patterns (with a suggested fix each), and judge feedback is collected. This + is what the optimizer learns from. It is drawn from train so the optimizer + never studies the examples it is then selected on (`reflect_on="train"`). +3. **Proposal.** The optimizer (see the table below) proposes up to four + candidate configurations per round from the incumbent and the reflection. + A candidate identical to one already scored is skipped for free: a weak + rewrite model often re-proposes the current prompt. +4. **Minibatch screen.** Each candidate runs on a slice of validation (30 %, + at least 5 examples) and is compared with the incumbent **on the same + examples**. Only candidates that beat it are promoted. +5. **Full validation.** Promoted candidates are scored on all of validation + and on the holdout gate. +6. **Acceptance.** A candidate replaces the incumbent only if every rule + holds: a minimum validation lift, statistical confidence that the lift is + real, and no sign of overfitting on the holdout gate. See + [Generalization & overfitting](generalization.md) for the rules and their + knobs. Every candidate records a `decision` (`duplicate`, `not_promoted`, + `accepted`, `rejected`) and a human-readable `reason`. +7. **Stopping.** Rounds continue until `max_trials` distinct candidates have + been evaluated (rounds = `ceil(max_trials / 4)`), `patience` rounds pass + without an accepted candidate (default 3), or a callback asks to stop + (`ScoreThreshold`, `EarlyStopping`, …). + +The result, a `PromptFitResult`, carries the baseline and best configs and +scores, the holdout scores, every trial with its decision and reason, +per-example results for the baseline and the best config, failure clusters, +suggestions and the settings used. + +## Optimizers — how candidates are proposed + +| `optimizer=` | Proposes | Good when | +| --- | --- | --- | +| `"rewrite"` | A system prompt rewritten from the failure clusters, plus variants with few-shot demos or answer tips | Instructions are vague or missing rules | +| `"gepa_like"` (default in `PromptFitter`) | Targeted edits driven by judge feedback | A judge gives useful feedback | +| `"mipro_like"` | Combinations of instruction variants and few-shot sets | Examples matter as much as instructions | +| `"few_shot"` | Demonstration sets selected from train | The format is right but answers drift | +| `"param_search"` | Grid / random / TPE over `model_param_space` (temperature, top_p, …) | The prompt is fine; sampling is not | +| `"apo"` | Critique of the full execution trace, then an edit | Multi-node agents; scope with `node_match` | +| any object | Whatever its `async propose(current_config, eval_results, dataset_sample, search_space, iteration, context)` returns | Custom strategies (see [`anti_overfitting.py`](examples.md)) | + +What an optimizer *may* change is declared by the +[`PromptSearchSpace`](prompt-fitter.md#search-space). + +## Epochs: `agent.fit(epochs=N)` + +With class agents, each epoch runs one complete `PromptFitter.fit` starting +from the previous epoch's best config, so improvements compound. After each +epoch the agent keeps the new prompt only if that epoch improved it, then +records `loss`/`val_loss` and your compiled metrics in `History`. + +`History.fit_results` holds one `PromptFitResult` per epoch; +`merge_fit_results(history.fit_results)` (and `generate_fit_report(history)`) +combine them into one result from the **original** prompt to the **final** +one. Looking only at the last epoch's result would compare the final prompt +with a prompt an earlier epoch had already improved. + +## What is never done + +* The **test** split is never passed to the fitter. It exists so you can + measure the final prompt on data nothing was selected on — evaluate it + before and after `fit()`. +* Nothing is written to `prompts.json` unless you call `apply()`, and + `apply()` refuses a result that did not improve or whose + validation/holdout gap is too large (override with `force=True` after a + review). diff --git a/docs/optimization/index.md b/docs/optimization/index.md new file mode 100644 index 0000000..a72bce9 --- /dev/null +++ b/docs/optimization/index.md @@ -0,0 +1,94 @@ +# Prompt optimization + +Agentomatic can improve an agent's **system prompt** (and, if you allow it, +its few-shot examples and model parameters) against a labelled dataset. It +scores the current prompt, proposes better ones with an LLM, keeps a candidate +only when it wins on validation data **and** carries over to data the +optimizer never selected on, and reports exactly what changed. + +It never changes live traffic. The result is a candidate prompt version that +you review, verify, and release through your normal delivery process. + +```mermaid +flowchart LR + D[Dataset
train · validation · holdout · test] --> F + M[Metric
deterministic · LLM judge · composite] --> F + A[Agent
reads its prompt via resolve_system_prompt] --> F + F[fit
propose → score → accept / reject] --> R[PromptFitResult
best prompt, every candidate, reasons] + R --> E[Evaluate the untouched test split
before vs after] + R --> H[HTML + JSON report] + R --> P[apply → prompts.json
refuses non-improving / overfit results] +``` + +## Pick an entry point + +All entry points drive the same engine, `PromptFitter`. Pick the one that +matches how much control you need. + +| You want to… | Use | Page | +| --- | --- | --- | +| Optimize a scaffolded project agent with one command | `python agents/NAME/train.py` (`train_and_report`) | [Class agents](class-agents.md#one-call-train_and_report) | +| Write the Keras-style loop yourself: data, metrics, loss, `compile`, `fit`, `evaluate` | `agent.compile(...)` / `agent.fit(...)` | [Class agents](class-agents.md) | +| Control every knob: datasets, metric, search space, callbacks, acceptance rules | `PromptFitter` | [Low level: PromptFitter](prompt-fitter.md) | +| Optimize an agent that is already running behind the API | `agentomatic optimize` | [CLI & legacy API](cli.md) | + +## A complete run in 40 lines + +```python +from agentomatic.agents import CallableMetric, MetricLoss, PromptFitterBridge, WeightedMetric +from agentomatic.optimize import LocalJudgeMetric, generate_fit_report, load_data + +from my_project.agents.support.agent import SupportAgent # any BaseGraphAgent + +data = load_data("agents/support/datasets/all.jsonl") # train / validation / holdout / test +agent = SupportAgent() + +judge = LocalJudgeMetric( + name="judge", + model="omlx/Qwen3.5-9B-MLX-4bit", + criteria="Correct, grounded in the policy, complete and concise.", + dimensions=["correctness", "completeness"], +) +facts = CallableMetric( # deterministic: the example's key facts are in the answer + "facts", + lambda ex, pred: sum(f in pred["response"] for f in ex.metadata["must_include"]) + / len(ex.metadata["must_include"]), +) +quality = WeightedMetric([("judge", judge, 0.6), ("facts", facts, 0.4)], name="quality") + +optimizer = PromptFitterBridge( + agent_name=agent.agent_name, + task_model="omlx/Qwen3.5-9B-MLX-4bit", + rewrite_model="omlx/Qwen3.5-9B-MLX-4bit", + llm_base_url="http://127.0.0.1:8000/v1", + optimizer="gepa_like", + metric=quality, # what candidates are selected on + max_trials=8, +) +agent.compile(data, metrics=[judge, facts, quality], optimizer=optimizer, loss=MetricLoss(quality)) + +before = agent.evaluate(data.test) # the split fit() never sees +history = agent.fit(data, epochs=2) +after = agent.evaluate(data.test) + +generate_fit_report(history, output_path="reports/support.html", + baseline_eval=before, final_eval=after, eval_dataset=data.test) +``` + +The runnable version of this, with every option listed in comments, is +[`examples/prompt_optimization/train.py`](examples.md). + +## What to read next + +1. [How it works](how-it-works.md) — the fit loop step by step, how a + candidate prompt reaches your agent, what epochs do. +2. [Metrics, judges & loss](metrics.md) — which metric to use, LLM-as-judge, + judge panels, composite objectives, how metrics, the objective and the loss + relate. +3. [Generalization & overfitting](generalization.md) — the data roles and the + acceptance rules that stop the optimizer from memorising your validation set. +4. [Datasets & augmentation](data.md) — the JSONL format, splits, and + leak-free synthetic training data. +5. [Reports](reports.md) — reading the before/after report. +6. [Examples](examples.md) — runnable scripts from "score some answers" to a + full training run. diff --git a/docs/optimization/metrics.md b/docs/optimization/metrics.md new file mode 100644 index 0000000..eb22982 --- /dev/null +++ b/docs/optimization/metrics.md @@ -0,0 +1,191 @@ +# Metrics, judges & loss + +A metric turns one answer into a number in `[0, 1]`. Optimization is only as +good as that number: if it rewards the wrong thing, the optimizer will find +prompts that do the wrong thing well. This page covers every metric, how LLM +judges work, how to combine them, and where each one plugs in. + +Runnable companion: [`metrics_and_judges.py`](examples.md) scores three fixed +answers with every metric below. + +## Three slots, one number each + +| Slot | Set with | Used for | +| --- | --- | --- | +| **Reported metrics** | `agent.compile(metrics=[...])` | Computed on train and validation every epoch (`judge`, `val_judge`, …), shown in `History` and the report. They do not steer anything. | +| **Loss** | `agent.compile(loss=...)` | The number `fit()` records as `loss` / `val_loss` and `EarlyStopping` monitors. `MetricLoss(metric)` = `1 − score`. | +| **Objective** | `PromptFitterBridge(metric=...)` or `PromptFitter.fit(..., metric)` | What candidate prompts are **selected** on. | + +Use the same underlying measure for the loss and the objective (for example +`quality` for both, as in the [overview](index.md#a-complete-run-in-40-lines)), +so the epoch curve and the selection agree. Without `metric=`, the bridge uses +your compiled loss's metric, else the first compiled metric. + +## Two metric families + +Every API accepts either family; they are adapted to each other +automatically. + +| | `agentomatic.agents` (class agents) | `agentomatic.optimize` (the fitter) | +| --- | --- | --- | +| Signature | `score(example, prediction) -> float` (sync) | `await evaluate(query, response, expected, context) -> EvalResult` | +| Sees | The whole `AgentExample` (input, expected output, **metadata**) and the output dict | The question, the answer text, the reference text, retrieval context | +| Returns | A float | `EvalResult(score, reason, metadata)` — judges add `dimensions`, rationale; failures set `metadata["evaluation_failed"]` | +| Best for | Checks that need structured output or per-example metadata | Judges and text checks | + +Adapters, if you need to cross explicitly: + +```python +from agentomatic.agents import OptimizeMetricAdapter, as_agent_metric +from agentomatic.optimize import as_optimize_metric + +as_optimize_metric(ContainsTermsMetric(["30 days"])) # class metric → evaluate(...) +OptimizeMetricAdapter(judge, name="judge") # optimize metric → score(...), evaluate(...) +as_agent_metric(judge) # same, idempotent +``` + +Inside `fit()`, `evaluate()` and composite metrics, a class-agent metric +always receives the example's `metadata` and the agent's structured output, +whichever family it is wrapped in. + +## Deterministic metrics + +Fast, free, reproducible. Prefer them wherever the answer has checkable facts +or format. + +| Metric | Family | Options | Scores | +| --- | --- | --- | --- | +| `ContainsTermsMetric(required_terms, case_sensitive=False, name=)` | agents | fixed term list | fraction of terms present in `response` | +| `ExactKeyMatchMetric(required_keys, name=)` | agents | output keys | fraction of keys present in the output dict | +| `ResponseSimilarityMetric(name=, fuzzy=True)` | agents | | fuzzy similarity to `expected_output["response"]` | +| `CallableMetric(name, fn)` | agents | `fn(example, prediction) -> float` | anything — e.g. `example.metadata["must_include"]` coverage | +| `ExactMatchMetric(fuzzy=True, threshold=0.8)` | optimize | | fuzzy exact match vs the reference | +| `ContainsMetric()` | optimize | reference = `"kw1, kw2"` | fraction of comma-separated keywords present | +| `DeterministicMetric(name=, checks=[...])` | optimize | check `type`: `contains`, `not_contains`, `regex`, `min_length`, `max_length`, `json_valid`, `starts_with` | fraction of checks passed | +| `CustomMetric(fn, name=)` | optimize | `fn(query, response, expected, context) -> float` | clamped to `[0, 1]`; `NaN` counts as a failure | +| `LatencyMetric(target_seconds=2.0, max_seconds=10.0)` | optimize | | 1.0 under target, linear to 0 at max; uses the measured run time (fails outside a fit / evaluate run rather than guessing) | +| `CostMetric(target_tokens=500, max_tokens=3000)` | optimize | | token-usage penalty | + +```python +from agentomatic.agents import CallableMetric + +def must_include(example, prediction): + facts = example.metadata.get("must_include") or [] + answer = prediction.get("response", "").lower() + return sum(f.lower() in answer for f in facts) / len(facts) if facts else 0.0 + +facts = CallableMetric("facts", must_include) +``` + +## LLM-as-judge + +A judge asks a model to grade the answer against the reference. Use one when +quality is open-ended (tone, completeness, grounding) and keep a deterministic +metric next to it. + +| Judge | Options | Returns | +| --- | --- | --- | +| `LLMJudgeMetric(criteria, model, name="llm_judge", temperature=0.0, base_url=, api_key=)` | a rubric | one overall score + reason | +| `LocalJudgeMetric(name="local_judge", model, criteria, dimensions=[...], weight=1.0, temperature=0.0, base_url=, api_key=)` | a rubric + named dimensions | overall score, a score per dimension, feedback, motivation, what worked / failed, improvement hints | +| `MultiJudgePanel(judges=[...], aggregation="average", weights=None, name=)` | several `LocalJudgeMetric`s (different models or rubrics) | one aggregated score + per-judge scores | +| `GEvalMetric`, `DeepEvalMetric`, `RedTeamMetric` | DeepEval metrics (`pip install deepeval`) | DeepEval scores, oriented so higher is better | + +```python +from agentomatic.optimize import LocalJudgeMetric, MultiJudgePanel + +judge = LocalJudgeMetric( + name="judge", + model="omlx/Qwen3.5-9B-MLX-4bit", # "provider/model" + criteria="Is the answer correct per the reference, grounded in the policy, " + "complete and concise? Score 0-1.", + dimensions=["correctness", "completeness", "concision"], + temperature=0.0, # same answer → same score every epoch +) + +panel = MultiJudgePanel( + judges=[ + judge, + LocalJudgeMetric(name="strict", model="openai/gpt-4.1-mini", + criteria="Penalise any claim not in the reference."), + ], + aggregation="median", # "average" (weighted) | "median" | "min" | "max" + weights=None, # per judge, for "average" +) +``` + +How judges stay honest: + +* **Reference.** Judges receive the example's expected answer (and, for + `AgentExample`s, a rich reference: expected output, rubric, required facts). + Write expected answers that state the facts a correct answer must contain. +* **Deterministic.** Keep `temperature=0.0`. A judge that scores the same + answer differently each time makes every comparison noise; the + [significance rule](generalization.md#the-acceptance-rules) exists because + judges are noisy anyway. +* **Failures are failures.** An unreachable judge or an unparsable reply + returns `score=0.0` with `metadata["evaluation_failed"] = True`, never a + made-up 0.5. The fitter counts failed examples as 0 for every candidate, + logs them, and flags an "evaluation blackout" when most calls fail. +* **Parsing.** Replies are parsed tolerantly (JSON in prose or code fences, + `score`/`overall_score`/`rating`, 0–10 or 0–100 scales normalised). +* **Panels.** `"median"` is robust to one outlier judge; `"min"` is the + strictest; failed judges are dropped, and the panel fails only if all do. +* **Endpoint.** `model="omlx/…"` routes to `base_url` (or the process + default set by `LLMCaller.configure(base_url=...)` / `OMLX_BASE_URL`). + `openai/…`, `ollama/…`, `gemini/…` and `litellm/…` are the other providers. + +## Combining metrics + +Two equivalent tools; pick by family: + +```python +# optimize family — for PromptFitter +from agentomatic.optimize import CompositeMetric, WeightedMetric + +objective = CompositeMetric( + [ + WeightedMetric(name="judge", metric=judge, weight=0.6), + WeightedMetric(name="facts", metric=facts, weight=0.4), # either family + ], + name="objective", +) + +# agents family — for compile(metrics=..., loss=...) and PromptFitterBridge +from agentomatic.agents import WeightedMetric as AgentWeightedMetric + +quality = AgentWeightedMetric([("judge", judge, 0.6), ("facts", facts, 0.4)], name="quality") +``` + +* Weights are normalised; negative weights are rejected. +* Each component is reported as a **dimension**, in the report and in + `EvalResult.metadata["dimensions"]`. +* A failed component scores 0 (the denominator stays fixed, so a dead judge + lowers the score instead of silently re-weighting the rest); the composite + is marked failed when at least half its weight failed. +* `CompositeMetric` also has `score(example, prediction)`, so it can sit in + `compile(metrics=[...])` directly. + +## Losses + +| Loss | Meaning | +| --- | --- | +| `MetricLoss(metric, name=None)` | `1 − metric.score(...)`; accepts either family | +| `CallableLoss(fn, name="loss")` | `fn(example, prediction) -> float`, lower is better | +| any object with `compute(example, prediction) -> float` | custom | + +`fit()` averages the loss over train (`loss`) and validation (`val_loss`) +after every epoch, plus a baseline row before the first epoch. +`agents.EarlyStopping(monitor="val_loss")` stops when it stops improving. + +## Choosing metrics + +1. Start with **one deterministic metric** that captures correctness for your + data (required facts, keys, format). It is cheap, reproducible and cannot + be sweet-talked. +2. Add **one judge** with a short, specific rubric and 2–4 dimensions for the + open-ended part of quality. +3. Blend them (judge 0.5–0.7) into the **objective**, and use the same blend + as the **loss**. +4. Report the components separately (`compile(metrics=[judge, facts, quality])`) + so a gain in the blend that comes from only one part is visible. +5. Always evaluate the untouched **test** split before and after. diff --git a/docs/optimization/prompt-fitter.md b/docs/optimization/prompt-fitter.md new file mode 100644 index 0000000..1efb5b4 --- /dev/null +++ b/docs/optimization/prompt-fitter.md @@ -0,0 +1,169 @@ +# Low level: `PromptFitter` + +`PromptFitter` is the engine behind every entry point. Use it directly when +you want to build the datasets, the metric and the search space yourself, +attach inner-loop callbacks, or optimize an agent running behind the API. + +Runnable: [`low_level_prompt_fitter.py`](examples.md) — every option below, +commented, plus a before/after test evaluation and a report. + +```python +from agentomatic.optimize import ( + CompositeMetric, Dataset, LocalJudgeMetric, PromptFitter, PromptSearchSpace, + WeightedMetric, generate_fit_report, load_data, +) + +data = load_data("datasets/support.jsonl") +points = lambda examples: Dataset(points=[e.to_datapoint() for e in examples]) + +metric = CompositeMetric([ + WeightedMetric(name="judge", metric=LocalJudgeMetric(model="omlx/Qwen3.5-9B-MLX-4bit", + criteria="Correct and grounded."), weight=0.6), + WeightedMetric(name="facts", metric=facts, weight=0.4), +]) + +fitter = PromptFitter( + agent="support_agent", + local_agent=agent, # in-process, no server + task_model="omlx/Qwen3.5-9B-MLX-4bit", + rewrite_model="omlx/Qwen3.5-9B-MLX-4bit", + llm_base_url="http://127.0.0.1:8000/v1", + optimizer="rewrite", + search_space=PromptSearchSpace(optimize_system_prompt=True, optimize_few_shot=False, + optimize_model_params=False), + max_trials=8, + baseline_system_prompt=agent.system_prompt, +) + +before = agent.evaluate(data.test, [facts]) +result = await fitter.fit(points(data.train), points(data.validation), metric, + testset=points(data.holdout)) # the holdout GATE +agent.compiled_config["system_prompt"] = result.best_prompt +after = agent.evaluate(data.test, [facts]) + +generate_fit_report(result, output_path="reports/fit.html", + baseline_eval=before, final_eval=after, eval_dataset=data.test) +``` + +## `fit(trainset, valset, metric, testset=None)` + +| Argument | Role | +| --- | --- | +| `trainset` | What the optimizer learns from (reflection, few-shot pool). | +| `valset` | What candidates are selected on. | +| `metric` | The objective — any metric of [either family](metrics.md#two-metric-families). | +| `testset` | The **holdout gate** (veto only). Omit to carve `holdout_fraction` of `valset`. Do not pass your real test split. | + +## Constructor options + +**Target and models** + +| Option | Default | Meaning | +| --- | --- | --- | +| `agent` | — | Agent name (reports, `prompts.json`, API routes). | +| `local_agent` | `None` | A live agent instance, called in-process. | +| `api_base`, `api_prefix` | `http://localhost:8000`, `/api/v1` | The running platform to call when `local_agent` is not set. | +| `task_model` | `"ollama/qwen2.5:7b"` | Model for failure clustering and optimizer analysis calls. | +| `rewrite_model` | `task_model` | Model that writes candidate prompts. | +| `llm_base_url`, `llm_api_key` | — | OpenAI-compatible endpoint for those calls (also the process default). | +| `local_judges` | `None` | Extra judge model specs appended to the objective as a panel. | + +**Baseline** + +| Option | Default | Meaning | +| --- | --- | --- | +| `baseline_system_prompt` | `None` | Start from this prompt instead of `prompts.json`. | +| `base_prompt_version` | `"v1"` | `prompts.json` version used when no baseline prompt is given. | +| `baseline_model_params`, `baseline_few_shot_examples` | `None` | Start from these params / demos (the class-agent bridge uses them to compound across epochs). | + +**Search** + +| Option | Default | Meaning | +| --- | --- | --- | +| `optimizer` | `"gepa_like"` | `rewrite`, `gepa_like`, `mipro_like`, `few_shot`, `param_search`, `apo`, or an object with `propose(...)` ([table](how-it-works.md#optimizers-how-candidates-are-proposed)). | +| `search_space` | prompt + few-shot + params | What may change ([below](#search-space)). | +| `max_trials` | `30` | Distinct candidates evaluated; 4 per round. | +| `patience` | `3` | Rounds without an accepted candidate before stopping. | +| `rewrite_passes`, `multipass`, `slm_multipass`, `llm_multipass`, `slm_default_passes`, `llm_default_passes` | auto: 3 passes for local models, 2 for frontier | Draft → critique → refine passes for rewrites. | + +**Acceptance (anti-overfitting)** — see [Generalization](generalization.md#the-acceptance-rules) + +| Option | Default | +| --- | --- | +| `min_absolute_improvement` | `0.001` | +| `min_confidence` | `0.8` | +| `max_generalization_gap` | `0.15` | +| `holdout_tolerance` | `0.02` | +| `min_transfer_ratio` | `0.25` | +| `holdout_fraction` | `0.2` | +| `reflect_on` | `"train"` | +| `reflection_size` | `8` | + +**Execution and output** + +| Option | Default | Meaning | +| --- | --- | --- | +| `concurrency` (alias `n_runners`) | `1` | Parallel agent / judge calls. `>1` disables `sequential` unless forced. | +| `sequential` | auto | Force one call at a time (kind to small local servers). | +| `callbacks` | `None` | Inner-loop callbacks ([below](#callbacks)). | +| `auto_report`, `experiment_dir` | `True`, `.optimize` | Write an HTML trial report per fit. | +| `dashboard` | `False` | Live terminal dashboard. | +| `drain_seconds` | `1.5` | Pause after the loop so in-flight calls finish. | +| `trace_store_path` | `None` | Persist execution traces (used by `apo`). | + +## Search space + +`PromptSearchSpace` declares what an optimizer may change; anything not +enabled stays as in the baseline. + +| Field | Default | Meaning | +| --- | --- | --- | +| `optimize_system_prompt` | `True` | Rewrite the system prompt | +| `optimize_user_template` | `False` | Rewrite the user-message template | +| `optimize_few_shot` | `True` | Choose demonstrations from train | +| `max_few_shot_examples`, `few_shot_selection_strategy` | `5`, `"diversity_weighted"` | Demo count and selection | +| `optimize_model_params` + `model_param_space` | `True`, a temperature / top_p / max_tokens grid | e.g. `{"temperature": [0.0, 0.2, 0.5]}` | +| `optimize_rag_params` + `rag_param_space` | `False`, `{}` | e.g. `{"top_k": [3, 5, 8]}` | +| `optimize_tool_params` + `tool_param_space` | `False`, `{}` | Tool policy knobs | +| `optimize_model_choice` + `model_choices`, `fallback_models` | `False` | Choose among models | +| `search_method` | `"random"` | `grid`, `random` or `tpe` for parameter search | +| `optimize_nodes`, `node_match` | all | Scope trace-aware optimization to graph nodes | + +!!! tip + Turn off what you do not intend to tune. With the defaults, a run may + also change few-shot demos and sampling parameters. + +## Callbacks + +`agentomatic.optimize` callbacks observe and steer the inner trial loop: + +| Callback | Options (defaults) | Does | +| --- | --- | --- | +| `EarlyStopping` (also exported as `OptimizeEarlyStopping`) | `monitor="score"`, `patience=3`, `min_delta=0.005`, `mode="max"`, `restore_best_weights=True` | Stops when the best score stalls | +| `ScoreThreshold` | `threshold=0.85`, `mode="max"` | Stops once good enough | +| `ModelCheckpoint` | `save_dir="optimization_results/checkpoints"`, `save_best_only=True`, `save_freq=1`, `max_checkpoints=5` | Saves accepted configs | +| `PlateauStopping` | `patience=2`, `factor=0.5`, `min_temperature=0.1` | Cools the rewrite temperature on plateaus | +| `TemperatureScheduler` | `initial_temperature=0.7`, `min_temperature=0.1`, `decay_rate=0.9`, `decay_type="exponential"`, `step_size=3` | Schedules the rewrite temperature | +| `NaNStopping` | `max_consecutive_nan=2`, `validate_output=True`, `nan_rollback=True` | Aborts on broken outputs | +| `ProgressLogger` | `show_prompt_diff=False`, `show_delta_chars=0` | Logs each trial | + +## The result: `PromptFitResult` + +| Field | Contents | +| --- | --- | +| `baseline_config`, `best_config` | `PromptRuntimeConfig` (system prompt, user template, few-shot examples, model / RAG / tool params) | +| `baseline_score`, `best_score`, `absolute_improvement`, `improved` | Validation scores | +| `baseline_holdout_score`, `holdout_score`, `generalization_gap` | Holdout gate scores | +| `trials` | Every candidate and phase: `name`, `phase` (`skipped`, `minibatch`, `full_val`), `score`, `decision`, `reason`, `confidence`, `holdout_score`, `system_prompt`, … | +| `baseline_examples`, `best_examples` | Per-example validation results (question, expected, answer, score, feedback) | +| `metric_deltas`, `param_suggestions`, `failure_clusters`, `suggestions` | What changed and why | +| `score_history`, `prompt_history` | Per-round curves and epoch learnings | +| `settings`, `dataset_sizes`, `early_stop_reason`, `duration_seconds` | How the run was configured and ended | +| `best_prompt`, `best_params`, `best_few_shot_examples` | Shortcuts | + +Methods: `summary()` (text), `to_dict()` (JSON), and +`apply(version="v2_fit", agent_dir=None, *, force=False, min_improvement=0.0, +require_generalization=True, max_generalization_gap=0.15)`, which writes +`prompts.json`, `runtime_config.json` and an audit line to +`fit_history.jsonl` — or refuses a result that did not improve or does not +generalise. diff --git a/docs/optimization/reports.md b/docs/optimization/reports.md new file mode 100644 index 0000000..346baa6 --- /dev/null +++ b/docs/optimization/reports.md @@ -0,0 +1,91 @@ +# Reports + +Every optimization run can produce an interactive HTML report (HolySheet, with +a self-contained fallback page when HolySheet is not installed) and a JSON +file with the same data. The report answers three questions, in this order: +*did it help*, *what changed*, and *why*. + +## Generating one + +```python +from agentomatic.optimize import generate_fit_report + +before = agent.evaluate(data.test) +history = agent.fit(data, epochs=2) +after = agent.evaluate(data.test) + +path = generate_fit_report( + history, # History, PromptFitResult, a list of them, or the agent + output_path="reports/support.html", # + reports/support.json + baseline_eval=before, # EvaluationReports on the SAME examples + final_eval=after, + eval_dataset=data.test, # questions + expected answers for the table + dataset_stats=data.metadata.get("augment_stats"), + model_name="Qwen3.5-9B-MLX-4bit", + optimizer_name="gepa_like", +) +``` + +| Argument | Meaning | +| --- | --- | +| `result` | The `History` returned by `fit()` (recommended — all epochs plus loss curves), a `PromptFitResult`, a list of per-epoch results, or the fitted agent. Multiple epochs are merged so the report spans the **original** prompt to the **final** one. | +| `output_path` | HTML path (default `.optimize//fit_report_.html`); the JSON sidecar is written next to it. | +| `baseline_eval`, `final_eval` | `EvaluationReport`s before / after, for the per-example table. | +| `baseline_eval_scores`, `eval_scores` | Scores-only alternative to the reports. | +| `eval_dataset` | The evaluated examples. | +| `keras_history` | `History.history`; taken from `result` when it is a `History`. | +| `dataset_stats`, `dataset_sizes` | Augmentation stats and split sizes. | +| `run_config`, `stack_name`, `model_name`, `optimizer_name` | Header and settings details. | + +`train_and_report` / scaffolded `train.py` scripts produce this report +automatically (`reports/train_.html`). `PromptFitter(auto_report=True)` +additionally writes one per fit under `experiment_dir`. + +## What is in it + +| Section | Shows | +| --- | --- | +| **Verdict** | Validation initial → final, holdout gate, test metrics before → after, a one-line conclusion, and the stop reason. | +| **Test scoreboard** | Every compiled metric on the untouched test split, before / after / Δ. | +| **What changed in the prompt** | The initial → final diff (long lines wrapped so changed sentences stand out), both full prompts side by side, and every accepted change in order with why it was accepted. | +| **Epochs** | Start / best / held-out / candidates / stop reason per epoch. | +| **All candidates** | Every proposed candidate: epoch, round, minibatch / validation / held-out scores, confidence, **decision** and **reason**; the diff of each distinct candidate prompt. | +| **Examples — before vs after** | Per example: question, expected answer, answer before and after, scores before and after, per-metric Δ, and the judge's rationale. Uses the test split when `baseline_eval`/`final_eval` are given, otherwise the validation examples the fitter scored. | +| **Data & settings** | Split sizes, augmentation stats, and every fitter knob used. | +| **Run configuration**, **Recommendations** | Model, optimizer, suggestions, parameter changes, metric deltas, a rollout recommendation. | +| **Curves** | Best score / loss per round, Keras `loss` / `val_loss` and metrics per epoch, the held-out evaluation. | +| **Prompt evolution** | Per round: score, whether accepted, the diff vs the previous version, and the epoch learnings (what worked, what failed, next focus, judge insights). | +| **Failure analysis** | Failure clusters from the train split and APO critiques, when present. | + +## Reading it + +* **Start at the verdict.** A validation gain with no test gain is the + signature of overfitting; a test gain that matches the validation gain is + the result you want. +* **Check the reasons.** Many `not significant` rejections → validation is + too small or the judge too noisy. Many `does not transfer` → the optimizer + is fitting the validation examples. Many `duplicate` → the rewrite model + keeps proposing the current prompt (try a stronger `rewrite_model` or a + different `optimizer`). See [Generalization](generalization.md#practical-guidance). +* **Read the examples table.** It shows *how* answers changed, which a mean + score hides. + +## The JSON sidecar + +`.json` contains `initial_prompt`, `final_prompt`, `prompt_changed`, +`validation` and `holdout_gate` (initial / final), `test` (before / after +scores), `test_examples` (the per-example rows), `epochs` (every +`PromptFitResult.to_dict()`), `keras_history` and `dataset_stats` — for CI +checks, dashboards or diffing runs. + +## Evaluation-only reports + +```python +from agentomatic.optimize import generate_eval_report + +report = agent.evaluate(data.test) +generate_eval_report(report, output_path="reports/eval.html") +``` + +Scaffolded projects ship `agents/NAME/eval.py`, which does this from the +command line. diff --git a/examples/prompt_optimization/README.md b/examples/prompt_optimization/README.md new file mode 100644 index 0000000..c14f99f --- /dev/null +++ b/examples/prompt_optimization/README.md @@ -0,0 +1,82 @@ +# Prompt optimization examples + +Runnable, commented examples of Agentomatic's prompt optimization, from "score +some answers" to a full Keras-style training run with an HTML report. They all +optimize the same small `SupportAgent` (`agent.py`), which answers Acme Cloud +customer questions from a policy knowledge base, on `datasets/support.jsonl`. + +| Script | Level | What you learn | +| --- | --- | --- | +| [`metrics_and_judges.py`](metrics_and_judges.py) | building blocks | Every metric family, LLM judges (single, per-dimension, panel), composite objectives, adapters between the two metric protocols, losses. Scores three fixed answers; no optimization. | +| [`data_augmentation.py`](data_augmentation.py) | building blocks | `prepare_dataset(augment=True)`: every strategy and option, leak protection, `augment_stats`. | +| [`anti_overfitting.py`](anti_overfitting.py) | mechanics | The acceptance rules on hand-made numbers, then a fit where a prompt that memorizes the validation answers is vetoed and a general one is kept. | +| [`low_level_prompt_fitter.py`](low_level_prompt_fitter.py) | low level | `PromptFitter` with every knob explicit: datasets, composite metric, search space, inner-loop callbacks, acceptance rules, before/after test evaluation, report. | +| [`train.py`](train.py) | high level | The class-agent workflow: `load_data` → `prepare_dataset` → metrics → loss → `PromptFitterBridge` → `compile` → `fit` → test evaluation → `generate_fit_report`. | + +Start with `metrics_and_judges.py` if you are choosing metrics, `train.py` if +you want the whole loop, and `low_level_prompt_fitter.py` when you need control +over every step. In a scaffolded project (`agentomatic init NAME --template +basic`), `agents/NAME/train.py` calls `train_and_report(...)`, which runs this +same loop from the project's stack configuration. + +## Running them + +Each example talks to an OpenAI-compatible model server. oMLX is the default; +llama.cpp, vLLM, LM Studio or Ollama's `/v1` work the same way. + +```bash +omlx serve --model Qwen3.5-9B-MLX-4bit # http://127.0.0.1:8000/v1 + +python examples/prompt_optimization/metrics_and_judges.py +python examples/prompt_optimization/data_augmentation.py --n-examples 40 +python examples/prompt_optimization/anti_overfitting.py +python examples/prompt_optimization/low_level_prompt_fitter.py --trials 8 +python examples/prompt_optimization/train.py --epochs 2 --trials 6 [--augment] +``` + +Shared flags (or environment variables), defined in `common.py`: + +| Flag | Environment variable | Default | +| --- | --- | --- | +| `--base-url` | `OMLX_BASE_URL` | `http://127.0.0.1:8000/v1` | +| `--api-key` | `OMLX_API_KEY` | `local` | +| `--model` | `AGENTOMATIC_LOCAL_MODEL` | `Qwen3.5-9B-MLX-4bit` | +| `--judge-model` | `EXAMPLE_JUDGE_MODEL` | same as `--model` | +| `--rewrite-model` | `EXAMPLE_REWRITE_MODEL` | same as `--model` | +| `--out-dir` | `EXAMPLE_OUT_DIR` | `examples/prompt_optimization/out` | +| `--log-level` | `AGENTOMATIC_LOG_LEVEL` | `INFO` | + +`train.py` writes `train_report.html` (+ `train_report.json`) and +`train_summary.json` to `--out-dir`; `low_level_prompt_fitter.py` writes +`low_level_report.html` and `low_level_result.json`. + +## The data + +`datasets/support.jsonl` has one JSON object per line: + +```json +{"id": "refund_2", "split": "validation", + "input": {"current_query": "I bought an annual plan last week. Can I still get my money back?"}, + "expected_output": {"response": "Yes. Annual plans can be refunded within 30 days of purchase."}, + "metadata": {"must_include": ["30 days"], "topic": "refund"}} +``` + +The four splits have four jobs, and keeping them apart is what stops the +optimizer from overfitting: + +| Split | Rows | Role | +| --- | --- | --- | +| `train` | 10 | What the optimizer **learns from**: failure analysis, judge feedback, few-shot demos. Augmentation only grows this split. | +| `validation` | 6 | What candidate prompts are **scored and selected on**. | +| `holdout` | 4 | A **veto-only gate**: a candidate that wins on validation but does not carry over here is rejected. Optional; without it a slice of validation is reserved. | +| `test` | 6 | **Never used by optimization.** Evaluate it before and after `fit()` for an unbiased number. | + +`metadata.must_include` feeds the deterministic `facts` metric +(`common.must_include_score`); any metadata you put there reaches class-agent +metrics during `fit()`, `evaluate()` and inside composite metrics. + +## Testing without a model server + +`tests/test_prompt_optimization_examples.py` runs every script end to end +against `tests/fake_openai_server.py`, a deterministic stand-in that plays the +agent, the judge, the prompt rewriter and the augmenter. diff --git a/examples/prompt_optimization/agent.py b/examples/prompt_optimization/agent.py new file mode 100644 index 0000000..96da5d6 --- /dev/null +++ b/examples/prompt_optimization/agent.py @@ -0,0 +1,140 @@ +"""Support agent shared by every prompt-optimization example. + +A small but realistic class agent: it retrieves the relevant policy snippets +from an in-memory knowledge base, then asks the model to answer with them. + +What the optimizer tunes is the **system prompt** (and, optionally, few-shot +examples and model parameters). The agent reads it through +:meth:`~agentomatic.agents.BaseGraphAgent.resolve_system_prompt`, which is the +hook every optimization path uses to inject a candidate prompt: + +1. ``system_prompt_override`` on the request (how candidates are scored), +2. ``compiled_config["system_prompt"]`` (what ``fit()`` keeps as the best), +3. the ``system_prompt`` class attribute below (the baseline). + +An agent that hardcodes its prompt inside a node cannot be optimized — always +go through ``resolve_system_prompt``. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from typing import Any + +from agentomatic.agents import BaseGraphAgent + +#: The facts the agent may use. Answers are only "right" if grounded here, so +#: a metric can check them deterministically (see ``datasets/support.jsonl``). +KNOWLEDGE_BASE: dict[str, str] = { + "refund": "Refunds are available within 30 days of purchase for annual plans only.", + "password": "Reset a password from Settings > Security > Reset password; links expire " + "after 15 minutes.", + "outage": "Service status and incident updates are published at status.acme.example.", + "export": "Workspace owners can export all data as CSV or JSON from Settings > Data.", + "seats": "Seats can be added any time; they are billed pro rata until the renewal date.", + "sso": "Single sign-on (SAML) is available on the Enterprise plan only.", + "support hours": "Human support is available Monday to Friday, 08:00-18:00 CET.", + "delete": "Deleted workspaces can be restored by support for 14 days, then are purged.", + "api": "The public API allows 600 requests per minute per workspace.", + "invoice": "Invoices are emailed to the billing contact and listed under Billing > Invoices.", +} + +#: Keyword patterns that route a question to a topic (a stand-in for a real +#: retriever — swap in a vector store via ``connections.py`` in a project). +TOPIC_PATTERNS: dict[str, str] = { + "refund": r"\brefund|money back|\bcancel", + "password": r"\bpassword", + "outage": r"\boutage|\bdown\b|\bstatus\b|\bincident", + "export": r"\bexport|\bdownload|\bbackup|\bcsv\b|\bjson\b", + "seats": r"\bseats?\b|\badd\b.*\busers?\b", + "sso": r"\bsso\b|\bsaml\b|single sign-on", + "support hours": r"support hours|reach support|\bsaturday|\bsunday|\bweekend", + "delete": r"\bdelet|\brestor|\brecover", + "api": r"\bapi\b|rate limit|\b429\b", + "invoice": r"\binvoice|\breceipt", +} + + +@dataclass +class SupportState: + """Per-run state of :class:`SupportAgent`.""" + + question: str = "" + snippets: list[str] = field(default_factory=list) + output: dict[str, Any] = field(default_factory=dict) + + +class SupportAgent(BaseGraphAgent[SupportState]): + """Answer customer questions from the knowledge base. + + Args: + llm: Any LangChain-style chat model (``.invoke(messages)``). ``None`` + makes the agent echo its retrieved snippets, which keeps the + examples importable without a model server. + """ + + agent_name = "support_agent" + agent_description = "Answers Acme Cloud customer questions from policy snippets" + + #: Baseline prompt. Deliberately vague so there is something to improve. + system_prompt = "You are a support assistant for Acme Cloud. Help the customer." + + def __init__(self, *, llm: Any = None) -> None: + super().__init__() + self.llm = llm + + # --- graph wiring ----------------------------------------------------- + + def build_graph(self) -> Any: + g = self.new_graph() + g.add_node("retrieve", self.retrieve) + g.add_node("answer", self.answer) + g.set_entry_point("retrieve") + g.add_edge("retrieve", "answer") + g.set_finish_point("answer") + return g.compile() + + # --- nodes -------------------------------------------------------------- + + def retrieve(self, state: SupportState) -> SupportState: + """Pick the knowledge-base snippets whose topic the question mentions.""" + question = state.question.lower() + state.snippets = [ + KNOWLEDGE_BASE[topic] + for topic, pattern in TOPIC_PATTERNS.items() + if re.search(pattern, question) + ] + return state + + def answer(self, state: SupportState) -> SupportState: + """Answer with the model, grounded in the retrieved snippets.""" + # The optimizer's candidate prompt arrives through this call. + prompt = self.resolve_system_prompt(default=self.system_prompt) + context = "\n".join(f"- {s}" for s in state.snippets) or "- (no matching policy)" + if self.llm is None: + text = " ".join(state.snippets) or "I could not find a matching policy." + else: + reply = self.llm.invoke( + [ + {"role": "system", "content": prompt}, + { + "role": "user", + "content": f"Policy snippets:\n{context}\n\nQuestion: {state.question}", + }, + ] + ) + text = str(getattr(reply, "content", reply)).strip() + state.output = {"response": text, "sources": list(state.snippets)} + return state + + # --- I/O contract ------------------------------------------------------- + + def input_to_state(self, data: dict[str, Any]) -> SupportState: + # REST sends ``query``; the platform normalises it to ``current_query``. + # Datasets may use ``question`` — accept all three. + question = data.get("current_query") or data.get("question") or data.get("query") or "" + return SupportState(question=str(question)) + + def state_to_output(self, state: SupportState) -> dict[str, Any]: + return state.output diff --git a/examples/prompt_optimization/anti_overfitting.py b/examples/prompt_optimization/anti_overfitting.py new file mode 100644 index 0000000..4f6b55b --- /dev/null +++ b/examples/prompt_optimization/anti_overfitting.py @@ -0,0 +1,212 @@ +#!/usr/bin/env python3 +"""Anti-overfitting in prompt optimization — watch a memorizing prompt get rejected. + +An optimizer that is rewarded only for its validation score will happily +"improve" a prompt by pasting the validation answers into it. That prompt +scores perfectly on validation and no better anywhere else. Agentomatic's +fitter guards against it with separate data roles and acceptance rules: + +* TRAIN — what the optimizer learns from (failures, judge feedback); +* VALIDATION — what candidates are selected on; a win must be *significant* + (paired bootstrap over the validation examples, ``min_confidence``); +* HOLDOUT — a veto-only gate: the candidate must not regress there + (``holdout_tolerance``), must not open a validation/holdout gap + (``max_generalization_gap``), and must carry a share of its validation + gain over (``min_transfer_ratio``) — the rule memorisation fails; +* TEST — never used by the fitter; measure it yourself afterwards. + +Part 1 applies the rules to hand-made numbers (no model needed). +Part 2 runs ``PromptFitter`` with two scripted candidates — one that memorizes +the validation answers, one that states a general instruction — and shows +which one survives, and what each would have scored on the test split. + +Run:: + + python examples/prompt_optimization/anti_overfitting.py +""" + +from __future__ import annotations + +import asyncio +import dataclasses +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from agent import SupportAgent + +from common import ( + DATASET, + build_parser, + make_llm, + must_include_score, + settings_from, + setup_logging, +) + +GENERAL_PROMPT = ( + "You are a precise Acme Cloud support assistant. Quote the relevant policy snippet " + "verbatim, then answer in one short sentence. Never invent policy." +) + + +def part1_rules() -> None: + """The acceptance rules on hand-made numbers.""" + from agentomatic.optimize import check_generalization, paired_improvement_confidence + + print("── Part 1: the acceptance rules ──") + print( + "incumbent: validation 0.50, holdout 0.50; rules: gap 0.15, tolerance 0.02, transfer 0.25" + ) + scenarios = { + # name: (candidate validation, candidate holdout) + "general improvement": (0.80, 0.75), + "memorised validation": (1.00, 0.50), + "gain does not transfer": (0.62, 0.52), + "holdout regression": (0.52, 0.45), + "gap blows up": (0.95, 0.60), + } + for name, (val, hold) in scenarios.items(): + check = check_generalization( + fit_score=val, + holdout_score=hold, + baseline_fit=0.50, + baseline_holdout=0.50, + max_gap=0.15, + holdout_tolerance=0.02, + min_transfer_ratio=0.25, + ) + verdict = "ACCEPT" if check.ok else "REJECT" + print(f" {name:<22} val {val:.2f} holdout {hold:.2f} → {verdict}\n {check.reason}") + + # Significance: per-example validation scores of incumbent vs candidate. + incumbent = [0.0, 0.0, 1.0, 1.0, 0.0, 1.0] + lucky = [1.0, 0.0, 1.0, 1.0, 0.0, 1.0] # one example flipped — could be judge noise + solid = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0] # every failure fixed + for name, cand in (("one example flipped", lucky), ("every failure fixed", solid)): + p = paired_improvement_confidence(cand, incumbent) + verdict = "ACCEPT" if p >= 0.8 else "REJECT (not significant)" + print(f" {name:<22} P(candidate > incumbent) = {p:.2f} → {verdict}") + + +class ScriptedCandidates: + """A custom optimizer: any object with ``name`` and async ``propose(...)``. + + Real optimizers ("rewrite", "gepa_like", …) generate candidates from the + train failures; this one proposes a fixed prompt per round so the outcome + is easy to follow. + """ + + name = "scripted" + + def __init__(self, rounds: list[tuple[str, str]]) -> None: + self.rounds = rounds + + async def propose( + self, + current_config: Any, + eval_results: list[dict[str, Any]], + dataset_sample: list[dict[str, Any]], + search_space: Any, + iteration: int = 0, + context: Any = None, + ) -> list[Any]: + from agentomatic.optimize import PromptCandidate + + if iteration >= len(self.rounds): + return [] + name, prompt = self.rounds[iteration] + # Change the prompt, keep every other setting of the incumbent. + config = dataclasses.replace(current_config, system_prompt=prompt) + return [PromptCandidate(name=name, config=config, source=self.name)] + + +async def part2_fit(settings: Any) -> int: + """A real fit where the memorizing candidate is vetoed.""" + from agentomatic.agents import CallableMetric + from agentomatic.optimize import Dataset, PromptFitter, load_data + + print("\n── Part 2: a fit with a memorizing and a general candidate ──") + data = load_data(DATASET) + + def points(examples: list[Any]) -> Dataset: + return Dataset(points=[e.to_datapoint() for e in examples]) + + agent = SupportAgent(llm=make_llm(settings)) + # The overfit prompt: the VALIDATION questions with their answers. + memorized = "\n".join( + f"Q: {e.input['current_query']}\nA: {e.expected_output['response']}" + for e in data.validation + ) + memorizing_prompt = ( + f"{agent.system_prompt}\nAnswer these questions exactly as follows:\n{memorized}" + ) + + facts = CallableMetric("facts", must_include_score) # deterministic → reproducible + fitter = PromptFitter( + agent=agent.agent_name, + local_agent=agent, + optimizer=ScriptedCandidates( + [("memorize_validation", memorizing_prompt), ("general_instruction", GENERAL_PROMPT)] + ), + task_model=settings.task_spec, + rewrite_model=settings.rewrite_spec, + llm_base_url=settings.base_url, + llm_api_key=settings.api_key, + baseline_system_prompt=agent.system_prompt, + max_trials=8, # 2 rounds (one scripted candidate each) + patience=3, + min_absolute_improvement=0.01, + min_confidence=0.8, + max_generalization_gap=0.15, + holdout_tolerance=0.02, + min_transfer_ratio=0.25, + auto_report=False, + drain_seconds=0, + experiment_dir=str(settings.out_dir / ".fit_anti_overfitting"), + ) + result = await fitter.fit( + points(data.train), points(data.validation), facts, testset=points(data.holdout) + ) + + print(" candidate validation holdout decision reason") + for trial in result.trials: + if trial.get("phase") != "full_val": + continue + print( + f" {trial['name']:<22} {trial['score']:.2f} " + f"{trial.get('holdout_score') or 0.0:.2f} {trial['decision']:<10} " + f"{str(trial.get('reason'))[:70]}" + ) + + # What each prompt would have scored on the untouched TEST split. + async def test_score(prompt: str) -> float: + agent.compiled_config["system_prompt"] = prompt + report = await asyncio.to_thread(agent.evaluate, data.test, [facts]) + return float(report.scores["facts"]) + + print(" test split (never seen by the fitter):") + for name, prompt in ( + ("baseline", agent.system_prompt), + ("memorize_validation", memorizing_prompt), + ("general_instruction", GENERAL_PROMPT), + ): + print(f" {name:<22} facts = {await test_score(prompt):.2f}") + kept = result.best_prompt == GENERAL_PROMPT + print(f" kept prompt: {'general_instruction' if kept else 'something else'}") + return 0 if kept else 1 + + +def main(argv: list[str] | None = None) -> int: + """Run both parts.""" + args = build_parser(__doc__.splitlines()[0]).parse_args(argv) + settings = settings_from(args) + setup_logging(settings.log_level) + part1_rules() + return asyncio.run(part2_fit(settings)) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/prompt_optimization/common.py b/examples/prompt_optimization/common.py new file mode 100644 index 0000000..dea5ac1 --- /dev/null +++ b/examples/prompt_optimization/common.py @@ -0,0 +1,177 @@ +"""Shared plumbing for the prompt-optimization examples. + +Every example accepts the same connection flags (or environment variables), +so you point them all at your model server once: + +===================== =========================== ================================ +Flag Environment variable Default +===================== =========================== ================================ +``--base-url`` ``OMLX_BASE_URL`` ``http://127.0.0.1:8000/v1`` +``--api-key`` ``OMLX_API_KEY`` ``local`` +``--model`` ``AGENTOMATIC_LOCAL_MODEL`` ``Qwen3.5-9B-MLX-4bit`` +``--judge-model`` ``EXAMPLE_JUDGE_MODEL`` same as ``--model`` +``--rewrite-model`` ``EXAMPLE_REWRITE_MODEL`` same as ``--model`` +``--out-dir`` ``EXAMPLE_OUT_DIR`` ``examples/prompt_optimization/out`` +``--log-level`` ``AGENTOMATIC_LOG_LEVEL`` ``INFO`` +===================== =========================== ================================ + +Any OpenAI-compatible server works (oMLX, llama.cpp, vLLM, LM Studio, Ollama's +``/v1``): only ``--base-url`` and ``--model`` change. Model *specs* passed to +the optimizer are ``"/"`` strings — ``omlx/…`` routes to the +OpenAI-compatible ``--base-url``; ``openai/…``, ``ollama/…``, ``gemini/…`` and +``litellm/…`` are the other providers. +""" + +from __future__ import annotations + +import argparse +import os +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +HERE = Path(__file__).resolve().parent +DATASET = HERE / "datasets" / "support.jsonl" + +# Let ``python examples/prompt_optimization/ + - + diff --git a/src/agentomatic/tasks/__init__.py b/src/agentomatic/tasks/__init__.py index f7ed193..0720652 100644 --- a/src/agentomatic/tasks/__init__.py +++ b/src/agentomatic/tasks/__init__.py @@ -43,7 +43,7 @@ create_event_log, register_event_log_provider, ) -from .manager import TaskInputValidationError, TaskManager +from .manager import TargetFailedError, TaskInputValidationError, TaskManager from .models import ( TargetType, TaskEvent, @@ -73,6 +73,7 @@ "InMemoryTaskStore", "NullTaskEventLog", "SQLAlchemyTaskStore", + "TargetFailedError", "TargetNotFoundError", "TargetType", "TaskContext", diff --git a/src/agentomatic/tasks/dispatchers.py b/src/agentomatic/tasks/dispatchers.py index 55c7a83..905404e 100644 --- a/src/agentomatic/tasks/dispatchers.py +++ b/src/agentomatic/tasks/dispatchers.py @@ -14,7 +14,7 @@ from loguru import logger from pydantic import ValidationError -from .manager import TaskInputValidationError +from .manager import TargetFailedError, TaskInputValidationError if TYPE_CHECKING: from agentomatic.core.registry import AgentRegistry @@ -278,7 +278,12 @@ async def run(target: str, payload: Any, ctx: TaskContext) -> Any: result = await endpoint.call(payload) else: result = await endpoint.handle(request) - return _to_jsonable(result) + jsonable = _to_jsonable(result) + if isinstance(jsonable, dict) and jsonable.get("ok") is False and "results" in jsonable: + raise TargetFailedError( + f"Endpoint '{target}': no upstream call succeeded", result=jsonable + ) + return jsonable return run @@ -386,6 +391,13 @@ async def checkpoint_cb(step_name: str, index: int, sub_result: Any) -> None: jsonable = _to_jsonable(result) if isinstance(jsonable, dict) and partial: jsonable.setdefault("checkpoints", partial) + # ``partial`` (some steps failed under on_error: continue) is a + # completed run; ``failed`` is not. + if str(getattr(result, "status", "")) == "failed": + raise TargetFailedError( + f"Pipeline '{target}' failed: {getattr(result, 'error', None) or 'see steps'}", + result=jsonable, + ) return jsonable return run diff --git a/src/agentomatic/tasks/manager.py b/src/agentomatic/tasks/manager.py index ec18a66..bbc057c 100644 --- a/src/agentomatic/tasks/manager.py +++ b/src/agentomatic/tasks/manager.py @@ -55,6 +55,24 @@ class TaskInputValidationError(ValueError): """Raised when a resource rejects a task payload before it is queued.""" +class TargetFailedError(RuntimeError): + """The target ran to completion but reported a failure. + + A pipeline whose status is ``failed`` or an endpoint whose every upstream + call failed returns normally, so its task used to be marked ``succeeded``. + Dispatchers raise this instead; the task is marked ``failed`` and keeps + the target's own result (step errors, upstream responses) for debugging. + + Args: + message: What failed, shown as the task's error. + result: The JSON-safe result the target returned. + """ + + def __init__(self, message: str, result: Any = None) -> None: + super().__init__(message) + self.result = result + + class TaskManager: """Run and track platform resources as uniform, cancellable tasks. @@ -390,6 +408,17 @@ async def _run(self, record: TaskRecord, batch_concurrency: int | None) -> None: if delay > 0: await asyncio.sleep(delay) + if isinstance(last_error, TargetFailedError): + # The target's own report (e.g. which pipeline step failed and + # why) — already what its HTTP route returns, so not sanitised. + logger.error(f"Task {record.id} failed: {last_error}") + await self._finalize( + record, + TaskStatus.FAILED, + result=last_error.result, + error=str(last_error), + ) + return logger.exception(f"Task {record.id} failed after {record.attempts} attempts") # Raw exception text routinely carries DSNs/paths, and this # record is served verbatim by GET /tasks/{id}, /result and the @@ -444,6 +473,8 @@ async def run_item(index: int, payload: Any) -> None: raise asyncio.CancelledError try: results[index] = await dispatcher(record.target, payload, ctx) + except TargetFailedError as exc: + results[index] = {"error": str(exc), "result": exc.result} except Exception as exc: # noqa: BLE001 - collect per-item errors # Per-item errors are returned to the caller too. results[index] = { diff --git a/src/agentomatic/tasks/progress.py b/src/agentomatic/tasks/progress.py index bae9f7d..79aa7f3 100644 --- a/src/agentomatic/tasks/progress.py +++ b/src/agentomatic/tasks/progress.py @@ -76,13 +76,35 @@ def report_stage_sync( total: int | None = None, message: str = "", ) -> None: - """Schedule :func:`report_stage` from a sync graph node (no-op if no loop/ctx).""" + """Schedule :func:`report_stage` from a sync graph node (no-op if no loop/ctx). + + Sync nodes run in a worker thread (see + :func:`~agentomatic.async_utils.run_in_worker_thread`), where there is no + running loop. The report is then handed to the loop serving the task, so + progress keeps flowing to ``GET /tasks/{id}/events``. + """ ctx = get_task_context() if ctx is None: return try: loop = asyncio.get_running_loop() except RuntimeError: + from agentomatic.async_utils import owner_loop + + origin = owner_loop() + if origin is None or origin.is_closed(): + return + + async def _forward() -> None: + await ctx.report( + stage=stage, + percent=percent, + current=current, + total=total, + message=message or stage, + ) + + asyncio.run_coroutine_threadsafe(_forward(), origin) return task = loop.create_task( report_stage( diff --git a/tests/fake_openai_server.py b/tests/fake_openai_server.py new file mode 100644 index 0000000..ca83f18 --- /dev/null +++ b/tests/fake_openai_server.py @@ -0,0 +1,212 @@ +"""A deterministic OpenAI-compatible model server for offline optimization tests. + +It plays every role the optimization stack sends to an LLM, recognising each +by its prompt: + +* **agent** (``Policy snippets: … Question: …``) — answers *well* (quotes the + snippets) only when the system prompt asks it to quote policy, and vaguely + otherwise. A better prompt therefore measurably scores higher, which is + what makes an optimization run meaningful. A prompt that lists + ``Q: `` / ``A: `` pairs is followed literally for those + exact questions — the memorising prompt an overfitting optimizer produces. +* **judge** (``You are an expert evaluation judge``) — returns the + ``overall_score`` JSON ``LocalJudgeMetric`` expects, scored by how many + words of the expected answer the response recalls. +* **rewriter** (asks for an improved system prompt) — returns + :data:`IMPROVED_PROMPT` after a ``---`` line. +* **augmenter** (``dataset augmentation expert``) — returns a JSON array of + paraphrased seed queries. + +Every request body is recorded on :attr:`FakeOpenAIServer.requests`. +""" + +from __future__ import annotations + +import itertools +import json +import re +import socket +import threading +import time +from typing import Any + +from fastapi import FastAPI, Request + +#: The prompt the fake rewriter proposes. Contains the trigger word ("quote"). +IMPROVED_PROMPT = ( + "You are a precise Acme Cloud support assistant. Quote the relevant policy " + "snippet verbatim, then answer in one short sentence. Never invent policy." +) + +_VAGUE_ANSWER = "Thanks for reaching out! Our team is happy to help with that." +_AUGMENT_COUNTER = itertools.count(1) + + +def _words(text: str) -> set[str]: + return {w for w in re.findall(r"[a-z0-9>:.-]+", text.lower()) if len(w) > 3} + + +def _section(prompt: str, header: str) -> str: + match = re.search(rf"## {re.escape(header)}[^\n]*\n(.*?)(?:\n## |\Z)", prompt, re.S) + return match.group(1).strip() if match else "" + + +def _agent_reply(system: str, user: str) -> str: + question = user.rsplit("Question:", 1)[-1].strip() + memorised = dict(re.findall(r"^Q: (.+?)\s*\nA: (.+?)\s*$", system, re.M)) + if question in memorised: + return memorised[question] + snippets = [line[2:].strip() for line in user.splitlines() if line.startswith("- ")] + if "quote" in system.lower() and snippets: + return " ".join(snippets) + return _VAGUE_ANSWER + + +def _judge_reply(prompt: str) -> str: + response = _section(prompt, "AI Response") + # A dataset reference nests its own "## Expected answer" section. + expected = _section(prompt, "Expected answer") or _section( + prompt, "Expected / Quality Reference" + ) + wanted = _words(expected) + score = round(len(wanted & _words(response)) / len(wanted), 2) if wanted else 0.5 + return json.dumps( + { + "overall_score": score, + "feedback": "Grounded in policy." if score >= 0.5 else "Too vague; cite the policy.", + "motivation": f"Recalled {score:.0%} of the reference facts.", + "what_worked": ["polite tone"], + "what_failed": [] if score >= 0.5 else ["does not state the policy"], + "improvement_hints": ["Quote the relevant policy snippet verbatim."], + "dimensions": {"correctness": score, "completeness": score, "relevance": score}, + } + ) + + +def _augment_reply(prompt: str) -> str: + """Answer a one-seed augmentation prompt with ``k`` distinct variations.""" + seed_match = re.search(r"## Seed example\n(\{.*?\})\n", prompt, re.S) + seed = json.loads(seed_match.group(1)) if seed_match else {"query": "question"} + count = re.search(r"Write (\d+) new", prompt) + k = int(count.group(1)) if count else 1 + keep = "must stay exactly the seed's expected_answer" in prompt + query = str(seed.get("query", "")).rstrip("?") + out = [] + for i in range(k): + n = next(_AUGMENT_COUNTER) + out.append( + { + "query": f"Variant {n}: {query.lower()}?", + "expected_answer": seed.get("expected_answer", "") + if keep + else f"Generated answer {n}", + } + ) + return json.dumps(out) + + +def respond(messages: list[dict[str, Any]]) -> str: + """Return the fake model's reply for one chat-completions request.""" + system = " ".join(str(m.get("content", "")) for m in messages if m.get("role") == "system") + user = str(messages[-1].get("content", "")) if messages else "" + text = f"{system}\n{user}" + if "Policy snippets:" in user: + return _agent_reply(system, user) + if "You are an expert evaluation judge" in text: + return _judge_reply(text) + if "dataset augmentation expert" in text: + return _augment_reply(text) + if "system prompt" in text.lower() and ("improv" in text.lower() or "rewrite" in text.lower()): + return f"Here is the improved prompt.\n---\n{IMPROVED_PROMPT}" + if "json" in text.lower(): + return json.dumps({"score": 0.5, "reasoning": "fake"}) + return "OK" + + +def _build_app(server: FakeOpenAIServer) -> FastAPI: + app = FastAPI() + + @app.get("/v1/models") + async def models() -> dict[str, Any]: + return {"data": [{"id": "fake-model", "object": "model"}]} + + @app.post("/v1/chat/completions") + async def completions(request: Request) -> dict[str, Any]: + body = await request.json() + server.requests.append(body) + content = respond(body.get("messages") or []) + return { + "id": "chatcmpl-fake", + "object": "chat.completion", + "created": int(time.time()), + "model": body.get("model", "fake-model"), + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": content}, + "finish_reason": "stop", + } + ], + "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}, + } + + return app + + +class FakeOpenAIServer: + """Run the fake model server on a free local port in a background thread. + + Usage:: + + with FakeOpenAIServer() as server: + base_url = server.base_url # "http://127.0.0.1:/v1" + """ + + def __init__(self) -> None: + self.requests: list[dict[str, Any]] = [] + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + self.port = sock.getsockname()[1] + self.base_url = f"http://127.0.0.1:{self.port}/v1" + self._server: Any = None + self._thread: threading.Thread | None = None + + def roles(self) -> dict[str, int]: + """Count recorded requests per role (agent / judge / rewrite / augment).""" + counts = {"agent": 0, "judge": 0, "rewrite": 0, "augment": 0, "other": 0} + for body in self.requests: + reply = respond(body.get("messages") or []) + text = json.dumps(body.get("messages") or []) + if "Policy snippets:" in text: + counts["agent"] += 1 + elif "expert evaluation judge" in text: + counts["judge"] += 1 + elif "dataset augmentation expert" in text: + counts["augment"] += 1 + elif reply.endswith(IMPROVED_PROMPT): + counts["rewrite"] += 1 + else: + counts["other"] += 1 + return counts + + def __enter__(self) -> FakeOpenAIServer: + import uvicorn + + config = uvicorn.Config( + _build_app(self), host="127.0.0.1", port=self.port, log_level="warning" + ) + self._server = uvicorn.Server(config) + self._thread = threading.Thread(target=self._server.run, daemon=True) + self._thread.start() + deadline = time.monotonic() + 10 + while not self._server.started: + if time.monotonic() > deadline: + raise RuntimeError("fake OpenAI server did not start") + time.sleep(0.02) + return self + + def __exit__(self, *exc: object) -> None: + if self._server is not None: + self._server.should_exit = True + if self._thread is not None: + self._thread.join(timeout=5) diff --git a/tests/test_all_methods_extensive.py b/tests/test_all_methods_extensive.py index 226c355..d5e9ab2 100644 --- a/tests/test_all_methods_extensive.py +++ b/tests/test_all_methods_extensive.py @@ -667,15 +667,27 @@ async def evaluate(self, *a, **k): metadata={"evaluation_failed": True}, ) + # A minority component failing counts as 0 at its full weight — it must + # never *raise* the composite (dropping it from the denominator did). comp = CompositeMetric( metrics=[ - OptWeightedMetric("ok", ok, weight=0.5), - OptWeightedMetric("bad", _Fail(criteria="x"), weight=0.5), + OptWeightedMetric("ok", ok, weight=0.8), + OptWeightedMetric("bad", _Fail(criteria="x"), weight=0.2), ] ) r = await comp.evaluate("q", "yes", expected="yes") - assert r.score == pytest.approx(1.0) + assert r.score == pytest.approx(0.8) assert not r.metadata.get("evaluation_failed") + assert r.metadata["failed_components"] == ["bad"] + + # Half the weight failing is not a measurement any more. + half = CompositeMetric( + metrics=[ + OptWeightedMetric("ok", ok, weight=0.5), + OptWeightedMetric("bad", _Fail(criteria="x"), weight=0.5), + ] + ) + assert (await half.evaluate("q", "yes", expected="yes")).metadata.get("evaluation_failed") all_bad = CompositeMetric( metrics=[ @@ -756,7 +768,7 @@ def test_augment_local_judges_wraps_and_extends(self) -> None: assert any(wm.name == "local_judges" for wm in aug.metrics) @pytest.mark.asyncio - async def test_evaluate_config_skips_failed_and_averages(self, monkeypatch) -> None: + async def test_evaluate_config_counts_failed_points_as_zero(self, monkeypatch) -> None: a = _DemoAgent() f = PromptFitter( agent="demo", @@ -792,7 +804,9 @@ async def _eval(query, response, expected=None, context=None): ), metric, ) - assert avg == pytest.approx(1.0) + # A failed evaluation counts as 0.0 — averaging only the points that + # worked let a candidate that breaks the judge on hard inputs win. + assert avg == pytest.approx(0.5) assert any(d.get("error") for d in details) def test_param_suggestions_and_baseline_load(self, tmp_path: Path, monkeypatch) -> None: diff --git a/tests/test_connections.py b/tests/test_connections.py index 32e4052..648c889 100644 --- a/tests/test_connections.py +++ b/tests/test_connections.py @@ -528,6 +528,41 @@ async def call_next(request): assert captured["connections"].get("svc") is not None +async def test_connections_middleware_resolves_a_slug_to_the_agent_scope(): + """URLs may use the slug; the scope is keyed by the registered name.""" + from types import SimpleNamespace + + from agentomatic.core.manifest import AgentManifest, RegisteredAgent + from agentomatic.core.registry import AgentRegistry + from agentomatic.middleware.connections import ConnectionsMiddleware + + register_connections( + "weather_bot", [HttpConnectionConfig(name="svc", base_url="https://svc.test")] + ) + registry = AgentRegistry() + agent = RegisteredAgent( + manifest=AgentManifest(name="weather_bot", slug="weather-bot", description="d") + ) + registry._agents["weather_bot"] = agent # noqa: SLF001 - as discovery does + registry._index_slug("weather_bot", agent) # noqa: SLF001 + mw = ConnectionsMiddleware(app=object(), registry=registry, api_prefix="/api/v1") + + captured = {} + + async def call_next(request): + captured["connections"] = request.state.connections + captured["agent"] = request.state.agent_name + return "ok" + + request = SimpleNamespace( + url=SimpleNamespace(path="/api/v1/weather-bot/chat"), + state=SimpleNamespace(), + ) + await mw.dispatch(request, call_next) + assert captured["agent"] == "weather_bot" + assert captured["connections"].get("svc") is not None + + async def test_connections_middleware_falls_back_to_platform(): from types import SimpleNamespace diff --git a/tests/test_dataset_augmentation.py b/tests/test_dataset_augmentation.py new file mode 100644 index 0000000..3c44b0d --- /dev/null +++ b/tests/test_dataset_augmentation.py @@ -0,0 +1,204 @@ +# pyright: reportMissingParameterType=none +"""``prepare_dataset(augment=True)`` must actually add usable, leak-free rows. + +Reported: "prompt optimization doesn't augment the data". Causes found: + +* one call per strategy asked a local model for dozens of examples under + ``max_tokens=2000``; the reply was cut off and parsed to *zero* rows with + no warning — only a clean, complete JSON array was understood; +* synthetic rows got a fabricated ``{"content": …, "next_action": "Follow up + with stakeholder."}`` label whatever the seed schema was, and lost the + seed's metadata — so every metric scored them wrong; +* rows were only de-duplicated against train seeds (copies of test questions + leaked into train), and without a train split, test rows became seeds; +* ``LLMCaller.configure`` was left set process-wide; ``augmented=True`` was + reported even when nothing was added. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +import pytest + +from agentomatic.agents import AgentDataset, AgentExample +from agentomatic.optimize import load_data, prepare_dataset +from agentomatic.optimize.llm_caller import LLMCaller +from agentomatic.optimize.synthesizer import parse_generated_points + +sys.path.insert(0, str(Path(__file__).parent)) + +from fake_openai_server import FakeOpenAIServer # noqa: E402 + +EXAMPLE_DATASET = ( + Path(__file__).resolve().parents[1] + / "examples" + / "prompt_optimization" + / "datasets" + / "support.jsonl" +) + + +@pytest.fixture(scope="module") +def server(): + with FakeOpenAIServer() as srv: + yield srv + + +class TestTheReportedCall: + def test_the_example_script_call_adds_rows_and_persists(self, server, tmp_path) -> None: + """The exact ``prepare_dataset`` call from the user's training script.""" + dataset = load_data(EXAMPLE_DATASET) + out, written = prepare_dataset( + dataset, + seed_path=EXAMPLE_DATASET, + augment=True, + n_examples=40, + persist=True, + persist_path=tmp_path / "all_augmented.jsonl", + model="omlx/fake-model", + llm_base_url=server.base_url, + llm_api_key=None, + strategies=["expansion", "paraphrase"], + ) + stats = out.metadata["augment_stats"] + assert len(out.examples) == 40 + assert stats["added"] == 14 and out.metadata["augmented"] is True + assert set(stats["by_strategy"]) == {"expansion", "paraphrase"} + assert written == tmp_path / "all_augmented.jsonl" + assert len(written.read_text().splitlines()) == 40 + + def test_only_train_grows(self, server) -> None: + dataset = load_data(EXAMPLE_DATASET) + out, _ = _augment(dataset, server, n_examples=34) + assert len(out.train) == len(dataset.train) + 8 + assert [e.id for e in out.validation] == [e.id for e in dataset.validation] + assert [e.id for e in out.test] == [e.id for e in dataset.test] + assert [e.id for e in out.holdout] == [e.id for e in dataset.holdout] + + def test_no_process_wide_endpoint_side_effect(self, server) -> None: + before = (LLMCaller._default_base_url, LLMCaller._default_api_key) # noqa: SLF001 + _augment(load_data(EXAMPLE_DATASET), server, n_examples=34) + assert (LLMCaller._default_base_url, LLMCaller._default_api_key) == before # noqa: SLF001 + + +class TestRowsMirrorTheSeed: + def test_label_preserving_rows_keep_answer_and_metadata(self, server) -> None: + dataset = load_data(EXAMPLE_DATASET) + out, _ = _augment(dataset, server, n_examples=34, strategies=["paraphrase"]) + seeds = {e.id: e for e in dataset.examples} + new = [e for e in out.examples if e.metadata.get("source") == "augment"] + assert new + for example in new: + seed = seeds[example.metadata["parent_id"]] + assert example.expected_output == seed.expected_output + assert example.metadata["must_include"] == seed.metadata["must_include"] + assert set(example.input) == set(seed.input) + assert example.split == "train" + + def test_generated_answers_use_the_seed_schema(self, server) -> None: + out, _ = _augment( + load_data(EXAMPLE_DATASET), server, n_examples=34, strategies=["expansion"] + ) + new = [e for e in out.examples if e.metadata.get("source") == "augment"] + assert new + for example in new: + assert set(example.expected_output) == {"response"} + assert "next_action" not in json.dumps(example.to_dict()) + assert example.expected_output["response"].startswith("Generated answer") + + +class TestLeakage: + def test_near_duplicates_of_test_questions_are_dropped(self, server) -> None: + train = AgentExample( + id="t", input={"current_query": "How do I reset my password?"}, split="train" + ) + test = AgentExample( + id="x", + input={"current_query": "Variant 999: how do i reset my password?"}, + split="test", + ) + dataset = AgentDataset(examples=[train, test]) + out, _ = _augment(dataset, server, n_examples=4, strategies=["paraphrase"]) + stats = out.metadata["augment_stats"] + # Every variant the fake model writes is a near-copy of the test question. + assert stats["near_duplicates"] >= 1 + assert not [e for e in out.examples if e.metadata.get("source") == "augment"] + + def test_without_a_train_split_nothing_is_generated(self, server) -> None: + dataset = AgentDataset( + examples=[AgentExample(id="v", input={"current_query": "q"}, split="validation")] + ) + out, _ = _augment(dataset, server, n_examples=5) + assert len(out.examples) == 1 + assert out.metadata["augmented"] is False + + def test_strict_mode_raises_when_it_falls_short(self, server) -> None: + dataset = AgentDataset( + examples=[AgentExample(id="v", input={"current_query": "q"}, split="validation")] + ) + with pytest.raises(RuntimeError, match="Augmentation added 0/"): + _augment(dataset, server, n_examples=5, strict=True) + + +class TestPersistence: + def test_persist_only_never_overwrites_the_seed_file(self, tmp_path) -> None: + seed = tmp_path / "all.jsonl" + seed.write_text(EXAMPLE_DATASET.read_text()) + before = seed.read_text() + _, written = prepare_dataset(load_data(seed), persist=True, seed_path=seed) + assert seed.read_text() == before + assert written == tmp_path / "all.prepared.jsonl" + + +class TestReplyParsing: + ARRAY = '[{"query": "a?", "expected_answer": "x"}, {"query": "b?", "expected_answer": "y"}]' + + @pytest.mark.parametrize( + ("reply", "count"), + [ + (ARRAY, 2), + ("Sure [ok]:\n```json\n" + ARRAY + "\n```\nHope [this] helps", 2), + ('{"examples": ' + ARRAY + "}", 2), + ('{"query": "a?", "expected_answer": "x"}', 1), + ('{"query": "a?"}\n{"query": "b?"}', 2), + (ARRAY[:-30], 1), + ("[x]" + ARRAY, 2), + ("no json at all", 0), + ], + ) + def test_reply_shapes(self, reply: str, count: int) -> None: + assert len(parse_generated_points(reply)) == count + + def test_truncated_replies_are_salvaged_during_augmentation(self, monkeypatch) -> None: + async def truncated(*args: Any, **kwargs: Any) -> str: + return '[{"query": "What is the refund window?", "expected_answer": "30 days"}, {"query": "Wha' + + monkeypatch.setattr(LLMCaller, "call", staticmethod(truncated)) + dataset = AgentDataset( + examples=[ + AgentExample( + id="s", + input={"current_query": "Refund window?"}, + expected_output={"response": "30 days"}, + split="train", + ) + ] + ) + out, _ = prepare_dataset(dataset, augment=True, n_examples=2, model="omlx/x") + assert len(out.examples) == 2 + + +def _augment(dataset: Any, server: FakeOpenAIServer, **kwargs: Any) -> tuple[Any, Any]: + kwargs.setdefault("strategies", ["paraphrase", "expansion"]) + return prepare_dataset( + dataset, + augment=True, + model="omlx/fake-model", + llm_base_url=server.base_url, + llm_api_key="x", + **kwargs, + ) diff --git a/tests/test_documentation_contracts.py b/tests/test_documentation_contracts.py index b54021d..7df058b 100644 --- a/tests/test_documentation_contracts.py +++ b/tests/test_documentation_contracts.py @@ -459,7 +459,7 @@ def test_local_documentation_images_resolve_from_their_published_pages() -> None def test_optimization_guide_documents_only_supported_cli_paths() -> None: """Prevent removed optimization commands and modes returning to the docs.""" - optimization = _doc("docs/guide/optimization.md") + optimization = _doc("docs/optimization/cli.md") assert "`PromptFitter`" in optimization assert "`PromptOptimizer`" in optimization @@ -550,7 +550,7 @@ def test_parsable_documented_agentomatic_imports_resolve() -> None: def test_optimization_guide_matches_current_cli_contract() -> None: """Keep mode names and important flags tied to Click's public interface.""" - optimization = _doc("docs/guide/optimization.md") + optimization = _doc("docs/optimization/cli.md") help_text = CliRunner().invoke(cli, ["optimize", "--help"]).output for token in ( diff --git a/tests/test_eval_blackout.py b/tests/test_eval_blackout.py index 44269f5..39442c2 100644 --- a/tests/test_eval_blackout.py +++ b/tests/test_eval_blackout.py @@ -158,7 +158,8 @@ async def test_one_success_clears_it(self) -> None: ContainsMetric(), ) - assert score == 1.0 + # The failed point counts as 0.0; the average is still a measurement. + assert score == 0.5 assert not fitter._eval_blackout # noqa: SLF001 @pytest.mark.asyncio diff --git a/tests/test_fit_anti_overfitting.py b/tests/test_fit_anti_overfitting.py new file mode 100644 index 0000000..98a7598 --- /dev/null +++ b/tests/test_fit_anti_overfitting.py @@ -0,0 +1,307 @@ +# pyright: reportMissingParameterType=none +# pyright: reportAttributeAccessIssue=none +"""Prompt optimization must not reward prompts that overfit. + +The mechanism, end to end: + +* the optimizer **learns** from train only (failures, judge feedback, demos); +* candidates are **selected** on validation, and only when the gain is + unlikely to be noise (paired bootstrap confidence); +* a **held-out** slice neither step sees can only *veto* a candidate — for + regressing, for widening the validation/held-out gap, or for a validation + gain that does not transfer (the signature of memorised validation answers); +* the ``test`` split is never used by optimization at all. + +The simulated agent below answers correctly only for the queries its prompt +"knows": ``GENERAL`` knows everything, ``MEMO: `` only those listed — +the textbook overfit prompt. +""" + +from __future__ import annotations + +import dataclasses +from typing import Any +from unittest.mock import AsyncMock + +import pytest + +from agentomatic.optimize import ( + CustomMetric, + DataPoint, + Dataset, + PromptFitter, +) +from agentomatic.optimize.config import PromptCandidate, PromptRuntimeConfig +from agentomatic.optimize.events import OptimizationEvent +from agentomatic.optimize.learning import check_generalization, paired_improvement_confidence +from agentomatic.optimize.runner import AgentRunner + +TRAIN = [f"t{i}" for i in range(4)] +VAL = [f"v{i}" for i in range(4)] +HOLD = [f"h{i}" for i in range(3)] + + +def _ds(queries: list[str]) -> Dataset: + return Dataset(points=[DataPoint(query=q, expected_answer=f"ans-{q}") for q in queries]) + + +async def _agent(query: str, *, prompt_override=None, context=None, invoke=None) -> str: + prompt = prompt_override or "" + if "GENERAL" in prompt: + return f"ans-{query}" + if prompt.startswith("MEMO:") and query in prompt.split(): + return f"ans-{query}" + return "no idea" + + +def _metric() -> CustomMetric: + return CustomMetric(lambda q, r, e, c: 1.0 if r == e else 0.0, name="exact") + + +class _Proposer: + """Duck-typed optimizer that proposes fixed prompts and records what it saw.""" + + name = "scripted" + + def __init__(self, rounds: list[list[str]]) -> None: + self._rounds = rounds + self.seen_eval_queries: list[str] = [] + self.seen_sample_queries: list[str] = [] + + async def propose( + self, + current_config, + eval_results, + dataset_sample, + search_space, + iteration=0, + context=None, + ) -> list[PromptCandidate]: + self.seen_eval_queries += [str(r.get("query")) for r in eval_results] + self.seen_sample_queries += [str(p.get("query")) for p in dataset_sample] + prompts = self._rounds[iteration] if iteration < len(self._rounds) else [] + # Like the real optimizers: change the prompt, keep everything else. + return [ + PromptCandidate( + name=f"c{iteration}_{i}", + config=dataclasses.replace(current_config, system_prompt=text), + source="scripted", + ) + for i, text in enumerate(prompts) + ] + + +class _Events: + def __init__(self) -> None: + self.events: list[OptimizationEvent] = [] + + async def on_event(self, event, data) -> None: + self.events.append(event) + + +def _fitter(tmp_path, proposer: Any, **kwargs: Any) -> PromptFitter: + fitter = PromptFitter( + agent="sim", + optimizer=proposer, + baseline_system_prompt="You are a baseline assistant.", + experiment_dir=str(tmp_path), + auto_report=False, + drain_seconds=0, + **{"max_trials": 8, "patience": 3, **kwargs}, + ) + fitter._runner = AgentRunner(agent="sim", agent_callable=_agent) # noqa: SLF001 + fitter._failure_clusterer.cluster = AsyncMock(return_value=[]) # noqa: SLF001 + return fitter + + +def _decisions(result) -> dict[str, tuple[str, str]]: + return { + t["system_prompt"]: (t["decision"], t.get("reason", "")) + for t in result.trials + if t.get("phase") in ("full_val", "skipped") + } + + +class TestAcceptance: + async def test_memorised_validation_answers_are_rejected(self, tmp_path) -> None: + memo = "MEMO: " + " ".join(VAL) + fitter = _fitter(tmp_path, _Proposer([[memo]])) + result = await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + + decision, reason = _decisions(result)[memo] + assert decision == "rejected" + # Validation 0 → 1 while held-out stays 0: vetoed by the safety net. + assert "Generalization safety net" in reason + assert not result.improved + assert result.best_prompt == "You are a baseline assistant." + + async def test_a_generalizing_prompt_is_accepted(self, tmp_path) -> None: + fitter = _fitter(tmp_path, _Proposer([["GENERAL assistant"]])) + result = await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + + assert _decisions(result)["GENERAL assistant"][0] == "accepted" + assert result.improved + assert result.best_score == pytest.approx(1.0) + assert result.holdout_score == pytest.approx(1.0) + + async def test_the_generalizing_prompt_wins_over_the_memorising_one(self, tmp_path) -> None: + memo = "MEMO: " + " ".join(VAL) + fitter = _fitter(tmp_path, _Proposer([[memo, "GENERAL assistant"]])) + result = await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + assert result.best_prompt == "GENERAL assistant" + + async def test_duplicates_of_the_incumbent_are_not_rescored(self, tmp_path) -> None: + fitter = _fitter(tmp_path, _Proposer([["You are a baseline assistant."]])) + result = await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + assert _decisions(result)["You are a baseline assistant."][0] == "duplicate" + assert not any(t.get("phase") == "minibatch" for t in result.trials) + + +class TestProposerIsolation: + async def test_the_optimizer_learns_from_train_only(self, tmp_path) -> None: + proposer = _Proposer([["GENERAL assistant"], ["GENERAL assistant v2"]]) + fitter = _fitter(tmp_path, proposer) + await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + + seen = set(proposer.seen_eval_queries) | set(proposer.seen_sample_queries) + assert seen, "the proposer must receive reflection data" + assert seen <= set(TRAIN), f"validation/holdout leaked to the proposer: {seen}" + + async def test_legacy_reflection_on_validation_is_opt_in(self, tmp_path) -> None: + proposer = _Proposer([["GENERAL assistant"]]) + fitter = _fitter(tmp_path, proposer, reflect_on="validation") + await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + assert set(proposer.seen_eval_queries) <= set(VAL) + + +class TestBudgetAndEvents: + async def test_max_trials_caps_distinct_candidates(self, tmp_path) -> None: + many = [[f"cand {i}" for i in range(10)] for _ in range(5)] + fitter = _fitter(tmp_path, _Proposer(many), max_trials=3) + result = await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + assert sum(1 for t in result.trials if t.get("phase") == "minibatch") == 3 + + async def test_round_end_is_emitted_for_rounds_without_candidates(self, tmp_path) -> None: + events = _Events() + fitter = _fitter(tmp_path, _Proposer([[], []]), max_trials=8, callbacks=[events]) + await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + starts = events.events.count(OptimizationEvent.ROUND_START) + assert starts >= 1 + assert events.events.count(OptimizationEvent.ROUND_END) == starts + + async def test_every_scored_candidate_records_its_prompt_and_decision(self, tmp_path) -> None: + fitter = _fitter(tmp_path, _Proposer([["no help", "GENERAL assistant"]])) + result = await fitter.fit(_ds(TRAIN), _ds(VAL), _metric(), testset=_ds(HOLD)) + mini = [t for t in result.trials if t["phase"] == "minibatch"] + assert {t["system_prompt"] for t in mini} == {"no help", "GENERAL assistant"} + assert all(t["decision"] in {"promoted", "not_promoted"} and t["reason"] for t in mini) + + +class TestGeneralizationRules: + def test_memorisation_does_not_transfer(self) -> None: + check = check_generalization( + fit_score=0.19, + holdout_score=0.085, + baseline_fit=0.085, + baseline_holdout=0.085, + min_transfer_ratio=0.25, + ) + assert not check.ok and "Does not transfer" in check.reason + + def test_a_large_real_gain_is_accepted(self) -> None: + check = check_generalization( + fit_score=0.8, + holdout_score=0.7, + baseline_fit=0.1, + baseline_holdout=0.2, + min_transfer_ratio=0.25, + ) + assert check.ok + + def test_held_out_regression_is_rejected(self) -> None: + check = check_generalization( + fit_score=0.9, + holdout_score=0.4, + baseline_fit=0.6, + baseline_holdout=0.5, + min_transfer_ratio=0.0, + ) + assert not check.ok + + def test_legacy_absolute_gap(self) -> None: + assert not check_generalization(fit_score=0.9, holdout_score=0.5, max_gap=0.15).ok + assert check_generalization(fit_score=0.6, holdout_score=0.5, max_gap=0.15).ok + + def test_paired_confidence(self) -> None: + assert paired_improvement_confidence([1, 1, 1], [0, 0, 0]) == 1.0 + assert paired_improvement_confidence([0.5, 0.6], [0.5, 0.6]) == 0.0 + noisy = paired_improvement_confidence([0.9, 0.1, 0.8, 0.2], [0.5, 0.5, 0.5, 0.5]) + assert 0.0 < noisy < 0.8 + + +class TestBridgeDataRoles: + @staticmethod + def _points(**splits: int) -> Dataset: + points = [] + for split, n in splits.items(): + points += [ + DataPoint(query=f"{split}{i}", expected_answer="x", metadata={"split": split}) + for i in range(n) + ] + return Dataset(points=points) + + def test_test_split_never_reaches_the_fitter(self) -> None: + from agentomatic.agents import PromptFitterBridge + + train, val, gate = PromptFitterBridge._split_three( # noqa: SLF001 + self._points(train=4, validation=3, test=3) + ) + used = {p.query for p in [*train, *val, *(gate or [])]} + assert not any(q.startswith("test") for q in used) + assert gate is None # the fitter reserves its own slice of validation + + def test_holdout_split_becomes_the_gate(self) -> None: + from agentomatic.agents import PromptFitterBridge + + _train, val, gate = PromptFitterBridge._split_three( # noqa: SLF001 + self._points(train=4, validation=3, holdout=2, test=3) + ) + assert gate is not None and {p.query for p in gate} == {"holdout0", "holdout1"} + assert {p.query for p in val} == {"validation0", "validation1", "validation2"} + + def test_train_only_is_split_without_overlap(self) -> None: + from agentomatic.agents import PromptFitterBridge + + train, val, _ = PromptFitterBridge._split_three(self._points(train=10)) # noqa: SLF001 + assert not ({p.query for p in train} & {p.query for p in val}) + assert len(val) >= 1 + + def test_a_non_improving_result_changes_nothing(self) -> None: + from agentomatic.agents import PromptFitterBridge + from agentomatic.optimize import PromptFitResult + + result = PromptFitResult( + best_config=PromptRuntimeConfig(system_prompt="from prompts.json"), + baseline_config=PromptRuntimeConfig(system_prompt="from prompts.json"), + best_score=0.5, + baseline_score=0.5, + ) + assert PromptFitterBridge._extract_config(None, result) == {} # noqa: SLF001 + + def test_few_shot_demos_are_served_as_they_were_scored(self) -> None: + from agentomatic.agents import PromptFitterBridge + from agentomatic.optimize import PromptFitResult + + best = PromptRuntimeConfig( + system_prompt="Be precise.", + few_shot_examples=[{"query": "q", "response": "a"}], + ) + result = PromptFitResult( + best_config=best, + baseline_config=PromptRuntimeConfig(system_prompt="base"), + best_score=0.9, + baseline_score=0.1, + ) + config = PromptFitterBridge._extract_config(None, result) # noqa: SLF001 + assert config["system_prompt"].startswith("Be precise.") + assert "## Few-shot examples" in config["system_prompt"] diff --git a/tests/test_fit_optimizer_matrix.py b/tests/test_fit_optimizer_matrix.py index 656a6db..44bd98b 100644 --- a/tests/test_fit_optimizer_matrix.py +++ b/tests/test_fit_optimizer_matrix.py @@ -322,8 +322,12 @@ def test_train_template_uses_run_train() -> None: assert "result = train_and_report(" in src assert "# compiled = compile_agent(" in src assert "# history = fit_agent(" in src - assert "# scores = evaluate_agent(" in src - assert "# generate_fit_report(" in src + # The held-out split is scored before and after fit, and the report gets + # the History (all epochs: original prompt → final prompt). + assert "# before = evaluate_agent(" in src + assert "# after = evaluate_agent(" in src + assert "# generate_fit_report(history," in src + assert "baseline_eval=before, final_eval=after" in src def test_eval_template_uses_evaluate_and_report() -> None: diff --git a/tests/test_fit_report_content.py b/tests/test_fit_report_content.py new file mode 100644 index 0000000..ad40bbd --- /dev/null +++ b/tests/test_fit_report_content.py @@ -0,0 +1,321 @@ +# pyright: reportMissingParameterType=none +"""The fit report must show what the optimization changed and whether it helped. + +Reported: "I cannot see prompts. I cannot see scores … I cannot see what +changed in the prompts, the initial ones compared to the evolutions". Causes: + +* every prompt panel was a HolySheet ``Accordion`` — its viewer renders none + of them, so prompts, few-shot examples and per-example details were hidden; +* with ``epochs > 1`` only the LAST epoch's result was reported: its + "baseline" was the prompt an earlier epoch had already improved, so the + report compared the final prompt with itself and said "no improvement"; +* no before/after numbers on the untouched test split, no per-example + answers, and no reason why each candidate was accepted or rejected; +* single-line prompts diffed as one unreadable ``-``/``+`` pair. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest + +from agentomatic.agents import AgentExample, History +from agentomatic.agents.types import EvaluationReport, ExampleResult +from agentomatic.optimize import PromptFitResult, generate_fit_report, merge_fit_results +from agentomatic.optimize.config import PromptRuntimeConfig +from agentomatic.optimize.holysheet_reports import _unified_diff + +INITIAL = "You are a support assistant. Help the customer." +MIDDLE = "You are a support assistant. Quote the policy snippet, then answer." +FINAL = "You are a precise support assistant. Quote the policy snippet verbatim, then answer." + + +def _epoch( + start: str, + best: str, + start_score: float, + best_score: float, + *, + name: str, + suggestions: list[str] | None = None, +) -> PromptFitResult: + improved = best != start + return PromptFitResult( + best_config=PromptRuntimeConfig(system_prompt=best), + baseline_config=PromptRuntimeConfig(system_prompt=start), + best_score=best_score, + baseline_score=start_score, + baseline_holdout_score=start_score, + holdout_score=best_score, + metric_deltas={"quality": best_score - start_score}, + suggestions=suggestions or [], + trials=[ + { + "round": 1, + "name": f"{name}_dup", + "phase": "skipped", + "decision": "duplicate", + "reason": "Identical to a config already evaluated", + "system_prompt": start, + }, + { + "round": 1, + "name": name, + "phase": "minibatch", + "score": best_score, + "decision": "promoted", + "system_prompt": best if improved else start + " (rejected edit)", + }, + { + "round": 1, + "name": name, + "phase": "full_val", + "score": best_score, + "incumbent_score": start_score, + "holdout_score": best_score, + "confidence": 0.97, + "decision": "accepted" if improved else "rejected", + "reason": "better on validation" if improved else "Not significant", + "system_prompt": best if improved else start + " (rejected edit)", + }, + ], + score_history=[start_score, best_score], + baseline_examples=[ + {"query": "Refund window?", "expected": "30 days", "response": "Hi!", "score": 0.0} + ], + best_examples=[ + {"query": "Refund window?", "expected": "30 days", "response": "30 days", "score": 1} + ], + duration_seconds=2.0, + agent="support_agent", + ) + + +def _epochs() -> list[PromptFitResult]: + """Epoch 1 improves, epoch 2 improves again, epoch 3 finds nothing.""" + return [ + _epoch(INITIAL, MIDDLE, 0.2, 0.6, name="e1", suggestions=["Quote the policy"]), + _epoch(MIDDLE, FINAL, 0.6, 0.8, name="e2", suggestions=["Say 'verbatim'"]), + _epoch(FINAL, FINAL, 0.8, 0.8, name="e3", suggestions=["No improvement"]), + ] + + +def _report(scores: dict[str, float], answers: dict[str, str]) -> EvaluationReport: + return EvaluationReport( + scores=scores, + example_results=[ + ExampleResult( + example_id=eid, + prediction={"response": text}, + scores={"quality": scores["quality"]}, + ) + for eid, text in answers.items() + ], + ) + + +TEST_SPLIT = [ + AgentExample( + id="t1", + input={"current_query": "Can I get my money back?"}, + expected_output={"response": "Within 30 days on annual plans."}, + split="test", + ) +] + + +class TestMergeFitResults: + def test_spans_the_original_prompt_to_the_final_one(self) -> None: + merged = merge_fit_results(_epochs()) + assert merged.baseline_config.system_prompt == INITIAL + assert merged.best_config.system_prompt == FINAL + assert merged.baseline_score == pytest.approx(0.2) + assert merged.best_score == pytest.approx(0.8) + assert merged.improved + assert [t["epoch"] for t in merged.trials] == [1, 1, 1, 2, 2, 2, 3, 3, 3] + assert merged.duration_seconds == pytest.approx(6.0) + assert merged.metric_deltas["quality"] == pytest.approx(0.6) + + def test_recommendations_come_from_the_epoch_that_produced_the_final_prompt(self) -> None: + merged = merge_fit_results(_epochs()) + assert merged.suggestions == ["Say 'verbatim'"] + + def test_the_last_epoch_alone_would_claim_no_improvement(self) -> None: + last = _epochs()[-1] + assert not last.improved + assert last.baseline_config.system_prompt == FINAL + + def test_inputs_are_not_modified(self) -> None: + epochs = _epochs() + merge_fit_results(epochs) + assert "epoch" not in epochs[0].trials[0] + assert epochs[-1].baseline_config.system_prompt == FINAL + + def test_needs_at_least_one_result(self) -> None: + with pytest.raises(ValueError, match="at least one"): + merge_fit_results([]) + + +class TestGenerateFitReportInputs: + def test_accepts_the_history_returned_by_fit(self, tmp_path: Path) -> None: + history = History() + history.record(-1, {"loss": 0.8, "val_loss": 0.8}) + history.record(0, {"loss": 0.2, "val_loss": 0.3}) + history.fit_results = _epochs() + path = generate_fit_report(history, output_path=tmp_path / "r.html") + sidecar = json.loads(Path(path).with_suffix(".json").read_text()) + assert sidecar["initial_prompt"] == INITIAL + assert sidecar["final_prompt"] == FINAL + assert sidecar["keras_history"]["val_loss"] == [0.8, 0.3] + assert len(sidecar["epochs"]) == 3 + + def test_accepts_the_fitted_agent(self, tmp_path: Path) -> None: + agent = SimpleNamespace(_fit_results=_epochs()) + path = generate_fit_report(agent, output_path=tmp_path / "r.html") + assert json.loads(Path(path).with_suffix(".json").read_text())["prompt_changed"] + + def test_rejects_unknown_objects_with_guidance(self) -> None: + with pytest.raises(TypeError, match="History returned by agent.fit"): + generate_fit_report(object()) + + def test_says_why_when_there_is_nothing_to_report(self) -> None: + with pytest.raises(ValueError, match="_last_optimize_status"): + generate_fit_report(History()) + + +class TestReportContent: + @pytest.fixture + def report(self, tmp_path: Path) -> tuple[str, dict[str, Any], list[dict[str, Any]]]: + path = generate_fit_report( + _epochs(), + output_path=tmp_path / "train_report.html", + baseline_eval=_report({"quality": 0.1}, {"t1": "Thanks for reaching out!"}), + final_eval=_report({"quality": 0.9}, {"t1": "Refunds within 30 days."}), + eval_dataset=TEST_SPLIT, + dataset_stats={"requested": 10, "added": 8, "by_strategy": {"paraphrase": 8}}, + ) + html = Path(path).read_text(encoding="utf-8") + sidecar = json.loads(Path(path).with_suffix(".json").read_text()) + return html, sidecar, _blocks(html) + + def test_sidecar_has_before_after_on_the_test_split(self, report) -> None: + _, sidecar, _ = report + assert sidecar["validation"] == {"initial": 0.2, "final": 0.8} + assert sidecar["test"] == {"before": {"quality": 0.1}, "after": {"quality": 0.9}} + row = sidecar["test_examples"][0] + assert row["question"] == "Can I get my money back?" + assert row["before"] == "Thanks for reaching out!" + assert row["after"] == "Refunds within 30 days." + assert row["Δ"] == "quality +0.80" + + def test_sections_exist_and_are_not_empty(self, report) -> None: + _, _, blocks = report + sections = { + (b.get("props") or {}).get("title"): (b.get("props") or {}).get("children") + for b in blocks + if b.get("type") == "section" + } + for title in ( + "Verdict", + "Test scoreboard", + "What changed in the prompt", + "Epochs", + "All candidates", + "Examples — before vs after", + "Data & settings", + ): + assert sections.get(title), f"{title!r} missing or empty" + + def test_prompts_are_visible_not_hidden_in_accordions(self, report) -> None: + html, _, blocks = report + assert not [b for b in blocks if b.get("type") == "accordion"] + code = "\n".join( + str((b.get("props") or {}).get("code", "")) + for b in blocks + if b.get("type") == "code_block" + ) + assert "-You are a support assistant. Help the customer." in code + assert "verbatim" in code + assert "(rejected edit)" in code # rejected candidates are shown too + assert "Thanks for reaching out!" in html + + def test_every_candidate_decision_and_reason_is_listed(self, report) -> None: + _, _, blocks = report + rows = [ + row + for b in blocks + if b.get("type") == "table" or "data" in (b.get("props") or {}) + for row in (b.get("props") or {}).get("data") or [] + if isinstance(row, dict) and "decision" in row + ] + decisions = {(r["epoch"], r["candidate"]): r["decision"] for r in rows} + assert decisions[(1, "e1")] == "accepted" + assert decisions[(3, "e3")] == "rejected" + assert decisions[(1, "e1_dup")] == "duplicate" + assert any("Not significant" in r["reason"] for r in rows) + + +class TestFallbackHtml: + def test_without_holysheet_the_report_still_shows_before_after( + self, tmp_path: Path, monkeypatch + ) -> None: + import agentomatic.optimize.holysheet_reports as hs + + def _missing(*_a: Any, **_k: Any) -> str: + raise ImportError("holysheet") + + monkeypatch.setattr(hs, "build_fit_holysheet_report", _missing) + path = generate_fit_report( + _epochs(), + output_path=tmp_path / "r.html", + baseline_eval_scores={"quality": 0.1}, + eval_scores={"quality": 0.9}, + ) + html = Path(path).read_text(encoding="utf-8") + assert "Test Scoreboard" in html and "+0.8000" in html + assert "Prompt Diff (initial → final)" in html + assert "Not significant" in html + + +class TestReadableDiff: + def test_long_single_line_prompts_diff_sentence_by_sentence(self) -> None: + before = "You are helpful. " * 12 + "Answer briefly." + after = "You are helpful. " * 12 + "Answer briefly and quote the policy." + diff = _unified_diff(before, after) + changed = [ + line + for line in diff.splitlines() + if line[:1] in "+-" and line[:3] not in ("+++", "---") + ] + assert len(changed) == 2 + assert all(len(line) <= 101 for line in diff.splitlines()) + + def test_identical_prompts_have_no_diff(self) -> None: + assert _unified_diff("same", "same") == "" + + +def _blocks(html: str) -> list[dict[str, Any]]: + """Every serialized HolySheet block (recursively) embedded in the page.""" + match = re.search(r'"blocks"\s*:\s*(\[.*?\])\s*,\s*"filters"', html, re.S) + assert match, "report JSON not found" + + def walk(items: list[Any]) -> list[dict[str, Any]]: + out: list[dict[str, Any]] = [] + for block in items: + if not isinstance(block, dict): + continue + out.append(block) + props = block.get("props") or {} + out.extend(walk(props.get("children") or [])) + for tab in props.get("tabs") or []: + out.extend(walk(tab.get("children") or [])) + for panel in props.get("panels") or []: + out.extend(walk(panel.get("children") or [])) + return out + + return walk(json.loads(match.group(1))) diff --git a/tests/test_fitter.py b/tests/test_fitter.py index 543552e..145404f 100644 --- a/tests/test_fitter.py +++ b/tests/test_fitter.py @@ -1326,7 +1326,9 @@ async def test_no_latency_data(self): m = LatencyMetric() result = await m.evaluate("q", "resp") - assert result.score == 0.5 + # No data is a failed evaluation, not a neutral 0.5 diluting composites. + assert result.score == 0.0 + assert result.metadata["evaluation_failed"] is True assert "No latency data" in result.reason @@ -1516,7 +1518,8 @@ def transform(self, input_data): agent = FakeAgent() fn = _wrap_local_agent(agent) result = await fn("hello", prompt_override=None, context=None, invoke=None) - assert result == "transformed:hello" + # Dicts pass through; AgentRunner derives the scored text. + assert result == {"response": "transformed:hello"} @pytest.mark.asyncio async def test_wrap_atransform(self): @@ -1529,7 +1532,7 @@ async def atransform(self, input_data): agent = FakeAsyncAgent() fn = _wrap_local_agent(agent) result = await fn("world", prompt_override=None, context=None, invoke=None) - assert result == "async:world" + assert result == {"response": "async:world"} @pytest.mark.asyncio async def test_wrap_output_dict(self): @@ -1544,7 +1547,9 @@ def transform(self, input_data): agent = FakeAgent() fn = _wrap_local_agent(agent) result = await fn("q", prompt_override=None, context=None, invoke=None) - data = json.loads(result) + from agentomatic.optimize.runner import _response_text + + data = json.loads(_response_text(result)) assert data == {"content": "foo", "next_action": "bar"} @pytest.mark.asyncio @@ -1577,7 +1582,7 @@ def transform(self, input_data): fn = _wrap_local_agent(agent) result = await fn("q", prompt_override="overridden", context=None, invoke=None) # response was generated with the override - assert result == "overridden" + assert result == {"response": "overridden"} # original prompt restored after call assert agent.system_prompt == "original" diff --git a/tests/test_jwt_signature_e2e.py b/tests/test_jwt_signature_e2e.py index 4176961..6a61665 100644 --- a/tests/test_jwt_signature_e2e.py +++ b/tests/test_jwt_signature_e2e.py @@ -87,12 +87,14 @@ def __enter__(self): def __exit__(self, *exc: object) -> None: return None - def _urlopen(url, *args, **kwargs): # noqa: ANN001, ANN002, ANN003 + def _open(_opener, url, *args, **kwargs): # noqa: ANN001, ANN002, ANN003 target = getattr(url, "full_url", url) assert target == JWKS_URL, f"unexpected JWKS fetch: {target}" return _Response() - monkeypatch.setattr(urllib.request, "urlopen", _urlopen) + # Every urllib fetch goes through an opener: ``urlopen`` builds one, and + # PyJWT >= 2.14 builds its own (redirects refused) instead of calling it. + monkeypatch.setattr(urllib.request.OpenerDirector, "open", _open) class _HttpxResponse: status_code = 200 diff --git a/tests/test_metric_honesty.py b/tests/test_metric_honesty.py new file mode 100644 index 0000000..f9468c1 --- /dev/null +++ b/tests/test_metric_honesty.py @@ -0,0 +1,388 @@ +# pyright: reportMissingParameterType=none +# pyright: reportAttributeAccessIssue=none +"""Scores must mean what they say — no clamped scales, no score *rising* on failure. + +Covers the judge-parsing, aggregation and training-loop defects found by +auditing the optimization metrics: 0–10 judge scores clamped to a perfect 1.0, +``{"score": …}`` replies rejected, a failed judge raising a composite score, +panel weights attached to the wrong judge, lower-is-better DeepEval metrics +maximised, Keras-incompatible EarlyStopping, crashed examples "passing", and +the same judge being called several times per example. +""" + +from __future__ import annotations + +import asyncio +from typing import Any + +import pytest + +from agentomatic.agents import ( + AgentExample, + CallableMetric, + EarlyStopping, + OptimizeMetricAdapter, +) +from agentomatic.agents.types import ExampleResult +from agentomatic.optimize import ( + BaseMetric, + CompositeMetric, + CustomMetric, + LatencyMetric, + LLMJudgeMetric, + LocalJudgeMetric, + MultiJudgePanel, +) +from agentomatic.optimize.llm_caller import LLMCaller +from agentomatic.optimize.metrics import ( + EvalResult, + WeightedMetric, + coerce_judge_score, + scoring_run, +) +from agentomatic.optimize.runner import RunResult + + +class TestCoerceJudgeScore: + @pytest.mark.parametrize( + ("raw", "expected"), + [ + (0.8, 0.8), + (1, 1.0), + (7, 0.7), + (85, 0.85), + ("0.6", 0.6), + ("8/10", 0.8), + ("80%", 0.8), + ({"score": 0.4}, 0.4), + ], + ) + def test_scales_are_normalised(self, raw: Any, expected: float) -> None: + assert coerce_judge_score(raw) == pytest.approx(expected) + + @pytest.mark.parametrize("raw", [None, True, -0.1, 250, "great", float("nan"), {}]) + def test_unusable_values(self, raw: Any) -> None: + assert coerce_judge_score(raw) is None + + +def _judge_reply(monkeypatch, reply: dict[str, Any]) -> None: + async def fake(*args: Any, **kwargs: Any) -> dict[str, Any]: + return reply + + monkeypatch.setattr(LLMCaller, "call_with_json", staticmethod(fake)) + + +class TestLocalJudgeParsing: + def _run(self, monkeypatch, reply: dict[str, Any]) -> EvalResult: + _judge_reply(monkeypatch, reply) + judge = LocalJudgeMetric(name="j", model="omlx/x", dimensions=["correctness", "relevance"]) + return asyncio.run(judge.evaluate("q", "r", "e")) + + def test_score_key_is_accepted(self, monkeypatch) -> None: + result = self._run(monkeypatch, {"score": 0.8}) + assert result.score == pytest.approx(0.8) + assert not result.metadata["evaluation_failed"] + + def test_ten_point_scale_is_not_a_perfect_score(self, monkeypatch) -> None: + assert self._run(monkeypatch, {"overall_score": 2}).score == pytest.approx(0.2) + + def test_dimension_only_reply_uses_their_mean(self, monkeypatch) -> None: + result = self._run(monkeypatch, {"dimensions": {"correctness": 0.9, "relevance": 0.7}}) + assert result.score == pytest.approx(0.8) + assert result.metadata["score_source"] == "dimension_mean" + + def test_nested_dimension_does_not_discard_the_overall(self, monkeypatch) -> None: + result = self._run( + monkeypatch, + {"overall_score": 0.7, "dimensions": {"correctness": {"score": 0.9, "why": "…"}}}, + ) + assert result.score == pytest.approx(0.7) + assert result.metadata["dimensions"]["correctness"] == pytest.approx(0.9) + + def test_string_fields_are_not_exploded_into_characters(self, monkeypatch) -> None: + result = self._run(monkeypatch, {"score": 0.5, "what_worked": "clear answer"}) + assert result.metadata["what_worked"] == ["clear answer"] + + def test_unusable_reply_is_a_failed_evaluation(self, monkeypatch) -> None: + result = self._run(monkeypatch, {"verdict": "good"}) + assert result.score == 0.0 + assert result.metadata["evaluation_failed"] is True + + def test_feedback_text_does_not_decide_failure(self, monkeypatch) -> None: + result = self._run( + monkeypatch, {"score": 0.9, "feedback": "Judge evaluation failed: not really"} + ) + assert result.metadata["evaluation_failed"] is False + + def test_per_judge_endpoint_is_forwarded(self, monkeypatch) -> None: + seen: dict[str, Any] = {} + + async def fake(*args: Any, **kwargs: Any) -> dict[str, Any]: + seen.update(kwargs) + return {"score": 1} + + monkeypatch.setattr(LLMCaller, "call_with_json", staticmethod(fake)) + judge = LocalJudgeMetric(model="omlx/x", base_url="http://judge:9/v1", api_key="k") + asyncio.run(judge.evaluate("q", "r")) + assert seen["base_url"] == "http://judge:9/v1" + assert seen["api_key"] == "k" + + def test_llm_judge_accepts_overall_score(self, monkeypatch) -> None: + _judge_reply(monkeypatch, {"overall_score": 9}) + judge = LLMJudgeMetric(criteria="c", model="omlx/x") + assert asyncio.run(judge.evaluate("q", "r")).score == pytest.approx(0.9) + + +class _Fixed(BaseMetric): + def __init__(self, name: str, score: float | None) -> None: + self.name = name + self._score = score + + async def evaluate(self, query, response, expected=None, context=None) -> EvalResult: + if self._score is None: + return EvalResult(self.name, 0.0, "down", {"evaluation_failed": True}) + return EvalResult(self.name, self._score, "ok") + + +class TestCompositeMetric: + def test_a_failed_judge_never_raises_the_score(self) -> None: + working = CompositeMetric( + [ + WeightedMetric("judge", _Fixed("judge", 0.8), 0.3), + WeightedMetric("format", _Fixed("format", 1.0), 0.7), + ] + ) + failing = CompositeMetric( + [ + WeightedMetric("judge", _Fixed("judge", None), 0.3), + WeightedMetric("format", _Fixed("format", 1.0), 0.7), + ] + ) + ok = asyncio.run(working.evaluate("q", "r")) + down = asyncio.run(failing.evaluate("q", "r")) + assert down.score < ok.score + assert down.metadata["failed_components"] == ["judge"] + + def test_negative_weights_are_rejected(self) -> None: + with pytest.raises(ValueError, match="higher-is-better"): + CompositeMetric( + [ + WeightedMetric("quality", _Fixed("q", 1.0), 0.9), + WeightedMetric("latency", LatencyMetric(), -0.1), + ] + ) + + +class TestMultiJudgePanel: + def _panel(self, scores: list[float | None], **kw: Any) -> MultiJudgePanel: + class _J(LocalJudgeMetric): + def __init__(self, idx: int, score: float | None) -> None: + super().__init__(name=f"j{idx}") + self._s = score + + async def evaluate_rich(self, *a: Any, **k: Any) -> Any: + from agentomatic.optimize.metrics import MetricResult + + result = MetricResult(score=self._s or 0.0, feedback="f", dimensions={}) + result._judge_failed = self._s is None + return result + + return MultiJudgePanel(judges=[_J(i, s) for i, s in enumerate(scores)], **kw) + + def test_weights_stay_with_their_judge_when_one_fails(self) -> None: + panel = self._panel([None, 0.5, 1.0], weights=[5, 1, 3]) + assert asyncio.run(panel.evaluate("q", "r")).score == pytest.approx(0.875) + + def test_zero_weight_survivors_fail_cleanly(self) -> None: + panel = self._panel([0.9, None], weights=[0, 1]) + assert asyncio.run(panel.evaluate("q", "r")).metadata["evaluation_failed"] + + def test_median_aggregation(self) -> None: + panel = self._panel([0.1, 0.8, 0.9], aggregation="median") + assert asyncio.run(panel.evaluate("q", "r")).score == pytest.approx(0.8) + + def test_unknown_aggregation_is_rejected(self) -> None: + with pytest.raises(ValueError, match="Unknown aggregation"): + self._panel([0.5], aggregation="vibes") + + +class TestDeepEvalDirection: + def test_lower_is_better_metrics_are_inverted(self) -> None: + from agentomatic.optimize.metrics import DeepEvalMetric + + class ToxicityMetric: # name matches deepeval's class + score = 0.9 + reason = "very toxic" + + def measure(self, test_case: Any) -> None: + pass + + pytest.importorskip("deepeval") + result = asyncio.run(DeepEvalMetric(ToxicityMetric()).evaluate("q", "r")) + assert result.score == pytest.approx(0.1) + + +class TestOtherMetrics: + def test_custom_metric_clamps_and_rejects_nan(self) -> None: + assert asyncio.run(CustomMetric(lambda *a: 7.0).evaluate("q", "r")).score == 1.0 + nan = asyncio.run(CustomMetric(lambda *a: float("nan")).evaluate("q", "r")) + assert nan.metadata["evaluation_failed"] + + def test_latency_is_read_from_the_scored_run(self) -> None: + metric = LatencyMetric(target_seconds=1.0, max_seconds=3.0) + + async def score() -> EvalResult: + with scoring_run(RunResult(query="q", response="r", duration_ms=2000.0)): + return await metric.evaluate("q", "r") + + assert asyncio.run(score()).score == pytest.approx(0.5) + + +class TestTrainingLoopSemantics: + def _agent(self) -> Any: + from types import SimpleNamespace + + return SimpleNamespace( + stop_training=False, compiled_config={}, invalidate_graph=lambda: None + ) + + def test_early_stopping_uses_keras_patience(self) -> None: + cb = EarlyStopping(monitor="val_loss", patience=1) + cb.set_agent(self._agent()) + cb.on_train_begin() + cb.on_epoch_end(0, {"val_loss": 0.5}) + cb.on_epoch_end(1, {"val_loss": 0.6}) + assert cb.agent.stop_training is True + assert cb.stopped_epoch == 1 + + def test_restore_best_restores_the_best_epoch_config(self) -> None: + agent = self._agent() + cb = EarlyStopping(monitor="val_loss", patience=5, restore_best=True) + cb.set_agent(agent) + cb.on_train_begin() + agent.compiled_config = {"system_prompt": "good"} + cb.on_epoch_end(0, {"val_loss": 0.2}) + agent.compiled_config = {"system_prompt": "worse"} + cb.on_epoch_end(1, {"val_loss": 0.4}) + cb.on_train_end() + assert agent.compiled_config == {"system_prompt": "good"} + + def test_auto_mode_checks_the_suffix(self) -> None: + assert EarlyStopping(monitor="gloss_quality").mode == "max" + assert EarlyStopping(monitor="val_loss").mode == "min" + + def test_optimize_callbacks_are_rejected_by_agent_fit(self) -> None: + from dataclasses import dataclass + + from agentomatic.agents import BaseGraphAgent + from agentomatic.optimize import OptimizeEarlyStopping + + @dataclass + class S: + q: str = "" + + class A(BaseGraphAgent[S]): + agent_name = "a" + + def build_graph(self) -> Any: + g = self.new_graph() + g.add_node("n", lambda s: s) + g.set_entry_point("n") + g.set_finish_point("n") + return g.compile() + + def input_to_state(self, d: dict[str, Any]) -> S: + return S() + + def state_to_output(self, s: S) -> dict[str, Any]: + return {"response": "x"} + + agent = A() + agent.compile(metrics=[CallableMetric("m", lambda e, p: 1.0)]) + with pytest.raises(TypeError, match="PromptFitter"): + agent.fit( + [AgentExample(input={"current_query": "q"})], callbacks=[OptimizeEarlyStopping()] + ) + + def test_crashed_examples_do_not_pass(self) -> None: + assert not ExampleResult(example_id="x", error="boom").passed + assert ExampleResult(example_id="x", scores={"m": 0.9}).passed + + +class TestJudgeCallsPerExample: + def test_one_judge_call_per_example_across_wrappers(self) -> None: + calls: list[str] = [] + + class _Judge(BaseMetric): + name = "judge" + + async def evaluate(self, query, response, expected=None, context=None): + calls.append(response) + return EvalResult("judge", 0.5, "ok") + + from agentomatic.agents import MetricLoss + from agentomatic.agents import WeightedMetric as AgentWeighted + + judge = _Judge() + example = AgentExample(id="e", input={"current_query": "q"}) + prediction = {"response": "same answer"} + OptimizeMetricAdapter(judge).score(example, prediction) + AgentWeighted([("judge", judge, 1.0)]).score(example, prediction) + MetricLoss(judge).compute(example, prediction) + assert calls == ["same answer"] + + +def test_duplicate_metric_names_are_rejected() -> None: + from dataclasses import dataclass + + from agentomatic.agents import BaseGraphAgent + + @dataclass + class S: + q: str = "" + + class A(BaseGraphAgent[S]): + agent_name = "a" + + def build_graph(self) -> Any: + g = self.new_graph() + g.add_node("n", lambda s: s) + g.set_entry_point("n") + g.set_finish_point("n") + return g.compile() + + def input_to_state(self, d: dict[str, Any]) -> S: + return S() + + def state_to_output(self, s: S) -> dict[str, Any]: + return {} + + metric = CallableMetric("quality", lambda e, p: 1.0) + with pytest.raises(ValueError, match="unique"): + A().compile(metrics=[metric, CallableMetric("quality", lambda e, p: 0.5)]) + + +def test_run_sync_reuses_one_loop_inside_a_running_loop() -> None: + from agentomatic.async_utils import run_sync + + loops: set[int] = set() + + async def probe() -> int: + loops.add(id(asyncio.get_running_loop())) + return 1 + + async def main() -> list[int]: + return [run_sync(probe()) for _ in range(3)] + + assert asyncio.run(main()) == [1, 1, 1] + assert len(loops) == 1 + + +def test_rewrite_text_keeps_fenced_blocks() -> None: + from agentomatic.optimize.briefing import extract_prompt_text + from agentomatic.optimize.llm_caller import _normalize_llm_text + + raw = "hmmOld:\n```\nold prompt\n```\n---\nNew prompt with:\n```json\n{}\n```" + text = _normalize_llm_text(raw) + assert "old prompt" in text and "```json" in text + assert extract_prompt_text(text) == "New prompt with:\n```json\n{}\n```" diff --git a/tests/test_metric_protocol_bridge.py b/tests/test_metric_protocol_bridge.py new file mode 100644 index 0000000..1616b34 --- /dev/null +++ b/tests/test_metric_protocol_bridge.py @@ -0,0 +1,237 @@ +# pyright: reportMissingParameterType=none +"""Every metric must work in every slot, whichever protocol it implements. + +Two metric protocols coexist: + +* ``agentomatic.agents`` — sync ``score(example, prediction) -> float`` + (``compile(metrics=...)``, ``evaluate``, ``agents.WeightedMetric``, + ``MetricLoss``); +* ``agentomatic.optimize`` — async ``evaluate(query, response, expected, + context) -> EvalResult`` (``PromptFitter``, ``CompositeMetric``). + +Handing ``PromptFitterBridge(metric=OptimizeMetricAdapter(judge))`` to fit +crashed mid-run with ``'OptimizeMetricAdapter' object has no attribute +'evaluate'`` — logged as a warning, so training silently continued without +optimizing. These tests pin the bridge in both directions. +""" + +from __future__ import annotations + +import asyncio +from typing import Any + +import pytest + +from agentomatic.agents import ( + AgentExample, + CallableMetric, + ContainsTermsMetric, + MetricLoss, + OptimizeMetricAdapter, + PromptFitterBridge, + as_agent_metric, + resolve_loss, +) +from agentomatic.agents import WeightedMetric as AgentWeightedMetric +from agentomatic.optimize import ( + BaseMetric, + CompositeMetric, + ExactMatchMetric, + ScoreMetricAdapter, + as_optimize_metric, + resolve_metrics, +) +from agentomatic.optimize.metrics import EvalResult, WeightedMetric, scoring_run +from agentomatic.optimize.runner import RunResult + + +class _StubJudge(BaseMetric): + """Optimize metric returning a fixed score (or failing on demand).""" + + name = "stub_judge" + + def __init__(self, score: float = 0.8, *, fail: bool = False) -> None: + self._score = score + self._fail = fail + self.calls: list[tuple[str, str, str | None]] = [] + + async def evaluate(self, query, response, expected=None, context=None) -> EvalResult: + self.calls.append((query, response, expected)) + if self._fail: + raise ConnectionError("judge unreachable") + return EvalResult(metric_name=self.name, score=self._score, reason="stub") + + +def _example(**metadata: Any) -> AgentExample: + return AgentExample( + id="ex1", + input={"current_query": "What is the refund window?"}, + expected_output={"response": "30 days"}, + metadata=metadata, + ) + + +class TestOptimizeMetricAdapter: + def test_speaks_both_protocols(self) -> None: + adapter = OptimizeMetricAdapter(_StubJudge(0.7), name="judge") + assert adapter.score(_example(), {"response": "30 days"}) == pytest.approx(0.7) + result = asyncio.run(adapter.evaluate("q", "r", "e")) + assert result.score == pytest.approx(0.7) + + def test_failures_score_zero_and_are_counted_not_faked(self) -> None: + adapter = OptimizeMetricAdapter(_StubJudge(fail=True), name="judge") + assert adapter.score(_example(), {"response": "x"}) == 0.0 + assert adapter.failures == 1 + + def test_rejects_a_non_optimize_metric_with_guidance(self) -> None: + with pytest.raises(TypeError, match="score\\(example, prediction\\)"): + OptimizeMetricAdapter(ContainsTermsMetric(["x"])) + + def test_unwrapped_for_the_fitter(self) -> None: + judge = _StubJudge() + assert as_optimize_metric(OptimizeMetricAdapter(judge)) is judge + + +class TestPromptFitterBridgeMetric: + def test_accepts_the_adapter_from_the_user_script(self) -> None: + judge = _StubJudge() + bridge = PromptFitterBridge(metric=OptimizeMetricAdapter(judge, name="judge")) + assert bridge.metric is judge + + def test_wraps_class_agent_metrics(self) -> None: + bridge = PromptFitterBridge(metric=ContainsTermsMetric(["30 days"])) + assert isinstance(bridge.metric, ScoreMetricAdapter) + + def test_fails_fast_on_an_unusable_metric(self) -> None: + with pytest.raises(TypeError, match="not a usable metric"): + PromptFitterBridge(metric=object()) + + +class TestScoreMetricAdapter: + def test_class_agent_metric_scores_through_evaluate(self) -> None: + adapter = as_optimize_metric(ContainsTermsMetric(["30 days"])) + result = asyncio.run(adapter.evaluate("q", "Refunds within 30 days.", "30 days")) + assert result.score == 1.0 + + def test_rebuilds_expected_output_from_a_rich_reference(self) -> None: + seen: dict[str, Any] = {} + + def fn(example: AgentExample, prediction: dict[str, Any]) -> float: + seen["expected"] = example.expected_output + return 1.0 + + reference = _example().to_datapoint().expected_answer + asyncio.run(as_optimize_metric(CallableMetric("c", fn)).evaluate("q", "r", reference)) + assert seen["expected"] == {"response": "30 days"} + + def test_sees_example_metadata_and_structured_output_inside_the_fitter(self) -> None: + seen: dict[str, Any] = {} + + def fn(example: AgentExample, prediction: dict[str, Any]) -> float: + seen["metadata"] = example.metadata + seen["prediction"] = prediction + return 0.5 + + run = RunResult( + query="q", + response="30 days", + metadata={ + "example": {"must_include": ["30 days"]}, + "output": {"response": "30 days", "sources": ["kb"]}, + }, + ) + adapter = as_optimize_metric(CallableMetric("c", fn)) + + async def score() -> EvalResult: + with scoring_run(run): + return await adapter.evaluate("q", "30 days", "30 days") + + assert asyncio.run(score()).score == 0.5 + assert seen["metadata"] == {"must_include": ["30 days"]} + assert seen["prediction"] == {"response": "30 days", "sources": ["kb"]} + + +class TestCompositionAcceptsEitherProtocol: + def test_composite_metric_with_a_class_agent_component(self) -> None: + composite = CompositeMetric( + [ + WeightedMetric(name="judge", metric=_StubJudge(1.0), weight=1), + WeightedMetric(name="terms", metric=ContainsTermsMetric(["30 days"]), weight=1), + ] + ) + result = asyncio.run(composite.evaluate("q", "within 30 days", "30 days")) + assert result.metadata["dimensions"] == {"judge": 1.0, "terms": 1.0} + assert result.score == pytest.approx(1.0) + + def test_metadata_metrics_inside_a_composite_score_the_same_outside_fit(self) -> None: + """``evaluate()``/``compile(metrics=[composite])`` scored a nested + metadata-reading metric 0: only the fitter exposed the example.""" + facts = CallableMetric( + "facts", + lambda ex, pred: float( + all(t in pred["response"] for t in ex.metadata["must_include"]) + ), + ) + composite = CompositeMetric([WeightedMetric(name="facts", metric=facts, weight=1)]) + example = _example(must_include=["30 days"]) + assert composite.score(example, {"response": "within 30 days"}) == pytest.approx(1.0) + assert OptimizeMetricAdapter(composite).score( + example, {"response": "no idea"} + ) == pytest.approx(0.0) + + def test_agents_weighted_metric_with_an_optimize_component(self) -> None: + weighted = AgentWeightedMetric( + [("judge", _StubJudge(0.4), 1), ("em", ExactMatchMetric(), 1)] + ) + score = weighted.score(_example(), {"response": "30 days"}) + assert weighted.last_component_scores["judge"] == pytest.approx(0.4) + assert weighted.last_component_scores["em"] == pytest.approx(1.0) + assert score == pytest.approx(0.7) + + def test_metric_loss_and_resolve_loss_accept_optimize_metrics(self) -> None: + loss = MetricLoss(_StubJudge(0.75)) + assert loss.compute(_example(), {"response": "x"}) == pytest.approx(0.25) + assert resolve_loss(_StubJudge(0.75)) is not None + + def test_resolve_metrics_accepts_instances_of_both_protocols(self) -> None: + resolved = resolve_metrics(["exact_match", ContainsTermsMetric(["a"]), _StubJudge()]) + assert all(callable(getattr(m, "evaluate", None)) for m in resolved) + + def test_compile_and_evaluate_accept_optimize_metrics(self) -> None: + from dataclasses import dataclass + + from agentomatic.agents import AgentDataset, BaseGraphAgent + + @dataclass + class S: + q: str = "" + out: str = "" + + class Echo(BaseGraphAgent[S]): + agent_name = "echo" + + def build_graph(self) -> Any: + g = self.new_graph() + g.add_node("n", self.n) + g.set_entry_point("n") + g.set_finish_point("n") + return g.compile() + + def n(self, s: S) -> S: + s.out = "30 days" + return s + + def input_to_state(self, d: dict[str, Any]) -> S: + return S(q=d.get("current_query", "")) + + def state_to_output(self, s: S) -> dict[str, Any]: + return {"response": s.out} + + agent = Echo() + agent.compile(AgentDataset(examples=[_example()]), metrics=[ExactMatchMetric()]) + report = agent.evaluate([_example()]) + assert report.scores["exact_match"] == pytest.approx(1.0) + + def test_as_agent_metric_rejects_unusable_objects(self) -> None: + with pytest.raises(TypeError, match="not a usable metric"): + as_agent_metric(42) diff --git a/tests/test_no_unhandled_500s.py b/tests/test_no_unhandled_500s.py index f7557c6..ba75717 100644 --- a/tests/test_no_unhandled_500s.py +++ b/tests/test_no_unhandled_500s.py @@ -13,8 +13,10 @@ None of those were caught by testing individual features, because each was only reachable in a configuration no individual test happened to build. This -module instead walks *every* mounted route and asserts none of them 5xx, in the -default no-store posture that a fresh `agentomatic run` actually uses. +module instead walks *every* mounted route and asserts none of them 5xx, both in +the default posture a fresh `agentomatic run` uses (no store configured, so the +platform falls back to a bounded in-memory store) and with that fallback +disabled (``AGENTOMATIC_EPHEMERAL_THREADS=0``, genuinely no store). """ from __future__ import annotations @@ -59,9 +61,8 @@ _CONTROL_TOKEN = "sweep-control-token" -@pytest.fixture(scope="module") -def swept_app(): - """A fully-featured platform in the DEFAULT posture: no store configured.""" +def _build_swept_app(): + """A fully-featured platform with no store configured.""" import tempfile from pathlib import Path @@ -84,6 +85,21 @@ async def echo(state: dict[str, Any]) -> dict[str, Any]: return platform.build() +@pytest.fixture(scope="module", params=["ephemeral-store", "no-store"]) +def swept_app(request): + """Both store postures: the default fallback, and the fallback disabled. + + The flag is read at lifespan start-up, which runs each time a test enters + ``TestClient``, so it stays set for the lifetime of the module-scoped app. + """ + with pytest.MonkeyPatch.context() as mp: + if request.param == "no-store": + mp.setenv("AGENTOMATIC_EPHEMERAL_THREADS", "0") + else: + mp.delenv("AGENTOMATIC_EPHEMERAL_THREADS", raising=False) + yield _build_swept_app() + + def _probe_targets(app) -> list[tuple[str, str, str]]: """Return ``(method, concrete_path, route_template)`` for every route.""" targets: list[tuple[str, str, str]] = [] @@ -136,9 +152,10 @@ def test_the_sweep_covers_a_meaningful_number_of_routes(swept_app) -> None: assert len(targets) >= 80, f"only {len(targets)} route/method pairs probed" -def test_store_dependent_routes_answer_4xx_rather_than_crashing(swept_app) -> None: +def test_store_dependent_routes_answer_4xx_rather_than_crashing(monkeypatch) -> None: """The specific regression: these need a store, and none is configured.""" - with TestClient(swept_app, raise_server_exceptions=False) as client: + monkeypatch.setenv("AGENTOMATIC_EPHEMERAL_THREADS", "0") + with TestClient(_build_swept_app(), raise_server_exceptions=False) as client: for path in ( "/api/v1/a1/optimization-runs", "/api/v1/a1/logs", @@ -150,6 +167,23 @@ def test_store_dependent_routes_answer_4xx_rather_than_crashing(swept_app) -> No ) +def test_default_posture_serves_thread_routes(monkeypatch) -> None: + """A fresh project configures no store — threads must still work. + + Studio's "New Chat" creates a thread before the first message; without a + store that answered 400 "Thread storage not configured", so no chat could + ever start on a freshly scaffolded project. + """ + monkeypatch.delenv("AGENTOMATIC_EPHEMERAL_THREADS", raising=False) + with TestClient(_build_swept_app(), raise_server_exceptions=False) as client: + created = client.post("/api/v1/a1/threads", json={"user_id": "u", "title": "New Chat"}) + assert created.status_code == 200, created.text + thread_id = created.json()["id"] + assert client.get(f"/api/v1/a1/threads/{thread_id}").status_code == 200 + assert client.get("/api/v1/a1/threads").json()["count"] == 1 + assert client.get("/api/v1/a1/optimization-runs").status_code == 200 + + def test_studio_graph_degrades_for_an_agent_without_a_graph(swept_app) -> None: """The Studio UI calls this for *every* agent; a node_fn-only agent has no graph to draw, which must not be a 500. diff --git a/tests/test_optimization_bugs.py b/tests/test_optimization_bugs.py index 0e0ff11..4a8efb2 100644 --- a/tests/test_optimization_bugs.py +++ b/tests/test_optimization_bugs.py @@ -520,9 +520,12 @@ async def evaluate(self, query, response, expected=None, context=None): expected_dict = {"answer": "42"} ex = self._make_example(expected=expected_dict) metric.score(ex, {"response": "r"}) - # expected must be the JSON representation of the dict + # The same reference every path uses (AgentExample.to_datapoint): the + # dict travels as its "Expected structured output" JSON section. + from agentomatic.optimize.metrics import _expected_structured + assert received_expected[0] is not None - assert json.loads(received_expected[0]) == expected_dict + assert _expected_structured(received_expected[0]) == expected_dict def test_score_does_not_raise_on_sub_metric_failure(self): class BrokenMetric(ExactMatchMetric): @@ -748,7 +751,9 @@ async def evaluate(self, query, response, expected=None, context=None): # -- error handling ------------------------------------------------------- - def test_score_returns_neutral_on_exception(self): + def test_score_is_zero_and_counted_on_exception(self): + """A failed judge must not invent a mid-scale score (it hid outages).""" + class FailMetric: name = "fail" @@ -758,9 +763,10 @@ async def evaluate(self, *a, **kw): adapter = OptimizeMetricAdapter(FailMetric()) ex = self._make_example() result = adapter.score(ex, {"response": "r"}) - assert result == pytest.approx(0.5) # neutral, not 0.0 crash + assert result == 0.0 # no crash, and no fabricated 0.5 + assert adapter.failures == 1 - def test_score_returns_neutral_on_sync_exception(self): + def test_score_is_zero_and_counted_on_sync_exception(self): class SyncFail: name = "sync_fail" @@ -770,7 +776,8 @@ def evaluate(self, *a, **kw): adapter = OptimizeMetricAdapter(SyncFail()) ex = self._make_example() result = adapter.score(ex, {"response": "r"}) - assert result == pytest.approx(0.5) + assert result == 0.0 + assert adapter.failures == 1 # -- integration with training stack ------------------------------------- diff --git a/tests/test_optimize_cli_search_space.py b/tests/test_optimize_cli_search_space.py index 9311147..2fb57bd 100644 --- a/tests/test_optimize_cli_search_space.py +++ b/tests/test_optimize_cli_search_space.py @@ -204,7 +204,7 @@ def test_bad_entry_raises(self) -> None: WeightedMetric([(1, 2, 3, 4)]) def test_component_missing_score_raises(self) -> None: - with pytest.raises(TypeError, match="Metric.score protocol"): + with pytest.raises(TypeError, match="not a usable metric"): WeightedMetric([("bad", object(), 1.0)]) def test_bare_metric_defaults_to_weight_one(self) -> None: diff --git a/tests/test_pipeline_condition_errors.py b/tests/test_pipeline_condition_errors.py index 5bbced1..54f27e8 100644 --- a/tests/test_pipeline_condition_errors.py +++ b/tests/test_pipeline_condition_errors.py @@ -158,3 +158,53 @@ def test_on_error_skip_still_lets_the_pipeline_continue(self, tmp_path) -> None: assert body["steps"]["branch"]["status"] == "failed" assert body["steps"]["after"]["status"] == "success" + + +PIPELINE_CONTINUE = """ +name: continue_condition +on_error: continue +steps: + - name: seed + transform: "return {'n': 4}" + - name: branch + condition: "ctx.get_step_output('seed').get('n') > 2 and missing_name" + transform: "return {'branch': 'long'}" + - name: after + transform: "return {'done': True}" +""" + + +class TestPipelineLevelContinue: + def test_pipeline_on_error_continue_applies_to_condition_errors(self, tmp_path) -> None: + """``on_error: continue`` used to be ignored for condition failures.""" + with TestClient(_platform(tmp_path, PIPELINE_CONTINUE).build()) as client: + body = client.post( + "/api/v1/pipelines/continue_condition/run", json={"input": {}} + ).json() + + assert body["steps"]["branch"]["status"] == "failed" + assert body["steps"]["after"]["status"] == "success" + assert body["status"] == "partial" + + +def test_the_scaffolded_pipeline_template_declares_real_contracts() -> None: + """``--template pipeline`` wrote ``input:``/``output:`` (ignored keys) and a + ``ctx.steps..output`` condition that always raised at run time.""" + from agentomatic.cli.templates import _pipeline_yaml + from agentomatic.pipelines.context import PipelineContext + from agentomatic.pipelines.engine import PipelineEngine + from agentomatic.pipelines.loader import PipelineLoader + from agentomatic.pipelines.models import StepResult, StepStatus + + config = PipelineLoader.from_yaml_string(_pipeline_yaml("scaffolded")) + assert config.input_schema == {"query": {"type": "string", "required": True}} + assert config.output_schema is not None + + ctx = PipelineContext(input_data={"query": "q"}) + ctx.steps["classify"] = StepResult( + name="classify", status=StepStatus.SUCCESS, output={"response": "billing"} + ) + engine = PipelineEngine.__new__(PipelineEngine) + for step in config.steps: + if getattr(step, "condition", None): + assert engine._evaluate_condition(step.condition, ctx) is True # noqa: SLF001 diff --git a/tests/test_prompt_optimization_examples.py b/tests/test_prompt_optimization_examples.py new file mode 100644 index 0000000..5c9c71b --- /dev/null +++ b/tests/test_prompt_optimization_examples.py @@ -0,0 +1,150 @@ +# pyright: reportMissingParameterType=none +"""Every script in ``examples/prompt_optimization`` runs end to end. + +Reported against ``train.py``: augmentation added nothing, metrics crashed +(``'OptimizeMetricAdapter' object has no attribute 'evaluate'``), and the +report showed no prompts. Each script runs here as a user would run it — a +subprocess with the shared connection flags — against a deterministic fake +model server, and its outputs are checked for the behaviour it teaches. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent)) + +from fake_openai_server import IMPROVED_PROMPT, FakeOpenAIServer # noqa: E402 + +EXAMPLES = Path(__file__).resolve().parents[1] / "examples" / "prompt_optimization" + + +@pytest.fixture(scope="module") +def server(): + with FakeOpenAIServer() as srv: + yield srv + + +def _run(script: str, server: FakeOpenAIServer, out: Path, *args: str) -> str: + env = {**os.environ, "AGENTOMATIC_LOG_LEVEL": "WARNING"} + proc = subprocess.run( + [ + sys.executable, + str(EXAMPLES / script), + "--base-url", + server.base_url, + "--model", + "fake-model", + "--api-key", + "x", + "--out-dir", + str(out), + *args, + ], + capture_output=True, + text=True, + timeout=300, + env=env, + cwd=str(out), + ) + assert proc.returncode == 0, f"{script} failed:\n{proc.stdout[-3000:]}\n{proc.stderr[-3000:]}" + return proc.stdout + + +class TestHighLevelTrain: + def test_optimizes_evaluates_and_reports(self, server, tmp_path) -> None: + _run("train.py", server, tmp_path, "--epochs", "2", "--trials", "4") + summary = json.loads((tmp_path / "train_summary.json").read_text()) + assert summary["prompt_changed"] is True + assert summary["best_prompt"] == IMPROVED_PROMPT + assert summary["validation"]["after"] > summary["validation"]["before"] + # Measured on the untouched test split, before and after. + for name in ("judge", "facts", "quality"): + assert summary["test_scores_after"][name] > summary["test_scores_before"][name] + + report = json.loads((tmp_path / "train_report.json").read_text()) + assert report["initial_prompt"] != report["final_prompt"] + assert len(report["epochs"]) == 2 + assert len(report["test_examples"]) == 6 + assert (tmp_path / "train_report.html").stat().st_size > 10_000 + + def test_augment_grows_only_train(self, server, tmp_path) -> None: + _run( + "train.py", + server, + tmp_path, + "--epochs", + "1", + "--trials", + "4", + "--augment", + "--n-examples", + "40", + ) + rows = [ + json.loads(line) + for line in (tmp_path / "support.augmented.jsonl").read_text().splitlines() + ] + splits = [r["split"] for r in rows] + assert len(rows) == 40 + assert splits.count("train") == 24 + assert (splits.count("validation"), splits.count("holdout"), splits.count("test")) == ( + 6, + 4, + 6, + ) + report = json.loads((tmp_path / "train_report.json").read_text()) + assert report["dataset_stats"]["added"] == 14 + + +class TestLowLevelFitter: + def test_every_knob_explicit_and_report(self, server, tmp_path) -> None: + _run("low_level_prompt_fitter.py", server, tmp_path, "--trials", "6") + result = json.loads((tmp_path / "low_level_result.json").read_text()) + assert result["best_score"] > result["baseline_score"] + decisions = {t["decision"] for t in result["trials"]} + assert "accepted" in decisions + assert result["settings"]["min_confidence"] == 0.8 + assert result["dataset_sizes"]["holdout"] == 4 + assert (tmp_path / "low_level_report.html").exists() + + +class TestBuildingBlocks: + def test_metrics_and_judges(self, server, tmp_path) -> None: + out = _run("metrics_and_judges.py", server, tmp_path) + assert "── 4. composite objective" in out + # The metadata-driven metric is scored inside the composite too. + good = next( + line + for line in out.splitlines() + if line.strip().startswith("good") and "objective=" in line + ) + assert "facts=1.00" in good + + def test_data_augmentation(self, server, tmp_path) -> None: + _run("data_augmentation.py", server, tmp_path, "--n-examples", "30") + rows = (tmp_path / "support.augmented.jsonl").read_text().splitlines() + assert len(rows) == 30 + + def test_anti_overfitting_rejects_the_memorizing_prompt(self, server, tmp_path) -> None: + out = _run("anti_overfitting.py", server, tmp_path) + lines = out.splitlines() + memo = next( + line + for line in lines + if line.strip().startswith("memorize_validation") and "rejected" in line + ) + general = next( + line + for line in lines + if line.strip().startswith("general_instruction") and "accepted" in line + ) + assert "1.00" in memo and "0.00" in memo # perfect on validation, nothing held out + assert general + assert "kept prompt: general_instruction" in out diff --git a/tests/test_studio_bundle.py b/tests/test_studio_bundle.py new file mode 100644 index 0000000..de83943 --- /dev/null +++ b/tests/test_studio_bundle.py @@ -0,0 +1,65 @@ +"""The prebuilt Studio bundle must be loadable and free of known defects. + +The bundle under ``src/agentomatic/studio/static`` is built from the +``agentomatic-studio`` repository. These checks run on the shipped files: + +* every asset ``index.html`` and the chunks reference exists (assets are + served ``immutable``, so a changed file is re-hashed and every reference + must follow); +* the Chat tab sends the typed message as ``query`` — a string message was + dropped from the request body, so every agent answered an empty question; +* message ids are normalised to strings — SQL / memory stores return integer + ids and ``id.includes(...)`` crashed the chat view; +* operation contracts resolve templated routes, so the Task Board can load + the input form of ``/api/v1/pipelines/{name}/run``. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +STATIC = Path(__file__).resolve().parents[1] / "src" / "agentomatic" / "studio" / "static" +ASSETS = STATIC / "assets" + + +def _entry() -> str: + match = re.search( + r'src="/studio/ui/assets/(index-[\w-]+\.js)"', (STATIC / "index.html").read_text() + ) + assert match, "index.html has no entry script" + return (ASSETS / match.group(1)).read_text(encoding="utf-8") + + +def test_every_referenced_asset_exists() -> None: + names = {p.name for p in ASSETS.iterdir()} + sources = [(STATIC / "index.html").read_text()] + [ + p.read_text(encoding="utf-8") for p in ASSETS.glob("*.js") + ] + referenced = { + ref for text in sources for ref in re.findall(r"(?:assets/|\./)([\w-]+\.(?:js|css))", text) + } + missing = sorted(ref for ref in referenced if ref not in names) + assert not missing, missing + + +def test_chat_sends_a_typed_message_as_query() -> None: + entry = _entry() + # The defect: ``query`` was only sent when the input *object* had one. + buggy = re.search(r"\.\.\.(\w)&&\((\w)\|\|(\w)\.query\)\?\{query:\2\|\|\3\.query\}", entry) + assert buggy is None, "string chat input is dropped from the run request" + + +def test_message_ids_are_strings() -> None: + entry = _entry() + assert "normalizeMessage(e,t){return{id:e?.id||" not in entry + chat = next(ASSETS.glob("ChatView-*.js")).read_text(encoding="utf-8") + assert not re.search(r"\b\w\.id\.includes\(", chat) + + +def test_operation_contracts_match_templated_paths() -> None: + entry = _entry() + # agentomatic-studio ``findPathItem``: a concrete path such as + # /api/v1/pipelines//run matches the published + # /api/v1/pipelines/{name}/run through a ``{placeholder}`` segment test. + assert "/^\\{([^}/]+)\\}$/" in entry, "no templated-path fallback in the contract lookup" diff --git a/tests/test_sync_node_offload.py b/tests/test_sync_node_offload.py new file mode 100644 index 0000000..7f28320 --- /dev/null +++ b/tests/test_sync_node_offload.py @@ -0,0 +1,188 @@ +# pyright: reportMissingParameterType=none +"""Sync graph nodes must not block the event loop. + +Template nodes call ``self.llm.invoke(...)`` — a blocking network call. The +async execution paths ran such nodes inline on the event loop, so while a model +answered the whole server stood still: an ``async`` task submission only +returned once the LLM had finished, and task polling, SSE progress and health +checks stalled with it. Sync nodes now run in a worker thread. +""" + +from __future__ import annotations + +import asyncio +import threading +import time +from dataclasses import dataclass, field +from typing import Any + +import pytest +from fastapi.testclient import TestClient + +from agentomatic import AgentPlatform +from agentomatic.agents import BaseGraphAgent +from agentomatic.storage import MemoryStore + + +@dataclass +class _State: + request: str = "" + output: dict[str, Any] = field(default_factory=dict) + + +class _SlowSyncAgent(BaseGraphAgent[_State]): + """A sync node that blocks like an LLM call and reports task progress.""" + + agent_name = "slow_sync" + agent_description = "blocking sync node" + delay = 0.6 + + def build_graph(self) -> Any: + g = self.new_graph() + g.add_node("work", self.work) + g.set_entry_point("work") + g.set_finish_point("work") + return g.compile() + + def work(self, state: _State) -> _State: + from agentomatic.tasks import report_stage_sync + + report_stage_sync("thinking", percent=50.0) + time.sleep(self.delay) + state.output = { + "response": f"done: {state.request}", + "prompt": self.resolve_system_prompt(default="default prompt"), + "thread": threading.current_thread().name, + } + return state + + def input_to_state(self, data: dict[str, Any]) -> _State: + return _State(request=data.get("current_query", "")) + + def state_to_output(self, state: _State) -> dict[str, Any]: + return state.output + + +@pytest.fixture +def client(tmp_path): + platform = AgentPlatform(agents_dir=tmp_path / "agents", enable_studio=False) + reg = _SlowSyncAgent().as_registered_agent() + platform.register_agent( + manifest=reg.manifest, + node_fn=reg.node_fn, + graph_fn=reg.graph_fn, + class_instance=reg.class_instance, + ) + with TestClient(platform.build()) as c: + yield c + + +class TestEventLoopStaysResponsive: + def test_async_task_submission_returns_before_the_node_finishes(self, client) -> None: + started = time.perf_counter() + response = client.post( + "/api/v1/tasks", + json={"target_type": "agent", "target": "slow_sync", "input": {"query": "x"}}, + ) + elapsed = time.perf_counter() - started + assert response.status_code == 202 + assert elapsed < _SlowSyncAgent.delay / 2, f"submission blocked for {elapsed:.2f}s" + + # While the node sleeps, other requests are still served. + health_started = time.perf_counter() + assert client.get("/health").status_code == 200 + assert time.perf_counter() - health_started < _SlowSyncAgent.delay / 2 + + task_id = response.json()["id"] + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + record = client.get(f"/api/v1/tasks/{task_id}").json() + if record["status"] == "succeeded": + break + time.sleep(0.05) + assert record["status"] == "succeeded", record + assert record["result"]["response"] == "done: x" + + def test_progress_from_an_offloaded_sync_node_reaches_the_task(self, client) -> None: + response = client.post( + "/api/v1/tasks", + json={ + "target_type": "agent", + "target": "slow_sync", + "input": {"query": "x"}, + "mode": "sync", + "wait": True, + }, + ) + task_id = response.json()["id"] + with client.stream("GET", f"/api/v1/tasks/{task_id}/events?since=1") as stream: + body = "".join(stream.iter_text()) + assert '"stage": "thinking"' in body or '"stage":"thinking"' in body, body + + def test_sync_node_runs_off_the_loop_thread(self, client) -> None: + response = client.post("/api/v1/slow_sync/invoke", json={"query": "x"}) + assert response.status_code == 200 + thread_name = response.json()["output"]["thread"] + assert thread_name != "MainThread" + + +class TestRequestPromptIsolation: + async def test_concurrent_runs_keep_their_own_prompt_override(self) -> None: + """Candidates scored concurrently must each see their own prompt.""" + agent = _SlowSyncAgent() + agent.delay = 0.2 + + async def run(prompt: str) -> str: + out = await agent.atransform({"current_query": "q", "system_prompt_override": prompt}) + return out["prompt"] + + results = await asyncio.gather(*(run(f"prompt {i}") for i in range(4))) + assert results == [f"prompt {i}" for i in range(4)] + # Nothing leaks once the runs are over. + assert agent._request_system_prompt is None + + def test_override_is_cleared_after_a_sync_transform(self) -> None: + agent = _SlowSyncAgent() + agent.delay = 0 + out = agent.transform({"current_query": "q", "system_prompt_override": "custom"}) + assert out["prompt"] == "custom" + assert agent.transform({"current_query": "q"})["prompt"] == "default prompt" + + +class TestBoundedMemoryStore: + async def test_oldest_threads_are_evicted_past_the_cap(self) -> None: + store = MemoryStore(max_threads=2) + for i in range(3): + await store.create_thread(f"t{i}", "u", "a") + await store.add_message(f"t{i}", "user", f"m{i}") + await asyncio.sleep(0.001) + assert await store.get_thread("t0") is None + assert await store.get_messages("t0") == [] + assert await store.get_thread("t1") is not None + assert await store.get_thread("t2") is not None + + async def test_recently_updated_thread_survives_eviction(self) -> None: + store = MemoryStore(max_threads=2) + await store.create_thread("old", "u", "a") + await asyncio.sleep(0.001) + await store.create_thread("mid", "u", "a") + await asyncio.sleep(0.001) + await store.add_message("old", "user", "still talking") + await asyncio.sleep(0.001) + await store.create_thread("new", "u", "a") + assert await store.get_thread("old") is not None + assert await store.get_thread("mid") is None + + def test_unbounded_by_default_and_validates_the_cap(self) -> None: + assert MemoryStore().max_threads is None + with pytest.raises(ValueError): + MemoryStore(max_threads=0) + + +def test_build_invoke_state_does_not_mutate_the_payload() -> None: + from agentomatic.core.agent_invoke import build_invoke_state + + payload: dict[str, Any] = {"query": "q", "temperature": 0.3, "metadata": {}, "context": {}} + state = build_invoke_state(payload) + assert state["metadata"] == {"temperature": 0.3} + assert payload["metadata"] == {} diff --git a/tests/test_task_target_failures.py b/tests/test_task_target_failures.py new file mode 100644 index 0000000..f705459 --- /dev/null +++ b/tests/test_task_target_failures.py @@ -0,0 +1,105 @@ +# pyright: reportMissingParameterType=none +"""A task whose target reports failure must be marked ``failed``. + +Studio's Task Board showed ``succeeded`` for a pipeline whose status was +``failed`` and for an endpoint whose every upstream call failed: both return +normally (with the failure inside the result), and the task manager only +looked at whether the dispatcher raised. +""" + +from __future__ import annotations + +import time +from types import SimpleNamespace +from typing import Any + +import pytest +from fastapi.testclient import TestClient + +from agentomatic import AgentPlatform +from agentomatic.tasks import TargetFailedError +from agentomatic.tasks.dispatchers import make_endpoint_dispatcher + +FAILING = """ +name: failing_pipe +steps: + - name: boom + transform: "raise ValueError('upstream exploded')" + - name: after + transform: "return {'done': True}" +""" + +PARTIAL = """ +name: partial_pipe +on_error: continue +steps: + - name: boom + transform: "raise ValueError('upstream exploded')" + - name: after + transform: "return {'done': True}" +""" + + +@pytest.fixture +def client(tmp_path): + agents = tmp_path / "agents" + agents.mkdir() + pipelines = tmp_path / "pipelines" + pipelines.mkdir() + (pipelines / "failing_pipe.yaml").write_text(FAILING) + (pipelines / "partial_pipe.yaml").write_text(PARTIAL) + with TestClient(AgentPlatform(agents_dir=str(agents)).build()) as test_client: + yield test_client + + +def _run_task(client: TestClient, pipeline: str) -> dict[str, Any]: + response = client.post( + "/api/v1/tasks", + json={"target_type": "pipeline", "target": pipeline, "input": {}}, + ) + assert response.status_code == 202, response.text + task_id = response.json()["id"] + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + record = client.get(f"/api/v1/tasks/{task_id}").json() + if record["status"] in {"succeeded", "failed", "cancelled"}: + return record + time.sleep(0.05) + raise AssertionError("task did not finish") + + +class TestPipelineTasks: + def test_a_failed_pipeline_fails_its_task_and_keeps_the_details(self, client) -> None: + record = _run_task(client, "failing_pipe") + assert record["status"] == "failed" + assert "Pipeline 'failing_pipe' failed" in record["error"] + assert "boom" in record["error"] + assert record["result"]["steps"]["boom"]["status"] == "failed" + + def test_a_partial_pipeline_still_succeeds(self, client) -> None: + record = _run_task(client, "partial_pipe") + assert record["status"] == "succeeded" + assert record["result"]["status"] == "partial" + + +class TestEndpointTasks: + async def test_all_upstreams_failing_raises(self) -> None: + class _Schema: + @staticmethod + def model_validate(payload: Any) -> Any: + return payload + + async def handle(request: Any) -> dict[str, Any]: + return {"endpoint": "e", "ok": False, "results": [{"ok": False}], "aggregated": None} + + endpoint = SimpleNamespace(get_input_schema=lambda: _Schema, handle=handle) + registry = SimpleNamespace(get=lambda name: endpoint, list_names=lambda: ["e"]) + ctx = SimpleNamespace(report=_noop) + + with pytest.raises(TargetFailedError) as info: + await make_endpoint_dispatcher(registry)("e", {}, ctx) # type: ignore[arg-type] + assert info.value.result["ok"] is False + + +async def _noop(**_: Any) -> None: + return None diff --git a/tests/test_template_chat_threads_tasks.py b/tests/test_template_chat_threads_tasks.py new file mode 100644 index 0000000..665182e --- /dev/null +++ b/tests/test_template_chat_threads_tasks.py @@ -0,0 +1,350 @@ +# pyright: reportMissingParameterType=none +# pyright: reportAttributeAccessIssue=none +"""Every chat-capable template must chat, thread and run as a task out of the box. + +Regression for a freshly scaffolded project (``agentomatic new`` + +``agentomatic init NAME --template basic|full|...``) where talking to the agent +from Studio failed with ``400``: + +* Studio's "New Chat" creates a thread first (``POST /api/v1/{agent}/threads``). + No store is configured in a fresh project, so every thread route answered + ``400 "Thread storage not configured"`` and no conversation could start. +* Studio runs never loaded the thread's history nor saved the turn, so a + chatbot answered each message as if it were the first and a reopened thread + was empty. +* The ``basic`` / ``full`` templates sent the model a single string and ignored + ``state.messages`` entirely. + +The project is scaffolded with the real CLI and served from its own +``main.py`` — the same app ``agentomatic run`` / ``uvicorn main:app`` serve. +""" + +from __future__ import annotations + +import sys +import time +from pathlib import Path +from typing import Any + +import pytest +from click.testing import CliRunner +from fastapi.testclient import TestClient + +from agentomatic.cli.commands import cli +from agentomatic.cli.project import scaffold_project + +#: template -> agent name. Names are unique so their ``agents.`` modules +#: cannot collide with other suites' scaffolds in the same interpreter. +TEMPLATES = { + "basic": "tct_basic", + "full": "tct_full", + "chatbot": "tct_chat", + "rag": "tct_rag", + "coordinator": "tct_coord", + "extraction": "tct_extract", +} + +#: Templates whose single LLM call is a conversation (system + turns). +HISTORY_AWARE = ("basic", "full", "chatbot") + + +class RecordingLLM: + """Stand-in chat model that records every payload it is sent.""" + + def __init__(self) -> None: + self.calls: list[Any] = [] + + def invoke(self, payload: Any, **_: Any) -> Any: + self.calls.append(payload) + + class _Reply: + content = f"reply #{len(self.calls)}" + + return _Reply() + + +def _texts(payload: Any) -> list[str]: + """Flatten an LLM payload (str, dicts or LangChain messages) to texts.""" + if isinstance(payload, str): + return [payload] + out: list[str] = [] + for item in payload: + content = item.get("content") if isinstance(item, dict) else getattr(item, "content", "") + out.append(str(content)) + return out + + +def _purge_agent_modules() -> None: + for mod in [m for m in sys.modules if m == "agents" or m.startswith("agents.")]: + sys.modules.pop(mod, None) + + +@pytest.fixture(scope="module") +def project_app(tmp_path_factory): + """Scaffold a project with one agent per template and build its ``main:app``.""" + root = tmp_path_factory.mktemp("tpl_chat") + project = root / "proj" + scaffold_project(project, "proj", force=True) + runner = CliRunner() + with pytest.MonkeyPatch.context() as mp: + mp.chdir(project) + mp.delenv("AGENTOMATIC_EPHEMERAL_THREADS", raising=False) + mp.setenv("AGENTOMATIC_LOG_LEVEL", "WARNING") + for template, name in TEMPLATES.items(): + result = runner.invoke(cli, ["init", name, "--template", template]) + assert result.exit_code == 0, result.output + _purge_agent_modules() + mp.syspath_prepend(str(project)) + namespace: dict[str, Any] = {} + source = (project / "main.py").read_text() + exec(compile(source, str(project / "main.py"), "exec"), namespace) # noqa: S102 + platform = namespace["_platform"] + llms: dict[str, RecordingLLM] = {} + for name in TEMPLATES.values(): + agent = platform._registry.get(name) + assert agent is not None, f"agent {name} was not discovered" + llms[name] = RecordingLLM() + agent.class_instance.llm = llms[name] + with TestClient(namespace["app"]) as client: + yield client, llms + _purge_agent_modules() + + +def _stream_turn(client: TestClient, agent: str, thread_id: str, query: str) -> list[str]: + """Send one Studio chat turn and return its SSE data frames.""" + with client.stream( + "POST", + f"/studio/agents/{agent}/runs/stream", + json={"query": query, "thread_id": thread_id, "user_id": "studio-user"}, + ) as response: + assert response.status_code == 200 + return [line for line in response.iter_lines() if line.startswith("data:")] + + +def _wait(client: TestClient, task_id: str, timeout: float = 20.0) -> dict[str, Any]: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + record = client.get(f"/api/v1/tasks/{task_id}").json() + if record["status"] in {"succeeded", "failed", "cancelled"}: + return record + time.sleep(0.05) + raise AssertionError(f"task {task_id} did not finish") + + +@pytest.mark.parametrize("template", list(TEMPLATES)) +class TestStudioChat: + def test_new_chat_thread_can_be_created(self, project_app, template: str) -> None: + client, _ = project_app + agent = TEMPLATES[template] + response = client.post( + f"/api/v1/{agent}/threads", json={"user_id": "studio-user", "title": "New Chat"} + ) + assert response.status_code == 200, response.text + thread_id = response.json()["id"] + assert client.get(f"/api/v1/{agent}/threads/{thread_id}").status_code == 200 + listed = client.get(f"/api/v1/{agent}/threads").json() + assert thread_id in {t["id"] for t in listed["threads"]} + + def test_studio_turns_are_persisted(self, project_app, template: str) -> None: + client, _ = project_app + agent = TEMPLATES[template] + thread_id = client.post(f"/api/v1/{agent}/threads", json={"title": "t"}).json()["id"] + for query in ("My name is Ada.", "What is my name?"): + frames = _stream_turn(client, agent, thread_id, query) + assert not any('"run_error"' in f for f in frames), frames + messages = client.get(f"/api/v1/{agent}/threads/{thread_id}/messages").json()["messages"] + assert [m["role"] for m in messages] == ["user", "assistant", "user", "assistant"] + assert messages[0]["content"] == "My name is Ada." + assert messages[1]["content"].strip() + + def test_thread_lifecycle_routes(self, project_app, template: str) -> None: + client, _ = project_app + agent = TEMPLATES[template] + thread_id = client.post(f"/api/v1/{agent}/threads", json={"title": "t"}).json()["id"] + _stream_turn(client, agent, thread_id, "hello") + renamed = client.patch(f"/api/v1/{agent}/threads/{thread_id}", json={"title": "Renamed"}) + assert renamed.status_code == 200 + forked = client.post( + f"/api/v1/{agent}/threads/{thread_id}/fork", json={"message_index": 0} + ) + assert forked.status_code == 200, forked.text + assert client.delete(f"/api/v1/{agent}/threads/{thread_id}").status_code == 200 + + +@pytest.mark.parametrize("template", HISTORY_AWARE) +def test_prior_turns_reach_the_model_in_studio(project_app, template: str) -> None: + client, llms = project_app + agent = TEMPLATES[template] + thread_id = client.post(f"/api/v1/{agent}/threads", json={"title": "t"}).json()["id"] + _stream_turn(client, agent, thread_id, "Remember this token: ZX42QV") + _stream_turn(client, agent, thread_id, "What token did I give you?") + last = _texts(llms[agent].calls[-1]) + assert any("ZX42QV" in text for text in last), last + assert last[-1] == "What token did I give you?" + + +@pytest.mark.parametrize("template", ("basic", "full")) +def test_basic_templates_send_a_real_system_message(project_app, template: str) -> None: + client, llms = project_app + agent = TEMPLATES[template] + response = client.post(f"/api/v1/{agent}/invoke", json={"query": "ping"}) + assert response.status_code == 200, response.text + payload = llms[agent].calls[-1] + assert isinstance(payload, list) + assert payload[0]["role"] == "system" + assert payload[0]["content"].strip() + assert _texts(payload)[-1] == "ping" + + +@pytest.mark.parametrize("template", ("basic", "full")) +def test_studio_debug_shows_state_and_history(project_app, template: str) -> None: + """The Debug view's State / Time-travel tabs were empty for class agents.""" + client, _ = project_app + agent = TEMPLATES[template] + thread_id = client.post(f"/api/v1/{agent}/threads", json={"title": "t"}).json()["id"] + _stream_turn(client, agent, thread_id, "debug me") + + state = client.get(f"/studio/agents/{agent}/threads/{thread_id}/state").json() + assert state["state"], state + assert state["checkpoint_id"] + history = client.get(f"/studio/agents/{agent}/threads/{thread_id}/history").json() + assert history, "no checkpoints recorded" + assert all(c["metadata"].get("node") for c in history) + assert history[0]["id"] == state["checkpoint_id"] # newest first + + +def test_agent_folder_pipelines_are_served(project_app) -> None: + """``agents//pipeline.yaml`` (the extraction template ships one) was + never discovered: the platform scanned the agents folder as a project root.""" + client, _ = project_app + listed = client.get("/api/v1/pipelines").json() + listed = listed["pipelines"] if isinstance(listed, dict) else listed + names = {p["name"] for p in listed} + assert TEMPLATES["extraction"] in names + + +def test_feedback_export_uses_the_default_store(project_app) -> None: + """With the default (ephemeral) store configured, export returned nothing: + the store dropped the query, response and correction.""" + client, _ = project_app + agent = TEMPLATES["basic"] + recorded = client.post( + f"/api/v1/{agent}/feedback", + json={ + "query": "What is the refund window?", + "response": "No idea.", + "correction": "30 days for annual plans.", + "rating": 1, + "user_id": "u", + }, + ) + assert recorded.status_code == 200, recorded.text + export = client.get(f"/api/v1/{agent}/feedback/export").json() + assert export["count"] >= 1 + rows = [line for line in export["data"].splitlines() if "refund window" in line] + assert rows and "30 days for annual plans." in rows[0] + + +@pytest.mark.parametrize("template", list(TEMPLATES)) +class TestTasks: + def test_async_task_succeeds(self, project_app, template: str) -> None: + client, _ = project_app + agent = TEMPLATES[template] + response = client.post( + "/api/v1/tasks", + json={"target_type": "agent", "target": agent, "input": {"query": "hello"}}, + ) + assert response.status_code == 202, response.text + record = _wait(client, response.json()["id"]) + assert record["status"] == "succeeded", record + assert record["result"] + + def test_sync_task_succeeds(self, project_app, template: str) -> None: + client, _ = project_app + agent = TEMPLATES[template] + response = client.post( + "/api/v1/tasks", + json={ + "target_type": "agent", + "target": agent, + "input": {"query": "hello"}, + "mode": "sync", + "wait": True, + }, + ) + assert response.status_code == 200, response.text + assert response.json()["status"] == "succeeded", response.json() + + def test_studio_task_board_defaults_are_accepted(self, project_app, template: str) -> None: + """The Task Board submits the defaults of the published input schema.""" + client, _ = project_app + agent = TEMPLATES[template] + schema = client.get(f"/studio/agents/{agent}/schemas").json()["input_schema"] + payload: dict[str, Any] = {} + for key, prop in (schema.get("properties") or {}).items(): + if "default" in prop: + payload[key] = prop["default"] + elif prop.get("type") == "object": + payload[key] = {} + payload["query"] = "from the task board" + response = client.post( + "/api/v1/tasks", + json={ + "target_type": "agent", + "target": agent, + "input": payload, + "mode": "sync", + "wait": True, + "retry": {"max_attempts": 3, "backoff": "exponential", "base_delay": 1}, + }, + ) + assert response.status_code == 200, response.text + assert response.json()["status"] == "succeeded", response.json() + + def test_task_input_is_not_mutated_by_the_run(self, project_app, template: str) -> None: + client, _ = project_app + agent = TEMPLATES[template] + response = client.post( + "/api/v1/tasks", + json={ + "target_type": "agent", + "target": agent, + "input": {"query": "hi", "metadata": {}}, + "mode": "sync", + "wait": True, + }, + ) + stored = response.json()["input"] + # A custom request schema (``full``) may not publish ``metadata``. + assert stored.get("metadata", {}) == {} + + +def test_a2a_task_round_trip(project_app) -> None: + client, _ = project_app + agent = TEMPLATES["basic"] + response = client.post(f"/api/v1/{agent}/a2a/tasks", json={"message": {"content": "hi"}}) + assert response.status_code == 200, response.text + task_id = response.json()["task_id"] + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + view = client.get(f"/api/v1/{agent}/a2a/tasks/{task_id}").json() + if view["status"] in {"completed", "failed", "canceled"}: + break + time.sleep(0.05) + assert view["status"] == "completed", view + + +def test_ephemeral_store_can_be_disabled(tmp_path: Path, monkeypatch) -> None: + """``AGENTOMATIC_EPHEMERAL_THREADS=0`` restores the explicit no-store posture.""" + from agentomatic import AgentManifest, AgentPlatform + + async def echo(state: dict[str, Any]) -> dict[str, Any]: + return {"response": "ok"} + + monkeypatch.setenv("AGENTOMATIC_EPHEMERAL_THREADS", "0") + platform = AgentPlatform(agents_dir=tmp_path / "agents") + platform.register_agent(manifest=AgentManifest(name="e1", slug="e1"), node_fn=echo) + with TestClient(platform.build()) as client: + response = client.post("/api/v1/e1/threads", json={"title": "t"}) + assert response.status_code == 400 + assert response.json()["detail"] == "Thread storage not configured" diff --git a/tests/test_train_api_and_holysheet.py b/tests/test_train_api_and_holysheet.py index 976caa2..edbab54 100644 --- a/tests/test_train_api_and_holysheet.py +++ b/tests/test_train_api_and_holysheet.py @@ -100,7 +100,9 @@ def test_build_default_metrics_shape(self) -> None: ) assert len(metrics) == 5 assert getattr(loss, "name", None) or True - assert fit_metric.name == "composite" + # Candidates are selected on the same blend the loss reports. + assert fit_metric.name == "objective" + assert fit_metric.metric is loss.metric def test_compile_fit_evaluate_pipeline(self) -> None: from agentomatic.agents.types import AgentDataset, AgentExample @@ -301,8 +303,16 @@ def _eval(*_a: Any, **_k: Any) -> _Report: ) result = train_api_mod.run_train(agent, config=cfg, dataset=ds) - assert calls == ["compile", "fit", "evaluate"] + # The held-out split is scored before AND after fit (before/after report). + assert calls == ["compile", "evaluate", "fit", "evaluate"] assert result.eval_scores == {"m": 0.7} + assert result.baseline_eval_scores == {"m": 0.7} + + calls.clear() + cfg.evaluate_baseline = False + result = train_api_mod.run_train(agent, config=cfg, dataset=ds) + assert calls == ["compile", "fit", "evaluate"] + assert result.baseline_eval_scores == {} assert result.optimizer == "rewrite" @@ -430,8 +440,12 @@ def _walk(items: list) -> list: if b.get("type") == "section" } assert sections.get("Run Configuration"), "Run Configuration section empty" - assert sections.get("Key Results"), "Key Results section empty" + assert sections.get("Verdict"), "Verdict section empty" + assert sections.get("What changed in the prompt"), "prompt diff section empty" + assert sections.get("All candidates"), "candidates section empty" assert sections.get("Recommendations"), "Recommendations section empty" + # HolySheet's viewer drops Accordion panels — nothing may hide in one. + assert not [b for b in flat if b.get("type") == "accordion"] # HolySheet interactive bundle is large; fallback is still richer than a stub. assert len(html) > 2000 diff --git a/tests/test_train_pipeline_fixes.py b/tests/test_train_pipeline_fixes.py index 71cdce4..9e1e4a7 100644 --- a/tests/test_train_pipeline_fixes.py +++ b/tests/test_train_pipeline_fixes.py @@ -244,7 +244,7 @@ def test_invoke_stream_emits_node_frames(self, tmp_path) -> None: class TestFitterEvalHonesty: @pytest.mark.asyncio - async def test_evaluate_config_skips_failed_judge(self, monkeypatch) -> None: + async def test_evaluate_config_counts_failed_judge_as_zero(self, monkeypatch) -> None: from agentomatic.optimize.dataset import Dataset from agentomatic.optimize.fitter import PromptFitter from agentomatic.optimize.runner import RunResult @@ -287,6 +287,8 @@ async def _eval(query, response, expected=None, context=None): ), metric, ) - # Only the successful point contributes → avg 1.0, not 0.5. - assert avg == pytest.approx(1.0) + # The failed judge call counts as 0.0 (not skipped): skipping it let a + # candidate that breaks the judge on hard inputs outscore one that + # answers them. The failure stays visible in the details. + assert avg == pytest.approx(0.5) assert any(d.get("error") for d in details) diff --git a/uv.lock b/uv.lock index 31195ca..3a060f7 100644 --- a/uv.lock +++ b/uv.lock @@ -10,7 +10,7 @@ resolution-markers = [ [[package]] name = "agentomatic" -version = "1.13.0" +version = "1.14.0" source = { editable = "." } dependencies = [ { name = "click" }, @@ -458,15 +458,35 @@ wheels = [ [[package]] name = "anyio" -version = "4.13.0" +version = "4.14.2" source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version == '3.14.*'", + "python_full_version == '3.13.*'", + "python_full_version < '3.13'", +] dependencies = [ - { name = "idna" }, + { name = "idna", marker = "python_full_version < '3.15'" }, { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/19/14/2c5dd9f512b66549ae92767a9c7b330ae88e1932ca57876909410251fe13/anyio-4.13.0.tar.gz", hash = "sha256:334b70e641fd2221c1505b3890c69882fe4a2df910cba14d97019b90b24439dc", size = 231622, upload-time = "2026-03-24T12:59:09.671Z" } +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" }, + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, +] + +[[package]] +name = "anyio" +version = "4.15.1" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.15'", +] +dependencies = [ + { name = "idna", marker = "python_full_version >= '3.15'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/d2/f4d173e22df740bc37b1db102b386ba719b66e95b0f0d751f556b387e6d2/anyio-4.15.1.tar.gz", hash = "sha256:9f28306018cbd6d329e64a36d58256edff76dd996fe423bc957326e578b82a94", size = 276966, upload-time = "2026-09-05T10:42:39.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/b8/4bd346e22b28902df4d651910f5242c28d84e4a5c2435ca5c3f797ed7e2e/anyio-4.15.1-py3-none-any.whl", hash = "sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101", size = 132079, upload-time = "2026-09-05T10:42:37.923Z" }, ] [[package]] @@ -523,7 +543,8 @@ name = "asyncer" version = "0.0.17" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "sniffio" }, { name = "typing-extensions", marker = "python_full_version < '3.15'" }, ] @@ -1706,7 +1727,8 @@ name = "google-genai" version = "2.8.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "distro" }, { name = "google-auth", extra = ["requests"] }, { name = "httpx" }, @@ -1993,7 +2015,8 @@ name = "httpx" version = "0.28.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "certifi" }, { name = "httpcore" }, { name = "idna" }, @@ -2520,7 +2543,8 @@ name = "langsmith" version = "0.11.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "distro" }, { name = "httpx" }, { name = "orjson", marker = "platform_python_implementation != 'PyPy'" }, @@ -2772,7 +2796,8 @@ name = "mcp" version = "1.29.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "httpx" }, { name = "httpx-sse" }, { name = "jsonschema" }, @@ -3353,7 +3378,8 @@ name = "openai" version = "2.41.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "distro" }, { name = "httpx" }, { name = "jiter" }, @@ -4846,11 +4872,11 @@ wheels = [ [[package]] name = "pyjwt" -version = "2.13.0" +version = "2.15.1" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/3b/81/58d0ac84e1ef3a3843791d6954d94c0b33d526c75eeb1efbce9d0a4c4077/pyjwt-2.13.0.tar.gz", hash = "sha256:41571c89ca91598c79e8ef18a2d07367d4810fbbd6f637794879baf1b7703423", size = 107515, upload-time = "2026-05-21T19:54:36.618Z" } +sdist = { url = "https://files.pythonhosted.org/packages/43/ea/5194e52748b0da83d71e082d75496eaec6e58f419f5e184786ded517e6a9/pyjwt-2.15.1.tar.gz", hash = "sha256:4f259e80cdfb6b3fc18a7de51fd1ef9ec79652f25019bae68975ca2468a34df8", size = 121252, upload-time = "2026-09-28T18:40:42.598Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a3/5e/ecf12fdb62546d64385c158514e9b2b671f7832108ef2ecd2020ce0af2d1/pyjwt-2.13.0-py3-none-any.whl", hash = "sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728", size = 31274, upload-time = "2026-05-21T19:54:35.362Z" }, + { url = "https://files.pythonhosted.org/packages/50/ca/44de4e75f8aadc457f0634be3b542815078ded46dca30efb960edeecad6e/pyjwt-2.15.1-py3-none-any.whl", hash = "sha256:42d59d631f7768a1028a64c7ff581a9bf7519804daf91fc5b6c56e30eec5e193", size = 33860, upload-time = "2026-09-28T18:40:41.429Z" }, ] [package.optional-dependencies] @@ -5058,15 +5084,14 @@ wheels = [ [[package]] name = "python-discovery" -version = "1.4.2" +version = "1.6.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "filelock" }, - { name = "platformdirs" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/0b/1a/cbbaf13b730abb0a16b964d984e19f2fe520c21a4dc664051359a3f5a9e7/python_discovery-1.4.2.tar.gz", hash = "sha256:8f3746c4b4968d22afbb97d36e1a0e5b66e6c0f297290f2e95f05b9b8bf18690", size = 70277, upload-time = "2026-06-11T16:10:42.383Z" } +sdist = { url = "https://files.pythonhosted.org/packages/0c/57/250bd238b966cece44328235eb85290045d059265fdaf7527a3a958123db/python_discovery-1.6.1.tar.gz", hash = "sha256:cf87d3627dfb4412437fdd5b13eae402607722998d21567993aedbc59b23c15e", size = 84338, upload-time = "2026-09-18T01:31:53.971Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/1a/82/a70006589557f267f15bd384c0642ad49f0d97b690c3a05b166b9dcbad3b/python_discovery-1.4.2-py3-none-any.whl", hash = "sha256:475803f53b7b2ed6e490e27373f9d8340f7d2eebf9acdaf645d7d714c97bb500", size = 33886, upload-time = "2026-06-11T16:10:41.192Z" }, + { url = "https://files.pythonhosted.org/packages/16/7d/e9ffbadfbf89c93848412d04594135c4ae8c1d37d9e053b9c3ed718fabc4/python_discovery-1.6.1-py3-none-any.whl", hash = "sha256:d43fcdef879fe795352bd13ccf8d185ba5a9f86f36cfcd00529f596e737442b3", size = 38664, upload-time = "2026-09-18T01:31:52.448Z" }, ] [[package]] @@ -5752,7 +5777,8 @@ name = "sse-starlette" version = "3.4.4" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "starlette" }, ] sdist = { url = "https://files.pythonhosted.org/packages/f7/2b/58abc2d1fd397e7dde08e947e05c884d8ef2f78d5e2588c17a12d42d6994/sse_starlette-3.4.4.tar.gz", hash = "sha256:07e0fa0460138baf25cdd5fb28683472c3995dc1642225191b3832d62526bcb0", size = 31819, upload-time = "2026-05-12T17:37:17.019Z" } @@ -5765,7 +5791,8 @@ name = "starlette" version = "1.6.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/b5/b4/205b0d5241d934e8add0c38aa924c4f9fb7330834ff11e5444db964ec3f9/starlette-1.6.0.tar.gz", hash = "sha256:d4e3ac5e546444960c710297a3c9fc3f7ebae1b7e963f3d36173b49da535be9b", size = 2716969, upload-time = "2026-08-08T18:27:57.512Z" } @@ -6075,11 +6102,11 @@ wheels = [ [[package]] name = "urllib3" -version = "2.7.0" +version = "2.8.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/05/b17359e1cefb4f909b5e40b1b90a496d987258916dbbf88e842c729f510e/urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63", size = 458972, upload-time = "2026-09-15T19:29:36.253Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, + { url = "https://files.pythonhosted.org/packages/92/9d/c4e665119135114480843e7ab388fa94d8480650450e6f8e26b70d323a4c/urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", size = 135717, upload-time = "2026-09-15T19:29:34.577Z" }, ] [[package]] @@ -6264,17 +6291,18 @@ wheels = [ [[package]] name = "virtualenv" -version = "21.4.3" +version = "21.14.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "distlib" }, { name = "filelock" }, + { name = "packaging" }, { name = "platformdirs" }, { name = "python-discovery" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/4b/50/7564c805bb8966d9771caaba8a143fa5e57c848ce4e7fdf2d55a1feb2ead/virtualenv-21.4.3.tar.gz", hash = "sha256:938ff0fd3f4e0f0d3a025f67a3d2f25e3c3aabbcd5857ea6170619138d72d141", size = 7644454, upload-time = "2026-06-11T16:47:04.843Z" } +sdist = { url = "https://files.pythonhosted.org/packages/50/67/b5d37693e5e666b68100db8fe34f00db28279330db55f739a3c799ff2449/virtualenv-21.14.1.tar.gz", hash = "sha256:719b189804e66678017d9f63bbfc590c44f6b96ab4829513806394f9b375929c", size = 5462838, upload-time = "2026-09-29T23:33:33.991Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a2/8d/84b0d07c6b5f685f85ddf6c87a59d3a8a895a3dfd89e759666fabe951b94/virtualenv-21.4.3-py3-none-any.whl", hash = "sha256:75f4127d4067397c64f38579ce918fec6bf9ca2cd4f48685e82952cc3c035840", size = 7625544, upload-time = "2026-06-11T16:47:01.78Z" }, + { url = "https://files.pythonhosted.org/packages/85/a8/22be2d14ef0d7b1e9e7dc1d839a85073fc20917be1747d6f1347d1cfc8c3/virtualenv-21.14.1-py3-none-any.whl", hash = "sha256:6fd04089fc0dc33549e7abdff70fc3b63d4e15799f2dbf3281f80d13b9fce522", size = 5486534, upload-time = "2026-09-29T23:33:32.333Z" }, ] [[package]] @@ -6309,7 +6337,8 @@ name = "watchfiles" version = "1.2.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "anyio" }, + { name = "anyio", version = "4.14.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.15'" }, + { name = "anyio", version = "4.15.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.15'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cd/41/5e1a4bb12aac5f1493fa1bdc11154eca3b258ca4eba65d39c473fe19d8e9/watchfiles-1.2.0.tar.gz", hash = "sha256:c995fba777f1ea992f090f9236e9284cf7a5d1a0130dd5a3d82c598cacd76838", size = 108252, upload-time = "2026-05-18T04:32:04.251Z" } wheels = [