From 6e4fe428d56394e4ca9b7f48aeb3e99c9aa845cd Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Fri, 17 Jul 2026 16:14:59 -0400 Subject: [PATCH] docs: add benchmark evidence to org profile; register openadapt-bootstrap Adds a compact Evidence section between the quickstart and the six-surface map: the live OpenEMR result (20/20, 39.2s p50, zero model calls) and the CI-reproducible MockMed control (100/100 vs 20/20; cost/latency, not success rate), each with its published caveats and a pointer to LIMITS.md. Also registers openadapt-bootstrap as Internal in the lifecycle registry; it was previously unclassified. Co-Authored-By: Claude Fable 5 --- REPOSITORY_LIFECYCLE.md | 2 +- profile/README.md | 22 ++++++++++++++++++++++ repository-lifecycle.yml | 1 + 3 files changed, 24 insertions(+), 1 deletion(-) diff --git a/REPOSITORY_LIFECYCLE.md b/REPOSITORY_LIFECYCLE.md index 21e9c4a..ed88610 100644 --- a/REPOSITORY_LIFECYCLE.md +++ b/REPOSITORY_LIFECYCLE.md @@ -42,7 +42,7 @@ The machine-readable source is [`repository-lifecycle.yml`](repository-lifecycle | Group | Repositories | |-------|--------------| | **Research** | `openadapt-ml`, `openadapt-evals`, `openadapt-retrieval`, `openadapt-grounding`, `openadapt-verifier` | -| **Internal** | `openadapt-ops`, `openadapt-wright`, `openadapt-herald`, `openadapt-crier`, `openadapt-consilium`, `openadapt-presenter`, `openadapt-telemetry`, `openadapt-viewer`, `openadapt-blog`, `openadapt-internal`, `openadapt-yc` | +| **Internal** | `openadapt-ops`, `openadapt-wright`, `openadapt-herald`, `openadapt-crier`, `openadapt-consilium`, `openadapt-presenter`, `openadapt-bootstrap`, `openadapt-telemetry`, `openadapt-viewer`, `openadapt-blog`, `openadapt-internal`, `openadapt-yc` | | **Experimental UI/support** | `openadapt-console`, `openadapt-tray` | | **Labs/forks** | `OmniMCP` (`omnimcp` locally), `SoM`, `PydanticPrompt` | | **Historical directions** | `OpenAdapter`, `OpenReflector` | diff --git a/profile/README.md b/profile/README.md index a1746b5..e91ebe0 100644 --- a/profile/README.md +++ b/profile/README.md @@ -30,6 +30,28 @@ This runs the bundled MockMed example and writes a human-readable `REPORT.md`. Use the [five-minute guide](https://docs.openadapt.ai/get-started/) to add lint, policy certification, drift, repair, and deployment. +## Evidence + +Published comparisons of compiled replay against a computer-use agent, with the +same external success check on both arms: + +- **Live third-party EMR** (OpenEMR public demo, fake patients, 18-step + add-patient-note workflow): compiled replay went **20/20 at 39.2s p50 with + zero model calls**; the agent went 10/10 at 70.4s p50 at about $0.55 of model + charge per run. Small sample on a shared, daily-resetting demo — not + CI-reproducible. + [Methodology and caveats](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/benchmark/openemr/BENCHMARK.md). +- **CI-reproducible control** (bundled MockMed task): both arms passed every + run (100/100 compiled, 20/20 agent), so the result is cost and latency, not + success rate — 4.9s p50 with zero model calls versus 37.5s p50 for the agent. + [Methodology and caveats](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/benchmark/BENCHMARK.md). + +Zero model calls on a healthy run means no model-API charge on that run; it +excludes authoring, review, maintenance, and infrastructure, and it is not a +production-reliability or clinical-safety claim. Read the +[limits](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/LIMITS.md) +before extrapolating either result. + ## Six Public Surfaces | Surface | Lifecycle | Start here | diff --git a/repository-lifecycle.yml b/repository-lifecycle.yml index 1146813..7d9724c 100644 --- a/repository-lifecycle.yml +++ b/repository-lifecycle.yml @@ -29,6 +29,7 @@ lifecycle: - openadapt-crier - openadapt-consilium - openadapt-presenter + - openadapt-bootstrap - openadapt-telemetry - openadapt-viewer - openadapt-blog