Skip to content

docs: add benchmark evidence to org profile; register openadapt-bootstrap #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
abrichr merged 1 commit into from
Jul 17, 2026
Merged

docs: add benchmark evidence to org profile; register openadapt-bootstrap #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion REPOSITORY_LIFECYCLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The machine-readable source is [`repository-lifecycle.yml`](repository-lifecycle
| Group | Repositories |
|-------|--------------|
| **Research** | `openadapt-ml`, `openadapt-evals`, `openadapt-retrieval`, `openadapt-grounding`, `openadapt-verifier` |
| **Internal** | `openadapt-ops`, `openadapt-wright`, `openadapt-herald`, `openadapt-crier`, `openadapt-consilium`, `openadapt-presenter`, `openadapt-telemetry`, `openadapt-viewer`, `openadapt-blog`, `openadapt-internal`, `openadapt-yc` |
| **Internal** | `openadapt-ops`, `openadapt-wright`, `openadapt-herald`, `openadapt-crier`, `openadapt-consilium`, `openadapt-presenter`, `openadapt-bootstrap`, `openadapt-telemetry`, `openadapt-viewer`, `openadapt-blog`, `openadapt-internal`, `openadapt-yc` |
| **Experimental UI/support** | `openadapt-console`, `openadapt-tray` |
| **Labs/forks** | `OmniMCP` (`omnimcp` locally), `SoM`, `PydanticPrompt` |
| **Historical directions** | `OpenAdapter`, `OpenReflector` |
Expand Down
22 changes: 22 additions & 0 deletions profile/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,28 @@ This runs the bundled MockMed example and writes a human-readable `REPORT.md`.
Use the [five-minute guide](https://docs.openadapt.ai/get-started/) to add lint,
policy certification, drift, repair, and deployment.

## Evidence

Published comparisons of compiled replay against a computer-use agent, with the
same external success check on both arms:

- **Live third-party EMR** (OpenEMR public demo, fake patients, 18-step
add-patient-note workflow): compiled replay went **20/20 at 39.2s p50 with
zero model calls**; the agent went 10/10 at 70.4s p50 at about $0.55 of model
charge per run. Small sample on a shared, daily-resetting demo — not
CI-reproducible.
[Methodology and caveats](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/benchmark/openemr/BENCHMARK.md).
- **CI-reproducible control** (bundled MockMed task): both arms passed every
run (100/100 compiled, 20/20 agent), so the result is cost and latency, not
success rate — 4.9s p50 with zero model calls versus 37.5s p50 for the agent.
[Methodology and caveats](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/benchmark/BENCHMARK.md).

Zero model calls on a healthy run means no model-API charge on that run; it
excludes authoring, review, maintenance, and infrastructure, and it is not a
production-reliability or clinical-safety claim. Read the
[limits](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/LIMITS.md)
before extrapolating either result.

## Six Public Surfaces

| Surface | Lifecycle | Start here |
Expand Down
1 change: 1 addition & 0 deletions repository-lifecycle.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ lifecycle:
- openadapt-crier
- openadapt-consilium
- openadapt-presenter
- openadapt-bootstrap
- openadapt-telemetry
- openadapt-viewer
- openadapt-blog
Expand Down