Skip to content

docs(evals): publish bounded OpenAdapt performance evidence#269

Merged
abrichr merged 2 commits into
mainfrom
eval/openadapt-performance-20260717
Jul 18, 2026
Merged

docs(evals): publish bounded OpenAdapt performance evidence#269
abrichr merged 2 commits into
mainfrom
eval/openadapt-performance-20260717

Conversation

@abrichr

@abrichr abrichr commented Jul 17, 2026

Copy link
Copy Markdown
Member

What this measures

  • Publishes row-level retained model-free evidence for Frappe Lending and OpenEMR: compiled GUI versus direct API, baseline and cosmetic drift, three trials per cell.
  • Separately summarizes the fixed historical WAA core-four comparison: legacy demo-conditioned computer use versus zero-shot, three retained trials per task and condition.
  • Adds strict validation and a clean-checkout byte-for-byte regeneration test using committed compact, non-sensitive inputs.

Results

  • Retained model-free subset: compiled 12/12 at 32.28s mean; API 12/12 at 1.63s mean; zero measured silent incorrect success, over-halt, model calls, or model cost.
  • Historical WAA: DC 3/12 and ZS 3/12 (0 pp reliability delta); 6 versus 5 silent incorrect completion claims.

Provenance boundaries

  • The model-free summaries came from a worktree later committed as 84c7a94, but the summaries do not embed a runtime revision. The report records this only as a post-hoc association, not exact Flow source binding.
  • Exact Flow 1.12.1 requalification and current Flow-versus-zero-shot remain unmeasured.
  • The paid agent arm was omitted, and the source matrices remain publication_ready=false and full_matrix_complete=false.
  • Historical model ID, runner revision, token usage, and cost are unavailable and remain explicitly unknown/null.
  • The compact legacy input excludes screenshots, action payloads, and model content while retaining task instructions, normalized rows, and hashes for all 48 raw summary/execution inputs.
  • No new GUI, cloud, or model runs were performed.

Verification

  • 5 passed
  • Ruff lint and format pass
  • Clean-checkout committed-input regeneration is byte-for-byte deterministic
  • Exact retained input and raw-manifest SHA-256 checks pass
  • Credential, local-identity, and secret-pattern scans pass
  • git diff --check passes

@abrichr
abrichr merged commit 7629ba5 into main Jul 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant