Repository navigation
Conversation
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9495 +/- ##
============================================
+ Coverage 91.57% 91.59% +0.02%
Complexity 5159 5159
============================================
Files 1300 1305 +5
Lines 58092 58258 +166
Branches 7615 7615
============================================
+ Hits 53198 53364 +166
Misses 2958 2958
Partials 1936 1936
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The agent's tests use a mock model, so nothing showed whether a change made the agent better or worse. `bun run eval` starts this checkout's agent service against a running Texera and a real model, runs six cases a few times each, and prints how many runs passed with the model calls, tokens and bytes per call, which a recorder in front of the LLM endpoint counts. The cases build a workflow, fix a failing operator, explain an execution error, answer from the docs, keep a change the user makes between two prompts, and explain the user's own failed execution. They work on orders made by a seeded generator, so every expected answer is computed.
bobbai00
force-pushed
the
test/agent-eval
branch
from
November 19, 2026 08:38
1127c95 to
6bd1b66
Compare
Contributor
Author
|
Closing this for now: evaluating the agent needs an effort of its own, so it waits until the rest of the agent refactor (#9453) has landed. The branch stays for when that work starts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
The agent's tests use a mock model, so nothing showed whether a change makes the agent better or worse. This PR adds
bun run evalinagent-service: it starts the agent service of the checkout against a running Texera and a real model, runs six cases a few times each, and prints how many runs passed and what they cost.build-workflowfix-failing-operatoramount, and the answer gives the right countexplain-execution-errordocs-questionuser-edit-between-promptsuser-execution-failedLLM_ENDPOINTcounts every model call: its size in bytes and the tokens the provider reports.agent-eval-orders, which later runs reuse.Today's score, on
mainwith Claude Haiku 5.5:The three failing cases are the gaps the other sub-issues of #9453 close: the docs (#9456), the user's edits between prompts and the user's own executions (#9464).
Any related issues, documentation, discussions?
Closes #9454
Part of #9453
How was this PR tested?
cd agent-service && bun test(384 pass, 42 of them new: the checks, the data and its expected answers, the report, the recorder against a stand-in endpoint, and the settings),bun run typecheckandbun run format:check, which now covereval/.bun run evalwithEVAL_REPEAT=3on a single-node stack of today's images with Claude Haiku 5.5, which printed the table above. Each case was also checked to fail for the right reason:user-execution-failedpassed while its error was guessable from the workflow (a cast of a text column), so both execution cases now use the Python operator whose error names an order.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5