Skip to content

test(agent-service): evaluate the agent on real prompts - #9495

Closed
bobbai00 wants to merge 1 commit into
apache:mainfrom
bobbai00:test/agent-eval
Closed

bobbai00 wants to merge 1 commit into
apache:mainfrom
bobbai00:test/agent-eval

Conversation

@bobbai00

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The agent's tests use a mock model, so nothing showed whether a change makes the agent better or worse. This PR adds bun run eval in agent-service: it starts the agent service of the checkout against a running Texera and a real model, runs six cases a few times each, and prints how many runs passed and what they cost.

Case What the user does The run passes when
build-workflow Asks for the three categories with the highest total sales The answer names them, in order
fix-failing-operator Asks to fix a filter on a misspelled column The saved filter uses amount, and the answer gives the right count
explain-execution-error Asks why a Python operator fails The answer names the order the error names
docs-question Asks which output port modes an operator can have The answer names the three modes in the docs
user-edit-between-prompts Asks for two changes, and adds a Limit in between The user's Limit is still in the saved workflow
user-execution-failed Runs the workflow, then asks why it failed, without running anything The answer names the order the error names, and the agent ran nothing
  • The cases work on 240 orders made by a seeded generator, and every expected answer is computed from them. The failing Python operator names the order it stops at, an id that only the execution's error holds, so guessing from the workflow does not pass.
  • A recorder in front of LLM_ENDPOINT counts every model call: its size in bytes and the tokens the provider reports.
  • It reuses the agent service's own code: the WebSocket command classes to prompt, the workflow client, the operator builder and the execution client.
  • Each run's workflow and agent are deleted when it ends. The orders stay in one dataset, agent-eval-orders, which later runs reuse.

Today's score, on main with Claude Haiku 5.5:

Case                       Passed  Model calls  Input tokens  Output tokens  KB per call  Max KB
-------------------------  ------  -----------  ------------  -------------  -----------  ------
build-workflow                3/3          4.3        56,984          1,402         37.9    39.8
fix-failing-operator          3/3          2.7        34,732            914         37.7    39.3
explain-execution-error       3/3          2.0        26,013            854         37.8    38.6
docs-question                 0/3          1.0        12,205            839         35.8    35.8
user-edit-between-prompts     0/3          4.0        52,409          1,037         37.8    39.8
user-execution-failed         0/3          1.0        12,693            739         37.0    37.0
-------------------------  ------  -----------  ------------  -------------  -----------  ------
total                        9/18          2.5        32,506            964         37.6    39.8

Score: 9 of 18 runs passed (50%). Costs are per run, averaged.

The three failing cases are the gaps the other sub-issues of #9453 close: the docs (#9456), the user's edits between prompts and the user's own executions (#9464).

Any related issues, documentation, discussions?

Closes #9454
Part of #9453

How was this PR tested?

  • cd agent-service && bun test (384 pass, 42 of them new: the checks, the data and its expected answers, the report, the recorder against a stand-in endpoint, and the settings), bun run typecheck and bun run format:check, which now cover eval/.
  • bun run eval with EVAL_REPEAT=3 on a single-node stack of today's images with Claude Haiku 5.5, which printed the table above. Each case was also checked to fail for the right reason: user-execution-failed passed while its error was guessable from the workflow (a cast of a text column), so both execution cases now use the Python operator whose error names an order.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file docs Changes related to documentations agent-service labels Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @renovate-bot
    You can notify them by mentioning @renovate-bot in a comment.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.59%. Comparing base (4db10cf) to head (6bd1b66).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9495      +/-   ##
============================================
+ Coverage     91.57%   91.59%   +0.02%     
  Complexity     5159     5159              
============================================
  Files          1300     1305       +5     
  Lines         58092    58258     +166     
  Branches       7615     7615              
============================================
+ Hits          53198    53364     +166     
  Misses         2958     2958              
  Partials       1936     1936              
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from 4db10cf
agent-service 99.58% <100.00%> (+0.02%) ⬆️
amber 85.44% <ø> (ø) Carriedforward from 4db10cf
computing-unit-managing-service 63.88% <ø> (ø) Carriedforward from 4db10cf
config-service 87.50% <ø> (ø) Carriedforward from 4db10cf
file-service 82.16% <ø> (ø) Carriedforward from 4db10cf
frontend 96.45% <ø> (ø) Carriedforward from 4db10cf
notebook-migration-service 83.77% <ø> (ø) Carriedforward from 4db10cf
pyamber 98.38% <ø> (ø) Carriedforward from 4db10cf
result-service 85.71% <ø> (ø) Carriedforward from 4db10cf
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from 4db10cf

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The agent's tests use a mock model, so nothing showed whether a change made
the agent better or worse. `bun run eval` starts this checkout's agent
service against a running Texera and a real model, runs six cases a few
times each, and prints how many runs passed with the model calls, tokens
and bytes per call, which a recorder in front of the LLM endpoint counts.

The cases build a workflow, fix a failing operator, explain an execution
error, answer from the docs, keep a change the user makes between two
prompts, and explain the user's own failed execution. They work on orders
made by a seeded generator, so every expected answer is computed.
@bobbai00

Copy link
Copy Markdown
Contributor Author

Closing this for now: evaluating the agent needs an effort of its own, so it waits until the rest of the agent refactor (#9453) has landed. The branch stays for when that work starts.

@bobbai00 bobbai00 closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-service dependencies Pull requests that update a dependency file docs Changes related to documentations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluate the agent on a fixed set of real prompts

2 participants