Skip to content

feat(agent-service): look up operator schemas with a tool - #9496

Open
bobbai00 wants to merge 1 commit into
apache:mainfrom
bobbai00:feat/agent-operator-lookup
Open

bobbai00 wants to merge 1 commit into
apache:mainfrom
bobbai00:feat/agent-operator-lookup

Conversation

@bobbai00

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Every model call carried the compact schema of each allowed operator type in the system prompt: 18.8 KB by default, 189.2 KB with all 168 types. The prompt also told the model to call a getOperatorSchema tool that did not exist, and addOperator accepted every type whatever the allowed list said.

main This PR
System prompt Each allowed type's schema Each usable type in one line: its name and what it does
A type's properties Already in the prompt get_operator_schema, for several types in one call
Allowed operator types Trim the prompt only One list for the prompt, the lookup and addOperator
A tool that throws, or a call the SDK rejects Recorded with no result Recorded as the call's failed result

This is the first part of #9481; edit_workflow and execute_workflow follow.

Any related issues, documentation, discussions?

Part of #9481
Part of #9453

How was this PR tested?

  • cd agent-service && bun test (352 pass), bun run typecheck and bun run format:check. New tests cover the lookup (several types, an unknown, a disallowed or a miscased type, an empty list), the prompt's list (allowlist, no allowlist, nothing known, no schema left), one list for the prompt, the lookup and addOperator, and a thrown tool error recorded on its step.

  • The evaluation from test(agent-service): evaluate the agent on real prompts #9495 with Claude Haiku 5.5, three runs per case, against main:

    Per run, averaged main This PR
    Runs passed 9 of 18 9 of 18
    Model calls 2.5 3.3
    Input tokens 32,506 20,587
    Output tokens 964 1,145
    KB per call 37.6 16.6
  • A first version kept the types out of the prompt behind a find_operator tool and looked up one type per call. It passed the same runs but made 5.3 model calls per run for 30,616 input tokens, since the model searched for each type before reading it, so this PR names the types in the prompt and reads several at once.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @oliarmak-web, @aglinxinyuan, @ShawnGarello
    You can notify them by mentioning @oliarmak-web, @aglinxinyuan, @ShawnGarello in a comment.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.57%. Comparing base (4db10cf) to head (7fd7ae0).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #9496   +/-   ##
=========================================
  Coverage     91.57%   91.57%           
  Complexity     5159     5159           
=========================================
  Files          1300     1301    +1     
  Lines         58092    58111   +19     
  Branches       7615     7615           
=========================================
+ Hits          53198    53217   +19     
  Misses         2958     2958           
  Partials       1936     1936           
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from 4db10cf
agent-service 99.56% <100.00%> (+<0.01%) ⬆️
amber 85.44% <ø> (ø) Carriedforward from 4db10cf
computing-unit-managing-service 63.88% <ø> (ø) Carriedforward from 4db10cf
config-service 87.50% <ø> (ø) Carriedforward from 4db10cf
file-service 82.16% <ø> (ø) Carriedforward from 4db10cf
frontend 96.45% <ø> (ø) Carriedforward from 4db10cf
notebook-migration-service 83.77% <ø> (ø) Carriedforward from 4db10cf
pyamber 98.38% <ø> (ø) Carriedforward from 4db10cf
result-service 85.71% <ø> (ø) Carriedforward from 4db10cf
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from 4db10cf

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Every model call carried the compact schema of each allowed operator type
in the system prompt: 18.8 KB by default, 189.2 KB with all 168 types. The
add tool also accepted every type, whatever the allowed list said.

The prompt now names each usable type in one line, and the agent reads
the properties of the types it needs with get_operator_schema, several at
once. The prompt's list, the lookup and the add tool share one list of
operator types.
@bobbai00
bobbai00 force-pushed the feat/agent-operator-lookup branch from 576416c to 7fd7ae0 Compare November 19, 2026 08:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants