Skip to content

Evaluate the agent on a fixed set of real prompts #9454

Description

@bobbai00

Task Summary

The agent's tests use a mock model, so we can't tell whether a change makes it better or worse.

Add a small set of real prompts, each with a pass check: build a workflow, fix a failing operator, explain an execution error, answer a docs question. One command runs them against a real model and prints the score and the tokens used.

  • It also prints the model calls per prompt and the bytes each call sends, and runs each prompt a few times, since a real model does not answer the same way twice.
  • Two prompts involve the user as well: one where the user changes the workflow between two prompts, and one that asks why the user's own execution failed. Today's agent cannot pass either; Let the agent join the workflow's shared editing and execution session #9464 is meant to.

Done when today's score is recorded, so each later sub-issue can report its own. The 25 questions in #9424 and the new-user questions offered in #9377 are a good start.

How an evaluation run works

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions