You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Evaluate the agent on a fixed set of real prompts #9454
The agent's tests use a mock model, so we can't tell whether a change makes it better or worse.
Add a small set of real prompts, each with a pass check: build a workflow, fix a failing operator, explain an execution error, answer a docs question. One command runs them against a real model and prints the score and the tokens used.
It also prints the model calls per prompt and the bytes each call sends, and runs each prompt a few times, since a real model does not answer the same way twice.
Done when today's score is recorded, so each later sub-issue can report its own. The 25 questions in #9424 and the new-user questions offered in #9377 are a good start.
Activity
bobbai00 commentedon Nov 16, 2026
/take