[dotnet-port-api] Add AI-judge loop evaluator - #1162
PratikDhanave (PratikDhanave) wants to merge 1 commit into
Conversation
Port the .NET AIJudgeLoopEvaluator: a loop evaluator that queries a separate
judge agent after each iteration to decide whether the user's original request
was fully addressed, continuing the loop with the judge's gap analysis as
feedback while the answer is "no".
The judge is queried with the original request contents and the agent's latest
response, and asked for a structured JudgeVerdict {answered, gapAnalysis}. When
the judge does not return structured output, the verdict falls back to the
non-overlapping VERDICT: DONE / VERDICT: MORE markers (MORE wins when ambiguous
or absent, so the loop keeps running). Criteria render into the judge
instructions at {criteria}, and the feedback template substitutes the gap
analysis at {gap_analysis}.
github-actions
Bot
added
area:agent
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Three moderate evaluator issues must be addressed before approval.
Review effort: Lite
Findings: None
What changed in this PR
Adds a Go AIJudgeEvaluator that iteratively judges agent responses, supports criteria and gap-analysis feedback, and falls back to text verdict markers.
Changes:
- Added evaluator configuration and verdict parsing.
- Added criteria and feedback-template rendering.
- Added structured, fallback, and nil-judge tests.
| File | Summary |
|---|---|
agent/harness/loop/loop_test.go |
Tests verdict handling, fallbacks, criteria, and nil judges. |
agent/harness/loop/evaluators.go |
Implements the evaluator. Three moderate findings remain: native structured output is not requested, non-text response content is omitted, and JSON extraction can reject valid verdicts surrounded by other JSON. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
github-actions
Bot
added
kind:code
|
Scope: public API, user-visible behavior Changed Go contract: New exported Upstream evidence reviewed:
Result: findings reported Overall naming, defaults (opt-in only evaluator,
Neither issue blocks the port conceptually — the overall design, defaults, and text-fallback semantics match — but both reduce parity with the documented .NET behavior and should be considered before merge.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Generated by Go API Consistency Review Agent · copilot · auto · 153.5 AIC · ⌖ 7.82 AIC · ⊞ 9.2K
| } | ||
| contents = append(contents, &message.TextContent{Text: "\n\n## Agent's latest response:\n" + loop.LastResponse.String()}) | ||
|
|
||
| resp, err := e.judge.Run(ctx, []*message.Message{{Role: message.RoleUser, Contents: contents}}, agent.WithInstructions(e.instructions)).Collect() |
There was a problem hiding this comment.
Parity gap vs. the upstream .NET AIJudgeLoopEvaluator:
-
No judge isolation.
NewAIJudgeEvaluatortakes a full*agent.AgentandEvaluateruns it throughe.judge.Run(...), which goes through the entire agent pipeline (default in-memory history provider, anyContextProviders,Tools,Middlewaresconfigured on that agent). Upstream'sAIJudgeLoopEvaluator(IChatClient judgeClient, ...)intentionally takes a rawIChatClientso the judge is, per its doc remarks, "queried directly (without any agent tools, session, or middleware)". Seedotnet/src/Microsoft.Agents.AI/Harness/Loop/AIJudgeLoopEvaluator.cs. If a caller passes a Go agent that has tools or session/history configured, the judge could execute tools or leak state across loop iterations — a structural guarantee the .NET type enforces at the type level that Go does not. Consider constraining the accepted judge type (e.g. a raw provider/run function) or explicitly documenting/enforcing the no-tools/no-session expectation. -
No native structured-output request. Upstream calls
judgeClient.GetResponseAsync<JudgeVerdict>(judgeMessages, LoopJsonContext.Default.Options, ...), asking the provider for structured JSON output before falling back to text-marker parsing only when the client doesn't honor it (seeTryGetResultbranch in the same file). This GoEvaluateonly passesagent.WithInstructions(e.instructions)and never usesagent.WithStructuredOutput/agent.WithResponseFormat(already available inagent/options.go/agent/structuredoutput.go), so it always falls back to manual{/}substring scraping (extractJudgeVerdict) rather than requesting the provider's native structured-output support like .NET does.
Ports the .NET
AIJudgeLoopEvaluator(Harness/Loop). Go's loop harness previously shipped onlyCompletionMarkerEvaluator; the parity doc lists the AI-judge evaluator as a gap.What it does
AIJudgeEvaluatorqueries a separate judge agent after each iteration to decide whether the user's original request has been fully addressed, continuing the loop (with the judge's gap analysis as feedback) while the answer is "no".JudgeVerdict{answered, gapAnalysis}.VERDICT: DONE/VERDICT: MOREtext markers, withMOREwinning when the verdict is ambiguous or absent (so the loop keeps running rather than stopping on an incomplete answer) — matching .NET.Criteriarender into the judge instructions at{criteria}; the feedback template substitutes the gap analysis at{gap_analysis}.Security
Mirrors the .NET type's security note in the doc comment: the judge is sent the request and latest response every iteration, so only use a judge you trust as much as the primary model, and prefer a stricter
MaxIterations.Tests