Skip to content

Evaluation: Improve summary identifiers #1174

Description

@AkhileshNegi

Is your feature request related to a problem?
The evaluation summary currently uses internal Langfuse identifiers, which reviewers cannot map back to specific questions without accessing Langfuse. Additionally, questions with duplication appear under multiple ids, leading to redundant entries in the summary.

Describe the solution you'd like

  • Provide the summary model with question_id instead of trace_id and update the prompt to reference questions as "Question N".
  • Upgrade EVAL_JUDGE_MODEL to a stronger model.
  • Modify the test in app/tests/crud/evaluations/test_run_ai_summary.py to accommodate the new payload key.
Original issue

Describe the current behavior

The evaluation AI summary (app/crud/evaluations/summary.py) keys every per-question judge trace on the Langfuse dataset item id (trace_id, formatted item_{row}_{dup}). The system prompt instructs the summary model to reference each question by that id, so the generated "Top 3 to check" list comes back as item_3_1, item_12_4, etc.

Two problems with that:

  • Those ids are internal Langfuse identifiers and mean nothing to a reviewer reading the summary — you cannot map item_12_4 back to a question without opening Langfuse.
  • With a duplication factor > 1, the same question appears under several distinct ids (item_3_1item_3_5), so the model can list what is really one problematic question as several separate items and burn the three slots on duplicates.

Separately, EVAL_JUDGE_MODEL in app/core/config.py is pinned to gpt-5-mini, which is weaker than what the LLM-as-judge scoring warrants.

Describe the enhancement you'd like

  • Hand the summary model question_id (the 1-based dataset row number already assigned at Langfuse push time and merged in app/crud/evaluations/merge.py) instead of trace_id, and update the system prompt to reference questions as "Question N".
  • Bump EVAL_JUDGE_MODEL to a stronger judge model.
  • Update app/tests/crud/evaluations/test_run_ai_summary.py for the new payload key.

Why is this enhancement needed?

Usability: the run summary is meant to be a go-verify pointer list for a human reviewer. "Question 3" is directly actionable against the golden dataset; item_3_1 is not. Collapsing duplicate runs of the same question onto one id also stops the "Top 3" from being padded with the same question repeated, so the three slots surface three genuinely distinct problems.

Additional context

question_id is already populated end to end (app/crud/evaluations/langfuse.py writes it into trace metadata, merge.py preserves and sorts on it), so this is a consumer-side change only — no new plumbing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions