Is your feature request related to a problem?
The evaluation summary currently uses internal Langfuse identifiers, which reviewers cannot map back to specific questions without accessing Langfuse. Additionally, questions with duplication appear under multiple ids, leading to redundant entries in the summary.
Describe the solution you'd like
- Provide the summary model with
question_id instead of trace_id and update the prompt to reference questions as "Question N".
- Upgrade
EVAL_JUDGE_MODEL to a stronger model.
- Modify the test in
app/tests/crud/evaluations/test_run_ai_summary.py to accommodate the new payload key.
Original issue
Describe the current behavior
The evaluation AI summary (app/crud/evaluations/summary.py) keys every per-question judge trace on the Langfuse dataset item id (trace_id, formatted item_{row}_{dup}). The system prompt instructs the summary model to reference each question by that id, so the generated "Top 3 to check" list comes back as item_3_1, item_12_4, etc.
Two problems with that:
- Those ids are internal Langfuse identifiers and mean nothing to a reviewer reading the summary — you cannot map
item_12_4 back to a question without opening Langfuse.
- With a duplication factor > 1, the same question appears under several distinct ids (
item_3_1 … item_3_5), so the model can list what is really one problematic question as several separate items and burn the three slots on duplicates.
Separately, EVAL_JUDGE_MODEL in app/core/config.py is pinned to gpt-5-mini, which is weaker than what the LLM-as-judge scoring warrants.
Describe the enhancement you'd like
- Hand the summary model
question_id (the 1-based dataset row number already assigned at Langfuse push time and merged in app/crud/evaluations/merge.py) instead of trace_id, and update the system prompt to reference questions as "Question N".
- Bump
EVAL_JUDGE_MODEL to a stronger judge model.
- Update
app/tests/crud/evaluations/test_run_ai_summary.py for the new payload key.
Why is this enhancement needed?
Usability: the run summary is meant to be a go-verify pointer list for a human reviewer. "Question 3" is directly actionable against the golden dataset; item_3_1 is not. Collapsing duplicate runs of the same question onto one id also stops the "Top 3" from being padded with the same question repeated, so the three slots surface three genuinely distinct problems.
Additional context
question_id is already populated end to end (app/crud/evaluations/langfuse.py writes it into trace metadata, merge.py preserves and sorts on it), so this is a consumer-side change only — no new plumbing.
Is your feature request related to a problem?
The evaluation summary currently uses internal Langfuse identifiers, which reviewers cannot map back to specific questions without accessing Langfuse. Additionally, questions with duplication appear under multiple ids, leading to redundant entries in the summary.
Describe the solution you'd like
question_idinstead oftrace_idand update the prompt to reference questions as "Question N".EVAL_JUDGE_MODELto a stronger model.app/tests/crud/evaluations/test_run_ai_summary.pyto accommodate the new payload key.Original issue
Describe the current behavior
The evaluation AI summary (
app/crud/evaluations/summary.py) keys every per-question judge trace on the Langfuse dataset item id (trace_id, formatteditem_{row}_{dup}). The system prompt instructs the summary model to reference each question by that id, so the generated "Top 3 to check" list comes back asitem_3_1,item_12_4, etc.Two problems with that:
item_12_4back to a question without opening Langfuse.item_3_1…item_3_5), so the model can list what is really one problematic question as several separate items and burn the three slots on duplicates.Separately,
EVAL_JUDGE_MODELinapp/core/config.pyis pinned togpt-5-mini, which is weaker than what the LLM-as-judge scoring warrants.Describe the enhancement you'd like
question_id(the 1-based dataset row number already assigned at Langfuse push time and merged inapp/crud/evaluations/merge.py) instead oftrace_id, and update the system prompt to reference questions as "Question N".EVAL_JUDGE_MODELto a stronger judge model.app/tests/crud/evaluations/test_run_ai_summary.pyfor the new payload key.Why is this enhancement needed?
Usability: the run summary is meant to be a go-verify pointer list for a human reviewer. "Question 3" is directly actionable against the golden dataset;
item_3_1is not. Collapsing duplicate runs of the same question onto one id also stops the "Top 3" from being padded with the same question repeated, so the three slots surface three genuinely distinct problems.Additional context
question_idis already populated end to end (app/crud/evaluations/langfuse.pywrites it into trace metadata,merge.pypreserves and sorts on it), so this is a consumer-side change only — no new plumbing.