Skip to content

fix: raise ValueError when retrieval evaluators get no questions - #13091

Open
Vamsi-2581 wants to merge 1 commit into
deepset-ai:mainfrom
Vamsi-2581:fix/evaluator-empty-input
Open

Vamsi-2581 wants to merge 1 commit into
deepset-ai:mainfrom
Vamsi-2581:fix/evaluator-empty-input

Conversation

@Vamsi-2581

Copy link
Copy Markdown

What happened

DocumentMAPEvaluator, DocumentMRREvaluator, DocumentRecallEvaluator, and AnswerExactMatchEvaluator average per-question scores with sum(scores) / len(questions). An empty question list (a filtered eval set, for example) therefore raised ZeroDivisionError instead of a useful error. DocumentNDCGEvaluator already rejects that input with ValueError.

What changed

Each of those four evaluators now raises ValueError when either input list is empty, before the mean is computed. A question whose own document list is empty still scores 0.0, which is the existing per-question behavior.

How it was checked

New tests in test/components/evaluators/test_empty_input.py cover both lists empty, one list empty, and a non-empty question with no documents. The empty cases raise ValueError (must be provided) instead of dividing by zero.

python -m pytest test/components/evaluators/test_empty_input.py test/components/evaluators/test_document_map.py test/components/evaluators/test_document_mrr.py test/components/evaluators/test_document_recall.py test/components/evaluators/test_answer_exact_match.py test/components/evaluators/test_document_ndcg.py -q
98 passed

Fixes #12543

DocumentMAP, DocumentMRR, DocumentRecall, and AnswerExactMatch divided
by the number of questions even when that list was empty, so a filtered
eval set crashed with ZeroDivisionError. Match DocumentNDCGEvaluator and
reject that input before the mean is computed.
@Vamsi-2581
Vamsi-2581 requested a review from a team as a code owner October 3, 2026 02:12
@Vamsi-2581
Vamsi-2581 requested review from julian-risch and removed request for a team October 3, 2026 02:12
@vercel

vercel Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

@Vamsi-2581 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Oct 3, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @Vamsi-2581, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from julian-risch October 3, 2026 03:44
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 3, 2026
@HaystackBot
HaystackBot marked this pull request as draft October 3, 2026 03:44
@Vamsi-2581
Vamsi-2581 marked this pull request as ready for review October 3, 2026 14:53
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @Vamsi-2581! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 of 5 Document*Evaluators crash with ZeroDivisionError on empty input

3 participants