fix: raise ValueError when retrieval evaluators get no questions - #13091
Open
Vamsi-2581 wants to merge 1 commit into
Open
Vamsi-2581 wants to merge 1 commit into
Vamsi-2581 wants to merge 1 commit into
Conversation
DocumentMAP, DocumentMRR, DocumentRecall, and AnswerExactMatch divided by the number of questions even when that list was empty, so a filtered eval set crashed with ZeroDivisionError. Match DocumentNDCGEvaluator and reject that input before the mean is computed.
Contributor
|
@Vamsi-2581 is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
|
Hi @Vamsi-2581, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
HaystackBot
marked this pull request as draft
October 3, 2026 03:44
Vamsi-2581
marked this pull request as ready for review
October 3, 2026 14:53
Contributor
|
Thanks for signing the CLA, @Vamsi-2581! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
DocumentMAPEvaluator,DocumentMRREvaluator,DocumentRecallEvaluator, andAnswerExactMatchEvaluatoraverage per-question scores withsum(scores) / len(questions). An empty question list (a filtered eval set, for example) therefore raisedZeroDivisionErrorinstead of a useful error.DocumentNDCGEvaluatoralready rejects that input withValueError.What changed
Each of those four evaluators now raises
ValueErrorwhen either input list is empty, before the mean is computed. A question whose own document list is empty still scores0.0, which is the existing per-question behavior.How it was checked
New tests in
test/components/evaluators/test_empty_input.pycover both lists empty, one list empty, and a non-empty question with no documents. The empty cases raiseValueError(must be provided) instead of dividing by zero.Fixes #12543