fix: keep the highest-scoring duplicate in merge and RRF joins

[linhongyu510]

Problem

DocumentJoiner has four join modes. Two of them state which copy of a duplicate they keep, and do so:

• _concatenate — "return only the Document with the highest score for duplicates"
• _distribution_based_rank_fusion — "If a Document is in more than one retriever, the one with the highest score is used"

The other two accumulate a fused score per id while writing the document unconditionally:

scores_map[doc.id] += ...
documents_map[doc.id] = doc   # _merge, and _reciprocal_rank_fusion in utils/misc.py

so the copy that survives is whichever list came last.

Duplicates share an id but not necessarily a payload — two retrievers can return the same document with different content or meta (a summary from one, the full text from another). The fused score in both modes is order-independent, so the result is a score that stays stable while the returned content silently depends on connection order:

list_a = [Document(id="d1", content="full text", score=0.9)]
list_b = [Document(id="d1", content="stub",      score=0.1)]

merge([list_a, list_b])  # -> score 0.5, content "stub"
merge([list_b, list_a])  # -> score 0.5, content "full text"

Measured on main (0defdcff), same input, all four modes:

mode	kept copy
concatenate	full text (score 0.9)
distribution_based_rank_fusion	full text (score 0.9)
merge	stub
reciprocal_rank_fusion	stub

Fix

Keep the higher-scoring copy in both, matching the other two modes.

utils/misc.py already contains _deduplicate_documents() — three lines above the RRF helper — implementing exactly this rule, so this aligns RRF with a convention the module had already settled on. Fixing the shared helper covers every caller, not just DocumentJoiner.

The score arithmetic is untouched; only which copy is stored changes. The -inf sort key _concatenate already used is lifted into a small _score_or_neg_inf helper so both joiner call sites share one definition.

Tests

• test_run_with_merge_join_mode_keeps_highest_scoring_duplicate — both list orders return the high-scoring payload
• test_run_with_rrf_join_mode_keeps_highest_scoring_duplicate — same for RRF
• test_run_with_merge_join_mode_keeps_scored_duplicate_over_unscored — a scored copy beats a score=None one, consistent with the -inf convention

Verification

• 48 pass in test_document_joiner.py (45 pre-existing + 3 new).
• Rollback counter-proof: restoring only the unconditional writes, with the tests kept, fails exactly the 3 new tests while all 45 pre-existing ones still pass.
• No new breakage: across test/components/joiners/ and test/utils/ the failure set is identical before and after; passing count goes 648 → 651, exactly the 3 added tests. (4 pre-existing failures on this machine are environmental and unrelated; test/utils/test_jinja2_extensions.py is excluded because it cannot be collected here for a missing optional dependency.)
• ruff check and ruff format --check clean on all three changed files.
• Checked the 64 open PRs for overlap with these files: fix: reject negative weights in DocumentJoiner #12718 (reject negative weights), fix: DocumentJoiner weights bind to Variadic arrival order, not connect order #12647 (docstring on weight binding) and fix: raise ValueError for reversed page ranges in expand_page_range #11492 (expand_page_range) touch them but none changes duplicate handling.

[vercel]

@linhongyu510 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[sjrl]

Closing, we state in our docs that IDs are used to de-duplicate so if they have the same ID then the content is considered to be the same. That is why our default ID generation uses the full doc dataclass to generate a unique ID

[linhongyu510]

Thanks for the quick call — that settles it, and I verified the reasoning rather than just taking it on faith: Document._create_id() hashes content, meta, blob, mime_type, embedding and sparse_embedding together, so two documents sharing an id genuinely do share the payload. My reproduction constructed the conflict by passing an explicit id= with differing content, which breaks that invariant rather than exercising it.

So the divergence between the modes is cosmetic under the documented contract, and the extra comparison is not worth the cost. Closing as you did — sorry for the noise.

One small thing that may still be worth a separate look, at your discretion: _concatenate and _distribution_based_rank_fusion both spell out "the one with the highest score is used" in their docstrings, which is what led me here. Given IDs are the de-duplication key by design, those two sentences are arguably describing an implementation detail that cannot be observed. Happy to open a docs-only PR dropping or rewording them if you think that reduces confusion; otherwise feel free to ignore.