Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Passage padding currently participates in the M3 training ColBERT MaxSim reduction.
_colbert_embeddingsets padding vectors to zero, but zero is still a candidate: when a query token has negative similarities to every valid passage token, padding wins the maximum and suppresses that token's gradient. The score then depends on the padded passage length.For example, a native tiny BERT encoder with normalized ColBERT vectors produces a score of
0.547694for a passage and0.652896after adding three padding tokens. The valid passage vectors are identical. The first query token's gradient norm changes from0.333333to0.The fix passes the passage attention mask through the unified training objective and excludes padding similarities before the maximum. The mask includes CLS, matching the existing query-mask convention; scoring removes its first position because ColBERT vectors already exclude CLS. This preserves valid zero vectors and leaves EOS handling, query-length normalization and temperature unchanged. The optional
p_maskargument preserves existing unpadded calls.Validation uses normal imports of the complete package with actual PyTorch and Transformers:
p_maskkeyword. The total baseline is 31 failures and 8 passes.These are synthetic-vocabulary, random-weight tiny-model checks; they do not establish pretrained BGE-M3 retrieval quality or GPU/distributed training performance.
Related to #515: multiplying passage padding vectors by zero preserves MaxSim only when the valid maximum is nonnegative. This change addresses training padding, independently of the inference EOS proposal in #1558.