Skip to content

fix: honor partial pairwise scorers in Python ListContains - #250

Open
BSAIPF wants to merge 1 commit into
braintrustdata:mainfrom
BSAIPF:fix/list-contains-partial-scorer
Open

BSAIPF wants to merge 1 commit into
braintrustdata:mainfrom
BSAIPF:fix/list-contains-partial-scorer

Conversation

@BSAIPF

@BSAIPF BSAIPF commented Oct 21, 2026

Copy link
Copy Markdown

ListContains calls pairwise scorers' private _run_eval_* methods, bypassing their public eval() / eval_async() wrappers. This loses arguments bound by ScorerWithPartial.partial() for custom scorers. A case-insensitive matcher that works directly scores 0 instead of 2/3 when comparing ["PARIS", "Rome", "extra"] with ["rome", "paris"].

Use the public scorer entry points for each pair. The regression test covers sync/async evaluation, scorer classes/instances, matching metadata, and the extra-entity penalty. This is Python-specific; TypeScript already invokes the supplied scorer function.

Validation:

  • All four regression cases fail before the fix and pass after it.
  • pytest py/autoevals/test_values.py py/autoevals/test_json.py -q: 12 passed.
  • Broader offline selection on Python 3.10, 3.12, and 3.14: 78 passed on each. The 16 existing live API cases were excluded; network sockets were disabled.
  • pre-commit run --all-files, python -m build, and python -m twine check dist/* passed.

On this macOS host, Python 3.10 used the official SciPy 1.15.3 macOS 12 wheel after the default macOS 14 wheel failed to load. No dependency versions or lockfiles were changed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant