Repository navigation
fix(python): do not send a default temperature from AnswerRelevancy - #254
Open
alexcastgonz (pcbeingused333) wants to merge 1 commit into
Conversation
AnswerRelevancy defaulted temperature to 0.5 and always sent it, so a judge
model that rejects a non-default temperature (Claude 4.7+ and 5.x) failed with
a 400 unless the caller knew to pass one it accepts. Default it to None and send
it only when given, as the other scorers in llm.py do ("If not specified, uses
the model's default").
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AnswerRelevancydefaultedtemperatureto 0.5 and always sent it on the question-generation request. A judge model that rejects a non-default temperature, such as Claude 4.7 and later (including the 5.x models), then fails with a 400 unless the caller knows to pass a value it accepts. The other scorers inllm.pydocument "If not specified, uses the model's default" and only send a temperature that was given, andjs/ragas.tsalready behaves that way.temperaturenow defaults toNoneand is sent only when the caller passes it; the docstring says so. This changes the default for every model: a call that relied on 0.5 now uses the model's own default. The score metadata reportsNonefortemperaturein that case. If you would rather keep 0.5 for models that accept it, I can instead skip the temperature only for the models that reject it; I went for the simpler convention used inllm.py.Test:
test_answer_relevancy_does_not_send_a_default_temperatureinpy/autoevals/test_ragas.pybuilds the question-generation request for a Claude model name and checks it has notemperature, and that an explicittemperature=0.2is still sent. Onmainthe new test fails (the default request carriestemperature) and it passes with this change. Inpy/autoevals/test_ragas.pythe other nine tests (thetest_ragas_retrievalcases and the custom-embedding test) fail in my sandbox both onmainand here, because they need an OpenAI key/network; I did not call a live model.This change was made with Claude Code.