You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
harbor: respect TrialResult.reward_valid when converting rewards to scores #1007
Sub-issue of #1001 (follow-ups from the 2026-10-26 dependency update, #998).
harbor 0.25.0 added computed properties on TrialResult: failure_cause (the error-policy cause of the first recorded exception) and reward_valid. reward_valid is false when the trial has no rewards, or when any recorded exception's error policy says the reward can't be trusted (for example a failure that happened after verification).
Today the Braintrust harbor plugin converts rewards to scores without checking it: _rewards() (integrations/harbor/plugin.py:139) feeds classify_rewards() (plugin.py:783, integrations/harbor/rewards.py:52), so a reward harbor considers invalid is logged as a normal score.
Proposed: when reward_valid is false and rewards are present, skip (or null) the scores, add a metadata.harbor.warnings entry, and record reward_valid / failure_cause in span metadata. Feature-detect the properties so harbor < 0.25 keeps the current behavior.
Tests: extend the harbor plugin tests with a trial whose recorded exception has a policy with reward_valid=False, at latest. The 0.16.0 pin should be unaffected.
Activity
AbhiPrasad commentedon Oct 26, 2026
released with https://github.com/braintrustdata/braintrust-sdk-python/releases/tag/py-sdk-v0.48.1
fix(harbor): respect trial reward validity (braintrustdata#1008)