Skip to content

harbor: respect TrialResult.reward_valid when converting rewards to scores #1007

Closed

Description

Sub-issue of #1001 (follow-ups from the 2026-10-26 dependency update, #998).

harbor 0.25.0 added computed properties on TrialResult: failure_cause (the error-policy cause of the first recorded exception) and reward_valid. reward_valid is false when the trial has no rewards, or when any recorded exception's error policy says the reward can't be trusted (for example a failure that happened after verification).

  • Today the Braintrust harbor plugin converts rewards to scores without checking it: _rewards() (integrations/harbor/plugin.py:139) feeds classify_rewards() (plugin.py:783, integrations/harbor/rewards.py:52), so a reward harbor considers invalid is logged as a normal score.
  • Proposed: when reward_valid is false and rewards are present, skip (or null) the scores, add a metadata.harbor.warnings entry, and record reward_valid / failure_cause in span metadata. Feature-detect the properties so harbor < 0.25 keeps the current behavior.
  • Tests: extend the harbor plugin tests with a trial whose recorded exception has a policy with reward_valid=False, at latest. The 0.16.0 pin should be unaffected.
  • Not planned yet; filed so it isn't lost.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions