Skip to content

fix(harbor): respect trial reward validity - #1008

Merged
Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
abhi/harbor-reward-validity-16730ec8
Oct 26, 2026
Merged

Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
abhi/harbor-reward-validity-16730ec8

Conversation

@AbhiPrasad

@AbhiPrasad Abhijeet Prasad (AbhiPrasad) commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

🤖 This text was AI-generated.

Harbor 0.25.0 marks a trial's rewards invalid when a recorded failure makes them untrustworthy. The Braintrust plugin previously converted those rewards into normal scores:

reward=0.25, reward_valid=False -> score=0.25

The plugin now checks TrialResult.reward_valid before converting rewards. Invalid rewards produce no scores or derived metrics:

reward=0.25, reward_valid=False
  -> no scores or reward metrics
  -> raw_rewards retained in metadata.harbor
  -> warning added to metadata.harbor.warnings

Span metadata also records reward_valid and failure_cause when Harbor exposes them. Valid rewards still convert normally, including after exceptions whose policies preserve reward validity. Feature detection keeps the existing behavior on Harbor versions before 0.25.0, including the supported 0.16.0 pin.

Fixes SDK-633.
Closes #1007.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-26T13:14:20.248288Z 177f196 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 141d4e8 into main Oct 26, 2026
98 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the abhi/harbor-reward-validity-16730ec8 branch October 26, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

harbor: respect TrialResult.reward_valid when converting rewards to scores

1 participant