Skip to content

feat(experiment): add trigger-shift replay evaluator#602

Merged
autogame-17 merged 1 commit into
mainfrom
sync/trigger-shift-current
Jul 17, 2026
Merged

feat(experiment): add trigger-shift replay evaluator#602
autogame-17 merged 1 commit into
mainfrom
sync/trigger-shift-current

Conversation

@autogame-17

@autogame-17 autogame-17 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • sync the current Evolver trigger-shift replay evaluator into the public repo
  • add paired replay tasks for wrapper trigger, temporal context, and instruction phrasing shifts
  • report train reward, shifted reward, and gap without any live GEP selection or trigger execution wiring
  • include a trigger-sensitive dummy policy regression test

Source

Ported from private source-of-truth merge:

  • EvoMap/evolver-private-dev#330
  • merge commit ffdf00a71325321508da521f96c0bff1a930d85c

Validation

  • PASS node --test test/triggerShift.test.js
  • PASS node --check src/experiment/triggerShift.js && node --check test/triggerShift.test.js

Risk / rollback

  • Pure experiment helper only: no I/O, no subprocess, no network, no store writes, no live trigger/selection wiring.
  • Roll back by reverting this PR.

🤖 Generated with Claude Code


Note

Low Risk
New isolated experiment code with no production wiring, I/O, or network; rollback is a straight revert.

Overview
Adds an offline experiment helper that scores policies on paired train vs shifted tasks (same objective, different wrapper trigger, temporal context, or instruction phrasing) to surface trigger/context overfitting.

evaluateTriggerShift runs policy.predict on each pair, computes binary train/shifted rewards against expectedDecision, and returns aggregate metrics plus per-pair gap (trainReward - shiftedReward). Report rows omit raw prompt text so diagnostics can be logged without leaking task bodies. smallTriggerShiftSuite ships one calibration pair per axis.

Tests cover empty suites, stable policies, a trigger-sensitive dummy policy, axis coverage, and that serialized reports include reward/gap fields without embedding full prompts. No I/O, live triggers, or GEP selection wiring.

Reviewed by Cursor Bugbot for commit 8f866c1. Bugbot is set up for automated code reviews on this repo. Configure here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@autogame-17
autogame-17 merged commit 55f0ec7 into main Jul 17, 2026
2 of 5 checks passed
@autogame-17
autogame-17 deleted the sync/trigger-shift-current branch July 17, 2026 09:54

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8f866c1. Configure here.


function labelReward(predicted, expected) {
return normLabel(predicted) === normLabel(expected) ? 1 : 0;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty expected yields false reward

Medium Severity

When expectedDecision is missing or only whitespace, normLabel turns it into an empty string, and labelReward awards reward 1 whenever both train and shifted predictions are also empty. Malformed or incomplete pairs can therefore look perfectly aligned and inflate meanTrainReward, meanShiftedReward, and understate gap.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 8f866c1. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant