Repository navigation
fix(benchmarks): add bounded verifier recovery for flash reruns - #51
Merged
minixalpha merged 3 commits intoOct 19, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add opt-in
retry_dependency_errors=truerecovery for recognized uv network download failures before official tests start. Retry the same output in the same container at most three total times, retaining the original per-attempt timeout and all attempt logs. Stop on test execution, a CTRF report, successful reward, or unrelated errors; exhausted dependency failures raiseVerifierDependencyErrorand remain unscored. The default remains timeout-only.Record the six DeepSeek Flash 1017 reruns and the authorized follow-up for two initially unscored tasks, including configurations, provenance hashes, reports, and results. WAL recovery, gRPC, and SQL optimization passed; regex chess failed. The follow-up generated fresh outputs because the original containers and task artifacts were unavailable: the sampler passed 9/9 on one verification attempt, while pipeline parallelism failed 2/4 after one timeout and a completed second attempt. The new dependency-error rule did not trigger in these native trials. Preserve follow-up
/appartifacts locally for future verification.Validation: recorded implementation validation passed all 178 Harbor workflow tests; all three CI checks pass at
5804708. The independent review passed all 35 verifier tests and configuration validation; its full suite had 171 passes and seven sandbox socket-permission errors. Both adapter READMEs document the optional policy. Agent code and the system prompt are unchanged.