Skip to content

fix(benchmarks): add bounded verifier recovery for flash reruns - #51

Merged
minixalpha merged 3 commits into
mainfrom
benchmark/deepseek-v41-flash-1017-rerun-20261018
Oct 19, 2026
Merged

minixalpha merged 3 commits into
mainfrom
benchmark/deepseek-v41-flash-1017-rerun-20261018

Conversation

@minixalpha

@minixalpha minixalpha commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Add opt-in retry_dependency_errors=true recovery for recognized uv network download failures before official tests start. Retry the same output in the same container at most three total times, retaining the original per-attempt timeout and all attempt logs. Stop on test execution, a CTRF report, successful reward, or unrelated errors; exhausted dependency failures raise VerifierDependencyError and remain unscored. The default remains timeout-only.

Record the six DeepSeek Flash 1017 reruns and the authorized follow-up for two initially unscored tasks, including configurations, provenance hashes, reports, and results. WAL recovery, gRPC, and SQL optimization passed; regex chess failed. The follow-up generated fresh outputs because the original containers and task artifacts were unavailable: the sampler passed 9/9 on one verification attempt, while pipeline parallelism failed 2/4 after one timeout and a completed second attempt. The new dependency-error rule did not trigger in these native trials. Preserve follow-up /app artifacts locally for future verification.

Validation: recorded implementation validation passed all 178 Harbor workflow tests; all three CI checks pass at 5804708. The independent review passed all 35 verifier tests and configuration validation; its full suite had 171 passes and seven sandbox socket-permission errors. Both adapter READMEs document the optional policy. Agent code and the system prompt are unchanged.

@minixalpha minixalpha changed the title test(benchmarks): rerun six failures with deepseek flash 1017 fix(benchmarks): add bounded verifier recovery for flash reruns Oct 18, 2026
@minixalpha
minixalpha merged commit ce28b1e into main Oct 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant