You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
However, with the release of Terminal-Bench 4.0, using a benchmark that DeepSeek V4 Flash has not encountered during training to evaluate the model offers greater discriminative power.
Of course, the lack of tokens and GPUs could be an issue. ASF does not provide tokens for DeepSeek V4 Flash, and the free tier of GitHub Actions Runners lacks GPUs entirely—yet running Terminal-Bench requires GPUs.
Desired outcome
Use Terminal-Bench 4.0 instead of Terminal-Bench 2.1 in the eval results.
Problem
Terminal-Bench 2.1, which is certainly fine.Terminal-Bench 4.0, using a benchmark thatDeepSeek V4 Flashhas not encountered during training to evaluate the model offers greater discriminative power.DeepSeek V4 Flash, and the free tier of GitHub Actions Runners lacks GPUs entirely—yet runningTerminal-Benchrequires GPUs.Desired outcome
Terminal-Bench 4.0instead ofTerminal-Bench 2.1in theevalresults.Alternatives or workarounds