RedRouterBenchmarks We Don't Run
benchmarks.redrouter.org continuously measures inference API speed, latency, reliability and price. Task suites run periodically, on separate schedules. A fast response tells you little about whether an agent can finish a job; a task score tells you little about today's API reliability.
We also need to trust the grader and afford repeat runs. Three prominent suites don't meet our current requirements for routine full runs, for different reasons.
DeepSWE: fix the scoring first
Epoch AI's September 7, 2026 review of DeepSWE v1.1 found scoring defects in at least 23 of 113 tasks. The reviewers confirmed false negatives by hand. In 18 cases, editing or adding tests could break the grader, although agents weren't instructed to avoid those edits.
That can turn legitimate work into a failure. We are deferring runs until we can pin a release with the defects addressed and check the revised verifier. More inference spending cannot repair a broken scoring rule.
CyberGym and CyberGym-E2E: infrastructure and permissions
CyberGym's setup instructions call for roughly 240 GB of task data and about 10 TB of full compilation/server data. A binary-only server option is about 130 GB, but omits the dynamic compilation environment.
CyberGym-E2E adds discovery, reproduction, patching and functionality checks across 920 instances. Its harness needs container images and isolated agent networking; we haven't established its storage requirement. The CyberGym storage figures should not be read as E2E measurements.
Permissions also need work. CyberGym's code is Apache-2.0, but its dataset card states no explicit dataset licence. We haven't established the E2E dataset's terms either. A harness licence alone doesn't settle rights to every dataset asset.
Terminal-Bench 4.0: long, expensive trials
The official run guide requires a sandbox with GPU access for GPU tasks. The 4.0 release notes set an eight-hour agent limit per task. Those are task resources, even when inference comes from an API.
Our full-run budget scenarios put the trade-off in dollars:
| Suite | Assumed input/output, million tokens | Inference per route |
|---|---|---|
| CyberGym, 1,507 tasks | 753.5 / 75.35 | $124–$2,769 |
| CyberGym-E2E, 920 tasks | 920 / 92 | $152–$3,381 |
| Terminal-Bench 4.0, 66 tasks | 2,574 / 42.9 | $328–$8,333 |
These are RedRouter planning estimates, not measured bills or benchmark authors' cost claims. The two price scenarios are $0.12/$0.45 and $3.15/$5.25 per million input/output tokens, without caching. Multiply each token column by its rate and add. Each scenario allows one attempt per task; three repeats triple inference spending. Storage, compute, grading and retries cost extra. Actual token use remains unverified.
What we run instead
Our initial selection for periodic task runs is:
- XSTest's safe set: benign over-refusal, weekly.
- OR-Bench Hard: broader over-refusal, monthly.
- CyberSecEval MITRE false-refusal: security prompts, with empty/error responses counted separately, monthly.
- BFCL non-live AST categories: tool-call arguments, monthly.
- Aider polyglot: executable coding and repair, monthly.
- Later pilots: Cybench and CyberGym's official ten-task starter set, with dataset terms resolved first. Starter outcomes won't stand in for full-suite scores.
This is the run selection, not a claim of completed results. We'll revisit DeepSWE when its scoring defects are fixed, and the larger agent suites when costs fall and dataset permissions are clear.
Sources and licensing pages checked November 5, 2026. Cover: NASA public-domain photograph; figure and budget calculations: RedRouter.