Benchmarks
Independent, reproducible measurements of the knobs you can actually set on a request: models, providers, search engines, and tool budgets. Every score links to the configuration and its cost.
18 benchmarks2.378.275 task evaluationslast run 24 Okt 2026
Agents & tools
| Benchmark | Quality | Value | Speed |
|---|---|---|---|
τ²-Bench Airline› Multi-turn service agents making tool calls under strict policy constraints. 150 models · 24 Okt 2026 | 79,9% CClaude Sonnet 4.5 | $0,009 LLlama 4 70B | 42s LLlama 4 70B |
SWE-Bench Verified› Resolve real GitHub issues end to end in a sandboxed repository. 96 models · 24 Okt 2026 | 72,4% GGPT-5.2 | $0,041 GGPT-5.2 mini | 3m 12s GGPT-5.2 mini |
Media
| Benchmark | Quality | Value | Speed |
|---|---|---|---|
Image› Image prompts built to fail: depth, direction, counting, and text in the frame. 59 models · 24 Okt 2026 | 84,1% GGemini 3 Pro | $0,007 GGemini 3 Pro | 2s GGemini 3 Pro |
Video› Six seconds of video, held to one duration and resolution across every model. 27 models · 24 Okt 2026 | 71,0% GGemini 3 Pro | $0,17 LLlama 4 70B | 30s LLlama 4 70B |
Speech› One sentence, every voice: did the words come back, and was direction performed? 35 models · 24 Okt 2026 | 96,2% GGPT-5.2 mini | $0,000 GGPT-5.2 mini | 1s GGPT-5.2 mini |
Artifacts
| Benchmark | Quality | Value | Speed |
|---|---|---|---|
Artifacts› Single-file apps rendered and interacted with, scored on behaviour not looks. 44 models · 24 Okt 2026 | 68,8% CClaude Sonnet 4.5 | $0,031 CClaude Haiku 4 | 48s CClaude Haiku 4 |
Reasoning
| Benchmark | Quality | Value | Speed |
|---|---|---|---|
GPQA Diamond› Graduate-level science questions written to resist lookup. 180 models · 24 Okt 2026 | 81,3% GGPT-5.2 | $0,004 GGemini 3 Pro | 6s GGemini 3 Pro |
Search
| Benchmark | Quality | Value | Speed |
|---|---|---|---|
Search grounding› Answer freshness and citation accuracy against live web results. 12 models · 24 Okt 2026 | 88,0% GGemini 3 Pro | $0,002 GGemini 3 Pro | 4s GGemini 3 Pro |
Routers
| Benchmark | Quality | Value | Speed |
|---|---|---|---|
Router efficiency› Cost per resolved task when the router picks the model. 8 models · 24 Okt 2026 | 74,6% aaiget/auto | $0,006 aaiget/auto | 9s aaiget/auto |