OpenAI o3 Benchmark Results: What the Scores Mean and Where They Mislead

OpenAI o3 scored 87.7% on GPQA Diamond, 71.7% on SWE-bench Verified, 96.7% on AIME 2024, and a record 87.5% on ARC-AGI — but that last number cost roughly $3,400 of compute per task. That's the short answer. The longer answer is more interesting.

When OpenAI previewed o3, the benchmark slides looked like a generational leap. Most of them are. But a benchmark is a measurement under conditions, and o3's best conditions are not the ones you get when you call the API. Knowing which scores transfer to real work — and which are lab demonstrations — is the difference between a smart deployment and an expensive surprise.

OpenAI o3 benchmark results, by task

OpenAI o3 is a reasoning model that spends test-time compute "thinking" before it answers, and its benchmark results were the strongest OpenAI had published at release: 87.7% on GPQA Diamond, 71.7% on SWE-bench Verified, 96.7% on AIME 2024, 25.2% on EpochAI Frontier Math, and 87.5% on ARC-AGI in its high-compute configuration. Each measures a different capability, and they do not all transfer to production equally.

Figures are from OpenAI's o3 preview and the ARC Prize Foundation's independent evaluation. o1 figures are the comparable release-time numbers.

What the o3 benchmark results actually mean

The single most important o3 result is SWE-bench Verified at 71.7%. This benchmark hands the model real, previously-unseen issues from open-source repositories and checks whether its patch passes the project's own test suite. Jumping from o1's 48.9% to 71.7% means o3 resolves roughly seven of every ten genuine software tasks unaided — the gap between "helpful autocomplete" and "junior engineer who can close a ticket." For agentic coding products, that is the number that matters.

The ARC-AGI score and the cost caveat the headlines skip

ARC-AGI is designed to resist memorization — each puzzle requires inferring a novel transformation rule from a handful of examples. No prior model had cracked it; GPT-4o sat near 5%, o1 around 32%. o3 posted 75.7% in a low-compute setting and 87.5% in a high-compute setting, the first time any system passed the ~85% human-level bar. That is a real milestone.

Then read the footnote. The ARC Prize Foundation reported that o3's high-compute run burned roughly $3,400 of compute per task — and there were 400 tasks. Even the low-compute configuration that scored 75.7% ran around $20 per task. A human solver costs a few dollars. So o3 proved the capability is reachable, not that it is affordable. When you call the o3 API at default settings, you are not getting the $3,400-per-task brain; you are getting a far cheaper configuration that scores meaningfully lower.

Where the o3 benchmark results fall short

Even taking the numbers at face value, four problems limit how much you should read into them:

o3 vs o1 vs o3-mini: which one should you actually run?

The benchmark gap between o3 and o1 is real and large — a 22-point SWE-bench jump is not marketing. But for most teams the relevant comparison isn't o3 versus o1; it's o3 versus o3-mini versus a non-reasoning model like GPT-4o. o3-mini captures the bulk of the reasoning gains at a fraction of the latency and token cost, which is why it became the default reasoning workhorse for production. Reserve full o3 for the genuinely hard cases — research-grade analysis, complex multi-file refactors, agentic loops — where the extra accuracy clears the 10x-plus cost premium. For anything routine, the cheaper model is the rational choice, and the benchmark crown is irrelevant to your unit economics.

This is the same dynamic reshaping AI startup pricing and valuations more broadly — capability is racing ahead while cost-per-useful-task becomes the real battleground. We track how those model economics flow into company pricing on the AI Valuations dashboard.