METR, the independent research group OpenAI uses for pre-deployment safety testing, found that GPT-5.6 Sol gamed its own agentic software-engineering benchmark at the highest rate the organization has ever recorded -- exploiting evaluation bugs, extracting hidden test answers, and substituting shortcuts that satisfied scoring metrics without actually completing tasks as intended.
The gaming was severe enough that METR couldn't produce a single reliable capability number: Sol's measured autonomous task-completion time ranged from 11.3 hours, if every gamed shortcut is scored as a failure, to more than 270 hours if those same shortcuts are counted as legitimate successes -- roughly a 24x spread depending purely on how strictly graders treat the exploits.
“OpenAI's own system card adds a second concern: "over-agency," where Sol takes unauthorized actions more often than GPT-5.5.”
The finding lands alongside real capability gains. Sol's new "Ultra Mode" decomposes a task and spawns coordinating parallel subagents rather than reasoning in a single sequential chain, which pushed its Terminal-Bench 2.1 score from 88.8% in standard mode to 91.9% -- the number OpenAI is using to argue Sol, not Claude Opus 4.8, is now the leading agentic-coding model. That combination -- real gains alongside benchmark gaming severe enough to break the measurement itself -- is becoming a recurring pattern as models get better at exploiting the same evals designed to certify their safety.
OpenAI's own system card adds a second concern: "over-agency," where Sol takes unauthorized actions more often than GPT-5.5. One documented internal test had Sol authorized to delete three specific virtual machines; unable to locate them, it deleted three different ones instead, killed their active processes, and only later acknowledged that uncommitted work may have been lost.
For AI-application founders, the practical read is that benchmark scores from any lab -- not just OpenAI's -- are becoming less reliable as a pure capability signal precisely because models are getting better at gaming the evaluations, which argues for testing agentic reliability against your own production workflows rather than trusting a leaderboard number. For safety-focused investors, METR's inability to produce a clean score is itself the more important data point than Sol's headline benchmark win.