For the past year, every new AI model has arrived with a chart showing it acing some benchmark. On Monday, the research firm SemiAnalysis published one that turns that chart upside down. On GDPx, a new test of AI agents doing real professional work, the top-scoring model, Anthropic’s Claude Opus 5.5, manages 12%. Google’s not-yet-public Gemini 4 Carbon comes second at 10.9%. On the finance and legal tasks, SemiAnalysis says “basically every model scores exactly 0.”
The announcement went up on X at 7:36am IST on 2 November and passed 990,000 views in about ten and a half hours. X had logged only a handful of posts mentioning “GDPx” on any day the previous week; by 6pm IST on Monday there were close to 1,500. Google CEO Sundar Pichai quoted it within hours, and it sat near the top of Techmeme all day. Here’s what GDPx is, what it found, and why 12% might be the most useful AI number of the week.
What GDPx actually is
GDPx is a benchmark, a fixed set of tasks used to compare AI models, built by SemiAnalysis’s Max Kan, Andrew Megalaa and Dylan Patel. Version 1.0 has 206 tasks across 11 domains: recursive self-improvement (an AI improving AI systems), hardware design, cybersecurity, biology, finance, legal, quant trading, chip design, software engineering, materials science and alignment, which here means whether an agent behaves like a trustworthy professional.
The name is a dig at GDPval, the OpenAI benchmark released on 25 September 2025 that is still printed on many model launch pages. SemiAnalysis argues GDPval was built “before the rise of agents.” Its tasks hand the model all the context up front and ask for a single document back, and its human expert graders agreed only 71% of the time on its gold subset.
GDPx flips that design. Every task drops the agent into a working environment and makes it find what it needs. One finance setup is a mock private-equity deal with about 4,000 anonymised files, based on a real transaction. Cyber tasks are built around zero-day bugs, flaws not publicly known before, found by the security firm Arithmetic. Agents get up to eight hours per task and run in their makers’ own harnesses where one exists: Claude Code for Opus, Codex for OpenAI’s models, Antigravity for Gemini. The headline number is the pass rate on a single attempt, averaged over three runs, with all 11 domains weighted equally.
SemiAnalysis built the tasks with 14 specialist data companies. It says it approached more than 30 and dropped any that failed its quality checks on the first try, “including some very prominent ones.”
The leaderboard
Here are the 13 models shown by default on the GDPx site, with their overall scores out of 100:
- Claude Opus 5.5 (Anthropic): 12.0
- Gemini 4 Carbon (Google): 10.9
- GPT-6 Astra (OpenAI): 9.6
- Muse Wonder Full: 9.0
- Claude Fable 5.1: 8.5
- GPT 6.1 Sol: 8.4
- Claude Sonnet 5.5: 8.2
- Gemini 4 Argon: 7.3
- Muse Wonder: 6.8
- DeepSeek 4.1 Flash (1017): 5.4
- GPT-6 Luna: 5.1
- Grok 4.7: 3.9
- GLM 5.3: 2.7
There’s an important asterisk. SemiAnalysis says Opus 5.5 and GPT-6 Astra are “sandbagged,” meaning their scores are dragged down, because their safety filters sometimes refuse work in cyber, biology and alignment. It also argues those filters matter when picking a model: by its numbers, Muse Wonder Full is “the best available cyber model today by a meaningful margin.”
No model wins everywhere. According to SemiAnalysis, Opus 5.5 is the best all-rounder and wins quant trading, hardware design and chip design. Gemini 4 Carbon is the best biology model (“likely due to weaker safety classifiers”) and narrowly wins software engineering. GPT-6 Astra wins recursive self-improvement and materials science.
Why Google is so happy about second place
The most-shared reaction came from Pichai, who quoted the launch post: “Great to see high quality benchmarks with headroom and look at that cost line for Carbon.” SemiAnalysis founder Dylan Patel went further: “Official apology to Google from me, they cooked with Carbon. It’s spikier in terms of domains and capabilities, but it’s really good in a few areas.”
Context matters here. As we covered when Google announced Gemini 4 Carbon last week, the model went to cybersecurity partners first, and most developers still can’t use it. SemiAnalysis calls Carbon “the only model in the same ballpark as Opus and Astra” and looks forward to impressions “once Carbon becomes generally available.” For now, the runner-up on GDPx is a model you probably can’t use.
Where the agents fall apart
The write-up is most useful for its examples of how agents failed.
- Finance. On one public task, building a “quality of earnings” workbook for a deal, no model succeeded in 25 attempts. The best single run passed 8 of 11 checks. Models often trusted whichever document stated a figure most plainly over the authoritative one, such as an old adviser schedule over a signed agreement.
- Legal. Asked to redline a startup’s Series B charter for the lead investor, none of 20 runs produced a usable document. Opus 5.5 averaged 44% partial credit and GPT-6 Astra 25%. All 20 runs missed that the company could use a subsidiary to get around the investor’s protections.
- Software engineering. On a Git LFS bug, 25 runs across five models produced zero passes. Across failed runs, the partner Datacurve found 76% of failures were in the requirements around the main fix, such as keeping existing behaviour intact.
The alignment results are the ones people are passing around. In five healthcare tasks, models worked as a doctor’s assistant inside a replica electronic health records system. In 100 runs, only 19 scored above zero. When the chart didn’t support what they had been asked to do, the agents filled the gap themselves. They marked a new consent form as signed by reusing an earlier signature, sent a patient a message saying his imaging was approved after the insurer had denied it, signed a clinical note in the doctor’s name and cancelled another doctor’s order.
Cyber was worse. Across 200 penetration-testing runs, 98% broke at least one rule of engagement, and 87% broke a rule rated “severely misaligned.” In the insurance tasks, 54 of 60 runs leaked confidential client information or committed another critical violation. If you’ve read our coverage of how AI agents quietly steer experts’ judgment calls, or of ARTEX, the AI hacking agent behind the ConnectWise breach, the pattern is familiar. Agents are very good at finishing the task, and not yet good at knowing when they should stop.
The fine print and the pushback
A 12% score doesn’t mean Opus 5.5 is useless at your job. GDPx is deliberately built to be hard, and it scores harshly. Finance and legal tasks are meant to be pass-or-fail, because a single wrong cell in a deal model can sink the deliverable. SemiAnalysis switched those two domains to partial credit only because every model otherwise scored zero. Critics on X seized on this: one noted that a model can get “>60% of verifiers correct yet still score zero %” under binary scoring, and another objected to the “all or nothing thresholding.”
It’s also fair to note who built the tasks. Several of the 14 partners are companies that sell training data and test environments to the same AI labs whose models are being scored, and SemiAnalysis says it is building a revenue model for OpenAI and Anthropic for its subscribers. None of that makes the results wrong, but GDPx is one firm’s view of which work matters most. It deliberately skips government, real estate and insurance, arguing the first trillion dollars of AI revenue will come from a few high-value fields.
SemiAnalysis calls GDPx a “living benchmark” and plans a new version every three to six months. That matters, because benchmarks stop being useful once models ace them, and as we saw with Kimi K3.1 Pro closing in on Opus 5.5, the gap between top models can close in weeks.
What it means for you
If you use AI agents at work, GDPx says something useful about how they fail. They are strong at the obvious part of a task and weak at the context around it: the document that overrides another, the rule that makes the goal off-limits, the person who has to sign. Keep a human on anything that involves a signature, a client’s confidential information or a contract term. And when the next model launches with a 90% score on some older test, remember that the hardest new test in town tops out at 12%.
Leave a Reply