🔍 Read the full analysis: How To Read Gemini 4 Carbon’s Security Scores For An AI That Can Break In And Fix A Lock on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Google says Gemini 4 Carbon scored 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench, covering vulnerability discovery, exploitation and repair. Google ran two tests itself; Collinear AI ran the third. The results point to potential value for cyber defenders, but differing test conditions and the lack of public independent testing limit direct comparisons.
Google reported scores of 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench. The tests cover different tasks rather than measuring one overall cybersecurity ability: triggering known bugs, turning crashes into working exploits, and finding and repairing flaws in code.
Google says it ran SEC-bench Pro and ExploitGym using its evaluation setup. Collinear AI, the company that built CWE-bench, runs that benchmark’s official leaderboard. On the private 120-task test set, Carbon’s reported 71% is three percentage points above the 68% reported for Gemini 4 Argon, GPT-6 Astra and Grok 4.7, and four points above Claude Opus 5.5 at 67%.
The ExploitGym headline score requires particular care. Google’s 54.8% result used a 10-hour limit; Google also reports 53.5% within six hours. The source material compares the latter with Anthropic’s 34.5% for Claude Opus 5.5 within six hours, but that figure comes from Anthropic’s own report. Results from different labs may use different tools, graders and settings, so they do not establish a controlled head-to-head comparison.
Infographic 01 · What Carbon's three scores measure
Find, exploit, fix: three tests, and only one outside referee.
Google says Gemini 4 Carbon leads all three security benchmarks it reported. Here is what each one asks of the model, and who produced Carbon's number.
1Find · SEC-bench Pro
- The model gets
- Source code, the area to search, the bug type and the expected error. Not the input that triggers the bug.
- It counts when
- The model writes an input that triggers a known, already-fixed bug in a browser JavaScript engine.
- Carbon’s score came from
- Google, 183-task May 2026 version, average of 3 attempts Google's own run
2Exploit · ExploitGym
- The model gets
- An input that crashes the program. Standard protections switched off.
- It counts when
- The model turns the crash into its own code running on the target and reads a hidden flag; an AI judge checks it used the intended bug.
- Carbon’s score came from
- Google, one attempt per task, 10-hour limit (53.5% at 6 hours) Google's own run
3Fix · CWE-bench
- The model gets
- A real open-source code base and one instruction: audit it and fix what you find. No hint what or where.
- It counts when
- The attack stops working and every existing test still passes. Models never write exploits.
- Carbon’s score came from
- Collinear AI, 120 private tasks, all models run by Collinear Outside run
Read the fine print
- Google tested without safety filters (“evaluated without guardrails”); Anthropic and OpenAI report their cyber results with safeguards off too
- Rival numbers come from the rivals’ own system cards, with their own time limits and graders
- SEC-bench Pro and ExploitGym use public bugs with public fixes; CWE-bench keeps its test set private
Finding is not discovering
- SEC-bench Pro tells the model where to look and what kind of bug to expect
- That is closer to a detective with a tip than to a hunt for unknown flaws
- Real attacks also need targets, delivery and staying undetected; real fixes need review
The same skill that proves a bug is dangerous is the skill an attacker needs.Thorsten Meyer · my reading
Sources: Google, 28 Oct 2026; Google methodology note, Oct 2026; CWE-bench leaderboard, 28 Oct 2026; SEC-bench Pro paper; ExploitGym paper; OpenAI GPT-6 Astra system card, 3 Sep 2026. Lead margin and counts as in the article; margin is my arithmetic.
Powered by Thorsten Meyer AI
Security Tools With Dual-Use Risks
One model’s ability to find and exploit weaknesses can help defenders locate problems before attackers do, but the same capabilities could assist someone trying to break into systems. That dual use helps explain Google’s decision to limit early access to vetted defenders, rather than offer Carbon broadly.
For security teams, the scores are an indication of possible utility, not proof that the model can safely handle production systems. The repair benchmark is a reminder that automated fixes still need review: even the leading reported result leaves nearly three in ten tasks unsuccessful. A repair must stop the attack while preserving existing tests, and an apparently plausible code change is not enough.
For the public, the report does not indicate an immediate change to consumer protections or software updates. It does raise a practical question for companies adopting AI security tools: how will they validate model-generated findings and fixes, and control who can use capabilities that also support exploitation?
Products Worth Considering
Dell UltraSharp 32 U3226QER 4K Thunderbolt Hub Monitor
Dell P2725H 27-Inch Full HD Monitor
Samsung Essential S3 27-Inch Curved FHD Monitor
As an affiliate, we earn on qualifying purchases.
Three Tests, Three Different Tasks
SEC-bench Pro is based on previously fixed bugs in JavaScript engines and, in its newer version, the Linux kernel. The model receives source code, an area to inspect, a bug type and the expected error, then has to produce an input that triggers the bug. It is a test of reproducing known vulnerabilities under guidance, not evidence that a model can independently discover unknown flaws across software.
ExploitGym uses 869 real vulnerabilities. The model is given a crashing input and must turn it into an attack that runs code and retrieves a hidden flag. Google’s methodology, as described in the source material, switched off protections such as memory randomization. That makes the result informative about performance under the test’s setup, but not a direct measure of success against every real-world target with its protections enabled.
CWE-bench asks a model to audit and repair a codebase without being told where the flaw is or how many flaws it contains. A fix counts only if the attack no longer works and existing tests pass. Its 120 tasks are private, according to the source material, to reduce the risk that test items have appeared in training data. The reported results show that five tasks defeated every tested model.
Google’s method note says the company calculated its SEC-bench Pro and ExploitGym results using the open-source evaluation setup. The source material says the SEC-bench Pro version contains 183 tasks from May 2026 and that Google averaged Carbon’s result across three attempts using its Antigravity agent tool. OpenAI separately reported 85.4% for GPT-6 Astra on that version, while Gemini 4 Argon scored 83.4%; the supplied material says no Claude score is available because of safeguard restrictions.
““self computed” SEC-bench Pro and ExploitGym “using the open source evaluation setup”.”
— Google, in its methodology note
Limits of the Published Comparisons
No public independent test of Carbon’s security skills is identified in the source material. Two scores were produced by Google, while Collinear AI runs the third benchmark’s leaderboard. The source says Artificial Analysis evaluated and published CWE-bench results, but that does not amount to an independent public replication of all three tests.
Comparisons also have uneven conditions. The ExploitGym figures use different time limits and are drawn from reports by the respective model developers; the source material does not establish that tools, graders or other settings were identical. SEC-bench Pro results are also not complete for every rival. The scores therefore cannot show with certainty how Carbon would perform against other models in a common, independently administered trial.
It is also unclear from the supplied material how Carbon performs on unknown vulnerabilities, on systems with real-world defenses enabled, or across a broader range of software and operational environments. A benchmark result is not evidence that a model has breached a live system. Google’s security claims should be understood as reported test performance, not proof of real-world attack or repair outcomes.
What to Watch in Access and Testing
The immediate next development is the restricted rollout. Google says Carbon is going first to vetted cyber defenders and is included in the U.S. government’s voluntary pre-release testing. The source material does not give a schedule for broader availability or identify the participating organizations.
Readers and security teams can watch for further details about access controls, the results of government testing and any public evaluation using shared conditions across competing models. Independent replication would help clarify whether the reported performance holds beyond the developers’ own testing arrangements.
Until more evidence is available, organizations considering AI-assisted security work should treat benchmark scores as one input, not a substitute for controlled testing and human review. The source material specifically cautions companies to budget for review of AI-written fixes. For individual users, it reports no immediate change: keeping software updated remains the practical step highlighted in the account.
Key Questions
What do Gemini 4 Carbon’s three security scores measure?
SEC-bench Pro measures triggering known bugs, ExploitGym measures turning crashes into working exploits, and CWE-bench measures finding and fixing code flaws while preserving existing tests.
Did Google run all three tests?
No. Google says it ran SEC-bench Pro and ExploitGym using its evaluation setup. Collinear AI runs the CWE-bench leaderboard, according to the supplied source material.
Is Carbon’s ExploitGym score directly comparable with rival models?
Not on the information provided. Google’s headline result used a 10-hour limit, while the source also reports results at six hours. Rival figures come from their developers’ reports, and tools, graders and settings may differ.
Can the public use Gemini 4 Carbon now?
The source says Google is initially providing access to a small group of vetted cyber defenders. It does not give a date for broader access.
Do the results show Carbon has attacked real systems?
No. They are reported results on benchmarks. The source material does not identify a real-world breach by Carbon or provide a public independent test of its security skills.
Source: ThorstenMeyerAI.com
