Skip to content
Cyberttak.
Autonomous agents· 3 min read

Gemini 4 Argon fixed 67.8% of real security bugs on first try in CWE-bench v2, with Claude Opus 5.5 close behind

The public test has 100 tasks across 91 weakness types, and even the leader fails 19% of them on every attempt.

By CyberttakPublished · 22:04

In brief

AI fixed 67.8% of real security bugs

CWE-bench v2 gives agents 100 real security flaws to fix. Gemini 4 Argon led 11 models. Even so, it fails 19% of tasks on every attempt.

A robotic arm repairs a cracked steel vault door in a workshop, with sparks flying from a welding torch beside padlocks and a clipboard on a workbench
A robotic arm repairs a cracked steel vault door in a workshop, with sparks flying from a welding torch beside padlocks and a clipboard on a workbenchCredit: AI-generated · FLUX.2 klein 9B

Collinear AI has released CWE-bench v2, a public test of how well AI agents can fix security flaws in real software. The post comes from Hugging Face, and the benchmark was run on 11 models.

How the test works

The agent gets a copy of a real open-source project and a short instruction. Its job is to find a security weakness and fix it without breaking anything that already works.

Grading is done by code, not by a human or a chatbot. So the results are the same every time, and anyone can reproduce them at no extra cost.

What is in the set

The public set has 100 tasks. They cover 91 different CWEs, which are standard labels for types of software weakness. Spotting a familiar pattern is not enough to pass.

Collinear says each task was checked by hand. The checks included whether the instruction sounds realistic, whether the starter code fails and the reference fix passes, and whether the known upstream patch is rejected. A separate held-out set stays private to track models on tasks they have not seen.

The scores

  • Gemini 4 Argon led with 67.8% pass@1 and 81.0% pass@4.
  • Claude Opus 5.5 followed with 67.3% pass@1 and 78.0% pass@4.
  • Pass@1 means solving a task on the first attempt. Pass@4 means solving it in at least one of four attempts.

Across all models, 95 tasks were solved at least once, so the tasks can be done. Even so, the leader fails 19% of the set on every attempt.

Cost

DeepSeek V4.1 Flash (1017) reached 57.0% pass@1 at $0.04 per rollout, meaning per attempt. Claude Opus 5.5 added about ten points at $1.83. Gemini 4 Argon edged ahead at $3.92. These three form the best trade-off between price and score.

Run it yourself

Every task is a self-contained Harbor task, with a Docker environment, an instruction and a verifier. Harbor can run the whole set straight from the Hugging Face repo. The dataset is gated, so you accept its access conditions first. The tasks are also tagged for reinforcement learning, so they could reward defensive agents during training.

The limits: the scores come from the benchmark's own authors and cover only 11 models. The held-out results are not published, and testing on it requires contacting Collinear. The source gives no sign of how these scores translate to everyday security work.

Source: Hugging Face Blog