AI can find your vulnerabilities faster than ever, but it still can’t be trusted to fix them

For the last two years, the security industry’s AI narrative has centered on offense: how quickly large language models can find bugs, chain exploits, and surface vulnerabilities that would have taken human researchers weeks to uncover. That capability is real, and it’s accelerating — efforts like Anthropic’s Project Glasswing have demonstrated AI-driven vulnerability discovery operating at a scale no human red team could match.

That success has pushed a growing number of security and platform teams toward an obvious next question: if AI is this good at finding vulnerabilities, can it also fix them? Vendors have raced to answer “yes.” OpenAI’s Project Daybreak, announced in June alongside a coalition of partners, was explicitly framed around using AI to “patch the planet” — closing the remediation gap at the same speed AI is widening the discovery gap.

New research from 1Password’s newly formed security research group, Off-by-1 Labs, suggests CIOs and CISOs should treat that promise with considerable caution — at least for now.

The finding: more than half of AI-generated patches for complex vulnerabilities fail or introduce new risk
In its inaugural study, Frontier Models’ Vulnerability Patches Are Often F.L.A.W.E.D., Off-by-1 Labs set out to answer a narrower and more rigorous question “can a model produce a patch that actually closes a real, recently disclosed, complex vulnerability — without breaking or altering how the application behaves.”

To keep the test honest, researchers deliberately selected six vulnerabilities disclosed too recently to plausibly appear in the training data of the frontier models being tested, including flaws in the Linux kernel, Apache ActiveMQ, Chrome’s File System Access API, Exim, Spring AI, and the Gemini CLI. Using two cyber-capable reasoning models — OpenAI’s ChatGPT-5.5 and Anthropic’s Opus 4.8, each operating under their respective vendor’s cyber-safety guardrails — the team generated 6,080 patches across varied prompts and environment configurations.

The results undercut the industry’s working assumption that AI-assisted patching is close to production-ready for hard problems:

Only 26.0% of patches fully resolved the vulnerability without materially changing the application’s behavior — the only outcome a defender should consider a clean win.

20.1% of patches did technically close the vulnerability but altered application behavior in the process, in some cases by silently swapping allow-list logic for deny-list logic or reimplementing file-local parsers.

A combined 53.9% of patches either failed to fix the vulnerability, introduced a new one, or both.

Perhaps most concerning for security leaders: even among the patches counted as “successful,” Off-by-1 Labs found that more than a third contained fragile, narrowly targeted fixes — patches that blocked the specific proof-of-concept input used to demonstrate the vulnerability while leaving the underlying flawed logic intact. In the Spring AI case, both models repeatedly produced patches that simply escaped the exact characters used in the test exploit, rather than addressing the root cause. Change the input string and the vulnerability reappears.

Cost isn’t the constraint — trust is
One data point should reframe how CIOs think about the economics of AI-assisted remediation: the inference cost per patch-and-validation cycle averaged $2.11 for ChatGPT-5.5 and $2.81 for Opus 4.8 — trivial next to the fully loaded cost of an engineer’s time.

But cheap and fast isn’t useful if it isn’t reliable. With roughly a 1-in-4 chance of getting a clean, behavior-preserving fix on a complex vulnerability, any organization treating AI-generated patches as a autopilot remediation pipeline is effectively gambling on production stability and security posture in the same move. A patch that appears to close a CVE but quietly reintroduces it under different input conditions is arguably more dangerous than an unpatched, known vulnerability — because it creates false confidence in a closed ticket.

What the AI labs themselves are saying
Off-by-1 Labs shared its findings with the model vendors before publication. Anthropic’s response, notable for its candor, acknowledged that patch generation capability has now outpaced patch verification capability, and argued the fix isn’t to slow down generation but to make verification execution-grounded — testing patches against real behavior rather than reviewing them by inspection alone — while keeping human domain experts as the final checkpoint at current model capability levels. 1Password says it “strongly agrees” with that framing based on its own results.

That’s a meaningful admission from a frontier lab, and it lines up with what security leaders have learned the hard way from every previous wave of AI-assisted tooling: capability and reliability are different curves, and they don’t move together.

CIOs and CISOs must treat every AI-produced patch as a draft from a fast, prolific, occasionally overconfident junior engineer — one that requires the same domain-expert review a human-written patch would get for a critical system, not less.

They must also build execution-grounded validation into your pipeline, not just code review. Off-by-1 Labs’ data suggests inspection-based review alone will miss the most dangerous failure mode: a patch that passes the specific test case but leaves the underlying vulnerability class exploitable. Regression and behavioral testing against the original exploit and plausible variants should be a hard gate before any AI-generated patch ships.

One can expect uneven performance by vulnerability class and codebase. The study found wide variance across the six CVEs tested — some vulnerability types were far more tractable for AI patching than others. Before rolling out AI-assisted patching broadly, pilot it against previously resolved vulnerabilities in your own codebase to identify where it performs well and where it doesn’t.

CISOs must also watch for “fragile fixes” specifically. Input-sanitization-style patches that block a known proof-of-concept string, rather than addressing the vulnerable logic path itself, were a recurring failure pattern. These are the patches most likely to pass a quick review and fail in production against a slightly different attack.

AI has decisively shifted the offense-defense balance in vulnerability discovery. This research is a clear signal that the same shift has not yet happened on the remediation side — and that the gap between the two is itself a risk. As Off-by-1 Labs puts it, attackers only need to be right once; defenders need to be right every time. For now, that means keeping a skilled human in the loop as the final reviewer of any AI-generated patch touching a system that matters — no matter how fast or cheap the model made getting there.

1Password has released its full dataset of generated patches, along with the tooling used to generate, validate, and compare them, so security teams can run the same methodology against their own codebases.