The first time my AI system caught a High-severity bug I'd have missed, I knew the game had changed. Most of the field back then thought security research will be one of the last things that get automated.
That was a year ago and things have changed dramatically since then.
AI is finding real vulnerabilities in production - in audits, in live bounties, on the contest leaderboards - for real money, today.
So the question worth our time isn't whether AI can find bugs in smart contracts. It can.
The real question is how good these systems get, and how you tell one that works compared to "find all the bugs" pasted into a chatbot.
This is a map of what AI for web3 security actually is, where it works, and where it doesn't.
What AI for web3 security actually is
Let's start with a simple definition so we're on the same page.
AI for web3 security is the use of AI reasoning systems to find vulnerabilities across the onchain stack - smart contract code, protocol design, cross-protocol economic attacks - with performance you can measure on real engagements.
If you want to build in this space, or judge what others build, two words in there are the whole game: reasoning and measurable.
Reasoning. Slither matches patterns: it knows the shape of a reentrancy bug and flags what looks like one. It only finds what it was taught to.
Reasoning understands what code is supposed to do and finds where the implementation drifts. That gap, the intent gap, is where the bugs that matter live.
The best ones you'll ever find aren't exotic patterns; they're a correct-looking function doing something its author never meant.
Measurable. A system you can't put a number on isn't a security system. How you measure one is the difference between a claim and a result. Just running your system in production is fine and it may very well find real bugs. But if you don't measure it, you can't conclude you're improving it. A system being measurable is a prerequisite for making it better.
The definition doesn't require a proprietary model. Most systems run on frontier models (Claude, GPT, whatever's sharpest), wrapped in the engineering that makes them reliable.
You can call that a wrapper, sure. But a model wrapped in a good system (or harness) currently gives you an edge compared to using just the plain models.
"It's just a wrapper" is a lazy take. Lazy because it doesn't require you to build anything and you can sit back and relax telling yourself and others "these wrappers will get extinguished with the next Claude update".
Everything is a wrapper. What matters is what you wrap it in, and whether it finds the one bug that matters or a hundred that don't.
The five levels of AI security systems
Currently, most of the SR practitioners are building their own AI auditor, systems, wrappers, harnesses, tools, whatever you want to call them. But the term "AI auditor" is extremely vague.
One "AI auditor" pipes your contract into a chatbot and prints whatever comes back. Another reasons across the whole protocol and catches the bug your audit team missed.
So let's give them some levels for defining where a tool stands on a scale.
| Level | What it is | What it finds |
|---|---|---|
| L0 · Explainers | An LLM that explains Solidity | Nothing on its own; you have to know what to ask |
| L1 · Pattern matchers | Pattern matching with a language model on top | Known bug shapes, in plain English |
| L2 · Reasoning agent | One agent, one pass over the code - may span a whole small protocol in context | Real logic bugs (the intent gap) |
| L3 · Orchestrated systems | Specialized agents + a judge + protocol context | Bugs across the whole protocol, filtered |
| L4 · Proven systems | An L3 system with measured recall on real ground truth | The same, but proven, not asserted |
Two things fall out of this. First: the label "AI auditor" tells you almost nothing on its own. The same two words cover L1 and L4, and the levels are how you tell them apart.
Second: the jump that actually matters is L2 to L3, where one clever agent becomes an orchestrated system with judgment. That's where a demo turns into something you'd trust on a real codebase.
From the outside, though, L3 and L4 look identical: both claim to find serious bugs. The only thing that separates them is whether the claim is measured - which is where recall comes in.
How do you measure an AI security system?
There are a lot of different approaches and different teams choose to prioritize one metric over the other. Recall, precision, efficiency, speed.
For me - recall is the most important metric.
A false positive costs you a few minutes ruling it out.
A false negative is a vulnerability that shipped, and then got exploited.
Precision is a cost optimization metric - it makes you, or other triagers, spend less time verifying issues. Which is fine. Cost optimization should be done.
But we don't measure the effectiveness of a tool by just measuring how cost-optimized it is.
Vulnerability discovery is what brings value. So the number that matters isn't how clean the output looks; it's recall: of the real vulnerabilities in the code, how many did the system actually find?
Measured against a known answer key: past contests, disclosed exploits, audits with a settled ground truth.
Precision still matters. A system that flags every line has perfect recall and is useless, so the real target is recall subject to a precision floor a human can triage.
Recall is the metric. Everything else is optimization. Deep vulnerability discovery is the hard part.
Triaging or judging has always been an easier task done by humans. You have to verify a statement. You do not have to think creatively and chain multiple small findings into big exploits.
That's why finding Critical, hard-to-find vulnerabilities has always been, and still is, very rewarding.
How to improve an AI security system?
- Get a database of ground truth - real codebases
- Measure your system rigorously
- Make adjustments
- Measure again
- Iterate
Iteration is all you need.
Choose hard targets. Do not be satisfied with mediocre performance.
I can tell you one thing - there are AI systems, currently existing, that outperform human teams on real production audits. Consistently.
You can build such a system too.
Human work is still very much needed. Domain knowledge is one of the most valuable and hard to get things you can have right now. Being a security researcher has given you invaluable skills like critical thinking, deep focus, and a sharp mind.
Getting good with AI can be done in a few weeks. Getting the domain knowledge in web3 security requires years - and AI can't speed that up.
I've spent years finding vulnerabilities manually. Now I build systems.
FAQ
Does AI just hallucinate vulnerabilities?
Yes. A single naive prompt does. It will invent plausible-looking bugs all day. That's what the judge layer and a recall benchmark are for: the judge filters findings for validity and severity, the benchmark measures how often the system is actually right.
Will AI replace human auditors?
No. It changes the job, it doesn't erase it. The responsibility, the judgment (to some extent) stay human, and the strongest systems are built by people who can find real bugs manually.
Can it find novel bugs, or only known patterns?
It reasons about what the code is supposed to do versus what it does, so it finds logic bugs no one wrote a rule for: the intent gap, not just pattern matches. Known patterns are the floor, not the ceiling.
The next era
The field is young, and that's the opportunity. The people who'll be best at this in a few years are the ones getting their hands dirty now, while it's still taking shape.
You don't need to be a senior engineer to be one of them. You need security taste: knowing what a real bug looks like, and what a protocol is trying to do. The models handle more of the coding every month; same for cybersecurity.
If you can find one bug by hand, you can direct a system to find a hundred.
I'm writing a hands-on series that builds one of these systems from the ground up, level by level. It isn't out yet, but that's where the how will live. Follow along at 0xasen.xyz/writing, or @asen_sec on X.
You're not late in web3 security. You're early for the next era.