Consumer Protection Tuesday: Better Together: AI and Human Expertise
Tl;dr: AI has fundamentally changed who finds software vulnerabilities, how fast, and at what cost. The most complex, highest-impact bugs still hinge on human judgment though. A recent vulnerability in how we reconciled Stellar withdrawals is a great example of that. It was reported by external researchers via our public bug bounty program and fixed quickly. No customer funds were affected. Our takeaway is simple: When it comes down to the most impactful bugs, AI and human expertise together are far stronger than either one alone. This is the model we are building our security program around.
The economics of finding bugs have changed
For most of the history of software security, finding a vulnerability required a scarce combination of skill, patience, and time. That’s no longer true for commodity bugs.
Frontier AI models can now read a codebase, reason about how its pieces fit together, and surface plausible security flaws at a speed and scale no individual can match. For most common bugs, discovery has shifted from being a function of human effort to being a function of how many tokens you are willing to spend.
That shift is showing up across the industry, and it cuts in three directions at once:
Defenders are integrating AI into every stage of the software development lifecycle, from code review on every pull request to large-scale analysis of existing products.
Researchers are pointing AI at systems from the outside, which correlates with a sharp rise in AI-generated bug reports. We are on track to receive 3X more reports this year, than last year, and last year was 2X the year before. Meanwhile, the number of valid reports has declined from 14% in 2024 to 4% this year. Industry data reflects a similar trend.
Attackers are using the same models to hunt for bypasses at scale, on their own timeline.
The uncomfortable truth is that no defender can match that speed with human effort alone. Staying ahead requires using AI as aggressively as everyone else in the ecosystem, including the people we are defending against.
We’ve transformed our vuln discovery pipelines across the SDLC lifecycle to leverage the best AI models available. As a result, our tooling can now continuously catch commodity issues at scale. AI cannot catch all issues, though. And this is why we’ve refocused our bug bounty program on the vulnerabilities that require genuine human ingenuity.
Where AI stops and human expertise begins
AI can accelerate vuln discovery, but human expertise continues to have an edge, especially when it comes to the most impactful bugs.
A recent submission to our bug bounty program illustrates the point well.
Two external researchers, Joe Almeida (hackerontwowheels) and Anh Nguyen (ledz1996) from Talaria Security Labs, reported a subtle flaw in our reconciliation of Stellar withdrawals. Stellar supports a native fee-bump feature that lets any third party take an existing transaction and “wrap” it in a new one that pays a higher fee without having to re-sign or manage sequence numbers. Fee-bumping is a legitimate mechanism designed to help others pay for your transaction fee or to help your transactions land onchain faster. The edge case was in how we reconciled a wrapped transaction after the fact: under certain conditions, the original transaction could be treated as failed during reconciliation even though the intended transfer had actually succeeded onchain, creating the potential for a spend to be double-counted internally.
A few things are worth being direct about:
No customer funds were affected. Our investigation found no evidence of real-world exploitation beyond the researcher's proof-of-concept and our own internal testing, and we moved quickly to pause the affected flow, confirm a fix, and restore normal processing.
This was not a bug that AI would easily catch. It lived at the intersection of a protocol-specific feature and our own reconciliation logic, the kind of "you had to understand both sides deeply" problem that rewards human creativity.
AI still plays a role. While AI did not find the specific issue Joe Almeida and Anh Nguyen reported, it did flag a related, albeit less severe, bug on the deposit side.
That is the whole thesis in miniature. While AI, in its current form, has become extraordinarily good at finding commodity bugs, it lacks the creative edge of humans. Identifying the most complex and consequential vulnerabilities continues to be an area where humans excel at, and the high-quality work by Joe Almeida and Anh Nguyen is one example of that.
Building for AI plus human, not AI versus human
We're designing our security program around the AI+human combination rather than betting on either extreme:
AI handles the scale. Continuous, automated reviews across sensitive services and new code changes, so that the well-understood classes of issues are caught early and consistently.
Humans do the judgment. Our security engineers and the external research community focus their time on novel, high-severity, system-specific vulnerabilities, i.e. the problems where context, creativity, and deep domain knowledge are decisive.
The two reinforce each other. Human-discovered bugs teach our automated systems new patterns to hunt for and automated analysis frees human experts to go deeper on the problems that actually move the needle.
This is an evolving balance, not a solved equation. AI models are getting better quickly. The threat landscape is shifting just as fast. We expect to keep adjusting where we draw the line between what we automate and where we lean on human expertise. These are genuinely hard trade-offs that no one seems to have perfectly figured out yet.
We are confident about the direction. The reconciliation case is a contained example of a much larger principle: the strongest security posture doesn't come from choosing between artificial intelligence and human expertise — it comes from leveraging both.
Thank you to all the researchers who partner with us! Thank you to Joe Almeida and Anh Nguyen for working closely with us to remediate the reconciliation issue quickly. The work you do is more valuable now, than ever before. And to all the builders and defenders in this space, our advice is the same one we are following ourselves: use AI to move at the speed of the threat, and never stop investing in the human expertise that tells you where to point it.
