AI security vulnerability scanning in Lighthouse

Transcript

[music] >> I'm King from Sigma Prime. Um, thanks for coming to uh another AI in security talk. So, hopefully we'll you know, we'll get through this one pretty quick. Um, Lighthouse has had 131 releases uh over the last six hard forks. And until 2026, we'd credited five vulnerabilities in our release notes. And then in 2026, we credited 21. So, you might be thinking, "Did we turn on the slop cannons? Were you just producing a a lot of rubbish and that's why we got all these vulnerabilities?" Well, I don't I don't think so because 16 of those 21 were in production code for at least 2 years.

Um, and it doesn't seem to be a Lighthouse-specific problem. The EF security researchers uh have told us that uh in the bug bounty program, they've had as many submissions this year as they had all of the previous years put together. You've all seen the uh CVE hockey stick graph. Justin threw it up the other day. It seems like a a pretty serious problem that we're dealing with. Um, but along with that graph comes a a lot of uh false reports. So, 19 of the 57 that were passed through to us by EF security researchers we deemed valid. So, that's not a great hit rate. That means two in three were invalid and we spent a lot of time triaging them. But, uh and this is the reduced set, right?

EF security have looked through and they weeded out all the obvious bad things. So, this is the the best of the submissions. Uh and we're still at we're still at like a a 1/3 hit rate. Um, why why are we getting so many vulnerability hits? Why why are we getting uh all of these bug reports? Both teams have the same tools, right? We have AI agents, they have AI agents. I have a Claude Max subscription, the guy the other guy has a Claude Max subscription. Why are we both uh why are we not on a level playing field? And I think it's the asymmetry of the problem. So, a blue team needs to cover the whole code base.

They need to find at least enough to make it uneconomically viable for the attacking side. Um, they don't need to be perfect, but they they have to be enough that it's no longer worth the while for the red team. But the red team, they just need one path in, and that's something that the AI agents are are pretty good at, right? That you give them a well-defined task, and you give them a goal, you get a cookie when the node crashes, and you leave it running long enough, they're probably going to find a node crash. Um, so we built Sexton at Sigma Prime to try and try and make the task for blue more like the task for red. And and how we do that is we take in a code base, we convert it to a types knowledge graph, and then we look for structural zones that might

represent those code paths into a vulnerability. So, we call those threat zones. Um, they might look like an entry point to an effect, or two entry points to an effect. Uh, it could be a race condition, could be like a shared cache problem. Um, and we take each of those, and then we enhance them with our previous findings from a bunch of our previous audits and some handwritten pattern cards from our security researchers. And hopefully, that makes the task discrete enough and enough like a red team problem that, you know, the the playing field is pretty leveled. Uh, one full review of Lighthouse, it uses about 3,000 review agents. Uh, it takes 5 hours and about 400 bucks to run, which seems like a lot, but then you compare that to a full security audit, and like, maybe it's not so expensive, right? Um, it also uh, that's

like a that's like a base to tip review as well. That's a whole code base review. A whole code a whole code base review. Um, but we can also scan diffs between commits if you want to do like a release a release version scan, or we can do reduced scopes. So, it's a quite a useful tool. Um, we've sent the same findings that we found from Sexton through the same uh, paths that the public submissions come through, so we could get an honest assessment of the accuracy. Um, from the first 25, 14 were real, and two were medium or better. Um, sorry, two were medium. Uh, and three needed trusted input, one was a duplicate. So, by the same yardstick that I measured the previous ones, which gave a one in three positive true positive rate, um, this one's much better. It's only like three in 25 false

positives. So, that saves a lot of time for people reviewing these things. Um, here is a not juicy finding. Sorry. uh, this is one's only a low, but like this is an example of how Sec would find a vulnerability. Um, [snorts] in this one we appear a node can ask a peer for a a block. The block the peer can respond with a block that's in a far future slot. We weren't checking the the slot height before we put it in the cache, so an attacker could continuously send us blocks in the future and we would fill up our cache and and use too much memory. Uh, that was found in batch 910 and that's an example of the kind of, uh, code path analysis that we're doing.

It's a very simple threat zone pattern. It's just like, uh, an entry point to an effect uh, and then we give the AI some guidance about what to look for. So, here don't do transitions before it could fail, right? Um, it's getting to the point where if you're waiting for a bug bounty submissions, you're waiting for a vulnerability disclosures, you're probably too slow. When we scanned our Glimesis and Devnet releases 12 hours later, we had a report of one of those findings. So, 12 hours between release and and, uh, discovery by an external participant, but we had already found it with Sec it, so um, we've also reported vulnerabilities in other consensus clients and we found them same week being tested on Devnet. So, they're real, they're being found.

Um, this is the same tool we use. It's live. It's, uh, we're giving away free scans for open source web3 projects trying to help the community. So, if you want to go to portal.sigma-prime.io, scan that QR code, you can you can have some free scans. >> Okay, thank you. Uh, we will run out of time. So, uh, inspiring, uh, experience with, uh, and, >> [laughter] >> let's, uh, look at the questions. So, uh yeah. >> I mean, there's there's a lot of doomerism, isn't there? Like there's a lot of if you if you listen to the wrong people, you're going to hear some very bleak news, but I don't think the bots and the clankers have completely replaced us yet. We just kind of have to adapt in the same in the same vein and use tools to secure our software in the

same way they're using tools to attack us. So, >> I have a question also, uh, interesting about you mentioned 3,000 agents 5 hours run. >> Yeah. >> You know, uh, by heart like what was the token spent? >> Uh, >> where where we are speaking quite roughly. >> It's about $400 in open weight model tokens. So, it would be in the It's in the billions of of tokens. Yeah, but >> We get another question from the audience. So, how do you ensure the scanning tools test all the loopholes in the code infra config infra, I guess, as different model of AI have different outcomes?

>> Um, I I think you're referring to like a coverage question of how do we ensure we we get complete coverage, and that's why we're using these deterministic code analysis tools first. I think a lot of people are going down the road of just prompting up an agent swarm, and that's a very stochastic approach, and you're going to get different results every time. But, we decompose the code base using deterministic tools to catch all those entry points that we can, and so we're going to get very similar results every time, and we can we can tune it that way. So, yeah, code base tools first, then agents where you need them, not agents everywhere is the is the approach. >> Okay, thank you very much. We made it on time. So, once again, please applause for Josh.

>> [music]