AI Won’t Replace Auditors, But It Will Replace the Lazy Ones

Transcript

Hey guys, apologies. My slides are very last minute and I think they're really just if you want to glaze over. Um, but my talk today is on how AI won't replace the audit industry at all. But we'll do what will do is it'll clean it up. Uh, so what does that mean? >> Uh, AI does not replace auditors. In fact, it actually helps the best auditors. Um, okay. Sorry. Down there. Um, okay. So when we started Olympics in 2022, about 90% of exploited contracts were audited. That's still the case. How does that happen?

>> Well, >> the top five, 20, 30% of auditors actually um don't get their hands on all the audits. And even when they do, what what falls short is that um they spend a lot a lot of time on things that teams internally can do. So um when we talk about AI kind of exposing the bad auditors, it's hard to see it right now. When we will see it is in this next wave once teams begin integrating AI and tooling correctly and we'll be able to trim the fat. Um okay, so uh what else?

>> Yeah, I think that's that's pretty clear. Um, okay. So, the industry problem today is is like I said, uh, a lot of auditors are relying on kind of a template or how do I do, you know, see a see a vulnerability that I've seen before as opposed to having the time to really think about what is this attack surface? Um, what is the intent and intended logic of this protocol and how can I exploit it? And that's something unique to a human. Now what ends up happening is they don't even get the time to dig into here because they're time crunched and um they're really spent just looking for patterns um taking existing vulnerability classes and and just trying to find them within the the audit which leads to again we're

coming back to the same problem. So it's really like chicken and egg of having to go and get a second third audit uh for these protocols. So it's really not working right now. Obviously the numbers speak for themselves with the amount of money exploited and the only solution moving forward is to integrate not just AI tooling but really true deterministic tooling and it can be leveraged by AI but within the CI/CD pipeline or ahead of the audit um or even let's say if there are multiple audits auditors um given to your report to have one of them focused on tooling results um and I personally on a biased view think the best way to approach this problem is to have internally within a team utilizing tooling and that tooling can be

leveraged with AI. Um or it can just be pure deterministic tooling. Um so uh what does AI do really well? Like I said deterministic. Um so anything like static if X is true um tell me why does that fabulously. Um what else does it do really well? Exhaustable. So again I'm not talking about like LLM rappers. I'm talking about when we're using agentic systems given really quality information. So maybe information from tools and there are things that LM rappers do well right like pattern driven if we see a logic vulnerability um or not a logic sorry sorry a a vulnerability that we can see on um multiple contracts and maybe each contract looks a little different and deterministically we can't solve that but from a pattern perspective an AI LM

rapper would do fabulous at that um but an AI agent would do really well taking results from deterministic systems and actually orchestrating and finding where the vulnerabilities can take place So there's a lot of things that AI does really well and also like exhaustive for example like if we use AI to find every path we can um f path we can fuzz or um exhaustive fuzzing um exhaustive uh exploration of of code paths. That's that's great. That's also not really feasible for a human to do. So, um, we're kind of splitting these two things, which is like what can a human do really well and what can a machine do really well as opposed to what can a human or machine do the same. Um, and I think when you begin to split it that

way, you begin to realize that the future of security in this space really looks like, okay, we have tooling that does all this exhaustive, annoying work, and maybe agentic systems use this tooling to produce really, um, easily digestible results, but also um, we have the human using their time productively um, and then that comes to like what AI can't touch. So oftent times I talk to auditors and we work so closely with so many incredible auditors and they say they spend way too much time trying to understand like what is the logic of this of this um what is the logic of this protocol and then how can I break it and when their time is spent on things that again going back a machine can do much better or can do much more

productively I'll just say it that way like mutation testing for example like no one wants to do that exhaustively on their own but if they can just leverage a machine great but when an otter actually has that time to understand the protocol logic understand misinccentive uh mislined incentive structures um economic attacks have that conversation and dialogue also with the team and protocol architecting it um they can do what a what a machine can never can never do um so I think um that kind of brings us just to the last slide which I'm sorry if I'm all over the place I'm not really uh a slides girl this is just something that I truly believe so I can kind of go on a a long a long string tangent on it but essentially the auditor of the future um what it really looks like is it means

internal teams audit themselves first with tooling with exhaustive measures and again this isn't just AI tooling but perhaps it's deterministic tooling where agentic systems help you or perhaps it's LLM rappers in some capacity if you can get to um a really good outcome with uh with false positive rate which is a whole another problem but it does not look like an auditor having all the load on their shoulder which is probably like the biggest shortcoming of today which is we don't set our best auditors up for success and we set our worst auditors up to find things even um if they just use tooling because you don't do it yourself and then you're not able to actually figure out who is the good auditor and um who is you know just medium of the pack which is really bad for the protocol for um every person who is

putting money into what you've worked so hard to build. Um, and it's it's bad for your company because again this the state of your security is the state of your company. And so you really have to think about who you internally own with tooling um the security posture.