Auditing with Machines: A Practical Exploration of AI-Augmented Security Work

Transcript

Hello. Okay. Hello everyone. I'm Yonus Vorel Gangu. I'm a blockchain security researcher at Open Zeppelin. I'm going to give like just a short very introductory talk about how to use AI to speed up your audit process, improve it quality, and make sure you have just a bit more time to focus on the complex flows that the AI can't handle. So, uh AI has become quite capable. It has this internal multi-step throcess that can split like a complex task into sub problems. It applies transformations on them. It retrieves relevant memory from the conversation you have with the AI and then sometimes it also does error checking between the process that it made and what was expected by you. It also has a bigger context window up to 400,000 tokens. Just a simple analogy

for the context window. This is the amount of information that you can flush through the AI in one go. The AI is like this big uh mathematical function with a lot of parameters. So the more relevant information that you can fit within uh those 400,000 tokens, the better it's going to perform. And then there are also all these powerful rapper IDs such as winds surf or cursor that already ingest part of the codebase as much as it fits and then add some uristics whenever you interact with AI such as better web searching, summarizing information, uh aggregating it and like efficiently spending tokens.

So this creates an opportunity to use it uh in your audit process just in very very simple ways that anyone can do just to understand the codebase better and we'll see a couple more examples. So here I have just a simple example. I downloaded the UNISOV v4 codebase and I asked a simple question. Can you just explain this codebase to me high level and give me the key components and in 30 seconds it was able to tell me that it has a singleton design. It has the pool manager call contract. It has five key operations. There's this hook system inside that executes custom logic or all these life cycle points and then it told me the key libraries and supporting contracts. So with just more simple questions, you can dive deeper into each each of these components and even go

through a flow like from user interaction through all the contracts. Uh second really simple way is like just understanding complicated code that um for example like this big block of assembly. This is again from unis v4 core. Most significant bit function is like giving you the index of the most significant bit within a uint 256. So at first it just looks a bit scary but the can easily tell you like this is just like a binary search in assembly. It has like five checks because 32 is two to the power of five and then the assembly like you see the or the bite and end is just a way to make that more gas

efficient. It would give you like a list of steps and this is just like a faster way to understand. Of course, you need to have like the knowledge to make sure this is right and you'd also have to check it again the against the op code just to make sure all the parameters are right and everything but it's just a lot faster. uh then you can also use it to identify simple vulnerability patterns and improvements. I've seen it uh and use it to like identify incorrect implementation of a documented process. So think of a cryptographic function for which you have like a text describing the steps. The is pretty good at that.

It looks at the implementation. It looks at the steps and is able to tell you you've missed something. A configuration parameter is wrong or a step was incorrectly computed. I've seen uh cases where it was able to tell you like a cross-chain message. You send it from source chain to destination chain and then on the destination chain there's nothing tying the message sender there to the message sender on the source chain. This again a pretty simple example and then things like incorrect order of parameters and gas improvements. Note that it doesn't work well on like complex issues that require more understanding of the codebase. you have to probably build something on top like either fine-tune the model but it's unlikely that you're going to fine-tune it better than open AI or try to

specialize it again through the context window. So uh something that was mentioned a lot was the primer documents where you specify like just all the vulnerabilities discovered in ERC 4626 volt so the AI is aware of them whenever it gives you answers. Then uh one other simple example to write simple tests and proof of concepts. You have a smell in the code. You want to check it quickly. So here again on the unis swap v4 codebase I just asked write me a simple test that creates a pool as liquidity and does a swap. It wrote me this in 30 seconds and then it was able it was even able to pick up that the unis swap codebase has all these helpers for deployment and for tests. So the deploy fresh manager and

routers is custom code and so on. and it made me like a very simple thing exactly like uh very similar to the UNISOP codebase. So just like some important last notes the AI speeds up your process a lot but you should have the knowledge to validate all the answers and ensure correctness you're ultimately the decision maker here then the AI is a bit like pretty dumb in some way in some ways is pretty prone to bias from the auditor. So if you ask it like uh are you sure 2 plus 2 is not three? You might actually convince it if you insist a bit. And then um try to keep it simple. Don't focus so much on what is the question you're asking but rather

what you're putting in this context window if it's relevant and try to fit it there. So try to either compartmentalize the codebase into different components and then strip it of less relevant files such as tests, helper libraries and view functionality. And then there is also another part just be safe whenever you use the AI but this is not a presentation about that but things like prompt injection and this kind of stuff whenever you put a codebase inside is going to act uh based on what it reads in that codebase but uh the red guild had a good presentation about it yesterday. Uh yeah this is everything I had. Thank you very much.

>> Awesome. Thanks very much for