Providing Ground Truth for LLM-Based Bug Detection Tools Using Slither MCP

Transcript

and about two dozen more tools that uh I didn't even want to list here. Uh if you're interested in seeing what we're up to, there's a QR code up there that goes to like some of our recent blogs, uh some of our tool releases. Uh it'll also show up at the end. So, first let's talk a little bit about Slither because this is like our oldest tool. It's the one that everyone knows about. Um it was built back in 2018 by Joselyn right there in the center. uh because we were sick of looking for re-entrancey issues. Uh re-entrancey issues were really really really common uh back then and it it it just sucks to have to look for them in an in an audit engagement and so Slither was created to

address this issue. Uh but Slither is also in a way a victim of its own success because over time as developers started to run this tool more and more uh there were less and less re-entry issues in solidity code bases in general. And there's other things that Slither does like it has other detectors but it's like if everyone has access to this tool that is really good at finding uh certain bugs uh those bugs are going to go away. So its usefulness as a static analysis tool over time should technically go down as the number of bugs go down. But this also means that here we are in like 2025 and if you run Slither as part of an audit, it's very unlikely that one of the bug detectors

is going to give you a finding that's actually a legitimate bug. Uh the value proposition over time has kind of shifted into the printers. If you've ever used a slither printer, it's a way of taking a codebase and printing uh certain information about it, like all the public functions, um what might be deployed, uh which storage slots are tainted by what, all kinds of stuff that's really useful in audits. Um but even then, it's like it's it's not that useful anymore. And the thing is about static analysis tools is they're very expensive to maintain. Solidity is changing all the time. Uh the you know, it's a very large tool. It's got a lot of croft and it's like what is the future of slither because right now uh auditors are not necessarily auditors

and developers don't necessarily get the level of uh usefulness that we would hope that they would get out of the tool. And fortunately things have kind of changed in the past year. Uh as you've seen from like all the talks everyone is all about LLMbased bug detection. There's a bunch of logos here that I literally off the top of my head, all these people are working on LLM based bug detection. And even if you aren't working with LLM based up bug detection, you're probably working with like cursor, you're probably working with cloud code, and you're probably operating using these tools to operate on some kind of slitty codebase. And that's all well and good, but these tools aren't perfect. Uh they have problems. Uh they aren't 100% correct.

They're only probabilistically correct. And so take a simple problem like what happens when you ask an LLM to analyze a codebase. So you're just giving it this solidity codebase. You're like tell me where the bugs are in this codebase. A very open-ended task. Well, the tools that the LLM has available to it are actually extremely limited. You get, you know, GP read file ls and in cursor they have like a a rag based search, but really it's just a it's just another way to search for things. So now imagine you're in the position of the LLM and these are the only tools that you have to analyze uh the codebase. So you have to ls and then read each individual file to try and build up this context and

understand what this codebase is doing. Uh it would suck. It would it would you'd have to make so many tool calls. You would never necessarily be sure if you have all the information that you need. You'd fill up your context very quickly. you'd have to rely on uh sub agents to like summarize certain parts of the codebase. It wouldn't be a very good time. And so here's a a good example of how LLM reliance on these tools uh can cause problems. So let's start with like a very simple project. We have an ERC20 like you know the basic ERC20 contract.

We have my ERC20 which is my ERC20 with something added to it. And there's a DAP file or a DAP contract that interacts with this ERC20. And the prompt is I want you the LLM to detect bugs in DAP.soul. And so as a part of this process, uh, the LLM has to figure out where is this function, this my ERC20.transfer actually implemented because it has to be able to audit this and take a look and figure out what's it actually doing. The first option out of the tools it could use is to simply use grap and like search for function transfer and uh you know try and find the implementation that way. But this is a flawed approach because uh that transfer isn't

necessarily the transfer that's being called here. It might misunderstand the inheritance hierarchy that's going on here and start analyzing the wrong transfer. The other option that it should probably do is follow the import path. So it sees, oh, it inherits from my ERC or it imports my ERC20. Let's go up to that. Let's read file. This one, uh, well, it's not implemented here. Let's go into the next one. And it goes all the way up into the top. And the problem with this approach is, uh, resolving paths like this can have very high error rates. Uh, but more likely than not, uh, it's extremely token hungry. So unless you're spinning off sub agents to do this, uh, it's it's not going to be a very good time.

So the the TLDDR problem here is that if you ask an LLM to do these like basic code navigation tasks, uh it it can't really do it with very high reliability and often it'll take completely different approaches. So like sometimes it'll do option one, sometimes it does option two. Each one of those options has a different probability of succeeding and it it's just a huge mess. It's not very good for reliability. And so this is where Slither can come in. Um, so this is a little TLDDR of how Slither works. So it takes a Solidity codebase, translates it into something called Slith IR, which is an intermediate format for Slither. Uh, and it can it's much easier to run static analysis on it. And then it uh does a couple of things. It turns the Slith IR

into CFG form, SSA form, and it gives you the ability to do like this data flow and taint analysis stuff, and stuffs all this information into uh something called the slither analysis object. And all that is like completely self-contained. So when you want to write a detector or a printer, all you have to do is interact with that slither analysis object. And so the nice thing about this is it's very easy to extend and add new things to it. And uh luckily for us, we can just stick an MCP server right in there and start uh exposing some of this uh static analysis engine for LMS to operate on.

So uh let's talk about an example problem that an LLM might have to do. So the problem is an LLM needs to find every code location that calls this iO.pric. Uh and so without slither MCP think about those tools that the LM has available to it. The most likely to succeed path involves doing a GP uh to identify all of the calls to price. Then calling read file on each one of those. Making sure that the par the arguments passed to price match address. uh then calling read file for each of the contracts and trying to see if contract implements iO this is the right way that an LLM should

do it but it's also like you know extremely token hungry um whoops spoiler alert uh it's it's extremely token hungry uh it's not reliable and ask yourself do you really trust the LLM to do this with 100% probability of success every single time and it just doesn't. But with Slitherm MCP, all the LLM has to do is call get function callers, give the signature, give the contract name, give the path, and that will give the result with 100% certainty because it's it's something from static analysis.

You're not relying on the LLM, this probabilistic process to produce the result. You have the actual thing. You know what it is. Here's another really good example. Um so the LM needs to understand what are all the functions that a contract has what when you deploy the contract what functions is it actually going to implement so without slither MCP it would have to call read file to read the contract then it would have to identify all of the inherited contracts then it'd have to call read file on all the inherited contracts and it has to recursively do that because you know this inheritance hierarchy is going to go especially if you're using like open zeppelin or thing. Uh, and then you have function overrides because there might be a

function defined way up here. It gets overridden here and then it also gets overridden here. And now the LLM has to figure out which one of these functions is the actual function that's going to end up in the deployed code. And so, yeah, you could have a sub agent do this. You could you you could have an LLM do this, but do you really expect it to do it with 100% reliability? Not really. And it'd also be expensive. But with slither MCP, you can just call list functions. You give it the contract name, you give it a path to the file, and uh that's it. It it gives you exactly what you need.

So there's a couple ways that you could use slither MCP. Uh if you want to use it in cloud code or cursor, you can just add it as an MCP server. Uh so like there's a way to add it with, you know, cloud MCP add. There's a way to do it with cursor. um cursor has like this weird thing where it sometimes can't resolve the uvx path. So you might have to run that sudo sudo command to make it work. Um but I think the really good use cases are those tools that people are building the bug detection tools because a lot of times those bug detection tools need a way to identify like uh give me the source code of this contract dot this function. uh give me the source code of this uh tell me the

function callers, tell me the function callies, give me a data flow analysis of this to that. Uh so you can also use this instead of just as an MCP you can take the individual tools and expose them to your agents uh using like a create get function source tool all kinds of things. This is like paidantic first. So if you're using pideantic it should be really easy to set up. And finally, you could also use it just as a simple API client to interact with slither because interacting with that slither analysis object is uh really challenging. Um if if if you haven't done it before, it it has a very steep learning curve. But this API that's relatively simple like you know the get

function source uh get callers, get callies, very simple API. So if you just need to use this as an API client, you can do that as well. And that basically wraps it up. Um, so that that QR code takes you to the Trailbits website where you can see how to like you can go to the GitHub for Slyther MCP can see some blog posts. But uh, other than that, does anyone have any questions? >> Hi, good job on the presentation. Um, I'm curious to hear your thoughts on one of my hesitations whenever I'm talking about AI tools or development is that the progress of the language models

themselves, the progress of like Anthropic or OpenAI in releasing new models um can often outpace by orders of magnitude, the progress of building tools on top of these systems. So like if you were to build this sort of thing for something like GPD3 and then GPD4 comes out um the progress difference from GPD3 with this versus GP3 without this uh comparing to GP3 and GP4 the latter will win every time because there's just so much more compute. So I'm curious about your thoughts on I guess how can you balance building these new tools that work well for the existing models while also trying to make sure that as these new models come out these new models are still going to be able to leverage these tools because that's where ultimately the biggest performance difference is going to come

from at least historically. >> That's a really good question. Um, so I'm kind of bearish on big models getting bigger and being better at solving some of these problems. Yes, the tokens will get cheaper and maybe it'll make all of this not matter. But the big problem right now is getting LLMs to even think in this idea of I need to understand all the functions a contract has because like right now if you plug this into like your uh your cloud code or cursor for the most part it doesn't actually use these functions. And that's something I'm trying to figure out why why will it not use these functions that we expose to it to navigate around this codebase

even though like we know that it has a higher probability of succeeding. We've we've indexed it or excuse me we benchmark it and we know has a higher probability succeeding. We literally say in the tool description use this tool for this task. You must use this tool for this task. It still doesn't do it. Um, and I think the reason why that is happening is because LLMs overall are really really leaning into how they are trained. So like when it comes to like cloud code and stuff like they're they're really leaning into read file, they're really leaning into GP even if those tools >> aren't the right solution to the problem and just making the model bigger. I I I don't think that it's going to work out.

I think the way that uh the whole industry is going to end up going is much smaller models with very specific like you asked earlier about models for specific tasks. I think that's the direction everyone's going to go into. >> So followup question if that's okay. Um I guess quick follow-up question. Do the models tend to use because there is definitely a complexity difference between without slither versus with slither in the number of steps that are required. So do the models tend to use the with slitter stuff where it's simpler more frequently than without slitter and like are they more successful in using your tools? >> Uh so when when we take away read file we take away GP and we only give it slither MCP it completes the task with a higher success rate than if it has slither MCP and read file GP.

>> Okay cool. >> And no matter what we do to the uh tool prompt that's not changing. Now, we could change the system prompt and say, "Hey, you should use uh the best tool for like the most specific tool for whatever task you're trying to do." And sometimes that'll work, but uh >> that's highly variable based on the model. >> It's very variable. Yeah. And also like you don't want to do have to do that like for an MCP server. It doesn't make sense. >> Okay. And then proper follow-up question. You were talking about smaller tools and stuff. Um in I don't think this exists much in the web three world, but in the web two world, sorry, I'm bad at holding this in my mouth. uh in the web 2 world there's a lot of like very very much smaller models so like using

BERT style or using um old not LLM but LM language model style training um do you think that if the LLMs are going to slow down in their progress that it's worthwhile looking back at these older styles of LLMs or like fine-tuning these smaller models for those very specific tasks to be able to properly leverage these tools or to be able to properly understand the context of solidity where the larger models might not have that level of fine-tuning associated ated with them. >> Sorry. What are the two options? So, the first is fine-tuning larger models. >> Well, I'm just curious what your thoughts are on fine-tuning because it's hard to fine-tune larger models just cuz they're massive. um but fine-tuning these smaller models like GPTO OSS um or potentially going even further back and

looking at models like BERT or Distilbert which are uh if you don't know there these tiny um >> uh encoder decoder models which are used a lot in the web two world and can be used for like >> token analysis and like finding vulnerabilities by literally just iterating over the tokens in a similarish skill set to what Slither can do um because it doesn't have the wider context. >> That's a good question. Um I so fine-tuning is a big lift because you need data to fine-tune like let's say we wanted to take uh get function callers and instead of having it implemented with slither MCP we want to fine-tune one of these like smaller

models to do it we have to that's an expensive thing to ask for and we have to be very sure that we know that we actually need this function because like so for slitherp I implemented a bunch of things I think that that LLMs are going to need. But some of these functions, they might not end up using them at all. And if we were to spend the time to like try to fine-tune a model to use a specific tool, we might like end up in a local minima, you know, if that makes sense. >> Well, not even fine-tuning to use a specific tool, but also trailits, I imagine, is going to have a whole bunch of audits at your disposal that are just this is the stuff that we've done in the past. This is the human benchmark. And so training the LLMs, the smaller LLMs

to follow in that same pattern in a way that the larger ones might not be capable of. I'm curious like >> have you looked into using your past audits as training data for the future LLMs or anything like that? >> Like in so the data question I think there's plenty of data out there like Yeah exactly. >> There's there what was the they've got a website with a bunch of solidity audits. Solidit. Yeah, solid. They just like released a way for you to download everything. So, like >> there's a couple of them on >> ke. There's plenty of data out there. Um, but again, it's like where do you best choose to use your time? Like it might be easy to get the data to like fine-tune, but like again like if I don't know that this is going to be useful yet, do I really want

to spend the time figuring out how to fine-tune it? Also, I don't know if the fine-tuning is also going to be the way that it works out that like we might end up just using GPT4 mini or GPT5 mini for back basically everything. That's what we do internally. like we don't really lean on the big models very much. >> Thank you. >> Yep. >> Cool. Anything else? >> Okay. If not, let's thanks Benjamin again. [applause] Okay. This conclude the AI session.

Thank you all for call coming. I think the next session will start in like 10 minutes or so.