Challenges of Replicating Historical Exploits

Transcript

[Music] All right. Hello. I guess I can start. Hello everyone. My name is Yan Gorsny. I'm a technical lead and co-founder at Circuit. And today I'm going to talk to you about a methodology for replicating historical exploits on EVM compatible blockchains, which is a bit of a mouthful as a title, but essentially it's going to be a way that we want to uh talk about how we can replay hacks. And why do we want to do that? not so that we can pull them off again, but so that we can test various things. And if you know what we do at Zirket, you'll know why I care about this. And if you don't, you'll know in a few slides. So let's get into it. So first I'll talk to you a little bit about about the background, then some related work, and

then we'll go the challenges, what we are going to do, and what's left to do. This is not just work that I did myself. Uh I am not that talented. I have a number of great co-authors. So uh if you uh see any of them, also feel free to ask them about this work. And you may have also seen this work um at Ixie, a workshop at Ixie, which is a software engineering conference. But I wanted to bring it here because we all know that sometimes academics and practitioners don't talk to each other. And I don't think that's right. So here I am giving the talk again. Actually, it's my first time giving the talk, but the topic is uh was given elsewhere. All right. So what do we need? What do we need? And uh why are we doing this? Why do we care?

What's the background? So first, let's say there's a hack on the chain. What can we do about it? The first thing we can do is know about it. The first thing we want to know is there a problem on chain with your DAP with your system you know whatever it is you have to monitor the chain there are solutions for doing this there are a lot um hyperative for scam sniffer they all target different things typically I mean there's overlap but you know scam sniffer maybe deals with fishing more than sort of exploits and hyperative maybe does the opposite maybe that's not true I don't know but there are certainly number of vendors each of them will tell you what they do very well or at least claim to do very well and I've used some of these and they are pretty good um by all means you

know check them out I I think monitoring is very very helpful and valuable, but they only detect a hack going on, right? They they typically um you know use analytics to figure out what's going on to see, you know, asymmetric value changes or something like this. Um and they operate passively to alert people maybe on Twitter or to reach out to their clients or vendors if they have a connection to them, which is very helpful. It's better than nothing by all means. Uh and you can get a lot of information. You can then go investigate and try to do something about it. But if you have the notification that a hack is going on, it doesn't mean you can necessarily fix it. Some of the systems take it a step further and it might include the list in the previous uh slide as well. I'm just spreading out

the logos because I don't work for any of them and I don't owe them anything. Uh but I do, you know, want to, you know, give everyone treatment. Um what you can also do is if you know something's going on, you can try to block the hack, right? So if you see a hack coming into your DAP, you maybe want to frontr run the exploit, right? you want to say, okay, I'll steal my own funds, which is annoying, but I'll steal my own funds because then at least I can give it back later, or maybe I confront want a pause on my smart contract so that no trades are possible, or maybe I do something else. Um, but this is um something you either need to have preconfigured or you need to be really really quick at getting your signatures onto the chain in order to prevent um,

you know, the hack from going through. So, you have a very limited time frame. And even then, if there's a private me or if there's, you know, some way you can skirt this, uh, it's going to be very difficult, but you need to be active. you need to be timely and when you have a decentralized team maybe that's possible maybe you have someone around the world or if it's automated you can do it that's uh a possibility but uh if you are always you know an eastern time zone and the hack occurs at 3:00 in the morning there's a good chance this approach won't work either uh so you know we want to be able to to look at both of these things and first monitor and and um block if we can in the situations where these are effective uh but we need to know how to test these

systems and that will be the latest latest part of the talk so if you're building a system like this how do you be sure you caught what it is. And then there's one other step you can do in order to uh protect network uh sorry protect your applications and that's build on Zirket. So Zirket takes it a step further. Uh and I won't shill Zirket too hard here but essentially um you know Zirket's a layer 2 network or rollup where we put an AI in the in the sequencer so that as trans as transactions come onto the chain we use an AI to detect them as uh malicious or benign. And so actually if there's a hack on our network the goal is to to not include that hack. um it'll be included in quarantine and then you know we'll we'll deal with it and so you

don't need to just be aware that it's existing and you don't need to be panicking that you have to quickly pause your contract or front run the hack. Circuit will take care of that for you. So that's that's the goal and why I'm interested in this research is because I want to make sure that I can convince you and anyone else in the room that Zirk is very effective at what it does, right? I want to be able to say it would catch all the hacks in the past as well as multiple new hacks in the future, right? Like we don't want to just say we can do this and then hacks get through. That's not the goal. And so this work is uh a little bit of a hint as to how we tested our own methodology in order to ensure we don't have hacks on our

network. Okay. Okay. So that's why we care. Um and if you do start looking at these, like I said, you need to know how how you can trust them and you got to figure out um how you can test them. So there's a bunch of, you know, concerns you have. If you have only a monitoring tool, um you know, you have this passive uh detection. So how do you test that? you know, maybe it works, but maybe you don't know it works under high TPS solutions or situations. Um, you know, maybe the monitoring can't keep up. For example, uh, if you're doing front running, it's not clear that you can always do this if you have private members. As I already mentioned, there's a bunch of challenges, right? And so, there's actually a lack of testing methodologies. And I I think this is sort of a false slide in the sense that

the people building these systems almost certainly have these testing methodologies. They're not going out there and making outright, you know, lies about what they're doing. But from you as an external user, you want to be able to figure out what they're doing. And this work is also in that direction. So we wanted to come up with that and start thinking about how to do this. And by no means is it a complete effort. But uh you know we'll talk about that near the end. Okay. So uh what the problem actually comes down to at least with the old hacks is just making sure that at least the old stuff is caught. Right? Security is a lot about growing and learning from the past. And while there are probably new hacks that are very difficult to uh figure out how they will work because no one's done them yet. Uh

we want to at least make sure all the stuff of the past is the stuff of the past. We don't want another re-entry attack anywhere. Surprisingly, the stuff still happens. People still don't learn. They don't take the time. But we absolutely don't want from, you know, the the most uh interesting point of view to to replay the old stuff and have that hack go through because, you know, that's the stuff we know how it should work. We should only be surprised by new stuff. And so what we want to do is we want to take existing hacks, use them as a testing uh data framework and rejig them so that we can replay them anytime we want with whatever system we want on whatever network we want to make sure whatever we're doing to identify or prevent that hack works. And so what we're going to do is we're going to

identify all the contracts involved. We're going to modify them so that uh we can put them on the network we care about and then we want to replicate the system and the uh hack um situation as best we can in order to test our own tooling. So, what are people doing that aren't just those companies I I talked about, and I'm sure there's more, by the way. Um, but yeah, so what can we do? Well, there's a lot of small steps that you need to take in between um, you know, none of this testing and a complete testing framework. So, in particular, you have to identify a whole bunch of issues when you start to look at forking a smart contract that was involved in a hack. You need to figure out where was it executed in what context. You need to figure out how

address was handled. If you're taking a hack from Optimism, which occurred on Optimism, for example, and you want to replay it on a different chain, do they have the same token addresses for the things that were stolen, does it matter if they do? Um, what's the bridging address? Does it matter if that, you know, changes anywhere? Uh, or, you know, what happens if you have crosschain deployments or what's the purpose of the attack? There's a whole bunch of really small stuff. In the interest of time, I'm not going to cover all of these because I'd rather get to, you know, the bigger pain points, but the point is there's a lot. In fact, there's even more. And uh and we have a real problem here because this is a lot of work, right? To replay a hack exactly the way it is and I will talk about this

I think in a couple of slides. It's fairly straightforward. You can always sort of fork the optimism chain at exactly the block the hack occurred on if it occurred on optimism and you can replay exactly that transaction. You can see what would happen. But that is not the point of this work. In fact, we want to be able to take the hack and replay it in different situations in different contexts because that hack exactly as it existed already happened and we don't care about it to some extent right we can't recover those funds or any you know it steps have been taken to overcome it already but what we want to do is if we saw something similar right hackers reuse ideas there some of them are creative others are very lazy if we use similar ideas can we catch it or if the hacker uses similar ideas so you

have to handle all of this so all the stuff on the last slide which I'll go back to right we have you know crosschain stuff purposes address handling all of that uh then what happens if you fork it you know did you fork it correctly and does it matter uh what happens if you can actually replay or things like this there's a whole bunch I won't get into this please you slow down the recording on the thing take a look or read the paper we have this in the paper so we can do that all right so all of that stuff aside I'm going fairly fast I do want time for questions um why is this hard because actually it does sound like we have all the data we have you know the transactions are public unless it's on a private chain It's another story. We

won't talk about that. But uh we have, you know, the the hacker's address, the transactions that were involved. People look at this right away. And we have the the smart contracts that were affected to some extent. Typically, the victims are always out in the clear because uh you know, you have D5 protocols that people studied and want to know how they understand or how they work. And sometimes the hackers only give us the you know, the nonverified version on Ether Scan because they're not trying to get themselves caught. But uh we still have some information. If we can do all of that, where can this go hard go wrong anyway? Sorry. Well, first you have to replicate the exact situation for this hack. So, you have to figure out what changes from the time that the hack occurred in the past that you're aware of to now when you want to do it on your

current chain. And you might not want to deploy it on exactly the same chain. Obviously, I'm from Zirket. I want to deploy all these systems on some version of the local, you know, development of Zirket so that I can see if my tooling will catch this. It does. Uh but you know getting it to a context that is relevant for your tools observation is very important because you might be building this for a very specific client right you might go talk to a specific chain and say I'm really interested in your chain because of x reasons xyz and I have this tool that's built for your chain then you have to also deal with address changes and uh contract um configurations right getting all the same state so that the the hack can be replicated is very very important and it's very hard so let's take a look at

motivating example so this part of some hack I believe maybe it's not it doesn't even need to be it's very simple and even this provides us with several challenges without um knowing anything about what's going on in terms of the actual offensive part of the stack you have a contract and you're just trying to redeploy it in a new context and already we have at least three problems first you've hardcoded various addresses which may not be the same on different chains if you have tokens like web it may exist on one chain at one address and another chain at another so that's unfortunate uh then you also have dependencies Almost all of DeFi has been marketed as money legos. And you know that's for good reason. That composability is really cool and really interesting. But that means you have dependencies you need to also have in your testing

environment. And then you also have things that um go bad when you do things in the uh constructor because maybe this is the hacker's contract and it can only be called by them, right? But you're not the hacker or at least you're probably not the hacker. And so you uh you know need to change all of this. So already we haven't even got to the interesting part of a hack and yet we have challenges to replay this. So this could just as easily be a software engineering talk. Uh okay. So we can tackle all these thankfully. Thankfully these are somewhat easy. We have a constructor that needs to be sent by a specific person. We can change that. If we have addresses that uh need to be hardcoded, we can change that. We just got to go find out that information. That's out of scope of the work. So you have to do the

mapping yourself. But it would be really nice if we could automate this. And I leave this as an open problem near the end. Uh but anything else in terms of constraints, we can get rid of those, right? We can simplify this process. We're trying to replicate the intent of the attack, not necessarily the specific um context. Although that might not be always true, right? Sometimes you are looking at this code for hints or you are using this stuff, but you can get around some of these. Um this will let you at least deploy the context for your hack, which is helpful. Then you got to go figure out all the changes uh on your smart contract addresses. So if you're attacking tokens or you're stealing tokens, they probably don't have the same one. I said this several times. you can change that too. That's not too big

of a problem. Uh which is very very nice. Uh but you also need to be able to change uh this in various ways. So smart contract developers are getting more clever and they're doing things in different ways. Uh so just make sure you catch all the places you do this. And then probably the hardest part which this slide does not do justice is to figure out you know how do the contracts store the data even in um you know various implementations of ERC20 tokens how addresses map and where things go changes. And so you have to be able to replicate this in any contract hack that you're also trying to replay. Again, this slide doesn't do it justice. Please see the paper for the more interesting parts of the challenges, but uh you know, you can also fix this if you're careful. Um and actually the way to do

this is the methodology, which I think I'm getting to right now. Perfect. And so we can use a lot of this information to do it. And I'll I'll go back to that example in just a second, but really what we're going to do is take the hackers contracts and stuff that's, you know, we already saw as the flow. They did a hack. We saw all that information. And what we're going to do is we're going to take all of that, modify the stuff that we need to surgically and then replay the test and make sure our tooling works. And so in this case, we got to identify exactly what's going on. Already went through the example, bunch of stuff that needs to change. Uh, and then we actually need to figure out anytime any of that's loaded into the EVM, for example, and we can start changing that,

right? So, actually, if you're doing it on a local version, if you're not trying to replay the hack on a public blockchain, which you probably wouldn't want to do because that would involve stealing real funds. But if you're doing this on your local version of it, you can go in and modify anytime any of these values load into memory and replace them surgically with the values that you need to make your replay of the hack happen, right? So, if the hacker stole, you know, a million of a certain token, maybe you only have a mock contract that has 10. I don't know if you're lazy like that. And, uh, you can change that anytime there's an S load. So that's basically where all the magic happens which I don't have time to talk about in 20 minutes let alone if I want questions but getting into these s loads

and figuring out um where things are being changed is very important and then you can also use proxies to get a little bit easier time with this. So what you can do is uh redeploy all of your hack contracts behind proxy so that you can sort of map token addresses a little bit more easily and and address reads. won't get into too much of it but essentially you know you can then upgrade and replay this and test as much as you want and then finally you configure all the storage as appropriate if uh you don't want to do it in EVM you can do it right on smart contract level and you redeploy the contract as appropriate there so you just be surgical you only set the context state that you need then you execute your replay and that's basically it so that was very quick I'm sorry I

I'm pretty much on time though so uh I will wrap up with some future work and um you Uh this is seemingly easy. What I said was just change the state to replay stuff. And that's a very very easy blanket statement. But as always, the devil's in the details. And uh in this case, it's a several page paper. So you have to go take a look if you want to learn more or come talk to me. I do have time for questions right now. Um but uh it's not done. We're going to keep working on this. We want to do some of this, you know, in in the open. And we also want to share what we've been doing and uh validate this approach. Um so even you know despite it being somewhat easy air quotes um it it's not perfect some things are very difficult to um

replay or replicate. So in particular stuff that happens in real time or based on time values in your blocks a little bit harder to do some of the surgical stuff. I imagine it's possible with a significant engineering lift but uh you know it is going to be some work and it's going to be difficult to replicate situations where you had a hack succeed or fail because of something like network congestion. If you had a lot of transactions and somehow the exploit uses that uh I don't know exactly how um that might be very challenging. It's not just about surgical state changes. It's about replicating all of that as well.

And the methodology doesn't do this. Nonetheless, we started doing this by looking at the damn vulnerable DeFi um hacks repository and starting to see if you know can we at least do the simple part of doing this state change um to replicate the hacks and detect them. We did detect them. it does work and we were able to validate this approach which means if you are interested in testing your tooling on hacks you know starting there and using this code which I I don't know if it's open source so come come ask me about that um is is a valid approach and one that you can actually uh succeed with ideally what we'd really love to do is replicate this in an automatic fashion so if we see a hack occur build some tooling around changing the surgical state to identify you know which parts of the state need

to change to replay it um that is going to be ongoing work and we're going to be happy to talk about that in the future. But um at least for now, we can manually go back and say at least on Zirket and with the tools that we tested, the hacks that we saw in the past won't be hacks again on our network or anyone using those tools to prevent um those systems, which I think is a good step forward. So thank you and I will I think have time for questions. Yeah, if you have a question, please raise your hand. Okay, I see. Thank you. Um I'm curious um for security practices for engineering teams um what external tools would you

recommend like slither orin for them to during their development life cycle kind of be checking proactively to prevent potential okay I would love to give you a really really good answer and it's been a while since I have personally used the smart contract code tooling slither was always one of the ones that worked really really well uh and I don't have any other recommendations at this Uh, a lot of people will tell you to use an LLM. I will say be very cautious. I think there is a good place to start or they're good tools to augment, but I don't have full confidence in just shoving it into like chat GPT. So, I will just give you that anti-recommendation and say come find me or I can recommend you to auditors who can definitely tell you better tooling.

Thank you. Thank you for the question. Any more questions? Yeah. Uh so you're you're changing a lot of uh the you're not replicating the the exact construction process of the contract obviously for this. Do you see a lot of uh hacks that occur uh like during constructor code uh because this would imply that you know that wouldn't work for this. So I'm just wondering yeah I I I I think uh you know you can also be more surgical on what you comment out right. So I think of the motivating example, we just killed the whole constructor there. But if it's just the requirement that the sender needs to be the attacker, you can change that and be more surgical. Um ultimately those types

of requirements also need to change not in the constructor as well, right? If if the hacker calls a function, which also needs to be gated by their address, that also needs to change. So it's not just a question of, you know, is the hack going to occur in a constructor. It's going to be, you know, can we surgically do these these changes to the attacker's contract that lets us replay it from our point of view. And so I I I think it's uh like yes, we can because you could as an attacker probably also move most stuff to the contract or to the constructor, right? So it's sort of like a a meaningless distinction of if the hack occurs in the constructor versus not except maybe one extra level of transaction call from the do you see that uh uh uh uh most of the hacks that

you see are are they uh not in the constructor code? I I would so yeah then that's a that's a good the perfect followup and I don't have the answer for that. I would have to ask my co-author who who took a look at it. Um, you know, there's actually a good chance a lot of it will be in the constructor because then they deploy and execute at the same time. And so if a system had static analysis tooling to detect old contracts deployed, you know, you wouldn't be able to catch something like that. And where can we find the link to the paper? Uh, that's a good question. I come I will tweet it after and I I will yeah share it. Thank you for the question. I should have put that on the slide. Yeah. Thank you for this to the speaker and thank

you. [Applause]