Dealing With a Hundred Million Dollar Live Vulnerability

Transcript

Hello everyone. >> All right. Uh first of all, thanks DSS for having me and thanks everyone for showing up. Um well, this is the title of my presentation. As you can see, it's a bit patchy. Uh and that's because of the recent events that have happened at Balancer. Uh this talk was initially about uh a previous exploit that we had uh on 2023, but I've adapted it uh to yeah also address what happened very recently. So first of all like uh you might be as you might be wondering like what is this guy even going to say after this huge exploit. Uh so yeah first of all I I need to say that we are very sorry about

what happened and we are doing a lot of effort to get uh back on our feet. Uh and besides that I'm showing up here because I do feel that I have to say something that you might find valuable uh and it could help your protocol to respond to these kinds of issues. So what we will cover first uh I will stick to my original uh script about the B2 boosted pool exploit in 2023 uh and then I will also address what happened uh very recently. So after this talk the key takeaways that I want to take you for you to take home is first why you should have a back bounty program. Uh these two exploits show very different uh scenarios where

you in the first one we could react because we did have uh a white hat report and the second one we didn't and you will see that the stark differences between the two of them. The second one is to reflect a bit the trade-offs uh about the trade-offs in smart contract design uh and what happens when your contracts are completely mutable when you have or you don't have posting functions uh and what are the trade-offs between the two of them. Um, yeah, we will dig a bit more into that uh along the talk, but yeah, there there's essentially a trade-off that uh I want to discuss with you. And then finally, the the main question here is how to improve resilience moving forward in each of these cases.

So first of all, like a very quick brief of how balance 2 works. Uh essentially in balance 2 we had like a vault pool architecture where the vault holds all the funds and performs like very basic functions as transferring the funds uh protocol fees and so on and so forth. And then you have the pools which implement pricing logic. Uh for the pricing logic we you had uh like different pool types. So you could have like weighted pool for uh uncorrelated assets or stable pool for correlated correlated assets and so on and so forth. uh and this is essentially the the architecture that yeah was present in B2 also V3 uh so it's a pretty common pattern also these days so what did the B2 stack look like uh

the core contracts were all immutable and non-custodial uh so both the bolt and the pools were uh immutable there was no upgrade mechanism uh post mechanism was not available by design in B2 we didn't uh I mean we we included a post window that was very short uh on purpose because uh maybe as as you also value like the the properties of of contracts being immutable uh and reduce governance and that's the design trade-off that we chose back then. Um and then well we also of course had audited both the vaults and the pools and a back bounty program with up to $1 million uh per critical report. So this is what the B2 stack look like. Um and

we will uh take a look a bit of what happened next. So the first exploit was more about linear pools and booster pools. Essentially linear pools were pool where you could trade tokens one to one uh and where you could after a rate is applied you could trade between underlying tokens and wrap tokens for example uh representing a position in a lending protocol. So, I don't want to get into the all of this complexity that much, but essentially what this pool allowed was for LPS to get uh fees from uh traders, but also yield fees from their deposits being uh in a lending protocol such as A.

U. So, what happened here? One Friday night, I got an MFI report. The there's the the s name of the of the reporter, God Shannon. uh and that was be unified. It was a Friday afternoon. So it was first marked as a high severity issue but the report was not very clear. Essentially it has some some steps to reproduce what was going on but it wasn't clear like how big the impact was whether this was really like an artifact of how the P was written or if it was a real issue. Uh so essentially we couldn't confirm and at the very first second whether things were really wrong or not.

So one thing that I mean gives you is uh not only a sony name but also like the ranking of the of the white hat and in this case it was a top 100 one white card and it was also one that had already reported a valid high severity issue in in for balancer. So we knew that this was serious. This is a very important hint that uh immunifi reports give you and essentially first tip is that if you get an immunifi report from a top 100 yhat you should address it immediately because it's probably very serious. So since we couldn't uh confirm the the validity of the issue I asked for for more for more information and I kept investigating on my side. A few hours back, a few hours later, the

Whitehead came back with a new different proof of concept that showed very clearly that you could extract, let's say, a million dollars from one of the pools in one single transaction. So, that was already pretty bad. And here I want to do a stop because this part is very important. Like, if you are an Munifi security researcher or if you are handling these kinds of reports, this should be enough proof for you to like fire all your alarms and start dealing with it immediately. like you don't need uh a proof of concept that drains all your pools in one transaction or or doesn't need to be like super efficient.

Uh but what is important is that if you get something like this where you can clearly see steps that drain at least a pool partially then that's very bad because essentially then the attack can be repeated several times and that means that all the TVL that is deposited in those kinds of pools is potentially at risk. So then the next step is actually assessing okay how bad is is this is this really uh what kind of PS are affected and what's the TVL we started that like right away uh it was late so yeah we started essentially at first first hour next day uh and we started asking for more information to the data team and to the rest of the

team and here one important one very important part is to keep everything like strictly need to know uh not all the rest of the team needs to know the exact states of the exploit. They just need to know okay there's something wrong. Uh and maybe you can give them direct directives such as okay I I need to know the TVL deposited in these kinds of pools um and things like that but without giving any extra information. You should only involve people uh who are really technical at first and once you get the understanding of what's going on and what kinds of contracts are affected then you can ask information from the rest of the team and again always keeping everything very strictly need to know after a data pass we confirmed that $240 million were was a

TVL at risk that was deposited back then in those pools we didn't have any uh mitigation mechanism available because we didn't have the pause window the contracts were immutable. Uh in some cases you could change the swap fee but that wasn't effective to pro to prevent this this problem. So essentially we were very short of options and here's where I wanted to reflect about the trade-off that we chose. I mean yeah immutable contracts are great like yeah if the odds go well and everything goes well it's fine like you don't have like upgra upgradability issues where you might introduce an error uh after it's deployed. But then if you end up in this situation, you're left up with very few options.

Uh after involving like a bit of more team members, we realized that actually uh one of the tokens that uh was used throughout many of the high value pools was actually upgradeable. So our contracts were not upgradable, but some of the tokens that were deposited in those pools were upgradable. And in fact those represented like 80% of the TVL at risk which was huge. So what could we do then? We had to upgrade those tokens but we didn't have access to them. Um so another important thing and I want to leave for here is that we got this report on like August 11 but we didn't need to like announce that immediately because the bug had been live for two years. So like the

chances of it being exploited the next day. uh I mean assuming that the issue was not not leaked were relatively small. So we had time to react uh and try to mitigate it as much as possible. So in this case we had to get some help in particular for from AB that they very kindly helped us get uh essentially a vote passed to be able to upgrade the tokens. And essentially what we needed to do was just break one of the functions that was in the swap path and that essentially prevented the exploit completely.

We took like uh about a week in doing all of that because yeah I needed to pass a vote. We needed to get uh all the all the permissions to do that and there were some time locks in the middle but in the end it worked. We could do all the upgrade without drawing too much attention uh from bad actors. Uh and yeah we essentially that meant that the moment we upgraded the contracts 80% of the original DL at risk was now safe. people couldn't swap through the pools, but that didn't matter because the the point here was okay, all that value is now safe and cannot be exploited.

Uh after we did the upgrade, we went through for the public announcement. Uh we had already saved like 80% of the value. So now it since we couldn't post the remaining pools or we couldn't upgrade the remaining tokens, it was time to tell the people to essentially get out. Uh here another important point is that you should give as the least amount of information possible like in this case it was okay there's this list of polls get out there's a problem and nothing else uh and essentially try to amplify that message as much as possible try to reach out partners that are known LPS and try to expedite that exit process as much as possible uh so that was a Tuesday uh we were pretty happy with the results I will

show some numbers uh in the next slide But the problem was that on Sunday the remaining value was exploited. In this case they didn't use the same exploit that was reported by uh via the white hat in Muny but another attack vector that was had some similarities but was slightly different. So we had to understand it afterwards. So even though this might look like a loss uh well of course then we did the deprecation of these types of pools the postmortem etc. So while this might look like a loss uh in summary like we had 240 million TVL at risk 80% was mitigated the moment we upgraded the contracts there were the people withdrew

very fast after a disclosure particularly because we reached out bigpiece uh directly and 5 days after a disclosure black has still figured it out even they had like very little information about the exploit itself if you want to see it like in in charts like this was the total DVL of the pool of the vault uh at the moment of of when we got the report. The yellow part represents the value at risk and the orange part represents the value at risk after the migration. And from that part all the green part was uh withdrawn in the first few days and only the red bit was the exploited part. So we can say

like even though some funds were exploited like the damage that was done was much smaller than what could have been. So what can we learn from this? Um so if you have an immunifi back bounty platform or any back bounty platform in general and you get reports through there even if they are critical and your li your options are limited you can always try to find something to do uh and that will help you a lot you that will give you time to react time to prepare and that is very valuable in this case the rounding error had been live for about two years we didn't have a standard response because the contracts were immutable But thinking outside the box, we could find something

that was worth doing and in fact it prevented like a lot of damage. Another thing is that it's very important to have partners uh and auditors at arms reach like essentially we should uh be able to reach out to big LPS as soon as possible. Also reach out to auditors so that they can jump in and verify that what you are doing makes sense. In our case, we had auditors uh review the appric that we were doing and they validated that what we were doing made sense. And another very important thing is that black hats will figure this out. Like if you just say there's a problem in these kinds of pools, even if you don't mention the type of pool explicitly, they will just figure it out from the pool list. Like if they draw your attention there. Yeah. Yeah. Take it for

granted that those will be exploited. So some tips from this experience, keep the bug bounty well funded uh and and updated uh address always the reports very fast especially if they come from top level white hats. There's a lot of spam these days but if it comes from a top 100 uh white hat it's probably something very serious. Uh another concern that is very important is that whenever you get this report keep everything like very strictly need to know. uh you don't need to involve all the whole team and you don't need to tell the whole team everything about it.

Uh so keep the details like very tight in these cases. Again do not discard any course of action like think outside the box if you don't have like standard measures to apply and also involve seal and the security community early on. So one reflection is that how did we improve B3 design based on this experience. So essentially we took all the uh the causes of the exploit which is essentially okay the pool had a rounding issue uh in B2 you could put the pool in artificially low liquidity levels and you could swap very small amounts which in principle is like yeah the the latter two things is like yeah why wouldn't you be able to swap like small amounts and in B3 we said like

okay that's not a valid use case if someone is trying to swap just a few way at the vault that's probably an exploit so let's just uh iron it out. We paid much more attention on the rounding and yeah essentially we also enforce minimum liquidity requirements at the base layer. So bottom line just restrict unnecessary freedom for the callers and this is something that we will keep reviewing in M3 and keep reassessing uh our security model. So now it's like after the first exploit I want to address the elephant in the room which is the second exploit that we had very recently. This one was much worse in terms of damage. It was above 100 million exploits. Some of that was

recovered. Some of them was paused uh and some of that was given back by some white hats. But essentially very quickly uh yeah why didn't we deprecate B2 pools after a first exploit? So here the answer is that we had no reason to believe that other pool types were affected uh back in the day. Uh people might see just the audits but the truth is that these pools also got a lot of attention via the back bounty program throughout all these years including many top level white hats. So there's a bit of this Lindy effect where after a protocol is live for many years yeah you kind of take it for granted to some degree that it it has to be safe. that wasn't the case. So, this is clearly a

lesson for all of us in this industry. Um, why didn't you disclose earlier like like last time? Well, in this case, we didn't have an immunifi report. I just woke up one day and like yeah, the attack was already done. Uh, so there was nothing to disclose in our case. It's not that we knew in advance. And why didn't you post the pools? We did post some pools. We had monitoring uh but only some of them could could be paused by design. The others uh the post windows have already expired. So they were immutable. There was nothing to do on that front. So more lessons learned. Uh the fact that a protocol is a long time live and in production doesn't mean that it's 100% uh free of bugs. That's a very unfortunate uh lesson from all of this.

But we have to face the reality and that's the case. The bug bounty efficiency like yeah in our case it was 1 million per critical. There's some arguments in crypto Twitter whether that's enough or not, but in any case, yeah, there's no such thing as 100% back bounty efficiency, no matter what the value uh of the rewards are. Uh and then lastly, that I think it's also worth focusing on a last barrier of defense in case like all of the others fail. Uh monitoring and pause like yeah, it's a trade-off. Again you can choose whether to make your protocol completely unstoppable or not but just know that in these cases it's very useful to have monitoring and uh something that you can effectively pause. Then the some

something interesting interesting that also happened is that there are front runners that are able to figure it out like on the fly essentially look at some transaction in one chain and maybe copy the same attack on another chain changing some values and actually rescuing funds. So I I find that just crazy in our case where we had some white hats that did that and returned the funds. One of them is bitf finding. Uh yeah like super interesting stuff that they are doing. Uh yeah the rest didn't want to appear here but yeah uh that that is something that exists and it's good to have contact with them. Um and then well again partners that can't take centralized action. So in some case they froze the funds such as monerium

stake wise uh and yeah after the the attack is consumed being able to freeze the funds eventually becomes uh important moving forward. Um yeah we will reassess our security assumptions. I think even if the code is immutable we should review uh our security assumptions from time to time and that's what we are going to do even with V3. Uh again there's nothing that we know that is concurrently broken but still we will just not take that for granted. Uh keep the channels open with your auditors. Uh prepare layers of defense and inform the users the best you can regarding the risks of LPing your protocol. So yeah, I'm running out of time. Uh but yeah,

I'll be around for questions if someone has one. [applause] Thank you.