Luca Winter - Improving Ethereum Resilience With Multi node Validator Setups

Transcript

All right. Hi everyone. Uh my name is Luca. Uh I'm the CTO at Serenita where we specialize in staking Ethereum. And I'm here today to tell you more about how we not just we but we as a community can do better and how we can improve Ethereum's resilience with multiode validator setups. So this talk will be about the topic of client diversity. We've heard a lot about clients. We're not done yet. Then we'll uh review the Hleski incident which was basically the the worst case scenario that can happen when it comes to client bugs. And we'll finish off with talking about how we can do things better going into the future.

So uh what's client diversity? Uh client diversity is the idea of um having multiple clients so that in case one client has an issue, Ethereum doesn't stop, we just keep going. Um this has been helpful in the past and we hope will be helpful into the future. Um, this idea has resulted in multiple clients being developed. So, we currently have, as you can see, like seven different consensus layer clients, seven different execution layer clients. If I missed any, sorry. Um, they're not all production ready, of course, but you can basically use any of those if you want. Um and if the entire network spread out the usage of these clients um all across the network then we would see

each client being used by about 15% of the network which would be an amazing thing. Um but of course we know by now that reality is different. So um in reality we see well first of all the client usage data is kind of vague. We're not completely sure how many clients uh how many validators are powered by which clients. So it's kind of tricky to tell what the situation is. But we know that we know for sure that it's not 15% for each client. Right? So um there is a question to be answered. Do we actually need all of these clients?

Do we need client diversity? Uh like I mentioned, we've seen client diversity help Ethereum in the past. So I made this overview of client bugs that have happened in the past. Um not all of the client bugs because there have been many. These are only the ones that affect consensus. So it it's basically only the the client bugs that were that had a opportunity to derail some of the clients from the correct chain. Um while doing this I run into the issue that um there were so many of these bugs they didn't fit in one slide. So I divided it up into the proof of work era and later on we'll say the proof ofstake era. So uh as you can see there have been a few.

Um this is not my way of saying client devs suck. Not at all. This is just reality. This is going to happen completely fine. It's it's a complex piece of software and bugs are completely inevitable. Uh well you could avoid them but it would cost a 100 times more. So this is probably better to just have multiple clients. Um, so this is the proof of work era. Moving on to proof of stake. It's been a little more quiet for a few years if you notice between 2020 2024. Not that many bugs then, but then back in January of 2024.

That was pretty interesting because if you, as you see like in the span of two weeks, we had two client bugs um, execution layer client bugs hit mainet. So that was a pretty interesting uh period and that triggered some node operators to start using different clients which was which was good and then we had uh the big incident hleski incident at the beginning of this year during the petra network upgrade on the hleski test net um we're going to talk a bit more about that uh but I prepared this slide earlier this year and since then we've had more client bugs pop up so I had to update the slide Um yeah, just to give you an idea how many of these bugs occur, it's it's a reality. So let's

look into and quickly review what happened on Hleski. The Hollleski Pectra incident. So um Pectra was the next network upgrade to hit Hleski. Um it was scheduled uh for February 24th at 9:55 p.m. And only six minutes later, something happened. So what happened? People started seeing these kinds of things in their logs. Invalid blog basically they were seeing invalid blocks being proposed on the network. Um but the funny thing was most of the chain kept going and so initially nobody was all that worried. Everybody thought this is going to be fine. Um do

we still have Marius in here? Yeah, there he is. So basically get Nethermind and Bessu agreed on one state of the chain and as I said the network kept going. It wasn't ideal. There were like being blocks being missed and so on but it kept kind of kept going. So people were not worried initially, but then um as people were investigating what is the cause of these invalid blocks, the mystery unraveled and uh by the time stamps by the way you can see this is about an hour after the bug appeared. So it takes a bit of time to figure out what exactly is going wrong in these kinds of scenarios. So at that point people figured out that Wrath and

Aragon were actually in the right there. And um yeah then the worry started. Okay what if this invalid CH chain finalizes um as as you can see some more confirmations came in. This is an hour and 20 minutes after the consensus bug appeared. So yeah it takes time to figure out what's going on. And the end result of all this happening was that um validators started to vote for this invalid chain to finalize. It did not get finalized all the way but they did vote for it. Then there were five forks and it was a whole mess and the complete end result was that the hleski test net needed to be deprecated

ahead of time before um it was planned. So, as we've seen, it it can happen. Um, and how bad these things get kind of depends on what the whole network does. In this case, a lot of the network ran those three clients that were affected. Um, that's why it was so bad. But uh just to quickly review the different cases um in terms of how many validators vote for the buggy chain, if it's only about 20% then the impact is negligible. The chain keeps finalizing and basically nothing bad happens. if the bug is contained to only 20% of the validators.

But as soon as you start hitting larger numbers, 40% then it already starts getting messy in certain edge cases. And even though um the 40% is unable to finalize their chain, it can still get quite messy and the people that vote for this 40% buggy chain can still lose up to 22 to 22% of their staked ETH, which is quite a lot. And for this kind of scenario to happen on mainet today only one client bug is needed a very specific kind of bug but still and then we review the hleski kind of situation where more than twothirds of the network is on the wrong chain and that's where it gets very tricky because you can't just go

back to the correct chain and you stand to lose up to 100% of your staked E. uh for this to happen on mainet right now two client bugs need to share this kind of correlated bug um going back to the hollleski incident in this case we had more than twothirds of the validators voting for this incorrect chain when is this going to happen on mainet right we hope it never does um so that's what I'm mainly here to talk about how do we assure that we don't get that situation on mainet so there's a few approaches um we can just stay ignorant about the issue and just act that it doesn't exist.

We can do this and I feel like that's what part of the network is doing right now. We hope for the best. We hope that it doesn't happen. Uh we just yeah but um that's not what I'm satisfied with. So um I'm going to show you how to can actually do things better and in practice. So, one of the things you can do is split your validators across multiple client pairs. And that would mean like if this is an example where you're running three 3,000 validators, you would split them up between multiple client pairs, uh, ideally minority client pairs, but basically split them up. Um, and if the whole network did this, then we would have a very safe

Ethereum in place. But the problem with this method is it relies on the rest of the network. So you basically need the whole network to buy into this which is not happening. So what can you do about this if the network is not co cooperating? You can protect at least your own validators and one of the ways you can do that is by using vouch. Uh vouch is a multi-node validator client. It allows your validators to connect to multiple client pairs at the same time and basically only attest to a chain if those client pairs agree. So this is very useful because if one of the clients ends up on the on a on an invalid chain, your validators are not going to vote for it.

Um, another thing you can do is use DVT. It's a similar concept. you basically uh combine the inputs of multiple client pairs and based on that you make a decision. The things that these two last approaches vouch and evt have in common is that they're pretty complex to set up. Um it's not that easy especially for homestakers to run multiple client pairs set up vouch with its own key manager or DVT with all of the complexity that that entails when it comes to splitting keys into partial keys. So it's not that easy to set up these two last uh approaches.

So my question two years ago when going all through all of this was can we do can we protect validators without all of that complexity and that's where the idea of barrel was born. It's a multiode validator client. It's similar to vouch uh but it's very simple. It's uses web3 signer so it's much easier to basically get started with. It's part of e docker. So it's very approachable much more approachable to homestakers as well. Um and just to show you this is the concept of a multiode validator client. Um it talks to multiple client pairs and you can configure a threshold that you're comfortable with like I want all three client pairs to agree on a state of the

chain or you can say I want two client pairs to agree on a state of the chain. And if one of those clients goes down, it's fine because the other two are going to keep things going. Um this is a similar overview of the pros and cons. Not biased at all. Um some it includes some nice graphana dashboards that you might like. Um has some uh advanced features like you can export traces as well. Um, one thing that not many validator clients have is uh slashing detection feature where Vero listens to all slashing events happening across the network and as soon as it detects any um any slashing event that affects any of

your validators, it stops performing duties. So if you for some reason misconfigure your setup, your first validator is still going to get slashed. But this automated mechanism prevents further slashings. So this is what it looks on the dashboard. Once any of the validators are slashed, it stops performing any duties. And if you're like feeling like this, um the good news is it's very easy to switch to Varrow as opposed to voucher or DVT. And I'm going to show you just how easy it is. Um say you're running this kind of setup. It's a very simple kind of setup where you have your execution layer client, uh consensus layer client, and your validator client.

Um, the first thing you need to do if you're considering to use Vero is move your validator keys into a separate remote signer. That's because Vero doesn't interact with validator keys directly. So, Vero is built like secure from the ground up. It never wants to touch validator keys if it cannot void it. So once you have your validator keys and a remote serer primarily probably web3 serer um at that point you can switch out the validator client for verarrow seamlessly within minutes and at that point your setup is no not much safer than it was before because you're still using one single client pair but at this point uh you should spin up some more client pairs once you happy with

their stability then you connect Vero to all three client pairs, set up your at the station um threshold that you're comfortable with and you're done. So that's it. You can do this thing while it's switching over in minutes. Like I said, uh this is what the command looks like. So it's it's very simple. Um with Veraro, if you set up this kind of three node setup, your validators are going to be immune to any single client bug, no matter how much of the network uses that single client. So that's very helpful. You don't need to worry about client usage data. You just spin up this kind of setup and you're golden.

Um if you want to take things one step further, you can set up a five node setup. And again, based on the threshold you choose, you can make this kind of five node setup immune to even uh four client bugs. So if four clients share a bug, you can have your validators safe from that kind of scenario, which is even worse than what happened on Hollleski. So the goal um not just my goal, but I hope for Ethereum is to get to a state where 40% of the proof ofstake network runs some kind of client bug resistant setup. Whether it's pouch, whether it's Pharaoh, whether it's DVT, a combination of these three. Um, and if we get to this state where 40% of the network uh

runs these resistant setups, we will never finalize a buggy chain. So, we will avoid the most catastrophic of scenarios. It's one thing to lose finality for a while, but it's much worse, of course, to finalize something that's invalid. Um, this is the setup we run. We run five different CL clients, five different execution clients, and um, if we can do it, anyone can. That's it from my side. I'm happy to take any questions now. >> Thank you. Um so for example, Lighthouse

already has this support that the multiple beacon chain support uh built into their validator client. like why would anyone switch from that uh to Vero versus just using Lighthouse and uh Lighthouse uh validator client also supports all the different CL interactions. So like >> the difference between the Lighthouse validator client and Barrow is that the Lighthouse and every other traditional CL client validator, it only uses one beacon node at a time. It's a fallback kind of setup. So if one of them goes offline, it switches to the next one. If that one goes offline, it switches to the next one. But it always uses one client pair at a time, whereas Vero uses multiple client pairs at a time.

>> Use the mic. Okay. Because I've worked extensively on this stuff. Um implementing not not writing implementing it in production. Lighthouse and all of the others cannot do a at a station threshold. So this idea that you say five of six need to agree on source on target during slot zero or we're not attesting to this because we got a major bug that does not exist there yet. >> Yeah. And that's the critical piece that avoids your validator from ending up on the invalid chain.

>> How much of the network do you think is already running uh barrow or like some of the others? >> Well, I'm aware that many professional node operators run vouch. That doesn't mean that they're running a client bug resistant setup necessarily because uh Vouch has multiple different strategies you can use. One of them is just a simple fallback kind of mechanism, but they also have a strategy that's called the majority strategy and that's the one you should use if you want to be uh immune to this kind of bugs. Uh I can't tell you exact percentage of of the network. I know Veraro isn't used that much yet because it's kind of new. Um, we know that uh DVT is starting to ramp up. I think SSV claims about 10% of the network if I'm

not mistaken, but they only well I don't want to get too technical, but they their setups weren't resistant to these kinds of client bugs so far. It's only included in the latest release. So, but I don't think we're at 40% yet. And if I had to give an estimate, I'd say probably like maybe 5%. Uh not really a question but Nimbus client can also support that for like two years but the delay traces looked very cool.

>> Thank you. Um Nimbus only supports multiple execution layer clients. So this is like CL and EL and if you use the Nimbus CL it can connect to multiple different execution layer clients but it's only half of the story. So it's a bit different. But in this kind of way that where it confirms the at the stations >> I'm not aware of that but yeah let's discuss it after I just just to clarify on if if you're running four different uh CL's and EL's and two of them pick one chain and two pick another you just go with the majority because you

>> depends on on the threshold you configure. Yeah. >> Um, by default it goes with the majority. So if it's two and two, it will just stop a testing because neither has the majority. >> Um, yeah. >> Yeah. Um, second question, uh, if you if you're running four pairs, do you get performance improvements? So, like for example, if a block comes late on one of the nodes, right, but it doesn't on one of the others. >> Yeah, you can get performance improvements and it definitely doesn't hurt performance. If you look at the raider rated leaderboard, you'll find us on the top.

>> Yeah. So, if you're proposing, you ask all four at once and then choose the best one. >> Yeah. >> Yeah. Cool. >> Thanks. Um, you had a little secure green chick. Um, kind of triggered me a bit. So, I'm I'm looking at your repo. Um, it's in Python. >> It is. >> Awesome. Have you got any external security reviews? >> No, not yet. >> Okay, cool. Thanks. >> Yeah. Look, keep in mind the whole slashing protection is like one of the important features that's handled by web3igner which has a audits.

>> Excellent. Uh I see a couple questions in the back here. >> Could you speak about the resource requirements um associated with running Barrow a little bit today? What's it look like in production? >> Yeah. um a validator client is very light in terms of processing needs. So even with this kind of multi-node validator client, it's still very light. That's why even Python works. Um all it does in terms of processing is make a bunch of API requests to the beacon node, make a bunch of API requests to the signer. That's that's most of what it does. So resource requirements are very low. You can run thousands of validators on a single

node. >> Awesome. Thank you for the contribution. >> Yeah. Uh, what's the right attestation threshold to be absolutely bulletproof? Four of four, four of five, five of six, seven or seven. >> That's the thing. You can't ever be 100% safe. What if all the clients contain the same bug? >> Well, then it's >> BSC, Bance Smart Chain, right? >> No. So, that's definitely a possibility. What I would recommend what we are doing is run a three out of five threshold on mainet um and around hard forks we increase this threshold temporarily so we go to five out of five and if we notice okay this is a bug but it's fine um we don't

want to follow that chain then we can go back to four out of five but yeah in general I say a correlated bug that appears the same way across multiple clients coming out of nowhere on mainet across like three clients is very very unlikely. So I'm comfortable with the threshold of three out of five and a temporary increase around hard forks. >> Excellent. We are out of time here, but uh please give a round of applause here for Luca. >> Thank you.