When Clients Disagree: Picking a Side Safely

Transcript

Hello everybody. Hi. Uh, thanks for coming for first of all. Um, I know Vitalic is speaking right now, so it must have been a hard choice. Um, my name is Luca. Uh, I'm in charge of validators at Serenita. And, um, I want to talk to you today about one issue that we face as validators on Ethereum, which is what to do when clients happen to disagree on what hap what's happening on Ethereum. Um, so just a quick agenda. We're going to talk about the situation when it happens. Um, then we're going to talk about what's actually at stake when it happens and and in the end we're going to take a look at different approaches at what validators can do when this happens. Um,

so let's start with the basics. Um, we're going to take a look at a blockchain. A blockchain is a chain of blocks, right? One following the other. Um, and this is kind of the happy day scenario where we see like one block builds on top of the other and everything is like great. But unfortunately there are situations when something like this happens. And in this kind of situation um we have two different forks of a blockchain where uh in this instance we see that after the second block 3A appears and this block 3A is deemed valid by some clients but invalid by other Ethereum clients. So when that happens they both move into

separate branches and they start they basically diverge and they start following different versions of the chain. So that's the theory of it. Um one important question that we need to answer is does this actually happen in practice? Um and it turns out it does. Um it's not like an everyday occurrence but it does happen from time to time. This is an overview of the uh this kind of bug. They are called consensus bugs. And this is an overview of those consensus bugs in Ethereum's proof ofstake era. So you see um we had one in 2020, then it was quiet for a while and then we had a bunch in 2024 and also a

bunch last year. Uh one of those is quite infamous. It's called uh the Hleski incident. It happened about a year ago. It happened on a test net and it had quite devastating consequences which led to the complete deprecation of the test net. Um so yeah um it does happen in practice. Why is that a big deal right? Um it's a big deal because uh like I mentioned Ethereum is in a proof ofstake era now which means that there are validators who have something at stake um and they are they stand to lose that stake when they take some kind of um invalid action or when they vote incorrectly. So in this kind of forking scenario

when validators end up voting for the wrong chain they tend to lose part of the ETH they staked. Um, if about 20% of the valer set does something like this, it's not a big deal at all. You can see the losses are pretty negligible. Um, nothing really bad occurs if only 20% of the validator set votes for the wrong chain. But this all changes when we increase the numbers and we see that like 40% of the validator set votes for the wrong chain. Um, in that case, those validators that vote for the wrong chain tend to lose 22% of all of their ETH staked, which is a decent chunk. And we only need one client bug, one client to have this kind of bug for this situation

to appear uh to happen. Um, so yeah, that's 40% already quite dangerous. Um but what's even worse if uh an even more significant network uh part of the network votes incorrectly votes for the incorrect chain um in that case like for this example with 70% of the network voting incorrectly um most of those validators will end up losing almost all of their stake. So yeah this is kind of what happened on the hleski test net last year. So now that this is clear that there is actually something at stake and validators have something to lose. Um let's move into the ne next part of the presentation and let's take a look at

what picking a site what picking basically one of those chains looks like in practice. Um and the this is like the most simple kind of setup that Ethereum validators use where they use their validator client and the validator client is connected to one consensus layer client which is then connected to an execution layer client. So this is basically the the most simple setup you can get if you want to be a validator. And when it comes to picking a side, what this kind of setup means in practice is that you pick a side before a consensus bug occurs. So you kind of pick your client pair, you pick your uh clients, and then

it's to a degree a matter of luck which fork of the chain your chosen client pair ends up going with. So this is basically only safe for your validators if you happen to end up on the correct chain, right? Otherwise, you're still not in a good position. Um, if you are a homestaker, it may just be like this may be the most affordable option, right? Running a single client pair. So if you are in this situation, the best thing you can do here is to run minority clients. So both on the CL side and on the EL side um there are a lot of people that are

using a little bit of uh more advanced setup where they use multiple client pairs and usually they use them in this kind of fallback um way. What that means is they have their validator client. It is connected again to a primary client pair which in this case is loadar and red. And if for any reason that primary client pair goes offline, um their validator automatically switches to the fallback client pair. So this protects your validators from going offline in case the primary client pair has an issue. So this is um a really good way to protect validators from going offline and making sure your uptime is high.

Um but when we get to the issue we're discussing here uh the different chain uh the different forks of the chain what happens here in practice is your validator basically by default is going to follow whatever your primary client pair is going to follow. But that's not all. Um it may also switch to the fallback client pair under certain circumstances. So this is kind of good for uptime but quite bad when it comes to the consensus bug issue because you may end up on either of these two chains.

Um the way the validator determines which of those client pairs to follow um this is from one of the validator clients documentation. It basically keeps track of the healthiness of both of those client pairs and then chooses to follow the healthier one of those. But following the one that looks more healthy is not necessarily the one that is on the correct chain. So this can still land you on the incorrect chain even if only your fallback client pair is on that chain. So good for an uptime from an uptime point of view, but from a safety against consensus bug point of view, this is actually quite dangerous.

Um, so these were two of the most common setups used by people also some institutional stakers. Um, is there a way to do things better? I'm happy to announce that yes, there is. Um, one of those ways is using what we call multi-node validator clients. And the principle here is that you connect your validator client to multiple different client pairs but not in the same way I described before where they're used as fallbacks but they're all used at the same time. So your validator clients um it talks to all of them at the same time and it makes sure that at least a threshold of these connected clients agree on which chain to follow. So this

this principle I like to call it attestation consensus because it happens when your validator is supposed to attest. Um so this kind of setup is actually safe even in the face of consensus bugs as long as a threshold number of your clients follows the correct chain. Um, this is probably uh still achievable for even homestakers to run three client bears, but especially for the larger players, I would recommend going even one step further because Ethereum has multiple different clients. So, I'd say go even further and run five different clients and then have like three of them agree on which chain to vote for.

Um, yeah. So this is uh basically the safest way you can run validators when it comes to protecting them from the dangerous consequences of consensus bugs. Um both Vero and Vouch are multi-node validator clients. Both of them support this kind of behavior. Uh apart from Vero and Vouch, there's also DVT. uh last year when I presented vero for the first time here DVT didn't support this this behavior this safety behavior but thankfully nowadays it does so if you want to protect your validators go with vero vouch or one of the dvt solutions um you do need to configure them correctly so for example for oal

you need to opt in to this safety behavior uh for vouch you also need to set a correct at the station data strategy um there's some trade-offs all of these clients make. Um, if you're already using vouch or dvt, uh, just make sure that you configure it correctly to protect your validators uh, against these consensus bugs. And if you're don't happen to be running either of them, then I think Vero is actually the easiest one to get started with. Um, I'm going to leave you with a couple of recommendations here. um split up into recommendations for homestakers.

So the first thing for homestakers is if you're using a single client pair, use a minority client pair. If you're using two client pairs, so you have two nodes at home or at a friend's house, it's safer to use a multi-node validator client um in the way that is shown on the right here. So you require both of those nodes to agree on the chain before you vote. that is safer than having a fallback client pair. Um, and no matter what you do, um, always go with the most minority client pairs you can.

And for the professional node operators out there, um, it's I would advise you to run at least three different client pairs. Again, the same kind of principle applies that minority client pairs would be the ones you get started with and then you add on as you move like to five different client pairs, it gets safer and safer to also add like the more used clients like Lighthouse or Prism or Teu. Um, yeah, the consensus threshold uh is configurable in both Vero and Vouch. So that's something you can play with. Uh it depends on your risk tolerance. So say you're running five different client pairs. You may choose to be okay with following a chain that's agreed upon by

two client pairs, but you also may want to play it super safe and require four or even five out of five of those client pairs to agree among each other before you vote for a fork. That way you can like be extra safe or a little bit more okay with a little bit of risk. Um yeah and the last point is uh increasing your at the station consensus threshold around network upgrades which is historically the most risky period when these consensus bugs are a little bit more likely to happen. So you might run like a three out of five threshold normally but around network upgrades you might want to like bump that to four or five. Um, yeah, the the only reason why you wouldn't always run with a super

high threshold is at some point it's going to affect the performance. So, there's kind of a trade-off. The more safe you are, the more impact it's going to have on performance. So, that's why it's like a trade-off you're going to have to make for yourself. Um, yeah, I should have one more slide here. sure if it's going to appear, but anyway, it's just some links to documentation. So, um, it's time for me to take your questions. Thank you. >> Thanks. When you say the threshold for DVT is not configurable, like how does it work in practice?

>> Um, I think it just uses the DVT threshold that it's that is in use. So, if you're running a four node cluster, it's going to be three out of four. So like the you'd use Varo for this like within the actual subnodes of a DVT cluster. >> Uh no, you'd actually still use the normal DVT software you use. So Karen or SSV and as long as you configure those correctly, they're going to also take this at astation consensus into account. So you don't use Varrow with DVT, you use one or the other. Given how bad the impact of a client bug might be and uh existence of Vero and vouch where you can basically safeguard

yourself from it. Uh why not to make it a mandatory thing to use Vera or vouch? >> Well, how would you enforce that? >> Um by consens rules. So for example, enshrine the fact that you should have like two different signatures, I don't know, something like that. I don't think you can do that. That's the same same reason you can't reliably um uh measure client usage. There's no way to really like on the protocol level prove that you're using a client.

>> Okay, gotcha. One more question from Maris maybe. >> Um, how would you see uh all of this happening or all of this moving into the ZKVM world? >> Yeah. Um, especially on the execution layer, we'll probably have the proofs relatively soon. So I think the idea there is that you'd have a similar kind of mechanism built into CL clients where they would require multiple ZKL proofs to be present before they validate the block. Um and if we go even further

where even the CL may be like a ZKL then I think it still makes sense to to like use different implementations and combine their output. So I think there will still always be room for something like this that combines data from different clients like Vero is a very lightweight layer. So something like that with a very small attack surface. Um another thing that I just thought of is that now we were going to block access lists. What we could do is just verify the block access list once on one m on one client and uh have all of the other clients uh statelessly verify uh the state transition function. Um so you

would only like you would get most of the reward with uh with not having to run five clients. >> Yeah. Yeah. That's something yeah that I agree with. That would be really nice if you would be able to run these lightweight stateless nodes. So you'd basically make it more feasible for even homestakers to run this kind of multiode setups without having to require like terabytes of disk space. Yeah. >> Okay. Thank you.