When Clients Disagree: Picking a Side Safely - Luca Winter - Serenita

Transcript

That's better. Hi everyone. Uh my name is Luca. Uh I'm in charge of our staking operation as at Serenita. And um I'm here today to talk to you about how to pick a site safely whenever clients end up disagreeing. Um so we'll first talk about what it actually means for clients to disagree when that happens. Then we're going to talk about what's at stake when that happens. And lastly, we'll uh look into ways how you can pick a site safely. So, uh clients disagree.

Um sometimes this is the like the the happy day scenario where we have a blockchain, we have a blocks coming in and all clients follow the same chain. But there are scenarios where some clients don't agree with other clients on which chain is valid and in that case we end up with having two different forks or even multiple more forks. Um, and what happens here is that on the blue chain we have block 3A which was proposed by one of the clients and for some reason like client A thinks that block is valid but client B thinks that block is not valid. So client B basically refuses to follow chain 3A and so on and also refuses to build blocks on top of block 3A and that ends up in

this kind of forky scenario where we have two different chains and yeah and different clients are going to follow those chains. Um do clients actually ever disagree? Does this thing happen in practice? Um this is a an overview of consensus bugs that have happened since Ethereum transitioned to the proof ofstake system. Uh you can see it was quiet there for a while between 2020 2024 but uh these clients these bugs do happen. Uh we've seen a few especially in January of 2024 was was kind of an interesting month where uh Basu and Netherm hit consensus bugs on mainets shortly after one another. Uh not all of those bugs that are listed here uh like

made it all the way to mainet. Some of them only made it to test net or were discovered even before. Uh but yeah, we do have these client bugs. They are a reality. So why is that a big deal? Uh it is a big deal because we Ethereum is in a proof ofstake uh era and that means validators actually have something at stake. Um and the the way this at stake thing works is if a validator does something that looks like an attack on the network, they tend to lose that stake. And when you as a validator vote for an invalid chain, that may look like an attack on the network and you stand to lose what you have at stake. And how bad exactly this gets depends on how

much of the network is voting for such an invalid chain. Uh if we only have like 20% of the network voting for an invalid chain, basically no issue at all. No issue at all. But once you go higher 40% there are already certain edge cases that can happen where validators tend to lose like 22% of their stake. And the worst case scenario is what we've seen on Hleski earlier this year on the Hleski test net also mentioned by Allian just now. Um and that basically in that scenario most validators would lose 100% of their stake because they voted for that invalid chain.

So, uh, we don't want that to happen on mainet, right? Um, so how do you actually choose a site? Uh, turns out it's difficult. Um, worst case like all clients end up on the wrong chain, right? But let's not go that far. Um, let's just assume some clients are on the correct chain, some clients are on the uh invalid chain. It's still difficult to determine which chain to vote for. Uh, you'll see messages like these in your logs. This is basically indecipherable. you know something's wrong but you don't know which side is actually correct and yeah you the thing is it also from this state from the moment you see these kinds of things in your logs it can take the core

devs uh a while to figure things out. So in case of the hleshki test net incident like everybody was on standby because it was during a network upgrade everyone was was already online. It took like about an hour to figure out which clients were in the wrong. So yeah, it'll take a while in practice. Um, so I'm going to begin with a couple of less safe uh ways to go about this. The first way is like this is like a very basic setup, right? You have a single execution layer client, a single consensus layer client, and a validator client. And in that this case, uh, your validator is going to do whatever this client pair decides is valid.

So this approach is only safe if the client pair you happen to choose follows the correct chain. Right? Um many homest stickers run like this. Not not a fault at all. Um if you do need to run this kind of system for any reason at all, the safest options you can take is to run a minority client pair. So basically try to choose the clients that are run by as few people as few other people than you as possible. Um then we have the fallback client pair option which is used by a lot of people uh also professionals. And the thing about this setup which we'll get to is this is what it looks like. You have a primary client pair. You have a

secondary plan pair. You could have even like a tertiary and multiple multiple pairs down there. Um the way this kind of setup works is your validator follows your primary client pair by default but when it notices your primary uh is unhealthy or something it switches to the fallback. Um in the case that your primary client pair ends up on the wrong uh fork, you're basically no better off than you were in the first case where you only run a single client pair. So it doesn't really help that much so far. But what's even more dangerous about this kind of setup is when this kind of forking scenario happens and say your primary client pair is on a correct chain

which um has low participation rate and the secondary client pair is on a wrong chain is on an invalid chain. your validator client may switch over to the fall back pair because it's like it sees more blocks than the primary client pair. It like the secondary client pair may look more healthy to your validator client than the primary. So it will switch over and that's that could be dangerous in case of forking scenario. So that's why um yeah this is probably the main takeaway I want to want you to get from this talk. It's dangerous in some cases to run this kind of fallback setup with

different clients. Um, yeah, good for non-consensus bugs. So, if one of the clients goes down, goes offline, that's fine. This is a good way to solve for that. But in case of consensus bugs, this is dangerous. Um, okay. Now, safely running safely picking a site is difficult. Again, um, there's an asterisk there because there's no way to always pick a site safely. Uh, what if all the clients have the same bug, right? if there's no picking aside safely in that scenario. But let's focus on the more uh realistic things and what we can actually solve for. So, and what you can do better versus the previous approaches I

mentioned is use a multi-node validator client. Uh, multi-node validator clients are special in a way that they allow your validator to connect to multiple client pairs at the same time and basically combine their output. Like take a look at all of those client pairs and see how many of them agree on a chain and basically by default multiode validator clients can follow whatever the majority of client pairs agrees on. Um, yeah. So this is this kind of setup is actually safe as long as that threshold number that you can decide on uh follows the correct chain.

You can bump uh take it a step further run like five client pairs you can run even seven client pairs on Ethereum mainet nowadays and decide on whatever threshold you're comfortable with. So like if you're comfortable with three clients choosing some kind of chain and you want to follow that chain that's fine. Yeah. So the threshold is configurable. Um yeah, this is probably the safest thing you can do right now. Um in case you have heard of vouch but not of Vero, I'm just going to explain uh the differences very quickly. The main thing is they both support this kind of at astation consensus mechanism where they talk to multiple clients decide on their combined outputs.

Um, Vero has support for the Ethereum remote signing API which is used by web3 Signer whereas Vouch doesn't. And Vero has this uh cool feature which is built-in slashing detection where it stops performing all validator duties as soon as it notices that any of its validators have been slashed. So it kind of prevents further validators from getting slashed in case you do misconfigure for some reason. Um, I'm going to leave you with a couple of recommendations. Uh some of these are for homestakers. So here I would start with running a minority client pair if you do have access to two machines or more. Um I'd say using a multiode validator client is preferable over a fallback client pair. And yeah, if you

do end up with a multiode setup, it's still a good idea to run minority client pairs as part of that multiode setup. And then for the professionals, um, I'd suggest really running at least three client pairs, defining a threshold that you're comfortable with. Again, prefer a multiode setup versus a fallback setup. Uh, DVT is also an option nowadays. As of recently, it also correctly accounts for these kinds of things. Um yeah, last uh bullet item there is uh recommendation to temporarily increase your atastation consensus threshold around network upgrades because those network upgrades are a bit of a more risky time when more consensus kind of

bugs could show up. Uh yeah, so like a day before the upgrade, you'd increase the threshold and a couple of days after the upgrade goes through successfully, you can just lower it wherever it was before. That's all from my side. Um here's some links if you want to get in touch. Uh if you want to run Vero, feel free to ask me any questions. Um yeah, that's it. Thank you. Excellent presentation, Luca. We actually do have uh one question uh in there for it and I'm kind of curious if you could expand on this a little bit.

Um could you describe or contrast the differences between Vero and Vouch or the the two together compared to DVTs? So Veraro Vouch versus DVTs. >> Okay. Um yeah, there's a lot of this kind of information in the documentation. Um but uh Vero and Vouch are multi-node validator clients whereas distributed validators are powered by like Oel or SSV. Um the main difference between Vero and DVT is that with Vero you have the key in one place. So you have the full validary key someplace and with DVT it's the key is like spread out in different locations.

Um that's probably the main difference. There's also like a difference with Veraro and Vouch. Vouch uses its own like ser and that ser allows you to also split up the keys into different locations. So it's like it gets very technical very quickly. Uh yeah, Yoric >> gonna run the mic here. >> Yeah, sure.

So both um OAL and SSV now also do this kind of um protection slightly differently. >> OB you need to opt in for now. That's important to know. >> Exactly. And of course the question is you know who's running which client pairs. So the way I think about it is if you're running the whole infra then a Vera web3 Signer or a voucher works great and if you use DVT that's just the same thing with an extra protocol and more steps and maybe a token but if you are staking for someone and they say I don't trust you I want you to have 1/4 or 17th of the key and I can't give you the whole key then that's where DBT shines.

>> Yeah. Yeah. DBT is like the best option if you have multiple node operators involved. Um, and I think if you have only a single node operator taking charge of all the infrastructure, I think Vero and Vouch are simpler to set up. Um, and Vero is part of ETH Docker. So yeah, I think Vero is the easiest option if your main goal is to protect from consensus bugs. And then depending on your exact use case, you may go with Vouch or DVT. >> Excellent. Another round of applause.