DAPPCON 2025: Protecting Validators From Client Bugs Using Vero - Luca Winter

Transcript

Hi everyone. Thank you for coming today. Uh, hi Gom. Um, yeah. So, my name is Luca for those of you who don't know me. Um, I'm the CTO at Serenita where we specialize on staking Ethereum. Um and today we'll be uh looking into how to protect validators from client bugs using a software that we developed and open sourced at Senita which is called Vero. So um there will be an active part to this workshop as well in a few minutes.

Uh in order to actively participate you're going to need a Linux machine with ideally Docker installed and about 4 gigs of RAM. like less should work as well but four gigs would be recommended. So let's dive a bit into the background behind Vero, why it exists, what it is and so on. Um the reason why Vero exists has a lot to do with something called client diversity. It's an idea in the Ethereum space where um Ethereum doesn't want to rely on a single client implementation in order to keep its blockchain going. Um, so the idea is we should have multiple clients that all

all track the same blockchain and that way Ethereum is much safer from these client bugs because if one client has a bug, we don't want the whole blockchain to like stop going. So that's a quick intro to the idea behind client diversity. Um, it's much more important to the whole system than many people think or are aware of. Um, there's a good article by Dankrat advised um on his blog or or his website which goes into the importance of um client diversity.

Uh so yeah that's a recommended read. Um so currently thanks to this whole idea and making it a priority for Ethereum uh we have six available consensus layer clients and six uh five clients on the execution layer which you can choose from and you can basically choose any of those. Um ideally that would mean that each of those clients would be run by a proportionate number of the network. So in case of the consensus layer clients, we'd be looking at about 70% of the network on each consensus layer client. And in terms of the execution layer clients, we'd ideally see about 20% of the network run each execution layer client. Um so

that's the theoretical and ideal scenario. The reality is a bit different. Uh on the consensus layer layer, on the consensus layer, we see um some clients uh are being used much more than others. uh especially prism and lighthouse which we see on the top here um are used by a lot of people probably a bit too many um this picture is not completely up to date like loadar is shown as 0.1% but it's higher than that as well as grandine but it's like a rough roughly accurate picture of things and then on the uh execution layer side it's uh a lot more uncertain uh all those orange blocks in the middle that

you see, those are areas of uncertainty. So, we don't know which clients some people are running. But um this situation on the execution layer side has improved a lot since about a year ago. Um so we are currently in a much better space in terms of client diversity than we were that one year ago. Um especially Nethermind has made a big rise in terms of adoption. So now that we're in in this uh state where all of the clients are likely used by less than twothirds of the network, are we out of the woods? Are we good now? Uh can we stop worrying about client diversity? Right? That's the question. And I'm here to say that we

really are not quite out of the woods yet. Um because of multiple factors. One of them is the client usage data. Like I showed on the consensus layer side, it's not 100% accurate. There were some changes in the most recent network upgrade which made measuring this like it yeah it made it less accurate. Um then there's also the fact if a client is used by more than 3 33% of the network so one-third uh it can still be dangerous in some edge cases. Um then there's also the possibility of two clients sharing a very similar bug like we saw recently on the test net. So if multiple clients share the same bug they

can still get over 66%. And then uh one more thing is kind of under discussed which are failover mechanisms. Um, many node operators run multiple client combinations in a kind of failover mode where if one of them one of their clients goes down, they switch to the other one. And this could um do the wrong thing actually because if you run these two clients and one of them starts following a different chain, you don't necessarily always want to switch over to another client as soon as something's up with the first one unless you know what's actually going wrong.

So that's the failover mechanisms. um as a node operator, especially as a professional one with like a responsibility for other people's money, you want to avoid um these client bugs as as much as you can. Um but the hard truth is we cannot completely avoid client bugs, right? The these are complex pieces of software. They're going to end up containing some bugs at some time. Um Ethereum is constantly evolving like the clients are optimizing. So there's always some room for bugs to be introduced but um we can reduce the impact of these bugs and the simplest way to do that is just by pointing some of like by using

multiple clients right and pointing some of your validators at one client combination some of your validators at another and so on. So in this case that we have on the slide here uh we have three client combinations and we basically split up our validators among these three combinations. Uh the pros of this approach um is that if everybody were to do this we would be in a very good space. Um so it's good for the network if you do this. Um it's a simple approach. It's relatively easy to set up and it still reduces the impact of client bugs. But in this case, if say one of these client combinations does have a bug, still like a third of your validators would go offline. So there is

some kind of impact, but it's not huge. It's pretty good. Then there's a couple of more sophisticated things you can do to protect the validators against client bugs. One of them is using vouch, which is a specialized validator client. Um it can it's it's a multi-node validator client which means that it can talk to multiple clients at the same time. So in this case it talks to like three consensus layer clients and three execution layer clients at the same time and combines their output. So if one of them like has a bug goes offline it can keep going using the other two. Um this software is used by a lot of

professional node operators. It's been around so it's been battle tested. It's a really solid piece of software. Uh it's highly configurable. You can do all sorts of things with it. It's open source also nice. Um the only cons I can think of is that it requires its own specialized key manager. So that key manager which contains the keys, it's not compatible with other software. So yeah, basically vouch requires Durk and uh the same way the other way around. Um and the other con is actually the same as the pro that is used by many professional node operators. Um the reason I mention it as a con is because all of those professional node

operators use it. A lot of Ethereum validators use it. And if vouch were to contain a bug, you would be offline with many other validators which would would well it could just cost you some money by being offline at the same time as they are. and you're basically forced to wait for a fixed voucher release. You can't just switch to a different client or something. Um then there is the other approach which is called DVT which stands for distributed validator technology. One of the implementations is from SSV. Uh and with DVT you basically run a single validator out of multiple places.

uh you kind of split it up and um yeah so in all this those places you can use a different client combination and again you can this way protect validators against a single client bug um uh so moving on to the next DVT uh implementation which is from Oel it's called Karen the high level idea is the same so you again split up the validator can use multiple client combinations in each place. Um, in this case, uh, Obel has, uh, Obel's current software has a business source license, uh, and it has a bit of a non-standard middleware approach. So, it's, yeah,

again, not inter interchangeable with anything else. So, all of these advanced approaches to avoiding client bugs have something in common, and that is that they're kind of their own way of doing things. they kind of lock you into their own ecosystem and it's hard to switch to this advanced solution. It's also hard to switch back from it. So at this point the question arises, can we like get to the same results? Can we avoid client bugs without all the complexity?

Um so yeah that's when uh I thought we should write a new validator client um with some design goals. One of them was that it should be able to talk to multiple nodes at the same time in able in order to um avoid these client bugs. The other thing I wanted to do is keep it simple which is always good when it comes to software um security. Obviously this is a validator client. It's close to private keys. So you want it to be secure. I wanted it to be um observable. So I wanted us to have a good idea of what's happening when it's happening, why we're having issues to make it easy to investigate any issues. Um I also wanted

it to be compatible with all client implementations and one important thing for me was making it compatible with web designer which makes it then much more compatible with the rest of the ecosystem. So that's how we came up with Pharaoh. um it's um like succeeded in terms of all of these design goals. It's a multi-node validator client. It's simple. It has less than 4,000 lines of source code. It's secure. It never touches validator keys directly. It's highly observable and it's compatible with um all the client implementations as well as web3igner.

So when it comes to a high level overview uh where does Varrow come in? So this is uh an example of is let's say more sophisticated setup with multiple clients. Um we have the execution layer client, CL client, the validator client and then a remote signer which is connected to its uh slashing protection database. So Veraro is basically the same thing as all of those validator clients as in the middle. You can just swap it out easily for any of those in this kind of setup. Um so uh when it comes to Vero being multiode that can mean multiple different things but um in the context of Pharaoh it just basically means that

it talks to all of these client implementations at the same time um and it follows the chain based on their combined output. So if we say have four um client combinations and one of them has a bug goes offline, a Vero is still able to follow the chain that the other three client combinations um agree on. In terms of observability, it has uh readable logs that you can follow. There's uh dashboards for graphana that you can simply install. The left one is just a high level overview of what Pharaoh is doing and but it also offers you the ability to like dive deeper into at which point in the slot are certain

actions taken and so on. Um Vera also supports exporting traces so you can really dive deep into uh this is a blog proposal so in case you miss a blog proposal you can really tell at which point it actually went wrong. Uh, one novel feature that Vero has and that um have that has only recently been started to introduce in other clients is slashing detection. Uh, Vero listens to slashing events happening on the entire network. And if it detects any slashing event for a validator that is managed by Vero, it automatically shuts down its validator duties. So that way it prevents like

further validators from being slashed. Uh this is what it looks like on the dashboard. So this is under normal conditions, no slashing detected, it's performing duties and as soon as any of its validators are slashed, it stops all activity. Um so with a new validator client, some people are worried about its performance. Ver currently powers thousands of validators. So we've like made sure that it performs well and uh we're currently like among the top on Ethereum in terms of um validator effectiveness as well.

So if you're worried about Vero's performance, you don't necessarily need to be um in terms of risks. Like I've talked about the good things about Vero, but maybe there's some bad things too, right? Um I tried to be objective here. Um, there's a few like possible vectors that you could be looking at like slashing, but Vero uses the battle tested slashing protection that Web3 Signer uses. So, that shouldn't be an issue. Um, there's also the slashing detection feature that I mentioned. So, it's kind of better in some ways in terms of slashing protection.

Um, in terms of validator key compromise, Vero never has access to validator keys. So that's you don't have to worry about ver leaking those. Um in terms of bugs obviously um every piece of software can have bugs but this is a small code code base. Um it has high test coverage. It's very simple like I mentioned. So um that should hopefully help us avoid bugs. There is no lock in. Um you can very easily switch to a different validator client at any time. And in terms of downtime, yeah, I mean it's possible that Vero has a bug which leads to downtime, but in that case you can just switch to a different validator client very easily.

Um, one more question is Vero for professionals only? It's not. Um, you do get most of the benefits if you run it with multiple beacon nodes and so on, but you can also just run it with one beacon, one client combination. So, um, easy enough. Um, Nexo wants to ask a question. Mika, could you pass the microphone? Thank you. You said you could use it uh with one with one machine and one one client combination. Why would you why would you use Vero for one client combination? Is there an advantage there? If you like Vero's metrics, for example, they're more detailed, I think, than other

validator clients um for the slashing detection feature, which some don't have yet. So, there are reasons to use Varrow even with a single client combination. Yeah. Okay. Thank you. Any other questions so far? Okay. Um, so yeah, there's a lot of good things about Veraro, not that many bad things. So, I would encourage you to try it out. Um, in terms of what's next, we're working on supporting the next network upgrade and also considering adding uh double gamer protection.

Uh, the end goal is basically for people to start running Vero in this these kinds of setups where they use a lot of different clients around Vero and that way they have a lot of like different clients um view of the chain that they can use to make an informed decision on which chain to follow. All right, let's uh move on to the active part of the workshop. So, um has anybody in here set up a machine that they can use? So, we have Yeah. What about you, Giam? You're going to join the active part?

Okay. It requires Docker. So, just follow me then. Okay. So, um, Luka, in case you want to follow it, this is a website where which lists all the commands that we're going to be using. So, we don't have to like, uh, manually transcribe. This is what it looks like. So, you just go to workshop.sarren.io and you can just copy paste the commands. So, we're going to get started with this. Oh, right. One more thing. Uh, this is a high level overview of, uh, of the setup we're going to be using today. So, we're going to be using three execution layer clients, three CL clients, and Pharaoh with the remote signer.

But to set all of this up today would be too much work. So, we're going to start off by only setting up Pharaoh. Um, and I've already like set up the rest of the infrastructure for us. Um, so it's time to begin. Um, the first command that we're going to run is this one. Um, it basically starts Vero and points it at the beacon nodes that I've prepared. Um, so yeah, let's do that. I'm going to copy the first command.

And um, let's paste it in here. And what this does is it runs the runs the validator client on the hoodie network. It uses this remote serer and then it uses these three beacon URLs. The last thing is just the setting the fee recipient which is required for VRO to start. Okay almost. Um, I'm going to need SW here.

So, it's going to pull Ver's image. All right. And we are basically live. You can see the first slot ticked off here and we already published some additation here.

It's as simple as that. if you already have the beaconos running. Yeah. So, yeah, you just need the addresses of the beaconos, a remote signer, and that's it. How you doing, Luka? Yeah. Is it working? Perfect. Okay. Um All right. So, that's not the end of the workshop. Um this was the the simplest way to get like running, right? If you only need this this single piece of this architecture, then it's pretty simple.

Uh we're gonna go one step further than that today. But I got a question before we Okay, so now we have run the same command and so what are you using to to not slash us just remote signers uh slashing protection database or something else? Yeah. So, um, these are the locks we've seen. Um, and that's what I wanted to mention. We're all using the same validator keys, uh, because we're all using the same remote signer and that remote ser has its own slashing protection database. Exactly.

So, even though we're running the same validators, we're not going to get slashed because there is a slashing protection database in place. So yeah, we could get into a point where um we tried to attest to different things, but because of the slashing protection, one of us would fail to do that. All right. Um you can now stop the process. It'll take a few seconds to stop because Vero has this um shutdown mechanism where it waits for um any pending duties to be finished.

All right. So um in this next part we're going to go one step further. We're besides Pharaoh we're also going to spin up uh the remote signer and it's uh slashing protection database. So the next commands to run are basically we're going to be using E Docker to do all of this to make things easier. So, we're going to clone the e Docker repository and um yeah, make some changes to it as we go along. So, we're going to run get clone this one.

change into the directory. Going to make a copy of its default environment file. And then we're going to make some changes to this this file. Um I'm going to be using nano here, but you can use any text editor. And the changes that we're going to make are the following ones. So, we're going to change the value of the compost file. So, it's this one.

And we're going to need to set the fee recipient value. Uh the network should already be set to hoodie. So we don't need to change that. We need to enable web3 signer which is right here. True.

And then there's one more thing. I think we need to change the CL node setting. So that that's the one that points VO to these different beacon nodes that are already running. So that one's all the way down. Where is it? Here it is. So, we're going to remove the original value and paste the new new value.

And I think that's it. So, we set these. We We're going to save the file. Um, how are you doing, Luka? All set. Okay. So now that we have changed the environment value, we can bring ease docker up using this command. This will take a second as it downloads some docker images again. So we're working towards this setting

this up barrel and web designigner. How do clients updates work? Do they get pulled automatically or do you have to wait for Vero to um incorporate them or anything? They work automatically. So, E Docker automatically pulls the latest version. So, as long as Vero like tags something as the latest version, then EORE will automatically like take that.

Still waiting for some images. You done? Okay. Also waiting. Okay. If anyone happens to be watching online,

um maybe you and send me a message if you want to actively participate because you're going to need a special thing soon. So, you can uh DM me at seneta_luca luca. Still pulling here. Um, it could be specific to uh what

Ecker does. It likes to build its own images. Like even if there's an official image, Eucker builds its own version sometimes. I think it has sometimes changes permissions and stuff like that. So yeah, as you can see right now, it's using the web resigner base image and then it's going to like perform some other things on top of it, right? Uh okay, now I have to use am I uh why? But yeah, like it's the web three ser itself that's taking like that's downloading and rebuilding all the layers because uh or is it just that the network is slow?

Um uh the network is a bit slow here, I'd say. Yeah, that's okay. Okay. Yeah, but also webiner is being built. I see it in my logs. So, okay. Yeah. And web designigner itself is very big big image if I remember correctly. Okay, understood. A lot of these docker images are a bit bloated. Yeah, that's fine. But looks like we're almost there. So, did you manage to build yet, Luka? Yeah,

exporting. So, I'm pretty much done. Okay, looks like I'm done, too.

So, ether is spinning up all of these services. This looks like it's done. So, At this point we ran east up and now we can run the logs to check the logs from the validator service. So if we do this we can see um you can see verarrow starting up. So these are all the it's it's a little hard to read. Let me just make it a little smaller.

Hopefully this is still big enough. Um so yeah it's started up and the only thing we can see is that we can see some warnings in the logs which say that there are no active or pending validators. So it's actually not performing any validator duties. Like the validator client is running but it's not doing anything because it doesn't have any validators attached to it. So that means it's now time to already did all this.

So yeah, we're seeing these messages warning us that update uh duties are not being updated because there are no active validators. So we're going to be adding some keys now. Um we're going to be using um these magic numbers here to import keys and yeah that will um like we will import validator keys into the e docker setup and then barrow will perform um the duties that those validators should be performing. So it'll look somewhat like this.

going to enter a magic number here. I'll uh give one over here. This is your password. So this magic number will allow you to um download some validator keys. So if it downloaded successfully, you should see like this kind of output. Um, if you see an error like this with a 404 status code, then it didn't download

successfully. Did you manage? Okay, perfect. So, now we have a zip file that contains all these keys. Uh, we're going to extract the keys into uh the correct directory for each docker using this command here. Um, so you should see the zip files being inflated. And next we're going to say eat the keys import. And this will import all of the keys that are located in the um in the directory that we just extracted them to.

So this will take a minute. And at this point uh Docker is asking us these are warnings. Yeah, if you're going to do this like for real, you should read all these warnings. Um, but yeah, it's basically saying uh it's warning you if you've used these keys before, make sure you that you uh turn them off and that you're using session protection databases and so on. So, we understand these warnings. We say yes. In this case, all of the validator keys have the same password. So we're going to say yes and we're going to enter the password which is just workshop

workshop. So ether is going to be importing all of these validator keys now. Um we're at this step. So at this point all of the validator keys have been imported. We're going to run e I think it's restart. Yeah, restart validator. That will make Vero um refresh its validator keys.

And if we now run the locks command again, then we're going to see Barrow's um logs and we're going to see that those warnings are no longer there. It's not warning us about anything anymore. It's actually updating duties now. And you can see already some addestation has been published here. So, it seems to be working for me. How about the other Luca also? All good. Perfect. So, yeah, that's the logs. Um, we can also like dive into metrics, but it's a bit more complicated and I don't want to keep you too long before the

panel with Vitalics. So, um, I'm going to like show you how it's done, but we don't need to necessarily go into the details because it also uses some other Docker images. So, we'd be just like waiting for a while. So, the the way it works is you add Graphana to your uh, East Docker compost file. you adjust the share IP in order not to share graphana with the rest of the world and then you just run eastd up again. Um it will pull some docker images and at some point it'll be done and then you can open uh local hosts at port 3000 in your browser and you'll have a graphana spun up. So you just log in to that. The

default credentials are admin admin and then you um click on dashboards and you'll find the vero detail dashboard. Um you select which instance you want to follow. You adjust the time frame and you basically get these nice charts that show you how uh Vero is doing. Um yeah, for anyone interested in uh traces, you can uh like talk to me and I can show you like what those traces look like. Um I didn't it's it's not possible to set it up as part of e docker right now. So okay um do we have any more questions? Uh I didn't get the part with the traces. So

if you can't get them from docker where do you get them from? Um, eucker doesn't have any like way any endpoint where you can send traces to, right? But so clearly you can extract them because you can show them you said. So where where how do you extract them? Um, you just add an environment variable to Varrow. You like that contains the endpoint where it should send traces to. Yeah. And yeah, we use tempo for that. Um, but yeah, Tempo is not part of um the East Docker stack right now. So all right that's all for um today. If you want to like learn more or keep in

touch um these are some contact details and locations where you can find more information. Um thank you for coming and uh I also saw that Lislaus is watching online. Hi.