Kody Sale - Correlated Failures and Network Impact

Transcript

All right, thanks for the intro. I'm Cody Sale. Um, yeah, I probably don't look as good as Vitalic on stage yesterday, but I'll try to keep you guys entertained. Um, currently leading product marketing at Oel. Prior to that, uh, built a little protocol on Ethereum called Smoothly, which was an ME smoothing pool for solo stakers. And before that was just a community member who was passionate about home staking. So I guess I'm uh one of those guys who just like started from the bottom now I'm here. Um oh yeah and today we'll be talking about correlated failures and Ethereum.

So from a high level you know what are the dependencies among a bunch of node operators that could either bring the network down or um you know things like that. There are three types we'll be looking into. So yeah, if you look at this heat map kind of shows you uh you know the geographic distribution of um nodes on Ethereum today, specifically like the east coast in the US and Western Europe are definitely hot spots. And so we'll be looking at cloud failures, stake centralization and client bugs as three different types of correlated failures uh today.

So for cloud failures, uh this is the distribution of consensus layer nodes on the network today. Um unsurprisingly, you know, hosting nodes dominate with a roughly 61%. Uh shout out to ether nodes for always having amazing data that we can pull from. But yeah, this isn't really surprising because a lot of the professional node operators are running uh uh in cloud and they run redundant systems. So that uh they also run DBT so you can fall back um in case something happens. But you know I was kind of surprised the residential is still roughly 35% of the network. So even though you know solo stakers only account for you know maybe between 3 and 5% of the stake today we

account for a lot more of the nodes on the network. And that's just a testament to like you know the impact running a node at home can really make. And so if we break down, you know, the hosting, you know, the hosting nodes, Amazon's pretty dominant with, you know, roughly 30%. And, you know, they don't have a third of the network. So that's pretty solid. But, you know, that's closely followed by, you know, I'd say two of the top cloud providers and bare metal providers that, uh, solo stakers use, which would be Heter and OVH. Like I'm a, you know, I run my node at home, but I also have an OVH uh, node in on bare metal just as a redundancy feature.

All right. So, yeah, while I was actually preparing for this talk, we got to test in prod what happens when like you know, one of the biggest regions of AWS goes down. So, on October 20th, 2025, uh AWS, I think it was US East one went down. Um, it was like maybe a 14 hour outage and yeah, crypto Twitter turned into a bit of a bit of chaos. You know, from a retail user perspective, it looked like, you know, the chain was down. You couldn't access anything.

That cat's supposed to be going like this trying to get a transaction in the network. Um, so yeah, from an opt optics perspective, it looked really bad. Um, [clears throat] but most people in here know that actually it went really well for the network itself. Um, we, you know, I'd say, you know, based on those two charts before, roughly eight 18 Hello. roughly 18% you, you guys still hear me? All right. Roughly 18% of Ethereum runs on AWS, but consensus participation only went down by roughly between 1 and 2%.

And so yeah, that was pretty amazing. I was actually, you know, before this outage happened, I was thinking that that would have went down by a lot more and that like missed blocks would have been a lot more as well. But it kind of speaks to just the maturity of the staking ecosystem and how incentives are actually working and um that a lot of the node operators are running redundant infrastructure like vouch DBT and um other backup systems. But yeah, other things went less well. So you know that like a lot of decentralized applications maybe are uh maybe they're dyno they're like decentralized in name only and so a lot of the DAPs frontends went offline and

so you couldn't access um you know any of your favorite decentralized applications to get your transaction in. And even if you could access the application, if you're using MetaMask with the default settings, then you're using Infura. And Infura's mainet RPC went down. And uh yeah, we kind of got wrecked. Like Coinbase went down, Robin Hood had an outage. Uh and a variety of other RPCs just um yeah, went offline for an extended period of time. So, it was one of those moments where we've had them over and over again when there's like a cloud outage, you can't access the network, and we always say we're going to do better, and then we never do. So, yeah, it's a kind of a repeat of history. But the good part is

we all know Ethereum just kept chugging along, and yeah, didn't even skip a beat. All right, next we'll look at stake centralization. This was originally called stake centralization not decentralization because we've actually improved so much from a couple years ago. So if you remember uh you know 2022 and 2023 Lido was getting you know close to 33% of the network and um the community you know we kind of turned to the social layer to call on the ecosystem to keep the network decentralized. Um, famously, you know, Danny Ryan jumped on a

bankless podcast and was like really aggressive uh about the community decentralizing uh stake and, you know, it's still up for debate. I, you know, whether or not LTO deserved that hate. Um, you know, they run multiple operators. Um, but I think the network's better for it. So, you know, now today LTO's at 23.7%. But not only did they reduce the share of the network, they implemented new modules like the um simple DBT module which moved them from you know 36 node operators at the time to I think over 250 after simple DBT and also uh now they have the permissionless CSM module.

So yeah, LTO's I mean moving in the right direction. Absolutely. And onboarding a ton of node operators and um yeah, it's great for solo stakers because you know like in CSM you can earn a lot more rewards than staking uh solo and LATA wasn't the only one. So because of like the social layer push, Ether also had a etherf solo staker program um and you know brought on over 200 node operators in that program as well. Yeah, we could probably all use a green chart today. You know, it's been a rough couple weeks. So, this is the state of the uh network today as far as um state goes. From a like Nakamoto coefficient

perspective, it now takes three entities um you know, assuming LITO is a single entity to hold hostage finality. Um and before it was really, you know, Lido plus one small entity. So, we're definitely improving stake decentralization. Um, yeah, it's more decentralized than ever. So, like although this talk, you know, is kind of sounds kind of bearish like correlated failures in Ethereum, I think as a community and as a whole, we've improved um a lot over the last couple years. And what that tells me is the social layer really had the push had lasting effects and just speaks the testament of like the impact the social layer can actually make.

All right. So, let's Yeah. The third type of correlated failure would be like a client bug. And now people are, you know, I guess correlating client bugs with the supermajority risk. So, this is an old chart. Um, you can tell because it looks like a chart from like Excel in 2012, even though it's pretty recent. uh in January 2022, GU, you know, had over 80% of the network and Prism, you know, was almost at two/irds of the network at 60%.

And this was another time when the social layer actually kicked into gear and was like, we can't, you know, this was before the merge, but after the beacon chain launched. And I think the goal was really to get Prism and GU below that uh you know, at least at least 50% before the merge happened. Um, and really that's that's what we did, right? There was a big push for client diversity, you know, adopt minority clients to preserve uh, you know, to help out the network. And, you know, it was really impressive. And so, today we're looking a lot better. Although, you know, Prism's down now to 22% and Lighthouse jumped up to almost 50%. So, I think it's always a work in progress,

right? Like, we can always improve client diversity. like we should all be moving, you know, to some Loadar, the Nimbus, the Aragon, the Basus, the Wraths. Um, but, you know, I'm confident that as a community, we can definitely make that happen. And just like the previous issue, the um, you know, the cloud outage, we also got to test this in prod. So during the Holski hard fork or the pectra hard fork on Holski testn net, we actually did see the supermajority bug happen and the network justified an invalid block and yeah it was pretty chaotic. Um you know we'd been speculating for a long

time what would happen in this case and um yeah it's basically like doomsday worst case scenario. Um eventually the network got to finalize but yeah a lot of the validators were slashed and because of that the exit Q extended to you know over I think it was 18 months. Um so if that happened on mainet yeah I'd say just ETH keeps we're just going to zero. Um but you know yeah it would be ugly and so what are ways we can actually play uh you know defend against this?

So, first I'm sure a lot of people in here if you're at the East Staker event are running a node at home. And so, if you're running a node at home and when cloud goes down, you can definitely, you know, use your own node as your local RPC. That's probably the easiest way to get your transaction in. But if your DAP front end's down, then, you know, maybe you need to pull up like Forge or something and use a uh use a CLI tool to get it in. But, but still, you can you can definitely get your transaction in through your local RPC. Next is run a halting mechanism. So after the issue on Holski um at Oel we started thinking about you know since we're using a multi-operator

architecture we could actually compare um you know the head of the chain among the nodes in that cluster and if the nodes didn't agree on what the head of the chain was um or if it forked off then that cluster could actually stop a testing um and you would have to manually intervene. So, we tested this on Fulu and it worked really well. So, that is one way to protect yourself from this like doomsday scenario. And I know other uh teams are also working on things like that as well.

The next would be stake with multiple operators. You know, don't put all your eggs in one basket. Find operators that run multiple clients. And the last would be choose a minority client. So instead of the guests and the nethermines and the lighouses, you know, pick a, you know, all all these clients are really mature at this point in time, you know, minus the maybe newer the few newer clients. Um, but yeah, run a minority client. Don't be part of the problem. And you know speaking of like the minority client if you're running DBT at OLE you know we've always pushed for this like endgame cluster architecture which is you know multiple operators in various regions around the world also

running different client implementations and then implementing this stake or this chain halt feature so that you know you're you're diverse around the world you're running various clients with different operators um but there was always like this single point of failure in such that Kon is a middleware so it sits between your consensus layer and your validator client but there was only one implementation and you know you have to start somewhere so that's what we did um but recently we released the DBS spec and invited uh the client teams and other builders in the ecosystem to start writing a second client and a third client um using the DBS spec so that Kon is not the only you know single point of failure in a DB cluster

And over the weekend, we announced at Stake Summit that Nethermind is the first team that's going to be building this. They're a longtime partner of ours, did research in multi-party computation um a long time ago and are definitely um you know, they've been they've been helping us with the security audits along the way. So yeah, happy to get Nethermine on board and yeah, looking forward to like the true endgame cluster architecture which is uh you know a multi-client at the middleware middleware layer as well. All right. Well, that's it from me. I'm Cody on Farcaster. If you want to hit me up on Telegram, Cody Sale. And I think we have time for questions. Probably [applause] questions from the audience.

actually uh had had one question myself here which is um you know out of the scenarios that you proposed are there any that you see as most likely in terms of probability >> the client bug issue was going to be near zero um and I say that it probably is still near zero but um that jumped up much higher than I expected. But cloud failure still seems like the most possible outage to me. Um because it's happened over and over. You know, we don't we have there's a precedent that like clouds will go down. Um but luckily like the network itself just keeps going and you know, from a retail perspective, they just users think that they can't

use the networks. They feel like it's down. But most of us know that it's chugging along. >> Excellent. Any other questions from the audience here? All right. Well, Cody, thank you very much for the presentation here. >> Yep.