A Brief History of Correlated Events (and What It Means for Ethereum)

Transcript

Hello. Hello. Does this work? Okay, great. Uh, my name is Thomas. I'm from the Ethereum Foundation. I work on the funding coordination team. And what we'll be covering is a brief history of corateed events and what it means for Ethereum. So stay tuned for the history and we'll look also at some data. So if anyone likes data in the room, we're going to have some data points around bugs, vulnerabilities, and how that gets mapped out on protocol security. All right. So, I want you guys to imagine uh driving. So, I assume most of you actually have a driver's license. If not, that's okay. Just imagine you are. And you have you encounter all these obstacles on the road. You have the weather. There's so many variables. You have other drivers. You have like holes on the road as well. There even animals to take into account. And there's even a

component of luck into the equation, right? And there's even a non-zero chance you may encounter a a dragon, right? Uh so why why do we bother driving with so many of these risks? Well, we manage them through education investment and that's including um what we do driving uh driving knowledge to the theory and also in the environment and investment into that. So it's not always the case. So it doesn't matter if you bought a Honda or a Toyota BMW because Takata was a recall that affected an inflatable bag in most of these cars. So, Takata had an airbag that affected I think around 70 million faulty inflators in 42 million US vehicles and that was the largest recall. They represented over 20% of the

US market share. So, what happened to them? They had uh that basically had a massive recall of 24 billion and that ultimately led to their bankruptcy. So, what happened in the end? So, because if you actually have a huge recall, the customers are unaware of this recall and they need an incentive and car makers actually need an incentive for them to actually repair these cars. So the U US government came in and actually a department of the US government came in and came out with a measurement. So they needed to create a measurement that ranked uh car makers publicly and that basically put a lot of pressure on them legally and also uh operationally right and in the end that actually led to a lot of investment. So because of that uh legal and and reputational risk that actually led to a lot of investment. So

they started canvasing repairing and ultimately that led to more airbags being replaced and safer roads over time. Baltimore is another example. So Baltimore had a fire that spread downtown and they didn't have enough fire engines to actually put out the fire. So you had cities in New York, uh, Philadelphia that actually came in to help out, uh, on on on the scene. The problem was that these engines were adapted for those specific cities. So when they arrived, the fire hydrant couldn't actually be plugged in. So in the end, 1500 buildings burned up in less than 24 hours. Uh, oh 30 hours, sorry. Uh, so what happened? So insurance companies basically came in and started assessing all these uh all these cities and they put them on a scale of 1 to 10. Uh they started pricing them and then you could actually

be able to price an insurance event such as fire for every property. That made a lot of businesses and individuals very very mad because they ended up paying very massive premiums because the city wasn't improving uh their conditions. So what what ended up happening? all uh individuals started loving the city and then in the end water supply alarms, better environment, better infrastructure, everything for uh mitigating the risks of fire and ultimately that led to lower risks and lower premiums over time. Third example and final in case everyone's bored of the history, heartbleleed. Um so hard is a great uh example of a dependency that was being maintained by one single developer on a $2,000 a year donation.

And what happened in the end was they got a 7.5 vulnerability uh on the CVE and that basically because it was such an ex like such a critical dependency for 20% of the web uh that basically ended up getting them funded because companies realized such as Google and and a whole bunch like realized that they needed to fund this critical dependency otherwise they would get screwed. Um so because of that they then realized that they needed to also measure other dependencies in their stack and that actually was a great example of funders getting um like being shown uh ways that they can fund their critical stack. Uh so yeah weak dependencies fixed before they break. So in every scenario we've seen here uh an

event occurs, a measurement comes out and an action is added. It's a it's a reactive it's a reactive scenario and we want to look at how maybe how does like the Ethereum depend on Wait, is this the right side? I think it is. Uh sorry, give me a second. It is not the right slide. Sorry guys. Here we go. All right. So where is Ethereum corated? Sorry about that guys. Um, so let's look at Ethereum stats and maybe even some some bugs. So, Ethereum since 2020 has only had 366 bugs disclosed. That is on public resources that I've encountered online and also through whatever public repositories are available. Two in three

are actually caught by bounties, audits, and fuzzing. And only nine in Ethereum's history uh since 2020 has actually reached mainet. And as a result, we've actually maintained a 99.997% time finality held. So we'll actually look at how uh at what cost and where like some of these bugs look are. Um so the reason why finality is held most of the time and we haven't actually had a disruption in the chain is because of client diversity. So client diversity is our friend here and as you can see here you'll you'll see a share of the stake that stopped voting correctly. So each of these numbers represent a time when one of these clients went down and how it affected the network. So you can see here some clients introduce a vulnerability which affected the chain and only some of them actually uh uh did

not like finality held because of client diversity but if multiple clients actually go down at the same time then you start having something called a final a finality delayed but because uh that only misses some blocks and pauses or even slows down the chain you're able to recover it quite quickly. Um so as a result uh we also don't look at the protocol layer as much because D5 is more exciting when you have 9.3 billion in losses. You don't actually end up looking at the protocol layer. But the protocol layer is a source of tele events that we should probably be looking at closely. So 550 ETH is in missed validator rewards. That's cost opportunity not in users actually losing funds. So just to clarify validators have never lost funds nor have users.

This is just in rewards that you validators expected to receive but didn't get. Um so here are some of the four outcomes that actually you end up getting from a mainet incident. It actually comes down to four outcomes. That's divergence, delay, something finalizing and halting. Uh most of them as you can see actually end up being divergence and delay. And the stakeholders are impacted are shown on the right side here. And you that these are stakeholders that you should care about a divergence or delay because that either affects them legally, operationally or um or uh yeah or like their risk stack. Um, so how do we protect these stakeholders? So if we have delays and divergences, how do we how do we actually like mitigate some of these risks? Well, if they are the ones that rely on Ethereum and they are actually exposed to reputational, legal,

and operational risks, how do we actually solve that? Well, there's two two ways to do that. There's sustained funding and insurance against event. So sustained funding is primarily the target area that funding coordination is in itself trying to solve for Ethereum. and insurance is a possible direction that maybe institutions maybe even users could buy capital from capital markets and insure themselves against the protocol. Um so here is actually like um here's a surface area for all the bugs um that I've condensed down for where bugs exist uh since 2020 um on Ethereum. So most of them actually come from the execution and consensus client. So again, like I said, diversity is almost like a firewall, but you still have uh so many of these bugs that appear and they're most critical to the pro the

Ethereum protocol. You do get bugs in crypto libraries, EVM, protocol rules, um and even deeper in the in the in the protocol layer, but for most of the time, client bugs that actually appear on the client actually disrupt finality. Um this is sort of like my own interpretation of where lean could probably fit in. Um, so if lean if clients get formally verified, which is an idea that Justin Drake has put out on Twitter, um, what will happen is that clients will likely get less bugs, but the risk doesn't get eliminated. It actually gets put down deeper into the stack. And you'll see here formal specs and proofs get created as a new category of risk. And if you concentrate all the risk in one specific area, then you basically have migrated client diversity and eliminated into the into into one uh

category. If you're only relying on one language and uh and one uh one spec uh this is where actually most of the mitigation comes in. So if bugs get reported what is actually the incentive for them to contribute to that? Well execution clients and consensus clients actually get the majority of the bug bounty program because there actually a lot of critical highest severity bugs that get um that get reported. Uh so since 2020 we've had almost $1 million paid in in grants or in boundary should I say uh across uh across the uh the EVM stack and you can see this is not a list that uh grows or shrinks. It's it it remains constant. If if anything it's probably growing at this rate. Um so this just something to to be aware of.

Um this is where all the Ethereum bugs sit in for just another visual if anyone likes data. This is like uh CVES which is just a a vulnerability standard um of of um a vulnerability standard that is funded by the US government. Then you have github advisory. Then you have also have the EF catalog which is an internal uh repo that the EF maintains. And then the postmortems are at the end here are the ones that didn't get through didn't get reported by a CVE or catalog except seven of the nine. And uh they they show here that like the the interesting trend here is that usually when like usually when a hard four goes live bugs tend to appear the most. So you can see here uh on 2024 a great instance of a hard for going that that's the lines are like a

hard for going live and then a bug's basically getting reported after that. So it really goes to show that having a stable code base uh actually reduces the bug surface area. Um so bugs are one type of signal that institutionals uh funders or even any funders could look at as a source of data. You have client share, you have network health, you have concentration, you have security posture. Those are uh just examples of where you can actually like uh point towards like being able to get data and and get that funded. Um also there's like a with with client diversity there's actually different measurements of it, right? there's different sources that tell you different data points. And this is a big problem because if you're trying to measure uh um if you're trying to come up with data or a measurement

for client diversity, you're going to get different data points based on a single provider. So there is also potential for this to get fixed, but there hasn't been an incentive till now maybe. Um so how do we actually shrink fragmentation? Uh well, possibly we could look at um creating a shared record of maybe we need to create a better incident format, right? like how is it that we have so many of these vulnerabilities that get reported in different ways? Maybe we can standardize that. What about client share? Maybe there's a way where we can standardize that as well. And what if we had open data and schemas run by the ecosystem that could actually shrink all this fragmentation into a single vector?

These are open ideas. Nothing I'm suggesting. Um so let's talk about insurance here. How what does it take to actually ensure tail events? Um so you'd likely need a shared record. Where are these bugs? How are they exposed the incidents? Right? like we already know the number of bugs that occur in the Ethereum protocol and they're shared across uh four records, but what if we had a schema that could easily point towards um open models. I think this is a really interesting category. If we were able to get the data of all the bugs that have happened in the past, maybe there's a way we can predict the future as well based on those events. Um so there's no public model of tail odds.

So what if we could predict based on the next hard fork what bugs could get introduced so that uh you could price them based on like onchain specific triggers, right? What if a specific commit or a GitHub or even just like a piece of code could affect the odds of uh of a till event occurring and capital that will actually unlock capital. So if you're able to actually price some of these events, you may even be able to unlock some new capital markets that you haven't thought of before. What if we could have a bond market? What if we could have insurance, bespoke insurance of specific products that could actually be bought uh by users or even institutions? And there is a growing number of these institutions that may actually want to buy coverage over a specific client that they implement or even just a protocol in itself.

So that's the big question that I'm trying to solve. Is there a way to share to build a shared measure of political health? If we can actually build this and we unlock capital markets and even a growing number of users may be able to on board safely and have their assets held in custody. If anyone's interested in the debate, I am actually holding uh several workshops throughout the week. Uh third is already kind of passed. fourth. We kind of pass today. There is one actually um at 5:00 pm upstairs at the Resilient Networking Hub. So, if anyone wants to show up, we can and have a good debate, we can have that. I'm also holding one tomorrow, not tomorrow, sorry, the day after on the 6th. Uh so, if anyone's welcome to join, you can do it. You can scan the QR code for more

details and uh you can also follow me on X if you have any questions. Uh yeah, and I also have a bunch of data here on like all the all the all the possible like things that we can discuss if anyone's interested in it. Um, so yeah, but like I've just run out of time. So anyway, thank you. >> Yes, thank you very much, Thomas. I really appreciate the data driven approach that you've taken to this. There was once more