Did Ethereum Break? with Preston Van Loon & Terence Tsao

Transcript

in the night with finality on the ethereum blockchain last week and uh we want to go all uh we want to go over that today with some experts who can help us answer the question as to whether ethereum broke or not what does it mean to break what happened what does non-finality mean I think we'll talk about that and what are the risk conditions that we are exposed to as a result of this how did ethereum handle this event is is really the central question in my mind David who do we have on today we have Preston Van Loon and Terence Stout of the co-founders of Prismatic Labs now in Arboretum now part of off-chain labs uh and we brought Preston and other members of Prismatic on to the show before to understand the world of

clients because it is a complicated world but not so complicated that we won't be able to understand it here on the show so some very important questions with some very smart people who have been with the ethereum proof of stake effort since it was Once Upon a Time called ethereum two but before we get into the show we've got to talk to our friends and sponsors at consensus because we've got to talk about this thing called diligence fuzzing it's a new fuzzle auditing tool just fuzz it yeah with a sick new name called diligence fuzzing so what's diligence fuzzing for smart contracts salinity devs Auditors Enterprises engaging with web3 that need to ensure that their smart contract is secure before proceeding to mainnet diligence fuzzing it's an audit grade security tool that serves as an automated way to find cone

code vulnerabilities for people without hosting their own infrastructure teacher or writing complete complex test cases so there are many ways to do automated testing but buzzing is a way brand new way with uh quite a lot of sophistication according to consensus that you can go and access right now so this is an extra layer of protection for your smart contracts in addition to and above and beyond typical code audits humans are messy and can miss the things but diligence fuzzing is diligence by robots fuzzy robots apparently yeah to get this uh to get access to this David can you imagine releasing a smart contract and not fuzzing it I just can't even imagine I prefer fuzzy smart contracts it's fuzz it uh all right so

um there was an event that happened last week and this was an event that happened after the roll up and I'm wondering if we could kind of tee this up before we bring Preston and uh Terence on so what happened I think maybe the first sort of mini event was was uh on Thursday and then again Friday if I'm recalling correctly all all my weeks are blurred David but give us the recap before we get in here yeah it happened twice and so ethereum failed to finalize which is an interesting sentence what does that mean uh blocks were still propagating transactions were still going through but finalization was not happening and just had a high level finalization is just something that happens when enough clients enough validators have given the

thumbs up about a certain Epoch epochs which are I believe Preston's going to come in here and and give me the whip I believe 64 blocks uh but it's a collection of blocks and then if enough validators uh give the thumbs up then that that Epoch finalizes and that is impermanently embedded into the blockchain when a when a Epoch is finalized it requires uh slashing conditions to go back uh and unwind those blocks and so if a block isn't finalized it's not as permanently embedded into the blockchain as it otherwise could be except with this event which happened twice uh ethereum failed to finalize because blocks were being missed validators were not uh doing their jobs um I am only able to explain it to some

degree at a high level and so we're going to go talk to uh Preston and Terence to get a deeper dive into what this world of finalization means should do a disclosure before we get in um of course now Prismatic is owned by off-chain Labs which is a sponsor of the bankless podcast very supportive of what they're doing Layer Two in client development work also David and I are notorious holders of the asset called eth so you should know that podcast that we are bullish on ethereum generally and are interested in finding out what happened today so guys we're going to get right to the episode with Preston and Terence to talk about whether ethereum broke or not but before we do we want to thank the sponsors that made this episode possible including our

number one recommend exchange Kraken go check it out Kraken Pro has easily become the best crypto trading platform in the industry the place I use to check the charts and the crypto prices even when I'm not looking to place a trade on Kraken Pro you'll have access to Advanced charting tools real-time Market data and lightning fast trade execution all inside their spiffy new modular interface kraken's new customizable modular layout lets you tailor your trading experience to suit your needs pick and choose your favorite modules and place them anywhere you want in your screen with Kraken Pro you have that power whether you are a Season Pro or just starting out join thousands of traders who trust Kraken Pro for their crypto trading needs visit pro.crackin.com to get started today mantle is a brand new high performance

ethereum layer 2 Network built differently from the other layer twos you may be familiar with mantle is a modular layer tool built on the op stack but uses eigenlayer's data availability solution instead of the expensive ethereum layer 1. not only does this reduce Mantle's gas fees by 80 compared to other layer twos but it also reduces gas fee volatility mantle has a decentralized sequencer set eliminating the risk of downtime and censorship on the network and because mantle implements multi-party computation nodes layer 1 settlement execution is shortened from seven days to as low as just one or two mantle is the first layer two built by a dow and is backed by one of the biggest Dow treasuries in the world bit down mantle already has sub communities from around web3

onboarded to help the growth of mantle like game 7 for web 3 gaming or edu Dao for the world of D side and buy bit for tvl liquidity and on-ramps check out mantle at mantle.xyz and follow them on Twitter at Xerox mantle immutable is at the Forefront of web3 gaming on a mission to bring digital ownership to every player offering the world's best games and game development platform immutable lets game Builders and players focus on great gaming experiences so build your next web 3 game on easy mode with immutable's leading full stack web3 gaming platform it's built-in ux features like the immutable passport are designed for games to scale to the next billion players coming to web3 with immutable players can sign up with an

email pay with a credit card and experience a frictionless purchase flow inside of games so discover your next favorite game and explore a network of 150 games building on immutable including such titles as gazan chains Guilds of Guardians eluvium Ember sword and metalcore so join web 3's largest ecosystem of games and players build play and connect at immutable.com bankless Nation Preston Van Loon and Terence Tao are the co-founders of Prismatic Labs one of the organizations that maintains the prism ethereum consensus client Prismatic Labs was one of the earliest client teams that was formed to start building what was then called ethereum 2 which we now just call ethereum and they've been very key to ethereum development every step of the

way Preston Terence welcome to bankless hey happy to be back thanks for having me yeah okay I I'm gonna look forward to being educated about some of the nuances of finality and security but first did ethereum break Preston no it did not break right okay but we did see um ethereum actually work this we have a time where ethereum has difficulty finalizing and we got to see it actually work in action right so this is not a uh like a problem with ethereum that specification did not break it actually it actually worked and it worked in a unfavorable condition and we got to see that happen in mainnet for the for the first time

so how would you summarize what actually happened uh and so I I did my best to explain it in the intro uh how would you describe to the Layman explain it like I'm five what actually happened here sure and how many blocks are there in an Epoch Preston most importantly yeah there are 32 blocks not 64. at least you got a power of two so that's nice what happened was um multiple clients prism specifically and techu had this problem where if it received these sort of old attestations attestations being what the validators are producing to say this is what I think the head of the chain is they received these old attestations

which caused prism in particular to replay a bunch of states right it it had us to go back in time to evaluate whether or not this attestation was valid or or how to count this vote ended up being a pretty expensive operation that somehow bypassed or filled the caches that we had so we weren't able to you know keep that work in the client that had to recompute it every time so essentially it was an accidental I guess like kind of denial of service where the clients were so busy doing these computations they weren't able to respond this is the beacon note part of the client the

beaconode was not able to respond to what the validators were asking to do the validator said here's the block I want to produce here's the attestation I want to produce these weren't getting out the door because the client was so bogged down with trying to process these valid but untimely attestations that's the gist of it this happened to compliance Preston yeah well two clients have released updates specifically to handle this scenario that's prism and Tech who Lighthouse already does it really well so a lot of us have sort of taken an inspiration from their model of how they handle these old as the stations in short they they basically dropped them they say

this attestation is not going to be very valuable to how we're going to update the fork choice it's not worth recomputing all this all this effort so we sort of copy their model of uh of just excluding that attestation from consideration not doing all the work and so why did this happen so suddenly well um we're still trying to track down the exact you know trigger to the event but what we think had happened is that some clients namely Lighthouse when they were when they lose a connection to the execution layer the layer that the client that handles transactions and the

the state of ethereum as we think of from you know the application layer that being like go ethereum or or whatever you use for your execution client when we lose access to that some clients will behave well they'll just withhold these attestations and then later produce those um just sort of later release them so what prism does is they just don't we just don't want to test during that time because we can't really come up with uh we can't Advance our node so we're not it just wouldn't really be a testing there other clients will produce these sort of old valid but you know not valuable attestations

and that was what actually triggered the event it's not a common path that happens but what we have learned is that it wasn't something that was nefarious it was just some sort of unexpected Behavior valid Behavior but unexpected that prism and perhaps Tech who didn't consider or handle very well I think a good like a question here it's like why it didn't happen on test net right like why it didn't happen before I think there are several reasons to it the first reason is that there's a lot of validators now there it's 600 000 validators and validator matters that the number of about their matters the

mobile that you have the more hashing that you have to do on the stage and that's slower and our test net only has 400 000 validators so even the test net is lower than the main net which is obviously a flaw here that's something that we're taking back and we're gonna revisit how do we do this type of test net the second thing is is there's a lot more deposit now right because because now we enable withdrawal and then the the policy key with and then people are kind of getting their top off eat they're gonna really deposit it and then with so many deposits there's a lot more hashing that you have to do they also increase the latency so those are the two big better and once we reach those two boiling point and we kind of just

create these stats that have spiral okay so to recap what I'm what I'm hearing both um there were are new conditions about the state of the beacon train a higher number of validators are increasing the resource requirements of nodes and then also from what uh from what Preston was saying uh prism and Lighthouse shared similar properties because prism just borrowed from lighthouse's code base to do this one part of the client that in prism and because that part of uh was shared this showed up this problem that was triggered by The increased computational requirements and and bandwidth or networking requirements of the more and more validators coming in that was triggered but then because of the shared

property of Lighthouse and and prism that showed up in two clients which is how it actually showed up as a non-finalization event in ethereum is that is that recap accurate uh sort of so we we the fix was to replicate what Lighthouse handles his behavior it was prism and taku that didn't handle this scenario very well so when the outage have happened right there were something like 60 or more blocks were missing so that's an indicator that more than one client is having issues and we believe that to be prism and techu because both clients have really sort of emergency updates or critical updates uh since the incident on Friday and Thursday okay

okay I have a question in general about what this looks like for ethereum uh to not finalize so here is um a beacon chain visualizer we've shown before on Bank list I think uh specifically when the you know hard Fork upgrades happen you sort of look for these things to go from from red to Green right these are proposed blocks that we're looking at so when the blockchain doesn't finalize what does when ethereum didn't finalize on I guess it was Thursday once for a period of time and I don't know how long that was and then also Friday what first of all what does this chart look like well the the main indicator is that the

the green bar you see at the bottom of each Epoch justifying or finalize that's not going to be greater than two-thirds you need it to be greater than two-thirds to justify or finalize okay in the case of the incident last week we were missing a ton of blocks so you can see a couple of red spaces here in the Box those are missing blocks this is the proposer whoever they may be failed to to produce a block at that slot and because there were so few blogs you know had more than half of them were missing it wasn't possible to pack in all of the attestations and also very likely the same nodes that were suffering from not being able to produce the Box we're not able to produce the attestations either

so two parts to it you have to have enough blocks to include active stations and you have to have enough attestations that agree with each other that being more than two-thirds to finalize and justif or Justify and finalize an Epoch and how long were these periods of time where this sort of degraded finalization service was happening and the first time on Thursday I think it was for four or maybe maybe five epochs that we had a gap in finality usually we're seeing every Epoch is finalizing you know two Epoch later so in this screenshot you can see Epoch 63 is finalized when the Kearney Epoch 65 finishes we're gonna see the 64 will

finalize and we just have this sort of continuing rotating um you know the epoch II epochs that goes finalizing well when it wasn't finalizing on Thursday it took five epochs to get back on track and then on Friday it took even longer I don't recall the specific number maybe Terence knows but what was significant about Friday is when it goes longer than five epochs we start to incur a additional penalty that everyone gets penalized and it's called a uh a liveless liveness league I think um Terence do you want to explain so we we have this inactivity so this

activity dig is essentially what make ethereum World War III resilience right so imagine we cut the internet in half and then there's this um say Asia and then USA and they cannot talk to each other right in that case we want we want either side of a validator to dig their balance faster to lose their balance faster so we can finalize faster on each side so you will begin to lose balance faster if we don't finalize for more than uh four to five E policy and then and then eventually after 17 days you will lost probably around 16 years then you will get ejected from the pool so this is the first time this in

activity um in activity did happen so I did some math earlier so we ended up burning about 28 even and I don't know if this is something that we should be proud of because everyone loves spend eat but it's slightly different so on average every battle did her if today if you are offline at that time you lost about one dollar so yeah okay just to double down on that mechanism the idea here is ethereum's philosophy is to withstand World War III so if we imagine the internet as a gigantic cell and that cell gets cut in half ethereum should find a way to to bridge the gap between the division of

the internet and the the way here is that if con if there can't be consensus between the nodes across the world there are the assumption is that there's going to be at least a few nodes that can span that Gap and for all the nodes that cannot span that Gap they will get inactivity leaked and down until they are kicked out of the validator set and then the few remaining nodes that can span east and west or whatever division of the internet there is they will be the ones left standing who are not getting inactivity linked because they are the ones still propagating the blocks and that's the mechanism that that entered is that correct yeah that's that's right well at least we got to test the inactivity leak right guys there's a silver lining here I don't think we've ever tested that in the wild

and so this inactivity leak came into play for those who are staking with with what was it is it just prism and takio and Lighthouse kind of carried on was it just a sort of a subset of uh client users I think it's hard to tell I think every client had some issue at that time because of just High CPU usage so they're sending their attestations but their attestations may not be received may not be processed so I think like every single client unless you that you run your client on a very very powerful machine that that you're likely Break Even or you would lose a few cents I got it okay so what happened that that's the

view of sort of from the from the client in high level blockchain perspective we've seen that what about the average user were what happened during this time were they able to get transactions through or what happens to users using the using the network did they even notice so I guess put it this way right if today it wasn't for peach and chain Explorer if it wasn't for that side will we ever find out the issue maybe yes maybe not but I don't think we'll find out the issue at that time right we'll probably find out the issue like a few hours later when someone was using this finality notion before right so so with that it basically we can say that yeah it probably doesn't affect most people

if today you're um you're doing a swap on unit swap you're buying something that he's those still goes on but if today say today you will actually using finality which is a pretty rare set of application I would say maybe there too maybe like maybe like it's changed as a big amount of withdrawal then that will be effective but that's a kind of a small set of the use case I mean was acceptable no I don't think it's acceptable for ethereum to be this Global Internet of money which we cannot have this right but I'm really happy that I'm really happy that liveness still goes on because that's probably like I would say close to 95

percent of the use case the transactions were going through completely it wasn't as if there was a pause in in sort of user transactions it was just this one finality piece that some sort of you called them settlement applications might use it might be a layer two or it's an exchange maybe doing a big batch process with a lot of money and they they want to wait until full finality is achieved before they before they settle it those are the kind of apps that were maybe affected even though that's even hard to see because this is so brief but users uh weren't affected at all like the regular users and liveness was preserved through this event um Preston looked like you were going to

add something to this yeah I wanted to add that the real impact to for the end users was that there was a diminished amount of blocked space right so when we're packing transactions into blocks what drives the gas price of ethereum is the demand for Block space and the availability so I did notice that at least on Thursday gas prices were slightly higher than the daily average it was something like 120 gray at least on the epoch that I checked which you know in terms of the scale of ethereum gas pricing is kind of a negligible increase maybe it was a hundred Epoch before now it's 120 we've

seen a lot of you know dapps push the gas price much higher for different reasons so it didn't seem like anyone really would have noticed right it wasn't astronomically high it's just slightly higher than average during that time due to less available block space for transactions but everything was still going through pretty in a timely fashion you may have noticed your your transaction took um up to a minute but where it normally takes you know 12 seconds but that was really really it you didn't really feel that it was down or broken in any way so in that case what's the big deal here like why do we care about finality what what would what have been what could have been a worse condition what what

could we play out a scenario where maybe this happens again and then something actually bad happens like what's the big worry here well the concept of finality in proof of stake is is one of the main features right it means that we have finality in the blockchain and that we can be assured that that point in time is is final and can never be um Rewritten or we cannot go back in time build on a prior block and sort of exclude this history that's happened uh as Terence was saying that's very important for some transactions so if you're doing maybe a very large purchase let's just say you know we're at a point

where you can purchase property in the real world you say I want to close on this property I'm going to send you my eth and then I'm or whatever it might be my tokens and I'm not and we're not going to consider done until there's finality right so there there is that aspect where we want it to be final cannot be Rewritten it's also part of the entire security of ethereum right it if if ethereum aims to be the global settlement layer you need to have this consistent finality in in a timely fashion and it can also affect you know layer twos or other networks that are leveraging ethereum security I was imagining the scenario or what if

this played out for a very long time right we have optimistic Roll-Ups out there with the a seven day I think a seven day window for the challenge period so you can imagine that if this incident went longer than seven days what does that mean for the challenge period even though it has resolved it's not entirely final because it's not final until it's finalized and I thought about that as being sort of like the the medium term worst case I mean the the obvious worst case this goes on longer and longer until we have so much inactivity leaked that a lot of validators are rejected eventually ethereum recovers uh l2s would recover and those challenges would eventually be finalized uh but you could see that would be a

major disruption if it went on longer than seven days where can you like put us into the headspace of Preston when this was happening like were you freaking out uh where you're like oh that's a curiosity uh and what just like sometimes um yeah we can talk about like the technical details of uh finality but what about the emotional response like how I can give you guys the response of people on Twitter which was mass panic freak out oh my God uh ethereum is broken that was the response on Twitter but yeah yeah so this is what happened on Thursday it was around like 4 P.M my time and I'd been starting my day very early and I was like I'm gonna go I'm

gonna go take a nap like I'm kind of tired right and then I just checked right before I go to make sure nobody needs me for anything and I see this message I never want to see again from Nishant he's on our team Nishant Das he sent me these a couple times he says maintenance stopped finalizing and I said oh my God what there's no way I'm gonna be able to sleep for days you know because this three words those three words about finalizing and this is how I found out about it the first time so this isn't the first time Maine has had a finality um delay or finality incident happened a few years ago where prison was unable to produce blocks and prison was a super majority thanks to client

diversity or not uh have that risk anymore but the same message he sent me he said for uh maina stop finalizing and at that time it took us I think three days to figure it out to figure out what the issue was and we were sleeping in rotating shifts like maybe a few hours at a time so I when I thought okay I'm gonna go take a nap and then he said those three words I said oh great I'm just never gonna sleep again so emotionally that was that was challenging well uh I think this uh does a very good job illustrating why client teams are so important and why client team maintainers are heroes that should wear capes uh so Preston Terence thank you for for always wearing a cape even

though uh you don't actually legitimately literally wear it wear a cape there's a lot more that I want to ask about uh but first we have to talk about some of these fantastic sponsors that make this show possible especially metamask if we have used jargon in this episode that is confusing to you I bet you metamask learn has an explainer for you on your product there is a link in the shout outs here we go learning about crypto is hard until now introducing metamask learn an open educational platform about crypto web 3 self-custody wallet management and all the other topics needed to onboard people into this crazy world of crypto metamask learn is an interactive platform with each lesson offering a simulation for the task at hand giving you actual practical experience for navigating web 3. the purpose of metamask learn is to

teach people the basics of self-custody and wallet Security in a safe environment and while metamask learn always takes the time to Define web 3 specific vocabulary it is still a jargon free experience for the crypto curious user friendly not scary metamask learn is available in 10 languages with more to be added soon and it's meant to cater to a global web3 audience so are you tired of having to explain crypto Concepts to your friends go to learn.menimask.io and add metamask learn to your guides to get onboarded into the world of web 3. hiring people worldwide paying them in crypto providing them access to benefits it also come complex but it doesn't have to be complying with labor laws payroll rules tax obligations and crypto regulations in every country

that you employ someone is difficult time consuming manual and costly and it's drawing more and more attention from regulators and governments but there is good news toku is here toku is the first employment and compensation platform for the crypto industry that makes this easy toku helps you hire employees or contractors and pay them in Fiat or crypto legally compliantly and with all the taxes handled and over a hundred different jurisdictions so whether you're an early stage company with just a team of two or you're an Enterprise with 200 toku has a solution that meets your needs toku is already working with the leading companies in the space protocol Labs hedera get coin and many more so transform your employment and token payroll operations with toku you can reach out to toku at

toku.combanklist or click the link in the show notes introducing ethx from stator ethx is a liquid staking token designed to maximize rewards all while securing ethereum with stator you can run in an ethereum node with just four eth an 85 lower Capital requirement versus the 32 eth required for solo stake with Staters four eth nodes you can get a 35 average higher yield since you charge fees to those who use your node to stake their eth by running a node with stator the ethx staking derivative token can get minted on your validators and can flow into the world of D5 which data is actively building Integrations and Partnerships into to increase the utility of eth X stator allows for both permissioned and permissionless nodes to

join the network maximizing its potential scalability for ethics while preserving the values of decentralization and openness behind its liquid staking token go to statorlabs.com eth and sign up to get access to the stator staking protocol arbitrim 1 is pioneering the world of secure ethereum scalability and is continuing to accelerate the web 3 landscape hundreds of projects have already deployed on arbitrum 1 producing flourishing defy and nft ecosystems with a recent addition of arbitrum Nova gaming and social adapts like Reddit are also now calling arbitrim home both arbitrum 1 and Nova leverage the security and decentralization of ethereum and provide a builder experience that's intuitive familiar and fully evm compatible on arbitrum both

Builders and users will experience faster transaction speeds with significantly lower gas fees with arboretum's recent migration to arborstone Nitro it's also now 10 times faster than before visit arbitrim.io where you can join the community dive into the developer docs Bridge your assets and start building your first app with arbitrum experience web 3 development the way it was meant to be secure fast cheap and friction free and we're back now Terence I'd like to turn to you uh what was your story what what is it like from your perspective to hear the words mainnet stopped finalizing as a client team death yeah Stephanie not great hearing that on Thursday afternoon typically I stop all my meetings after Thursday so I was

excited to get the coding have some personal time read some paper but um so I yeah so like President did I heard from the news I heard from Telegram and to lose 65 of participation it's it's a little crazy like when I first read that I just couldn't believe that I was like just how would that happen right it's like imagine like 65 of the no just disappear right it has to be like this sounds very like catastrophic if you really think about it so in my mind I just I started piecing everything together quickly right because like 65 then I know it's not just one client just like thanks to climb diversity like every client is

pretty spread even my president has sorry uh prison is 33 Lighthouse has like 33 so I know it's not just one client so it's not a client bug then I felt okay maybe it's like uh maybe it's like a target attack right and then maybe it's like they're already like someone is attacking the network finally someone's actually taking a network and then but likely like the whole thing like went back to normal after like three to four Epoch which is like 20 minutes so they kind of give us more time to think about what the issues is right and then I thought okay well if it's a target attack like why did it stop if you I mean if they could have done for longer like why stop there so I so I didn't really think it was a target

attack at that point then I thought okay maybe there's like a really big exchange or staking pool that runs something on a gcp and AWS and then they shot down or something like that so that was another reason but I'd like to the I think the whole thing took about like four to five hours until we came to the Giza gist of it that's basically our software was a little bit um under of under optimized so how hard of a problem was this to fix as soon as so it sounds like diagnosing wasn't too difficult but what was the actual solution process like it wasn't hard to fix we could have

fixed the problem within the first 30 minutes but we kind of wanted to do the perfect fix right because everything is straight off client software is a trade-off if you optimize one thing you cannot lose another thing and this is why Clan diversity is nice right so I'm gonna shout out client diversity again just lighthouse what fine because of the trade-off they they made right today network is under another scenario Lighthouse may underperform very like versus prison because everything is a trade-off here right so we kind of we kind of picked the face that we thought had the right trade-off that where we like and then again this is a team effort like like

probably like five to ten of us like we got together we talked about it and it was like okay this is the fix that we want we also share our face our idea with five other awesome client teams as well right and then because we're all learning from each other we are all study each other's design so yeah one interesting observation I have about this is this is um this is sort of a finality event but it was but it's also like a non-event right like uh when we asked the question at the beginning of this episode Preston um did ethereum break and you were just emphatically no it didn't break and so this is maybe a demonstration of the resilience of the

ethereum network and I I what came into play like how did the ethereum network show resilience during this event like that Terence you just talked about multiple clients was was that the key thing is that the thing that that saved our ass I guess I would say or are there other elements that demonstrate the the resilience of ethereum first to uterence that's one thing right if today it was three years ago when prison was the majority client I will be really really worried right but today we're just like a piece of the pie and uh I'm uh so therefore I feel like I can sleep a lot better the second thing is that we really need to thank like this

um this thing is called lmd ghost this is designed by bitadik and a few breathing researchers from the ethereum foundation so lmd goes is this consensus mechanism is this Fortress rule that we use for E2 or or the chinchen basically this is so essentially uh the beacon chain ethereum proofers taking senses is made up two part the first part is lmd goes essentially it favors liveness it basically ensure today if we don't have safety we don't have finality the blockchain keeps growing and this is a very important right as it turns out because before I I always question why do we need this why can't we just have

safety that's it what can we be like a tandemic channel what can we be like maybe like the other blockchain they just have safety right and this is literally the reason this this is why because like if things go wrong if we don't have penalty we don't want to affect the use we don't want to affect the ux right we want people still be able to buy another piece they buy coffee they ultimate trade so essentially to answer your question this lmd goes Fortress rule was the one that saved our blood it prioritized liveness that's what it did specifically Preston what would you add to that what saved us yeah I think Terence said it well it's the specification of ethereum lnd ghost and these things that make ethereum uh

the way it is combined with client diversity right like the specification worked here and when there's an incident it can recover it can live on and because there was an incident and we're having client diversity it was minimized to a subset of the network right it could have been worse if it was more of a majority prism it brings me back to the days when we had these early test Nets like madasha it which was 100 prism and it was just on fire it was it was terrible when there were incidents and you know thankfully client diversity we've done a really great job over the last two years or three years however long to smooth that out and we could see it here today that there was an incident

and it recovered on its own without intervention within minutes and that we were able to release mitigations and hot fixes and client updates within days to say okay now it's not going to happen again so I think that went really well it's a great test in Maine so I'm going to pull in uh something from my bitcoiner world um they'll always love to say this is good for Bitcoin uh and the spirit of that that response is that anti-fragile cryptoeconomic systems what doesn't kill them makes them stronger so Preston um was this good for ethereum yeah I think you know like we're saying this is the first time we've seen it in

mainnet in production we knew it would work we've done it in testnets but I actually see it in in action um it just goes to show the resiliency that when things are going horribly wrong it can still be okay and it's still um eventually finalizes and that the mechanisms and the specification and the design of ethereum proof of stake is resilient so I thought that was great so like ethereum is an evolving protocol as we know right it is still constantly heavy under heavy r d right and then we definitely came out here with like a very better understanding on just like very little detail of things that we

want to improve of even like future research spec as well just I heard your podcast on um MVP boom right that's a very good podcast but because of this right we're kind of looking at Mev brand we look at PBS to see how we can do better there as well because everything is interconnected like they all they basically all touch each other and I definitely feel like we definitely got out of this very strong and not only that but clients have discovered a weakness that is now removed correct yeah cool nice I love anti-federal systems my my last question is this and this it

goes back to uh Terence something that you were saying about kind of the the design of ethereum proof of stake with lmd ghosts that's like weighing the weeds for people but like the priorities that the ethereum network makes uh as we all know it's not high transactions per second on the base layer on mainnet right instead ethereum has made other trade-offs in one of those trade-offs is uh liveness aliveness trade-off and I want to just focus for a minute on um some other outages that have happened in crypto with other layer 1 blockchains and we don't have to name names it's like I don't want us to uh like anyone to feel like uh you know the ethereum's

punching down that kind of thing okay so like but like just just think of the first name that comes to your mind about a blockchain but but also just like just the facts okay so like um I think when uh people were talking about and by the way Bank listen included we're sorry uh everyone listening for the uh the Tweet headline all right we've made amends on that um that happened this week but when people equate um ethereum breaking to sort of the the downtime and the breaking or temporary halting of other blockchains it's much different so I'll just throw um the name Solana out there just just for a minute so we can talk about that as a just the facts kind of use case here um when Solana was down because it did not prioritize uh liveness when it had

some some issues it was actually offline right like you couldn't get a transaction through for a period of time I'm wondering if you guys could just just weigh in on that briefly because it is a useful contrast Point okay ethereum doesn't have the highest transactions per second on the main net but um it does have this feature of liveness Terence could you talk about that for a minute yep so I just wanted to add right like there's nothing like what we are doing today right we're working on open source software and this is the open network that means anyone can read any code that we write we're like chefs we're cooking in the kitchen right and anyone can join the network right so it makes you very like vulnerable to attack therefore

right therefore I think Feathering liveness is very important just because like you want your life to keep going right say today you have a country that's built on top of ethereum I want to buy coffee and then I I mean today ethereum is done cannot process transaction they I cannot buy my coffee then I will be very upset right so I would say likeness is a very important property probably the most important property if you want higher throughput there is Layer Two right and that's what their two is for but ethereum has a base there definitely liveness yeah you know I think that's what makes a blockchain great is to have liveness right it's not useful if you can't use

it so to have live news is I think the most one of the most important uh aspects of ethereum more so than transactions per second right we have solutions for that we can't abstract two layer two or or other clever mechanisms but if you don't have wideness then none of that works you'll have zero transactions per second so it is the most important aspect of ethereum beautiful parents yeah thanks for summarizing this thanks for jumping on the bankless podcast explaining this to the bankless community and the crypto Community uh did ethereum break the answer is no this was maybe whoever tweets that is dumb yes this is maybe ethereum's uh biggest liveness test yet

though and uh so it was good to come out on the other side and see a resilient network uh pretty unique I think something that's that's very special in the space and worth preserving and as always gotta thank you whenever we have client uh devs on on the bankless podcast we would be remiss David if if you and I did not express our thanks both to you Terence and to you Preston and also to any other client Dev that is doing this hard and essential work to build this incredible Freedom technology network uh that we care so much about thank you sincerely open source superheroes for you we appreciate you guys going on yeah it's our pleasure to get to work on

it really it is an exciting opportunity we know that ethereum is going to change the world and to be working on it at the protocol level there's nothing better that we could imagine doing so it's our pleasure to be here plus one to one person said Thank you for having us Chris and disclaimers guys uh of course gotta let you know ethereum is resilient but uh ethi asset is risky so is all of crypto so is D5 you could definitely lose what you put in but we are headed west this is the frontier it's not for everyone but we're glad you're with us on the bank list Journey thanks a lot