Testing The Merge - Parithosh Jayanthi

Transcript

[Music] so peritosh give me one second i'm gonna introduce you and then you can take it away um first up we have a good friend um from the ethereum foundation who is a good good friend to chainsafe and close collaborator with our lodestar ethereum consensus client team paratorsh is in fact in possession of a very exclusive chainsafe merch the black chainsaw hoodie not many get not many get a chance to get one but for peritosh we make sure to pull some strings and who knows one day if you contribute and help chainsafe out you too can don one of our super rare black hoodies i digress um paratush is a devops engineer over at

the ethereum foundation he's currently based in berlin germany where he completed his masters of science in communication engineering at the technical university of munich he got his break into the world of blockchain when he received a scholarship into consensus academy's developer program before joining slot.it which if some of you know your trivia had a dow that was originally involved with the contentious ethereum heart fork of 2016. but that was well before peritosis time no worries there um nowadays he serves the ethereum foundation faithfully by helping out with all things verge related and of course that is in reference to ethereum foundation's upcoming switch over to proof-of-stake consensus

as some of you may know the expected merge date is ephemeral and changes often but the recent rumors are september so naturally that piques my curiosity a little bit so in a continuation from last month's ef talk with marius i invited pari um again from the ef today to do another talk to give us a low down on the merge so without further ado i'd like to invite peritosh onto the screen uh unmute your microphone and take it away okay thank you so much for that awesome intro i really love my chainsaw footy i do not know why i didn't remember to wear it for the for the talk i just wore the ef one instead but um next time i will

definitely wear the chainsaw um yeah also fun fact i was wearing the chainsaw hoodie for um eat lisbon and everyone just assumed i was working at chainsafe but at some point i just looked like a piece of tape put it on there and then just wrote ethereum foundation instead because i had ef much but it just wasn't as good as the chainsafe much so i still wanted the chainsaw merch for the ef logo there you go it's super exclusive let me just show my screen um yes can you guys see this okay yes we can perfect awesome so today i'm going to be talking

to you guys about the merge hang on yeah so what is the merge to begin with it's the switch that ethereum is completing from proof of work to proof of stake so we already have a proof of stake chain but it's currently running in parallel to the proof of work chain and the merge aims to bring these two together what's so complicated about it well there's about 20 different client combinations we have about four execution layout clients that's the each one notes that people know and love and we have about five consensus layer clients of which

load stars won um as tim mentioned so this means that we end up with 20 different kind combinations and regressions can sneak in very very easily um all the interfaces are really well defined but in aims to fix the spec implementation with one kind it's relatively often that something else might break so we need to be really careful with testing to make sure that this doesn't happen and honestly we've tested all 20 clan combinations so just choose any one of them that help decentralize the network you don't have to use a majority client you can choose both minority clients we've tested them all equally

uh communicating and debugging becomes extremely difficult and this would tie into the last point i have that competences are really different we have the proof-of-work world with the execution layer clients that have been debugging and securing the ethereum network for years and then we have the proof of stake world where they've been um live on mainnet for like a year and a half almost two years now it's not extremely easy for the two to debug the others clients especially when we're adding that dependency right now so we've actually spent a lot of time on all of these test nets to make sure that there's multiple people from each client side

on the table so we at least know what the other side expects so if the er teams notice an issue they make sure to capture some of the information that they think the cl teams might require and then pass it on so that's actually been a lot of the complications relating to debugging the merge that we didn't really expect beforehand the second thing is how to help you test all of these right up until now we just had spec tests on the cl world and we had hive tests on the el world each client team could do whatever type of testing they wanted i'll go into what those two words actually mean later on um but we needed a way to test all of it together we needed a way to test it in the in a manner that we can

reuse this when it comes to sharding when it comes to um prototype sharding or when it comes to state pruning what workable trees whatever it is we wanted to build it such that we built it properly now the testing infrastructure and the testing framework such that we can keep using it we don't need to redo all of this work we're already inheriting a lot of complexity we might as well do it right so a really high level view of um of what the cross client testing infrastructure looks like so this just to note does not take into picture what the client teams are testing themselves this is just the testing that happens across client teams apart from this each and

every client team has their own testing division or their own devops people or their own ci whatever that they have or their own local test nets um all of that is completely separate from what what i'm gonna talk about today what i'm gonna talk about today is just what's common in testing infrastructure across all the client teams and i've tried to get them same names um so hopefully it's a bit easier to follow what's happening so i've tried to also visualize it as a circle i'm going to tell you right now it's never that clean um it's never that one of the tests happened before the previous one that's just going to be way too much um

time being spent on waiting for the previous step to be real almost all of these are happening at parallel all the time and it's a lot more chaotic but i've tried to bring some semblance of logic to how the testing infrastructure looks so the spec tests are typically where we're going to begin the cl world is really well defined in the spec implementations um so you can actually the entire um cl spec is actually python executable so you could just build your client and execute it against the spec and it should technically work and that should technically work is also what we test so what we do is on every commit to the ci spec to the consensus

layer spec repo we run a bunch of tests we make sure that um all the tests are passing and this is just a sanity check to assure that we're not changing the spec in our manner that the clients themselves cannot implement it um the client teams themselves would have their own ci that then post the latest version of the spec and test their client against that version of the spec so those are like the two asynchronous things that happen but the client ci would typically run probably before releases whereas the specters we've migrated to doing it nightly as well as on almost every commit npr by now i think the next step is hype tests so hive tests this there's two versions we've

got the hive test in head test tool uh the hive test to have also involved the consensus layer nodes by now but it's essentially a simulator that starts up the clients and runs tests against a predefined interface so the clients did have to make some minor modifications to define these interfaces but we have a couple hundred test cities and we've made this pretty much a minimum requirement before we could have have the much happen these define extreme edge cases for example propagate a block have the merge transition invalidate something and trigger some weird edge case and at the same time the network should still be fine

um this is run for client basis of course and it's defined in different suites so you have some group of tests for the engine api some group of tests for the transition and each of them work in different manners this would fall under the class of integration and regression testing and this pretty much runs continuously so the idea is that if you update your um el in such a manner that it's failing one of the edge cases then you know you probably shouldn't be making a release now that we've looked at the integration side of things let's assume that clients have all made the releases now we want to make sure that all of

these individual client releases do work with each other so we use this test tool called kurtosis that spins up a local test net with a predefined like it's just json config where you can say cool so i want 10 nodes of load star with nethermine and five nodes of prism with gef and i want them to spin up on this one machine go through the transition and once the transition is done make sure that finality is achieved make sure that the base fee is a certain number or make sure that there's transactions or make sure that whatever it is we we have like a verification

file that's also provided over there so it asserts that all of those things are true and if it's true then we consider that the test net is a success but as you can imagine this is really just a happy case there's nothing bad happening here it's just spinning up a fresh test net it's going through the transition and it's making sure that once the transition is done it's okay so we kind of do this in two ways we have one consensus layer node along with one execution layer node as a majority so you have a chain that's just gets lighthouse or just load star lighthouse to make sure that we're eliminating cases where there might be incompatibilities between clients and then we have multi-client test nets as

well we typically just upload the logs and once the logs are up there if you notice something's wrong client teams can just copy out the json file and rerun kurtosis locally to try and trigger the same issue again so that's also the nice thing about having these json files that just define all the how the test needs to look because if there's an issue we can just give someone doesn't say run this it ideally should be reproducible and that happens once releases are done right the other thing that's really important to make sure once releases are done is that we can actually sync a real network so we have a sync test coordinator um and what the sync test coordinator does

is it spins up again every one of those 20 client pairs and syncs uh one of your favorite test nets so either robsten or sepoya or kiln um just we're just thinking test nets because it's lighter and make sure that they reach uh both the consensus layer as well as the execution reach um head and the reason we need to make sure that both of them reach head is post merge the consensus layer uh so the execution layer depends on the consensus layer to know what head is to know what to sing the sync target so there is the sort of dependency between the two so we need to make sure that sync isn't broken um but again this is just a happy case

we've also defined some edge cases over there and those edge cases are say your eth1 or your execution layer node is synced ahead and for whatever reason you need to you want to change your beaker node from keku to prism right so suddenly your prism node is sinking from genesis you need to make sure that the two are in different states and yet they can both continue as functioned so we've defined all of these edge cases as well and we make sure to assert that all of them pass again this looks pretty much like an integration test and it's just to ensure that we're not making a big release that suddenly no one can sync the network anymore

and this is kind of the meaty part so till here we've been looking at like ci nightly happens in the background sort of tests the next set are test nets and these are more active um these require client teams to participate these require someone to set them up um we've kind of divided the test net phase into three we have devnets that are just small scale expectations that client teams take part um probably like 20 nodes or something it's fine if a client pair doesn't take part whatever um the main reason is those are lightweight and we just want to make sure that things go fine there we actually haven't really needed to do a devnet since like

december because at some point no one wants them anymore they're too easy um that's why we moved to bigger tests so bigger test nets have more public fanfare we asked the general community to run validators there's just more transaction load uh the example of those would be like the kinsuki and kiln test nets that you might have heard of the merge test nets um fun fact killing test net even had i think a three or four day period where it was processing more transactions than main net because i think there was an nft drop or something expected um so it was it was insane our faucet was completely drained the explorer couldn't keep up with the load

because we never expected anything like that to happen um but yeah like a good portion of time where we were just doing handling more transactions would that mean that even um yeah so we haven't done a fresh testnet since kiln that's about four months ago i think time flies uh since then we switched to another form of testing called shadow fox and an example of what the shadow fork looks like is present on the right um so we have the regular chain example of girly and we take the girly config and we add uh we modify it to include the terminal total difficulty that's when the nodes

know when the merge is supposed to happen so what happens is when that condition is triggered these nodes sort of fork off from the traditional gurley network and they start forking into so they basically merge into a beacon chain that we spun up so it's just a fresh beacon chain but because we haven't unpaired from the old nodes and we're still connected by gossip we still receive transactions and because we still receive transactions we're still touching state we're still producing blocks we're replaying everything so it kind of inherits all the stresses that girly has but tests the benefits that the merge would look like

so it was a cool idea that mario's had at some point in jen and we tried it out like oh cool this is actually a perfect way to test stuff and i i don't even know how many dozens of bugs have fallen out because of this and how many assumptions were wrong we were extremely off about how long it takes to build a block we were quite off about how networks behave when there's multiple terminal blocks being proposed we were off about i don't know there were like a good two or three different things that we were off about but the nice thing is we've we fixed all of it now um and you kind of get to see how the test net merge would look had it have happened in the background

so we've had i think like five or six of these girly shadow forks um and since then we've migrated just doing midnight shadow forks all together because main net state is massive you inherit the weird transactions that we probably wouldn't think of that maynet has and we're testing on a wider scale network that's actually something that has been really really fruitful over the past year um this as you can see would trigger real world edge cases it allows sometimes we actually do involve external participants on the shadow fox as well especially if someone has a specific thing they want to test

if they want to test one of their setups if they want to test their backups whatever it is we just create a bunch of validators for them and add them in there one example would be we're testing mev boost on some of the shadow forks so we want to make sure that um mav boost works as expected when it hits prime time so to say and the last in the cycle is antithesis so antithesis is a company that we work with that offers a deterministic hypervisor what this means is you're able to translate you're able to go forward and backward in every single step every single execution step that the

client would have this means we can the setup allows us to create like a network split or simulate packet loss at the same time fuzzing the client at the same time if we notice a crash we can capture the the state at that crash and we're able to trace it backwards figure out what packet triggered it figured out what state the client was in and then sort of just it's like having like an interactive debugger at that point um this as you can imagine helps us get like really really edge cases hkc scenarios like the network was split into four parts with 20 packet loss and at the same time there

were like four terminal blocks noticed and this node was dropping i don't know couldn't manage it to find its disk how does this network then heal after after a while um the whole idea is that the fuzzer is able to electric a bunch of the edge cases and figure out whatever needs to be figured out i do think we found a couple of api related issues there we found quite i don't know i think there's a list of like a dozen or two dozen issues we've been able to find because of this and a couple not all of them are even merge related um that's actually a really nice thing about the merge efforts it's one of the first times that all of

these clients are being looked at with such scrutiny that we've been able to fix bugs that have just been dormant for potentially like months or years and ethereum nodes but just because we're doing that level of testing now we've been able to find them um and finding bugs and antithesis especially in these edge cases help us reevaluate if we need to make any major changes in the spec or if it's just a bug in an implementation then we're happy about it and we fix that implementation and we're good to go um just to give you some numbers to date we've had i think like six devnets mostly internal we've had three public test nets five golly shadow forks ten minute shadow fox

and robson and sepoyer have both successfully merged so that's uh i wanted to compile how many validators that is and how many machines that is or whatever i wasn't able to get those exact numbers for you right now but hopefully in a future version of the talk i'd be able to um but that being said if that's the testing plan so far uh where are we we're kind of i think finishing up or or mid quarter three of 2022 we have one test night left to go uh the test that's scheduled to merge i think between the 9th and 12th of august so that's going to be girly update your notes the ef blog post announcing everything should be out i

think the next day um once scully is done we're gonna have probably an off cycle all cardaps to decide the terminal total difficulty and when minutes gonna happen so it's getting real we're really really close there's just like uh i i think the way the eat sticker community defines it four steps left so you know three steps left so the girly merge bellatrix on mainnet and the main net merge so those are just the three steps left out of i think six when we started in may so yeah how can you guys help with the efforts so if you notice i've been using

the hashtag testing merge um we do have a list of different tests that people could run either running your validator in a weird manner or whatever it is there's a list of like 20 different um topics in the document a lot of them aren't even to do with just testing it's also we need better ux in terms of knowing when the merge has happened or we need some tooling we do have like a bunch of links as to hey it would be great if someone could build a tool to i don't know figure out which block number corresponds to which slot number because you always have to do that math manually right now um or there's been a lot of tooling requests

around traces if i'm not wrong um and just have document stuff make stuff easier to run at least the last couple of shadow folks have shown our last couple of test networks even have shown us that right now we've tested stuff so much that the typical failure scenario someone just forgot to configure something correctly so figuring out how to tell people to configure stuff is kind of the long tail of the problem to solve right now and if you guys have any questions about anything a bunch of us are available on twitter happy to answer each r d discourage um eat sticker discord cat heard us discord server it's a bunch of places where a lot of us are actively answering

questions for you and that's about for me does anyone have some questions amazing thank you paratorsh um you guys are doing god's work um with testing emerge and making sure ethereum successfully moves over to proof-of-stake um if anyone has any questions please feel free to unmute your mic or ask it or you know just type it into the chat box emerge um i guess i have a kind of a non-technical dumb question for you like what's

at this point like if you were to assign a percentage for like a successful merge like what like what's the percentage or can you say anything other than 100 i mean i think 100 is really hard to achieve because at least every fork has shown that there is someone who will forget to update their nodes there is someone who wouldn't read the documentation and to be fair if that leads to network participation of i think 80 percent for a while before they figure out their setup i'm okay with it i'd consider that 100 success um and i i also don't think that's completely unlikely right like someone will at some point forget to update

something or they haven't checked whatever but the nice thing about mainnet is if they forget there's an incentive to fix it as soon as possible because they're missing out on rewards they're missing out on transaction fees um they hopefully have notifications set up so i'm quite confident that things are going to go well that's amazing uh yeah so with all of the client um diversity testing you guys are doing um i'm curious i'm sure you're impartial or not partial uh for sure but i'm wondering if there are any like uh caveats like strong caveats with running any particular combination or any um specific client that you guys have encountered that's something that we're

thinking about um you know because we're not gonna start with all of them right so which one could be more troublesome to uh to operate yeah um so i don't know how much light has been put on it but there's a the barcelona supercomputer uh group i think um the there's a group called amiga lab over there uh and they do a lot of testing in terms of different client types and they did the tests in terms of to figure out who you are and what client type might make sense for you so they kind of enumerate okay these are the type of rewards you get if you use this client pair or these are the resource requirements you have or these are the

um i don't know this throughput you need if you choose these client pairs so based on what your requirement is you could pick out a client plan from that so if you're someone who's like i have a ton of money i don't care i'm just going to choose the client that earns me the most money then you could look at those docs and figure out which the client that earns you the most money is and choose that or if you're someone who's like i'm in a resource constraint environment i need to figure out the one that's using the least bandwidth i don't care about the hardware itself but my home internet's too slow then you could also use that guide to figure that out they they i think it's a medium post they they do a great job about that right on right on i'm sure the

winklevoss twins will definitely appreciate the resource constraint approach uh but yeah need the labs yes definitely here's another uh dumb non-technical question for you what's been which team has been your favorite team to work with i'm just kidding i'm just kidding they've all been amazing like what's been the most challenging thing about like like trying to wrangle like because you i think you mentioned earlier in your presentation like there's 20 clients that you have to test for like so like what what has been the most challenging thing like how do you like you know what what are some learnings you've taken away from it

i think having to do this in a decentralized and distributed manner has been one of the hardest parts because these client like 20 client teams none of them are almost none of them in the same time zone so trying to figure out how the hell you coordinate all of them to get on the same page about a lot of things especially when you add the complexity that we've been divided into like two groups when there's an issue you need to figure out someone from the el side who's gonna help out someone from the cl side who's gonna help out and at the same time you don't want to just rely on that crutch forever you want to slowly try and make it such that both the teams also know how to debug

some of the other side um i think that's been really hard but we did have like an in-depth session about it during uh depth connect where we we had like a list of things and we weren't even like create like a basic run book cool if a cl team notices that this is what's off they of course already know what to fetch from the cl front like it's an invalid block try and dump fork choice or try to try and jump where the attestation votes have been so on they know all of that stuff but a list of things that the elders would like and typically that's around the topic of traces or dump the block and stuff like that or vice versa if you're an el team who's at the incident like the first responders right what's

the checklist of things you have to do to make sure that whenever the other team is out there they have all the things they need to start i think that's been really challenging to try and achieve right um yeah i know i feel like that's definitely why you know like things like the interop events are so helpful too right when you get everyone in the same room and you just get so much more done um versus like that decentralized um thing but cool uh anyone else have any questions um dan i don't know you you work on a alternative client for polka dot like is there some right that's cool client testing question that you have sorry to put you on the spot oh that's actually a nice uh nice segue as well so all the stuff that i've

mentioned today or almost all the stuff that i've mentioned today we've developed it in such a way that it's super easy for anyone to fork and reuse for whatever networks they have um i don't think i have it in this presentation but uh the mobile network guys use some of our testing tool the gnosis chain guys are using i think all of our testing tools which is awesome like one day i woke up to like 30 prs on one of my repos because the previous day i was bitching about a couple of issues i had and they were like oh you know we're using all of your stuff so we might as well do you the favor of fixing all of it that's been awesome um and i think the i don't know there's like a bunch uh we actually reuse some of the stuff for the

vertical test nets um vocal testing and vocal test nets so there's like a lot of stuff that's reusable though excellent it's good to hear um i don't know if anyone else wants to like chime in with a comment or two um here if not then oh dan i see you unmuted ben also yeah cool question of mine for testing um just for the more technical tests like the merge hopefully in a month and a half um if like someone like myself were to commit to trying to get involved to test how much of a learning curve is it is it going to

be like three weeks just to get up to speed and now i can actually start helping like crap like the merchant three weeks ago anyway um you know or like how you can just like start today or someone smarter than me just start today and actually start helping i think some of the tests are really easy to pick up so if i was to list it in order of complexity right i think the hive tests probably going to take a couple weeks before you're up to speed and might not make sense to start right now if you want to contribute to those but on the other side of things like testing the merge we have like this list of different issues that you could take up or there's a there's a github issue on the ethereum pm repo as to nice to have tools that we'd like to

someone to develop right and i think those are really low hanging fruit it's just no one's had time to do that so i think we have like the spectrum wherever you fall on that perfect um okay if no one else has any questions carrie like thank you so much for taking the time out today to present um really appreciate it you are you're one of them keep it up let's get the merge done yeah it's gonna happen soon just throwing up that excitement it's exciting but yeah thank you very much feel free to stick around