Galen Marchetti - Spin up a multi-client, full featured private testnet for dev and testing purposes

Transcript

presentation is designed to be uh like a follow along Workshop uh so uh it will make the most amount of sense if you have your laptops out and are following along with us we originally intended for you to actually install cryosis and run everything on your laptop but never know if the internet is good enough and the amount of Docker images you would have to download is about 6 gabt so decided to do everything on git pod so that's how we'll do it um so you'll just need a laptop and and a GitHub account uh to actually walk through everything so welcome to our Workshop uh it's all about custom ethereum devn Nets uh for consensus and validation testing the workshop agenda first we're

just going to run a very basic test net uh it's going to have one full node only it's going to be a GU and Lighthouse pairing um and then we're going to move on to something a little more complicated uh with three uh three full nodes we're going to change the client uh pairing between the El and the CL layers um and we're going to show how we can have 128 uh different validators because each node uh can have multiple validators uh and uh then we're going to show you what it looks like to run with full uh PBS emulation we're going to run the full me flash Bots uh infrastructure set and while we're doing that I will show you how it looks for kubernetes you're not actually going to set up your

own kubernetes cluster in this demo but I'll show you that ptosis you can run it over Docker which is what most of these things are over kubernetes as well uh without having to worry too much about changing anything so like I said before what you're going to need is just a laptop and internet we have the internet hopefully you're connected to the the event internet um and a GitHub account so to start um we'll run this via a git pod um instance so the URL is here it's a short url. lowercase GJ SZ and then uppercase q um you can run

into that that git pod uh instance to open up ketosis we also have uh a bunch of kosis team members around the audience uh to help out um if anything is going wrong going poorly uh just yell at them and they'll they'll handle it uh so I'll leave this up here again and repeat it real quick for people who might not be able to to see it too well up here uh it's short url. slash lowercase G j s z uppercase q give a little bit show you what it looks like here

inside of that short URL great right you should see something like this um it might prompt you if you want to submit your email address you don't just if you hit enter with nothing it'll be fine uh that's optional let's see what I'm my connection is working here there we go okay does is anybody still

waiting for a screen that looks something like this to pop up got one cool the very first thing we're going to do is we're going to run the it's called the ethereum package um the command is going to look like kosis run github.com kosis DTE ethereum D package it's just going to look like that I'll try to zoom I'm going to zoom in a little bit here let's see if it

works there we go it looks something like this kosis run github.com crosis ethereum package so the way cryosis works is environment definitions are in things called packages and a package kind of contains all of the setup logic that's involved for spinning up in this case an ethereum test net and with kosis you just point it to a GitHub locator and it's going to pull everything it needs from that GitHub locator uh to spin up what you need so when we run this interesting I lost my cetosis let me see

SEC hopefully that didn't happen for anybody else let me I'm going to blow away my my workspace real quick and then do a new one apologies so to repeat that command here it looks like assuming your workspace worked and mine didn't uh kosis run github.com kosis

teum D package is anyone attempting to run this and not able to do it or hitting problems awesome apologize while you're doing that I'm going to keep reinitializing my own git pod here now what you should get out of uh running this command is you'll see an output that kind of lists all the things that cryosis is doing it's spinning up

uh an e client a CL client a bunch of helper Services it's putting the Genesis data where it needs to go it's getting you the validator keys you need to to get it's going to do all that stuff and at the end you're going to see a bunch of uh an output that uh tells you all of the stuff that's available and you should see if you're in git pod you should see a list of URLs um and the URLs are are Port exposures so any relevant application ports um including Explorer front ends and all that stuff they're going to be available to you on your local machine if you're in git pod you're going to use the git pod port forwarding and it's going to be it's going to work fine and I think I am almost back on track

here yes okay all right hopefully this one's going to work so you can see it live on

screen here all right so you can see it's downloading all of the um ethereum uh uh relevant images it's going to need to download uh e panda Ops python web 3 uh you have some uh observability tooling um and some spammer tooling spammer what I mean is like transaction fuzzing and like sending uh activity through the network that's kind of just basic activity so you have some degree of transactions to work with grafana Prometheus and uh a couple different explorers this uh the entire package that we're running by the way on GitHub is an open source

asset um I'll show you where it is on GitHub it's uh it's it's an open source asset that is uh jointly maintained between uh I'll show you between curtosis and uh a couple of the folks uh in the ethereum foundation uh Pari Barnabas PK 910 Sam cm and we have contributors from uh across uh client teams and and different block builders that have that have contributed to this so uh it's a community effort that's uh still growing and um as you as you get comfortable with it you can also become uh one of the contributors or uh maintainers so back to uh my cetosis

that's running now with the default configuration um it's as it prints it's printing out all the stuff that it's doing it's showing me exactly what cryosis is adding as it's adding and then here's the view that I wanted to show you earlier you can see that I have uh Port bindings exposed everywhere um I have a CL uh uh Lighthouse gu um service and that what that is is that's a you know represents a lighthouse node um and then I have the Val the beacon chain kind of part of Lighthouse and then I have the validator part of Lighthouse and then I have the GU node and the RPC ports are exposed and I can show you that Dora uh which is a a lightweight Explorer comes with the whole system and you can click into Dora and you have a

nice Explorer for your test net um we are currently pre Genesis you can see that the Genesis is going to happen in 34 seconds here um so we're not going to see much activity yet um but once Genesis happens you're going to start to see slots come through and you will also be able to go to your validators uh screen and you'll be able to see all of the validators that you've deployed in your network um there's one node running here but each node has uh 64 uh validator Keys tied to it and that's why you see that's why you see all of these uh all of these entries here

yes great does this make sense so far is there anybody still working on this step and if not we can go to something a little more fun Giani cool what we'll do in the next step is and this is one of the the big benefits of of using this package that that we've built is you can pass in a highlevel Json configuration and that Json configuration will specify things like the elcl client pairings you have in your network how many instances of each elcl client pairing you can also specify additional Services um like more advanced explorers or um or like the the

me infrastructure um so we'll show you some of that so to start doing that we're going to do something a little more complicated so if you can go to this uh GitHub URL it's github.com krosis text Summit and once you're in there you'll navigate to something called complex. um that complex. yo is is uh a starter configuration file for something that's going to be more exciting than what we just saw um and then once you have that open and on your machine uh or in your git pod uh then we're going to change it together a little bit so github.com oop sorry github.com

kosis SLS Summit and then go to the complex. and in just a second I'm going to move my screen over to go to GitHub to show you where that thing is cool.com sosis texts Summit and once we're in there we have complex. yaml and you can see uh a yam configuration that this is going to give us a different test net um and you can see that there is a GU Lighthouse pair a nethermind prism pair and a w teu pair in this test net so three different nodes and we also uh have a couple of different uh additional Services

specified here yes yeah hopefully that's good yeah it's going to be easiest to be able to copy and paste this because I'm going to move out of the screen just a little bit to show you what we're doing but let's let's make um in git pod let's take this and let's make a file let's make a file here called uh complex Ty AML keep it simple and then let's paste the contents of the file in GitHub into that complexo here cool now once you have that is

there anybody who's still getting to the spot great give a little pause and if you're if you're curious uh as we're as we're uh catching up if if you're curious about what the exact uh options here for thisam or this Json are if you go to the um ethereum package in GitHub you can see a full spec o where is it this in the editor here oh there's a typo let's see is it everywhere it is okay on line nine of

your ammo if you see net oor Network that's not a word it's going to be network uh Network prams yes thank you Ah that's why I didn't catch it got it got it cool network uh netw uh o Arc Rams it's going to work cool and now uh so with this as the base what I want to do is I want to edit this a little bit together um to make some changes so you can see what it's like to do a little bit of configuration on your ammo so on line eight um we're going to set this key oop we're going to

set this key ethereum metrics exporter enabled uh to true on line 10 uh will'll change the number of validator keys per node to 128 and then online Evan online 11 under additional Services uh we'll add the following We'll add full uncore Beacon chain undor Explorer and that's going to give you a more advanced Explorer than what we saw with Dora uh it's a little more heavyweight but um it's going to have a different set of visualizations that is going to be nice useful for you so I'm going to pop back and forth

now as I do this too so ethereum metrics exporter enabled going to change this one to true on line eight and then the next one is changing the number of validator keys per node to 128 so what I'll do is go to number of validator keys per node change this to 128 and then also under additional Services I'm going to add a new additional service and it's going to be called Full Beacon chain Explorer like

this make sure I got that right full uncore Beacon chain Explorer okay this makes sense and now we can go to our terminal this up and we can run uh cetosis run it's going to be github.com kosis Tech SL ethereum package and then we're going to have uh an additional command line argument-- ARS file and then you're just going to pass in complex .l which is the args file that you just uh just created and if you run that then ketosis

is going to take in those arguments and it's going to change the setup of your test net uh you don't have to stop the previous one they can run simultaneously side by side and you should be okay so it's going to download a couple of new images that it didn't have before because we've added new um El and cl clients we've added a new Explorer uh so

we're grabbing a couple of these images this is the main reason why we're doing it on git pod instead of on our own laptops because this would have been a nightmare over spotty internet to try to do all at once and it's going to do the same thing it did before the thing that it's printed out is called the plan every cetosis package creates a plan and the plan is a sequence of steps that are uh kind of the fundamental building blocks of spinning up an environment a multi-container environment um so that's what it's spinning out here and it's it's a bunch of stuff that handles the devops complexity involved in passing all the different values is involved in setting up one of these test notes between all the nodes so that they have

all the information they have to have in order to start up for

for great now we're starting to get the beacon train Explorer and while it's running I can uh I can give a shout out I believe the Dora the Dora explorer was built by um was built by PK 911 hey yes thank you for that he's in the audience we can give him thank you it's awesome appreciate it and then Beacon chain was was B chain no awesome uh so that's what we're going to check out here uh we're g to uh we're

going to get we're going to check out the Beacon chain this time the previous one we saw uh was was Dora which is still running it's great right we're almost to the end of this let see great okay let me just make it a

little smaller so everything fits into this spot okay so now we got a bunch of services um but as I mentioned we have a bunch of different elcl pairings here um and all of their most important ports are exposed to you in your local machine so that you can you can play with them um we have now uh a new set of uh exploration tooling available via the beacon chain Explorer and if we go to that front end uh you can see uh the beac chain UI as before you can check out your validators and you can see all the the public Keys uh in this view we don't see

um you know what type of valid they are like whether it's Lighthouse gu or uh or sorry Lighthouse or um uh uh gu pairing or like teu prison pairing like these things are going to be uh uh not immediately available in this in this view but you do have uh a different view for your transactions and for your mempool uh and your eox and Slots now uh it's very we're on slot n here apparently cool another thing that I wanted to show is that we have built in grafana and Prometheus dashboards in these um in these test Nets uh so if we pop into grafana I'll show you a bit of the dashboards the community has put

into um has put into the package so that you can get some some monitoring out of the box so we just pop into the dashboards General and I'll show you we have um we have the ethereum uh metrics exporter overview and it's split into real quick I Collapse these real quick it split into an overview the consensus layer and execution layer uh it's split into uh consensus spec overview consensus layer and execution layer um and as you dig into these um you get kind of the sets of monitoring that you would expect heads slot um Epoch I believe in the overview

we have memory usage and CPU usage um per per uh per node and you can use this to check what's happening um as you run your tests uh so that's the that's the ethereum metrics exporter overview it has aggregation of all these different validators together there's also another dashboard which is the single view uh where you can actually pick the pairing that you want to look at um and then when you're in that pairing uh you have a more in-depth view per node so just quickly to uh review what we just covered um when I

run um when I run the the test net I have a couple of things I can configure the elcl pairings I can configure how many of each pairing that I get and once it runs I have I have uh flexible selection of explorers I can use Dora the Explorer at the start I can use uh uh the the beacon chain Explorer which has a bunch of different services that are involved in getting the thing going but it has uh uh different views that I can use and then when I want to do my observability and my metrics uh style stuff I have graphon and I have a couple of dashboards uh that show me uh uh per views either on the aggregate or

per uh El or per CL client cool now there's two more things I'm going to show um one of them we'll do together the other one I'll just show you while that one is running um but before I move on is there anybody who's still playing around or getting up to the step or am I good to move to the next spot awesome great so the third thing we're going to do is we're going to we're going to run the the me uh hard uh software um in in the test net as well uh so the first thing I

want to do is I want to run kosis clean- a to remove the stuff that I've already ran uh it's going to free up some space for me to run uh kind of a a more uh heavy uh test net here with the flash poot stuff running so going back here cetosis clean- a it's going to wipe away all of the containers and the uh the files that were generated in the creation of these test Nets it's going to clean up both of those things great and now what we can do is we can go back to the summit uh GitHub

where we have the configurations and we can go to M.O and in M.O we have a different setup where we've configured a full me to be running and we've also configured some parameters uh that are uh one going to make it uh I think move a little faster for the demo um and also um going to set uh exactly which images are going to be used for MV flood and for MV relay so you can pop into that GitHub and just copy and paste that into a new yaml inside of your gitpod called mv.

yo and this one doesn't have a typo so that m dyo and this one we're not going to edit together it should work uh just the way it is uh for what we want and then all we'll do is we'll run a very similar command as before kosis run github.com cryosis Texan package DJ args file in order to specify the args file and what I'll do is instead of running complex.

ammo I'll just run M.O cool now this one is going to take a hot second to run um because I want to show you me working and um it's going to take a couple of epoch um in order for that to happen uh so so that's going to run and while that's running I'll show you how to do stuff over kubernetes um and uh hopefully by the time we're done with that we get back then everything will be clean and good to go uh with this real quick just to check

great okay I'll pop over into kubernetes real quick to show you what it looks like um and I'm going to do this on my own laptop uh so just to show you ketosis runs not only in git pod but also on any laptop that you bring um and ketosis it runs over it can run over Docker or kubernetes and the way that you choose between these things um is with the cluster setting so if I was to just set my cluster uh to Docker let me make sure my Gateway is not running yes so if I set it to Docker then I'm running over my local Docker engine and if I set it to remote cetes that I'm running over a remote kubernetes engine and uh oop

and the way that works is in my Cube control file I have a remote carbonates cluster set up so I did that beforehand um that's why that's why I can just type this but then what you would do is you would make sure that you have your Cube control on your own laptop configured to your remote Kates cluster and ketosis will just use that to connect to it um so now I'm on my remote Kates cluster I'm just going to run one more one more process here it's called cryosis Gateway and cryosis Gateway enables port forwarding from everything running in cryosis into my local laptop uh it'll just make the user experience a lot nicer and I'll feel the same thing that I saw before where all of the important ports that are running inside of my Enclave uh which is the the the test net

those are all going to be mapped towards my local machine so that I can access them and do my workflows there so now that I'm on uh let's see this kosis just to make sure ketosis cluster LS I am indeed on a remote kubernetes and then I can run the the package exactly the same way that we saw before uh kosis Tech SL ethereum package I can run it exactly the same way as before just pass in the GitHub locator and I'm going to have the exact same user experience but instead of the deploy Target uh being Docker it's going to be uh it's going to be kubernetes on the the remote cluster um so it's going

to show me the same exact thing the steps that it's doing as it's spinning up my network and then once we get to the bottom of it it's going to show as well the the list of servers uh services that are in the network and the local port mappings and just real quick I'm going to check the progress on yes me

yes yes um and our my CTO is is here as well so he can give you some more detailed answers uh the short answer is that right now um right now the the administration of the kids cluster is completely handled by uh by the end user um rather than us and the extent to which you can do very customizable things are um inside of the kosis package you can specify the resources you desire per per thing um per Service uh and then you can also specify custom labels and that's how you do stuff like chaos mesh uh workflows um because you can specify the labels that get tagged into things and then your other systems will pick it up um and we're working on persistent

environments so these are designed to be ephemeral um but we're working a lot in the persistent use case and Kevin might be able to give some better uh details on that so the balance we're trying to strike is not exposing an overwhelming amount of configs so be making the abstraction useful but also giving yourself an escape hatch in order to get down to being able to mess with things that you might need um so the balance that we have right now is being able to specify uh Min and Max resource limits of CP and memory uh and custom labels we don't have the ability yet to do affinities and taints so being able to specify which node that you want to get allocated or looking at that now what would be a good design for that

something that's not going to overwhelm the user um and then uh beyond that like for example being able to inject service meshes and stuff like that we don't have that ability yet it's something we're also ter uh the question was what about terraform um yeah so uh we would um the way to use terraform to to automate all of this would actually be that you use terraform call down to curtosis uh so that way curtosis knows about all the resources and everything that you're you're being to able to spun up uh if you want the very lowlevel control uh then terraform directly being able to like call Direct kubernetes resources kto is not going to be a replacement for that so if you need like very deep control the idea is that

we're trying to provide a much more user friendly abstraction on top of this it's uh looked like you had different kind of clusters that you could set with like kubernetes and local or Docker I mean uh would it be possible for someone to contribute their own cluster configuration I I can this one uh yeah yeah so one of the um uh the question was how difficult would it be um that it would definitely be not insignificant amount of work um but two of the ones that we've contemplated is maybe a bare metal uh cluster back end and then uh potentially

a hashy COR Nomad uh back end we don't have either of those we don't have plans uh the bare metal one we're kind of exploring a little bit um but but we don't have that yet but but it would be possible we have an open flag you know you can go to the rep GitHub and and what these values are um you can put any number of kubernetes clusters onto this setting so you could have eight of them if you have eight of them um that's very much a user set thing but right now at least it has to be either Kates or Docker um yeah so like I said just to close the loop on that I mean all this random CES here but um it is mapped to my local machine just like uh it was earlier over Docker in gpod um so I still can like

click through for example to Dora and then we're going to see we're going to see d the Explorer and Dora the Explorer is going to show us you know the slots and the validators that we would expect to see well and then the the same thing works as well if I want to clean up everything just get rid of um all of those containers and uh files and everything that's kind of messing up my kubernetes cluster I just run cryosis clean- a and it's going to clean up all that stuff and my Kates is going to be clean from the stuff that I that I put there before let's pop over and see if the timing worked out uh favorably where is my git pod I may have lost it oh here we are

great timing did work out favorably so uh my me flow has has ran and um you know I have a couple of of services here that are related to uh the flash blots infastructure M flood M Rel API housekeeper website all of this now I click into the me website this is where I'm going to be able to see um whether payloads have been delivered um from The Flash Bots relay and we haven't had any payloads delivered yet and if we want to see perhaps why we can pop into I guess Dora is going to tell us that 28 more

slots Let's see we are on the the third Epoch we need the fourth Epoch so 60 more 50 more seconds 84 more seconds we can count down together if we want to um while we're waiting for that let me just pop into the presentation real quick let's see yes uh so just to recap the stuff we saw in kubernetes while we're waiting for that to happen um we have people using um kosis to deploy these test Nets on kubernetes and then using chaos mesh on the same Kates cluster to do uh to do to do chaos testing um there was a tool by uh trailer bits that was showing that

at the trust X talk yesterday um to reproduce a finality incident that happened last year on mainnet um and you can do a lot of other stuff there uh and it also is useful for doing stuff a higher scale of course if you're going to do it remotely you can scale it up and Kates will handle all that for you great we're on the fourth Epoch and if we go to flash Bots let's see what we got for the Boost relay we're starting to do payloads yes so we have uh flash Poots actually delivering payloads to the network um now that we are on the fourth Epoch of uh of our beautiful test net great now that is it for the uh for

the workshop um you can open it up to this flexible question question um if you like and also we have a QR code so you can scan it and get to the ethereum package I imagine you if you wanted to go there you've already gone there probably uh over the course of this um but the cetosis folks are on the crowd yes um so how useful would you say is it for execution layer testing um because there's a lot of kind of existing clients for testing think hard hat or um envil for example found um how useful is it or is it possible to add those into this stack and kind of have like a multi

both consensus layer and execution layer testing spe kind of combined I know that there probably a lot of trouble hooking that stuff off to each other but maybe you have a guess yeah um I think a lot of the execution stuff works really really well with like um basically like evm simulators um and people that really focus on on the evm testing side of things um so I haven't yet seen El or like evm style or like dap layer uh testing working well with curtosis the way I would do it is I would try to combine those tools into the network um to the extent that any of those tools can connect to like full nodes that then have different consensus layer configurations uh the other like place

where it would be very interesting um is if you're doing things with like oracles or other like side services that you would want to have hooked in to a full node and you want to see how that thing changing impacts how your smart contract is is moving um because then it's like more realistic uh cool love to hear so basically uh entire HS which are being spin up on every release on every testing phase of the El and cl are using CIS as well so the CIS is spinning out the local test net and the transaction

spamming is happening and a lot of plenty of thousands of tests are being executed for each DEET each before Shadow Forks Etc so this like highly appreciated for us and very helpful and on other side which I'm doing right now we are helping FL Ste directly about comparing two versions of Builders that's in the performance and we are using and we help we get tons of help from the carass team about the m flat so we are using it very heavily and I guess it's already starting to help the flash team about uprs and Builder start is open source right now so this would be some kind of like improv like performance comparision of the latest version comparing to the pr which will

be pushed to the build so that's the current use case which we have like closing C with those are the things and I can add one more comment on this so the packages are designed to be written in a dialect of python and you can import packages into other Python scripts and then build extra stuff on top of that so basically we have an import system and then you can say hey like slend together any bits and pieces that you want uh to form new packages so if you wanted to add something on top maybe like add a a dab server or something like that you more questions and let's thank the speaker again and

thank [Applause] you