How to Build a Production-ready Ethereum 2.0 Client

Transcript

our next speaker Preston Van Loon is actually the co-founder and tech lead of prismatic lab which is one of the first teams that started looking at implementing East 2.0 and we also like just yesterday they eath 2.0 yo so it will be really exciting to hear about one of the East 2.0 veterans if there is such a thing alright thanks I guess I have underscore in my last name now so that's cool I want to talk to you guys today a little bit about what we're doing at prismatic labs our focus is of course on a dream 2.0 but we like to say

that we're targeting a production ready client so hopefully by the end of this you'll understand what it means and how we're going to get there this is prismatic labs we're a team of independent watching developers currently there's four full-time people or two part-time people but we also have 40 40 or so open source contributors a lot of them come from get coin bounties programs so that's super cool that we can engage the community and sort of get everybody involved in a building dream 2.0 let's talk about why are we doing this and how are we going to do it well

it all started I guess at the end of 2017 rollin Raoul Jordan and I had seen this starting FAQ and we said wow you know this is a really great idea it seems pretty complete it seems kind of stable maybe but we wanted to get involved and what we do understand was that aetherium doesn't scale especially at that time when we were seeing things like crypto kitties and and other applications max out the blockchain already on a global scale the amount of users on aetherium was relatively low and the amount of capacity being used was extremely high

so we we saw this as a really critical problem and we want to get involved and we wanted to start doing it so we did and today we have almost 2,000 commits I think I took this snapshot on Tuesday but we've we've been really really trying to put this forth to put this together and just just build it and help build it aetherium 2.0 so let's talk a little bit about the decisions we made on why we're building the client this way so why use the go programming language well it has a lot of things that are that are just built in that are really helpful so it has a mechanism for documenting your code in in such a way that you already by default have a

documentation page and that's a go Docs org everybody who has a github project has one it's also type safe so you don't really have to wonder what does this function take what are the inputs here and what are the outputs you sort of you you can be certain of what's what's going on and when you're reviewing code in in without context that's extremely helpful because you don't always have that background or the the knowledge that the author had if you can defer that from the code that's excellent it also has built in garbage collection so we don't really have to think about of course we think about our memory allocation but we don't have to think about returning it it's just done done for us so that's great and the standard

library is amazing there you'll notice there are very few framework or framework like competitions for further or forego at least compared to things like JavaScript where there's like a framework for everything like left pad where you're just padding you know a number so that's great for a concurrency so it seems to you know be a great candidate for us given that aetherium 2.0 is an entirely new system and we're rewriting everything we have this greenfield opportunity to just sort of choose more robust tools so what we've done here is we've taken some of the powerful

technologies such as basil which is a build tool so that you can get really reproducible incremental builds and what's great about it is it's it's language agnostic so if we have some code in here that's go or some solidity or Java or whatever we can build it all using the same command additionally we have G RPC which is a framework for client-server communication uses protocol buffers which are really nice for having just performant and you know like type safe and reliable wire protocol also using the p2p that seems to be the best candidate for for

peer-to-peer communications and then we're using Paul TB for our back-end database one thing that's we've come to learn over the last year is that the code quality is super important we've actually already gone through several rewrites of the entire system having to start over for for one reason or another and finding that we're going to be the maintainer of this code for a long time it needs to be excellent because we're gonna be looking at it for a long time so this image on the right here is a snapshot from from one of our team members github page where you can sort of see what does their activity look like well the majority of what everyone's doing is code review or

usually reviewing more code than we're writing and that that way everybody is on the same page about what's going into the project is it the best idea and is it excellent some of the things we enforce so that if you fix something you need to have a test so that it doesn't get unfixed a lot of times you might have a merge conflict and someone says I'm not sure what this line is I'll delete it all the tests pass it's probably not important and yeah and you need to write the code so that you can understand it without having this additional context and if you have new features you need to have reasonable test coverage it is not okay to just do one test to see that it doesn't panic or explode it actually needs to go through

all the behaviors you can think of especially any edge cases that might get you later so some of the road map so far we started it in January 2018 after row and I met in New York City to kind of get involved with etherium and then in March 2018 there was a youth 2.0 workshop I think the first of of those kind and since then we've had several other major milestones we had merged our research team merged shardy and Casper we came up with our proof of concept this was a client that that worked locally on your own machine without any network peering we're really close to having our test

network the spec freeze is coming that's great and then we'll have these cross client test networks so the real question is you know really win-win test net this is the this is the new win moon win test net so we actually had one for over a month now it just doesn't last very long so when we're talking about test nets we have two types we have short live types that can span from just a few days or weeks or or months and then we have long-lived test networks that we expect to run until all of eternity or until they stop being useful these ones are so

short-lived that you'll come back an hour later and it's dead so that's not very useful yet but we're getting really close every day it goes a little bit farther if we learn something new we find a new bug it's getting super close so how do we orchestrate this naturally this blockchain system is a network of machines so one idea is that we could just take four computers here that are on the same network and connect all them together and this doesn't scale very well because you need to buy new machines or you need to spin up new new virtual servers or whatever so we've taken to a framework called kubernetes and sort of use this tool to

you know manage our orchestration of our containers here so there's a lot going on in this picture but basically we can any of these processes if we wanted to scale them out so if we wanted more beacon notes we could just change the number from fifty to a hundred and then they will get scheduled and as long as we have enough resources available and likewise with the validators we can easily scale them up and sort of see new values come in and out and and really architect our test network in sort of a one-click deployment so what do you need for the test network well you need some kind of scalable infrastructure like we

just saw you need some sort of reliable to 41.0 test network so we're using the Gourley network because we want this to be real deposits coming from a real etherium like chain we want to make sure that it works locally of course we're not wasting our time we want it to be accessible to outside peers so we can connect to it it's not just a simulation and you want to make sure you don't have any plans for a few hours because you're going to be just staring at I watching it as soon as you launch it so we check all of our boxes we're ready to go let's do it let's deploy let's deploy test network so we do it we slap this big blue button and then we say alright wow

the logs are flowing look at all these colors I'm just in euphoria right now this is amazing but this is just one of the validators and there were 16,000 so how do I understand what's really going on in my system well we have metrics instrumentation and we can build dashboards like this so what we see here is we have a very nice uh pin to the right which is this the blockchain hide over time we really want to see that be going up really nicely we're getting finalisation justification we're seeing new validators come in and get deposited there are deposits are included in blocks this is all looking really great

we have some healthy network traffic things are looking good but then sometimes things don't look so good so all of these charts look super scary and it's kind of hard to see this kind of information just from logs alone so we find this is a really really powerful tool but if we find ourselves in this scenario we say oh wow something is just churning through resources we have a quarter million subroutines running concurrently so what what is going on how can we even like further dive into some of the things wrong with the system so we actually have this idea called request tracing and we're using a

framework called open Census to to achieve this and here we can see the life cycle of a request from start to finish so we can see when a validator proposes a block well how long does it every step of that take including propagation through our network so that can be really nice for finding issues we can see that the this block took almost 300 milliseconds to propose sometimes we have errors so we'll one a query for okay what was an error when did it happen was sort of on what method and and why that can be super helpful and then sort of the next steps like after

we have all this stuff together we have our any Testament work we can really understand what's going on well we don't we don't want to break it so we're going to be enforcing a lot of rules so we already have enforced unit tests on a pre submit this is going to include the U 2.0 conformance testing so we'll be conformant with what the specification requires and we'll be in line with other clients but we'll also have more full fledged into and testing so you can imagine standing up two or three beacon nodes on your own machine and then running them with messages passing back and forth and things are getting validated and finalized and and whatnot and that can be part of the unit test

suite that's required on pre cement or at least required before new release can be tagged and then going one step further as we want to actually launch a potentially launch a mini cluster like we saw before with some non-trivial amount of validators and just see that even in you know larger conditions it still works and then finally now we'll have things like canary deployments so deploying new a new version out to the test Network and then maybe routing like 5% of the traffic to it to see that it's not knocking things over that validators not getting slashed we can monitor this and then gradually roll out everything so

just to recap like what does it mean to us to be production ready we need to hold yourself to really high standards because you are the maintainer of your code your team is the maintainer if you don't do it you're just gonna be upset with yourself in the future because you're gonna say what what is this who did this and when you look at the blame it's actually you so it's just better to hold yourself to these high standards you need to be able to understand your system somehow just using loglog ingestion alone is sometimes not enough it's it's still super helpful but it you sort of need some high-level picture of what is going on with your code and you need to be able to dive into a deeper this is useful for identifying those

regressions from new deployments or just identifying bottlenecks things you need to fix some examples you might have these failure rates and your understanding oh why did this take too long or what's taking too long and also areas for redundant work you might see that in one one request you're fetching the same information from the database a hundred times when you could have been caching that data because it's not actually changing and the most important thing to take away here is once you get it running don't break the build once we get this testing network running our our goals are not to knock it over it's to to advance it to to make it faster to make it a little bit more performant and to keep it going and that's it for me

thanks everybody