All Core Devs - Consensus #55

Transcript

so [Music] [Music] [Music] [Music]

here is the agenda and if you are listening in on youtube just drop me a line in the chat if you can hear us okay happy new year uh you know 2020 was an interesting year for many accounts but for you and ethereum i think it was a great success thank you all around um and we have a lot of work to do and a lot of exciting things to accomplish this year so here we are um i flipped it a little bit today i think we'll start with client updates then we'll talk about

um a proposed mid-year upgrade and some of the things there i think um there are a couple of proposals floating around we need to get this kind of written up in a more concrete way so we can engage with it better but there are some some pr's in the repo that i'd like some input on we can get to that a little bit later um then i want to just kind of give you the broad on q1 r and d which will um kind of i think client teams can dip their toes in do some education here and some some input which will help set the stage for upgrades

kind of expected in the latter part of the year and early next year and then we can talk about just some general stuff and see where we're at so let's go ahead and get started we can start with client updates and teku get started sure hey guys um so we merged in a new community contributed feature the option to load graffiti dynamically from a file at runtime we incorporated the recent risk api updates to the debug state endpoint so we now return the requested state as json or ssc depending on the accept header we reworked our validator client to optionally use the dependent

root fields which were added just before mainnet to detect when duties need to be invalidated we currently have this toggled off by default but testing on piermont is going well we've reviewed the upcoming protocol upgrade changes and have started some refactoring to prep for it and uh we've been doing some general cleanup bug fixes because to get performance tests up and run running some hardening of the p2p layer and some code quality related refactoring uh that's it for me great thank you let's go ahead with nimbus all right we've released a new release

1.0.6 the most significant uh new feature is that we now have reproducible builds for arm devices the other important fix in this series is that we've improved our subnet walking to logic we now receive a much more amount of unnecessary stations and this leads to reduction in gold bandwidth and cpu usage because you're not processing these suggestions we are working now on some quite interesting feature which is a new form of slashing protection where the nodes the attached validators to the node are inactive

for several epochs which is a configurable parameter and during this period you monitor the network for potential activity from the same validator running on a different node and if you detect such an activity you would enter either disable the validator running on that note or quit and present some kind of message on the user urging him to investigate the situation we are also finally starting work on the rest api as you know we so far we had json rbc api so we were not really spec and we are finally addressing this and also finally we are now having

a proper mailing list where people can subscribe and get updates about our releases [Music] nice uh that as superfiz called it that doppelganger doppelganger detection uh i think is a fantastic feature so from the command line the default is some number of epochs in if you don't specify and then you could drive that number in any direction even down to zero if you wanted to override well we have uh kind of configure parameters for advanced users

but uh so far our thinking thinking is to make this really simple for end users and we don't um offer many options just either you have the protection you don't have it gotcha our parameter is hidden so far and we are also deploying this right now in production in a special mode where it produces only a warning so our thinking here is that we want to make sure that there won't be any false positives so we monitor the system for a couple of weeks to see that these warnings are not triggered on our own fleet because we know that we don't have a doppelganger validators there right nice and that that override uh

command line pram can have in caps unsafe cool that's that's awesome but um isn't it like okay can it not be triggered if you go offline and immediately go back online like on a restart well when you restart the process is expected to repeat yourself so you can potentially enter this restart rule if you're running with a supervisor such as systemd but we've decided that that's a reasonable enough because it should be safe right i'm just saying when you just when you restart immediately after it looks

like there's a double gang all right so it will wait for another uh and ebooks and we will see again and so on and on right see you even if it does happen that's kind of harmless because all that happens is that that it shuts down right and then the user has to investigate whether there is a oh yeah i was just talking about the idea that you just checked that the the warning is never triggered but i thought it can be triggered even if you're running only one yeah yeah and then that that that's a little bit why we don't want to deploy it you know in in the hard mode immediately we want to

make sure that the user experience is reasonable as well nice cool anything if anybody is interested there is a pr up with this feature uh we can link to it later where you know there's a little bit of documentation and obviously then the code details which are actually very very easy to read um okay cool uh moving on appreciate it lighthouse hello um so we've been coordinating as a team to determine how we can

best apply ourselves to the phase zero maintenance and also the shutting and merge works i've written a blog post about this with our intentions and we should be releasing it next week um since the last score we've published um a few releases lots of fixes and improvements there um we've added a new system for monitoring validators that's in a pr now so this is on the beacon node side it has a list of validators and it performs additional logging when your evaluator has a block included or a test station included it has a lot of prometheus metrics um to do like fine training tracking of front grain tracking your validators like the delay when you're getting their attestations and blocks and things um

so you can supply the beacon node manually a list of validators to track or you can get it to detect them automatically by subscription calls to the api um that's doing the pr but we should see it soon we're adding support for weak subjectivity sync and this turned out to be quite nice since it's making our database more flexible and generic we've been reviewing the media upgrade prs um and thinking a lot about how we can best support hard forks with minimal code duplication and complexity and we're also thinking about how we can support the sharding and merge experimental works inside the lighthouse repo without you know jeopardizing the stability of the production code that's about it from us

great thanks paul um lodestar hey uh so uh we've been working on um just getting making the uh our beacon node more stable and more friendly to use so we spent some time uh speeding up our epoch transition um and figuring out how to store fewer states to disk uh we also refactored our request and response code which will help us revisit our syncing so concurrently we're working on updating the sync so we can like download and process blocks in parallel it's a lot better to do that

and we're also now shifting some of our team to be looking at future hard forks so we've been looking at how to uh update our database to more easily support hard forks and different types of data and we are planning our next release for monday with all of the things that we've done so far great thanksgiving and prism hey guys happy new year and so in the last couple weeks we have made our slashing

protection on validated on the validated client side more performant it's working better without the decline that holds over more than hundreds of keys this is on track for um version 1.1 release next monday and we also have a bunch of box fixes that has gone into the release and then on the e2 api implement implementation side we are almost done with the now with the networking stack um in parallel we are also uh experimenting um like cleansing uh committees and then reforming accounting with the participation based changes and then we're also revamping our

test uh test uh utilities for uh better test setups and that's it thank you thanks ernest okay i believe that is everyone um so i think we've talked about this across different channels um the current intention would be to do a minor upgrade to the beacon chain in mid year early summer would be the target uh which would include a couple of nice to have features and some um

cleanups in the way state and things are managed uh to help with just the maintenance of the beacon chain and maybe some edge cases for example processing empty epochs uh this currently is in the form of some pr proposals and stuff in what's called the light client spec folder that would be renamed to be the name of whatever this upgrade ends up being called and generally i think we want input especially on some of the state reform uh stuff from engineering teams because this

ultimately should make things cleaner but also might depending on the path taken represent uh technical debt so we kind of want to find the right balance here so please please there are a couple of pr's up for review uh accounting reform and global penalty quotient these are probably the two more invasive things with respect to state and management and validators but would love for you to take a look the other big one there is which is already in dev under that light client folder is adding this notion of a sync committee which is a very similar to a beacon committee but larger and longer standing that um proposers can

include the signature of uh which they just kind of sign over the latest block and what that does is provide like client support kind of as a first class citizen in addition to some of the beacon chain changes to support that feature there is a sync protocol file which demonstrates how you can use this to construct a liteclasing protocol analyte client that's reuses a lot of the components that we already have the notion of committee subnets aggregate signatures that kind of stuff um and it's a really a big win so i think it's a nice feature to get in in addition to that

some things that we might want to discuss are maybe a new message type for reactivation or other things like that which i think we can discuss in the in the coming weeks if there are any additional good features to get in there beyond that there are a couple of like network iterations network fixes that are coming out which i believe should be released prior to such a fork and um a couple of security fixes to the fork choice which are under some internal review right now and i will

share those with you once we have concrete proposals which should be in the next week or so that too can be released prior to this kind of mid-year upgrade and probably should be on all of that i'm happy you know open up for discussion here and also um take the discussion to the repo i think we need to figure out how to kind of package all of this for discussion and review other than just hammering out these pr's and the spec repo and obviously that can be a bit opaque uh so in the next week or so i'd like to get you know this just written down um kind of bulleted at least

in some form so we can talk about it in aggregate better any questions or discussion on that this also serves as something of a warm-up gets our hands dirty in being able to upgrade the beacon chain in a production context i was going to have to you know about to do that on test nets and then do that on mainnet before a couple of the more ambitious and larger upgrades which would be the merge and charting further down the line right i guess uh one thing to add is that the ideally we want to kind of

come to consensus on like what we what we wants to go into heart well i guess what we're calling hard fork one kind of fairly quickly um just uh so that uh we don't need to lose any uh lose any time on moving forward on it uh so i'm not in the list of uh kind of items that can potentially go in as i understand it as like number one sync committee is numbered to the incen um accounting reforms um where there's a yeah enough a lighter version and then there's a a somewhat deeper version uh that replaces effective balances um or

it changes the way that effective balances works and that kind of moves quotient some out and makes it easier to process an empty e-box so that's three and then also fork choice changes there's two tweaks to um resolve some of the issues that people have come up with over the um and published some papers about over the last few months but like i guess um this will be written on work well this is these things are already in the issues and there's already some documents about specific things but we'll come up on oh we will um help kind of package that together into a more unified list and it would be i guess

nice to make uh some a lot of pro uh progress on kind of at least some decision making over the next couple of weeks yeah i agreed um i think we set ourselves up for moving quickly uh if we can kind of agree on this in the next couple next two weeks i guess by the end of january um and so next call yeah yeah i'll if i haven't heard from you on from your team on on especially those accounting reforms uh i will knock on your door pretty soon uh i had a question about that i mean um

part of the reason of doing the accounting reform is because we feel that some of the processing and the epoch transition is a little bit heavy um but i was curious have you looked at how much of that processing is really in the spec like in the spec function and how much of it is um due to finality occurring and therefore clients needing to uh prune trees save databases and and things like this across the board i mean i know from nimbus like we the the difference between processing a block

and processing an epoch isn't that significant if we only consider the part in the state transition hmm i yeah i remember benchmarks from uh like months earliest it could just be a long time ago but that said that at the maximum possible validator count uh it takes six something like once a lot to fully process and the e-book processing function and so like even if it's not even if it's not the worst thing it still seems like something where if i i guess if we can't alleviate it we should possibly all right part of it

is we shouldn't i'm just curious if if if you have any numbers on on what's actually going on that people transition like how much we can how much benefit we can expect yeah absolutely better of course requires like you know the latest optimized versions of clients where which have implemented pretty much the same accounting already in caches yeah um i mean any numbers from clients on that would be great additionally it also make there's two other wins uh one is throughout the epoch the state actually makes it uh is much more aware informationally

about what's going to happen which i think could help serve users and more sophisticated apis and it actually makes spec writing and spec reasoning simpler for example there is an issue currently where the intended proposed reward is just quite frankly incorrect and that was probably it was an oversight but is also because there's not there wasn't like a very clean way to uh reason about the rewards uh with respect to kind of how things were configured and one of the things there is the way rewards are handled um and based reward is handled in the

accounting reform makes it very clear as we extend and modify rewards and subsequent phases when there's additional validator duties uh that rewards and issuance are kind of like you can reason about the relationship between rewards and things to each other much more cleaner but yeah i i don't have a great intuition on the ratio of state transition to general cleaning uh right today and the other thing worth kind of restating is that the the deeper reform the quotient reform is like particularly attractive because it's uh in greatly increases the efficiency of

processing empty or nearly empty chains right which today depending on how you handle those types of things can be a pretty big dos vector all right great thanks i guess another thing to note is that the um validator set size is probably going to grow so optimizations here are valuable but you're right a lot of the optimizations a lot of what this reform does is

takes some of the uh the way things are cached and optimized on epoch transitions today and actually just kind of chucks it into state as canonical yeah from my looking at it i would be surprised if the accounting reform one um saved this much time in in the actual epoch processing but i think it's got uh potential to save us time in tree hashing uh and some other some other kind of edge cases on the pr little list of things that i think it might improve right well i can i can just briefly mention that i know from from from experience in left numbers that our slowest processing is definitely um persisting the state

on finalization and on epoch transitions really like that just so massively outweighs everything else that that's what we're focusing on right now with you know with more uh clever ways of of storing the state so that we don't duplicate as much information every time but that's going to be highly client dependent i suspect right yeah the nice thing about um the accounting reform one is that it makes a little bit easier to i think it'll make it easier to optimize the database because you're storing this kind of list of of of bits that it's quite easy to reason about versus a list of

weird attestations that are quite you know it's really hard to compress them um so i think i think it'll be handy in the long run that's good to hear okay so let's refine this conversation in the coming two weeks and plan on being able to make a decision uh on what's in here at the end of january um again any input especially on this global penalty quotient which is a little bit deeper surgery uh maybe for some big gains it

would be great okay moving on um i wrote in the agenda q1 rnd and this is really what's happening the next couple of months next few months to better solidify the path for the two larger upgrades that we'd like to do in the coming couple of years one of which is the merge the unification of ether1d2 and the other is scalability through sharded data availability both of which have been under very active r d over the

past like half year and something of specs exist in some capacity on both specs on the sharding r d are up in prs right now and merge work has largely been talked about more on each research today even though there are kind of some general articles that miguel wrote recently that at least show directionally what's going on there um and both are in different ranges of r d um there's some prototypes and even uh miguel has some like local test that's running on the merge stuff um i think in terms of client resourcing there's stuff to dig into

here which can help the r d effort and help spec effort and help refine these and then i think the goal would be to at the end of q1 have a pretty good handle on both have something looking like pretty good specs on both and being able to at that point make a decision on what to drive on more concretely in production engineering obviously at that point i think we'll be working towards this first mid-year upgrade um but after that mid-year upgrade we want to be able to put the pedal the metal on one of these two major upgrades um so point being is there's some r d work to do there so

in terms of resourcing on your team part of it i think probably continues to work on phase zero beacon chain optimizations um and we're looking towards this upgrade and then part of the team should probably spend some time on education and digging into the r d to flesh out some of this other stuff to that end we do want to do an educational kind of information exchange session at the end of january maybe the beginning of february to help bring teams up to speed on this you know a few hour workshop presentations and and q a shall always reach out to most of the team i think the teams here

to help schedule something there and i think just in general dig into what the resources are so far and start thinking about it providing feedback and thinking about what your team can do to engage with it to help refine both of those it's a bit long-winded um essentially we have merge work we have sharding work both are in like heavy r d and kind of in the early spec phases and testing um and we're working actively to refine them and can use help at the same time bring you up to speed i know everyone's been pretty heads down on on shipping the beacon chain and there's some fun stuff to learn about and start engaging with

questions thoughts comments on any of that um i can add to the merge topic um we plan to have some testnet launch in q1 some like more of uh more of what we have so far like not local but um some widely used uh test nets also and probably it makes sense to start work on the specs however there are some open questions um mostly on the

h1 side like the block hash um stuff right some of the op codes what's the best place to follow the merge work mikael um i can share i have like a document let me just drop it here um yeah actually what we have so far is the research post first and uh this hackmd dropped in chat with like the communication protocol actually you can take this document and

and see what's required on the site to communicate with the ethon engine right so we've i think up to this point been relatively quietly working in um you know a working group and driving on this and i think it's it's definitely time to compile what we have worked on uh into digestible information and also kind of opening up this working group so we're not just working in a silo and welcoming moran uh so in the coming few weeks i think that that'll be the primary goal of the working group is just to get all this information out there and begin engaging more publicly there is a merge channel

on the discord it's probably a good place to ask questions and start digging deeper um and mikhail and others will work on making sure that everything's well documented so we move from there thank you something of note is some fun cryptography related to the sharding and date availability these are pate commitments if you have a crypto

wizard on your team that wants to dig into something new before we do you know one of these workshops and better more information exchange digging into kate commitments it would be would be good uh this is some stuff that's gonna need to be written uh the kind of extension of some libraries um and so yeah that's that's definitely something to highlight some work to do there i think proto has some go reference implementations that he's working on there that he can share about the crypto yeah so

there are two repositories one about the data availability prototype and one about the category commitments and a whole lot of virtual functions that help extend data and recover data from samples so if you want to look at that then i'll share the link in the chat great um so we're going to work on getting a date together for a three four hour session where we can go over a lot of this material

and share things um and until then uh dig into what's there and ask all the questions so we can help get everyone up to speed and working on this stuff anything else here um just kind of wanted to lay it out i'm working on a big blog post i think paul also in his blog post is talks about some of the different r d efforts that are uh so hopefully that'll help orient as well okay any

research updates people would like to share today i guess broadly merge and sharding work is ongoing anything other than that and i guess one note here um and it's more of a roadmap strategic thing is that basically there's this two types of features that where we have you know planned for the future um one is like functionality you know things like the merge and

sharding and the other thing is security features and we have you know four or five different security features we have things like data availability sampling like uh proof of custody like um secret leader election vdfs and and whatnot um and i guess um you know one of the the realizations is that the the security of the beacon chain is probably good enough um you know in the in the short and medium term is not world war three grade we can make it better with all these you know fancy cryptographic um security features um but maybe we should be focusing on on

the on the functionality and i guess um you know from a from a research standpoint i think we want to try and keep um you know keep making progress internally at ef uh in terms of of specking and prototyping but maybe um limit the the expectation that uh implementers have to understand all this stuff and and and and be on top of things and and implement it in the short term so i'm i'm hopeful that kind of this this separation of of concerns between security and functionality will allow us to really go full steam on on

functionality um when it comes to implementation um as opposed to having to worry about the the more fancy cryptographic stuff which can come after we've implemented the the functionality yeah absolutely and even just from a the way that we will think about the ordering of specs and how things are released regardless of you know when security features are added to mainnet if they were shipped with the functionality features at the same time or later you'll see us prioritizing shipping in terms of specs the base functionality so that you can kind of engage with that and then

security features are extensions of that so it's a very reasonable way to think about these things and prioritize work okay anything else here great um networking age put up an issue uh there is in the way we do aggregate attestations um there's this optimization in the validations so that you ignore an aggregate

attestation if it has the same exact aggregate if it's essentially the same attestation but different aggregator this actually this was a nice to have optimization that ultimately actually makes more work on the network today because all the message id is actually related to the outer message and not that interactive station and so you end up everyone has different i want i haves for these what look like the same aggregate message and you halt the gossip on the repeats but you end up wasting a bunch of energy on i want i have to recover so um take a look at age's issue it explains it uh

ultimately the optimization is a nice to have and it probably makes sense to remove that that line unless there is a like clean way to to keep that obviously in which based on a little bit of back and forth between me and age i don't know if there is um at least well i want i have still works the same way it does and message id i think you could do it if you like message id hooked more into the application layer but then all of a sudden you're like mixing layers in in a weird way and uh you can't very quickly and efficiently calculate message ids so take a look at that issue um

if we don't have uh any brilliant thoughts on how to keep that nice to have optimization in there next week we'll probably put up a pr and remove that line um in addition to that this box sync every week subjectivity yes jastic please can i just say something about the previous one how about we just drop i have and i want that's an excellent optimization i yeah i mean i think that that's a valuable conversation to have um is especially in some of these transient attestation channels it might not be

helping us that much that said i think if we reduce the number of message ids that we include in ihave i want and rely on each node kind of randomly sending some of them that we greatly reduce the overhead of there and still get a lot of the benefit but i would love to quantify that benefit on mainnet because i i i'm not as skeptical as you but that benefit might be marginal and might just be wasting bandwidth and having additional complexity so well i mean there's what say 50k validators um thousand validators per slot whatever

right um thousand messages thousand message ids we select five of them what benefit is there really if we select the thousand of them that's a lot of traffic right if you select five of them and everyone selects five of them and you have 100 peers there's 500 of them but coming from all different directions um i guess that's not the right assumption because people aren't different subnets but there is a there is potentially a play at getting some value out of it with some randomness but look i hear you i if somebody wants to quantify what i want and i have is doing for us

today on mainnet that would help us make a decision here does anybody have any thoughts on that both on kind of a theoretical standpoint and also if we've observed anything on mainnet that will help inform us i suppose one of the benefits is in more of the tail risk and attack scenarios recovering from the event of censorship attacks and different things like that so the unintended consequences might not be

in the happy case they might be in the attack scenario case the unintended consequences of removing it i'd love to have this conversation because you're right uh there might be unnecessary bandwidth usages here uh and maybe removing it on disabling it on certain subnets could make sense if not all yeah let's take to the issue and deserves thinking absolutely absolutely okay um

i think there might be one other standing minor network issue oh well i don't know if after our last call somebody opened up an issue about um error codes and error handling but uh that's an ongoing debate uh those couple of minor network adjustments uh i would expect in a minor release rather than waiting for like a mid-year upgrade so keep your eye on that and provide any input on the spec repo if you have it any other networking items okay uh general spec discussion any

questions thoughts on anything in there that we want to talk about today okay and open discussion closing remarks anything on anyone's mind um test nets we we said we would review peer mentors the end of january are we still happy with paymont or should we be doing something different yeah i guess just a couple of things one would be test sets serve a couple of purposes for

us one staging our upgrades and also providing like staging engineering upgrades and also providing um a quality of service to the community to be able to test things i think that it currently is still serving our purposes although the more the community uses its tests and then maybe doesn't exit validators the we could see a degradation in quality of service there i'm a little bit of the mind if it ain't broke don't fix it but maybe have a plan for when we want to light it and replace it with something more sustainable i haven't put a lot of thought into this

i haven't put any work into it uh it's working but that's about all i can say does anybody have any thoughts on this one i'm not hearing anything particularly negative about piermont i mean it does occasionally stumble but this this is fine it seems to be serving those needs just uh trying to get a sense of clarity for planning um you know for people like infuria want to know where they stand if they adopt peer month whether they're gonna have to adopt something else straight away and stuff like that right

one way to potentially keep paramount in check is as a function of community that joins you know we we join in a similar ratio with uh validators we control to kind of keep something like 75 to 80 in our control um i don't think we've added any validators in a long time we could consider doing so that'll also bump the load up a little bit on the network which would be good for testing i don't know is anyone unwilling to say pyramid at least for the next couple of quarters maybe into this mid-year upgrade the main problem is that you know a

girly whale troll could uh mess up pyramid overnight without us being able to control that but maybe that's not too much of a concern uh one one minor point is that any large players like exchanges or stuff who want to do testing are not going to be able to use pure mod today in all likelihood just because if they want to test set up similar to what they will eventually go on maynet but my inclination there is just kind of make them keep them waiting because they're

may not necessarily yeah it's it doesn't seem like super high priority so when things get sorted they get sorted right and if you need to test more than a couple hundred validators maybe you should run your own tesla um okay maybe we investigate us doing auto deposits as a function of community deposits to keep the ratio good is anything wrong with that idea why is it that people contest with gamma oh they can but if you want to test uh ten thousand validators

well it might be hard to get gorilla eth and all of a sudden you might disrupt the network um which some of these exchanges we know do want to run such tests and a faucet and a girly whale or at least a responsible really well probably wouldn't give them that much ease what was that uh i was just saying i i i think the concern is more just disrupting finality of pyramid um yeah absolutely but if they show up with ten thousand validators and they're a

good citizen there's no issue there but you know there's also nothing at stake for them to remain a good citizen i think if if um exchanges and these big players if they're actually expecting to have a really decent chunk of validators then um i think perhaps we should be open to spinning one up for them um just to protect our own interests as clients by ensuring you know this good two thousand um validators run well um so i don't know if there's any big big players out there listening then they can approach me at least and we can just spin you up a test net we

used to do it all the time it's quite easy so you know we can also just get people to spin out their own test nets yeah sure perhaps i need to make it more accessible okay long live piermont uh consider us padding the deposits that client teams control as a function of new entries so that quality doesn't degrade okay anything else before we close the

meeting not for me great well thank you everyone um please have someone on your team take a look at uh the open prs that we'd like engineering feedback on uh please uh begin to dig into sharding and emerge resources um to bring yourself up to speed and uh we will talk to all in two weeks and then do uh some sort of educational information exchange workshop shortly after thank you take care thanks cheers

[Music] [Music] so [Music]

thank you [Music] [Music] [Music] [Applause]

[Music] you