All Core Devs - Consensus #51

Transcript

[Music] [Applause] [Music] foreign [Music] all right uh streams transferring over

here's the agenda um and let's get started testing and release updates um the i know there's an issue with a malformed test vector that is fixed on the one oh candidate release branch but i have not yet released another update to that i know a lot of you have a temporary workaround just by modifying that file manually there's a couple of additional test factors i like to get merged in and then i can get those up additionally we are looking into

kind of some test vector reform to reduce the overall size reduce the amount of time to generate and to get those four choice tests integrated and kind of built off by spec um shall we and myself are gonna dig into that over the coming weeks so hopefully some good stuff um we'll try not to like pull the format out from underneath you before mainnet but um probably soon after we will release kind of a reformed test vector generally the same structure but um with some slight modifications um other testing and

release updates okay cool um moving on i'd like to talk about the state of madasha uh i know we've all kind of been chatting about it i just want to see where people are at uh the leak is working no fundamental consensus errors but it does look like we are seeing uh instabilities probably every like four to eight epochs um i know that prism is looking into some

optimizations and what might look like kind of a dos vector that's popping up are there any other kind of fundamental issues or resource consumption issues that have arisen with any other clients so we've been suffering high resource usage on teku but has been constant wouldn't explain the periodic drops and is now fixed so i don't think that relates gotcha thanks

very cool um let's it's obviously something we need to fix um let's here with the let's hear the client updates and then um we can talk about this more outside the call thank you so let's move on to client updates uh tekku can start us off hey guys um so our audit is complete we're happy with outcome and we'll share the final report when it's available um we pushed a few performance improvements related to madosha not finalizing um to reduce memory consumption we reworked our state regeneration logic so that q

q tasks are no longer holding on to state references we eliminated some extra processing that we were doing at finalization that could cause the node to pause after a long period of non-finalization basically we were previously regenerating finalized states so that they could be stored and instead we just we now just store finalized states that are immediately available we also found a nice optimization related to our get ancestor calculation we move that logic closer to the proto array code so that the lookups are a lot quicker and more efficient other updates outside of modasha we've

updated to discovery version 5.1 we've been working to implement the standard rest api and handling a few issues raised related to that we've done some work to improve our ssc serialization for large beacon states we've seen a 20x speedup we started work to sync the client from a recent state this is in progress that those flags are currently hidden and related to this i posted a question about uh how we should be handling blocks by range requests when his historical data is missing um i can go into that now if that makes sense let's talk about it during a networking

section okay cool um yeah and then that's it for us great um congrats on some of those optimizations 20x speedup is huge let's go on to lighthouse hey everyone um so for lighthouse we've updated uh we've made a release that updates us to the version one of the specs so we are training taco on on modasha with that um also we've updated to dysview 5.1 so we're also on board with 5.1 on modasha we've been finishing up some of the f2

standard api endpoints um there were some discrepancies so we're working through those to hopefully get everything uh sorted we've also been adding http api endpoints into our validated client uh this is to uh kind of facilitate a ui that we're building to help onboard users and make it a little bit easier to use we've been monitoring for quite a while now our gossip sub 1.1 scoring kind of uh parameters um so we've got a node or a few nodes that are actually have those implemented and just seeing how they score other peers so our plan is to kind of merge those down but in order to do that everyone has to kind of agree on on the latest gossip

subspec we've also been doing a number of performance updates similar to everybody else specifically in reducing some memory and we've been tracking various bug fixes um bugs that have have occurred due to some of the b2p updates and some various updates through the code basis as things haven't finalized that's pretty much it from us gotcha um and i might bring up those one one parameters during network section as well thank you uh lodestar hey uh so we've been working on getting our pub sub up to spec uh turns out we didn't have the uh

signature policy uh strict science trick no sign implemented so uh we're resolving that um we merged in uh our validator slashing protection interchange um and kind of refactored that to handle all all slashing cases um we uh are finishing up with the standard rest api we thought we were finished but we hadn't implemented the state uh pieces pieces under the state and um we are trying to finish up a refactor of our

core chain logic because we're wanting to pull that out into a separate module that our separate package that could hopefully be used independently of ludstar and we're kind of currently still trying to gear up for another release and once we get all of our in-flight prs merged we'll should be ready got it thanks kevin prism hey guys so we had our first um uh beta release promoting from alpha to beta that was a really big jump for us we we've never really done breaking changes all throughout the process so it was the first time we had

a chance to really get rid of legacy code promote features that were in v2 to just the default um and overall just make a lot of improvements um you know regarding the media institute yes we're investigating i think we we might have a potential fix found by nashant um so mostly has to do with kind of like retrieving information from disk in times of no finality and some mismatches between uh cash retrievals and maybe times where we don't fetch from the cash so yeah so you know we'll see if that if that uh helps a resolution today uh also we have been working on the standard e2 api just been implementing the endpoints by category and that's been that's been going okay

um aside from that yeah like we're just uh trying to knock out as many issues as possible before mainnet i think we had like 100 open issues and a lot of things going on and now we're i think around 70 or less than 70. so overall just trying to get into mainnet with as few open items as possible just knocking them out one after another the trailer bits on it and went pretty well so we've been resolving all those audit related items and just trying to harden security as time goes on you know just taking it one day at a time and uh trying to trying to just um do as much as we can got it thank you and nimbus hi like everybody else we've deployed

discovery 5.1 and vos v4 on madasha we are used the current state of madasha to [Music] test how they must behave when the situation is brought we introduced some new optimizations and currently we are running pretty stable even though the network has long penetration period but our memory usage is stays roughly the same we are working on some new benchmark targets that make our targets that you allow everyone to run our benchmarks some more easily last week actually reported that we are

working on a way to start the beacon node from a workweek objectivity checkpoint state and in this mode there is one relatively complicated problem of how you produce new blocks i mentioned it last week but now reduced the required data to this to just uh the state of the deposit contract uh i'm again curious if any of the other teams has worked on this problem because i see some potential for standardizing the format of the checkpoint states that all the clients could work with and that's it for us thank you

and i believe that is all the client teams correct me if i'm wrong um and we can uh when we bring up networking and talk a little bit about blocks by range as i read you can pose your question then um see if there's any feedback on onto research updates does anybody want to get us started um i can talk about updates to the phase one uh and of data availability work that we've been doing um so i guess the uh the update there is that um so my cell uh

so recently we've been um exploring this kind of uh expedited uh data availability focused phase one direction which will uh bring in shards as you know explicit data shards and and use cad day commitments and some other tools to make sure that we can very easily do a data availability sampling and and so lately we've been looking into exactly what kind of subnet structure data availability sampling requires proto has been starting to make a concrete implementation of both the erasure coding and commitment side um and

the subnet side um i wrote up a in the in the process of writing up a of writing up a document and so that work is kind of proceeding proceeding quietly but well um at this point soon well one of the challenge um uh things that we're thinking about right now is exactly what the concrete proposed uh structure in terms of in terms of proposals will look like and uh this is as good an opportunity as any to try to see if we can do things like staggering to try to get the kind of short block time down to like much less than once watch so i published a blurb on that youth research about

that problem and we're talking about possible solutions at the moment and so that is i guess still going and the goal of this is to move as quickly as we can towards something that we can live test in some kind of shorted peer-to-peer environment thank you vitalik uh other research updates uh one question um which type of ratio codes are you looking at um just standard read some menu codes using a finite field with the curve being the same as the vls

curve we don't really have actuate regard because we do because we needs to be in if compatible with with the categories which we will have to use bls 12 through 81 for thank you other research updates um okay so this will probably should go in networking but i would say just to give an update uh we keep working on the network roller um and um so

at this point we are able to gather a bunch of information with this very light client that is based on rumor uh we get peer id not id client type and version ip multi-address we can also figure out the country and the city of each client and latency of each client and we subscribe to all the five gossip subtopics i will put on the chat a complete report of what we are able to do with the color after this point that's it thanks cool yeah it'd be great to see that data any other updates before we move on okay great moving on to networking um

one of the big things uh to talk about is what meredith brought up which is uh once you've started from wake subjectivity state uh what do you do about serving blocks because you might be getting block requests outside of since you've synced the chain presumably you've at least have the blocks since that state that you started from but that state could be pretty near the head this is in a slight hand waving this is handled kind of in the spec it says that blocks by range requests should be served within the week subjectivity sync period

the intention there and this was written probably well over a year ago and needs to be refined based on these discussions but the intention there is um if the week subjectivity period is two weeks and you started with something from two days ago you should backfill uh the blocks through those two weeks and be able to serve through those two weeks um you could also backfill all the way to genesis but that's not requisite for the function like the baseline functionality of the protocol um that can and should probably be refined and made more explicit um my intuition on the actual granularity of the question of if somebody makes a request and you

should should you what should you do if you don't have the blocks um yeah we need to think through that um i think what i'm what i responded to is more like what is the expectation when you're actually sinking from somebody and what is like kind of you can can you consider malicious behavior or not and really being able to serve through that range is kind of the baseline expected functionality but if you don't have a block um we do need to figure out which of those to do um does anybody do here right now yeah meredith yeah i was just going to say real quickly um

like we're planning to backfill blocks but even if you're back filling blocks there'll be some period of time where you're still in the process of doing that right so like you still need to be able to like you know figure out what is an explicit way to handle this case yeah absolutely are other teams handling this case currently we're not handling it but i i thought a general idea might be that we could indicate what blocks we support in either the enr or the metadata so that you know what blocks appear supports you shouldn't ask it for ones that it doesn't right metadata does this seem like a

reasonable place um maybe the absence of information implies that you support at least through the subjectivity period which we can make a more explicit definition in the spec and then an explicit explicit data and metadata would imply something more explicit like you export support beyond that range but still there's kind of the corner case of your backfilling uh yeah suppose when you're backfilling you could also uh and you have less than the week separativity period you could still specify as such um but it still

begs the question of if somebody asks you something outside of that metadata you need to respond with something which is either error or empty i would presume error a more explicit error rather than emptiness because emptiness uh might imply that there's just nothing on that range and then when you get something else from somebody else in that range it looks malicious yeah that's kind of the direction we were heading um also in terms of like signaling what blocks are available it i think ideally it would be nice if that was on the status message because um we kind of query um status periodically so we have an

updated view of where our peers are at so you know you could kind of update your range as you backfill um but we don't have that right now so right how frequently is metadata passed what are the rules in that i'm looking up myself right now i guess there are no rules when are people practically calling metadata is anyone using get metadata at all

okay i will circle back with y'all outside of this call if we don't know right now adrian you keep on muting do you have anything yeah i i can't remember it's related to the ping right we check the ping and then we get the the number and then if it's out of date we update it we do a ping every 30 seconds right okay we we don't do this for example what was that sorry we are not using pink yet that's what i wanted to mention i see but yeah you're right adrian um in the ping you send your metadata sequence number and if it's out of date you update um meredith are you all using that

protocol um yeah i was just trying to look up and find the actual number um it looks like we're pinging every 10 seconds yeah we're paying like twice an epoch so but we mostly just um look at peers that are subscribed to the particular subnets from gossip sub and also use that mostly right for the nets things yep gotcha okay so this should land either in status or metadata

um let's i'll take a look at both those protocols and write a quick issue on the spec repo and we can talk about it there and try to refine how to handle this uh edge case sounds good great zari you again had the question of when you've done weak subjectivity sync and you don't necessarily have information through genesis how do you handle block production especially with respect to eth1 data right can you pose that question more concretely

when one one option is to discuss it offline i started kind of a message for this in the wake subjectivity telegram i believe everyone is there but the short version is that you need to produce miracle proofs for the newly added deposits in new blocks and to produce these markov proofs the current spec says that you need the entire set of deposits from the start from kinesis it's not very complicated although a bit tricky to create an algorithm that doesn't use all the deposits but only the state of the deposit contract the node array which is incremented that we use to create

the deposit route and i mean from one one hand we can share like how we did this and then it would be probably valuable if we standardize what is the checkpoint state that each plant should obtain to start operating it includes the beacon state this deposit data in the compressed form and potentially some metadata for the network so you are not mixing networks by mistake right i see so if you if you have a a branch if you have a branch anywhere

in the tree and you append to the right then you should be able to form the root right so you're saying in addition to in addition to the beacon state you should also have a recent deposit and proof essentially a branch in that tree so then you can always kind of grow to the right by appending new deposits well it's roughly equivalent to proof in size but more precisely it's exactly the state of the deposit contract so it's this you have to examine it more carefully exactly right but it's similar in

spirit right because the state of the deposit contract is carrying essentially one of those one of the the most recent branch and growing from there yeah that makes sense okay yeah we can um are there any more thoughts here um otherwise we can talk about it in the telegram okay great and were there any other networking items that there might be oh uh gossip subview one one um i uh ben from sigma prime posted something

on the specs repo a while ago and kind of the generation of the params i haven't reviewed that yet um but i do think that it's about time for um more than a few of us to review it and get this into the specs age can a do you think is it is it safe for just a few nodes to deploy the rewards parameters uh or does it need to kind of happen all at once we i mean ben can comment more on this um if he wants to but we we have a few nodes that have it and you can kind of see how they score everybody else on the network so

that's fine they don't really affect anything else um but in order for the network to function everyone kind of has to be on the right on the same spec level um you can't have like people get um penalized if they're not sending messages when you expect them to so if people are subscribing to topics while they're thinking for example then they'll get kicked out of the mesh and get get scored badly or if we're not aligned on our message ids things like that you can can cause um your nodes to score other nodes down so if if lighthouse merged into master for example and all the lighthouse nodes had scoring we it's likely that we could uh segregate ourselves via gossip and just only talk to other lighthouse

notes right it might be worthwhile spinning up um a small network before we push out to madosha uh with a bunch of the clients to see if there's any like bad behaviors that we're not catching right now yeah it's a good idea um one one problem or one one thing we always have to think about when about when talking about gossips or parameters that it depends strongly on the number of validators how many messages we see right so this is also um why maybe the behavior in madasha is different than on a small test network um for those parameters

right all right is there and i think we've talked about this a little bit before but is there a way to dynamically adjust those as the validator size adjust the scoring parameters as a function of validator size like can the application this is basic oh sorry yeah yeah go on this is this is basically what we what what i what we tried and um why i proposed in this issue uh a python script that basically computes the or basically a bigger formula for computing the parameters based on the number of validators and currently we are just updating them regularly on in lighthouse yeah or in the tests in peers we are testing it currently

so updating them manually like via a different release of the software oh no no this is this is done really application there during runtime okay gotcha is that function custom functionality that you had to build on uh russell p2p or is that standard functionality with respect to the new gossip sub v11 um since we basically implemented the gossip v11 for libya to appear we also added this functionality gotcha yeah i think it goes planning on doing as well right resetting the scores yes there was all the talk about that

yeah okay so there might be a little bit more leg work to do to be get this in other languages okay um ben has an issue up a comment a recent comment and an issue on the specs repo if you are one of the networking folks from your team please take a look at the script uh we need a couple of eyes to review this before we define these parameters and you know we could use some help i'll share the link in a minute uh any other

thoughts on v11 scoring params or anything else okay great any other networking related items great so this is the comment in the issue that i'd love for people to take a look at uh ben's put in a lot of work to get this script running um and we need to vet it before we make

it part of the spec so take a look at that i might knock on your door in the next day or so if you don't thank you moving on general spec discussion anything here great um open discussion closing remarks anything else on your mind how's the progress on the bls blst audit uh it's great um there's been a couple

of minor issues in reports over the past couple weeks and we expect to have not the final completed report done early next week but the kind of final wave of feedback at which point i think we can pull the trigger on everything cool anything else donkered you unmuted you got something for us um i just wonder if someone knows like looking at medasha it seems like there is like

the inactivity leak in progress but it doesn't seem to correspond to like an increase in participation that you would expect does anyone know why this is so we talked about this a little bit like beginning um our best guess right now is uh an instability in prism uh it seems that every two four or eight epochs there's a high resource consumption which is causing a kind of instabilities you see the participation rate grow and then you see it uh kind of distinctly diminish and at least on the prismatics nodes they're seeing this high resource consumption it's being investigated but

um that's our best guess at the moment yeah i guess one interesting analysis anyone have has the easy um access to that um would be like to see what if you if you take all the validators that have been online for like say 100 epochs what is the what's the total percentage and if it's like much more than what we're seeing then that indicates that there's a pretty big problem and that might be related to the forking that um alex was saying i don't know yeah um i'm gonna spend some time kind

of poking around uh prismatic team is working on an update to release on the resource consumption issue they're seeing and i think uh barnabas might have some scripts that will allow us to easily check out the kind of overall participation so maybe we can get that number by tomorrow thank you anything else here great well

keep up the good work talk to you all soon thank you thanks everyone guys thank you you