PeerDAS Breakout #17

Transcript

e e hey Matthew how's it going

oh I'm on talking to mute I I realized uh my window is behind another window how you doing man I'm good happy to be able to attend the p p ass breakout calls finally only the best breakout calls for the best people obviously [Laughter] I feel like I'm so behind I haven't been I haven't really been keeping up with this stuff for a while to be fair I mean you have your fingers in an awful lot of podt so like I'm not too worried about you yeah dealing with me Bo stuff at the moment so fun it's it's my favorite did you ever read my uh ordering thing oh I'm so sorry I did not know the

thing you brought up in uh Thailand so I put up a I put it up in E3 search I put up a post about it you link did uh George or anyone else respond at least no because I kind of put it up right about like when everybody was doing Devcon so like I think it kind of just kind of just got buried okay let's see where how's the rest of your New Year's it was good can't believe the year the current year is already like 10% over it's crazy it's crazy like the the older I

get the faster they go is the part that like is a little scary because they already go fast yep hooray and we are sink load star is part of the devet now how cool congrats oh hi hello dude thank you so much for uh uh for uh you know I I just everything you do it's wonderful thank you course glad it

worked Jim okay uh should we start off else yeah um I know where I don't think we noticed that we're overlapping with fossil every second week so we might have to see what we do about that fossil for the win we cannot we

cannot steal that Thunder fair enough does anyone know if the recording is already on on this call I don't think I'm the host I was the first one in and it did say it was recording uh when I came in it says it's recording on the top right for me about the time I don't know maybe what if we keep it at this time on every other week and the week that it overlaps we go back to the pruse time is that too confusing I mean also just I was thinking this because maybe that allows at least one every two weeks the Australia people to join more

easily I think that makes sense uh we've had that issue in in the past where we had different um different times for different calls um I think people just usually don't like it if there's one person leading the call all the time but I think now there's Jimmy as well as me/ Barnabas and few more people who are willing to lead the call so it should be fine um if you are working on PR you are not allowed to work on fossil you have to focus on this I'm sorry I veto that oh and also uh Jimmy cannot attend this uh hour anymore so I'll be covering for

him yeah exactly so if you were to follow franchesco's suggestion then Jimmy would still be able to attend every alternate [Music] call cool um Franchesco could you post a message about that and if people are okay with it we can go with that suggestion thanks uh okay so here do so first step I just give you guys an overview of what's going on with the death net so since the last call we actually do have a DEET up and running uh roughly 98% and we're almost 1,000 EPO in so this is

looking significantly better than what we had in the last couple of deets um we I I think around the 3 to 5 day range is what stuff started breaking down so we seem to have cleared that CLE so congratulations um grandine is aware that they have an issue I think the person who's normally working on Grandy in pias code base is on um out of office right now so they might not fix their um no in time but uh since okay Matthew do you want to give an update about load star and I have a followup question for you sure should I leave out that somebody forgot us on the devet config oh my God I forgot about that sorry um I

I went by the spec sheet and the spec sheet didn't have load star on it so I didn't set up a loar validator sorry about that no honestly totally totally sweet uh so we we did uh get everything going got it debugged uh we're spinning up a couple nodes on our on our clusters um and uh they're they're sinking now so I'm about to uh pull it up on the dashboard and and you can go ahead and deploy that p as Branch it it seems like it's working fine now uh but it's it's with the GU uh P Das um y that's perfect um and if you want until grandine has a fix I could point the validators at you guys sweet okay um I'll take care of that

later and reach out to your confirming the image and so on um and Matthew also brought up a good point so G has merged in a couple of their changes that enable EF however they haven't gone the entire way like it doesn't do much but it seems to be changing some structures um and as a result we're going to have to lock an older version of GS that's compatible for for pias usage so I've created an created an image e panda op SL get golden pias I think so up until we actually need active El involvement please use that image for all testing so that implies all of the kosis configs

ETC we now need to specify this as the E image that you're using um and just for sake of clarity we're doing the exact same thing on the eof side we've pinned a version of Lighthouse that doesn't have the pias um code base but at the same time enables it doesn't fail on Fu um and that's what the UF guys are using for testing their test St okay um any uh clients want to give an update about first is going with form Lighthouse is holding well we found a sync issue that's preventing us to syn

in sometimes where we add appear to our sync before we get the metadata so we don't consider it a super node but it has an e and should be much so um I can talk about prism so we are continuing to work on validator custody it's a little little bit more complex than expected but uh we have some good progress and also yeah you have a bug about Fu bug which is actually imp impacting Electra because the fu Fork is the first one where the fu State and Fu block is exactly the same than the Electra State and the

Electra block and because in prism we do some du tying I mean we if something looks like Fu we consider it like we consider we consider it as Fu so sometimes we had some issues with Electra and yeah we are focusing on on it aome thank you um we already have from loar is there any other CL teams um I don't think did from MERS yeah so we're working on Parallel reconstruction we're testing it in a local

Deets um we do have some latency in validating gossip like since this uh network is an all super node Network um there are some f Forks that we see it's usually um happening when Nimbus is taking some time to make the block available for attestation there is a time difference um we didn't have get blobs implemented yet so uh basically like a couple of days back I implemented get blobs and I'm again testing it on local Dev Nets and this latency issue seems to have gone

but ideally in a real Network where um let's say we have even a 20 30 40% density of super nodes I don't think this issue should come up um so that is that is pretty much from our side we started some work on validator custody as well uh it's still work in progress yeah yeah and um I would say our sync algor is a bit optimistic um it relies preferably more on super nodes um we don't have rewriting the sync on a higher priority than other things but we are still

considering to like have a totally separate pure pool um for um peers who are subsets uh like subset of our custody um so that is that is a bit of a longterm uh thing right now but in terms of Dev Deets I think sync should work just fine um in which we where we like kind of optimistically uh syn using super nodes first um if they are preferably there in more in number in the network so yeah that's that's update from my

side good thanks for that um yeah I guess that brings us to the next two points uh and also one of the comments we do only have super nodes on the network right now uh we we only wanted to start with super nodes in the beginning because we've had quite a bit of trouble in the past with um just having a network that's stable so we wanted to First do it in Easy Mode so I guess now 3 4 days it's been stable we should be uh replacing all the not all but at least some of the super into full nodes um I see Manu put a proposal so we do about one3 um full nodes and 2/3 sorry 1/3 super nodes and 2/3 um full

nodes is that acceptable for everyone if so then I can take care of that should we do 8020 so we're not that close to losing finality if the super noes fail yeah um yeah I don't think the CL distribution symmetric anyway so I can make sure that we probably that we have around um 80% super modes I think before I think before I think before having full nodes in we should test um withhold data columns so that uh we can like sort of forcefully enforce like all

of the other nodes to reconstruct and leave it for like two three days and then uh like drop the super node density gradually although like um yeah I mean you can you can do both of the changes at the same time but I think if you do withhold first then you'll make sure that every client every like every peer is actually reconstructing okay does every client already have a um withhold data col them flag with them that I can use I think only Lighthouse and prism has just for Te we have reconstruction and

syn only we don't have reconstruction when we are on goip but we should syn yeah so it's also my point before stopping some NDS and one thir of NS and converting them from Super node to full node we should it's my first step in my in my message actually we should I guess stop only one node per uh client to ensure that uh every client is able to S up to the tip because else we will lose the finality losing finality is something I would like to see after but not not now yeah that makes sense um one caveat

am I doing Genesis sync or am I doing uh checkpoint sync right now um the node are State full right I mean if you restart a node on the devet it will it it won't uh wipe the database oh no no yeah of course it won't uh but I would um delete the database of one node of every client and then I would either need to choose if it's checkpoint sync or if it's Genesis sync I think check more point is more reasonable it's what's going to happen on the

network okay um I can start with checkpoint sync and make sure that all the nodes can get ahead to begin with and once that's good um switch to uh add the withhold data column flags and then once that's looking good add some full nodes and report back I guess rect never try the chainpoint think on Prison maybe it works maybe not but uh I'm sure that Genesis works either with empty database or either with a database already close to the tip it will work yeah I think it's a good idea to try

Genesis because uh you could at least uh like syn across three Three Forks um that I think that that thing counts and uh like there must be every client which should have like let's say a request response domain for blobs and then columns so that entire feature would be getting tested as well across the sync so I think that thing is important because I think there might be people like I don't know but there might be even us U even we might just ask um a post Fu blocks like a post Fu blocks blobs by range who knows like I'm not

sure if you're absolutely correct there so I think it's it's it's a good idea to do genesis yeah okay um I can maybe first try checkpoint syn to make sure that sync itself works and we have a fallback um and then I can use the same node again for Genesis syc and ping you guys on bugs and once that looking good we can switch to full notes okay um so I think that's most of it uh the second one was uh Jimmy mentioned testing distributed block building blob building um so I guess this is part of the withholding data

columns that you mentioned at at Nish uh no no no distributed blob buildings is I think it includes engine get Globs V1 Whoever has implemented that yeah Jimmy's message mentions lighthous is adding some test Flags to artificially delay publishing blocks in columns um that plan do you have some more info on this yeah I guess um I couldn't attend the previous one but everyone in the room agrees that we should include the theg proofs in the transaction so my my only concern with

that that's why I ask in the Discord group is if we change the cite in the future we would also have to change the the way the kg proofs are computed on the user end uh is that something that we are concerned about do we know what kind of user really use blobs I mean is it mainly 99% of the time is it L2 or do we have some standard user using blobs if it

is Major l2s I guess we can r on the fact that they will be able to do the needed work I think it's mostly l2s I think so then yeah I think that was my only objection to that but I see Jimmy's point that otherwise it's going to be a bottleneck if we scale uh quite High then I guess in terms of Thailand is something that we want to see on the release of b or something we could do in the next for could you describe what the change

would need to be don't think I fully grasped the change yet yeah so what what we're talking here is now now if you have to propose a block you get uh just blobs and commitments from from the El and it's your it's the duty of the C on the path of block production to compute the kg proofs for every single block on every single cell uh now that's a lot of kg proof so it takes a decent amount of time uh I don't know the benchmarks but maybe on the order of some hundred some hundreds of milliseconds so instead if the center of

the transaction sends the kg provs along with the transaction then that time doesn't have to be spent and the issue with this time is is if you want to do any sort of block production locally it's even worse if we scale to a very high number of blobs it's going to be pretty bad so now we shift the burden of this competition to the center of the transaction and we we don't have to spend a time is that sumary correct uh yeah I just wanted to mention also that that's correct just that it's not just uh local block production that that's affected but also distributed block building uh um like if uh you send out just a block and you're kind of hoping that the network will do the rest

for you if you don't have the proofs already in the mle you either need to have people pre-compute everything or you you need them to compute it in the critical path like I get your block and now I need to basically recreate The Columns which I mean is just sort of pulling the blobs from a m pole and putting them in column shape but also getting the proofs and so either you've like spent time to to precompute it or you just uh have to compute it at that moment um so that also kind of gets in the way of of block building kind of in general like like I guess both forms of local block building like both actual local block building and distributed block building um so I mean yeah we could do precompute to make it better but it seems like the I mean at least I

understand there's other complexities and involving the Y etc etc but um other than that it definitely seems like the right way for this to happen would be for it to happen at at the edge at the transaction Center level um then yeah I mean I don't have too much of an opinion on these other um complexities but I I hope that like if we choose not to do it it it is because we're really sure that it's not going to be a blocker to increaseing The Blob count yeah and I want to say from Lighthouse team we are very eager to see this spec and happening as soon as possible so we have a complete P spec uh hopefully this month and we can get the

B rolling so I think Jimmy is drafting a document now and he wants to present to alls execution because I think that's that's the first time that patas spills into their layer yeah so as far as I thought as well um Jimmy did want to bring it up this week um he asked to add this as a discussion Point um and the main open question was like you said who's going to work on the specification as well as who's going to work on e/ East implementation if we actually need to have it I think those were the two big points I think Jimmy

was ready to work on the presentation to um eleps but like you said this is the first time they're really engaging with uh pias in general so having as many pieces of information for them such that they can get started ASAP would be great has anyone looked at uh E's implementation on this stuff okay does anyone want to look at eeld implementation does e have mol related things I don't know I'm actually even not even sure if that exists to be

honest like I would look but I I I don't know what I've seen in eels is more like just the transition like that stuff I I haven't looked for the other stuff but uh if it doesn't yeah I don't know I can see if it exists or not but okay so I um I can also maybe ping Mario and ask him if eels or east is even the right place for this and if not um where would be the right place I think the intent is mainly that once the El have implementations that they could locally test it somewhere so that they know they're doing it right before we jump directly into inter okay so I will ask Mario about that and I've made a note for that uh

Jimmy would work on the specification SL making the case at ACD um is there anything else left for that topic is this still a contentious change or is everyone agreed that this is something that should go into here us as soon as possible so we can finalize the spec I me Jimmy also brought up in Discord this question of um more like longer term question of do we want to change the cell size and is that going to make it harder to do um I I mean my feeling is we might want to change change the cell size um I don't know if yeah I I I and I don't

have too much opinion on like how much harder this would make it I I don't think that it's something that we want to do like continuously like you know that every Fork we want to make like these kind of parameter adjustments so I wouldn't think that it's a good enough reason not to do it if we generally think it's a better design like I I think maybe might might want to change cell size want to reduce it but other than that I I don't know it just doesn't seem like something super yeah that should happen more more than that um not sure if there's other contentious parts to it um does that sound like something we

can then discuss on the pias testing chat and then figure out that um are there any other major changes we should be talking about before um we're talking about the next p or the next month is of theas I'm not sure if this is related to now or to the next topic but about validator custody I think in your respect franccesco you wanted to set such that uh public you don't announce the updated

cgc until you complete backfill my issue with that is that I suspect that in the real network with the final values that we choose for R Field syncing would be really slow so consider a note that has connected to it a lot of validators when it starts it doesn't have any information about this bators being there so it will act as a regular note then 1 minute later bors connect to it so it triggers the pack field process to become a super note with via validator custody this thing could take I don't know but maybe a week so during a week you would have a note that could act as

a super note to the network at least on the tip of the chain but it doesn't because it's completing backfield so the network loses one super note for one week because of this requirement I'm not sure how impactful that is in terms of the network but it's something that slightly concerns me I you could still subscribe to all of the um subet that you want to subscribe to but just not announce them which I guess in the current design doesn't really do too much damage because we don't have peer sampling I guess it would be a different story if we did but um

yeah I I see what you mean so I guess yeah the effect would be more for recovery on short Forks when peers need to get samples via RPC they would not ask you for them oh I see okay right yeah I mean on the other hand I guess so if we didn't do that and we said you just immediately announce it then can you end up just with people disconnected from you because they they try to get this older data and you don't have yeah so in this case if we choose not to

wait for the back fil to announce a new custody subil groups we should definitely not don't score and B ban a peer if the peer cannot uh send back the data Curr but that would increase too much of spamming on the network now I think yeah go ahead yeah it may a little bit but actually I think it depend of how many nod are actually back um currently backfilling I'm not sure at the same time there is a lot of Nod back filling their the their

blobs so normally this this period of transition for one n should not be too long and not too many NS should be in this in this transition period at the same time yeah this issue happens today you you can have a pier which is very useful for you has amazing band with great mesh score but it doesn't have the backfill data yet so you may choose to disconnect for it because you cannot backfill but it's actually pretty useful here for gossip so at the end we have to reconcile these multiple dimensions in

one number and at least Lighthouse would do that if we if we check sing and we try to back fill and you don't have the data we will disconnect you but we could be disconnecting a very useful here so ideally I think there should be like two separate pure pools one which has all the useful blocks like can serve blocks by range and everything and one who is the correct subset of columns for us like whoever is the local P who is trying to sync or whatever and I think the scoring algorithm should uh behave

differently like let's say a bad block by range shouldn't be down scored as much like should like a a bad column by range should not be Downs scored as bad as a bad block by range I don't know if that makes sense so like yeah Nimbus is right now trying to think of an idea like this where we have two different GE pools because again the one thing that you said that if if it doesn't have the right columns to serve columns by range it's still a very useful tier how would it how would it then work

if you have two PS cist or how how does that how are you going to be maintaining that it should just like be two different peer pools like with different logic or with the same so for bad columns and like let's say no columns the scoring will be very different the rate limiting criteria will be very different to like bad blobs or no blobs or bad blocks or no blocks so there will be a proper divide in both the pure pools

because it's like if we ever have a unified pure pool where we think that okay um this column doesn't like this spear doesn't have the subset of columns that we need um it's it's of no use no that's not of no use it's pretty useful to do everything else other than column syncing so that's that's the basic motivation that makes sense um I guess we know more once it's live to see how stable that approaches I don't think anyone else has tried doing that

right um not yet but I have been I have been discussing this with um Manuel for time um yeah like over over the last few months or yeah we we have a kind kind of similar approach where we have preference for peers that have data that we're looking for and kind of give them a heavier weight uh but it's not necessarily A gossip thing it's how we do our uh our peer management okay um anything more on the

top the other like um idea that one can think of is let's say you started on a podos devet on Genesis and like it's been eight8 10 8 to 10 days and now you want to sink one approach can be like you started syncing you didn't do column syncing at all you just sync blocks you went till the tip and while you're going till the tip U as a background process or something you are managing the pece who has like like the correct subnets and everything um so in this uh while you are like filtering out the peers um

as you are syncing blocks U basically you have every correct like every correct Peer who can serve you the right column and uh like there are odds like let's say the pure got disconnected or whatever but there's still a very good decent probability that once you have those spear filtered in a separate pool as you are syncing blocks once you start column syncing it should be pretty fast regardless even if there is sort of lesser supernes so that can be one approach of column syncing yeah that makes sense

um yeah I guess from my side once we try it out we learn more I'm still a bit intrigued to see how that would play if you have nonf finality and a few Forks running around cuz I guess in the happy case either approach will probably work mostly fine especially in a network with a lot of super nodes um the real challenge will be if you have multiple forks and you don't have many supernotes on your fork I think if you're if you're syncing blocks first even if you have multiple Forks um while you're column syncing logic you can just always compare the block header or or whatever and just

keep investing based on the correct block header in the correct slot like basically you you B you basically sync blocks and you remember the map Which slot what did you sync and while you start column syncing just apply that same map for the same slots for same columns like ask the ask the columns only for those same slots it shouldn't be like very complicated yeah yeah um any other topics that we want to discuss relate any other big changes I guess the sort of main root problem we

want to get at is um are there any other big topics or big changes we need to do similar to shifting computation to the transaction Sendo or figuring out um multiple here scoring of peer pools or things like that uh that would prevent us from really sprinting on pias post Petra basically if there's something we should be working on on the specification side of things that could happen earlier than later I'm not aware of anything yeah

okay that's good um then that sounds like it's mostly validate capacity distributed blob building and um computation to transaction centers those would be the three big topics we would need to tackle over the next weeks months and then just stabilizing the network right okay sounds sounds good um a yeah so there's like there's one point that I see from the last week's doc that is um if we were to like test um get blobs in all the CLS one thing we could do is we

could either um do delayed publishing of columns every slot or we can publish less than 64 or like less than 50% because like let's say if we publish less than 50% then every CL would be enforced to check da first and then it would be like okay there is no da but that shouldn't be the case they should reside on uh get blobs instead um to make the block available for attestation if that makes sense so um that can be a test of A Sort if anybody agrees yeah I agree and even publishing

um less than 50% is quite interesting because for sure normally all super node would consider the block as unavailable because they will be totally not able to reconstru the the whole block but some full NDS should consider still the block as available because if in the let's say 90 49% of the col publish there are all the needed columns by full node so the full no we think this this block is available and then normally it should be reor after if if for example it's the full propos of block just on the slot

after so it can be a quite interesting test to to run yeah yep what what we can essentially do is like if we could somehow make the full nodes not subscribe to column topics at all like for a while while the publishing happens um let's say we delay the entire process of subscription for like let's say 2 seconds into the slot um then the full nodes would be very enforced to declare the block is available based on what it sees in engine get

blubs technically but that would be a weird test to do yes maybe we can start with the withholding tests and then reconvene either in chat or in one of the breakout rooms to figure out how to continue that test stream and it seems like the lighthouse node adding the flag to artificially delay publishing blocks and column should achieve that or would I be wrong there yeah um

line yeah I wanted to add uh that we we are planning to add a few new flags for testing so we want to add block publishing DeLay So we temporar withhold and then release uh then also data column publishing delay and another one to do to advertise a false cost grab count group count such that today if we use the withholding the data is still available from from RPC so even if we don't have reconstruction the peers can just query the original publisher and get the data but if we publish from a note that's not

a super node and it withholds the data the data is never made available and such that the network can only progress if someone reconstructs yeah that sounds great um once you have a branch could you let me know and I can already deploy it and we can run around of tests yeah what would be the use case of delaying block publishing like I understand the use case of delaying column publishing but not block publishing exactly I guess more Network chaos I mean we essentially need to make

sure that we can recover from all sorts of weird edge cases right yeah and also people can be malicious validators if they want to be um okay cool I think then we have enough Clarity on tet 4 I'll write down a few more few of the test scenarios that we agreed on doing um we I I've shared the link to the deet five specs uh are we in agreement that we want to wait for defit 5 as in we get make sure everyone has validat custody we do the distri BL building stuff we

engage with El and so on before we do devnet 5 and until then we just continue on devet 4/ whatever variation of it it looks okay then I guess we just need some information and I guess once people start working on some of these topics we'll know how far out they are and if it's worth waiting for them or not oh and third topic was that there's no other big open research topics we need to talk about so we should be good on the shipping front for PS great anything else people want to discuss today yeah I want to give a heads up to

all the implementor teams that at least from our end validator custody is really complex and potentially the most complex part of the design so you guys should start looking into that as soon as possible it's just it's just an engine by the way I mean it's just from an engineering point of view it's it's just complicated I totally agree with Lion the spec is simple but implementing it it's not so simple at all uh do which clients have valid data custody implemented already uh probably

n to be fair it's a very recent like spectr yeah I mean we've just started looking into it as well and uh yes it's it's complicated because like um the beacon node and all the dcode things are like on one side and then uh bringing the data back back from validators that's that's another like it it just doesn't adhere to most of the design um system design in general yeah and being being pragmatic I don't uh I see the point of being uh feature complete as soon as possible but it's this is not in the path of

stability and even if one client doesn't Implement Valor custody we can still go ahead and for on it to say if if 70% of the nodes do have Bor custody that will be probably enough super nodes yeah that makes sense and I guess this is also something that we can deploy already on any devet we have live and we don't need to wait until the next one so yeah if if you Manu or anyone looking into this this if you have a

design that you are happy with would you share it on the Discord because I'm I'm Keen to see how do you deal with the complexity yeah yeah yeah I don't have any document but definitely I will write one and yeah also I I I will be happy to see on your side how how you deal with it yeah we don't have a design yet that we're happy with okay some um sounds good to me anything else we want to discuss today I see there was a lot of chat with

between the D plan and Francesco I don't know if you guys think it's worth surfacing that do you I leave that up to okay um I'll take that as uh anything else yeah K yeah uh so I guess the computation metric should be deprecated right if if we move everything to you why would they need to be deprecated right

now uh um maybe not right now maybe later uh but just want to draw your attention that um if you move to computation to Y because we have a computation metric now is this part moving to Y that's that's at least four months from now at least like in the least uh estimate okay and I guess there will be new metrics forid dat custody later yeah I think once client teams

have started implementation they'll have a better idea of what might be equipped to have a symmetric what not uh that will be great to share some ideas in the chat later if someone has any is it will speed up the process of metric synchronization yeah um I think who first with their design maybe post a list of things you would like to have as a metric so that we can just align on

that and we don't already have people implementing different things okay great thanks okay um anything else people want to talk about today and thanks everyone for joining and have a nice day bye everyone bye everyone than okay bye

e e for