PeerDAS Breakout #15

Transcript

e okay

e e he hey hello everyone hello hello so I'm not sure if sh is going to be

making the call today um Jimmy did she reach out to you by any chance or should I be leading the call oh yes hello um yeah um she did mention that she's going to be um away unavailable today and next in the next break breakout call um yeah um and I'll be helping to um just to just to moderate these two calls um so it looks like we've got enough people here maybe we should get started um yeah so this is uh pet break room for those that just joined um I think we've got okay we've got few teams here um maybe we can get started

with client updates um I can probably start with Lighthouse um we've made quite a bit of progress in the last two weeks and um I think we have pias definite full um ready I mean at least for interrupt U we've got P activation at Fu and we are running the latest Electro code um we've got something decoupling working as well um yeah so um I've tested like local de Nets um and it's been working pretty well but um I've also tried interrupting with prism and um unfortunately uh when we transition to fulu um we we're getting a state rout

mismatch so I'm still looking to that um yeah so that's pretty much my update I'm going to continue to look into um the state R mismatch issue um and maybe we can talk about more when man talks about his prite test scenarios um yeah that's that's it from ligh house um okay who else is here I'll pass over to Manu yep give hello everyone uh so during the last two weeks uh we implemented the custody group count in place of custody subet count also we work on the M of the C library by the C goaz

library right now the goaz library still needs about two times more time to compute uh the C steps uh it's probably because the go does not use the pre compute why the C use the pre compute and Kev is working on on that uh also we started to imp mon uh validator custody uh yes it's very early right now and uh so yes uh I will try to investigate why we have this state truth

mismatch with the lighthouse today uh and that's all for my updates for prison thank you maybe uh uh take yeah uh we have not finished uh all needed updates so we we have transition to full but uh we have not completed uh decoupling uh decoupling custody groups I'm currently working on it uh so I think uh I will finish it very soon and start start testing uh that's all thank you

demry um pass to lar um maybe Matt or someone else lar y uh so got stuff rebased to fulu uh had a bug in testing uh that took a little bit of issue uh to resolve but it actually wasn't in the flu code it was uh somewhere else uh so got that resolved uh but it stopped us from finishing the um decoupling so we're going to continue working on that this week nice thank you um have we got gring here I I can speak Paul uh for Grandin

so Grandin right now uh we already uh moving the P code activation into Fu and uh we already implement the the couplings uh Network subnet and right now we are um try to stress test our reconstruction flow because uh right now there are some a few thing that uh seem to be like a Baki code so we try to stress test our the Reconstruction flow nice thank you yeah thank you have we got um n don't see anyone

from Nimbus here right okay it looks like don't have nimus um we can talk about um devet um we don't have a defet yet but um I think last time we agreed on a um devet spec and everyone's been working towards it um there's anyone have any comments on the current DEET spec um are there any PRS we should be tracking or adding to this by the way I'm not Weare of any any PR I think

the only thing that I know that's work in progress is V custody and it looks like um Manu and Justin made some progress on the B custody PR um um yeah I think I think for the B custody is currently optional and if teams want to work on it um we're free to work on it because it's compatible um do we know what's the status of that PR does anyone want to give an update yeah the p i just uh I did some modification because was working with u validator

is a sorry custody subnet count and not custody group count and yes just the modification I did is to modify to to yeah to to move to C group count but apart of that I don't know okay in that case I've updated the consensus spec version to use 1 15 beta 0 so that should be the last consensus spe release that's happen and is compatible with backet 5 as well in case there some missing PS then let IJ know and I think we can make a

release unless I'm mistaken there is no the test the custody group test in beta0 there are in Alpha 10 but not in beta Z ah got it would you prefer then pin Alpha 10 I mean I don't know if there is other differ that this uh test custody group test what what I did in PRM I just I mean I I use beta Z Plus the plus the test that's all I have no preference okay um maybe I can then text Justin to have something released for the pias folks so that there's one version or One release land that is

easier to track nice great thanks um is the um the custody group um spec test part of beta zero or was it part of alpha 10 they are part of Al 10 currently not beta Z okay okay great so there's not there's no change in beta Z okay cool cool nice um so it looks like most team are still working on um subnet decoupling um and we're probably not ready for devet until

another another I don't know how many weeks um does does anyone want to give an upd there I think I think I think at the moment Lighthouse and prison are ready for interrupt or we're running into issue but we we continue to look into that um I think looks like teu Loa and Gran all working on um subnet coupling and fulu um so do we think maybe we can like talk about this in the next breakout call or do we want to talk about a date seems like we probably need to get some pred definite testing going first right yeah I think I'd be more comfortable if we did um integration testing uh kosis style before we really

go to devet cool sounds good so we'll continue to work on that yeah one thing I do want to mention is I think over the last day the Standalone prism DEET also yeah like 6 hours ago uh started failing so I can do an enclave dump and send it across but it does still look like the four the five to six day Mark is when Dev net start to fail this is prism with itself so there's no interr complexity there's nothing else this is just yeah can you send logs yes I can do an enclave dump and send it across nice

thanks uh but that also then sounds to me that even once we have uh or the moment we have a lighthouse uh prism interrupt we should just throw it on this kubernetes cluster and just leave it alone for like four five days and as in when more clients come in we just keep increasing this number and once we have some confidence I think we're ready for a public Tet sound good thank you Perry um any more devet updates or um otherwise we can move on to spec discussions if there's anything yeah about the testing approach

I I have some uh document I want to share after par yeah that would be awesome um just mainly seeing if this it feels like we should change some approach we're taking five days to figure out that something's PL and feels weird yeah yeah I agree with you uh maybe I don't know I have to check but maybe you have some memory leak should be the most uh potential issue because yes you know five days for a memory leak is still possible yeah um yeah if it's a Memory leak then I think I can still live with it I think if it's more of a question of

some sample came in some weird order that triggered a bug that feels a bit more scary cuz that feels harder to kneel down definitely yeah K uh yeah hi everyone so uh I've done with the prison metrics uh if Manu has some time to check it uh probably before the devet we can integrate them and uh we will have Lighthouse and uh prison metrics for the devet thanks sure I I will check your request today thank you uh one more sort of add-on thing cuz

uh manuji posted the um test scenarios we have this thing it's just a make file in a script that can do a sync test so just modify the curtosis D config yaml file to include the pias config and then you can do I think make sync test or make run and what it does is it spins up the network um it waits until a defined Epoch it turns off one node then it waits a defined period of time and then it starts the node back on so it makes sure that you're syncing back to head and if you syn back to head then it passes otherwise it fails um so this should cover scenario 2 and it should mean that you can also test this with

just a prism Network just a lighthouse Network then do cross and yeah increase the coverage space thank you can we somehow adest reconstruction or is just a local storage um is there any particular scenario that you want to test cuz reconstruction should be happening pretty much like all the time or like every once in a while on super nodes oh okay then then okay I have in my document I have a reconstruction

specific scenario it's scenario number four okay great where you uh let me repass the the the Link in the chat well yes you you are sure that uh all your fix no does not uh does all the F do do withhold data colums on purpose when proposing a block to force the tested not to rec constrict and Reed missing data columns it is scenario number four scenario number one just a standard

word nothing special and scenario number two and three are scenario when you want your Tes not to be able to think uh from Genesis up to the tip of the chable yeah thanks and the last question uh do this uh custody group somehow um changes or custody metric or not I don't know Jimmy maybe you know ah maybe I have to check soorry which uh which metric the custody column number it's the last metric uh I will

quickly share the link is that the the metric that counts the number of custody columns in the database yes yes exactly last one making custody colums count sorry go sorry yeah I did that come in because I was curious if we have a standard metrics for tracking how many

how many columns and noce custy or how many cust group The the no is is um is assigned to um I think we do have some M on that but I don't know if we need to standardize it does anyone have any thoughts on that maybe yes we can Soliz a number of custody groups uh especially with custody could be a useful metric we can try on pris then and see maybe Lighthouse well okay I will look through that thank

you and so the idea of this pureet scenarios is I think before it's always quicker to debug uh debug stuff in a local curtois devet than in a real devet and so if ideally each Cent team could run all the scenarios against all other client team and so to get the Matrix uh all green normally if everybody does do that at the end the official devet should run ideally without an issue hopefully

and for sure it's it's faster to debug when there is some issue on your local computer than than waiting or something to fail on the on the devet so man are you saying that um we can we can basically look at the scenario you listed and update that table that you have on your document yeah yeah yeah for so there is one table per per client uh on the Crome table for example you can see that prism pass the scenario of pass all the scenario with prism and prism does not pass the scenario with lious the scenario 1 a and for all other scenarios we don't know

yet and yes if each client team can have this the same uh table and ideally with all the green green check check marks it's can bring some confidence before uh launching the next devet it's of course it's optional there is nothing monor yeah nice um I think one the one thing I noticed when I was doing local definite testing is um whenever Fork happen it's usually like at least for Lighthouse it's usually something to do

with with peer scoring um like we usually F when we when we like ban and disconnect a peer um so I think scoring is probably something that um that we probably want to keep a close eye on um we fix a few bugs in Lighthouse regarding scoring so hopefully um Can Survive a bit longer once we get the interrup issue fixed all right great um does anyone have any open topic any topic for discussion good

K yeah I just I would like to make sure uh it was the here do break room somewhere before the where we decided that we don't need a qg single verification metric and just to make sure if anyone agree with that uh so I will finally delete it no one use uh single K verification uh I think I use single K verification when I receive a colon from gosip or we can try um to put it on the

dashboard for those clients who have it I don't know if it makes sense but we can just try to monitor probably it would be helpful for someone I don't know like uh not necessary for implementation but for those who have it I think light house also uses single PG right um yeah I'm not 100% short okay that's it thank you

right cool looks like we have a relatively short one um and it looks like we're on track now to um at least once we have custody group count implemented and F activation implemented we probably in a good state to try to stabilize sync at least and interrupt bit more um and then um I guess the last bit to to have a production ready pations about a custody um I think we might have some some more discussion maybe some some more next week because we also just started looking into the ver custody design as well um it's like from the

from our first look with um with Lion it like we think it's a bit um there's a bit of complexity there because we have to basically like dynamically adjust the um the custody group and dynamically adjust adjust um the count so it's it's a bit tricky um I guess there's a a few scenarios so I think what we're thinking is maybe we we could do something like only allow it to in allow the custody group count to increase but not decrease um has anyone have any thoughts about that um why not allowing to decrease I mean it's it's simpler to allowing to

decrease and to increase because if you want to decrease you don't have to to backfill all the blobs you just need to yes to decrease it and to sub unsubscribe to some subnets and to announce via metadata and and not record that to custody less group than before I I don't see the difficulty [Music] to to reduce the custody group count I do see the difficulty to to increase it but not to decrease it yeah yeah I guess I I guess the the the challenge is when you update the the

cgc in the the metadata to your PE um you tell your peers that you you custody those columns right so so then you have to go and backfill um like that like those new columns for the last like 18 days and then I think dynamically dynamically changing that is is kind of uh uh you basically have to man manage the backfill process um if if it happens frequently then you might run into issues I think I think I think it's a bit of complexity to manage there but um but we haven't look deep enough to um to have a solution yet and just a question

um what method do you use for the beon not to be aware of all the validator Cent uh he manage do you use I guess uh big beon API call from Cent and what method do you use um I'm I think we we use an mpoint um that whenever we we attach our Val data um it it sends a post request to the beacon committee API okay I think um yeah we used to do that sorry the beon committee

subscription yeah I think so I think we used to do that for ATT station subnet as well um but it was removed after like it was removed the last year or something but um I think it might be that end point but I'm not 100% sure cool I wanted to to use this as well okay great um anyone else have anything to talk about before we close the call nice um so I think um our next step

is basically all team trying to get get dead net b um working locally um with his with their own client then try to interrupt do interrupt testing with other clients you hes and try out my new scenarios and then um we can start doing more interrupt and hopefully we have a definite in next couple weeks um yeah that's all for today um so we're finishing early thank you everyone see you guys in two weeks thank you all thank you by thank you byby byby