PeerDAS Breakout #9

Transcript

e e Oho

hello e hello he

hi everyone happy perfect uh Square day uh start the call um okay that's start with the c update um anyone wants to start I see Reon yes hi everybody um so right now um about the P2 prism is still uh offline um we had some issues with um with granding which um published inv

valid um data colon gossip and in this case and this invalid data col gossip was also republished by I guess te I guess everybody but but lighthouse and in this case prism was because message was invalid prism was just disconnect disconnecting basically every Pi because PR P received 128 invalid goip message from data C uh so we deactivated the pure disconnection because of for that and also right now yes PRM has very strange issue when grounding is in the mix

uh Discovery V5 is really really really slow uh when prism tries to iterate over Pierce and it only happens when um rounding is in the mix uh we have a work on for that it's not a real fix but something which allow us to parp to the dev Nets and also prism had some issues when thinking um when the devet uh is not finalizing which is the case actually uh I have a fix for that I tested it locally and locally I am able to to to think again uh I tested I tested it for full N I want to test it

for super n before sending every before yes releasing a new version of prism to the devet um yep pretty all for prism thank you Manu okay then next one L star yeah uh so as I mentioned before as well as in the channel uh so lotar was restarted uh because of some uh configuration issue of

uh basically the persisted ident persisted pair identity parameter was not provided which basically put lar nodes into syncing but then it could not none of the lar node could sync for the issues that I have already raised in the channel I'm not sure whether other clients have got on to see it but uh I think few of the clients uh provided data column at some slot in which there was not a blob and then there was few client there were few clients who did not provide data columns which were supposed to be super

nodes uh because I checked and as per the G metadata they they advertised themselves as supernotes but they were not replying with the data columns so none of the lots are not could sync it uh I I hope other clients have looked into into the issue I myself have debugged the lotsar calls and lotar processing with lots are only nodes and it seems to be working fine so since this is since uh these are the issues mostly from uh super nodes so I don't basically see any issue of node ID or data columns

uh or uh wrong mismatch of data calculated data columns yeah I wrote something in the chat I don't know if it is expected but the boot there is a boot N is a network and this boot to advertise itself as a super node with a CSC of 128 um I that's yes it is TE our boot Noe is a te boot Noe but it's not s to head so for each of the clients I was

seeing issue I posted uh basically what where I where lots are got the error and uh so if any of you guys have debugged it sort of figured that out because I know that ages Nimbus has been debugging uh uh ners response and uh yeah and I also double checked on lotar using lar only nod so as per my understanding Lota code seems to be working fine gajinder uh next we

have here uh yeah uh okay uh so we have updated Ras kg and I have noticed that it's not parallel by default like CG uh you need to put in the Constructor parameter of paration and we don't do this in Te yet but I have had it an issue and I see in benchmarks that if it's par parallelized good enough like CG it's on par with c g I got the same timing

inle also uh we have greatly improved our syn uh we were sampling uh slot by slot before during syn and now we have dedicated uh sampler for for sinking uh we investiga in issue with when Tech distributed but the column side cards I have checked everything everything looks good and I'm not sure where it leaked but I have had get data column side car rest API so we could check next time uh on the Note what data column side car is in

database doesn't exist and I join the question I think man man ask it in Channel how how to how to reach rest API of the node at least your own client node and also we have Ed in our field in get Pierce we were ask it for it uh that's that's all cool um I can go next um um if

that's all um on Lighthouse side um we've identify a few issues on definit to um I think prison raised one which is the invalid Val columns not getting rejected by Lighthouse um I've confirmed that it's an issue and um we we'll fix it this week um and also there's been some observed um stuck lookup issues on death net um that's that's why a few of the lighthouse know got stuck with and getting a sync and not able to recover um L's been looking to that um not sure if that's been fixed yet um I think another issue was the um issue raised by

ginda um one of the LI house supernode is not serving columns um I haven't had a chance to reproduce it um so I think I'm going to give it a try sometime this week as well um yeah so a few issues on Deet 2 um um other than that um I think we're also going to update the Russ kg Library um K also told me that it's there's been some improvement in the computation time um it's been brought down from like an average of 200 millisecond to 150 milliseconds so that's pretty exciting um I'm going to give you a try and other than that we've also

released some testing metrics for p with 16 blobs um I've published a block post um can share in the chat anyone hasn't seen see it yet um but that's all from L house thank you Jimmy um anyone from nus here anyone from granding here okay um next let's goes to the day

update so think client have sh some no issues um anyone wants to add something so regarding de 2 uh we are not looking very good I think we haven't been stabilizing for over like five days and possibly that also causes some issues I'm not sure if any of the uh prism images will be able to syn up to head even when the chain is UNF finalized for

so long we could possibly give it some time till the prison team has a new image ready and then we could try to come back and try to revive this chain but otherwise we could do a relaunch of two spec and call De 3 and then possibly do the devet 3 spec that we have discussed before with the Reas on top of pectra devet 4 spec uh for a future perer deser net um yeah for prison so so as I said earlier I'm able to locally um s to the Head s to the

Head um for a full node for a super node it's currently in progress uh I think it's interesting not to drop the the test net uh even after long period of lack of finalization just to to see if we are able to to go back to a finalization mode yeah in that case I think we can keep uh the deet around for a bit longer and uh see if we can revive it and then possibly wait for Pure DEET 4 uh for it to have a final spec and then we can do the rebase on top of that for the

relaunch possibly two three or four weeks from now out I mean f the B side I'm looking at the death n planning document this one um so think we can include the two PRS to the next release in this week so that I

mean so that could be a temporate um temporary desate Target or whatever it is well and so I'm curious so I mean CS have problems that the interrupt uh but when it is uh in a Solo client environment uh did you did you still having this problems at like not serving columns things like that basically I wanted I wonder if it's

a interrupt problem or it's a networking and more complicated problem so is the question related to Lighthouse not serving columns so I missed the question no just generally like all the uh if it's a solo uh client environment I mean solo client Network um is is it does it work smoothly as we expected or we also facing this similar problems um we have actually seen that

um I seen that issue on a Lous only Network um but I I think we could do more testing I'm not 100% sure if it's if it's like a in problem or a um a lighthouse problem I think it might actually be Lighthouse but but um I'll I think I'll do more testing to confirm thank you Jimmy anything else regarding the definate planning oh

mention sorry shall we um Yeah you mentioned the um the pr to decouple network up nets um yeah I've raised my thought um so want to see like what other teams think um I'm kind of interested to hear what what people think of it I mean decoupling is is is nice but um I just I'm just not sure like what what the like where is in terms of priority because we also have VI CL violator custody coming up which we want to implement and I assume that's probably like higher priority but um I'm kind of kicking to hear people's thoughts on it would this change be um difficult for

Lighthouse um I haven't really look into what the change would be I guess it would be um probably the same as everyone else I don't imagine it being too different uh yeah I think I think the shame will not be like huge right yeah at least for us I don't think it'll be um be that big just looks like we would need to have a abstraction for the network subet but other than that I don't think it's too

bad uh is there any other reason not to do it or um maybe that say okay do you want to delay

the decision to maybe Thursday for the cor Dev score to finalize it or mean I don't want to wonder people have that line yeah I wanted to Echo the concerns that Jimmy raised of being uh an unnecessary abstraction at the moment because it's we are unlikely to raise uh the subnet count for potentially a long time so yeah I don't I don't really see the point on having to deal with it now I know it's not a big thing not a super strong opinion but

we are fine with what we have now so Trying to minimize complexity when it's not necessary Cent want to come in guess that's also depends on the def three timeline right so like do we have

more time to I mean refactor it um or we want to spend more time on improving Network we could also possibly just do the rebase on top of pure for DEET 3 and then we could do DEET 4 with new new features because I think the rebase itself is going to be a significant amount of work for some teams

um um I'm curious to know what's the um estimated timeline for teams to do rebase I think pris already got it ready um Lighthouse also has it but we haven't tested it um so there shouldn't be much work for us um what about um Tak and other teams Poco we need I think two weeks at least to rease on pecta what cler for

rease yeah yeah I guess uh one two weeks would be fine with us yeah I think uh pure DEET 4 will launch uh possibly next week and then maybe a week after the pectra de 4 launches we can aim to do beus de 3 launch with the same spec as pure ad 4 on the consensus spec wise unless we need to make a new release for changes I'm not I'm not sure if we're going to have any changes uh in P death

that will require a higher uh consensus back so um if def I mean Petra defin is done next week then I'd say we probably won't include Petra updates in this release um so that we can release in today or tomorrow or Petra and then next release we have mean

it'll be more um pure Des focused does that make sense there there's one more release uh scheduled for Petra right I mean yeah this week we plan to have a release for Petra this week mhm uh um so I guess H we can in I mean we can include this to your desk PS or not um but if it doesn't matter much for clientes to to welcome P

desk then and and we don't have to include it this I I I would actually include the 3893 uh just so we have the rebas in there and then client teams can actually test against the same consensus speack as for Petra 4 yep um we can do that um so just delay the decapo one so yeah yeah I I would do that I I would focus on uh just the fact base and then we can do the the decoupling in a future DEET when everyone is on board okay SS good to me oh yeah because this no

this trend will this release will include some pectra updates and so P definite has to follow pectra um status after this release this FYI be ready okay think we got some conclusion any objections before we change the subject okay um next um any

other expect discussion or open discussion we want to talk today I'd like to follow up on the validated custody change um I don't see any progress on the on the pi itself I was wondering whether that's ready for implementation or are we still doing further research on that um let me try to find the pr yeah this

one yeah so it looks like the last approval was in was end of August by Alex I don't know if there's any further progress made to this um feature um so I was wondering when do we want to include this all do we want to wait for later debet is it hard to implement this um it's um some yeah yeah I think there will be some work um I don't know exactly how how much work but um there will be some work it's not super involved but it will require some some work and some

testing and um but the thing is this is this is kind of like I think what Francesco mentioned would be required for us to ship P so U be would be good to have it completed and tested early I have a question about validator custody and more specifically about validator custody accountability um what's the point to implement validator custody if we cannot guarantee that the node which are linked to validator clients or to validators are not penalized if they do not custody the amount of colon they should I mean currently with this spec I

guess every valid every client would simply cheat and do not subscribe to to the correct not subscribe to more than the minimum amount of subnets even if they have a lot of validators attached to it there is no penalty basically not to respect this rule this validator custody rule uh so that's my take I'm not sure if that's a complete answer but we we only need a sufficient amount of super nodes in the network to provide security so even if some people cheat that's fine

we just need enough nodes to not cheat so yes if that was a widespread problem it could become an issue um but I think we can we can deal with a minor amount and I would imagine from the point of view of a large operator uh the cost the bandwidth savings probably do not offset the development cost of M the fork plus there is the um I guess honesty Assumption of operators just wanting to contribute to CH in a positive way I guess mhm

okay I guess that the the same argument can be done of why isn't everyone playing timing games some people are some people don't it's a it's a matter of um where every single operator lays on the trade-off of uh say G technical knowledge and capacity to execute customizations yeah but I think playing timeing games little bit different because um the worst case if you play timing games for you for you if if if you meet the block and so you have a direct incentive not to play timing timing game too much um about validator Cody if there is

too many people who cheat we may I mean I guess the worst case would be to finaly a block an available block uh no I don't think that's the case like if if we don't have super nodes the network could stall if we are not able to search the columns but I I don't think we will ever finalize something that's not available oh I mean back to the

pr maybe give people a week to comment and then we will revisit it okay that okay thank you uh any else you want to comment

or we will move on um any other topics to discuss today everyone yeah sorry go onl uh no please please you speak first yeah I just I would like to discuss uh metric specs uh if you had any chance to look through the pr um I will um I'd like to share the link for the dashboard if you have an access um to it pands

grafana um you can check this drop dashboard uh so you can see some of the metrics uh for liouse sorry yeah K would you like to share screen for um I can't see the uh I can actually but yeah I can't see the button to share the screen do you need oh yeah I see okay okay um can you see my screen

yes yeah so uh we already have the draft dashboard and you can see some of the clients on uh One dashboard uh so you can compare the metrics um and uh you can filter um by the client uh for example Lighthouse but these are only a few metrics uh we have um they are uh the names of the metrics are uh from this uh PR we're trying to discuss uh so if other clients already have this metric but maybe the names of the metrics are different so you can just

rename them and you will see them on the dashboard um and you can compare with the other clients for example or check anything else you want so I would like to disc discuss the prefixes of this metrix we already raised a discussion here in the pr I will share the link uh but is it um is it a problem uh to rename um metrics for example um because in every client metrics are implemented differently um for some clients it's easy just to rename a metric and for other clients it can be for example enumeration for

prefix so um I would like to know what you think uh of it and we have a three types of prefixes um bacon uh Gossip sa and lip top RPC so this the latest one lip top um there was a discussion should we rename it to rec rasp or not um so so I'd like to to know the client's view thank you Kya any comments hey so as for uh renaming I

think at least internally in Lighthouse we consider the remaining of metrics are breaking change because it can break the the the setups of other people both alerts and dashboards so that's going to be complicated but that's one of the hardles of any standardization effort uh these are the new metrics yeah if they are not used in production then it should be fine yeah we definitely we discuss yeah we discuss like P P Matrix only um okay then should be fine yes I think there also some like gossip um

gossip metrics and some of them could I'm not sure but I think some of them potentially in the LI P2P library for us so I don't um if if it is then it will be a bit more difficult to rename but uh I think uh yeah probably yeah I think it would be useful

to try to identify the key metrics that we want to standardize um a few of them things like gossip bandwidth would be um I think it would be interesting to see if if all the clients have the similar issues like having the the metric collection in the library itself um but for p specific metrics I think it should be a bit easier should P specific metrix be prefixed with P name maybe in that way we can like nicely identify P the specific metrix I think that that will not scale well because we then we have to do

that every for I mean the general name of P death like it it would not be a breaking change or or what you mean it would scaleable but I think the convention normally is just prefix with the name of uh the binary or model of the thing so in this case would be uh the propos the proposals that we had either liquid top RPC Pon RPC something like that uh if we mix now specific

feature uh once we move to to Tas or some other feat then we have toist that uh nclor usually we do the model so Li to prpc and then the metric that with timing so data columns uh bites per topic things like this so I'm not I think adding I'm not sure how we could put p with respecting the C naming convention so we can currently leave it as it is uh

the naming of this metrix uh is it listed now in PR I think if you want to come to consensus on these metrics it's best to do it on the pr uh because we don't have full representation of the of the scull of everyone in what so we are uh leaving this PR open for now right because um for example currently I'm trying to implement some metrics for clients and I use these names uh I have

in this PR uh but um as it is still not in the production we can rename them later for example or if if we agree on something else on prefixes um yeah I would prefer to not have to rename down the line so it's best to do uh good effort now to come to consensus on what makes the most sense and just commit to it yeah I think sorry um I think it

would be nice if we could break down the pr just to include the the P specific metric which is the first section it contains the dark col kg custody Matrix I think it would be nice to have that first because that those are all new metrics and the remaining gossip sub and request response metrics are kind of um existing on so some some clients might already have them implemented so it will be hard to rename them um yeah I think we should probably I think those one will take probably longer to come to consensus but uh the first section the dollar columns it might be easier to to get them done yeah yeah

I would say as an advise for you Kya this is going to be a difficult effort because I think clients already have metrics and probably we all have conflicting uh naming schemes so I would recommend you to uh do Outreach with every single client and maybe draft in a dog the current situation with everyone and hopefully find the least um the path of least resistance in to something that's somewhat standardized if possible um got it okay and uh um uh what about uh other clients uh like Nimbus and uh

prison do you already have uh some metrics do you use them for for p d specifically yeah so I posted in in the channel but I I think for some of these metrics we have it in a different name so it's going to be difficult to standardize this because U these are like generic prefixes for all P2P messages you know across all topics so know having it different just for pias is is kind of a weird case but um I think some of the much newer ones like you know reconstruction time and stuff like that

uh that we can standardize on but I don't think we can s sanitize on everything um yeah we can start from p d matrics at least and uh look what we've got um I think it's a good idea to create um to to split this PR in in into small PRS and uh probably then you will raise issues for for your clients um uh so yeah we can we can try to build a summary dashboard and compare the metrics thank

you got uh could I ask all the client devs to make a comment on this uh PR or possibly in the new PR with the smaller changes and just let kcha know what is the corresponding uh metric name otherwise it's going to be a huge battle for kcha to find those out for every single different client uh especially because everything is written in different languages so it would be just a huge help uh if everyone could comment look look through the pr and and see which metrics you already have which ones you don't have and the ones you have and you don't want to rename uh uh Ju Just so we can create a dashboard that would work for every single client

even if it requires a different name uh we can just make the dashboard a bit more uh complex but that shouldn't be a big deal yeah and I'm also going to update the curtois dashboard for pust metrics so uh in this case if you run CIS locally or uh you will see the dashboard at once for standardized beard metrics uh so it probably will be helpful also lion you mentioned that you don't want to introduce or you do a rename for specific matrixes uh matrices but would

you be able to consider to add an additional metric for the same value under a different name I'm not sure our code base would support that we would I mean everything is possible but we would have to yeah architect something custom look a bit ugly um okay could be done but if we canot do that that would be best yeah I think we can start from pust metrics for new metrics and uh we'll see the result uh how it works together um

all the metrics for clients and I mean at least before renaming the P2P Matrix uh we can start from P Matrix the standardize them um I think with the um existing metrics that teams can't rename like the RP see bandwidth and the gossip numbers I think potentially there might be some grafana features to allow you to map them into the same metrix but I'm I mean

I'm not an expert but I think there might be ways around it without having to add the metrix into the client code or even at Prometheus level yeah I can check it actually that's a good point new fresh new uh new bottle forri

pick yeah sorry uh I wanted to bring up again uh this effort led by Michael and Jimmy if you I guess you are all familiar with engine get blobs B1 uh in the context of the current discussions of distributed block building and raising the block count I think it's an incredibly important feature that we should prioritize and it it has a lot of intersection with uh Pi D at least I think in maybe Jimmy correct me if I'm wrong we have already Incorporated uh engine get blobs B1 into our pias block

production flow so I wanted to know if other clients have even implemented this for den and if we should start incorporating or testing it for uh p as it can influence how much can we push things without screwing up solo stickers yeah uh so for prism we've been discussing the general design on this um is definitely a priority but one thing holding us back currently I don't think all the execution clients have like implemented this if I'm not get hasn't so it's a bit more difficult to test on our end but yeah uh

I would agree it's definitely something that we should have sooner rather than later yeah I think it will actually be great to ship that before we get to pias um given that potentially there might be a lob increase in I'm sure I'm supposed to mention those those words but um if we do increase the blood then we definitely want to have that just to be I mean just to to to keep the network stable um even before P so I wouldn't I kind of I think it would definitely help help p a lot more but it would be nice to even

consider doing that before that yeah I want to say I don't understand why this is not prioritized more it's a incredibly useful lwh hanging FL that we can do to significantly help Network because consider to the other options that has been discussed uh for solo stakers uh such as ADD think a flag to the execution clients where they uh say do not do not ever produce a block with more than two blobs or do not include blobs at all or only include blobs if they pay a specific fee uh if the execution client is smart enough and with this feature only includes blobs that have been properly propagated to

the network to the home Staker it can include as much blobs as we want with zero done with cost which is exactly what we want um not have solo Staker sensoring the network but have a backbone that can include as much blogs as as necessary and I don't know to me but it's just the perfect solution so I really hope that we can prioritize this as soon as possible uh do you know why the pr has not been merged it looks like it's been open for a while well I guess it's probably the same issue that K is facing when had to come to

consensus we just need someone that has merge permissions and then they can merge it it looks like it's already been implemented by three clients though I think um on the E side um and it looks like ethereum JS is also got a PR open so maybe just GI that's that's left yeah thanks thanks thanks for thank you

any comments on it if not um any other copies to discuss today has anyone did the 7742 thep 7742 to uh be able to send different number of blobs Target and uh and Max to the E so is that in any of the defet targets

yet not yet but we could possibly push for it uh I heard that Mario is going to implement it in G and then we can start testing that I'm not sure when it's going to be done from the side because it sounds like it's a bit more tricky to implement it on the El uh I'm just curious if any client teams have uh taken a look if there's any problems maybe with it we could also Target for like devet 4 regarding that is the idea to include the new uh

maximum Target in the config by the way or how are we going to change Max because if if we do that then every time we want to bump it we're going to need like add a new prefix or something for each Fork right um yeah both the configuration you for this conven SP the prefix is required are we going to have like

version one version two version three or or what is the plan or we just call it by the name of the update and then we do Max with that update like pure or whatever the next work is going to be called I guess it depends on if if Petra will update it right I mean in the end we we s it a way to to increase this down the line so is

it just going to be like the name of the update with Max blob and Target blob because right now we just have a single value without any prefix right um so do you mean that we should add this uh configurable parameter in the uh PS B right now just for most yeah would be compatible yeah so right now we have Max

blobs per block and we don't have any flag for Target blobs per block so it would be good to have something like Fu Max blobs per block Fu uh Target blobs per block and that way those values can be read uh by the CL teams and can pass that uh assuming that 7742 will be included in full lo I can make a draft V of it to see if make sense for the for yeah

or is there any other way we could do this other than set it in the config because this this is going to add a bunch of different values if we like want to bump this every single update but it it's supposed to change at the for boundary then I I mean for consens speak I don't see there

option but client might have Fier configuration setting I don't know any anyone wants to share your client impementation no okay uh okay I will make a drop here

other then we will see how many lines we will effect at least found the six okay so they were bit running out of time before we end this call I want to quickly share um updates um last Rudolph and I recently looking uh looking at how to know improve P des and future for project management or um so we

brandstone some ideas this is a link to know describe some like basic things we can do from scratch or for one of this to create a g Des contributor list and also we can try to make you know collect the G Des reading materials research in a more structured way um hopefully you can make a like a website or notion page like work code.

info so um assist the initiative I asked some people to to fill in the the contributor list and I will sh the a form later after this c um yeah so if you have any comments uh please let me know or can share it right now but we are running out of time yeah any store time Le

bance okay thank you everyone um think we let's call it day see you soon bye byebye thank you bye byebye bye bye bye