PeerDAS Breakout #3

Transcript

e e e

hello Ione hello oh is uh what are you doing here Antonio I mean I mean I in order for me to understand the more the better no oh I see I didn't know you attended the these like breakout room calls I do not usually but again for me it helped me understanding context right that makes sense are you keev by the way uh yes I think I'm the only one who's I change it yeah my iPhone just

usually says iPhone [Music] um I have I have the same issue by the way when I connect from my from MyPhone yeah yeah I'm just going to leave it I don't really know how to change it on my phone um but yeah this should be good sh hello hi

K hey uh and Mel you are an iPhone you know uh yeah my phone doesn't uh have the name when I join by Zoom I don't know why to change Antonio hi did I pronounce it correctly hi show yeah

than hello everyone thank you for joining the

Cod I just posted the genda and meeting notes in the chat so um let's start with the Cent updates anyone wants to share the progress in the past two weeks pron yep um I can start so for uh the last two weeks uh yes we added a new PR where data cols are activated from the e 7 5944 aoch

before data columns were activated at denb Epoch it's not anymore the case uh also about sampling we did only verify that the root and the colon index was corresponding to the expectation when something we not check at all um the KJ in inclusion proof or k side car proof and now we test them um Al so yes we did some unit test for robustness and uh currently we have some issues with on

p d testest one where we cannot think past given point and we are working on it um yeah basically it is updates from the last weeks thank you and then Lighthouse hey hello everyone um um most sleeping out last couple of days but um last week I spent um most of the time working on sync um we fixed an issue with our RPC limit and previously it was preventing us from serving Dooms by range requests and that's working so I

think I believe ligh house was was serving the by range request uh for a while until it broke um and then um I've also got Lighthouse to sync um work on our Branch but we haven't deployed it yet there are still some minor issues that we wanted to fix before we deploy it there it's very slow at the moment so we're still looking to sync um but in terms of serving data um my house is able to serve data now um but um if the latest defet one is currently broken um um because um the change with disabling the inbound rate limiter um kind of review an review a a bug in a sampling process so one of our

Lighthouse uh superno actually ended up sending a large number of sampling request with the other super node um eventually causing that that one to be out of sync as well um without the rate limiter so um pan from our team's been looking to that um I think that should be a pretty easy fix um and but we're not sure if if we we still want to revive um DEET one or just relaunch it um want to see what people thought are um the was uh re-enabled by the way so I was hoping that we would see something else but we just have too many Forks at this point I don't know if

we can still recover from the stage yeah um I think it's possible but it probably require a bit of pack uh because we still have all that data been active um it might be easy to just relaunch but um I think I think depend on what everyone's preference is because I think some people are still debugging issues with sync so maybe want leave it on um not sure but um yes today I've been trying to use um a tool um a manual HTTP sync tool that uh Michael from our team creat it um it's it's worked in the past but I haven't been able to get it to

work on lighthouse in DEET one um I could try to run other clients if we want to so what it does is it basically um downloads blocks from a an up toate note and then push those blocks to to another node and which which in turn will cause a big um reor so then they'll eventually be on on the same chain um it's kind of a hack um so as why I was thinking it might be easy to to relaunch um but yeah we can can try if you want to um but on the lighthouse side we we're we're looking into fixing the sampling bug um other than that we have um tested CS Russ kzg Library um which is like a

like a like a replacement alternative to ckg um it's worked pretty well and I've done a bit of testing on it um the metrics so far seems pretty comparable to ckg at a high level um when the conditions great great so so um we're looking to the moat as well um yeah I think that's pretty much it from Life House side um we'll continue to focus on getting sync to work um and to just try to stabilize our our D's Branch um yeah that's it from my side thank you B yeah Jimmy uh you mentioned that the

rate limiter Bo could be very easily fixed uh would that would it be possible to get that fixed ASAP then we would relaunch with defet 2 let's say because this is yeah it's it's hard to recover from this point and then uh because this rer really limits uh capabilities of other people thinking to do themet because we don't have too many nodes so other PE are able to serve the state but uh it it's just not fast enough I see um I was wondering if we could like kind of raise the rate limiting like the rate limits are to

like a very very high limit um instead of just disabling it completely just in case if if yeah again um we might kill it again um so I think if we don't set a limit there's still a chance that we will kill it so it's better if we just raise the limit to to a high enough limit high enough for people to download stuff but yeah but we can talk offline um I can send you the flags needed to raise the limit Jimmy any other Cent updates L

star yeah I can go for L Shar so uh basically last week I was able to syn lar to the deet one head when basically the apis were working and I was also able to stay snc stay sync with the Hedge for some time and uh was basically using supernes and uh right now yes I have not baked in any validation slash proof validation uh even though so but but uh what I'm trying right now trying to do is basically change the sync mechanism so I can also sync so that lar can also sync

from from non supernodes and uh through our various mechanisms that we have uh rain sink by root and uh when we don't have the parent so we do some unknown blocking AR which uses the root bus there are some mechanisms that we have to pull the blocks and the data for it so I'm sort of getting it uh working and then load Stars can basically take part take part in the defet itself once that is done um I believe can also very well serve uh serve the data columns because uh I mean on the

local test net I'm just using l l to lot lot combo to uh do all the debugging and to fix all the issues uh and uh so if basically this this goes well then I will also sort of integrate the validation and grou uh so we can basically then participate in the D itself thank you any other CL update okay then um let's talk about the

death Net One um I I me I think one of us to give us an overview yeah sure so last week uh when I discipled the the rate limitting that's when really the whole netk just went haywire and uh right now we are participate we we have six different Forks so each client is sitting on its own Fork uh it's let me send you to the URL so we have prism on their own Forks lighthouse on its own fork and te seems

te is the only one that is able to agree with itself but every other client just seems to for away even from its own uh kind implementation so I think a relaunch is uh worth it uh we can probably do that once we have a rate limit a higher rate limit option from Lighthouse and when Lo star is ready to be onboarded maybe end of the week sounds reasonable yeah will be ready by end of

the week to join the 11 Jimmy yeah um um I was just curious um got a question for prism um because I think prism's supernote also died around the same time when the lighthouse supernote died so I wonder if if we know what happened there was it because because if if Li housee somehow somehow does the prism node um prism would probably have some some sort of rate limiting to prevent that from dying so does anyone know why like what happened to the prison supernes I think it would

be great to have like at least two clients supernote working before we do a relaunch because I think having just only one one super node client is a bit risky no we should we should have de and prism super nodes enabled as well so we had six Super nodes not just Lighthouse yeah yeah I was wondering what happened when when Lighthouse disabled red limiting did the lighthouse no kill all the other supern as well um so I I think for prism we have introduced a few changes over the last week to our limiting strategies so I think when Lighthouse died it uh kind of triggered a few

bugs so I'm trying to track down the bugs right now so that um we would effectively be able to serve you know all the columns all the nodes rather than simply rate the everyone hopefully by the end of the week uh we have the fix in okay uh yeah just trying to understand so each client is on their own fork and is it because of a different reason or is it that the rate limiter got removed and then um they just

started disconnecting from each other um I I think we had a few bugs so one of the bugs is our rate limiting strategies so we have these custom limits for data columns and I think the other bug Manu is trying to track down maybe he has more info on that actually a bug I'm trying down right now is a bug about thinking when you start a new node trying to connect to the existing Network um so it's it's a different bug oh okay right uh but I guess did all of

this the fork start when the rate limiter was removed or were they on their own Forks beforehand it it happened when the r material was removed yes okay I see um and okay so this implies that if you have a network with just like White House nodes or a network of just prism nodes then once the rate limiter is removed then it it will also Fork away from each other and it's just a teu noes that don't do that is that correct that's what it looks like currently yes okay I see uh is there anywhere I can get more Contex context on the rate limiter issue if it seems to affect every node

yeah I'm actually not sure what's the root cause of it um but I think when Li housee got restarted one of the supern not started sending out like huge number of sampling requests um and and that kind of um overwhelmed the other lious supernode so that means both house super not like stop serving other stuff so I don't know if that's the reason why the rest of the network um started having issues so it basically lost the two liouse at the same time um when when it was turned off um I think it's not really a direct issue with the r limiter it was just a uh kind of review a a barain sampling um

code right right cool yeah I mean at least yeah it can be sort of reproduced of just like Lighthouse or just prism Lo which means that it's probably easy to test locally in debug I think or at least easier than sort of testing across multiple nodes I think it might have also been uh because of the enr change there was like a some enr issues uh post the launch of the different clients and then we rolled

out an update to Lighthouse and with with the fix and I think that also had some issues uh yeah another thing I wanted to um to raise I think maybe we have a spec discussion uh um session after like later but um kind of related it um eni is not always available so it means if other nodes connect to us we don't know their supernes so it's not a super reliable way to identify supernotes so um Li's got a p to consensus back to add a um metadata V3 query um maybe we can

talk about this later but um I think that would help maybe yeah I'm not sure if we want to make this part the DEET to um spec I think it might be useful so uh Wonder so if we want to restart I mean relaunch definite to the s or we have like more time to add more features to it or

not let me uh so jimy you mentioned that having extra request response API is nice to have or it's a like a a must here um I think it it depends on how um nodes are like depends on how clients are are done identifying supernes I think heu looks at um looks at peers are subscribed to the subnet um I think that would still work if we're purely looking at enr and if

enr doesn't if enr is not available we just fall back to the default then that's that might not be enough which is the the previous behavior in Li housee um so if other clients are all just looking at subscribed um Pi on the on the subnets then then that's fine um I think we can live without having the metadata query for now is it uh 3821 yep how much implementation would this

add for client teams it it wouldn't be too much but it would be a bit of work so um if we do have it as a requirement for DEET 2 I don't think we could launch it by end of the week yeah sounds reasonable I think I think I think having a DEET to test is probably still like a bit more useful so

maybe we can leave that to the next DEET um if we can launch relaunch a new devet next week okay so let's aim for to relaunch with the same spec as the previous one with all the client fixes in hopefully at the end of the week and we should probably include loar if they ready as well that's

good any other thing we want to discuss with the def if then we can start with the SP discussion so we have two items on the genda first one is that Pap has new proposal yeah uh the The Proposal is to decouple the the the network subnets

from the the core protocol like uh for exle when when we think of the editation subnet right uh we have the communties it's like uh uh for when when you send thetion when you broadcast thetion you send it to the committee right you don't send it to you you you you you don't think that you send it to the sub you you send it to the committee and and uh and the community is just like an abct group of people and and yeah I I want to do the same with as well because I don't want uh when you send the columns I don't want you to think that you send it to the subnet but you send

it to to like a group of people yeah so so that so that so that we can we can uh decide the the underlying Network like in the way in in any way that we want uh does that make sense so so it's like uh uh when you when you uh uh clearly when you take a custody right you you you you you take a custody of of the column in the subnet but but I just rename the subnet to something else like

to custody groups is like when you custody the column you you instead of a subn yeah yeah pleas take a look at the pr thank you pop so it's um it's to make the P Des part more General more abstract uh yeah it's like make the network layer and the call protocol like decouple to each other

so it's looks nice and we I need more kind things to to help review it and to give feedback on it before we merge it and I think to be the target of Def three if possible yeah any feedback or suggestions on this PR yeah some from my side it's just just

what I wrote in the in the pl as well is that so conceptual cing is I think a good idea uh the my my concern was that we it seemed to me that the the only reason we are actually grouping The Columns I'm just trying to use the right energy here why we grouping The Columns is because of the uh presumed inefficiencies in the in the topic based gossips of topic based distribution so so we decoupling it but we are doing the whole thing just because of that um that's how I understand it now but I

have to look into it in more detail and for me even if if we would do it just because of that maybe the coupling makes sense but uh but yeah I have to I have to look more in detail any suggestions feedback okay if not then please comment on the pr itself uh the next one we have yeah we like to revisit the um the BL the BL limit PR again

so think it think for the previous code ACD and PS breakouts that I seem to prefer the one that not handle gas free computation in the C site so this PR and this still has other steps to do like changing the engine apis and and also needs e Cent to support it um

yeah and any I mean suggestion feedback on this PR like um how much we want to include it in Petra wouldn't this be a p death related EIP or would you want to include it in Petra anyway whether pus makes it in or not I mean it would be good to have it

in Petra because then like if we do p d we don't need to F the E to to increase the number of blobs right that's the logic of mean being part of the P TR is it would make PES um update cleaner it seem to me that this issue is not related to P death right it's just it's like more related to 4844 right but uh just giving us

a a foundation of like increasing BL limit easier in the future last we didn't get any push back from the so I feel like it could be pushed into pcture even yeah like uh implementation wise it' be a lot easier for us if it's like with pectra because we've been doing some work on trying to implement the separate activation Epoch after Electra

and it's just a lot cleaner that way good um so I suppose the next step is uh having an VIP and changing the engine API interface and that might also push another push again again in the the co

on Thursday to make sure the E people are aligned with it and uh another items that stalkes that the message is do we realistically see ourself changing the FL Target to anything other than Max divided by two if we do then we should also have the CL and Target along with Max values any sorts on it I think it would be better to send

both that's going to give us a lot more flexibility down the line even if we just even if you don't change anything uh in the foreseeable future I think it's better to send both thank you and anyone else uh I'm not sure but but I think the idea of dividing it it by two is from the EIP

like the idea is from the EAP 1559 right I think I think the idea is adopted from that that pass e maybe like what what what is the reason for dividing if by two like in the EIP 1559 maybe if anyone if if anyone know the reason Maybe uh we can answer that question like easier there's no specific reason that it has to be by two it's just yeah it's just a parameter choice I mean I think the idea is that uh it means it's robust to even to up to half of the

proposers uh wanting for example to boycott 1559 by fixing the block size and there for the gas and extract instead so basically we lose some robustness to attack if we increas it um about any other

suggestions if not then we can start a open discussions any thing we like to discuss today um I have two things um the first one was uh regarding metrics I spoke to Barnabas um about this I think last week or yesterday um is there a time where I think where clients are looking to sort of implement all of the necessary metrics in order to sort of get a good view on what the P2P

Network looks like um maybe this is like too far in the future I just want it to get sort of a handle I think for us we we have a set of uh metrics for all topics that we apply for so I think it's relevant to for p 2 are there any specific metrics that you particularly looking for I posted a Discord uh message we we created a list of the essential metrics that we probably see we would need oh okay all right

cool yeah I think we have most of this U or a few of them but maybe not all um and I guess liouse has most of them as well is that right jermy um yes I think we do have most of them although like some of the names might not match exactly but um I think we have most of them I think the challenge now is to actually gather gather the metrics because we do

actually need it like a good size Network a lot of noes you want to really really get like useful metrics for bandwidth like for example if a full node proposer want to see how much the proposing a a block and blobs take then we need at least um like like enough number of mesh piers in each subnet as possible to to get like some realistic metric that's going to potentially look like in practice if we only have like one one like one Pier which is a super node then it's probably not going to be a realistic representation of what's going to happen practice so yeah we kind of need a large

Network to get like real metrics I'll just look right that makes sense um I guess for load star um we're still trying to sync so the metrics aren't really a priority at the moment is that Y correct cool I mean metrix can become prior maybe next week I think okay cool that makes sense don't see any other C uh sius

I don't if you're on yeah hi guys do you hear me m yeah so we actually kind of doing experiment with P development uh uh so we have a bunch of uh fellow uh guys from from fellowship uh and uh essentially they trying to to deliver something so so we are a bit uh I would say out of sync of the whole thing and likely will not participate in the next uh devet uh so I cannot comment any further lightly M okay yeah that's great thank

you okay uh I don't think teu is here maybe I can just ask offline um yeah I guess my second question was um teu has merged our uh rust sort of implementation for the cryptography is it possible to um allocate some nodes that sort of switch that use the uh the CLI flag to enable it um so we can see sort of a network with both implementations yeah could you please put a message about it in the testing

and then I I can get to that mhm all right thank you um Kev um Lighthouse is potentially going to merge that PR soon as well um so we potentially going to switch to the the Russ Library as the main one um temporarily at least so yeah I think ligh house might be running that the next step net okay cool um maybe we can syn offline cuz I would like to figure out how to collect metrics on the local devet um I think I was messing something up locally have you tried to use kosis by

the way uh yeah I'm using ketosis um the metrics that I would that I was getting from the endpoint seemed a bit off uh so I think maybe I was just doing something wrong any other topics to discuss today um I've just got something just to

mention um it may not be like 100% relevant but um Lion mentioned a um um approach to um decentralized blood building um which is also mentioned in um danr post um um Lions wrote a hackmd page about it and uh Michael and I have created a v to the execution API spec um I think it's been approved I'm not sure if it's merged yet um if anyone's interested um it's been implemented in ref but sort of it's on Michael's B so if anyone wants to experiment with it um if we have some band with we're likely

going to experiment with it soon as well um but yeah with got the spec Pi open on the execution API um our repo yeah that's it I wanted to yeah can I get a minute here yeah so I wanted yeah so I wanted to ask what is the current uh situation in optimizations of the specific uh uh cryptography function so so previously

seems that the uh most of the time was spent in msms and is is there a big change in in this or um this is still the the slowest part generally uh you're asking about the cryptography in particular ER yes I mean for the cryptography the do specific cryptography MH um yeah the main part would sort of always be in the msms because the msms are group operations uh where is everything else is just like field operations right yeah and it's still it's still the same I mean there was now some

new like very different alos introduced that reduces uh uh the usage of msms um not as I mean there is a there is a technique to reduce the usage of msms um by sort of amortizing uh you sort of batch across different msms so in PIP and uh maybe this is maybe this is something for offline but essentially you can batch the inversions for group editions across different msms um that's the only technique that I know that we're not using at at the moment um it's sort of a a con a standard technique used in P Ander

nowadays so you do addition and aine formula and then you batch across the uh the inversions that are needed okay so so we just got merged U GPU Cuda optimizations for msms in rest casg uh but unfortunately at the moment we don't have that specific um uh functions ported to RG and even so the result is that even for smaller msms it makes sense to use I mean it is faster to use on to to calculate on GPU and I was just thinking is there I mean was there some discussion on the potential on trying to

use gpus uh here at least on some nodes like a super nodes or or so and if yes then we likely will U once we have a d specific functions uh it will be very easy to hook up uh uh these Cuda msms um so I guess my opinion on this is that currently it doesn't seem to be a bottleneck um at the moment so gpus is not currently needed um Cuda in particular is for NVIDIA gpus so I'm a bit more hesitant on adding those in because then now the super nodes need to have an Nvidia GPU right um if it was

like web GPU I think that might be a bit more acceptable since that works on every uh GPU uh but yeah yes maybe I can ask differently so so for sure there is some vendor looking and so on but uh let's say there are some nodes that runs on gpus and are doing faster uh msms on compared to to high CPU so is it going to be a a good thing for the network I mean yeah I mean I I guess generally I'd say yes because you'd be creating the sell the proofs faster

right um but instead of going to GPU wouldn't it just make sense to just add more uh add more threads to to it initially um I did some benchmarks and for I think it was eight or six 16 Freds we go down to 100 milliseconds um and this is not paral this is not paralyzing the fft um like the fft algorithm this is just paralyzing the MSM part um like in my opinion I don't think it's currently a bottleneck but maybe the like if it becomes a bottleneck then I think maybe GPU should be considered yeah so so my my initial impression actually is that uh uh it is

possible to to kind of Stack a lot of small msms like we have in in ethereum the msms are quite small and uh uh and this is at least from my view it seems It's significantly faster than uh even if you load a highend CPU fully uh so so but this is the initial observation from from the results from the benchmarks we got so anyway so we will keep working in towards this direction and once we have something working I think we can just uh uh join a network and see is there any changes in in metrics or you know is there any effect of uh of actually doing

msms faster okay yeah that sounds great um yeah if you have any preliminary numbers it'd be good to know how much the GPU is sort of making proof recovery or Computing proofs faster um so then we can sort of say uh you can get a GPU or you can get like eight Freds and you get sort of the same performance uh so we can make comparisons yes yeah I just wanted to say if it it it plays in if we have larger Bob counts because the the construction as we doing it now on the full column so the full data it's getting more and more

expensive as we have more more blocks uh which uh if you have the faster the construction then then we have nodes which can do faster Rec construction and and feed those those in so feed those channels basically feed those topics uh which are reconstructed which were not um propagating well so having nodes in the network which do faster reconstruction is is definitely helpful um with the current blob count I think it's it's helpful but it's not you know it's things work without uh with with more blobs it can be timelines are extending yeah I agree I just think

that it is a sort of another requirement to say that you now need an Nvidia GPU um so we should sort of wait we off the tradeoffs between more Freds and the Nvidia GPU yeah so I I would I would change to 2D instead of scaling in that Dimension but uh because in a 2d case you would not need that uh actually but again whenever you have faster Rec construction it's it's it's good for the network so it's it's a benefit for the network it's benefit for energy consumption and for anything else so um yeah any other topics for today

okay now then thank you for attending let's talk soon on Thursday bye bye byebye bye bye see you