All Core Devs - Testing #39

Transcript

Loading developers...

[Music] And people can still filter in slowly. Um and yeah, thanks will for mentioning I think we do have to rename the um the testing cog. Okay. So the first thing we Yeah, thank you. And we should be live. Uh the first topic I wanted to bring up was um Petra retrospectives. We have our blog post lives that we that I've shared in the chat here. Um one second. Yeah, we have a blog post live

Pectra Retrospectives: Discussing blog post on blobs count estimation

that I've shared in the chat here. And the blog post talks about uh the approach used for figuring out what the blob count was for Peterra. So the proposal was 69 and we've had the blog come live for about a month now um or a couple of weeks at least and Sam and Andrew from the team have done an analysis based on data from homestakers to figure out how that number actually played out on mainet. Um TLDDR of the post the estimation was actually quite good. Um I think 69 sort of hit the perfect mark of what the network is able to handle potentially today. um which is good news for pure dust estimation topics.

Uh yeah, we'd suggest people to have a look at that. Uh if there's any discussion then please bring it up in the ACDT issue next week or we can discuss on ETH R&D if um people want more data or more information about any of the estimations. The next topic I had was um the gas limit. So we've had gas limit at 60 million on both Solia as well as hoodie for about a week, two weeks now. Um there was an initial post made by Sam looking into the data and in general it um does any has any other client team

Gas Limit Discussion: Considerations for increasing to 60 million

spent some time on the topic? Anything you guys have observed, anything we should observe over the next week or two. Okay. Um then I guess over the next couple of days we should slowly start signaling a gas limit increase officially as well. Uh, but we can do that async and it would be great if client teams can spend some time over the next days looking through the metrics and making sure we're not missing anything.

GM everyone. Yeah. Um, okay. So, the next topic I have on the list is Fusaka Devet Zero. Uh uh par before we go to that can we first signal to 48 million then to 60 instead of jumping straight to 60 on May not I mean yeah do fan teams have an opinion on the steps what would you guys prefer?

Yeah, come in and then Ben maybe. Yeah. Uh, in my opinion, I we already see like 17 16 or 17% of validators already signaling 60 million. So, I guess they won't go for sure lower. Uh, we do not have probably anything problematic for now to go for 260. uh both from our tests and from the research done by some seems like 60 is like a reasonable option would be for sure nice to see smaller steps but I don't think it is necessarily needed for 60 like 60 we can go directly we should consider smaller steps moving forward

not not like very smaller steps but for sure not going 30, 40, maybe going by every 20, 15, 20, but 60 seems like a no-brainer, at least for now. The only reason I bring up this point is because we are almost doubling the gas limit now, which is like a very, very big change. Uh, yeah. Ben, do you want to go?

Um yeah, I was just going to say something similar to how came landed that 60 million seems uh postra with the cold age change um seems okay. above 60 we should you know think about a more cautious um you know if you want to go to like 120 I don't think we should double again. we should take it in more cautious steps. Yeah, I think the argument that was presented that also made sense to me was that uh gas limit changes tends to have a decent chunk of coordination effort and if we haven't noticed any big

problems until 60 million, then we can basically go to 60 million and then spend that extra coordination budget for smaller jumps post 60 million. So we could even do 10 millions instead of expecting a larger jump beyond that number. Seems to be so far that client teams are okay with that approach. Are there any objections to that approach? Yeah, I'm bringing up a point Tony posted in chat. um CL if it's true that the CL max limit size is 2 1/2 MB um which is something we should spend some

time figuring out as well uh then we can hit uh 10 MB only at 65 million so 60 should actually be safe okay then I think this week we can spend some time on getting some statistics for the max CL size and um it seems like that shouldn't be a blocker for the next jump at least if anything it would be the blocker for jumps after that.

Okay. Um any other updates on the 60 million topic at least for now? Uh we will again talk about gas limit uh towards the end but just to close out the 60 million topic do we have anything else for now? Okay. Um moving on to the next agenda item. There's one uh pas announcement which is um we've had a podas testing channel up until now. We've decided to sunset that um as well as um sunsetting podas devet 7. Um all pas testing moving forward will happen on the fusaka testing

stream. So we have fusaka tnet zero and that also implies that all the chats will be moving to the interop chat. Um so please move your conversation there and keep them in threads as much as possible so that the there's an easier overview for the channel. and I'll hand it over to Bangabus for Fusaka DNet zero updates over the last week. Do you want to go? Sure. So Zero uh started out with uh electrogenesis and we forked uh flu in epoch 256 where we kept the maximum count at 9. Then 256 epochs later in epoch 512 we have

increased the uh max count to 12 with BPO with the first BPO then in the second BPO 256 epox later we increased it to 15 then to 18 then we went back down to 9 and uh now in a few minutes we're going to go up to 20 and so Are all the BPOS's worked and did all the BPOS's lead to all clients successfully upgrading and downgrading or was there any bugs found?

We we didn't see any forks. Uh we saw some other appearing issues in different clients but none of them seem to be related to uh BPR itself. Awesome. And I think before uh Fusaka went live, you also sent the newer version block types to the network, right? Uh was that test also conducted as expected or was there anything founder? Yeah, there was no issues. Awesome. Very promising, man. Thank you. Um anything else for FAP at Net Zero?

What else is planned for the network? Um, is it just the last bo and then some sync testing or uh are you going to be doing malicious column withholding? What's the plan? Uh, I think the best would be to do sync testing and see if clients are able to back you know with checkpoint sync. Awesome. And we also have the problap tool uh that yeah I'm not sure if it was presented on this call yet but we have a tool uh built by the probab team that would request certain columns that the uh node is advertising um at least based on the CSC to make sure that the

advertised columns are actually being delivered on the network. So it should be integrated I guess over the next few days into the testing stack and then we'll I guess reach out to you if your node is live. Yep 100% this is uh ready. I'm actually going to share the link in the u the GitHub repository link in the chat and happy to receive feedback and of course reach out. Oh yeah, someone shared it already. Yeah, Barnabas. Thanks.

Yeah, we already building uh UI for it. Excellent. Thank you. Yeah, let us know uh in case of problems or whatever else. Okay, thank you. Um perfect. Um moving on to the next topic. Uh could we get an update on where client teams are with uh data custody? Um I can start for prison. Uh so yes we implemented validator custod uh but we did not yet implement uh the back field. That means that if you start

Client Team Updates on Data Custody

uh with a given validator count and you increase it, we won't back fill uh the already uh past blocks after having increased the validator count. And is this something that you guys plan on implementing or what is the it is under imple? Yes, implementation is in progress right now. Is this causing issues in implementation? Is this something that we need to have?

Um let me check if Case is on the court. No. Um yeah, as far as I know there is no no issue with implementation right now with back and yeah I think one issues with implementation the worry was that it's been taking so long. Um could we maybe hear from other clients as well as to what the status is and like what the ET is? Is this a feature we're expecting to have feature complete by the next cycle or will this take another month or two months to implement?

I think for Lighthouse we're expecting to have it done uh by end of this week like at least before the uh interrupt event. So yeah like it is a little tricky to implement mainly because of the back fill like Manu mentioned and uh also like uh dynamically when the validator uh when the custody count changes based on the number of validators you need to uh backfill accordingly before advertising the updated uh before updating the advertised CGC. So, it's a little bit tricky implementation wise, but I think we should be able to get it done before

the interrupt event. Okay, thank you. Um, maybe we first finish client updates and then I can um allow Francis to speak if that's okay. Uh, are there any other client teams here? Is it nish or someone else here? Yeah. Yeah. Yeah. So, uh from Nimbus, we have validated custody. We have the access column back filling, but I think there was a recent PR.

Yeah, I think Francesco went and uh yeah, so okay, this Okay, this is M. So I I don't think um I don't I don't Yeah, I don't think we do need valid validated custody backfilling uh because um you can check the you can check this new PR. I think it says that only update the new CGC after the DA retention period something like that. uh but we do have the mechanism there. Um there are few issues um I think I raised

this at least two weeks ago that is um with validator custody we have u a dynamic DA and the DA can like uh let's say if you're using key manager based setups u the DA can change or like the validator custody can change um at any given time slot and let's say it can even change um like um let's say at slot start and in the first two seconds. So um it it can it can be a situation where your custody count is 20 and right at like 800 milliseconds into the slot it can become

like uh 54 and you have like around 200 milliseconds or 250 milliseconds to um maybe get the excess columns into your DA from gossip or something like maybe u so there are there are uh different uh ways you can tackle but I think I think the best approach would be um to have a very um a constant um DA condition either I do samples per slot or I do number of columns if I'm a super node because if validator custody moves all over the place I'm not very sure about the

network stability. isn't the idea that the CSC etc only changes on the epoch transition? So even if your validator custody changes, you will have a certain amount of time before you have to make that change, right? It's not that you would have only a couple hundred milliseconds to react. Uh I I'm not sure if that is clearly mentioned in respect. I think it's uh more of a CL implementation detail although I I may not be sure but yeah Franchesco do you want to go maybe you have some insight on that. Yeah. So I mean one one thing is uh I guess

everyone uh probably should take a look at this PR. I think we discussed it in in a thread. So uh maybe this might also be why uh Agnes might be the only one that that has seen it already. Um but yeah, basically the PR I think should simplify the situation uh quite a bit and I I guess it was Justin already merged it uh because it essentially doesn't require uh back fill when your custody increases. It just says that you can essentially you're free to just wait until the custody period is passed and you actually have already kind of organically um done the not back failing but like you you know you've kept up with the with the chain. Um, so you you already have all of the things that you would need at that point to to increase your your your advertised custody. So I

think that that kind of simplifies like both of the questions that that were posed like both on the back fill in size and on the side of u basically it's not quite as dynamic because it doesn't it doesn't really jump all over the place. Like even if it increases by, you know, first by one and then by three and then by five, it doesn't really matter. you're you're still going to wait until uh the two weeks or 18 days or whatever has passed before actually making any changes. Um that's on the increase in side, but yeah, the decrease in side is is generally easier like at least on the on the uh you know back filling part.

You just don't throw away anything and there's there's nothing to to back field because you already have more than you actually need. Uh, so to me it seems like this actually um takes care of both of the concerns like both on the back filling size and on the on the side of the of things being a bit like too jumpy all over the place. Um, but yeah, for the second one uh if this is not enough for some reason I so the spec doesn't say things like you you mentioned party around only changing uh at transitions or something like that. I personally would prefer not to make the spec too rigid around these things but I think maybe we could have some note that essentially just says that is a bit up to implementers. Um again if needed on

top of of already this PR that was merged because yeah in principle it's definitely not it shouldn't be the case that I mean even in the old spec it wasn't meant to be the case that if the custody increased uh you know.5 seconds before you were about to attest now you have to throw away your attestation and and make sure that you like you know scramble to to subscribe to new subnets or or or think something like that like it was definitely always meant to be that there is some flexibility to implementers to kind of handle this um um kind of changes in custody in in a way that makes sense for for for them.

Um but yeah, I don't know. I think first of all, I guess people should take a look at this and then if something is needed on top, we can discuss that. Awesome. Thank you. Um Manu, do you want to go or should we have Justin first present the PR and then if there are questions discuss it after that? Uh yeah, as you wish. Yes. Um I have some concern some remark about this uh deep pull request but yes maybe uh just so that we are all on the same page about what this PR aims to achieve and then we can do questions.

PR Presentation and Concerns

Hi. Yeah. Uh sorry I didn't realize you guys weren't aware this was merged. Uh I mean brought up some concerns with me privately and I made a PR to what I thought would resolve those concerns. Um, I mean it kind of just adjusts when and uh like when we drop custodied side cars or like the columns and also when we start act filling like uh Ignes brought up that like when custody changes if it changes more frequently it might be an issue to like constantly delete and reback fill. So, we sort of came up with like a a less um I don't know how to say it like

something that wouldn't require back filling like every other epoch or something. Uh I mean to me it makes sense. You just uh you're required to custody columns which uh at pretty much any point in time. Uh sorry, you're required to continue keeping columns which you were required to custody at any point in time. And then like after the blob uh the retention period is over for that blob you can drop the column and um I mean yeah pretty much oh man what questions do you have or concerns?

Yeah. Um actually I know some big node operators have two set of nodes. For example, they have um let's say one beacon node A and one beacon node B. The beacon node A is connected to some validator clients. Um and so we'll have a high CGC and be not B is not is a standby and is not connected to any valid data client and have the minimum CGC when they want to to upgrade let's say a validator client and or a beacon node so they in order to minimize the IP loss um they will first upgrade the beacon B

and then they will connect the validator client to the beacon not B and so beonode A will immediately or more or less immediately decrease the CGC and beet B won't increase its CGC before 18 days and if they change you know connecting from A to B more often than 18 days if I'm not wrong with this uh put request both become nodes will And yeah, so both big mechan will won't be able to serve more than than four data columns

which kind of defeats the purpose of validator custody and if we require the biton node to back fill I think there is no more this kind of issue. Um I mean I don't know what to say. I mean uh you could choose to back fill if you want to. I mean I guess the spec does say should not backfill but uh I mean we can change that. Yes the spec say should not. Yes. Exactly.

Yeah. I mean I can change that to may or we can change that to may if necessary. But even if it if it is the main node and if every clients does not implement backfield, it may be an issue for n operators that switch from become a to be b quite often because you know none of them will advertise high CGC. Okay. I mean yeah we can change that. Uh I see some suggestions in the comments for like may and may not like I think both kind of mean the same thing.

Uh, but also I feel that this is somewhat of an implementation detail and like I'm not sure if it should really be super specific in those specs. I don't know because my question is if no client team implemented I if yes let's imagine no cliented no client team implement the back field and uh every big node operators do this big node A to be node B switch in this case we will we will lose a lot of super nodes I mean a lot of nodes which should uh behave like a super node will actually behave like no full node.

I mean how often does that happen then a switch from beacon of a to I mean as soon as there is a new new new new new new new new new new new new new new new new new new new new new new client version as soon as wouldn't these like big sticking operations just run as super node anyway? Sorry, I'm sorry for background noise. Uh wouldn't these like big sting operators run like a super node anyway? like it would this even be an issue for them? It's not an issue for them but it's maybe an issue for the network if there is not enough super node.

My concern is with this with this pull request with this change in the spec we I think we will have less super nodes than expected. Yeah, Pawan, do you want to go or Anish? Yeah, I was just Yeah, sorry. I was just wondering like uh Anu like for Prism like what uh like what endpoint do you uh use in Prism to figure out like to figure out how many validators are attached to a beaker

Discussion on Validator Custody and Network Stability

node? Is it like the subscription endpoints? Is it the uh attestation publish endpoints? How do you figure out how many validators are attached to a node? Do you take it as input from the user? How does it work for you guys? It is endpoint prepar uh beacon proposer for the beacon API beacon proposal like we we were considering using the subscription endpoint because it usually in multibbn setups the subscription at least for lighthouse they are sent for like all the connected beacon nodes from the validator client. So in that case like

uh usually most of the time the since the subscriptions are sent to everybody like all the BNS in the multi-node setup sort of become uh like have the same number of validator nodes consistently across them. So that is what we were thinking for lighthouse. Okay. Um, let me check. But I'm not sure it will it will change uh it will, you know, do anything better regarding the beacon A to be B switch from some not operator.

It will it will probably behave the same way. Yeah, Anishh, do you want to go? Yeah. So, my concern is like maybe a request to the research team that is can there be potential attacks if there is uh like if somehow a BN is not really enforcing the validator custody count to be very accurate like I understand that validator custody is sort of more like a huristic.

Um, can there be any kind of network partitioning attacks or any kind of like potential instability cases that can happen like say for these multibbn setups or something similar where the validator custody count is um is basically a virtue of the fact that these validators are pretty much being listened over by m multiple beacon nodes uh or like not very inaccurate like not very accurate uh calculations of uh validated custody count. Can we can we like uh can there be any kind of uh attacks or like any kind of uh problems

because yeah I mean we we came at it as a heristic but I don't know if if the number is enforcable or not in this is it's strict or not strict. Uh so I just have a question around that because u I understand Pawan's point and uh we have been talking about this internally in our team for quite a while now. Uh but it's like whenever this discussion comes we just at the very end the conclusion is oh it's just a huristic it's it's maybe not a very enforcable number. So that's that's basically the question.

Um, can there be any risks if it's not enforcable and if it's just a huristic? Um, would someone from the research team be able to take that up? Maybe Francesco or Raul? Yeah. I mean, I I guess I don't entirely understand what you mean by um like not very accurate like some Yeah. I don't I don't quite understand. Yeah. What not very accurate means, but in terms of the the risks, um like the the main point of V custody is really just to ensure that there's kind of a baseline of super nodes in the network.

like that's probably like I don't know uh well I don't know about a percentage but like it's it's like a lot of its value just just that. So yeah, as long as that's kind of insured, but like with with quotes, it doesn't need to be insured in the sense that it's impossible for it not to be the case, but just you know, if enough node operators are not uh modifying their code and uh I mean you could also include some something like maybe if not all of the node operators are running like very strange setups or something like that then hopefully that's the case. And then another part of of of it would be that in principle it you can get like just stronger assurances about um basically the

percentage of the stake that can be made to vote for uh something unavailable for something unavailable being available like to incorrectly vote. Uh so for that you kind of would actually want to maximize the amount of uh nodes that are voting with with respect to their actual custody as opposed to something much lower. Um but yeah but it's not like it's not super crucial. Um so yeah I I I don't know then what that practically mean. I mean I guess I don't know what exactly do you have in mind in terms of um yeah um so I think a better question would be is uh let's say the counting of validator

custody is strictly based on the fact that the validator is tobn relation needs to be one to one or maybe one to many uh like let's say um in multiv setups um where like let's say um in multibbn setups these BNs are detecting okay we have the validator here okay one more epo we don't have it um what I'm trying to say is is it strict one to one or like in cases for key manager based setups or in in a case where even Pawan tried to

explain And um I think um at a very granular point of time it's probably one to one but since we are um counting the custody over a like decent period of time it's seemingly one to many um I hope I could explain but that's that's basically the concern like yeah yeah I mean in principle there's ways to mitigate this like we could just say I don't know for example or you use the max of the validators that you ever detected in the last however long in the last 18 days or I don't know some period or like since the last time you started it or I don't know whatever we want I

guess uh or some kind of time weighted average I don't know probably the max is the thing that would make most sense so that like yeah if you run a multi- beacon node setup um with like a thousand validators that sometimes are connected to one sometimes to another then uh all of those beacon nodes will the super nodes all the time. Um I don't know that could be a solution. Um maybe we do Manu and then Raul. No, sorry. Uh it was from my pre previous intervention.

Yeah. Uh then Ro. Yeah, mine was also responded in the chat. Sorry. Oh, perfect. Um good. Then I guess we still have the open, this is still an open topic, right? Or are we changing the should to may or may not. And is that something that everyone's okay with? Uh, you go. Sorry. Yeah, sorry. I was just going to say I do think we we need So there's a bunch of edge cases here that I think warrant more like deeper analysis to to like think through how things would react.

So, I think we should take it offline um and and come up with something. Franchesco, I'm happy to be on this and Justin. Yep, that sounds uh prudent to me. Um for now, it seems like we have some understanding at least to have me uh that could be the PR and then we can have a discussion on that PR um and then decide what we do with the topic. Um, yeah. And regarding Barnabas's point, could we do May and hope to have validator custody by next week if that's okay?

Oh, yep. Let's do it. Okay. Um, then Barnabas, once there's a PR, could you add it to the Faka Devet one speech, please? Thank you. Okay. Um I guess we also need to figure out what edge cases happen with um with with the valid data custody changing, right? And I would assume we would discuss that on the PR. So yeah, more coming next week, I guess. Um continuing with uh Fouseaka DevNet one topics then we have um EIP7917 that was SFI on last week's AC/DC. Uh

EIP-7917 and Implementation Updates

this is the deterministic proposal look ahead. Um do CL client teams want to chime in on this one? Are there any open topics? Is this an understood EIP that just needs implementation time? And do note that this introduces um beacon state changes. So this is also a warning to all the tooling side because that will all break.

Is is 7 917 a target for the next? Yes, that was the idea. Um, unless there's severe push back on either timeline or the EIP on this call. From what I remember Mark telling me, we like he had already implemented it in Lighthouse, but I'm not completely sure if it was like a full implementation or like more of a P or like so I'll check with him, but I think we should be able to uh get it done like we can talk about it offline if there are any issues.

Okay. Um, other CLS, do you guys have an opinion on 7917? um working implementations or in progress. Do we have the spec ready for this and already merged in? Uh no, there's an open PR for the spec. Uh I would appreciate if CL teams did review and approve or disapprove. Uh I don't know. We're planning on merging it. Said he's going to take a look at that today and then maybe we can merge it in tomorrow and then we can start aiming for that next Monday. Uh pus has already left his review.

Um Prism has um in progress implementation for this uh EIP also in progress for load star. Awesome. That sounds good. Then uh so I guess the two major changes for Fusaka Devnet one will be at least on the CL side um the validator custody change pending a PR that we discussed today as well as 7917 right uh are there any other big changes I'm missing in the in the devet okay um are there any big changes for

the CL sorry the EL uh not sure the transaction gas limit cap and the blob base fee bounded by execution cost exactly uh do we have some EL devs here who can talk about how their implementations are going do you need something there open PR that need to be pushed through um yeah thank you Gabrielle Bezos is on track could we hear from Nethermind get wrath Aragon perhaps.

Okay, Nethermind's on track. Cool. Um I guess we'll uh Rex is also on track and we can follow up with Aragon and get um Async. Great. Um, looks like we're still on track for Fusaka DNET one next week then. That sounds good. Awesome. Aragon's also on track. Perfect. Uh, the next topic would be uh Yeah, Raul, do you want to give us an update on the BO networking changes that was uh to be?

BO Networking Changes Update: Discussing fork digest and EIP agreement

Yes. Yeah. So the final shape that this has taken is basically having a updated compute fork digest function that takes into account the blob schedule um and bit uh bit ma masks the the current um maximum blob um per block blobs per block and then adding a few more fe adding one key to the ENR to signal the next intended fork digest that the node is going to take. Uh there's currently an EIP uh that seem that most people that have chimed in seem to agree with.

We made a few a few tweaks here and there and now uh there's going to be a consensus spec patch coming in later today. Uh and do we want this included for DevNet one? Yeah. Um, how does that look from the implementation side of things? Is this relatively quick for client teams to do or should we table this for DevNet one and do it in DevNet 2 instead? So, I guess it it depends on the kinds of tests that we intend to conduct as well in DevNet one. Um if so far there have been no uh I guess network partitions or network splits observed so

far I don't think it's necessary it's urgent to implement it right now. So if there are other things to implement for devet one I think this can wait for devet 2. Yeah I'd also tend to agree right now. Um, also as of now, peer-to-peer changes are still a bit of a pain to debug. So, it would be nice if we're not doing that along with a bunch of other changes in a oneweek time time span. Uh, I would also support uh pushing it to DevNet, too, so that we have more time to review the PRs.

Awesome. Thank you. Um, and Justin, would it be possible to have whatever PRs we've discussed merged in this week and potentially a release or tag or something that client teams could use for testing? Uh, yep. I've already mentioned that somewhere in the comments. I'll share it again. Uh, planning to do a release on Tuesday, pretty much after the deterministic proposer look ahead PR is approved and merged. Awesome. Um, yeah, the next topic we have, uh, sorry, is there anything else to the BPO networking change that people want to chime in with? Um, everyone's okay with

it. What's the feeling? Okay. Um, I'll take that as acceptance. The next topic we have is um metrics implementation. So we have a couple here. Uh Katya, do you maybe want to talk about them? Um if if someone currently implements get blobs v2, uh I added the metric for it. So probably you can add it to the client's implementations. Uh that would be great to see. I'm currently building it for loadar and also it's not clear what what will

Metrics Implementation: Discussing new metrics and client integration

be with the custody. Um so yeah we'll see how it goes. I actually have metrics for it but if there will be any changes so maybe the specs will change as well. That's it from my side. Thank you. Um should we add the metrics change to DevNet one? Um feels like a relatively minor change for client teams to implement over the next week.

Yeah, it's it's still in this uh PR14. So probably we need to merge it and uh to open new PRs for any further changes so it will be easier to follow. Okay, perfect. Um Barnabas, can you take lead on making sure the metrics PR get merged then? Sure. Thank you. Um and yeah, I think someone from the Sunnyside Lab team wanted to talk about the same metric but for the E side.

Yeah. Hi. Um I'm Jay. Um yeah so we have raised um metrics for e side and then um um previously Marius has suggested changes in the naming and we've asked E teams but they did not yet respond to it. So um if they can have a look and if they are okay um we can go ahead and make changes. Um, we've already implemented the original versions to refer and also EWMJS and G is on Open PR. So, we can probably

change it quickly. Okay, thank you very much. Then, um, yeah, let's add the E metrics change as well to DevNet one. That way we can get some progress on the metric side and then we can still refine it and change it for the future. Sure. Thank you. Uh Kya uh yeah Perry regarding to EL uh metrics uh may we have a repo for for them as well. That would be great to just to share open PRS and discuss them.

Yeah. Is this uh is this just going to be execution metrics similar to consensus metrics or Yeah. Yeah, I guess so. Okay. Um yeah, we can sort that out and um get back to you on that. Yeah, thank you. Um and to address Justin's point, uh the ask on the EL side is to align on um get engine get blobs v2 metrics for all ELS so that we can actually visualize what's going on on the hit rate of the requests or whatever.

Okay, I think Katcha's ask probably solves what I need. Thank you. Yeah, exactly. So we'll we'll get a place where we can track this. There will be PRs. there'll be a metric back and then we can move from there. Okay. Um I think that should be most of um of that. Uh we have a Sunnyside Lab team testing update from the 30th of May. Do you guys want to give a brief intro on what's going on?

Um sure. Um may I share the screen if you don't mind? Yeah, of course. Uh figure out you should be able to share your screen now. Yep. Um let me just quickly share the link in case you guys don't have it. Uh I got that. You can start. Uh, sure.

Um so yeah we've done some um we've run seven dev nodes over the last 10 days prior to this release and um we were focusing mainly on um the impact of get plot enablement on different um combinations of CL's and we also run a dev um the same spas stabina 7 but without um load sty nimbus whether it becomes more stable and reaches higher block count.

Um to briefly go through the performance um without nimbus and load star we actually got lower block counts but block time was more stable. So we don't know what happened why but it was what it was but yeah we've we were more focused on the bottlenecks so that we can try to find out like how we can improve and um reach higher overall block counts.

So we looked into like each client in particular and for lighthouse um we got higher um 53 blob counts per block um when get blob is enabled and 45 without it. So it was like about 15% higher and it was similar for prism as well. So 40 while 37 without and then for grand um we somehow couldn't test it further than um more than 10 blobs and grand team confirms that we were using

um older image which had some bug. So we will try new grand image later with Zaka deadnet zero spec. So moving into analysis um this is the part where it becomes more interesting um that we got some ironic results. So um first of all this one is not ironic but we found out that Brendine was requesting more than once when it fails while other seals request get blobs just once regardless of like whether it fails or not. And we could see that on

Analysis of Blobs Request Patterns

Grandine when it request multiple times it were able it was able to get blobs occasionally. So um if other CS also want to um implement this optimization then it might be worth checking it um for the data column processing. Um on lighthouse um when get blobs was enabled um the number of column side cost that was submitted for processing decreased.

um when um when the block counts increased, it was quite interesting to see why, but maybe there was an optimization or some maybe there was a bug on metrics. Um we don't really know why, but this was something new that we found. And second thing that we found regarding the data column processing was that um on lighthouse gas combinations um when we disabled get blobs we didn't get any data column failures but when we enabled get blobs we got so

many column failures. So um we cautiously assume that um guest was handing wrong columns to lighthouse for some reason. Um so this week we're going to test this with RAS as well. So so we can check whether this is from E or not. And we got we also got similar results with prism as well. So with prism when get blobs is enabled with gas um we got more processing failures.

This was interesting question. Are you are you sending uh V1 transactions or V2 transactions? Um B2 Yeah, because like it could be that if she send we then we will not return the correct thing but like that's I had this error and everyone complained that there's this error thrown so I just like silently ignore it now but u that was one of the one of the one of the things that came up is that there was still being a bunch of transactions sent with uh with PL we want um and so we

didn't return the plots on but I can I can okay that would be appreciated and Pan please. Yeah, I think the failures could also just be uh like duplicates like do you know like what specific failure it was on lighthouse? Because if you are getting columns over gossip and also getting them over get blobs it might just be getting registered as a duplicate column and that is also like an error. We don't like uh do anything with the error but it is still registered as an error would be that

probably. Okay. Yeah, that that might be one of the reasons because how we measure this is just total request minus um number of successes. Um yeah, that's a good point. Um maybe we need to count it separately. Um moving on to the next point that we found interesting was um the column processing time was inversely proportional to the number of

Column Processing Time and Optimization Issues

processings on average. So the first chart is the number of requests and you can see that Grand has the most and Tekku has the least. But then when we measured the timings, Teu has taken the most and Grand has taken the least. and Jay would it be possible to summarize in the next minute because we have like one more topic we want to do and then the rest of the do async.

Yeah. Yeah. Um high reconstruction time on lighthouse was already discuss discussed on the discord and then block size um I think we need to increase it on g I think 42.6 6 megabytes is bit too small because the blob size will be 9 mega megabytes for 72 blobs. So um that was it and for CPU RAM. Yeah, our blob pool is 1 GB pool. Yeah, it's just one one sender can only send a few blocks.

Okay, size of the of the block is sparking the question. Okay, let me check with you offline and the rest is yeah it was all right. Um grand guess um I think because of the glo size we had problem along with like not using the latest image. Yep. Thank you. Thank you very much for the analysis. Um, could you maybe create some threads on the interop chat for whatever topic that you wanted to discuss and then we can discuss with the respective client teams and sort them out one by one.

Yeah, sure. We'll do. Yeah, thank you very much. Um, and I guess you guys should also migrate to the Fusaka DevNet zero spec next for the next version of Yeah, we've just enabled it today. We've been starting to test it. Perfect. Um, okay. But at time there's one more small topic I wanted to bring up after, but there's uh one announcement. Is everyone in general okay with um uh cancelling next week's call because we will be in person with a interop event and I think ACD also might be cancelled next week.

Okay. Um we'll still discuss that and post a message on all core devs as well. Um but yeah I think if anyone needs to drop then please feel free to drop. Um the only last topic I wanted to bring up was uh do we have column side cars in the beacon API event stream already and if not are we planning on adding it? Um I'm not sure is there some CL dev or someone owning the API who's here for the event stream. I I I think we don't have it uh in in beckon API but for for

tech we have custom implementation for example. Okay. Uh then I think we might need to add it to the event stream cuz uh we just started looking at the zatu metrics for it and we realized that the event was missing which makes sense. Uh cool. So we we we'll make a PR and then we can take from there. Okay. Thank you everyone for coming then and hope you have a nice day. Thank you. Bye.

What's up? Thank Heat. Heat.

Heat.