All Core Devs - Consensus #152

Transcript

[Music] This is layer call 152 uh issue 1323 in the PM repo. And yeah, there's a lot to discuss today. Uh we have some test nets to to investigate. So uh let's go ahead and get started. Uh petra is the name of the game and yeah, let's go ahead and just get into heski. So uh I think we all are familiar with the latest on hleski. Um I did just look at the chain a bit while a few minutes ago and participation was much higher than I was expecting which is good. Uh anyone here would like to give an update on Halki any further? Maybe some client teams on what they're seeing. Uh and then from there we should discuss uh recovery.

I guess I can ask like our clients feeling good about how they're performing on Helski. Uh so I guess I would say from Lighthouse we at this point have we've made enough mitigations for Lighthouse nodes to be manageable I believe. But um uh a feature that we've added is manually finalizing your node, which is essentially just like allowing you to prune your hot database in case it's getting too big. Um, but there's bigger more structural there's like a big

feature we had in the works that would help here um that we will get in in like you know the relative short term but not like immediately. So I guess we're we have enough mitigations to feel good but we're not um fully there and addressing like fundamental issues. Um yeah, put it. So it's tree states right now. What we have is we have tree states in the freezer our freezer DB so our cold DB and we also have it in our in memory structure but we don't have it in our hot database and our hot database holds unfalized states.

So that's why like disk is going out of control, right? So yeah, I think most of the CL clients are taking up a lot of resources during this period of non-finality on Alleski. So the sooner we can get to finality, the better. Um, yeah, I guess maybe we'll go then to this uh doc here from Manu. Do you want to say anything, Manu? I think you did uh some investigation into an estimate. Yeah. Hi. Uh yeah. So you can click on this link. Basically it's a model which estimate when finalize the game by so it take into account the inactivity leak

the in/ flashing penalty the correlation flashing penalty and the validators leaving the queue and it has also some hypothesis. It's on the first tab of this document readme and you can explore it and tweak it if you want. And basically um with the condition of this morning uh we can expect finality for Hleski uh on 28th. So in 28th of March so in 22 days. Yeah. Okay. This looks nice. Thanks for putting it together.

So yeah, I guess that's the next step then is like if we won't finalize until the end of the month, uh we should discuss if we want to wait for that or try to pipeline a bit. Uh there have been some ideas around either new test nets or like yeah different variants of Hushki. One big thing is that we don't really want to go to mainet until uh different users and different uh stakers and things can test out the new features that we have in Pekra. And yeah, right now they can't really do that. Uh, Helleski was the place that did that and Hleski is not really usable right now.

So, um, yeah, Perry put this link here in the chat, a path forward for Heski and laid out some options. Uh, Perry, I don't know if you want to say anything. Yeah. Um so I think the first question is do we want to um continue using holki once it has finalized or are we discussing replacing holleski with another network and that other network is the path forward. So even if holski finalizes we essentially um abandon it once it's done. I think once we answer that initial question then there are about four out options outlined in the document. um each one with a bit more detail um as to how we envision a

recovery scenario to look. But I think the core question is still is Holski worth keeping around after it's finalized or is it not? So I would say we should recover Hushki because it's like a good exercise to go through um separately. I think having another network to drag people to for testing is a good idea. Yeah. Um and also just to let client's point um is there value in syncing a network in about 3 weeks time the network would finalize again. So at that point it's a regular network again. Um so you would be syncing it as you would any other network.

At least getting a temp check from client devs earlier. It sounded like we wanted to keep holski around longterm. um clients were okay with doing maxb deposits and bumping up the amount of stake on holleski. Uh so essentially it sounded as if we just need a testing location for the next couple of weeks. So is that the right reading or would people prefer another approach? I guess one point here is if we have a second network it'll be double the infra and is just for a temporary amount of time but yeah there's some cost there.

How do client teams think about this? Because you know I think ideally there'd be some configuration to separate the new hleski from the old one. You'd probably need to support that in your client cover releases and this whole thing. Yeah, Terrence, what do the stations think about this? Just like I prefer I I presume they're the biggest like uh users here and they have some existing setup on Holi as well, right? So, they do want to test it. So, yeah. Do they like have ha has has anyone talked to them yet?

Yeah. Yeah. Um and Ivan from LA here. So uh for us uh Hleski is uh really critical uh and uh without Hleski finalizing we'll have a very hard time or like almost impossible uh to test factory related changes to to a protocol. Um but also to test uh whatever integrations we have with other protocols that use Lido. Yeah. So um once Hleski gets to finality, we will also need some time to make sure that the protocol uh is able to digest whatever happened on Hleski because it

will likely go into what we call a bunker mode which is a very special mode of operation that results from mass slashings and uh delayed oracle reports. uh once the protocol is recovered, we'll also need time to um uh test all our monitoring and uh new releases of oracles and so on. So um yeah, for us it's it's really critical that it works and you don't think you could test the same things with the second network?

um depends what is on the second network because a lot of uh like Lido has a lot of dependencies. uh and that that second network would have to be I guess not worse in terms of features and the deployed uh protocols and also the numbers and types of the uh node operators and validators as haski because for example we have dbt module in uh that require certain features like um SSV or or oval we have uh ID layer and and a integrated with Python and they have to network. So all of that should be on that second network,

right? Yeah. So we could basically clone Heski. So it would look the same as it does today. You would just have a network that works uh at least on the contract level. So the tooling level would of course depend on the Oracle providers, RPC providers, etc. to support the shadow fork. But the a contracts will exist. Everything that is onchain will exist at the exact location that it has existed since the last few days, weeks, whatever. Yeah. And um in terms of u the I know the number of validators, how many do you think will be from this new network?

If we're doing a shadow fork, we can aim to have the same validator sets as um there's also two options. We can even use a large majority of the um genesis validator set where we have the private keys. Um there's also another option where we come with a new validator set but in both options we would have the same number of validators and and in terms of client diversity how how will it look like? I think we basically have free reign on that. So we can design it um to be as close to host or mainet or whatever you would like.

Yeah. Yeah. Yeah. I think like it sounds like if it's totally replicated then I don't see why we would would be not okay with it. Um so yeah but then the question is would it be faster than wait for to finalize? Yeah, I mean the shadow fork is something we can get up and running as early as next week. So it would be about a few weeks faster than whole. Uh the downside would be all the external dependencies. So stuff like oracles RPC providers were not sure what their timelines would be.

Okay. Yeah. Yeah. I mean why not? Um for us having this also is important because if something goes wrong with on mainup we need some kind of a place to test uh fixes to to the the protocol but if it's a new network like a shadow fork I guess it it covers that need as well we don't have like unless you're working it's probably also not ideal. All right. So then it sounds like we should parallelize and just start. Well, we should do both. So have the new network and presumably we can get all

the infer and things that we need up uh sooner than Hleski will finalize and then obviously once Hleski finalizes we could either tear down the new network um perhaps keep them both. We'd have the option to to do what we want. Um but yeah, that way we can keep moving forward on Spectra and not add uh you know say another couple weeks of delay just because of the status of Heski. us. Yeah. So from one clienting perspective at least um we have the following problem which is we we are certain to have a bunch of bugs on attestations. We fixed some uh but our these bugs are being shadowed by all the problems that

clients are having right now. So, so it seems that hleski as it is today is not really useful for us to diagnose spectra itself which is what we wanted to do and that's why we we had this buffer of a month running on a good test net to actually decide on mainet uh to decide of a go on mainet and we we don't we don't have this now and a shadow fork would would only be a test it wouldn't really be a test net in all of the P2P side of things so so that's the the the trade-off that I see on like rushing without really having diagnosed our bugs that are existent. They they are there and you couldn't investigate these

things on the shadow fork. Well, you can investigate a bunch of things on a shadow fork that is stable, but definitely not things that depend on the centralization. Hleski is a is a is something of the size of mainet and is actually decentralized. I mean nodes are everywhere. uh shadow fork. Well, one point I will add to that though, uh Holski is large, but the order of nodes is around 500 to a,000. Um it is decentralized, but it's about 10x smaller than mainet.

So on mainet we we we're fine with these numbers, right? Like a thousand nodes is thousand nodes. Yeah. Yeah. Uh but the shadow fork we're talking about would also have between 500 and a,000 nodes. The main difference is that Holski has setups in the wild. For example, a test in vouch would be running there. Of course, we can ask Jim to run on the shadow fork as well. But I don't know if someone has forked their client and is running on Holski, we'll never be able to replicate that. Um, but in terms of pure node size and distribution, we'll aim to be as close to Holski as possible.

So, you can keep like a a thousand nodes running for a month. That's what I've been told we're okay to do. I mean, this is important. So, I think we should we should go ahead and do it. Um, assuming there aren't any like technical bloggers, radic. Yes. So, I have two comments here. Um, I'm a little bit worried about starting a new network. Although I think it's a good idea but I'm a little bit worried that if we start a new network then people will start like abandoning holleski meaning that for example people will uh I don't know about client or maybe others will just start running

nodes on the new network and not run any validators or nodes anymore on hleski and then participation will drop blocks will be missed etc. So, you know, because the new network works, people will just stop using the old one. And this is something related to what Ivan said from Lido that they have some kind of a bunker mode. And I guess uh bringing Hollleski to finality and keeping Lido on Holliski uh and checking how if it works would be a test if this bank actually works. Whereas if for example Lido Lido switches to the new network and just abandons Holliski then they will never really be able to like actually test like this bunker mode

because um you know because they will just just just move over. So what my my um I guess my um little worry is that because of the new network we will just you know treat Hollleski as as like a second class citizen and maybe even you know um it will just degrade over time or uh yeah or it we will not be able to properly test like refinalizing uh both in client teams and external um users because they will just start using the new network which is uh stable. Yeah, in terms of like internal bandwidth, like we could have half of a client team work on this new net and half focused on Heski. Does something

like that make sense? Yeah. I mean, yeah, of course that makes sense. I'm I'm also thinking about you know external like not client teams but other you know like um staking pools and whatever that they would not stop like caring about holleski move to the new network and then they will never be able to like test what happens when there's a you know final like when the f when the chain starts finalizing it or stuff like that. just saying that you know this will be like um

not retested exactly as it would happen on mainet because we will probably have a new uh network which is stable and people will just move there. Yeah, I mean we could commit to tearing down the new network in say a month. Um but yeah, I hear what you're saying uh in flag. Yeah, I think I would agree with that because right now it's really hard to say. Is it just buggy clients that make the network unstable or people being offline? Because over the weekend the network was in a lot better uh worse shape than it is now. So I guess it's maybe both. But uh looks like also just people not watching their nodes and keeping them up. So just from a testing perspective would be nice to still keep

a good focus on the current Hleski network. Yeah. Ivan. Yeah. From the Lido side, uh it would be ideal if we keep focus on whole escape because we have all our processes tailored to that. Uh but we probably would be able to also find resources like you like split the team into two parts while working on bles working on the shadow fork but we won't be able to do that for long because of of the limited bandwidth.

All right. So, I guess something else to keep in mind, I think there are some chat comments. There's ephemery, then there's also the DevNet 7 that we spun up. I'm not sure if anyone wants to speak to those. Uh, but they could take some amount of this testing load. Yeah, maybe just a brief about DevNet 7. Uh, it's already live. It's about the same validator size um as mainet. So, you have about a million validators running on about 500 nodes and geo distributed. Uh the downside is like Ivan mentioned, it's a fresh network.

There's no dependencies there. So if you depend on the a contract, whatever, unless you can convince them to do the deployments, you won't be able to use it. But if you're a staking pool or a regular homestaker who just needs a location to test, then it would be useful. Um however, it seems like the benefit right now is minimal to keep it around. So we'd rather take it down. And if you're someone smaller who's looking for a location to test um validators, then please go to ephemery instead. Yeah, we got a plus one here to temporary hleski shadow fork. Uh which is I think the strategy I prefer at the moment.

Okay. So we'd spin dev7 down. Yeah. Okay. Any other questions or comments? It feels like we can save some time by having the hles shadow fork. So I would lean towards doing that. Ben, uh just since you mentioned spinning Devet 7 down, has all the consolidations etc been tested on DevNet 7 and is it happy? Yeah. On seven we have parents.

Yeah, one last comment about the shadow for holies. Um, I think it's important to know that client need to run something, right? So, even after holy is finalized, those slash clients doesn't have keys anymore. So, we need to also deposit redispute the keys and that also takes time too. So we need to cons. So we probably shouldn't uh like uh erase the shadow fork unless we get to like 1.8 million validators the basically the basically the previous state.

Right. Yeah. Um one comment on that. I think because of the uh churn limit to get back up to 1.8 million validators might take significantly longer. It might be roughly a year. I'm not sure if we want to keep the network along around for that long. You're going to do support till 28 anyway. Holki shadow for no not forks. Yeah, holki we'll keep around. I'm talking about holki shadow. I think we should spin that down. So exactly personally I think hoski shadow frog should be spinned down once hoski is finalizing.

Yeah. So approximately hopefully in three weeks. And I think anytime there's a need for a really large network, we're happy to spin one up. But the main downside is that these networks cost a lot of money to keep up. You had your hand up. Oh, I was going to ask a question. I thought there was there was much less slashings that we were expecting. So, I thought that the the exit of the validators that are in the wrong chain still or just didn't update was going to be the main driver.

Yeah. I mean, if they haven't been slashed, they could even finalize like they could help finalize the good chain, right? Yeah. That's so that's the thing. So, if if the slasher is actually effectively off now and we finalize, then those the clients are going to keep the the the keys. So, it's fine. Clients are still at testing are still producing blocks. So they're active. They're not leaking. Uh SAS. Yes. So I'm just wondering so is there a big need for um for the other parties except the core there was uh uh to run a large number of validators uh as they did previously on Hleski. So because if if they are if

they are also testing this then uh because it will be it will take a lot of time to to get a lot of validators again on hleski it's uh it's not going to work I mean hleski anyway not going to work uh for longer for them so so apart from you know apart from execution layer uh state is is there some other entities that are heavily testing on on the large number of validators on hless So is is it known thing or not? Well, yeah. I mean there all the staking pools and then all like the solo stakers and home operators. So I mean unless he has decent usage.

Yeah. But for let's say for lido. So so I I think they they mostly were interested in the E state. Is it is it true? Ivan, are you still uh at the Yeah. No. No. For us it's both both are important because we have many node operators run. So both EL state is important but also whatever happens on the consensus layer. Okay. So, so actually so actually if if we if uh the even if the network finalizes uh in a month or or whatever it will be the date it's still not enough for you because you you will not be able to have a large number of

validators uh that you can run is it that's right uh so once finalizes uh what will likely happen is that will go into this so-called bunk remode because of uh slashings and also the fact that Oracle reports haven't been coming through. Uh it will take some time to restore the protocol from the bunker mode to the normal operational mode. Uh and once that is done, we'll also need to run some tests of the of the new features like to make sure the protocol works well with uh petra and after that we will also need to make make sure that all our monitoring tools work and the

oracles work. uh that other uh parties run and so on. Um and uh after that we'll also need to prepare for uh the voting to update uh the lid of protocol. So we'll need to know the exact epoch for the mainet and uh we'll need to make sure that voting uh lido argon vote passes and passes like from one go to make sure the protocol is actually updated at all times. Yeah. But but all you said is unlikely. It doesn't need to to have a million of valid active validators on the network, No, it doesn't need a number.

Yeah. So so likely there is a a quite low need to have a huge number of of validators on the Hleski. So, so you would be essentially you would be happy with finalizing network that has a relatively small number of validators like a 100k or or 200 or so. Probably yes. the the most important for us is that we have um as similar setup as we can to mainet uh in terms of at least like the client diversity and in terms of the types of validators that are being uh run. So for example, we

will definitely need DVT to be tested with both SSV and mobile because that's like an integral part of the of the LDOP protocol. Um that doesn't require millions of validators, but it's more about the like coverage uh covering like in test coverage you need to cover all the different cases that uh that are relevant. the big number of validators u is also a test in itself but I'm not I don't think that for lido specifically it's going to be a a blocker okay thank you so if if anyone will know like a real cases where the relatively small number of validators is a blocker

then yeah let us know because it it's important and Ivan just a clarification do you feel like you could do productive testing on the shadow fork in the meantime. I I think so. At least currently I don't see any blockers to that. Okay. Yeah. And then you may want to do other things once we finalize key. But uh yeah, I guess we'll cross that bridge when we get there. Yeah. So my main understanding is that you know we have Hleski in the size that we have so that we can test features that we then will roll out to mainet right and especially hard forks and so with

regards to when we want to uh remove the shadow fork I think the shadow fork now becomes our test ground for mainet for spectrum mainet which means that if we finalize holleski but we're still not 100% sure that what we tested on the shadow fork is good enough for us to say yes let's uh you know let's go to mainet then I would say we should not tear down the shadow fork right we should basically I think the shadow fork will become like the test net for petra and then the hleski can become test net for fusaka because it will probably be um be um healthy as hleski was uh before the accident

by the time that we will roll out the next hard fork. So that's my opinion about uh you know timing the the the tear down of the shadow fork. Okay. Uh in uh so my comment is related to what PUS mentioned. So um basically regarding the tester slashing. So we had a quick discussion on the testing call on Monday and I mean the only reason right now why we are not slashed is because uh slashes don't work properly. They produce like one to three indices per slashing even in huge ones. So we will never slash them. So the outcome there was we considered to handcrafting good slashings so we can actually see the

spec mechanisms play out. And I guess the second comment related to that, I think it would be a mistake uh to finalize Oleski right now just because the slasher doesn't work. So personally, I would prefer for the spec mechanisms to work out or play out. So we can see this actually works and we are not basically cheating just because it's broken. So and yeah, I don't see an issue at least for us to keep the non-finalized chain running for a few more months. So at least right now we are not aware of any ticking time bombs that will kill our client.

So yeah um it feels like the slashing issue however is not necessarily unique to host key right that's also something we can provide an alternate testing ground for you um and the main it is definitely something we should be tracing down and fixing. Um I think I just disagree with the location purely because there's a lot of dependencies on Holski that we can't control. Whereas with the slashing bug, there are no dependencies. We can control everything. We can create the bug. We can fix the bug. We can um retest it. Whatever we want um in a location that allows others to continue their work basically.

Okay. So basically the goal you you mean would be to finalize Hleski ASAP. basically try to get 66% online right now or ideally yes. Okay. And then we would uh spin up a NFT devet to test this scenario. I think the uh whole exercise has showed us that NFT devets are just so long for non-fality devets are uh something we can't sleep on any longer. So we will be putting a lot more resources and time on that anyway.

Okay. My only worry there would be that a lot of the bugs we had were not popping up on the NFT defet. So we need to figure out how to get this in a state where we see those bugs. Yeah. Yep. We could just reuse another version of GU that trigger the bug. Yeah, we can actually just trigger the bug the exact same way we did on whole scheme. Okay, so it sounds like we like the idea of the shadow fork. We'll move ahead with that. Uh in parallel, we get finality back on Alleski. I think a provisional date to spend down the shadow fork would be once we finalize the Lushki, maybe plus a week or so, and

we can always revisit that as we get closer uh and decide if we need to keep the shadow fork around for longer. Does that sound okay with everyone? Yeah, I I do think we should um I mean line with pus I think we should um push for a longer lived shadow fork, but we can see we can evaluate that later. There's just a lot of ops time and effort on our side. Um yeah, we can have that debate later. I think this is a good first step.

Okay, great. Um there's one more uh minor announcement we wanted to make. Uh is there any reason to keep the Mong test net up and running? Uh it's running an older version of uh of the spec. It's been up for a long time, but it feels like we've move everyone has moved on Spectra testing to new locations and we'd rather repurpose those machines if So I guess the question is do people want DevNet 7 to continue running as well as the MCOM test net? If there's no answer, then we'd like to take both down by end of day tomorrow or today.

We got a chat comment, no interest. And yeah, that sounds good to me. Okay. Uh we'll have a blog post on the EF blog as well just announcing this and give people some time to react. But um we go ahead with that. And maybe just one more small comment, a bit of a happy note, I guess. The Chiato test net worked earlier today with Petra and it seems to be successful. Yeah, good to hear. Cool. Okay, anything else on Alleski?

Uh, we can touch on Sapolia because we had a Seapoleia petra fork yesterday. That was uh also kind of fun. Okay. Yeah. So then for Solia, the fork was yesterday and yeah, there was an issue with how some of the EEL clients were processing uh essentially a custom deposit contract on that network and it caused some issues. Uh this was yeah with the 6110 mechanism and the execution request. So luckily we were able to recover it uh after some quick onhands debugging. Uh but yeah, otherwise Solia looks good and it actually is the healthiest I've seen

it I think ever. Um so yeah, I guess this is another note if you're listening and need to test um especially any like application layer things, Solia' be a great place for that. Any other comments on Seolia? Mario. Yes. Can you hear me? Uh so I think we should discuss a little bit on the root cause and u there's still some differences that uh we can address uh between EL clients uh particularly the AI encoding that is happening uh of the deposit contract. Um

there's an app a PR that's has been opened by Yahim and I think it's very important that we uh all come together into deciding the that we parse the deposit contract blocks the same way. Um basically I mean it's not a problem here. It's not going to cause a problem in mainet. Uh but um I I think it would be nice to not have like AVI AVI encoding enshrined in into the protocol in some way or another. Um so yeah my suggestion is that we should like uh not use this this encoding and basically just try to extract the deposit information that we require directly from the b the the bite array. Um, I

think that would lower the u the the yeah the the risk of for example running into a a a a library bug uh due to the AVI encoding and that causing some sort of consensus issue. Um so yeah that that is um yeah I think Joh shared the the PR and um I will encourage uh eel deps to look into it uh and so we can have a a a way to proceed. Yep. Do we want to formally get this into the petra specs?

I think so. Yes. And we going to update the tests um to include this scenarios. for example, um the scenarios with the AVI encoding uh because we yesterday we generated one test first without it and it caused uh some some some problems for for some EELs to to pass the test. Um so I think we can just regenerate uh the tests with this in mind and to make sure that the E AI encoding is not like any any cause of issue uh ever. Yes. Yeah, that makes sense.

And one related thing here uh male made an update to I think this was 6110 um to basically just specify uh the strict behavior. So that's been merged and yeah anything else on Solia? Okay. If not, then uh I did want to touch on mainet timing, but we've kind of covered it already. It seems like we need information uh we need more information from how heski goes, how the shadow for goes before we can really uh set a concrete date. Uh anyone else have

any comments on this? Yeah, I'll come. Yeah, just one comment because this is about uh like the AI decoding. Yes or no? Like Roman just posted a comment. Um we should really decide if we want this or not because this will differ in the test. And if we want the AI decoding uh like enshrined into uh well the protocol basically then g is also going to need an update. Yeah. So my my view on this is that we uh we don't want to enshrine um the AI because that would mean we would need to um need to enshrine our AI uh library.

Um I don't know about your guys implementation of the AI uh decoding but ours is pretty and we don't want this to be in the critical part of the consensus. I completely agree. So let's please go with the the offsets and the uh the sizes like the hard uh the hardcoded version which GE has. So no AI decoding. Okay. So this is uh sorry maybe okay I can just start speaking. Sorry. So this is Felix from G. Um so as far as I understand in your PR you are proposing to formally codify the approach that is also being used in GU and some other

clients with the offsets but it is an implementation detail I think so it for me it doesn't really make sense to put it in the spec exactly as is because the only thing that really has to be specified is just like how we get the information from the from the from the like how we get the information from the contract output is not so important I think like the mo most important thing is that we produce correct output but the spec doesn't really have to tell the method by which the output is extracted okay then I will have to investigate further because then I don't understand

why uh clients passed the first version of the test yesterday yes or no Um I will have to investigate and I will get back to that later. Yeah. Okay. If I can add something. Uh well ABI is ABI format. It's already part of consensus because logs come with this format. So it's already enshrined in consensus. Now there is a question do we what do we how do we want to decode it and do we want to enshrine any enshrine it's implementation detail? Do we want to use for consensus any libraries? And the answer is uh like kind of agree this is implementation detail but if you like hardcode the uh the offsets if there's

something different with the ABI you kind of fail and you you parse garbage and we want the um like there's a change in AI and do we want uh the clients to potentially parse garbage on production networks. I'm kind of like leaning to let's parse this message properly as it was encoded and go with that. Uh hello.

So yeah uh apparently we have switched the mic setup mid call. So now I'm speaking into a different microphone regarding uh this thing that was just said. Um it's kind of an interesting question. What does it even mean to like enrine the AI? But basically what the case you're talking about is kind of an impossible case from the mainet point of view. And then the only thing that could change is like if the contract produces an invalid output, which is inherently like an sort of undefined behavior case, I would say I don't know if there's any precedent for this type of undefined behavior in the spec, but definitely

yeah, it's just kind of weird. I think we should just define it. And I think the the the right way to define it is based on on the offsets like uh yes this this might be valid AI that we're we're generating as the as the logs but um no one has ever said that this has to be passed as as valid AI. It can just be some binary output that we are passing with the with the offsets that we Okay, but then this would basically mean we go with the PR like the question here is like is the PR good or not? I think the PR I think the PR is good and I think it should just be enshrined so

that in the case that in the like case that will never happen um that uh this is triggered then we all do the same thing because because I I think it's way more likely that uh some AI library will uh will break at some point and um is in the consensus critical path and then the whole node will shut on. I think that is more of my worry than that the contract somehow returns some some miss uh some some valid API that is different from uh the format that we're hard coding and we are uh parsing

garbage like this can happen with if if the output can if the output of the contract can change anyway um then uh this will be the case regarding is whether we use AI to decode it or whether we hardcode the offsets. Also, also I pushing back on the on the stuff that we are hard coding AI anyway in consensus. We're not that the logs are just uh whatever you put in memory. You can uh solidity hard codes or solidity defines the AI and how logs are done but the EVM does not. So we have not hardcoded uh ABI in in the EBM just just to be explicit what the AI is

it it's all the fields are length prefixed so they say how long the following data is so in theory you know if if they decided to trim the zeros that would still be valid but it would screw up all the offsets but it would still be valid with the length prefixes Yeah, but it's not packed by default on logs, so it won't. Yeah, but but that that's um that's just convention. It could be packed. And with the AI, it would it would be fine, but with the offsets, it wouldn't, you know, if somebody say L2 decided, well, shouldn't we trim these logs to to

save in receipt size? Amma, do you have a comment? Yeah, I I I don't I don't understand why we're discussing implementation detail and we want to specify in the spec what every client needs and how it needs to implement this particular part of the spec implementation mechanisms are always the choice of a client saying that using a specific library can then lead to this library breaking or whatever. This is not a reasoning that we usually use to choose implementations. Every client is always free to choose whatever implementation suitable uh for a pre-ompile for example um that there is no enforcement to use a

specific library. Yeah, the problem the problem is not the specific library. The problem is that uh the testing team wants to hardcode this test case this impossible test case that will never happen with the with the contracts that we have. But they want to make sure and they want to hardcode this test. And um we kind of need to make a decision whether everyone should pass this test or whether everyone should fail this test. And but I I think we can also take this offline uh because this is consensus layer call. But yeah, that was going to be my next point. Um so perhaps we can handle this async and try to make a decision on what we want to do by next ACDE. So next week

that seems like a good way forward. Otherwise, let's see. So there were a number of bugs that have kind of come up here and there in the last week or two. Uh there was like a BLS issue that was found but has been fixed too. Maria says uh there was also uh some concern about the uh 7702 fee market that people are actively discussing. These are also primarily EL issues. So, I kind of would think to hold them off until next week. Um, but yeah, I don't know if anyone wants to touch on these right now.

I can quickly touch up on the on the BLS stuff. Um, there was an issue that was kind of in the spec. Um, Nether might have created a uh a a section in the spec differently than from what it was originally intended. Um, and I don't think that this issue was was with Nezerman. The spec was like very weird weirdly written. Um, that was the the one issue and the second one uh the second issue was an issue in EVM1.

Um both have uh both have been found by uh by uh fuzzing. Um and uh we added way more test cases to the um specs repo um the eels repo um sorry no east repo and uh we're doing fuzzing of all of the pre-ompiles in order to um to catch more of those. But I think hopefully the two are the only things that we we need to catch. Yep.

Thank you. Uh great. So yeah, number of Yale things uh we can touch. Oh, there is a hand. Oh yeah, Muel just I can briefly go um on the over the fee issue um kind of thing. Um there is an analysis um in the EIP assets um that uh the report of the analysis which highlights like most important things here. Uh first of all there is uh there is really um a kind of griffin vector um in the fe mechanism but uh uh first of all there is no dose vector in it. Uh the the the the logic here is that um kind of the model of

attack would be uh so how much it will cost for attacker to block the withdrawal functionality for certain period of time and uh it means that uh the attacker would need to keep the fee keep the excess at a certain level uh which at some levels of fee that are kind of prohibitive for rescuing um deposit of of a valer um at the prohibitive fee levels it will cost attacker a lot and uh even for 1 hour uh you can take the numbers from from this report. Um on the other hand there is the griffin vector

indeed. Um so there can be spikes in the fee um which are relatively cheap to to induce and uh one of the um one of the counter measures proposed is that we can get a um fee update per each request not per block as we do now. And uh if we consider like griffin surface uh with this uh improvements in the uh fee update mechanism in the fee mechanism will just reduce the surface of griffin not entirely eliminate it. Um so

uh there are there are other other approaches uh to like kind of uh get uh um reduce the griefin surface would be to increase the target so the uh fee goes down uh faster than it is today. Uh but uh if we do this it would mean that we stress CL side with more requests and uh CL will um at this point at these uh parameters CL kind of um parameters are optimal for from CL perspective to process those requests uh whether it's a partial withdrawal requests or a full uh withdrawal requests. uh and if we kind

of increase target uh and stress CL more so there will be low um lower room for for processing those requests. So it's a kind of like really um I would and recap is just yeah let's let's keep it as is. Now this is just my you know my my um opinion on that and happy if other people take at this kind of analysis and take a look at those numbers. Okay cool thanks. Yeah, I tend to think we should leave alone as well, just given, you know, there'd be a round of updates and testing, potentially an audit that would just add more delay,

which we should avoid if we can. Uh, Pis, you have your hand up. Yeah. So, my first reaction was the same. Uh, and and it still is, but but I perhaps it's farfetched, but I would like to see the other side of this report. So I'd like to see a report not by a core but say for someone actually in the dark forest or in the market side of things and and see if you can actually leverage the delay attack by shorting and running on on on state ETH or these kind of of things and see if you can actually finance the attack by shorting on the other leg. Uh, I haven't seen this this analysis done and I don't know if it's viable, but it smells like

the kind of things that you can't you could do. Why would the asset be relevant? like you suggested some kind of short with state well just a run I don't know a runup and then forcing Lido to not being able to return to redeem state ETH which would just lower the price DPEG uh state ETH and which would like probably cause a chain reaction on D5 I don't I I honestly don't know I'm not I'm a core dev not but I but I claim that I haven't seen an analysis by the people that actually work on these things on this topic yeah my understanding of the issue is it's not that severe. But yeah, that's a good point. Um, I'll try to find someone

to take a look. Yeah, this griefing uh what uh regarding staking and pools, I believe that sten pools will submit partial withdrawals for consolidated validators in batches and uh if if they can be gripped by someone and partial withdrawals can be delayed. This is a potential attack vector as well, but I don't know how severe is it. It is. Um, so it's like not delaying by I don't know by a day. It's delayed by some uh some amount of time like an hour um until the fee goes down again. I don't believe that.

But if you can if you can actually call if you can actually call a DPEG on state ETH and and actually do an arbitrage on that that I mean you could probably finance a very long attack. So that's why I would like to see an analysis done by someone that actually thinks along those lines instead of us that we typically think about other kinds of attacks on our protocol. Yeah. Yeah, makes sense. Agree. That'd be great to see. Okay. So yeah, I guess it is good that we're discussing these types of things in terms of uh I guess attacks out on the risk curve because it means the core of Peterra is stable enough. Uh but yeah, so there's a few things like this

uh everyone's keeping their eyes on and in the meantime, yeah, so to recap, we will keep Heski going. Uh we will finalize it and in the meantime, we'll launch a shadow fork. uh anyone who wants to test on hleski can test there and yeah I think the next sort of decision point would be finality back on heski and at that point uh we'll have more data to make a call on the next steps around mainet and launching pectra so I think that's it for pectra today anything else going once Okay, so next up would be PIDOS. Uh

let's touch on that briefly. Uh we canled the breakout this week. A number of people couldn't make it. Um we had some discussion around throughput the for Fuaka and I think explored this idea of like a BO uh roll out a bit more in depth. Um otherwise, yeah, I'm not sure how the DevNets are going. If there's anyone who could give an update here, potentially no update. Um, the definite is going well. Um, actually 100% participation, but however, all the nodes are super nodes right now. So, all the nodes custody all the data columns.

That's it for the net. Oh, sorry, I didn't quite catch that. Can you just repeat what you said? Yeah, sorry. Uh the devet so pure devet um pure does devet five is currently running for uh two weeks now. Um every client's team perform very well um 100% of uh participation. However, uh all the nodes are currently super nodes, meaning that clients do custody all the 128 colons. So basically there is no really shorting.

Right. Right. So yeah, so I think this is like a step along the path, right? So we just get this with the new mechanism up and stable test out and then yeah perhaps the next iteration of the devet we'll move to a lower distribution of super nodes. Cool something else. Yeah. Maybe that's maybe you can do that next week. Yeah, I hope so. Okay, cool.

Uh let's see anything else on Puros. Uh yes, I left a comment on the agenda about um a couple of open uh issues PRs that uh could use some attention from execution layer teams. Um, one of them is engine API spec changes and then one is related to sending uh the KZG proofs with transaction submission I believe. Um, but yeah, I think both of those some CL teams have looked at but it would be good if execution layer teams looked at.

Okay. Yeah, I can make sure this gets in front of them. Uh, and definitely it'll be on the agenda next week. Um, yeah. Thank you. Anything else? Okay. If not, everyone, good luck with Hasheski and the rest of Pekra. Uh, it's been a lot of work and we're in the middle of it, but uh, we will get we'll get it back.

And yeah, if there's not anything else uh for today, then we'll go ahead and wrap up. Okay, thanks everyone. I'll see you next time. Byebye.