All Core Devs - Testing #78

Transcript

Mario Vega: Hi, everyone. We'll just give a couple more minutes, to give time to join, and we can start.

Alright. Do we have everything to start the recording?

David (dionysuz): Yeah, it's… it should be ready to go.

Mario Vega: Alright, thank you. All right, so welcome, everyone, to ACDT number 78. Today is April 20th. Mainly we'll focus on discussing DevNets today. So, yeah, let's just get started. And the first topic is blob DevNets. Blood Delta Serum, and I think Barnabas, that's, Your domain? Do you have an update on this?

Barnabas Busa: There's no, currently, any updates. I think most of the CL teams are focusing on, shipping ePBS. We currently have still just Prysm and Lighthouse deployed on them. Last week, we have enabled partial cells on every single node, and we're experimenting with that.

Mario Vega: Alright, thanks. Any points to be raised by any of the client teams? We're gonna need a lot of minutes.

Barnabas Busa: It would be good to start onboarding other clients here as well.

Mario Vega: Which are the ones that are currently, joined and not, and who is missing?

Barnabas Busa: It's Prysm and Lighthouse only. Everyone else is missing.

Mario Vega: Alright, anyone from Nimbus? Lodestar, who want to chime in?

Nico Flaig: Yeah, for us, there's still a few pieces missing we need to get done, probably this week.

Mario Vega: Alright, thank you. Anyone from Nimbus? This is blob.net server, by the way.

Nico Flaig: Oh, never mind my comment. I was thinking ePBS, Definite.

Mario Vega: Oh, I see. Got it. Alright, I think… I don't think there's anyone from Nimbus. Should we jump into the next topic, then? Or, Barnabas, do you want to… .

Barnabas Busa: Yeah, because my wife.

Mario Vega: PowerPoint Alright, thanks. Yeah, let's move on straight to, blog level access list, DevNets. There was a comment in the agenda today by Marius that, he suggested that we should jump directly into Glamsterdam.devNet 0 instead of going into, Block Level Access Lists.devnet 4. So Marius, would you like to, explain on the topic?

Marius van der Wijden: Oh, give me a second, I'm in the room, and my fiance is also in the call, so… Okay, so… Basically, what we want to do is we want to move, to a world that has all of the changes for Glamsterdam as soon as possible. And there's two ways to do… of doing it. We can either go to VelDefNet 4, or we can go to Limited DevNet 0. What we would gain by going to BAL DevNet 4 is that there is a… That, it's, like, regardless of whether ePBS is happening on… on the consensus layer, we can, we can continue testing stuff on the, on the execution layer. But what we can instead also do is just go to glimpse at M, definite0,

And then… Get an old version of the consensus layer, to create a network. Based off of that, if we feel like the contents layer is not ready. for, for Glamsterdam yet. Yeah.

Mario Vega: So, Barnabas, you have, the other, suggestion, right?

Barnabas Busa: Yeah, so basically, we want to make sure that the PBS branch is as stable as possible, so we're not going to be doing any kind of pausing yet in the early days of Amsterdam DevNet Zero, for sure. So, if you want to do fuzzing and benchmarking and stuff like that, I think that should be done on Ball-specific DevNets, like BAL DevNet 4. And ideally, Glamstreet and DevNet3 should be, based on top of, ball DevNet3 plus, the PR770 from Execution APIs.

Mario Vega: Yes, produce?

Potuz: I want to support what Barnabas is saying. I think, the Glamsterdam DevNet Zero should be thought of as an ePBS DevNet. It's not going to be stable. And it doesn't matter for us that it's not so stable, and it doesn't matter if Block Level Access Lists wants to be testing, like, new features on that DevNet, because it's very unlikely that we're gonna be hit with problems that are gonna be… more complicated for us to debug, but I suspect that for EL devs, debugging on an unstable consensus chain is going to be much more complicated. So, my advice for EL devs would be to just have a second, just purely EL, just purely bald DevNet, while we test consensus on the Glamsterdam at DevNet0.

Mario Vega: Okay, Johnny?

Toni Wahrstätter: Yeah, or I can also just echo that… what Potos just said, and also what Barnabas said, just with one small caveat, because Barnabas, you said, we should then build, glow as definite. or Amsterdam Definite 0 on top of BAL DevNet 3, but we are currently working on BAL DevNet 4, so I guess you mean we build on top of BAL DevNet 4, right?

Barnabas Busa: No, because I don't want any 8037 HAVOC on Glamsterdam, that net zero. That's why I.

Toni Wahrstätter: But we have it on Definite Free already.

Barnabas Busa: No, those are using static values. Yeah, but those are using the static values, not the dynamic ones, and as far as I know, the dynamic values are the ones, that, like, require a huge, rebate.

Marius van der Wijden: you know.

Barnabas Busa: the execution of the airplanes.

Marius van der Wijden: No, no, no. No, that is just changing one value. I'm doing a rebase because, because we've merged some stuff, into… into… into master and… and rework some of the stuff. That's why we have to do rebase. It's not because of the dynamic value for the… for AT37. Basically, like. whether it's a static or a dynamic value shouldn't make a big difference. The calculation is pretty easy. The… the bigger point is… is, whether, is whether 8037 is correctly implemented, and that is… Testable regardless of what the cost per state byte is.

Barnabas Busa: Alright, but then, could maybe Dragon mention what he means under disruptive changes?

Dragan Rakita: Maybe that's true, forget, but for our EVM, we still need to implement those dynamic changes, and there is additional, like, 4 or 5 additional changes that needed to land on the state gas. So yeah, there is… Yeah, there's multiple trains that need to be done.

Marius van der Wijden: Yes. Like, there are changes that need to be done, also for the GAF side for Definite 4, but… Like, those are not super traumatic. God, I would say. Anyway, like, it seems like everyone is, is, yeah, for making definite for… the next, thing for… well, definite 4, and… but I, I think we should not make Glamsterdam DevNet. 0 based on, BAL DevNet 3. That would be a pretty huge mess for us.

Mario Vega: Alright, let's do Maria, D'Antonio, and Spencer.

Maria Silva: Yeah, I think I was just going to… to add that the complication on the dynamic cost was mostly around the test, so the Steel team had to rework a lot of the previous tests to, to be able to take these dynamic costs into consideration, and my understanding is that most of the work there is being done. Then, I would agree with Dragan, so we do have some spec changes between, BAL DevNet 3 and Bold DevNet 4. But I feel… I don't know, it's still up to the clients to decide, but I think it would be easier to manage to just have Paul DevNet 4, and then have Glamsterdam feed from… from that development branch, because implementations will be slightly different, and so it's… it's a bit weird to…

be using, like, old implementations of AT37 in the… interop, DevNet, and then, on the side have, like, continued to work on the new, specs. But yeah. Happy to also hear your thoughts there.

Mario Vega: Tony?

Toni Wahrstätter: Yeah, I agree with that, what Marius and Maria just said. We should do DevNet 4, and then have the first Glamstadom DevNet also use DevNet 4 instead of doing Definite 4, and then have the first Glamsterdamdam Defnet, basically. Go a step back, and… be a DevNet-free implementation.

Mario Vega: That's true.

Spencer Taylor-Brown: Yeah, I guess from my side, I think it depends how… like, how quickly do we want to test a glance at RamdevNet? If we just want to quickly test the integration between the EL and the CL, then using Belle DevNet 3 feels good, but if we want to use BAL DevNet 4, then it means the first glance at MDevNet's probably going to be delayed. And… Probably towards the end of next. next Friday.

Parithosh Jayanthi: I think at least timeline-wise, we should definitely be aiming for Late this week, or early next week, because… If we have stuff in person, then it's going to be most productive if we're not waiting till the end of the week to have it done.

Barnabas Busa: Exactly, and that's why I proposed that we would do, bold.net 3, instead of Bold.net 4. Like, we ideally want to start the Glamsterdam DevNet.

Marius van der Wijden: Barnabis, like, we… like, we have most of the changes for BAL DevNet 4 already, and, like, it takes us another day or two to get a version of BAL DevNet. for done, for all clients. And then it makes no sense to… to test a version of… of AT37 that is, like… I don't know.

Parithosh Jayanthi: But then where does Spencer's timeline of end of next week come from, if the changes are 1-2 days?

Marius van der Wijden: I don't know. Like, definitely for us, it's not… it's not one to two days. It is… sorry, it is one to two, two, two days. I need to do a rebase of, Of the stuff, and then I need to implement the seven changes that we discussed on top. And that should be done in a day. Maybe two.

Spencer Taylor-Brown: I guess I'd add the reason I said, it'd be a bit later is just… each individual new spec change, so I think there's 7 total to 1837, has, like. added already a lot of… a lot of extra testing that has taken… taken a while, and I think… I mean, maybe we'll be very slick in person, and It'll be very quick for us to… Like, iron out all the bugs in person.

Marius van der Wijden: like, do you have… do you have tests for BAL DevNet? 4… already?

Spencer Taylor-Brown: No, no, not yet. I was hoping to have it released for this call, But yeah, we're… we're still ironing out, some… some of the AT37 changes. But yeah, well, I think we're just filling the tests now, and trying to fix… And the remaining changes. Yeah, and I think most of the fixes are… Due to the dynamic… Cost per state byte, addition. But we're close, we're very close.

Mario Vega: Yes, Stephan?

Stefan Starflinger: Yeah, I would also propose, to have… to launch, BAL DevNet 4. We can have it very short, but then to give clients to iterate a little bit, and then quickly afterwards launched Glamster Dam. Definite zero, that would be my suggestion. It doesn't have to be very long-lived, but just… To get some fixes in, and then focus on GameStream DevNet Zero, or we can have both running then at the same time as well.

Barnabas Busa: My main question would be, is there any other EL teams that feel, sufficiently ready, basically, to have 10.4 changes? In a feature branch?

Toni Wahrstätter: I guess the question is by end of week, right? Or what's the timeline you're now thinking of, Parabas?

Barnabas Busa: Well, I'm trying to do local group assistance tests before we launch a .NET, so that usually requires at least another day. ideally by Wednesday, if we would have… like, Mario said that, they should be ready by Wednesday, so that's why it would be good to have at least another client or so by Wednesday.

Spencer Taylor-Brown: We'll have tests, tests out by the end of today, at the latest, early tomorrow. Yeah.

Mario Vega: So, to add up to that, I think, we can reduce the number of tests by reducing the coverage, if something is not… does not go quite smoothly. But we'll try to have something. Bye tomorrow. Or Demophore. Ball metaphor.

Marius van der Wijden: Instead of… instead of reducing the number of tests, can you just undo the, the dynamic state cost thing? Because, like, that would be, for us, very easy to test. We would just undo that change. We would… we would run all of the tests, all of them would be passing, and then we would just… Redo the gas cost change, and then… Like, if this is the… if this is the issue with the… why the tests are not… are not published yet, then, I think that should just be undone. Because, as I said, I don't see many… Issues happening with that. Well, the, like, the behavior changes that we are doing to 8037 are more complex and would require, like, specific test cases.

Spencer Taylor-Brown: I think… I think we'll be able to get them in. Like, we're… we're… we're basically there, yeah. M.

Mario Vega: Just to be clear, that's a 4 spec, Are we all on the same page? All teams? Alright, so, just to reiterate, we're aiming, for spec. For Block Level Access Lists, DevNet 4, which will become Glamsterdam DevNet0. Is that… is that correct? Barnabas, or anyone else?

Barnabas Busa: Yeah, I'm still waiting for at least one more client to speak up about it, because last week we were still hearing a bunch of people that were doubting that they could be even done, by end of next week, which means end of this week.

Mario Vega: Can other clients comment on this? How about Nethermind Besu? Yeah, Besu, please.

Daniel Lehrner: Yeah, I think as long as we get the tests, we could be ready. So, I mean, we have most changes prepared in draft PRs, basically. I'm a bit afraid of the 80-47 changes without tests, because those are really easy to get wrong. But I think if we get those, we should be ready, yeah.

Barnabas Busa: Yeah, in that case, we can definitely do DevNet4, with the new two new EIPs and the dynamic, changes plus, yeah, plus the PR from 770. Which we should discuss next, I think.

Mario Vega: Alright, before we jump into that, there are, any other ideal clients, with objections to… The current, spec, that we are aiming for. All right. If not, I think, just to reiterate, we're aiming DevNet4, so full 83.7 for DevNet 4. And this will become Gamson Devidence Zero. And the last top… the last, PR to discuss would be the Execution AP's, PR, so let's… let's discuss that. I just linked that in the comment, so, for everyone to see. Who is in charge of this, PR? Who can give us an overview?

Mikhail Kalinin: I can give an overview. Can you hear me?

Mario Vega: Yes, perfect.

Mikhail Kalinin: Alright, so… this PR, basically. Changes the, Oh. the behavior of EL when CL is, like, kind of in sync and way behind the head that EL is aware of. Previously, if, FCL, Sends the head block. that EL already has validated many, many, much time, a lot of time ago. In the case when this book is from the canonical chain, it's ancestor of the head that the EL wears of. Basically, this back was… Saying that. In this case, the folk choice Update. Can be skipped. And in the case when it's skipped, there is no payload build process triggered. And, yeah, that was before Gloss, and worked before Gloss.

But now, since gloss, we need to… build payloads, and we need to rework out payloads, canonical, like, from the canonical chain, from the point of view that EL thinks that it's canonical, because it was previously, pointed out as the head. So we need to rework back to, like, ancestor of the head of canonical chain, and be able to… to build payloads on top of ancestor of… Of the head of canonical chain. Yeah, that's… that's what this PR is about, and also it introduces some limitation on how deep this reorog could be. Which is 32 blocks, basically, and… Yeah, but this limitation only… matters for the case where we do rework from the head of canonical chain to one of its ancestors, but not to any other kind of reworks.

Mario Vega: Marius?

Marius van der Wijden: Yeah, so, I, I left this PR, I, It's, it's, it's a 5-minute thing for… for every client to, to, to change. Basically, the way we, we, we have discussed it, in the gas team is, So the way where we had this rule not to reopen the head block was, in order to get, yeah, to filter out faulty or late messages from the… from the CL, that would, yeah, basically, show a weird state to the user, by going one block back. what we're doing here now is just to track more closely the fork choice of the CL, and we hope and we trust that they're not doing weird things with it, so, yeah, but this is anyway an assumption that we have. So, yeah, this…

should be really a no-brainer. I've… I've implemented it twice today. And in… in… within 10 minutes, so… I think every, every client should be able to do this. Yep, that's it.

Mario Vega: Maputos?

Potuz: Yeah, so I think the phrasing on the PR of, like, the block… the limits being 32 blocks for this reorg, it's off. ideally, what EL clients should support is a reorg up to the finalized hash. And if clients cannot deal with very deep reorgs, that's fine. The spec says that you should be able to support up to the finalized hash, and if you can't, then just put in your implementation some limits that are reasonable for the implementation themselves. But I think it's… not correct to have the specification, allowing for this sort of reorgs up to a, like, fixed… any fixed number of blocks.

Mikhail Kalinin: I think it's not, like, fixed. I mean, yeah, it could be… any number. I think we need some sane limit. And finalized, realled back to finalized. Yeah, I'm just, you know, it's up to EL implementation, so what do they think about it? And, for instance, what happens, then the question, what happens if… If EL does not support a reorg of, you know, of 10,000 block depth, What should it… Return, in this case. Probably it should be an error.

Potuz: I don't…

Mikhail Kalinin: Or something like that.

Potuz: It's fine, because it's only on the proposer side. If you're proposing locally, you're self-building and your EL cannot support this. then you will fail just proposing your local block, but you can just still take bits, and you can still propose a block. And for anyone else in the chain, it doesn't matter, because if there is such a long reorg, anyone else that actually gets that payload will see a non-S reorg, not one of these going back to head. So it only affects the proposer side and in the self-building path. So I just don't see any reason why we shouldn't just put there that you're… you should be able to reorg up to the finalized hash.

Mikhail Kalinin: Then what will happen? I don't know.

Potuz: Then you get a blog.

Mikhail Kalinin: Yay.

Potuz: And then you have a branch, and that's just the usual case in which the EL should be able to support the reorg up to the finalized hash. That happens today. Right. The EL today should be able to reorg up to the finalized hash.

Mikhail Kalinin: Yeah, just imagine the case where we have a long period of non-finality. And then…

Potuz: Happens today. This is today!

Mikhail Kalinin: Right.

Potuz: specification.

Mikhail Kalinin: But, yeah, so we had this… In this fact, so not… not do a reorg if the CL, like, syncs with, like, the chain, right, with the network. And it tries to catch up with the head, and it sends already known payloads and payload hashes, and yeah, and FogChoice updated as well, pointing out already known and validated in the past blocks by EL. So, and in this case, if there is a long period of non-finality, in this case, EL will have to do a reorg to a finalized block regardless. So, yeah. I tried… actually, this PR just tries to do something in between, still allows for this optimization. Let's name it this way. And, allow for… For reorgs to, you know, if it's, like, quite close to the head, to…

Yeah, to address the case for a PDS.

Mario Vega: Tony.

Toni Wahrstätter: Yeah, my question was, just answered by Marius, so I was curious, like, because Podos mentioned it's only for the proposal, but I was interested in, yeah, how would this work for a test desk? Would they have the state on the EL site available? But Marius just, replied in the chat that EL clients would basically need to resync in such a situation. So I guess it's not super smooth if we ever had such a long reorg, because this would mean The nodes could not attest, timely, because they would need to re-sync in order to get the… state. Of that pivot block.

Potuz: That cannot really happen. If clients cannot support a 90,000 block reorg, then that chain will not have the attestation weight to actually cause the reorg. So that's… I think it… we're just discussing over nothing.

Marius van der Wijden: Yeah, so, I… I like… I like the limitation to this, one, because what we have seen in the past was faulty CLs just feeding us old blocks and… and causing… and then, like, we ignored them, but this would now cause us to reorg back to old blocks. So I think it makes sense to me, to have a limitation there, how deep these… Like, these reorgs within the same branch can take. And yeah, as… as Mikhail already said, this doesn't… This doesn't, impact how reorgs to a different branch, take. Right.

Potuz: But the guy thinks… you guys are dismissing the fact that we need to change FordChoice to put this kind of limitation, because otherwise, we actually might get into sort of, like, a deadlock. If FortChoice is saying, you should reorg up to, like, say, 64 blocks. and the EL cannot handle this, for whatever reason, then…

Marius van der Wijden: Why…

Potuz: CL, not advanced, because that's…

Marius van der Wijden: Why would…

Potuz: We should be building on top of.

Marius van der Wijden: But why would the fog choice ever need to reorg 64 blocks on a… A chain without any new block.

Potuz: No, no, so 4Choice would need to reorg on whatever… that's the assurances that we have. We can always reorg back up to our finalized checkpoint. It may not happen, it may be very difficult to happen, but this is what 4Choice is. That's what… that's what FFG guarantees to us.

Mikhail Kalinin: You mean, like, you… you need… you are talking about a reorg on the same chain, back to finalized block? Without, like, you know, introducing Without, like, introducing a competing block, right?

Potuz: Correct.

Mikhail Kalinin: Why would… why would, this kind of rework need it?

Potuz: Well, because this is what fortune is. We don't have a proof of fortjoice, we don't have a proof that this is impossible to do.

Mikhail Kalinin: Yeah, I'm just trying to find… find a way, you know?

Potuz: It might be impossible, that's… that's the whole point, and… and it's… It's fine if the spec says we should be able to reorg up to finalize, and implementations cannot handle this situation. Today, the spec says you should be able to sync any block on top of an unfinalized block, and that's what the spec says. Actually, the spec doesn't even say that it needs to be, well, it doesn't matter. In principle, clients should be able to sync any block. On top of a block that is not finalized. And we just can't handle this. We get dust, and therefore, we ignore some blocks. That's just fine. So, I don't see why we would put an artificial limitation where clients can just deal with those limitations. As Mario's saying, the clients already are not following the spec, and they cannot handle a reorg that is deeper than 90,000 blocks.

Toni Wahrstätter: I think they can handle it, they just need to re-sync. That's the… that's the thing.

Potuz: No, that means not.

Marius van der Wijden: No, no.

Potuz: Because if I don't get a… if I don't get an accepted, or if I don't get a valid there, then we just can't, we can't be in sync.

Marius van der Wijden: Yeah, but the… like, that's… that's the thing, we always had this… we always had this guarantee that the folk choice will never… Flip as long as there is no other chain, no other block. Where, where, like, it will just not go…

Potuz: Like, what happens if we're not finalizing for 90,001 blocks, and I just produce a beacon block on top of that one, like, 90,000 old block? You're telling me that you cannot handle this?

Marius van der Wijden: I'm telling you that we cannot handle this, but this is a different… this is a different scenario from what we've been talking about. is, is, reorgs within the same chain.

Potuz: I understand, but the point is that there's already a situation in which the spec says something, which is what ideally clients should do, and what actual clients can do is true.

Marius van der Wijden: Yeah, yeah, I agree, I agree, and the thing is, we can handle 90,000… guests can handle 90,000 blocks. Good luck trying that with other clients. I think they can handle, at most… maybe a thousand blocks, if you're lucky. So… And even for us, the 90,000 block reorg takes several seconds. Probably minutes, by this point.

Mario Vega: you know, Mark.

marek: Yeah, so what I wanted to say is that, on one point, we brainstormed the idea that basically, in case of long reorg, because it's really problematic for us to handle such long reorgs. But on the other hand, we can just switch to, let's say, archive node in this case. And basically, we will use more storage. But that's it. We will handle this anyway, so this is not fully implemented in Netherlands, we can do it like this, and I think I agree with Potus that this is what we should do.

Mikhail Kalinin: So it also kind of implies that the spec since Paris, was incorrect about what we are talking now. I mean, like, with respect to what we are talking about now. So, it should not have this optimization at all. And what worries me is that, for some reason, we just remove it now, and yeah, we're required to get reoralk to final… back to finalized. Which is probably… Fine, if finalized block is not too far from the head. But then what happens if, you know, if this starts to… if this… Is the case on some… testnet, or whatever. That we've seen in the past. So just… I'm just trying to think about potential implications of that.

Marius van der Wijden: Yeah, the bigger problem is that we don't have this finalized state until, until… basically until we get to the head. Like, we don't know when… when is the final… what block is final until we get to the head? And so there's… there's, like, this… there's many cases where we don't even know Where the final… where the final block is, and what… where the finalization point is.

Potuz: relevant in this case, right? Because this is only for a proposer that should be already in sync. And for what it's worth, unless we are under an attack.

Marius van der Wijden: Okay.

Potuz: I think this limit could be just 2. Like, we would never need to reorg more than just to the previous block.

Marius van der Wijden: Yeah, I agree. The problem is, if we, if we, if we don't put a limit there. Then… and the CL tells us weird things, then we might just not be able to sync. Because we keep… we keep getting, like, just imagine having, having tools.

Potuz: It's fine, that's an EL that is already a faulty EL, and you shouldn't be syncing in the case that the EL is faulty. And if the EL is not faulty, then we are under a situation in which we should be reorging. So, it'll cause a deadlock either way.

Marius van der Wijden: Just, like, just imagine, like, right now you can run an EL off of two CLs. like, you have two CLs that feed you different points, you have one CL that is at the head, you have one CL that is at, not at the head, that is currently syncing up, and it works, because we specifically just ignore all the updates from from the one CL that is out of sync. If we don't have this limit, what will happen is that this one CL will go forward, the other CL will tell us, hey, you actually need to go 5,000 blocks back. We will happily go 5,000 blocks back, then the other CR tells us, no, you have to go 5,000 blocks forward, we will go 5,000 blocks forward.

So we will not ignore the, ignore the blocks from the out-of-state CL anymore. This is… this is why this… this limit is, is there.

Mario Vega: It sounds… Barnabas, it sounds like there's… needs to be a change in this… in the PR. At least to me. Is that correct?

Marius van der Wijden: I don't think so.

Mikhail Kalinin: Yeah, my… my… Yeah, my opinion is, I'm just… I'm fine with, like, introducing an error, which will say that we will not reorog, you know. Beyond some. And yeah, yeah, and beyond some, you know, some depth, MDL will just… each EL implementation will pick up Own, limitation on that. We're gonna say that It must be at least till the finalized block, which is some… Yeah, which is kind of like… for EL finalized means that it actually doesn't know where a finalized block is, right? So it's, like, arbitrary, Depth. So we can just have some kind of 90,000 block limit on that, or whatever. So… Yeah, but that's, like, up to EL client developers to decide on what should be spec'd out there. Again, this PR tries to find a middle ground between the optimization that we had since berries, and the new requirement that Gloss actually has.

That's all. If that's gonna be an error, whatever, yeah, let's just do it.

Mario Vega: Alright, can we, wrap this discussion and continue on the PR? Does anybody else have any questions about this? Alright, the follow-up question would be, is this going to be introduced in Devna 4? Is this a requirement? .

Barnabas Busa: Yes.

Mario Vega: I feel like we're leaning towards… yes, okay.

Marius van der Wijden: Well, no, it's not a… it's not a requirement for… for BAL DevNet 4, because it's not used by BAL DevNet 4, but if we ever want to do, ePBS testing on top of BAL DevNet 4, then… then we would need it, so I would also say that we should do it, but it's not a… it's not a requirement for BAL DevNet 4.

Parithosh Jayanthi: I'd also say we're probably not stressing this exact path, right? We need quite deep reorgs, and right now we're still doing relatively happy case testing anyway.

Marius van der Wijden: No, ePBS, triggers this path quite. Easily.

Parithosh Jayanthi: Cool.

Barnabas Busa: Yeah, we had this issue already on .NET1, or .NET 0.

Mario Vega: Alright, regarding the requirements, is it okay, Barnabas, that we create, the… the note with the description above them before, without this PR, and then on Glamster then zero, we add this PR as a requirement for EO clients to be able to join. Can't… Okay.

Parithosh Jayanthi: And is this implementable by all clients by next week?

Barnabas Busa: This week, end of this week.

Marius van der Wijden: Yeah, this should be implementable by all clients by the end of the call, and we have 20 minutes left, so get to it.

Mario Vega: Alright. Great, so we have agreement. Good. Do we have, do we need to break into… into break… into the breakout rooms for, discussions in specifics to block global access and ePBS? Yes, Potus?

Potuz: Before any break, I would like to talk about syncing, again, just 5 minutes.

Mario Vega: Yep, okay.

Potuz: Oh, can I do it now?

Mario Vega: Yes, yes, please do.

Potuz: Alright, so it's the same thing that I raised on ACD, and I would try to raise it again. I want to… and no one has returned on us with this. I really want to have some eyes on, potentially removing completely the payload envelope syncing… range syncing on the CL. There is… So the question is to analyze the security, or lack of thereof. from range syncing on the CL side without even requesting the payload envelopes. the CL could perform all of the state transition on the CL side without ever taking any payload, without even knowing if the payloads actually existed. And then, when it catches the head, as soon as it sends a new payload, the EL should be able to tell if it's invalid or not.

So what I'm proposing is to change the notion of optimistic sync in this setting. So the CL would not even check the block hash of the blocks, because it wouldn't even take the payload envelopes. Another Problem with this would be that, the CL… would not verify the signature of the payload envelope, because it's not even requesting those payload envelopes. On backfield, if the CL wants to backfill and get those envelopes, then we could do a batch verification of those signatures. So I'd like to understand what is the trade-off space here, because CPBS allows us to sync without ever requesting a payload.

Mario Vega: Alright, are you looking for input from the ELs, CLs, or what's the, approach here?

Potuz: From anyone, but mostly from… the EL, because Optimistic Sync is something that is quite complicated on clients, and I believe that most of the edge cases had to do with the merge itself. So, I believe we can just remove and simplify quite a bit of optimistic thinking. We can just sync without a payload, and when we're on gossip, validate payload just on Gossip.

Marius van der Wijden: Hmm, that's it. Does it matter for the ELs? Like, as long as you, at some point, give us a payload, we will be doing SnapSync anyway.

Potuz: Well, that's… so if we trigger SnapSync, that would be a big problem. So the security assumption that we do not want to lose is that an EL that had the state at some point, and we're just, like, restarting or whatever it is, we do not want to trick an EL into following a finalized but invalid chain. If this can happen, and if this can happen because of this change, that would be a problem.

Marius van der Wijden: Okay, so what you're saying is that the year is… is… is at some point? you shut it… you shut it down, you… you restart the CL, the CL does some… some sync to the head. Without giving us any payloads, and then it tells us this is the block that you're syncing to. And then, in that case. What happens is the ER will go and… and fetch those… this data from the network anyway.

Potuz: Exactly. And today, for example, my house, like, fetches it twice.

Marius van der Wijden: Yeah, that's… Sounds unnecessary.

Potuz: Yeah, so what I would want to do is not even fetch it, so not even check the block hash, not do anything on the CL on this, and just give you, later, when I catch up, give you a block, and make sure that if this block is a descendant of an invalid block, even if it was already finalized on the CL, I want to make sure that the EL would just tell me this is invalid.

Marius van der Wijden: Yeah, I think that is already working.

marek: Yeah.

Marius van der Wijden: for us.

marek: But is it, like, the problem with some concrete EL, or you just want to make sure that That's how ELs work nowadays.

Potuz: Yeah, so I want to make sure, and I want to, like, to have it written somewhere so that we can actually get into agreement. It sounds to me that there is no reason for the CL to keep payload envelopes, and there's no reason for the CL to even serve by range payload envelopes.

marek: So, my only comment is that it makes sense to me, it's how Nethermind works nowadays, but the only thing is that it's quite a tricky scenario that you cannot easily trigger, so we need to make sure that it is really well covered by tests, and there were… some of the tests were, Flaky a little bit for us. because there are many testing hives that are checking it, that you… you will return correct, invalid hash, and so on. So, I guess it should be fine, generally, what you are talking about.

Potuz: Yeah, so what I would want… so this is the concrete request. The concrete request would be to EL devs to analyze the situation, analyze what are the security, the problems that this could cause. For example, one that would be dramatic would be if such a situation can trigger all ELs into snap syncing. Because if that happens, then my understanding is that, that could force… that could trick the EL into treating a finalized but invalid block as valid. And this would be an exception. acceptable.

marek: So, sound…

Mikhail Kalinin: Please repeat that, sir.

marek: Yeah, go ahead.

Mikhail Kalinin: Just, photos, could you please… could you please do a recap on why C… for CL with, like, in a PBS case, it's… it becomes a problem? Don't exactly…

Potuz: It's the other way around. ePBS allows… today, the CL cannot sync. Without getting the payload, without downloading the payload, because the payload is within… is with the block. So the way we sync when this EL is not catched up is we verify the block hash, either we do it, or the EL does it for us, and then we continue syncing. But we download the full thing. ePBS allows us to sync without ever downloading the payload. to which we would fail… we would just no longer verify the block hash, and moreover, we wouldn't verify the signature on the CL side that the payload envelope carries. But I believe that this is safe, anyways, because of what EL devs are telling me of the way they sync.

Mario Vega: Alright, I feel we have to park this conversation. Bottus, can you, coordinate to do a, write-up on this topic to see if we can keep discussing it.

Potuz: already written… most of the scenario up, and I've already posted it on ACD. I'll ask to, again, bring this up on ACD next time. I'm hoping to get someone to, I mean, ELDFs to look into this and actually produce some written statement.

Mario Vega: Yeah, sorry about the rush, it's that just we have to go into breakouts, because we have barely enough time. Is that okay? Alright. Yeah, let's try to jump quickly into the breakout rooms. One for ePBS, one for Block Level Access Lists. And, probably if people have more time, we can stick a little bit longer. I know these… I know these have already 15 minutes, but let's just try to… Let's try to finish the discussion topics that we have today. Alright. Who's helping today with, breakouts?

Parithosh Jayanthi: I think it's Tony and Justin.

Justin Traglia: Yep.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.