Block-Level Access Lists Breakout #8

Transcript

Toni Wahrstätter: Hello, everyone. Let's wait for one or two more minutes. Give people some time to… Sure.

Perfect. I think we can get started.

Akash | ECH: Real life.

Toni Wahrstätter: Great! Welcome, everyone. Today is December the 3rd, EAP7928 breakout call number 8. Let's get started. I just posted the agenda into the chat. Today, it's very lightweight, I would say. We have 3 topics to discuss, starting with one change in the specs and the EAP, more a clarification around how to treat, calls that send value but run out of gas. The second one is storing the block of access list in the block body versus not storing them. We discussed it already last time, but wanted to double-check with everyone. And finally, going through, optimizations needed for the benchmarks we wanted to get in order to determine if the read values are worth it to include in the block lab access list or not.

Perfect, and directly… let's directly jump into the first topic, which came up as there were some differences in how client implemented The gas accounting when it comes to coal. to a recall opcode. And basically, it… Depends on when do we check, when do we do the static checks, versus when do we actually go to the state. And… I will post this, PR into our chat. One second… There it is. And I'm curious what clients think about it. So right now. What we do is we, in the specs, we now make sure that We first check all the static checks, which is memory expansion, access costs, and the transfer costs, and only if they succeed, we go to disk. And the tests showed, like, differences between

I think it was between, Beso and REF versus Geth. I think Geth did something different there. And I wanted to double-check with, clients, if this is something we can agree on, how it's now defined in the EAP. I think it only needed changes for GAF, right? Is Jared on the call? Maybe Jared, Do you have some insights there, or is this something you guys are about to change?

Jared Wasinger: I just joined, what, what, what are you talking about here?

Toni Wahrstätter: Yeah, it was the topic with the call, so basically, checking all the static gas costs around, memory expansion, access cost and transfer cost before actually doing the state access.

Jared Wasinger: Yeah, actually, I'm working on that right now, and I'm hoping I can get the test coverage around that, fixed pretty soon. In fact, I think that there's a possibility that that might be the cause of the chain split that we see when we try to run local DevNets between Geth and Bisu, so, hopefully fixing this also, makes the devnets more stable. But yeah, I'll have an update on that, soon. That's, like, my top priority right now.

Toni Wahrstätter: Awesome, thanks a lot. I'm just relaying Stefan's message here in the chat. For memory expansion, Nethermind also has an inconsistency versus Ref and Besu. Someone from NetherMind who is already aware of the topic, And Nicole?

Marc Harvey-Hill: Yeah, sorry, what was the question, sorry?

Toni Wahrstätter: So it's basically, in the specs so far, it didn't matter that much when you do the state access versus when you do the static checks for memory expansion, access cost, and the transfer cost. And what came up during the testing was that, for example, if you have a call that forwards one EV, But the account only has… 0.1 EV, Then the question is, should the target address that is about to be called, should it be included in the block level access list or not? And I think Beso and Rev did not include it, which seemed right, because you can do all the static checks before actually going to disk and actually accessing this account. So, it's… So it was probably NetherMind and Kef that

Had this state access before doing all the static checks.

Marc Harvey-Hill: Right, yeah, I mean, I don't have… any opinion on it, really? Like, I'm happy to change to what Bessu and Wrath are doing.

Toni Wahrstätter: Okay, great. Maybe, Belipe, as you're also on the call, is there anything you would add to this topic? Because I… We have been… Going back and forth in the specs regarding that, so is there anything you'd like to bring up?

Felipe Selmo: to make sure.

Toni Wahrstätter: We don't forget.

Felipe Selmo: No, no, I think we covered it. I am a little curious, though, because… We do have… This is a test, and another mind is… Passing all of them. So I'm not sure… Stefan, where they have the inconsistency.

Stefan Starflinger: That's for memory expansion. I'm not sure if I shared the right link. When there are different times of, gas accounting, and never mind, reads the target. I think it's the target. I'm not 100% sure, though, what exactly NetherMind reads before doing the memory expansion gas accounting. The other gas accounting seem to be in line with Bazel and REST, just the memory expansion seems to be a little bit delayed.

Felipe Selmo: Yes. Okay, yeah, you didn't share the right link. This is the PR that you added a test for, and I merged. I'm pretty sure. Yes, exactly.

Stefan Starflinger: This is… this should be the right order.

Felipe Selmo: You know…

Stefan Starflinger: So this is just an inconsistency I found with memory expansion. I think that's the last thing so far, and then Netherlands should be, aligned. I think you just merged this recently, so it might not be yet in the tests, because I think you have to make a release I'm not sure if you've done that.

Felipe Selmo: Perfect, yeah, no, I thought we were talking about the, the state access. I remember this one. This is not yet in a release. We're waiting for… One more PR… And then we're… I think we're also going to be able to turn on all the old Ethereum tests with BALS, and so… I'm trying to iron out some of the inconsistencies there. And then we can get a release out with even more tests. That we can try to get consensus with all clients on.

Toni Wahrstätter: Awesome, this is great to hear. So, it looks like the specs are correct, and the test should also be correct in this case. Because then we are basically ready to test all the clients and make sure, we are all aligned. Awesome. Is there anything else regarding this topic that we should discuss? It's a very interesting topic, by the way, because it's… it's nothing that broke consensus so far, but can break consensus in the future, because suddenly, like, when do we access state will become part of consensus. Is there anything people would like to add to this topic?

Ben Adams: Yeah, I think it makes sense, to do the checks. To bring the checks that might cause it to go out of gas. Before the state access, because if you're gonna go out of gas. You know, don't bother accessing the state. It makes sense from a performance standpoint.

Toni Wahrstätter: Yeah. Thanks a lot for that, yeah, I agree. Also, a lot of thumbs up here from many people, so… seems like we have a very broad agreement here, regarding this topic, which is great. Okay, so then I would suggest we move on to the second agenda item. Which is, summarized as storing the block cover access list in the block body. Versus not storing them in the block body, plus defining a retention period. And there is this comment from Lukash, if you scroll down at the agenda. Lukash made a very nice, summary of the pros and cons that each option brings. Maybe, Lukash, if you want, do you want to quickly summarize your thoughts there?

Łukasz Rozmej: Yeah, so this is basically simplicity versus… UX, probably, simplicity of implementation, and of everything. So, if we keep the balls in block bodies, we risk… so, we will be generally increasing the gas limit, right? So, we will be increasing block bodies already quite, quite a lot. And if we… if we store balls there, we probably risk increasing it at least 50% more, Then… than what they would already weigh, and… I'm not sure if I trust this number, 50%, you can, Tony, chime in, later about it. I'm, like, my gut feeling is it will be even more than 50%, but… that's the number you provided, so I'm… I'm not sure if I trust it, to be honest, but

That's the problem. Another thing that you basically, for those blocks, keep kind of archived data in the block history, so… That seems wasteful for me. And… If you have an archive node, you probably still need to re-index this data in a different way, so again, this is quite useful for you, so you blow up doubt even more. Okay, good point, that balls can be then used to quickly, sync archive node later, so that's a point, to just index this data. So that's the pros of kicking balls. So, basically, you get… Kind of the best of the both worlds, so pretty fast, tip. processing. But lower resource footprint.

But it's… if we do that, it introduces a lot of potential issues, because… Things like, do we actually want to sing those balls at all through DevP2P? If yes, then we need to create some kind of sub-protocol for it. If not, we can just ignore it. What if we don't have balls? We need to have a fallback mechanism of processing blocks if we don't have a ball. Which is fine, because we can… we can verify the… ball hash by generating it. While processing, but it makes things more complicated, right? What else? Potentially, it reduces the speed of doing things like catching up to the head of the chain, or…

Darpc, for example, right? So, there are a lot of drawbacks, that we… that this causes. I'm not sure how balls are currently distributed on, on the P2P side, but if they are in the block bodies, they are probably just automatically distributed with blocks. If they weren't in the block bodies, it would have to come as a sidecar. Which could have potential good and bad, well, bad, it's, again, more complicated, but potentially, maybe it's easier to distribute two things that are… Still relatively heavily, both body and ball, rather than just one thing that is heavier, so it might actually, in the end, potentially reduce latency. On the P2P side, so… yeah, there's a lot of things to consider, and in my opinion, it should be, like, a conscious decision. To be honest, I was under the impression that we won't do balls in the block bodies, because that was the…

like, when initially discussed, and I didn't, like, that, that was, like, idea I proposed. And then I didn't, like, pay that much attention, sorry guys, to this, because Mark is leading the charge here on the Nethermind… championing the CAP on the Nevermind front, so I recently read it and thought, okay, that… I'm not sure this is the best decision. It might be, but I think it needs to be a very conscious decision, what we want to do. So, just throwing this out, are we sure we want to do that? Because it's, like, we're increasing the history footprint probably just through increasing the gas limit, let's say, by… 3 to 4 times? 5 times, maybe? And then if we add balls to it, we can increase it instead of 4 or 5 times to more around even 10 times. So… do we really want to do it that big?

That's my question.

Toni Wahrstätter: Yeah, thanks a lot for this summary. I remember, like, this has been a topic in Berlin, and right after Berlin. And as you already said, like, one of the strongest arguments for keeping the blocker access list in the block party was the simplicity. So, I remember, like, this was, by the time, the argument, why we ended up putting the ball into the block body, but I can totally see your arguments, and yeah, would be curious if other clients have an opinion on this already today.

Mark Holt: So, I think from an archive perspective, we're probably gonna just drop the bowels from storage. And the reason for that is we can reconstitute them. Because, effectively, we've already got the access history. We record, you know, we record that anyway, so if we need to reproduce a bowel, it's quite likely we'll just reconstitute it from our… From our access history. Because otherwise, we'll store it twice.

Łukasz Rozmej: Will that be on sync? Will you reconstitute it on Saint?

Mark Holt: No, we'll re… well, so here's the question is, is there going to be an RPC call that's basically getting the bow?

Łukasz Rozmej: Yeah, give me block body. Like, on the DevP2P.

Mark Holt: Yeah. Yeah, but that's DevP2P, not RPC. Is there an RPC call? Are people going to want to access the… block bodies historically, via the RPC.

Łukasz Rozmej: I think there is a… but I think it's a debug feature to get RLP off the block, and BAL should be there, right? But that's, like, not a normal thing to…

Mark Holt: Yeah, yeah, well, I'm just saying, that's… from our perspective, that's what we'd be thinking about, is there's a trade-off between storing them and reconstituting them. Because we'll have the information to reconstitute them. And… if… Because re-execution's actually quite cheap, right? Because we do it all the time for our PCs. So we've got this general trade-off that… If it doubles the disk storage, we'll probably drop them. for example. Because once you're keeping them historically, that's quite… that's quite a big disk budget.

Łukasz Rozmej: Yeah, exactly. Yeah, I think it's not doubling it.

Toni Wahrstätter: I think it looks like it's adding, like, 50% on top, but not 100%.

Łukasz Rozmej: But, yeah, this might not matter in the internet. This is analysis with a current gas limit, right? Ehh.

Toni Wahrstätter: No, this was already worth $45 million, so I remember, like, back then, at the very.

Łukasz Rozmej: At the beginning, we analyzed the.

Toni Wahrstätter: 30 million, and there it goes 50%.

Łukasz Rozmej: Tony, I don't really care. If we are 10xing the gas limit to around 300 million, that's… analysis is, in my opinion, worthless, because the gas usage pattern might completely change.

Toni Wahrstätter: Right, but the ball size should scale sublinearly. So the ball, because it's very well aggregated, it will never scale, super linearly. So if it was 50% extra at 30 million, it cannot become 100% at 300 million. It can only get you, like, 40% on top of it.

Mark Holt: Yeah, so, just to be clear, that's 40% over the top of the cost of the block with all of its transactions in it. Yeah.

Toni Wahrstätter: Yes, and I will actually have to redo this analysis, because I said in the chat, I posted the most recent analysis in the chat. But…

Mark Holt: As you can see there, the bow size is, like.

Toni Wahrstätter: 56… Kilobytes compressed, that's the median. This was not yet at 60 million, so this was still at 45 million. and… I just compared it with this chart that you get when you Google, like, eFroscan block size. And the block size there, which only considered the EL payload, was… is currently at around 120. plus kilobytes? So that's where the 50% are from. But I will definitely double-check that and provide, like, clear numbers so that we know, is it, like, closer to 50%, 100%? But right now, I would assume it's closer to 50%. But yeah, I'm not even sure if this changes something in this discussion here, because in the end, it's still, like, significant data. So yeah, curious, if we should go for the simple approach, and… Just,

Keep them in the block party. And then drop them, and be able to recompute them.

Dragan Rakita: Just to add here, I talked with the red team. Basically, we are supporting from dropping the ball from the block. And for us, it is simpler solution for the ball to not be included in the block, not all the way around. And, yeah, I'm echoing the thing about the database size and complexity on the P2P. So…

Mark Holt: From red point of view, dropping ball from the block makes a lot of sense, and it's something that we would like to see. I mean, it is a trade-off for syncing, though, so… You know. Because if you're… if you're a syncing node. you probably want the bowels on DevP2P.

Dragan Rakita: That's true.

Mark Holt: I'm… yeah. I'm just pointing out that from our perspective, we've effectively got the same data stored. Because we store… Accesses per transaction in our database. So… We just physically won't store the bowel, we'll reproduce it, because we've got an optimized version of it already. which I'm… I'm point… I'm not… I think we're relatively neutral for that reason, is basically what I'm saying. I think, you know, at an implementation level, we probably won't physically store the bowel in our database, but we will be able to reconstitute it if somebody asks for it. And I would expect anybody who's syncing will want the bow, because it'll speed syncing up.

Karim Taam: I see a real benefit to have a block access list also during a sync, because, as I said in the chat, if you don't have to replay a block, you don't… have to, to get all of the EVM4 version, so I think it's real benefit. We will not have bulk from the Genesis, but… It's, it's still a good thing to not have to replay all of the block from the ball fork.

Łukasz Rozmej: Okay, guys, but what kind of syncing are we talking about? Because if we are getting new payloads, we will be getting the balls with them, right? But do we need them on the FP2P?

Mark Holt: Yeah, I think it's if you're doing dev P2P recovery. Is what I'm talking about. If you're a… if you're doing a backward download via DevP2P. It's quite useful to have the bowels.

Łukasz Rozmej: Okay, but for example, we would have to store them in our, like, especially non-archive nodes. We don't have a way of reproducing them as easily, we don't store the archive node as… as Ref and Aragon, so… we cannot guarantee them. If we want to drop them, we drop them, and we can have a retention period. But we cannot guarantee to have them.

Ben Adams: I mean, for… for syncing, so… so what would… what are the benefits? Say we have a retention period of, like, 256 blocks or something. What are the benefits of keeping them? So if you snap sync, there's no benefit to them.

Jared Wasinger: The benefit is that you don't need to heal, you can just download the range of blocks with access lists, and… Once you…

Ben Adams: Dude. If you've got, like, the last 256, you can't snap-sync. Further back than that, because you're too far away from the head.

Jared Wasinger: Yeah, I guess that's why the retention period would need to be… The suggested retention period would need to be reasonably long, right?

Ben Adams: Yeah, but only, like… 256 blocks, or something. It doesn't need to be.

Mark Holt: Yeah, that's not very long. Right.

Ben Adams: Yeah, because you won't be able to get the steak.

Łukasz Rozmej: Well, we should… we should keep them up to finalization barrier, to be honest, right? Maybe.

Ben Adams: Yeah, but that's even… that's even less. That's, like, 64 blocks.

Łukasz Rozmej: Yeah, yeah, but… no, the finalization barrier is, like, can move… Right?

Dragan Rakita: How… how can this be enforced on the P2P? We are talking about the limit of the last 10 blocks while the syncing is happening, and we don't exactly know Like, what kind of number are we talking about?

Ben Adams: Okay, well, hey, say a week. I'm just beyond that. Period. Is there any benefit? I mean, I can see an archive node, yes, but…

Dragan Rakita: We don't need it.

Ben Adams: So they're quite rare, they're quite rare.

Dragan Rakita: From Brett's side, we don't want Bolt to be transmitted over P2P. You're just fine having it in engine, and that's it.

Łukasz Rozmej: So, so there are… there are other things, right? So, if we are definitely, like. Our node hanged, we are restarting it, we are several thousand or 10,000 blocks behind. Then, when we… okay, we won't get that from new payloads, probably. So we need to go through… DevP2P, probably, right? So we are getting the blocks. Now, the blocks are very big. So, if we are single-threadly just replaying them, it might take quite a long time to catch up, right? So that's without balls, right? So that's a problem, potentially. That the catching up process will be way slower. If we don't have a ball, and we need to reprocess something, it is way slower. Another problem is that how the balls will be now distributed. If they are going as a CL sidecar, this is now a CL change.

And we need to involve more CL devs into prototyping that, into speccing that, and it increases complexity on their side. So generally, it increases complexity a lot, if we… if we drop the balls. But… yeah. That's kind of it.

Ben Adams: But we can… we can delete them from… From the block itself. And change the… RPL.

Łukasz Rozmej: Not sure if I understood.

Ben Adams: Well, the… as I understand it, we're not… we're not, generating a hash multiple times. We're not… You've got the bowel hash, which is the hash of the bell, and the header, but then you're not… You're not rehashing the entire block to get a block hash. You know what I mean, it's the sum of all the hash… it's the hash of all the hashes. So, you can just erase it from the lock at some point.

Łukasz Rozmej: Oh, okay, yeah, you could, you could. Okay, so maybe it comes from… okay, so we now have, like, a storage and networking part, but that would kind of…

Dragan Rakita: Doesn't… couldn't ball, like, behave something like sidecar? I'm not exactly sure about CL details there. That shouldn't be acted like sidecars, the blobs are currently… done.

Mark Holt: Yeah, so… I guess the question is, are you talking about dropping them from the EL, or dropping them from the EL and the seal?

Ben Adams: Because the CL has the EL block as…

Mark Holt: Yeah, yeah, yeah, yeah.

Dragan Rakita: CL will still need to have them.

Łukasz Rozmej: I would drop in both, because…

Mark Holt: Yeah, but is it…

Dragan Rakita: Maybe. I'm not sure, I'm not sure… maybe not, my comment's not good.

Mark Holt: Yeah, because I… well, sorry, I'm just thinking this through now from a… Kaplan archive node, it'll have… so we might drop them from the Aragon archive, but end up having to keep them in the, In the… in the CL archive. Which is interesting.

Ben Adams: How… how much do CLs outside of archives normally keep? I mean, do they keep all the box forever?

Mark Holt: I don't know that… yeah, that's a good question.

Ben Adams: Well, do they already have something like Full Force?

Mark Holt: Well, I guess that's a question we should ask. So I… I don't know, because basically. how our use case is archiving, and I think we just keep everything. So… I'm not sure what everybody else does.

Ben Adams: Because I know they do the, checkpoint sync, then. But I don't know, beyond that, what they do with their books.

Łukasz Rozmej: But if CLs keep it, we are definitely more than 10x-ing the storage requirements, and even if we have a retention period of 44s, like, half a year or something, that still will be quite a big amount.

Mark Holt: Yeah, so I think it's an open question. You need to check with the CL devs. I mean, they might not have even thought about it yet.

Łukasz Rozmej: Anyone from Bezu or GeF want to chime in? I'm not sure if anyone here…

Karim Taam: In Bezos' side, I think we need, A minimum amount of block accesses for SnapSync. as I said, I see some benefit to have block access… all of the block accessories, but… It's increasing the size.

Łukasz Rozmej: So, for SnapSync, I don't see this as a problem, because I expect block access lists will be coming through new payloads, right? So you get a new payload, you start snap syncing from that new payload as your pivot for snap syncing, and you can accumulate all those balls, or apply them on the fly while you snap sync. And then you finish SnapSync, you go to the healing, verify you have everything or not, and be done, right? So…

Karim Taam: If you don't have everything, you need to find a way to retrieve what is missing.

Łukasz Rozmej: If you don't have what?

Karim Taam: If you don't have everything. Who you guys?

Łukasz Rozmej: That would be normal healing, right? That would be normal healing.

Karim Taam: Hmm.

Łukasz Rozmej: But you generally should have everything, unless your CL didn't send you the new payloads, right? At some point. Because that's how you will get Baal on the… train tip, and during the snap sync, you are following the train tip and repivoting, so… I don't see this as a problem for SnapSync, right? It can be…

Jared Wasinger: We're back… How far back will the CL give you new payloads? Like, if you… if you stop your EL node and then reconnect to the CL.

Łukasz Rozmej: So, if we stop EL node, we generally restart SnapSync from scratch.

Karim Taam: At least that would never mind us, I don't think…

Jared Wasinger: Yeah, but I mean…

Łukasz Rozmej: Mbezu, okay.

Karim Taam: No.

Jared Wasinger: Yeah, I guess I'm wondering, like, if there's a lapse in the… like, will the… Will the CL node give you new payloads for, like.

Łukasz Rozmej: Yeah, I don't know, but generally, if it doesn't, for some relatively small amount period of time, the healing at the end of SnapSync should just handle it, right?

Karim Taam: Yeah, but…

Łukasz Rozmej: There will be a problem if that's a big period of time.

Karim Taam: For Bezoo, it will change significantly, because, for example, we have a healing of the tree that is general fast, but we have the flat dB healing that can take more times. several hours, so if we have block access lists, it could be really fast. If we are just missing one block access list, we'll have to do a… old healing version, and it will take a lot of time. So it will be a huge difference, just for example, because we are missing one block access list.

Łukasz Rozmej: Hey, but we could have then a retention period in ELs of, let's say, a day of block access list, and that should be… Fine.

Karim Taam: Yeah, if we have this minimum retention, and we have a way to retrieve that, yeah, it should be okay.

Jared Wasinger: So, I guess from Gat's side, we, like, I voiced before that we want… We would be in favor of some retention period that enables a reasonable probability of… of, of… of… successfully syncing and using the BALs to heal without, needing the heal phase, and also, I think, like, if we're gonna scale the gas limit to hundreds and hundreds of millions, it… there's… like, the healing phase could become a bottleneck there, so… keeping… so, using BALs as an avenue to, sort of, address this… scaling bottleneck with the syncing is important, I think. So, I mean, we… we would like them… well, I say we, but it's mostly just me. I think they should be, retained for some period of time, and also transmitted.

over DevP2P.

Karim Taam: I see also, but maybe it will be okay with a genu pod, but for example, if you stop your node and you restart your node after sometimes. Instead of doing, backwarding and executing everything, you can also use block accesses to reach the head. But maybe in this case, we'll have a genu pair outside. It'll be okay.

Jared Wasinger: Yeah, I think… I think that, my… like, what I envision the retention period being, like, sort of tuned for is, like, the… To capture enough, portion of the cases where maybe people, like, have some kind of a network event that prevents them from sinking… I… I… I think there's always gonna be… if we have a retention period at all, there's always gonna be a case where somebody Turns off their node, it powers it back on, and they're kind of… out of luck, because they're just too far behind the chain, and they would need to restart the SnapSync, but…

Karim Taam: And also, yeah, backward sync, it's using P2P, so… Yeah, it would be nice, for example, in order to sync, when you restart your node, having this block access list.

Łukasz Rozmej: One downside of this is actually, I think the complexity increases Quite substantially, and this might be… Not sure if I would say delaying, because we don't have, Time, but require more effort, and potentially, delivering the forks slower.

Jared Wasinger: I mean, are you saying that implementing the… the… The replacement for the… healing phase in addition to the… yeah, okay, so…

Łukasz Rozmej: Yeah, like, just handling all the cases of baclovastasis available, not available, having some retention period, having, the FP2P that might transmit them, might not transmit them. Having CLs potentially having a sidecar for them, right? It's complexity over complexity over complexity, having a way of process blocks with and without them. Potentially might be needed, so… a lot of complexity, right?

Jared Wasinger: Yeah, for sure. I think whatever we decide on, I mean, because we can always… if we could create a… if we could figure out a way to avoid ballooning the complexity of the current EIP further, and then kind of push the… The question of… Yeah, I mean, I think we should go with whatever the simplest option right now that keeps the future open for potentially replacing SnapSync.

Karim Taam: Yeah, I think it's more, just to define if you want to keep block access lists During a specific window, and after each client. will do what they want. If they want to use this block access list to implement new features, new optimization, but it's not really related to the fork, we can have just the block access list consensus to say, okay, we are keeping block access list during a window. a specific window, and after, client can add more optimization later, etc, so we are just… Opening the door to more optimization and features, but… I don't think it will delay, the… the fork.

Łukasz Rozmej: Well, the second thing is how it should be, you know… transmitted, right? Should it be with block bodies, like it is now, or should it be as a sidecar? And that complicates the CL site, right?

Ben Adams: We probably should just follow up with CLs and see what their… Retention on boxes. That was… Rather than having them trying to understand the EL payload. Cause if we… if we expire them in… block. Say, after some… some time, the… if the CL… Just keeping that. Forever. Ben. Yeah, it doesn't completely solve growth.

Toni Wahrstätter: Mmm… Dragon, you have your hand up?

Dragan Rakita: Yeah, just wanted to add, if the ballots are needed for the snap sync, maybe adding the additional endpoint to the snap sync protocol is maybe way forward. It can be like, get me the ball of this block hash. Because what I'm hearing is, this… It's mostly needed for those snap syncing. Or the best use case for the ball that people want to use it is for the snap singing, or recovery from snap singing.

Toni Wahrstätter: Okay. And let me just quickly summarize what we just discussed. So it feels like, the status quo will just stay the same for now. We will keep the block lab access system, the block body, and the main argument is the simplicity it comes with. Because removing it from the block body would mean… would mean, like, all the things that Lucas, just iterated on, like defining retention periods. Having different paths for what to do if the ball is there or not. And then we will also check with, the CEOs what their opinion is on the topic, but of course, we should like, the CL side of this work will already be quite big and quite complex, so…

If there is a way. of course, must be efficient, but if there is a way that we don't need to create new sidecasts for the CL, it will definitely be beneficial for Glamstadom. You effin'?

Ben Adams: Yeah, I just think, one thing to check with CLs is what their retention period on blocks is… is it forever? Or do they have, like, a mechanism to expire? But I do think if we keep the bells, then it's going to cause more pressure to bring up The 4-4's expiry time, so… Have… have blocks expiring much faster.

Łukasz Rozmej: think regardless. It will. This would be a very, pressure on… for Force. If we want to increase the gas limit a lot.

Mark Holt: Yeah, so I guess the question is, given the current 4-4's retention period, what's the… What's the, you know, how much more disk, in practice, do you end up needing? Right, if it's… you're saying it's 40% of the current blocks? It potentially is?

Toni Wahrstätter: Yeah, might be an optimistic guess, so if you want to be pessimistic, I would do with a 50%.

Łukasz Rozmej: Yeah, it would be good to have some kind of calculations with different block gas limits, right? What kind of things you need for, and what's the expected growth for… What, 100 megahs and… 200, 300, right? And just… Showing that. Because… The numbers might be huge.

Mark Holt: That's the simplest…

Toni Wahrstätter: Yeah, go ahead, Mark.

Mark Holt: I was gonna say, the simplest way of dealing with it is just to say, after bowels, there's a shorter retention period. I'm just thinking in terms of managing the EIP, because it does sound like there's a lot of work if you start… if we start dropping things now. Whereas, by keeping things, we're just, like, punting the problem. Basically.

Ben Adams: I assume it would just… just… rather than…

Mark Holt: Expiring.

Ben Adams: thing would be just to say the bells can be optional.

Mark Holt: That's true. Yeah, and in practice, that's optional across… dev P2P, right? Because that's the use case that you'd be using them, is when you're syncing them. When you're syncing, you may or may not get BALs. And then if you don't get a bow, you're just gonna have to do a full. execution.

Karim Taam: Not adding complexity in this case, because we… We need to manage to kind of, sync, depending on If we have block X, this is all not swoopy-tookie.

Mark Holt: That is true, yeah.

Ben Adams: Yeah, but you still need the capability to execute a… execute a block anyway. Unless you're just ignoring processing and flying bells in the… in the main… new form of ZK.

Mark Holt: Yeah. Yeah, then you keep the bell and drop the… drop the… drop the transactions. I mean, to some extent, that's a reason for… for not doing anything at the moment, till it's kind of… you let it bed in, because you… you could theoretically end up with a set of… a class of nodes that just… just want the bowels. They don't care about the transactions.

Karim Taam: But technically, yes, you need to equal translation just to validate the block, but In all of the other cases, you can just use block access list.

Łukasz Rozmej: What are the case… what are the other cases? Well, you can probably assume beyond finalization barrier, you don't need to execute the block.

Karim Taam: Now, before all of the block, Before the… after the finalization, I'm saying. So, yeah, as I said, backward sync, all of the sync, so when you need to reach the head, reach the last finalized block. So… as I said, archive node, or something like that.

Mark Holt: Yeah, so as I was saying, I think archive nodes need to just store the data anyway, but they're a special case.

Łukasz Rozmej: The archive node requirements after increasing gas limit to 300 or something will be…

Mark Holt: Yeah, yeah, that's the problem. Yeah, yeah. So, I mean, there's a bigger problem here, is once you've got a massive, Once you… once we've got more states. I think, you know, it's part of what we're thinking about is what does an archive node look like after this? And it will get to a point where it's impossible, you know, it's virtually impossible to have You need an archive virtualized in order to support the data growth. Hmm.

Łukasz Rozmej: Whatever that means.

Mark Holt: It means you're not storing everything on one machine.

Łukasz Rozmej: Okay, okay, so this…

Mark Holt: you get to that.

Łukasz Rozmej: Okay.

Mark Holt: Yeah, distributed archive, yeah. I think is where… I mean, I just think that's where we're gonna end up, basically.

Łukasz Rozmej: But the latency would be really bad.

Mark Holt: It depends on your… Yeah, so that's the… that's the… that's the design space. It's like, can you… can you… can you end up with an archive node where you can dis… I mean, it's a bit like… it's a version of the stateless client problem. Right? Because you've got… You've got network latency, not just this latency, to take into account. But, yeah.

Toni Wahrstätter: So, do we agree that we keep the balls as is for now? Or… Yeah, Ben?

Ben Adams: Yeah, I think… I think the summary, since there's… there's a couple extra use cases, like, you might want to drop the transactions, keep the valve. Then… Yeah, it might be better to not make the perfect the enemy of the good. keep them in the… in the blocks for now, and then revisit in a… in another EIP. When we see how everything plays out.

Toni Wahrstätter: Yeah, this is my feeling, too. And just, to answer the question from Jagan in the chat, like, what is the as-is case? It's keeping the block of access list in the block body. And not having them as a sidecurve.

Łukasz Rozmej: So, my gut feeling is that we need a better analysis on numbers, and come back with numbers, and then see how many people yell at us that this is unacceptable numbers. In terms of storage growth.

Dragan Rakita: How would that affect P2P? Or database, or painting.

Łukasz Rozmej: Well, everything will bloat, right, by the size of the bone.

Dragan Rakita: That's what I'm saying.

Łukasz Rozmej: That's the current design.

Dragan Rakita: That's not that great.

Łukasz Rozmej: Yes, exactly, that's why I brought it up.

Dragan Rakita: Yeah, yeah, I know. I, I, basically, my… Cleased my mind from the Berlin, interrupt that we did, we decided that the ball was going to be dropped, and I had that mindset from that time. So, yeah, I'm not sure if it's a good decision to… like… To have balls inside the block body.

Łukasz Rozmej: So, in my opinion, we should postpone decision by a few weeks, gather more data, gather more projections, talk with more people, talk with CL people, what do they think, talk with, people that are more focused on… other parts of scaling, too, like, lusanskar here, probably… Not… what does he think about it? Yeah, talk with Ansgard, talk with, maybe some, also, other people that think about storage. Not sure if there are other people that are outside of this… Outside of this meeting that are focused on storage that much. But get also sections.

Dragan Rakita: In general, the feel that I get from this call is that it's this useful for the snap syncing, or the syncing on the tip. But I think it's not worth it, because it will blow up the database.

Mark Holt: 100.

Dragan Rakita: In case it will… it will…

Mark Holt: Blowing out the database is just an implementation detail. Right, because I think, effectively, what we're saying is that the RLP of the bodies that you get across DevP2P will include the bowels. Whether it bloats, I…

Łukasz Rozmej: No, it isn't a implementation detail, because basically you say each node is an archive node, regardless if you want it or not. Because if you need to serve balls through DevP2P, you need to kind of have an archive node for those blocks.

Dragan Rakita: AI sucks.

Łukasz Rozmej: So you, like, for us, if we don't do archive node, this kind of use case just becomes extent. It doesn't work.

Mark Holt: Right, because you've got to re-execute the block to get the…

Łukasz Rozmej: Yeah, and I don't have state, I already pruned the state.

Mark Holt: Heh, right.

Łukasz Rozmej: So, our kind of validator-optimized setup Just doesn't work.

Dragan Rakita: To be honest, I was a little surprised that we store balls in the block, and… That was the reason, what Lucas said.

Łukasz Rozmej: Well, it's… the AIP says that both are part of the block body. Currently. That's the current…

Dragan Rakita: I just thought that that'd need to be updated. Okay.

Mark Holt: If you… yeah, but if you've got a validator-optimized node, surely you can't serve the blocks, either.

Łukasz Rozmej: Well, depends on the…

Mark Holt: Because if you prune the blocks, you can't deliver them.

Łukasz Rozmej: Yeah, it depends on the lever optimization, but, you know, 44s, dictate us how much We need to serve, right?

Mark Holt: Yeah, yeah, so that's what I was saying, I think the question is, in a 4-4s retention period. How much practically more disk space does that add to a minimal node?

Łukasz Rozmej: Well, 50% on the blog bodies.

Ben Adams: Even if the sales are storing. It's…

Łukasz Rozmej: Oh, CLs don't… so you're unstarting it in non-archive node, right? So, it's fine for them on non-archive node. So don't worry about that, that's only an archive note question for them. If they are storing it.

Mark Holt: It's not pissed.

Łukasz Rozmej: Yeah, so I think everyone wants to more or less drop them. But… It complicates stuff, potentially.

Toni Wahrstätter: And dropping them comes necessarily with taking them out of the block body, right?

Łukasz Rozmej: Well… either that or saying block body is optional… has optional balls, right? In certain circumstances, right? Because this block body is kind of… what comes from CL, block body is what we transmit through DevP2P, so…

Toni Wahrstätter: And also, like, dropping them, which means we have to put them in a sidecar, right?

Łukasz Rozmej: Maybe? Yeah, probably.

Ben Adams: Not, not necessarily. We just erase them from the block. You know. After a retention period, go back to the block, load it, remove the bow, save it.

Łukasz Rozmej: But then you have, kind of, two block bodies formats, with balls or with opals. Yeah. Block bodies can live… can live… In both states, right? Which is also might be fine, but… trade-offs, right? Also, question mark, if for CLs it wouldn't be more beneficial to transmit block body and bulk separately for latency. I don't know.

Toni Wahrstätter: Yeah, this was what I was thinking of initially. Like, you… we keep them in the block body, and you can still erase them. Because then we would not need to activate the CO to have a new set car.

Łukasz Rozmej: That's… that's doable, just needs to be specified, right? Right now, I didn't see anything from it in the AP, I think, so…

Toni Wahrstätter: Right, yeah, this doesn't exist yet. So, how would… what would you say… how would we best specify this? So, we would basically say there is a… A given retention period. And after that, you can… you can delete them from…

Łukasz Rozmej: period, blog bot, these could be, like, transmitted through DevP2P without it, right? Something like that. And we need.

Toni Wahrstätter: Is this…

Łukasz Rozmej: potentially mark that there is a null there, so there will be probably a null pointer where Bowles is, right? Just to…

Toni Wahrstätter: Is this something other clients are fun with, too?

Dragan Rakita: I need to track, but seems okay.

Karim Taam: Yeah, it's a good trade-off.

Milos: I have a question, Would… if we would switch, like, if we would make a progress on the 4-4s and do some kind of, like, a rolling window, 4-4s, would that solve all these problems? And we can keep it in a body in that case?

Ben Adams: I would put pressure on making that rolling window very, sure. Shorter.

Milos: I mean, why does it need to be specifically short? Any kind of, like… currently, the 4-4s dictates you have to keep it since merge. So even something that is, like. I don't know, a couple of months, or even a year, would definitely significantly help with, with this, and sure, you can go shorter, but I don't think you need to be super short in any kind of form.

Łukasz Rozmej: I think at the moment we're keeping since the merge.

Ben Adams: Is that right?

Milos: No, I'm thinking, actually, going to some rolling window where we keep only the last, I don't know, since the last, maybe… I don't know, 3 months, or how much is, subjectivity period, or something like that. So, going to, like, last couple of months of worth of Bodies and receipts and balls. And keep from…

Łukasz Rozmej: From my perspective, one, yeah, it could be a workaround, right? You still need to keep it, you still don't really need those balls, really, in practice, in any way, because they can be regenerated from those blocks. And archive nodes still need to either double the data, because they still need to keep it properly, or they will actually still remove it internally and regenerate it, so… which they will probably do. So, it's… like… Trying to ignore the problem where, like, say it's an implementation detail problem, which… It's fine. But… yeah, I'm not sure.

Dragan Rakita: Regenerating would be a problem, in general, because you can DDoS the P2P, or drop, or not respond a lot of P2P requests, because it's not just, like, fetching something from the database, it is execute in that block. If there's few blocks, that would be fine.

Mark Holt: Depends how you… depends how you store the data.

Łukasz Rozmej: Yeah, you could probably fetch it and just compile it from the fetch data, or something.

Dragan Rakita: You're still for the ball, at least from… I'm not sure about… I'm not sure if Aragon stores the read.

Łukasz Rozmej: I… yeah, that's exactly…

Dragan Rakita: Pro.

Łukasz Rozmej: I wanted to say that the Ridley's is the problem, because the right Yeah. So it's actually, question mark, do we have read symbols? Because if we do, then, yeah, the only thing is re-executing, and if we don't, then… Then you can probably compile it from the archive data.

Dragan Rakita: Then you're basically forcing everybody to have, like, archive kind of note.

Łukasz Rozmej: Exactly. Yeah, that's not that great. There was a reason why it's dropping, yeah.

Dragan Rakita: It opens, like, all of these…

Mark Holt: Yeah, so I… yeah, I agree. If you start doing this stuff with bowels, which is why I sort of said at the beginning we're neutral, but other people probably aren't, because If you do start having to do this, it's a big piece of work. That's the problem, basically. Because you have… you end up having to have incremental data… an incremental data store. Which is not an insignificant amount of work. And it feels like dropping them, given the status quo, because presumably everybody at the moment can process Blocks without bowels. Right. So the implementation cost is quite small.

Łukasz Rozmej: Well, the problem is that we are currently processing small blocks without bowels.

Mark Holt: Yeah, yeah.

Łukasz Rozmej: And we'll be processing huge blocks with all balls. So, for example, I'm currently implementing block STM-like processing regardless of balls, and I expect we will be processing them relatively quickly, but yeah, it still complicates things, right?

Mark Holt: Yeah.

Łukasz Rozmej: Can be huge.

Dragan Rakita: It's even bigger than the archive node, because we have additional reads.

Toni Wahrstätter: Yeah, this seems like a topic that we definitely want to discuss, next time, on the next, breakout call in two weeks again. I would suggest we… Don't change the EAP for now, and get more information, especially on the size. But also, if it feels like we will definitely have to go the path, of having the balls as a sidecar. Then, of course, the better, the earlier the better. we change something and involve the CL.

Ben Adams: I'm just…

Toni Wahrstätter: So someone…

Ben Adams: Just to…

Toni Wahrstätter: Sure, go ahead.

Ben Adams: I'm, like, not entirely serious. Oh, but it could be. Reserve one… one blob per block for… One, yeah, one blob per block for, about… we already have a mechanism to do a sidecar.

Toni Wahrstätter: Yeah, I mean, technically, it would work like this, right? You would just create a new sidecar, and then the ball would come in that sidecar, probably need a new Gossip topic?

Ben Adams: Yeah, but it could be… it could actually be a lot… a blob.

Łukasz Rozmej: But then you have to differentiate, etc, probably. Messy.

Milos: Thank you. can be a blob, because if you do, like, a peer does, and clients don't download entire blogs now, you break that.

Toni Wahrstätter: Yeah, it would not be, like, a real blob where you do, like, KCG stuff with it. It would just be a blob of data that is, like. Transmitted separately from the block body. Is there any to-do for the EAP right now, until we resolve that? Should I clarify anything in the EAP already? Like, regarding… Because it feels like there is also some confusion in the chat around what does it actually mean if we keep the ball in the block body. can we just erase it out there? After some retention period, or is that not even possible?

Ben Adams: Yeah, because the… the block hash is the hash of the hash in header, isn't it? So, it's not… you don't… you don't hash all the hashes and then rehash the RLP. So you should be able to erase it. And then, when you re-execute the block. You generate the bell anyway, so that you can verify that… It's correct. And it hashes to the bowel hash. So, you know, there's already… you already regenerated it, so you could delete it.

Toni Wahrstätter: Right, yeah, I get that. Yeah, this seems like, to me, at least for now, like, the… the best solution we have, right? And… If, like, if… and the second best would then be the… the sidecarve. Which, of course, would come with a lot of complexity, because I remember, like, this was a discussion at the beginning. Ball in the block body versus ball in the side curve. And… By the time selders were involved, and many people, also networking people, were all for keeping them in the block body for simplicity.

Ben Adams: Just to follow up on your question, I don't think you should change the EIP at this stage.

Toni Wahrstätter: Okay, yep. Yeah, we're already at the very end of this call anyway, we're already over time, I will definitely put that topic on agenda for the next breakout call in two weeks again. Is there any… Last words that people… Wanna say anything else we should, we should, add to that?

Mark Holt: Yeah, so the only thing is, I'll… between now and the next call, I'm going to look at how much we care about reads. Because I think we should probably include the, do we want reeds in the bow in that overall conversation?

Toni Wahrstätter: Yeah, this was also on the agenda for today, and to make… to summarize it very quick, what we need to benchmark the reads is basically the batch I.O. implementation. So before, we don't know how much we get from reads, we wanted to test it on actual client implementations, and for that, we need two optimizations, the parallel execution and the parallel I.O, because then we can create, like, worst-case blocks and test how much faster are we actually, verifying those worst-case blocks.

Mark Holt: Yeah, so…

Toni Wahrstätter: Having the wreaths and don't having…

Mark Holt: Yeah, I didn't want to have the call. I'm just saying what came out of this conversation for me was these two questions are actually a bit related, certainly from an Aragon implementation point of view, because that's a good point. I don't think we're recording reads. which I hadn't… I'd forgotten about the read stuff, so… I need to do some research, basically.

Toni Wahrstätter: Okay, anything else we need to discuss? Otherwise, I would say we end the call, we're over time anyway.

raxhvl: Yeah, I'll quickly go over the comment that Ben had for using ZAR for… storing nuns. I think so ZAR, essentially.

Toni Wahrstätter: Now, let's do that next, let's clarify that async. Because this was already a discussion point at some point, I would say we clarified that, I think.

raxhvl: Alright, Sean.

Toni Wahrstätter: Because we're already 10 minutes over time.

Łukasz Rozmej: Hurry!

Toni Wahrstätter: Then fi… Thanks, everyone. No, it was… it was a great discussion. Thanks, everyone, and… See you at the next breakout call in two weeks.

Dragan Rakita: Bye.

Stefan Starflinger: Thank you. Bye.

Łukasz Rozmej: Bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.