All Core Devs - Testing #73

Transcript

Danceratopz: Hey, Akash, do I need to be made the host, or does it not really matter, or…

Akash | ECH: N-no, you can… No need to make me a host, just let me know whenever you are ready so I can switch the stream.

Danceratopz: Okay, thanks a lot. And do I need to be the host? Because, as far as I know, I'm… I guess maybe I am automatically from… From the clever setup that we have.

Akash | ECH: I don't know, I think you don't need to be a host.

Danceratopz: Okay. I think it should be okay. Maybe Mario knows. Mario, if you create the issue on Ethereum PM, are you automatically the host?

Mario Vega: I'm not sure, I hadn't thought about that anymore. I don't know.

Danceratopz: Don't worry, I'm clearly overthinking it. And we won't worry about that.

Mario Vega: Do you have an issue, or…

Danceratopz: None yet, Mario, none yet. Thank you.

Spencer Taylor-Brown: Hey guys, how you doing?

Danceratopz: Thanks, good morning. Good morning, everyone. We'll just wait to see if a few more people come in.

Oh, Pari's the host. Hey, Pari.

Parithosh Jayanthi: Yay! I've made you co-host, and you should be able to do everything you want.

Danceratopz: Thanks a lot. Awesome. Alright, so… We seem to have quite… yeah, we got quite a lot of people in here. I think we can get started, Akash, please.

Akash | ECH: Yeah, real life.

Danceratopz: Alright, good morning, everyone. Welcome to ACDT number 73. It's the March 9th of 2026. I will just drop the agenda in the chat for everyone to have a look at. And we've got a lot of DevNets running, or about to launch, so I think that'll take up a lot of time today. Let's get to it. First off, the status of blob DevNet Zero. Would someone like to speak up about that, who has a bit more context?

Parithosh Jayanthi: That should usually be Barnabus, but he's out today. Last I've heard, there hasn't been any particular change, so I would say it's roughly the same as last week. I know that he was looking a bit into bottlenecks with higher blob counts, but I think we have an agenda item later to talk about, engine API and SLZ.

Danceratopz: Yeah, thanks a lot, Perry. That would actually be more or less now, if we want to talk about it now, if someone would like to add a… I know that Barnabas did a bit of benchmarking. If someone else would like to pick that up, there was quite a bit of discussion in Ethereum R&D on this.

Stefan Starflinger: I mean, just to summarize the issue, so there has been tracing, like, OTEL tracing. OpenTelemetry added to the clients, and then Barnabas looked at the bottlenecks, and he saw that Serializing is quite the bottleneck. And since we want to switch to SSC anyways, it would make sense to have that as a way to communicate between the EO and CO. And that's kind of the proposal. And through the capabilities, it's not really a breaking thing, just if the clients support it, then they can, basically use that, otherwise fall back. And that's kind of necessary to scale it further. That's kind of the context of the PR.

Danceratopz: Awesome. Thanks a lot, Stefan. I guess this would be a decision that would be deferred to ACDE or ACDC? Would any client teams like to speak up to this suggestion and strongly oppose it, or welcome it?

Giulio Rebuffo: I will. I think, because… I have been doing some work on this over the weekend. I actually… I just… I… this is not, of course, this is a discussion for ACD, so I'm not gonna take time too much, but I did implement the spec for… Prysm, Lodestar, Lighthouse, Teku, Erigon, Nethermind, and Geth? I also run some benchmark… some benchmarks through Kurtosis. It works. It can do… it can be done very easily in, It can be done very easily. Optionally, so… yeah, I just wanted to give this quick update. I'm gonna present it maybe more on ACD, but it's basically done… I mean, it's not done, but there is already… there are already a lot of… there is already a proof of concept, and probably if you want to test it on blobdevNet0, you can take one of my PRs from one CL and one EL, and then just yellow it there, and you should see if that helps or not.

Danceratopz: Awesome, thanks a lot for that, Julose. I think that's really helpful, to help make a decision.

Parithosh Jayanthi: The proposal was also do it as an optional thing, right? So, worst case, we can roll it out asynchronously, and whichever clients have it implemented, it would be used, otherwise it doesn't.

Giulio Rebuffo: So, the way it works is that when you send a request to anywhere via HTTP or via the web, you can put your content type that you expect in the header. So, if you put the application JSON, what happens is that you're going to be redirected to JSON RPC, This is all execution layer side, right? So if the engine API, if the layer puts that JSON RPC, it will be routed to the JSON RPC. If not, it will be… if it puts Octream, it will be redirected to a REST full API, that uses SSD instead. So that's why it's optional. So the CL can, you know, can test if there is the… if there is…

the CSS support by seeing if it doesn't get an error, if he puts OctetStream, and then he can, you know, he can decide himself where to go by default. You know, maybe at the beginning, of course, since this will require maybe Hive tests to be adapted, you will have it basically disabled, even in CL, and be enablable by a flag. But, yeah, I think it's, you know, but… But it is optional. It's very easy to make it optional, and it's already optional as I already implemented them.

Danceratopz: Alright. Thank you. I think we'll leave that there, perhaps, to be taken up for further discussion, another call. The only other topic we had there was… there was some, issues on E69, but I think it was just some mainnet peers sneaking in. I don't think… I don't know if anyone wants to say anything to that. I think the problem's solved. Would there be any other topics for Blob DevNet Zero that anyone wants to talk about today? Okay. Thanks a lot, Gilo and Parry, for the context. Moving on to Glamsterdam. Bavnet, sorry, BAL DevNet2 is still, running, and I think Stefan just posted, in the issue below. That you'd like to, hi.

Stefan Starflinger: I can go over… I can go over it, if you would like. So basically, I got around to adding some more nodes. These nodes won't have any validators attached to them, they're just different configurations with the prefetch enabled, sequential, or with all of the valid optimizations enabled, I've now rolled that out on DevNet2 just get and Bezo, so we can capture some benchmarks. I will update also the Grafana dashboard with these nodes. They're currently still syncing, and that will give us some insights. I'm not sure how long I will be running them, but we will have a little bit of data for how good the optimizations are, and I think DevNet2 also has a decent state by now. I mean, it's not mainnet, but it's something since we've been spamming continuously on the network.

Since it started. But otherwise, I think, regarding nodes, it's going pretty smoothly so far. Just one issue with the Bezu tracing that is already being looked at. And Erigon, hasn't, given an update regarding, their progress on DevNet 2. I think they switched over, focusing on DevNet 3. about Definite 3. But I think that's all from… a definite tool, and I think we can… If it's okay if anyone else has something to say for Definite 2, otherwise we can move on to Definite 3. Okay, then, for Definite 3, we've been doing a lot of, local testing and kurtosis, trying to, find any inconsistencies with the spec or, with the different implementations.

So it's still an ongoing, process, especially for 837. These gas changes also kind of make it difficult, for… the ball, because quickly there are consensus issues with the blocker access list and gas accounting, so that makes it a little bit harder, but it would be good if we could get an update from the clients regarding the progress. on, about definite 3. Maybe we can start with, Geth?

Danceratopz: I see your PR got merged, Stefan. Does anyone from the call want to… from Geth want to speak up to that? Looks like Marius merged the fix. Okay, if not, shall we move on to Besu?

Daniel Lehrner: I… I can give an update. So with the latest spec test, 5.3… We have, 7 filling tests. Two of them are false positives, where BESU recognizes the block as invalid, which is expected. We just returned the wrong error message, so I need to… to make a PR for the execution specs to fix that. The other 5 were 8037 differences between the 5.2 test and the 5.3. I have been fixing them this morning. I think it should work now, so… If everything is correct, I think we should pass all the tests, but I need to… I need to double-check. As I said, except maybe for some false positives, but for the consensus-specific stuff. It should be… it should be correct.

Danceratopz: Awesome. Thank you, Daniel. Just for context, Ben detected some gaps in AT37 coverage, created an issue, and added some tests himself, and Spencer added a few more tests, for AT37, which resulted in another execution specs release, 5.3, which was released, like, late on Friday. And just a heads up that that release, includes all the tests filled. Well, it includes Osaka, but it actually includes, all forks. So not just Amsterdam. So yeah, then, if you see any, issues with the tests, please ping us in the Discord and let us know what needs to be changed. Then… Stefan, you also found an issue in Nimbus that…

PR also seems to be merged.

Stefan Starflinger: Yes, we're making some… some small progress, but I've also been testing with all the clients with a basic payload of creating a large contract and EVM fuzzing, and those networks don't seem very healthy yet, so there are still a lot of things to uncover. That, I'm also working on, and would be great if, client teams Can, run kurtosis locally to see If they can find some issues as well, still. I updated the Kurtosis configuration in the spec sheet. To the latest one, and also updated the Ethereum Genesis generator to include the Nethermind, Timestamps. as well. But I think, is there anyone from Reth, that could give an update as well? That would be great.

Emma: We've got Dragon working on 8.37, I think most of the changes are there, it's just, like, testing and some fixes, but otherwise, I think, on the Reth side, we're passing all the tests, so… I think we're looking good.

Stefan Starflinger: Okay, another mind, a quick update, that'll be great.

Ben Adams: Yeah, we have all the features implemented, we're planning some tests at the moment. And the BALS optimizations we're thinking, will come a bit later.

Stefan Starflinger: Okay, great. I think then we're making pretty good progress. It's mostly about ironing out all the… small issues with gas accounting that basically straight away result in consensus issues because of buzz. Is there anything we missed on DevNet3?

Danceratopz: Do you have a feeling, Stefan… sorry, do we… do you have a feeling, Stefan, for when we could potentially launch? Is there any, date? Targeted for that.

Stefan Starflinger: I think Marius just raised his hand, but regarding the deadline, I'll try to get it for Wednesday, again. Let's get it working well. Marius?

Marius van der Wijden: Yeah, so, we just sat in a call for, like. an hour, and I think they said for two more hours before that. Looking over the 1837 changes, it's… like, the stuff that we have is partly implemented by me, which are the good parts, of course, and half of it is implemented by AI, and it's just so garbage that we basically have to, yeah, sit down and redo a lot of these. I suspect we will not be able to finish, or I will not be able to finish that by 2. by, by Wednesday. The… yeah, the version we have right now, it has a… still not passing a bunch of test cases, like a thousand, but, yeah, might be able to participate on the network, except for, yeah, if there are any transactions on it, basically. No, if there are, if there are,

complex, interactions. So, yeah, I've tried to do my best to, to get there, but, yeah, it's not super easy. And, yeah, the whole… logic of 1837 is also… It's… yeah, it's… it's just not super nice, and it… Every time I think more about it, I… I want to change stuff, but then I think more about it, and I don't want to change them anymore, and so it's… yeah. It's not an easy process.

Danceratopz: Okay, thank you.

Stefan Starflinger: So, yeah, it's hard to say what the deadline will be. I think we're… we'll make… some more progress until Wednesday, and I think then, we can see, and maybe… Have some more specifics.

Danceratopz: none?

Ben Adams: Yeah, just to follow up on that, the implementation for us has been, like, an extremely deep change that, like, has tendrils that affect everything. And so we don't like it from that respect. But then, from the… from another aspect. Don't really think of anything. That's fair. So, yeah. is what it is, I suppose.

Danceratopz: Okay, alright, thanks. for the additional context, Marius and Ben, I think Wednesday does seem a bit ambitious. As always, I think we can lean on Highview and the test results to see how clients are doing, this week. And obviously, Stefan's amazing ketosis testing as well. To see if it reveals some more gaps in coverage. Okay, Is there anything else anyone would like to discuss for BAL DevNet 3? Or we… can we wrap that up now?

Toni Wahrstätter: Maybe, maybe just a quick question, regarding the PAL optimizations. So Geth and Basil have been ready already with the parallel I.O. I wanted to quickly get a feeling how far other clients are with regards to the parallel I.O. optimization. I think I saw in the chat that, was it Ben or Lukash? Someone mentioned that Nethermind is… It's getting there. Very soon, so I'm wondering, is that already, like, Definite Free ready? And also hearing from other clients, like Erigon and Rap, how far they are with regards to the hardware.

Łukasz Rozmej: So, parallel transaction processing is almost ready, and we are experimenting with parallel I.O, but our current experimentations are not really netting any major benefit, and almost any benefit, to be honest, so… But we are not sure, maybe we still will. Managed to get something.

Ben Adams: Yeah, we're more focused on correctness than optimization at this stage, I think.

Toni Wahrstätter: Makes sense, yeah, thanks a lot for the update. Is there also someone from REF that could quickly… I guess Dragon is not on the call, but is there someone else who has insights into parallel I.O. on REF?

Emma: We've got the batch I.O. work in a PR. Otherwise… .

Łukasz Rozmej: Does it give you any… mmm… benefit?

Emma: Oh, no, we're just doing pure implementation at this point, no optimizations.

Łukasz Rozmej: But I mean, have you benchmarked it? Does it give you any benefit? No, we haven't been to it. Okay.

Emma: But I can do that today and report back.

Toni Wahrstätter: Awesome, thanks a lot for the update. Yeah, and we have some tests prepared, some in the… it's in the… In the testing repo. There are some tests specifically written for benchmarking. That could be used, I can, I can post them into the Discord, so we could. Yup. Catch up async.

Emma: Coop.

Danceratopz: Awesome. Thanks a lot, Tony and everyone, for chiming in there. I'm thinking that's it for about DevNet 3? moving to the CL, Would someone like to give an update on the ePPS DevNet?

Parithosh Jayanthi: Yeah, so EPBS DevNet Zero launched last week, and we had a couple of early issues. Portos has a write-up on his blog as to what the issues were. I've linked it over here. But besides that, we aren't particularly testing or stressing the network, it's just making sure that the nodes are up-to-date and rolling out patches from clients. I guess more serious testing would kind of happen on EPBS DevNet 1.

Danceratopz: Thanks, Perry. I think POTUS wanted to talk about forcing the EL to trigger block production on older heads. And the issue of PTC lookout. POTUS, would you like to speak up on those topics?

Potuz: Yeah, so one is sort of, like, mandatory, and the other one, we need to make a decision what… how to deal with. So the first one is on the EL side. Today, there's, on the engine API, There is a May that I think needs to become a must. Which is the fact that when we send a four-choice updated message to the EL for a previous head. then the EL just ignores it and gives us a payload… an empty payload ID. But on EPBS, the problem is that we actually might have an honest reorg on the CL That, does a four-choice update to a previous head on the current branch. So I… currently, because of the issues that, well, we have, like, partial fork choice in the DevNet, this is actually being triggered on Prysm essentially every few blocks.

And yes, so, well, that's the comment. I need input from the EL. Marius already replied something, but then that sort of died on Discord. I need to know from the EL if this is hard, if this is trivial, if we can have it soon, so that we can actually start testing, these kind of things.

Łukasz Rozmej: And alleles work like that?

Potuz: Well… All the else, as far as I know, give us an empty payload ID if I give you an FCU to a previous head. At least every time that we test, the EL gives us an empty payload ID, they ignore an FCU to a previous head. And they don't trigger blood production. When I do an FCU for an old head, And that breaks CPBS.

Marius van der Wijden: Yeah, the program.

Alex Stokes: thing, or… Like, do you think it'll get better with multiple choice progress?

Potuz: No, no, no, I mean, this has to be the case. I mean, a reorg… I might reorg to empty, like… there's an honest behavior, which is voting for the empty block, and in that case, the CL needs to reorg. Right. A current head, and it needs to be able to produce a blog on top of a previous head.

Alex Stokes: I bet.

Potuz: The thing that is triggering this… I mean, this shouldn't happen on production, on mainnet, regularly, right? So we want to continue building on full, but the problem is that Lighthouse and Lodestar are voting for empty everywhere. And they, they outweight us, and so we take this as a signal that we need to reorder the payload, and we try to build on empty, but then when we try to build on empty, the EL just grabs an app on us, and we get reordered.

Danceratopz: Marius, did you want to add something?

Marius van der Wijden: Yes, so… The problem is that we, the spec does not allow FCU To, a… a parent of… little block, because that basically means we are reorging back the chain. The payload production is a different service. To allow… Dude…

Potuz: Yeah, it's… I can't hear Marius.

Łukasz Rozmej: Yeah, we cannot hear anything.

Marius van der Wijden: So… No? Goddammit.

Łukasz Rozmej: Every Friday.

Marius van der Wijden: Good Lord.

Łukasz Rozmej: Maybe.

Marius van der Wijden: Okay, different…

Łukasz Rozmej: Might be the internet.

Danceratopz: Okay, we'll see if Marius can resolve his connection issues. Are there any other EL devs that would like to speak up to this in the meantime?

Łukasz Rozmej: Yeah, I'm taking a look, and I explicitly see this in our codebase, that yes, if we have, Fork choice to a… payload that is on the same chain as the current head, we ignore it, so… not sure what should we do, and how's the specification. Phew.

Marius van der Wijden: Yes, this was part of the spec. I hope you can hear me better now. I'm in a different Wi-Fi. Hopefully, I will get better internet soon. So, the problem is that we are forbidding these reorging the reorging the same chain. What we could allow is, if you specify older, if you specify the new FOC choice. but older payload parameters that we are building, these older blocks for you. We could do that, or we could allow for reorging the chain. But I think there was an issue with reorging the chain, why we specified it this way, but I'm not sure.

Łukasz Rozmej: I will add interesting… context in our… HiCore plugin, we actually have this feature disabled, so you can reorg to all the train. Which is interesting. But it's not in the mainnet implementation.

Danceratopz: Okay, it seems like this topic will need to be followed up on.

Potuz: I just want to stress something that… I mean, it's not a matter of something that might be optional. There are reorgs on the EL that will happen where we roll back the chain. And that… that will be the head. Even if no block is produced. We are going to call the yel, saying, you just need to roll back your head.

Marius van der Wijden: Yeah, this is so horrible, UX. I… I… I know that this… the way it's going to work now, but this is so horrible to UX, because we are telling the users that, hey, this is the new head, oh no, actually, the previous head is the new head. Or actually, no, we have now another sidechain that is the new head.

Potuz: But isn't that the way that it works now?

Marius van der Wijden: Yes, but it never happens.

Potuz: Is this behavior when you reorg a block?

Marius van der Wijden: Yes, but reorgs never happened. And it seems that with.

Potuz: Yeah, reorgs will never happen on mainnet either, but we need to deal with them.

Marius van der Wijden: Okay, okay, then, then it's fine. Like, we can just prepare a branch that removes this check. For… for Bell DevNet 1, or 0, or wherever you guys are at. I think that is… that is not a problem. Like, I can do that right now.

Danceratopz: Okay, are there any other young, client devs who would like to speak up to that, or… Does this discussion need to be taken in more detail elsewhere? BOTUS, are you happy with the comments you've received?

Potuz: Yeah, definitely. If we just have an implementation that we can test, that'll allow us to test for choice. I mean, we don't expect to see many of these kind of reorgs, but we need to deal with them, and we need to test our for choice, so we need to have a branch that does this.

Danceratopz: Okay.

Ben Adams: I do it. Where are you?

Potuz: no rush at all. We are not going to test for choice until two DevNets from now, so there's no problem at all not having this behavior now. We're not thinking on testing these things, even on DevNet 1.

Ben Adams: Is it… is it purely a testing thing, or would it need to… We expect change for me.

Potuz: It happens that we reorder blocks on mainnet. It will happen that we will reorder payloads on mainnet, which is done… We don't want to see it regularly, as they just don't happen today regularly. But it does happen that we have reorgs on mainnet.

Danceratopz: All right, POTUS, when's the next EPBS breakout?

Potuz: We don't have breakouts since we have this meeting.

Danceratopz: Okay, good. But I think the message was pretty clear. Thank you. Did you want to talk about the PTC look-ahead as well?

Potuz: Yeah, this is an issue that Nico found, which… which is, yeah, it's a flaw in our spec. We… We need to… PTC attestations are sent during one slot. And then they are included in the next slot's block, and they are validated when we process those… that block. The problem is that, the PTC committee is computed with the balances of the given state. And that means that if you attested being on the PTC on slot 31, And your attestation got included in slot 32. you are validating the block that included that PTC attestation, and you're looking at the effective balances Of the current slot, which is 32, it's in a different epoch. And those are… that may have changed the PTC, the full committee. So an attestation that was included in the blog, because it was violent then, might just become invalid now.

And to deal with this, we have a few options. I opened the most naive of the possible PRs, which is including a sort of like a look behind. We just cache the full PTC committee for the current epoch, well, the previous epoch and the current epoch on the beacon state. It is mostly useless, because this only happens at that particular slot. It's just one slot that is needed. But… Clients, anyway, seem to cache the full epoch to have duties, so it seems… on a first run, it seems that clients don't mind this. But I don't know. I wanted to have a small discussion, at least on what are the options, or what people prefer. I really have no strong opinion on the topic.

Perhaps Nico is here, who found the topic, if you can just say something.

Nico Flaig: Yeah, so I'm not sure what's the cleanest spec design here, but as for Loadstar, we currently cache the current and previous epoch, so for us, it wouldn't be extra memory usage. Also, I think it's the equivalent of I think 64 validators or something, so it's pretty small memory-wise.

Potuz: Yeah, so the current PR, I think it's 256 kilobytes per state that is added.

Alex Stokes: Does the spec right now say… Like, I'm sure, like, the spec is correct, right? In terms of who you validate when and where?

Potuz: No, no, the spec is incorrect, actually. The spec, the specs… I mean, it's clear on whom you need to validate, but the problem is that, some.

Alex Stokes: It's using it wrong.

Potuz: It's valid in the previous epoch is not valid in the current epoch.

Alex Stokes: Right. Okay.

Potuz: This is a bugger.

Alex Stokes: So it's… it's using the wrong state at that point in time.

Potuz: Yep.

Alex Stokes: Okay. then yeah, we should fix the spec. But I think what you suggested makes sense. Yeah, like, I don't think we need to be more invasive, have some cash explicit in the state. Clients can just cache it locally. But I should just be clear, yeah, who you want to validate against.

Potuz: The cache is actually quite trivial. We are… it turns out that Prysm is not caching the full epoch. Prysm is only caching the committee and computing slot by slot, but moving to the full epoch is trivial for us, so it's fine.

Alex Stokes: Do you have a link to the PR?

Potuz: Oh, yeah, let me find it.

Alex Stokes: Okay.

Justin Traglia: To be clear, this solution that you guys just talked about, would that mean not adding anything to the state? Like, caching it in the clients outside of the state.

Potuz: No, no, no, this needs to be added somewhere. It needs to be clear what is the state that you're validating with, so… the PR caches it in the state.

Justin Traglia: I see.

Alex Stokes: Yeah, big 10.

Justin Traglia: 256 kilobytes is kind of a lot to be added to the state each. Is it a slaughter epoch?

Alex Stokes: I think it's each epoch, but yeah, why can't it just say, you know, use the state? In this case, I have the last slide. Like, you just should just use the state that corresponds to the validators you're.

Potuz: Well, the state transition function is a function that takes one state, one block, and recovers a new state. If you… change the signature, that's a much deeper change. You're gonna be saying now, take two states.

Alex Stokes: Yeah, but we're seeing…

Potuz: Give me a new stake.

Alex Stokes: Hmm… Yeah, I don't know… Is there nowhere else we don't do this?

Potuz: Yeah, there's no… no, no, this is… there's no way we're doing this, and… and even… we tried, like, milder changes before, of having one state and two blocks, like, to use the previous block as part of a state transition function, and… Danny almost kills us, and forced us to, like, cache… whenever we are in this kind of situation, we add an extra cache to the state. That's why we have headers, and we have a bunch of stuff in the states that we could just easily take from the previous block.

Justin Traglia: I think I'd prefer a solution that just adds the least amount of data to the state, whatever that is.

Potuz: So there is a solution that only adds, So, two committees? That is… that will be, a thousand, indices, so it's 8 kilobytes, I think. And that works. But then that forces clients to be, like, updating this So, to keeping the cache in memory, and then updating this state every slot. I can open that PR if you want. It's easy to open. I don't know what CL clients would think it's… I mean, there's a trade-off of complexity versus, like, having the space in the state or not.

Justin Traglia: Yeah, I would hold off on adding a PR for that. I think we should just continue discussing this more. There you go.

Potuz: This one is a little bit more urgent than the previous one, because we do want to test PTC attestations, and we suspect that if we send PTC attestations now on DevNet0, we're gonna split. So I think that's the last thing we're gonna do. When we tested all of the existing structure. we are going to say, well, okay, so now we're fine by splitting, let's just send PTC attestations and see if we actually split, or everyone implemented the same kind of sort of cache.

Justin Traglia: Got it, yeah, let's make a decision on this, this week, and I can also… make a SPAC release with this change as well.

Danceratopz: Alright. Thanks a lot, Justin. I think we can continue that conversation async. Just in case you missed it, POTUS, Mario's already pushed a fix to the, GEF branch, the fix for reorgan to parents in the child wealth. Is there anything else that we'd like to discuss on the ePBS DevNet before moving on?

Potuz: Can we quickly, like, decide parties here of what sort of, like, things we can… We can send, or… or it's fine if we just, like, comment on the chat. We want to, like, stress now EPBS.net zero. We can probably kill it by… but we can… we should, like, test I don't know, withdrawals and these kind of things. So for this, we just need to make sure that all… that we have clients that can handle, or that Think that they can handle, and what structures they can handle.

Parithosh Jayanthi: Can we make a thread on EPBS, and then we can have a list of things, and we'll take care of it.

Potuz: I'm sorry, Barry, I didn't hear you.

Parithosh Jayanthi: Can we make a list on the ETH R&D EPBS channel? And then, once we have a list, we can just go through it one by one? I'd rather first test it out locally before just breaking the live network.

Justin Traglia: Maybe it'd be nice to get some verbal, Discussion on this really quickly, though? Like, what can clients not do? Is there anything? work. I don't know. I kind of agree with Botus that it might be nice to talk about.

Parithosh Jayanthi: Yeah, maybe withdrawals is the easiest first one. Is there any client that can handle withdrawals on EPPS DevNet Zero right now?

Potuz: Well… Withdrawals will happen once we have, Yeah, I don't know. We need to… so withdrawal request, you mean, right?

Parithosh Jayanthi: Yep.

Potuz: I think we should be able to ha- to handle them. Not sure. We can try.

Parithosh Jayanthi: Okay, we can create a local test and then share the outcome. I would assume it's the same for deposits, blobs, transaction load, everything, so we can just try one test locally with everything and see what breaks?

Potuz: Right, so I'd like to try, Current structures that exist, whether or not we can continue to handle them, handling them. Especially if there is a missed slot, or a reorg of a payload. So, now the depth net is perfect, because there aren't that many missed lots. So… So it's good.

Parithosh Jayanthi: Yeah. Yeah, what we can probably do is Asserta can turn on and off block production, and then we can trigger missed slots via that, and then make sure that the network's still fine. Yeah, I'll set up a test for it.

Justin Traglia: When you say builder deposits, or sorry, when you say deposits, does that include builder deposits?

Potuz: No, no, no, just the usual structures that exist now, and then once we know that we can handle those, like normal deposits, then we can just start dealing with builder deposits. I think that should be easy. And after that, we can just kill everything with BTC attestations, or trying to reorder just payloads. Like, we can start, like, withholding payloads, and these kind of things, and see if people, like, reorder the full block, or just build on empty. And then I'm sure we're gonna kill the devnet. I'm sure we're gonna be split then, but that's fine.

Justin Traglia: Okay, we'll continue discussing this on the ETHRNG Discord. Dan, are you there?

Danceratopz: Yeah, sorry, everyone. I had a bit of a sound problem for a minute, but I think I'm back now. I could hear you, Justin. Sorry about that. I lost the last minute of whatever was said or not said. So, I have one short… in my agenda, I mentioned that Teco and Nimbus are not running on the DevNet. Is that… is that not true? Are all clients running on the DevNets at the moment?

Potuz: Techo and Nimbus are not running.

Danceratopz: Okay.

Potuz: But, Stefan, I don't know if Stefan is here, but Stefan said that he might be able to hack a branch to sync.

Stefan Bratanov: So if we, if we start…

Potuz: from Genesis, I think, both work, but syncing is a problem, or checkpoint syncing is a problem.

Stefan Bratanov: Yeah, I'm working on it at the moment, but haven't done it yet.

Danceratopz: Okay. I was just wondering whether, it's time to up the ante on testing on the DevNet when some client teams haven't joined yet, but… I'll let you, decide that based on local testing, if that's how you're gonna move forward. Are there any other topics for EPS DevNet Zero before we move on? Alright. Then, Tony brought up on the agenda he wanted to talk about a networking protocol change for Snap 2. Tony, would you like to speak to that?

Toni Wahrstätter: Yes, I will be quick with that one. Gary and me have, began inspecting SNAP2, which is, SNAP protocol using block access list. It's an early draft, so I would love to have more eyes on it. Basically, instead of, using getTryNotes, it will use the block level access list to… do the healing phase in SnapSync. That's kind of the TLDR. And, yeah. More to become. But would be great to get some feedback on it.

Danceratopz: Sorry, yeah, thanks a lot, Tony. Tony just dropped the link in the chat if anyone wants to provide feedback. Please take a look. And I think Joachim's question moves on to the next agenda point, so if, just quickly. If anyone would like to mention something now about SNAP2, please speak up. Otherwise, we'll move on to EV70, EV71. Okay. So, Tony had another agenda point for ETH71. To clarify the response size cap, would you like to say something to that?

Toni Wahrstätter: Yeah, this was, this was a mistake I made, so I was just, thinking of the 10MB limit, which we had in EVE70 with the receipts, but, thanks to Besu, who pointed out on EVE Magicians, that GEF is using, for example, GEF is using a 2MB softcap. And EVE71 for block cloud access lists now also have… has the same soft cap of 2MB. So this was a… Very recent change I just did today, just as a heads up.

Danceratopz: Awesome. Thanks a lot for the information, Tony. Is there anything to discuss for EV70? Does anyone have context on this? Is this blocking anyone? Okay. If no one… Has anything to say there? then I would move on to… Benchmarking and state bloat, in particular performance DevNet 3. Would someone like to give an update on the Perf DevNet 3?

Louis Tsai: I can give a quick update. So, for the blotting process, we already completed blotting for EOA account and ERC20. We have a smaller size ERC20 that targeted, necessary benchmark. And, we have an updated version of a related benchmark, including account access benchmark and Storage Operation Related Benchmark. And we're now moving forward to generate the payload using guest benchmark tools from Nita and my team. And collecting the data using Benchmarker. On the agenda, I see there's some questions regarding the snapshot management. I'm not entirely sure about the details. I see Johong on the call, maybe Johong could chime in for more details?

Jochem Brouwer: Yeah, so on the… can everyone hear? That's true. Yes? Oh, great, great. So, on the DevNet, well, to take a snapshot, you of course have to have a synced client. As of now, we have, the clients Rev, Nethermind, Besu, and Kev. Here we go as a problem, but the plan is, if it is synced, then we will take another snapshot, and we will use that snapshot for the snapshot item.

Danceratopz: Alright, thank you for the updates. Lewis, did you want to add any other… did you have any other questions for client teams regarding…

Louis Tsai: Adding note from my side. Thank you.

Danceratopz: Okay. Jocham?

Jochem Brouwer: Yes, one more point. So we are currently targeting the benchmarks for the repricing. But, I will also ask, to start learning the block-level access list benchmarks. We have some… benchmarks for that already. I think they are not merged yet, but I will focus on that this week to get these merged, and to also add additional benchmarks to also target the block level access lists. I think we will target the Perth DevNet, because then we can also focus on the state. To see what the impact is on the block access list there. So, also building clients with the block access list optimizations and without them to see the, Well, what they optimize and what the impact of these is.

Yep, thank you.

Danceratopz: Awesome, thank you, Joachim. Is our all client synced to the DevNet now? Or is it…

Jochem Brouwer: No. Is there still some points missing? Here we go, it's not fixed, but I see on Discord that there is a fix ready. Except that it needs a resync, so this is going to take some time, but this is not a blocker for the benchmarks, we will bring them on the synced clients, and we will try to get everyone synced, and to also add those to the benchmarks. And one of the reasons, just to add this, is that the, well, not the reason, but one of the causes is that only DevNet, we really, well, we raise, like, the gas limit by a lot, and this stresses the clients, of course, and, well, this might run into unforeseen issues.

Like, I'm not going to speculate, but one of these things are, like, out-of-memory issues where, well, there's, like, a lot of state changes happening, and these state changes are all loaded into memory, and this causes an out-of-memory. And this, of course, only a thing which happens if you, like, raise the guesstimate by a lot, and then start to spam this network with a lot of these bugs. But it's, I think, still nice to see… to find these kind of bugs and to fix these now. So we are basically ready for, like, a 5-giga-gas network based on these kind of stats, yeah. But I'm not saying we should move to 5G, I guess, on mainnet, of course, yeah.

Danceratopz: Awesome, thanks a lot for the clarification, Joachim. All right, any other topics for Perf DevNet 3, or for… Other benchmarking topics in total? There's, of course, an informal gaslighting committee meeting tomorrow, if client teams wants to join. Just drop a line to Lewis. Or Maria, or Jocham? And us to join. Otherwise? Any other topics on benchmarking or stay below? Okay. So, we're very close to the end of agenda. The only other topic that I would, that I added to the agenda was basically there was some discussion about the ACDT cadence, and whether we should reduce this. I think…

I would just say, given the amount of discussion we have on this call at the moment, I think it's… it's merited to have it weekly. But of course, it's… I understand that it's a diverse scope of topics, and that not all topics are directly relevant to a lot of people on the call. I added the link to the thread in the ETH R&D Discord server thread. To the agenda, please chime in there, but if anyone wants to add quickly their two cents before we finish up for today. Please feel free to speak up.

Parithosh Jayanthi: Yeah, maybe one thing I was thinking of was taking a call about next week's ACDT by roughly, like, Wednesday or Thursday. If it feels like there's mostly just topics to discuss on one specific, feature, for example, if it's EPBS or whether it's block-level access list, then we could just defer the call to an EPBS breakout, so that the teams can go deeper into those, those niches. Yeah, portal.

Potuz: I'd suggest that we have a call that doesn't have a topic, and whose topic is chosen every week. So, for example, now we could say, benchmarking this or that feature on block-level access list requires a discussion, let's just take that spot on Wednesday. So that that call actually is a technical call, and only is attended by people that know that they want to be there. No, Nick, so it's not really ACDT, because ACDT now has I don't know, 20, 30, 40 people talking about completely separate topics, and none of them actually gets to be discussed in detail. Well, at least none of the ones that I know about.

Danceratopz: So, this suggestion would be similar to how ACD got split into the EL and the CL calls. We could do the same for ACDT and rotate.

Potuz: No, because that… that keeps them… that keeps them every week, already forced, everyone from ACDC to go there. What I say is. we have a topic, we decide now, today, or an ACD, or whatever, or even an async, we have this topic, let's cover it in that call. And let's just have a focused call that is a very technical call.

Danceratopz: So just reserve the spot for breakouts, and define the breakout subject based on demand. Okay, Tony, did you want to add something?

Toni Wahrstätter: Yes, we also wanted to stop doing the Brock Cloud Access List breakout calls, and move them to ACDT. So as of… as of today, we are still on Wednesday. Every two weeks is the Block Club Accessless breakout call, and every other week there is the repricing breakout call. And we could also use that spot, so I think that's basically what Podos said. For example, with all the benchmarking that is still needed and might still need a breakout call, we could move it to do it weekly, or have that new spot that the block of access list breakout call had to do other, breakout sessions.

Danceratopz: Okay. All right, thanks a lot. Yep, sorry, did someone want to speak up?

Parithosh Jayanthi: Yeah, I think part of the issue might just be that it's a seasonal thing. Maybe right now, the better option is to seed ACD in terms of, sorry, ACDT in terms of more breakout sessions, because it seems like there's more interest in diving deeper into topics rather than making quicker decisions. Maybe in two months, that's not the case anymore, and we just need to meet to make quicker decisions. So yeah, we… Mario and I can maybe take a call on it on a weekly basis. If it feels like there's a lot of piled up decisions across different streams, we can say that we do ACDT, but on weeks that there aren't, then we can, cede to…

switch to a breakout format. Does that sound like a fair outcome?

Danceratopz: That sounds fine. Thank you, Perry. I think we can leave it there. Continue a discussion async, perhaps in that thread. Thanks, Justin, also for offering to help, that's awesome. We're at the hour, so thanks a lot for everyone for coming. And for your contributions, been a really good call. Thanks a lot, see you next time.

Potuz: Bye-bye.

Louis Tsai: Thank you.

Łukasz Rozmej: Bye, thank you.

Toni Wahrstätter: Thank you, bye-bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.