All Core Devs - Testing #99

Transcript

Mario Vega: Hello, everyone. Let's give just a couple more minutes until more people join before we can start. Thank you.

Okay, let's get started. I'm gonna share the agenda in the chat just now. Welcome, everybody, to ACDT number 99, today's October 5th, 2026. We have a small agenda today, consisting only of devnet updates and mainly the Sepolia upgrade thing. We'll do the EL and CL breakouts, after that is finished. So yeah, let's just get started. On the first… Topic of the agenda today, regarding devnet updates, do we have anyone from PandaOps that could chime in and share the latest updates of this?

Stefan Starflinger: Yeah, sure. I think, The Shadow Fork updates would be most relevant now, that we ran last week. Where we shadow fork Sepolia, and we did two rounds, and in the first round, we found some issues Especially, with Nethermind, where the release didn't include some fixes yet. And, Besu had some issues, where, there were empty blocks, that came from, peering issues on the Shadow Fork. We also had, some issues with, ethrex, because the peering on ShadowFork is a little bit difficult. And then there were some fork boundary-related, PTC issues and builder onboarding, that Lighthouse looked into. And, figure it out.

And then in the second run, Nethermind went quite smoothly. I think the builder onboarding also went well. We had also some deposits. And, Overall, it seemed, it worked pretty well, and a lot of the issues are being worked on. And we have Sepolia Shadow Fork, no Sepolia Normal Fork tomorrow.

Mario Vega: Awesome, thank you. Thank you, Stefan. Do we have, any comments? I think, in particular, from Nethermind? Yeah, ethrex and Lighthouse, do you folks want to chime in with any comments on this, or is everything resolved?

Iván | ethrex: I can start on the ethrex side. I don't know if you guys hear me well. I'm Masri. But… Yeah, the P2P issues we have are only related with this shadow fork, and shouldn't be a problem. Or tomorrow, or in the other networks.

Ben Adams: But Nethermind, it was fixed in our 2.1 release, 2.0 had some, some of the late ads to the ELS tests. We, we haven't included in that release, unfortunately, because they cut off And our QA was before that. Yeah, but they were already in our 2-1 release, which we intended to. Released before the Shadow Forks, but hadn't. And yeah, it's now available.

Mario Vega: Nice, thank you, thank you very much. Anybody else, who wants to chime in? Any incidents found or comments from other clients? Okay, I don't think so then. Stefan, anything else regarding devnets? Anything that you wish to share, or… Let know, everybody.

Stefan Starflinger: I think that's mostly it. Yeah. FOCIL devnet is… is down. If you want to talk to that part of us.

Barnabas Busa: Yeah, so I was just rolling out a new image update and, enabling QUIC on each of the CLs. And, Due to the fact that we don't have any supernodes online, we only had 100 validators on each of the nodes at reconstruction. Didn't go as planned. So, we ended up, going into different forks, and now, some of the columns are lost forever. So, in the future, then that's where you're gonna make sure that we're gonna have enough supernodes on the network.

Mario Vega: Okay, thank you. Ben, do you want to chime in?

Ben Adams: Yeah, one thing about Sepolia, so I was having a look on ENR Scout to to check how many people were on the previous version, and not upgraded to 2.1, and I've noticed, at least for Nethermind, there's a lot of nodes that are on a very old version. So, is there any… Any way we can tell people to upgrade? I mean, they aren't on the version that even has the Sepolia, releasing it.

Parithosh Jayanthi: I mean, we can push on the validator chat a bit, but other than that, I don't know if there's any other way to convince people to update.

Łukasz Rozmej: They will notice tomorrow.

Mario Vega: Okay, great. I see people already chiming in with, updates to Sepolia, so let's just jump into that. I'll start by saying that, Sepolia upgrade is for Glamsterdam, of course, it's, less than 24 hours away. It's happening tomorrow. at 13, 13 hours UTC, with 53 minutes. So I just wanted to… See if there's any last comments from clients today. And I see already one from Prysm that they are going to be doing another release for Sepolia by James. James, do you want to just say that out loud or do you want to share anything else?

james: Yeah, this should be a patch release. There were some last minute configuration issues that that were found. For… for builders, and so for… most nodes, it shouldn't matter, but if we're testing builders, we'll need to use the patch. And with this update. It'll also include the partial data columns, by… by default, so I guess that would also be nice to test for the… For the fork.

Mario Vega: Nice, thank you. Do you have a number of the version?

james: Yes, it's… I'll type it in chat, but it's, Version 7.2.1.

Potuz: wait.

james: And currently.

Potuz: Aren't we going to release 7.2.2?

james: 7.2.1.

Potuz: Oh, I just posted the wrong one on the Sepolia Validators channel. We'll change that.

james: Okay. Unless we're we're doing a second patch, then it would be that. But Our last release was 7.2.0, so this would be 7.2.1.

Mario Vega: Awesome, thank you. I'm glad that that got clarified today. Great. Alright, so I'm just gonna do a roundup of clients, just to check if any… if there's any other comments from clients. I'll start with the EL side of things. So, Geth, any comments for tomorrow, or… Version updates or whatnot? Rather just an acknowledgement that we are ready for tomorrow. If that's… if that's okay. Is there anyone from Geth in the call? Let me check.

Marius van der Wijden: Alright, so what…

Mario Vega: Just, just, readiness check, for tomorrow, Sepolia?

Marius van der Wijden: Yeah. Well, we've just had a new fix. We have a release that is ready for Sepolia. It would be nice to get that fix in as quickly as possible. So, most people should be running with the fixed version, but yeah, there's nothing to stop us from from using the old version for Sepolia.

Mario Vega: Alright, just to clarify, that fix is already out, or is it planned?

Marius van der Wijden: Yeah, it's planned, it's not out yet.

Mario Vega: All right, thank you, thank you. Following up with Reth, are we ready for tomorrow? I don't see anyone from Reth. I'll just skip them. Nethermind, you already said 2.1.0. Anything else you want to add or are we ready for tomorrow?

Ben Adams: I'm ready for tomorrow on that version.

Mario Vega: Okay, thank you. Erigon? All right, I'll just skip, Besu, any, anyone from Besu? Do we have a readiness check for tomorrow, Sepolia?

Daniel Lehrner: Yeah, we already… I also checked our own validators on Sepolia, they're all upgraded, so… Should be good.

Mario Vega: Alright, perfect, thank you. And is ethrex everything ready for tomorrow?

Iván | ethrex: Yes, we are in a similar position as Geth. Today, there was something… there was some finding from Nico, from Lodestar, and we are working on the fix, but we should be good for tomorrow.

Mario Vega: Perfect. Excellent. I'll just jump into the CL clients now. Prysm, you already mentioned that you have a new version out or planned, but anything else? From you, from you guys?

james: I don't think so.

Mario Vega: All right. One question from Barnabas, will the Prysm release have the 200 million gas set?

james: Yes, so the… it'll have the scheduled, the Sepolia scheduled, Gas limit stuff.

Mario Vega: Awesome. Thank you. Anyone from Lighthouse? Any,

Pawan Dhananjay: Yeah, I'm here. We have released 8.3.0, release candidate 0, that's the one for Sepolia. The Pandas did find a bug in the Shadow Fork, because of which Lighthouse would Not, published proposal preferences correctly before the… just before the Sepolia fork, so we have fixed that on Unstable, but I don't think we, I'm not completely sure if we will do a additional release for that. The worst that will happen is that we just do local builds for the first epoch of Gloas, which I don't think is the worst thing. So… If we do another additional release, we'll just post it on the channels.

Mario Vega: Excellent, thank you. Also, same question for you guys, do you have the 200 million gas set?

Pawan Dhananjay: Yes.

Mario Vega: Awesome. Thank you. I'll carry on with Teku. Just release check any comments or anything from you guys. First of all, yeah.

Enrico Del Fante: Everything is fine. We have 96.9.1 ready for Sepolia. I'm checking right now if the gas limit is in. I will follow up on chat, because I'm doing this double check right now. I don't have it in my… right in my mind.

Mario Vega: And one comment from Barnabas in the chat. He says that he doesn't think that the Teku has a 200 million gas.

Enrico Del Fante: Yeah, because I believe we merged the PR, but it's not part of the… of the release, so I was checking if we were… if we merged it after of… or before the release, probably it's after. But we are planning to do a release, but it won't go in, before Sepolia, so we won't release the .2. So now it's believe there is an You can, you can specify by, by command line anyway. But we can follow up on that in chat.

Mario Vega: Perfect. Thank you so much Nimbus. Anyone from Nimbus who can get on.

Caleb: update.

Mario Vega: delivery.

Caleb: Yes. So, we are… we are ready. So, I'm just checking our latest release candidates, which is 26.10.0. And I think we are ready to go.

Mario Vega: Excellent. Thank you. Lodestar, anyone can give us a readiness update?

Nico Flaig: Yeah, all good from that. We have the gas limit as well. And, I'm planning to run a builder on Sepolia as well. So if people want to connect to that, you can share the URL here. The only caveat we have on Sepolia is that it will not be possible to get them active at the fork, but I guess that is fine. I let EF DevOps speak to that if we want that, but I think right now we don't want it, so…

Mario Vega: Awesome. Thank you. Also, same question, is Lodestar ready with 200 million gas limit set?

Nico Flaig: Yeah, with the gas schedule.

Mario Vega: Awesome, thank you. And lastly, Grandine. Do we have anyone from Grandine? I see a comment in the chat, we're working on some fixes, but nothing major. Current release version 3.0.0 is ready for Sepolia. And they have the gas limit set, perfect. Awesome. I think that's everyone. I'll circle around Reth. Has anyone from Reth joined who wants to chime in about readiness for Sepolia? Reth or Erigon. If not, we can sync, via Discord or Async. Great. I think it seems like everyone's ready for tomorrow. Do we have…

Any other comments from clients? Do we want to discuss anything regarding tomorrow's fork?

Łukasz Rozmej: Let's go.

Mario Vega: Yeah, let's go. All right. Okay. Aw, Pooja. Yeah, go for it.

Pooja Ranjan: Thank you. It's not from clients. I think it's a special request for all the authors whose proposals are getting into the Glamsterdam upgrade. Since we are having Sepolia in less than 24 hours, I can understand the priority cannot be processed and making the PR, but once you are available, kindly take a look at it. Ideally, all proposals should be moved to last call, because on testnet, we would want to have all the proposals on the last call. So I have added a comment, if you could make a PR whenever you get a chance, and if that is before the next EIP Editing Office Hour, please add your PR as a comment, and we would try to merge it before the next ACD call.

Thank you, Dean.

Mario Vega: Excellent, thank you so much. Yes, please take a look into this issue. Barnabas, please go ahead, yeah.

Barnabas Busa: Yeah, I would like to raise one thing. So as Nico mentioned, we are not going to include any builders. before the fork. This is due to the fact that we have a special deposit contract on Sepolia that is Limiting us to the Bepolia token, and We basically just don't want to deal with making more deposits, validator deposits, to create the queue in order for us to have builders available at the fork. We have tested this on the Shadow Fork, and it was actually working well there. But for the fork tomorrow, I don't think we will be sending out Bepolia tokens to all the builders that would like to be participating before the fork. So, what I would actually advise is.

If builders want to test this, then, to do that on Hoodi. But yeah, everyone can just make a deposit immediately after the fork. That's when the deposit contract for the builders will become live. And just normal Sepolia ETH will be sufficient there. Oh, yeah, the deposit contract is already live, I guess. 2 weeks ago. Nikki has deployed the contract.

Mario Vega: Awesome, thank you. Spencer?

Spencer Taylor-Brown: Yeah, just a more of a question from my side. Just around versioning for testnet. So my understanding is that We have the blog post with all the versions for all the clients. But what's, like, the process that goes on, like, when there's a new version that needs to be bumped? And, like, how do clients actually announce the versions that should be used? And as a general, is this something that could be improved? And I guess I'm asking this now because… Sepolia's tomorrow, and we have Like new version bumps happening, I guess, within the space of a day.

Nixo: Probably bring pinging Alex.

Spencer Taylor-Brown: So the blog post is like the default source of truth.

Barnabas Busa: Yeah, I would say so.

Spencer Taylor-Brown: I guess I'd ask as well, like, for me, if I'm updating my version, like, I would look at the blog post once and be like, it's done. Is it expected for people to… Continuously monitor the blog posts.

Parithosh Jayanthi: I think it depends on maybe the network, so for Sepolia, we'd also be posting these updates on the validator chat, and you'd be quite sure that all the validators know about a version, so even a day is not too crazy. For Hoodi, it's also roughly the same. I think majority of validators are on the validator chat, and everyone else would basically hear from either the blog or Twitter. For mainnet, that's where it gets, I guess, a bit fuzzier, because there's no central location to coordinate people, so it would just depend on client release channels and Twitter and the blog posts and a combination of everything. I think it changes per network, basically.

Spencer Taylor-Brown: Thanks, yes. Sorry if that's a random question. I just figured it was, for me it was useful, so thank you.

Mario Vega: Nice, thank you. Any last questions, last thoughts or comments on Sepolia? From anyone. Alright, yeah, it seems like we are ready for tomorrow. Yeah, good fork, everyone. Yeah, I think that's it for today's, general. Call, and we are ready to go into the breakouts. If I understand correctly, Jihoon, are you in the call? You are leading the CL breakout for today, right?

Jihoon Song: So, Wilby, could you help with creating the breakout channel?

Mario Vega: The breakout channel, do you mean the call, or…

Jihoon Song: Yeah, they're cold.

Barnabas Busa: We have another URL for that, no?

Mario Vega: Yeah, I think Pooja just shared the link. Perfect. Awesome. So everyone in the EL call, just stay here. We'll start in a bit and please everyone from the CL jump into this different call. All right. See you, CL folks. And I guess we can just get started on the EL topics. The first one. Was brought up by Maria. There's a message in… let me see which… ethereum magicians post that discussed the default execution gas limit for the ETH call RPC method. So, Maria, would you like to, go for this topic?

Maria Silva: Yeah, so this was, this is actually something that, is not a consensus change, so this is something we could do, without any consensus breaking, but essentially, and this was something that was only… confirming gas, so I don't know what the other clients did for ETH call. But essentially, for Geth, ETH call is, implementing the same Gas limit for execution as the, transaction. As the… the transaction limit, so the… the 16 million, gas. And essentially, there are, cases where people want to simulate calls that take more than the 16 million, Gazan for them. This is kind of, breaking their… their use case. So before ETH call was already… didn't have, the limit that a single transaction has.

And so… the point here is that, so Gary from Geth already confirmed this and said he was fixing it in Geth. The question now is, like. What did other clients do? And, should we make it consistent? So my idea was initially to maybe add a comment on the EIP itself to describe that this should be the behavior for ETH call. But yeah, I would like to also see the opinions from the other clients.

Marius van der Wijden: Shouldn't this just be specified in the execution APIs repo? I think that's at least, like, the place. that this should… Probably be in.

Maria Silva: Right, so you say it doesn't make sense to be in the EIP. At all.

Marius van der Wijden: Well… I'm not saying that, like, it can be added to the EIP, but I think the important bit is to add it to the execution APIs repo. Because it's, like, the canonical specification for the APIs.

Maria Silva: Do clients have any comments here? Yes, so essentially I want to know if… everyone is okay with just keeping the current behavior of ETH call, because my understanding is, like, this would be changing it.

Mario Vega: Luis, if you're speaking, we cannot hear you.

Luis Pinto: I was waiting, I was waiting for you to give me a… Oh, sorry. Okay, yeah, so I'm wondering, because if the maximum transaction gas limit is reached, what's their use case for simulating an ETH call that does not work on chain from now on?

Marius van der Wijden: It's, it's, it's, heavy view functions in contracts. So, for example, there are some few functions that, that paint a picture of the, of the NFT, and so you want to be able to do that in the front end, via the deployed code, but you don't, like, you will never do this on-chain.

Maria Silva: Sorry, I would just add on that comment, on the EIPMA decisions, they specifically refer the, the. the renderers for, for SVGs, so what's the use case that Marius was saying, but also, like, quoters and, aggregators, so, yeah, but essentially, that's… that's the idea. Chase, to follow on your comment. So if Geth is implementing these in the right way, I would assume then that would be translated to the execution APIs. Repo at some point.

Chase Wright: Yes, that's correct.

Maria Silva: Okay. Yeah, I posted on the 8037 channel on Discord, and Gary said I already have a fix for that. And it says it has a fix for ETH call, trace call, and ETH simulates. So, yeah, just need to make sure that it's included, but I'm guessing it should be. But I can follow through with that. Are there any more questions, concerns, or is everyone happy with this? Go, jochem.

Jochem Brouwer: Yeah, so I'm reading this comment on magicians, but And… Does that not imply that we should actually change this API to have both state gas and execution gas? This is what we are proposing here.

Maria Silva: My understanding, maybe it's something we can double-check with Gary, but it's just the fix was just lifting the limits, not necessarily… Removing the two counters, but I'm not sure.

Jochem Brouwer: Okay, because I think this is somewhat problematic, because, so for magicians, the thing is that for SVG vendors, so this is an execution, which needs, like, a lot of execution gas, but for ETH call, we also have to, the purpose of ETH call could also be what would happen if I would execute this on-chain. So… That means that, a call, which, on-chain would create, like, a lot of state. That would mean that… if we lift this limit, then the question is, okay, but how do we lift this limit? So, do we put all the gas in the execution gas, or do we put all the gas in the simulation… oh, in the state gas, right, but if you put… Yeah, it needs…

Maria Silva: It needs to be all in execution gas, because it's the one that can be used by both state and…

Jochem Brouwer: Yeah, I see, I see. Okay, then, okay, then I think that's not a problem. Yeah, okay, sorry, yeah, thanks.

Maria Silva: Yeah, and I would say this behavior already happens today, right? Like, if you want a transaction that takes more than 16 million, it would fail. Like, the ETH call would execute and would be… Let's say, valid, but then you would actually submit the transaction, and it would fail, so… That is the behavior today. Okay. Okay, any more… Questions, concerns about this? Okay, if not, then I will reply on… Eth Magicians to communicate that, essentially, this would be fixed with an… with the execution API release.

And I will double check with Gary, to see if the thing is already merged or not, and yeah. I think that's it. Mario, I don't know if there are any topics.

Mario Vega: Yeah, I think… no, there are not. This is the last one. I think we can… we can end the call here, then. If nothing else. Yep. All right, we can. We can end the end. The call, then. Thank you very much, everyone.

Marius van der Wijden: Thank you, bye bye.

Mario Vega: Wait.

Jochem Brouwer: That's all. Bye bye.

Mario Vega: Goodbye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.