All Core Devs - Consensus #165

Transcript

Alex Stokes: Okay, Josh, are you helping with the stream today?

Josh Davis: Yep, you're alive.

Alex Stokes: Okay, thank you. Hi, everyone. Welcome to ACDC 165. This is issue 1716 in the PM repo. And the main topic for today will be Fusaka, with a few other minor notes for Amsterdam, but yeah, let's go ahead and get into it. So, to kick us off. It'd be good to touch on Fusaka, DevNet 3, and DevNet 5, and yeah, kind of where we're at there. I wonder if, Perry or Barnard, Barnabas or Perry have any, updates there, or anything they'd like to call out?

Barnabas Busa: Sure. Let's start with DevNet 3. DevNet3 has been, mostly very stable during this past week. We are over 96% participation now. Teams are still pushing out fixes into their trunk branches, and they get automatically deployed onto this network. We haven't been doing any kind of… attacks or any splits in this past week, because we have been mainly focusing on Cosaca.net 5, in this, past week. 7 at 5? Started out with, 1,500 nodes, and we did a fulloo last week's Thursday, then we did a BPO a day after the birds. the… Went up all the way to 72 max blobs, and we did 48 on the day before. Unfortunately, we had a… bug in the configuration, which led to the full node getting a super node flag, which basically meant that our full nodes were

At 100% bandwidth utilization pretty much all the time. Which led to quite some issues. Yesterday, we fixed this by wiping all the full nodes and re-initializing their databases and their… and this also reset their, CGC value. And we ran an analysis with, 15 max blobs, 21 max blobs, and then 33 max blobs, one more time. But, because of the time window, it's so short between, that time and ACD, The numbers are not really, representative of, Of many, many slots. Full nodes are representing 90% of the number of nodes, but only 10% of the validators on DevNet 5. Nevertheless, some from our team did an analysis. Pari just linked it in the chat. Hopefully, most of you had the time to take a look before the call.

And the conclusion of our, analysis was that, max values of 15 and 21, seem to be reasonable for BPO1 and 2. And next value of 32, might lead to struggles into publishing blobs without any V-release or full nodes, with limited bandwidth. We certainly will have to do another analysis. It's probably going to be physical.net 6, maybe a couple of weeks from now, but, we feel like we should be good to go with Holosky and with these, BPO numbers.

Alex Stokes: Okay. Yeah, thanks for the update. Yeah, this analysis, looks pretty good. I'm just looking at it now, but… everyone definitely take a look there, and… yeah, at least I think that will then confirm, these first two EPOs, which I think was the main thing we wanted. So… yeah. Any other known bugs, with any of the clients, or any issues we should touch on with the DevNets?

Barnabas Busa: Yes, Prism has a known bug that they are investigating right now, that leads to a bunch of, often, blocks. We are rolling out a fix, or a potential fix right now. I'm not sure if they have anything more to share.

Emmanuel Nalepa: Yeah, I can share something. So yes, we have a bug in prison with a lot of orphan blocks. It seems to be only the case with high blood count and locally build block, so there is no issue if it is… if there is MEV, or no issue with low block count. We are still in the process of narrowing down the root cause. What just we know right now is When we produce the blog, the blog is produced in very well fashion, like a few hundred milliseconds after the start of the slot. And what we see at the end is when we broadcast, the columns, the tech columns are broadcast very late, into the slot, like 6 or 7 seconds.

So there is something fishy between… after the end of the blog bidding and before the start of the data collection broadcasting. And yes, we are in the process of finding the root cause in this process. That's all I have to say.

Alex Stokes: It seems like this is a regression. Does that track with your understanding? Or do you think something… do you think something new was introduced that caused this, or is this a regression from what we had in the past?

Emmanuel Nalepa: Actually, possibly, it's not a regression possibility, it's an issue which is here for a long time. However, if I'm not wrong, we… the best… We mainly tested, high blood count with MEV, which does not cause any issue, and I think we really started to see the issue when we… when, you know, MEV was deactivated on the… On the… on the DevNet 5, with high blood… high blood count. So, possibly, it's not so… it's maybe something totally unrelated to pureness. A bug which was here from, you know, a long time, but which was not visible with a low, low blood count. Yep.

Alex Stokes: Yeah, frozen period?

Raúl Kripalani: Yeah, so I think this could be, just kind of like a potential lead here, could be the fan-out mechanism. So basically, when you're a builder, you are forced to publish to 128 subnets, simultaneously, which means that you need to have peers in all those subnets that are peering… that you're peering with, that are part of your mesh. And this usually requires a little bit of preparation ahead of time to make sure that, you know, given a particular peer count, active peer count, you have sufficient density of peers in each of those subnets. I am aware, I think we've been looking into the Prism codebase as well, and the peer-to-peer networking team, and we did find out some events in which

Prism can, like, spend 5 to 6 seconds or something like that, discovering peers when it doesn't have sufficient peers, and if this is part of the critical path, that could justify the delay that you're seeing here. And this goes, like, in general, as an advice, I think, to all CL clients, is ensuring that, you're correctly Predicting when you're gonna be up for builder duty, for proposal duty, and if you're gonna… if you're a local builder, then ensuring ahead of time that you have sufficient peers in your meshes, so that when the time comes, you don't spend any time actually conducting the discovery during the critical path.

Emmanuel Nalepa: Yeah. So, so for information in a presumably, as soon as We… we have one active validator. We ensure to have, at least 6 peers, in our fanhood for our data collection subnets. So we don't compute, nor are we going to… Are we going to propose a block in one way? Just, yes, if we know we have validators, we always make sure that we have peers in all the technical subnets. Paramount. Yeah. So normally, I mean, we don't have to search for a new peer when proposing a block. And if we do, we have luck for that, and it doesn't seem to be the case. But, yeah, I will double-checked, for sure.

Parithosh Jayanthi: Yeah, one of the things I wanted to mention was, in the past, we would either have MEV completely running throughout the lifespan of the network, or we would have it turned off. We didn't really have a fine-grained mechanism to change this, mid-BPO, for example. But now we've added a scripting approach, so if a BPO lasts for one day, then 12 hours of the day is gonna be with MEV and 12 hours without MEV. So just in case any client dev is looking at a DevNet and wondering why there are no MEV proposals, it's probably because we've disabled it, so that we can test both local block building as well as MEV workflows for all blob numbers.

Emmanuel Nalepa: And, yeah, and additionally, and from additional information, currently, I know that I checked one hour ago, DevNet3 had LED on. And DevNetri has no issue about offhand blocks for poison. And, yeah, the next 5 had MED off, and there was a lot of orphan issue.

Barnabas Busa: DevNet3 has a significantly more share of super nodes, though, and also much less blobs. They have a next blob count of 20 at most, and I think 50% of the network is super nodes. I'm… on DevNet5, it's also possible that some of your peers will be only peering with other non-super nodes. So, peering will be significantly more difficult to… to find enough peers on… enough columns. That has all the columns that you need, basically.

Emmanuel Nalepa: Okay, I will double-check that. Should not be the issue, but yes, worth to check again.

Barnabas Busa: We might need to increase the default peer count on some of the CLs, but I think Lighthouse also increased from 100 to 200.

Emmanuel Nalepa: Yeah.

Alex Stokes: Okay. Anything else on the DevNets? Okay. If not, then, next on the agenda, so there was an issue that came up with CKZG. Justin had a note here in the chat, I don't know if you want to add anything to that, Justin, otherwise I can, Give a summary.

Justin Traglia: Nothing really to add, just that… there was a misunderstanding of how, Blast used points at infinity, or… yeah. Pretty much we fixed it with a new release, shouldn't be an issue anymore.

Alex Stokes: New release of CKTG.

Justin Traglia: That's correct. Version 2.1.3.

Alex Stokes: Okay, gotcha. And… The understanding is that everyone should be using 2 and 3. Or at least not using 2-12.

Justin Traglia: Correct.

Alex Stokes: Okay. I don't remember if there's a way to link this in the specs, but there either should be, or… yeah, I think we should make this clear somehow. How have people handled this in the past? What was that, Felix?

Felix Lange: Sorry, I don't think it's necessary to, like, link specific library versions in the spec.

Alex Stokes: Fair. So then we should just remove 212 from the repo?

Justin Traglia: You mean, like… or what do you mean, exactly?

Alex Stokes: Well, it sounds like 2 and 2 is just broken, right?

Justin Traglia: Yeah, sure. You wanted me to, like… Okay. Remove it as a release from TKZG?

Alex Stokes: Yeah, I mean, I want to prevent what happened, where I think some clients upgraded to it, because there was a new release, and then it caused some issues.

Justin Traglia: Yeah, I mean, we can discuss it. Personally, I don't think that's necessary, but .

Alex Stokes: Right. I'll follow up with you later, then. Then… I guess that's everything there. Just be aware of that. Okay. Let's see here, let me look at the agenda. Okay, so, yeah, if that's everything we need to talk about in the DevNuts, The next thing I wanted to bring up was the testnet schedule. So, I have a proposal here. And… let me just run through it, and then we can discuss. So… The proposal would be next Monday, which is the 22nd of September, effectively code freeze. This could be, like, a branch or some commit on, say, you know, a trunk branch, but ultimately, you know, this would be the code that would go into releases for testnets.

If we have code freeze then, I believe the PandaOps team is willing to do some validation on this code. Just to make sure things, you know. look okay, and yeah, maybe once I'm done talking, they'll have something to add here. On that point. But essentially, aim for Code Freeze next Monday. Then if we aim for testnet releases for all 3 testnets on the 25th, which I believe is next Thursday… let me just double check that… It is… Then from there, we would aim, following sort of the process stuck that we had, at something like 2 weeks, For every testnet from there. So then, Heleshky would be on 2nd of October, Sepoli on 16th of October, Hudi on 30th of October, and there are fork epochs that align with all the different parameters we need, here on the repo.

And… I guess one thing to note is the Sepolia fork that would work on the 16th of October is during ACD, so I'm not sure if we want to move that. Even assuming we can, you know, agree to the schedule here. But that was something to flag. And, yeah, one other note, we would use the BPO schedule from .NEF5. Again, it sounds like from the analysis we had, we can at least validate these first two BPOs with the numbers that were given. From there, I think everyone knows the deal. You know, if there's a test network and it doesn't go well. This is basically, you know, everything after that's on pause. If everything does go well, then even when we get to the last testnet with Hoodie, mainnet would be, you know, 30 days after that, so…

That's a proposal, it keeps things moving, and yeah, also generally follows this process document that we've been discussing over the last few ACDs. So… I see… yeah, Christine here put this in the chat, thank you for that. And, okay, yeah, Barnabas, do you want to discuss your adjustments?

Barnabas Busa: Right, so my adjustments would be that we don't do a fork on a non-Wednesday, basically. Okay. Wednesdays are probably the best, most suitable days for a fork, because it allows us to debug before and afterwards as well. And it doesn't clash with ACD as well. My proposal would be we would do Holsky. Fusaka on the 1st of October. Then, a week after that, we do the first BPO, then a week after that, we do the second BPO. Then, a day after that, we do Sepoya Fusaka. And, we basically just keep repeating this pattern, throughout all the different testnets. So we wait one week for each BPOs, and, we're gonna ship all the BPOs before we proceed for the next testnet,

book. So this allows us to run analysis on the different testnets, just to make sure that everything is still looking good before we, pull the trigger.

Alex Stokes: Yeah. Okay, I mean, yeah, I'll validate the numbers you have here, but… That seems reasonable. I think, most importantly, I would like to hear from the clients what they think. I reached out to a number of different, I think I reached out to every client team over the last couple days, and I think generally they're okay with this. I think the main thing to figure out for now is just really next week, in terms of code freeze and releases. But yeah, I'd love to hear, if anyone Has an issue with this, or thinks we should make adjustments somewhere. So, I've gotten thumbs up from all the client teams before, so if no one says anything, I will just assume it's still thumbs up.

Okay, now there's a lot of thumbs up on the Zoom screen. So… Okay, that's very exciting. Does anyone… I guess one question here is… yeah, like, do we just go with Barnab as a schedule? I'd have to look and see… But it seems good.

Barnabas Busa: All the epochs are on the boundaries. So…

Alex Stokes: Okay, yeah, that's mine.

Barnabas Busa: The official room fight, also, I made sure that the BPOs are on the boundaries as well.

Alex Stokes: Okay. Yeah, I guess what we could do, because this might take a little double-checking async, is just also have this finalized by… Monday, along with Code Freeze. And… I would say lean towards go with this, unless someone wants to say otherwise. But yeah, I just…

Parithosh Jayanthi: The next step needs to be a PR to the ETH clients repository, updating and setting the forks, and then we can make sure that all the clients have, looked at that before we merge it in.

Alex Stokes: Yeah. Okay, yeah. Oh, Barnabas says you can do that. Great.

Barnabas Busa: So, cool, yeah. One more thing that, I suggested that we're gonna have different release deadlines, instead of trying to get everyone to include all the timelines in the release. I'm not sure if they want to. But, like, we could have a… you could basically have all three raised deadline by the 24th, then Sepolia raised deadline a week after that, and two weeks after that, the Holy One. I'm not sure if it's really needed. Or whether we just want to pre-schedule everything in one release.

Alex Stokes: Yeah, I think it's simpler for everyone if there's just one release, but… Yeah, I guess so. I'd love to hear from clients how they feel. Is that helpful for anyone?

Ameziane Hamlat: So, for Basic side, it would be helpful, because we are working on some performance improvements that we would like to include for Fuseka. So we may need to have a different release between the one for Holesky and… Let's say the one for a hotel, for example.

Łukasz Rozmej: So, I think, like, we can definitely do one for all, but we might, Also, like, do some service releases after that, But I think that's fine, and that should also be fine for Bess, so I don't really see this being a conflict. I think if we can agree that it's probably fine to have a release that supports… as a baseline that supports all testnets, but then have subsequent releases after that. That shouldn't be a problem.

Alex Stokes: Okay, so, well, everyone will aim to have one release, but then, yeah, if things update, then that's all perfectly fine. Okay. Cool. Does everyone feel good about this?

Matthew Keil: Yeah, I can second that sentiment, that, we're ready for a release, and, but the flexibility in case something comes up during test sets. To be able to put one more out before mainnet.

Alex Stokes: Yeah, I mean, you can always, I think, Cut a release. But just for the test nights, given there's so many, sometimes it's simpler just to aim for one, but… Yeah, definitely, we should be able to accommodate, At least 3.

Dustin Brody: I mean… Practically, this means that there's… unless there's a… things go catastrophically wrong, there will not be a schedule delay. That's not necessarily good or not necessarily bad, but… but once those releases are out there, and now… Essentially, the dynamic that happens is it becomes partly as risky to delay the release, even if you think it might go wrong, than it is to… the fork schedule, than it is to keep it going and just make it a… an emergency fix, if all of the releases are scheduled at the same time, and this has happened before. So, I mean, as long as people go into that knowingly, that's… Fine. I don't know. It's a choice. But…

And this is… but this is one reason why I know the… in other discussions, this has been… Yeah, a trade-off discussed.

Marius van der Wijden: Yeah, I would say it's fine. I think, Because these testnets are only run by a very small group of people, we can always ask them to just use the newest release. Yeah, and the important thing is that we don't bake in the mainnet release into the tested release, so that the mainnet release will be must be a different release. And I think that's… that's it.

Alex Stokes: Yeah, sounds good to me. Okay. Again, I know this has been a huge lift to get here, so, I guess congrats, everyone. And let's hope the test notes go well. So… Cool, we'll get this PR up for the schedule…

Barnabas Busa: Should I make the schedule then based on the timelines that I've suggested?

Alex Stokes: Yeah, I think so. I'll double check, just to make sure, you know, things are aligned to these boundaries and stuff, but yeah, let's go ahead and go with what you had. It looks good. Cool, okay. Anything else on Fusaka? I think this is pretty much… what we have today, on Fusaka, just because, yeah, we're in the process. Anything else?

Parithosh Jayanthi: The audit went live, do we wanna maybe mention that?

Alex Stokes: That would be good. Frederick, I don't know if you have something to say there, otherwise… I can go try to find a link to it. I don't have one handy. Here, I'll just… oh yeah, thanks, Justin. Okay, so yeah, I was looking at this blog post here. So… Yeah, you know, every upgrade, we usually do one of these security competitions, we have one live for Fusaka. There's a $2 million bounty here. And, yeah, check out the post, especially if you are in the security space and listening to this recording, or watching it live. This is super useful. We've found things in the past that, you know, are direct impacts to mainnet security, so… This is a super good program, just to find any lost bugs lingering.

And… yeah, I think that's all I have to say. I don't know if anyone wants to add anything else. Mario says it projects 20,000 submissions by the end of the contest. I don't know where that number's coming from.

Marius van der Wijden: Yeah, that… I don't know that…

Alex Stokes: odds.

Marius van der Wijden: Yeah, the algorithm is kind of weird, but there have been some submissions, and the security team is looking at them, but we also want to encourage all the client teams to look at them as often as possible. So, yeah, these things don't go stale, and we don't receive, like, duplicate submissions because the code hasn't been updated.

Alex Stokes: Okay. Cool. Anything else on Fusaka?

Barnabas Busa: One more thing, like, it would be very good to get an update on, backfill from different clients, because I think this was one of the main points that we, still have to ship. I think Lighthouse now has working backfill, but I'm not sure about any other client teams that have, finished implementing Backflow.

Alex Stokes: Okay, Bota says Prism has it. Yeah, there's an implementation they're working on.

Fredrik Svantes: I can say something more about the competition quickly. There is, for the client teams, so they're aware, There's a bit of an issue right now with the… Judging aspect, but that's gonna be fixed. Tomorrow, most likely, so that's… Brian teams can provide comments, and yeah, feedback on the severity levels, etc. And then, yeah, if you need any more people from your team to have access to the competition, to provide this feedback, then you can just reach out, and we will get everyone added.

Alex Stokes: Alright, thanks. Yeah, Barnabas, back on the backfilt topic, do you just want to follow up async?

Barnabas Busa: Yeah, sure.

Alex Stokes: Do you have any other questions right now? Okay. Did we want to touch on this earliest available slot thing? Because that also did come up.

Barnabas Busa: I'm happy to bring it up, but I think that can also be discussed. Oh, awesome.

Alex Stokes: Okay. Another comment here in the chat from Solus, Grandine also has backfill, merged very soon. Cool. Then with that… Glamsterdam, and then, yeah, a few announcements from A.B. Kotap. Let's make sure I say that correctly. So yeah, with Amsterdam, I think the main thing to keep in mind for now is that we said we want to have non-headliner EIPs proposed. by Fusaka Midnight Releases, which, with the schedule today, gives you some more clarity on when that would be. So that's good. Otherwise, yeah, I don't think we really need to get into it, just because I think we should all stay focused on Fusaka for the time being. Yeah, if there's, like, very brief… announcements or something anyone would like to provide around breakouts or something, I think that's admissible, but…

Otherwise, I think that's all we really need to say there. Then, on the announcements I mentioned, so… we do have an ETH Magicians post here. for the name for H-star, Which, yeah, I forget… Okay, there was a poll… here, let me grab the link here. There was a poll here, and I guess, yeah, we just need to actually make the decision at some point. So yeah, take a look there if you want to chime in. And there's another request here for a name for the replacement testnet for Sepolia. I will… Share this, because it was added to the agenda. I… Myself, I'm not convinced we need to replace Apolia, even though I know that was sort of what was written down with, this LTS situation.

But I think that might be a conversation for a different day. So, in any case… Following the schedule, yeah, we would basically sunset Sepolia, Soon-ish, and then there would be some replacement, and this post is just asking for a name for the replacement.

Parithosh Jayanthi: I do want to maybe make a point here, we're still helping a lot of people migrate to… from Holsky to Houdi, so we don't want to add to the chaos and immediately have everyone also migrate from Sepolia. So, even though Sepolia is supposed to be end-of-the-year terminated, I think we will add, like, a decent grace period. We'll probably update wording, and even if there is a replacement test for support here, we'll make sure that both are running at least for a considerable time, so that no one's, overburdened with having to migrate their entire stack. Multiple times in a year.

Alex Stokes: Yeah, that sounds good to me. I mean… Yeah, again, maybe a topic for a different day, but I am not sure why we need to actually tear down Sepolia at this moment. But yeah, again, different, conversation. In any case, yeah, I think what Perry said stands, so there'd be plenty of grace period, just so there's not, yeah, more pressure there. Okay, with that, this was everything we had on the agenda today. Anything else?

Barnabas Busa: I would like to discuss, Glamsterdam testing, and, possibly splitting up, all DevNet 0 and EPPS DevNet 0, and keeping them separate as long as we basically can, and we want to. Personal has a 90% done implementation that would support, the block access stuff, and I think, we can just basically do a prism-only, DevNet for all the ELs that would be supporting it, and that way we don't have to burden any of the other CR devs with this. And they can implement it whenever they want. So, at least for the first couple of iterations of, while DevNets, I think we can just have it with Prism. And then, on the other side… So that would not trigger any EPPS testing, and then we could have the EPPS testing without the ball testing, basically.

I just want to make sure that all the CR teams are focused on EPBS testing and writing EPBS stuff, and so we can start testing in parallel as soon as possible.

Alex Stokes: Yeah, that part makes sense. I was gonna ask, is this okay with the Prism devs? Like, are… Can they produce something that's good for BLs?

Barnabas Busa: It's already done. Like, it's 90% done.

Alex Stokes: Okay. Okay. Okay, and yeah, just to be clear, that would have the EPPS features in it, and they would do BLs on top of that with EPPS?

Barnabas Busa: No, it's without the PPS.

Alex Stokes: Okay, interesting. Cutters?

Potuz: Yeah, so the issue is that, block access lists, without EPBS requires a change in the beacon state, and this is the problem, and why we were petitioned on having different branches. With DPBS, block access list doesn't require any changes on the beacon state, because we've removed the header, the execution payload header from the state. So now, in the future forks, whenever we change the payload, we do not need to change the beacon state anymore.

Alex Stokes: Okay, Francesca's also asking in the chat here, we might not need CL changes? For BL.

Francesco D'Amato: Yeah, it's just what was just said by Purdus.

Alex Stokes: Oh, sorry, okay. My bad.

Barnabas Busa: Yeah, but the question is, can we… can we check, ER functionality without… like, testing any EPBS functionality.

Alex Stokes: So what would that look like? It would be a Fusaka release with no PPS on top?

Barnabas Busa: Yeah, so the PR that I linked is that. It's basically implement EIP7928 as gloss fork without concentrating.

Alex Stokes: Okay. Does this work for the AL teams, then? Because then, yeah, they can use this for the CL, and then we can move ahead on VL DevNets. Podos, did you raise your hand again?

Potuz: Yeah, so I was… I wanted to suggest, I mean, it's absolutely fine to use this branch now, but I would suggest that as soon as we have some minimal implementation of EPBS, that we can… if we could, switch to doing this on top of EPBS, because maintaining this branch, even though it's mostly generated code, it's going to be a mess. It's… I mean, it's… it changes everywhere. Whenever you change the beacon state, it changes everywhere. So I think it would be better if we switched to doing block access lists on top of EPBS so that we maintain a single… and we actually start merging into our main branches these changes.

Parithosh Jayanthi: Yeah, we just need it for now, so that we can unblock stuff. We don't need to maintain this long term. The moment EPBS has minimum viable client support, we're gonna switch over to that. We just want to ASAP start with block-level access lists, and also just for the other CL teams, you guys don't have to do anything. We just need one CL team to take on the topic, because that will unblock us, and the rest of you can continue work on EPBS, and Mudge everything later.

Alex Stokes: Right. So it sounds like we have a plan, then. We can use this PR here, 15712. And move ahead with BLs, and then, yeah, I guess another sort of point here is not having this become, like, a persistent branch. Plan being to get things merged sooner rather than later. And… yeah, okay. Are we good there? Marvis?

Barnabas Busa: Yeah, I have just one more thing. So, if we can have, client releases, done by maybe next week, Monday or Tuesday or something like that, and we plan to do a shallow fork. And this shadow fork is going to be for Hudi, one for Sepulia, and one for, Koski. And then… probably we can also do a mainnet shadow fork, but maybe not next week. So ideally, as soon as the client teams are ready, please let us know, because we would like to have this shadow fork up and running as soon as possible.

Alex Stokes: Yeah, there was also a question here from James, about this, so… I don't think it matters to you, but would you prefer a release, or, you know, is, like, a commit okay? Or is it more up to the team?

Barnabas Busa: I would personally prefer a release, yeah.

Alex Stokes: Okay. That might take a few more days, though. And is that okay?

Barnabas Busa: When do client teams plan to have OSCI releases, if we plan to roll it out by the 1st of October?

Alex Stokes: So, I think, generally, we're aiming for Thursday next week.

Barnabas Busa: Okay. Yeah, I mean, we can do… we can do that. If we have, releases by next Thursday, we can do host t-shadow work next week, Friday.

Parithosh Jayanthi: Yeah, so maybe some, context on that. We already have all of our Shadow Fork, playbooks up and running. I'll run them through a mock version tomorrow, so that when the client releases are done, it should take us a matter of, let's say, hours before we'll have the Shadow Fork update.

Alex Stokes: Okay, so I guess the point is that… Pandops will do some validation here with shadow forks and things, as soon as they have, you know, something from your client team, Yeah, I think from there, like, if we can hit Monday, that's great. the reason to separate, like, code freeze on Monday from, say, a release on Thursday was just as… was just because some of the clients, I think, have a few days to, like, soak the code before they make a release. So… Yeah, but then I guess you're also saying it's fine if you don't get that… And implementation until Thursday, right, Barnabas?

Barnabas Busa: Yeah, that's fine.

Alex Stokes: Your thumb is still up. Okay, cool. Everyone clear on that? Any questions?

Dustin Brody: I'm wondering on Guido, I mean, has this been tested at scale? I mean, I say as Nimbus was what Nimbus and Techu were a couple of the clients who discovered this, Nimbus was… exposed to this in the DevNets for a while, and we were performing worse because of this. This was fine-ish on the DevNets, that's what they're for. We don't… but typically, we would not want to do a release with… who knows? I mean, especially if it's BLAST and CKCG at the same time. I would say, except for emergency situations, this is precisely the kind of dependency bumps that I would wait till just after a release like this to do, so we had not just two days, but weeks, or at least a couple of weeks, which is clearly not going to happen in this case, for this to, soak, to use the verb people are using here.

So, that… there's some wariness on my end from that, I'll say.

Alex Stokes: Yeah, because you're concerned with an issue in the newest CK, did you release? Or just simply doesn't that time?

Dustin Brody: Become Lindy. Yeah. Yeah, that… well, if it were only the CKZ Geo, it actually wouldn't be as bad, because, if it… that only really affects Fusakra to begin with, then that's a lower stakes thing for now. That's test ad only. But we don't do test and only releases if we don't have to, and we won't have to otherwise here. And so this will also be a mainnet release, not the Fusaka fork mainnet release, but the mainnet release. And BLAST is a absolutely critical dependency. we don't update with P2P, like, 3 days before a release either. That's… Stupid high risk. why… I don't know that we would update Blast, and that's why I asked before, and got the answer from, Justin, saying, well, no, like, they go together. And they don't… Blast doesn't have a release out yet.

Justin Traglia: That's true. This is just hybrid from my end. Would you prefer if we make a new CKG release that adds back the pointed infinity filtering to CKZG, so that you do not have to update BLAST?

Dustin Brody: Yes.

Justin Traglia: Okay. We'll do that. I think that'd make… Everyone felt better, so… Yeah, consider that done. I'll do it today.

Dustin Brody: Okay, great.

Alex Stokes: Okay. Anything else? Okay, if not, then let's go ahead and wrap up today. Good luck with MSACA, everyone. And I will see you next time. Oh, this is a fun question from Perry. Did anyone do anything cool to celebrate March Day? I imagine we were all working off Osaka. I did. Yeah. Okay, I'll see y'all next time. Thank you.

Potuz: Bye, guys.

Justin Traglia: Bye, Ron.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.