PeerDAS Breakout #22

Transcript

I apologies for not updating the calendar invite. I overlooked that part. I didn't know I had to add access to it. Um, yeah, we realized yesterday that the Zoom link uh was going to shave and from an old meeting that she inherited. So, and the automation is not working currently for the uh the repo. So, we were trying to come up with a temporary solution that would not end up in Shaw's lap. Um, well, almost got there, but anyway, uh, my name is Will and I joined EF uh, not too long ago as a research coordinator and have been asked to help moderate these calls and, uh, do some

overall coordination efforts. So yeah, I tried to spend last week um going and meeting with a couple of individuals and just kind of getting a a highle overview of the status of the project and then just trying to distill all that into the the agenda uh issue that's in the PM repo that I shared in the chat. Um so nothing really new is going to change about the the course of the meeting today. We're going to first start off with client updates and then go to pure uh sorry DevNet updates um and then jump into uh specifications and testing and all that stuff. So uh I

will you know I'm still trying to get a sense of who all the representatives are for each client team. Uh I tried to put like a my best understanding of a roster in the the chat uh or sorry in the issue. Uh but if I got anything wrong or overlooked anyone uh know my attention just really trying to get a a better understanding of of who all is contributing and and who works for who. Um, but that said, if there's someone from the Prism team, uh, I don't know if we just want to go around the horn and, uh, get a sense of where we're at with client updates.

Yeah, sure. So, uh, hi, welcome. Um, I'm Manu from the Prism team and, uh, I, yes, I'm the main contributor on Prism about Purus. um about the last week. So you may know Prism had some issues with the way it stored uh data columns in the database. We needed too many uh we needed one file per data columns which was an issue. So I spent the whole week working on redesigning a new database. Um there is something working right now. I was able to sync a node on the dev net for my local computer and Baronabas or Perry

are going uh to roll out this new version to the pure devet 5 which should normally get back to live sorry um prism super nodes. Um at the same time I spent a lot of time was quite um missing unit test in prism about pureas. Some unit test were still missing and I pushed them up to far future epoch and so I decided it was a good time to do it now before it's too late. Uh yes, it was my main contribution and uh Francis from Bass worked on uh uh get blob v2 and uh

the equivalent uh uh API uh uh when proposing a block um locally. That's pretty all for prism. And maybe I br is here from T. Yeah. uh t uh we have finally released validator custody. We have tested it internally uh started testing with devops. Now I have some issues but we are in in contact so the issues will be resolved. Uh also we have added new peras metrics with the help of ka and are working to resolve resolve some

bugs. That's all for tech I think. Uh, Lighthouse. Uh, Junior. Hey, thanks Ditri. Um, from Lighthouse side, um, we are still like fighting through sync issues. Um, we are focusing on getting um, our scoring um, like improving our scoring when we do sync. So that should um, that should improve um, Blackout's abilities to sync to head. Um at the moment the dev the devet 5 node um lighthouse g2 is still down.

Um I think we we've got to the bottom of the the root cause already. Uh we just haven't been able to merge the uh fixes yet to um to our main branch. So we're just going through reviews at the moment. We've got a bunch of sync improvements. So we're working on that. And um on the distributed publishing side of things, we've got the spec PR um created um and I see I've seen some um some comments on the PR already. Um feel like it's good to go. Maybe another round of review if anyone's interested.

Um but on the Lighthouse side, we we're also working on um making Lighthouse compliance with the complant with the spec. So at the moment um a full node doesn't do distributed publishing and um I think I've described in the discord as well it's um it kind of hurts the propagation of blobs because we we uh when we receive a a blob in the e um and we convert to data columns and we we we mark it as seen on gossip but if we don't forward it then um it kind of like stop the propagation if if future um data column arrive to to lighthouse So, um yeah, we're going to work on that fix this week. And other than that,

um yeah, but the custody is also going well. We still haven't um haven't had we don't have a implementation yet. I think it'll probably still be um a week or two away um because we have to do some refactoring. Yeah, that's all from my side. I will pass it on to Lostar. uh made good progress with the refactor and getting it unit tested uh kind of the same as uh man who mentioned and then uh we're spinning up a new contributor in order to help with the validator custody. Uh so thanks Derek uh if you're out there and uh came up with an interesting issue uh I think Jinder

brought it up on ACD about uh people make making sure we have peers uh that are listening to all of the subnetss to make sure that when you publish a block it actually gets uh received uh by somebody on the other end. Uh so we're uh looking at tuning how we manage our peer pool in order to ensure that uh we always have somebody listening um and uh we have an interesting opportunity or or uh idea in order to um how we manage uh walking uh disbed so that when we we need to find new peers we're readily able to connect

to people who are listening to the correct topics. Um, still probably uh uh like Jimmy mentioned a couple weeks out from validator custody, but we're making good progress getting the new contributor rolled in. And uh you know, things look pretty good on the DevNet, though. And uh I guess maybe Grand Dean, I don't know if I heard that name or actually or Nimbus I guess would be next. Uh I don't see Agnesh on the call so maybe we skip Nimbus and Grandine's

given an update on chat. Um so they started implementing offload proof computation with um edging get blocks v2. It's not working yet and they're investigating and they parallelize crypto functions and PS. So yeah, I think that would be all the Thanks Barry. Uh do you want to go into DevNet updates? Yeah, sure. Um so Paset 5 is still up and running. Um we or I'm sorry actually was there do we normally go through uh EL client updates or do we go right to up until now we actually haven't but now

that there are some E client teams here maybe we can give um give them a chance and then we can do nets sounds great. Um yeah I see u FLCRL from Nethermite so maybe you want to go first. Uh yep. Hello guys. Uh so uh we have a proof of concept uh branch uh with uh um API methods uh implemented and uh I I I guess it should work uh uh in part of uh P2P uh transaction propagation uh and we want to test it in multiclient uh environment. Uh so we'll be glad to join

any devet as a execution layer client. Uh yeah I guess that's it. Uh Roman from rest, you chime in. Uh yes. Hi. So we uh not much has changed since last week. Like we have all of the types implemented. Uh what are we really looking forward to is locking in the specs and making sure that uh none of the u message uh type decoding or the way we treat like any request types or network message types is fork dependent.

And once we logged in, uh I think it's like one or two days out to us being ready. For what it's worth, uh latest changes from branches uh look pretty good so far. And uh I will have a couple of questions regarding those changes, but but I guess we can go over them after the client updates. And I see Felix from Gth.

Yeah. So, not a big update from us. We have an open pull request about this uh uh encoding stuff, but we haven't really reviewed it. A lot of people are on vacation this week, so we actually kind of we haven't really actively worked on it. Okay. Um maybe I can do DevNet 5 updates then we can talk about the open PRs and then we can do DevNet 6 planning. Does that sound good? Sounds great.

Okay. Um so DevNet 5 we have been gradually increasing the number of malicious um peers. What they normally do is either withhold columns or delay when they publish stuff. Um the last update I think has been about a week ago. So roughly 30% of the network is acting malicious in some way or form. Um and this is in a network with already 60% full nodes and 40% super nodes. Um we are still waiting for some fixes. So there's the prism fix and ban is building and rolling it out now. So once that's back up then we'll continue increasing the number of malicious nodes. But as of now, I think um there's

nothing else major planned for testing on DevNet 5. So it's mainly helping teams get to a stage where we can talk about loading uh launching Devet 6. Excellent. Um in terms of the open PRs, uh how do we want to handle that? I I've got a list in the of spec related ones uh in the issue or Perry, what's the the best way or how do how do we typically proceed through that or Jimmy?

Yeah, I think we can just go um with the ones in the issue if those are up to date then that should be perfect. Yeah. Um so I shared the issue at the top of the chat. I'll drop it in again here. But really, this is starting with the We can come back to the EIPs, I guess, as part of a discussion. Um, in the specs, I've got the basic validator custody. Franchesco, is there anything regarding that that you would like? It looked, it appears to be done, but I just wanted to flag it and see if there's anything that you wanted to

highlight. Um, no, not really. I mean, I think there is a there's an open PR by Gabriel and or Alex um to maybe clarify some of the language, but I don't think it's Yeah, I I'm not sure it's uh worth this. Okay. Uh and then G11 tech the make validator custody static for beacon node run session uh appears to be ready for review with but uh there's still some open comments on it and they might not be on the call. Uh

that would be goodender. Oh gender. You want me to give a summary of my comments? Uh yeah. Or if you could just uh help help me understand the status of of that PR and what's the blocker with getting it merged. Ideally, uh which particular PR we talking about? Uh the make validator custody uh static for beacon node run. Oh yes. Uh so uh the stratus is that we uh I think uh Alex was going to look at it to see whether we would want to uh change uh

spec a bit uh in the sense to remove validator indices uh from uh uh from where we are saying get validator custody and uh replace it by supported stake as well as uh in is available data. I think I'm not sure whether it's in this PR or there is another PR where is available data also has valid data indices. So basically uh we so my recommendation was that even if we are not doing static or uh we want to do dynamic because of the UX we still clean up uh the specs and uh put supported stake over there and uh then obviously also have uh a way to specify how the

that the validators have to register with the beacon node and for beacon node and a small helper function. to uh calculate the supported stake uh uh for the beacon node from the validated indices which taken state as the input. Uh so I think uh uh I I I can basically rework the PR uh if basically we want to go in this direction. Uh so I think Alex wants to give some feedback on this or someone or anyone else who wants to chip in is welcome as well.

Uh, did anyone else have thoughts on this PR or feedback for the gender? Okay. Um, I'll drop the next one in the chat. So, this is uh this is picking up an issue that Stokes had filed. Uh, I don't know, Francesca, it's related to the validator custody specs. I'm not sure if Gabriel's on this call or not.

Um, but if there's anything that people feel we should discuss here, let me know. Okay, Alex just has to review it. Thank you. Uh, that was for the previous one. Um, I'm not sure about the complete validator specs. Is there any blocker for merging it in? Uh it I don't believe so. Yeah, it just looks like Alex is awaiting the review there. So I'll ping him and let him know that that's uh in the queue for him.

Yeah, I just wanted to hop on the call said u he just wanted your opinion and if everyone here agrees with the PR, we can move ahead and merge it. Great. Thanks, Gabriel. Um, this is a one from Justin. He's not on the call. Um, regarding the get blobs data column sidecar.

Um, this is actually just an open issue. So, I don't know if there's anyone that feels like this issue is something that needs to be picked up or what the status of it is. Apologies for the na naive questions. I'm just trying to get kind of a a level set understanding of where things are at now. So, I'm just um Yeah, no. Yeah, this is it's uh it's something that just came up in in Discord a few days ago. Um it's kind of just a small optimization um that uh yeah basically it's just saying uh you don't need to wait until you get

the block to to pull blobs from the GM pool. You can do it as soon as you get any data column sidecar. Um so yeah in principle if you get at least one column before you get the block uh it would be faster to to trigger the uh meool pull then. Um, so that yeah that that's all the issue is about. I think it's it's most likely worth like uh doing this like small spec changes to to kind of clarify that this is a possibility and that um it's encouraged uh behavior to do this and yeah I think that's that it's not it's not it's nothing like super um urgent but okay great contender I see you have your

hand up. Yeah, I mean the counterpoint is that we basically till we get the block there is time for E to sync the blobs, right? So if we for example get the first blob and start triing get blobs on the E and it might not get enough time to basically pull or get other blocks and basically serve get blobs more complete more fully. Uh so I think uh I mean in terms of my implementation judgment I think when we see a block then is then it's the best time to pull the blobs because otherwise uh uh I mean we can't delay any more than that. So uh or basically

when we when we get the block again we do a pool of get blobs at least that has to be done twice then so I I kind of disagree on this being better because um it's not just about your own uh custody being satisfied in which case yeah like you could do it at the latest possible moment and it wouldn't make a difference other than like the small latency of of actually getting it from the yellow uh but you're also like kind of meant to contribute to repropagating these this these columns like like reconstruct columns from the blobs you get from the L and um contribute to propagation. So I think it's it's really it would be much better to just do it as soon as possible um so that like you're kind of maximally

helpful to the to the network. So what we are saying is that we do get blobs and as soon as we see something and if we have all the uh blobs so basically we can construct all the columns then we receed the network with those columns. Is that the intent of the PR? Well that's not what the PR says but like the distributed blob uh publishing or or whatever it's called PR that we were discussing before from Jimmy. It does clarify the language to be that. I think maybe it already was that way. But anyway, like yeah, the the that's definitely part of the distributed blog publishing uh spec that you should be treating these things as you as if you got them from gossip and

repropagate them. And then uh yeah, what what Jimmy just linked. Um and then yeah, this issue from from Justin is just u like a small optimization that given that existing behavior uh makes it in principle better. Um, does that make sense? Like it just allows you to contribute in in this way um to to column gossip earlier than you might otherwise do if you just do it on on block instead of on any column. I mean can we also piggyback these API calls to trigger to basically indicate to the EL which uh blobs or which uh columns it should

aggressively pull. If it we are going towards distributed uh blob blob building and if we are not going towards distributed blob building then maybe it can aggressively try to pull the blobs it doesn't have from its E network. if it's I mean if if we want to piggyback on this to sort of uh strengthen the P2P network I didn't really catch that. Uh not sure what you mean by so when we do get blobs uh but we can't do it for all the commitments unless we have the block. Uh so I was no you can do it you have all the commitments when you get a column like a column contains all the

commitments. So as soon as you get a column you can get all of the Yeah. Right. Right. So basically then we can send all the commitments to the EL and maybe E can also start aggressively looking for it from the peers if there is some sort of a peer announcement uh for the blobs uh that the peers might have right. Yeah. Okay. Do you mean like yeah making like also like anite kind of u contribution to the to looking for it? Yeah. I think that that would also be be good.

Cool. Um, so yeah, sounds like maybe we're at a uh a compromise there or have selected a direction. Great. Um, the next one is in the execution specs. This is really just a a variable or a type change to variable. What was it? update the blob gas used. Um I think this was back propagating into forks beyond when blobs were introduced. Uh nothing major here.

I believe the community contributor is still working through it but uh just flagging it here in case anyone wants to discuss it. It's not. All right. Um and then the the next group these are I believe these were all uh related to sorry there's a test new one and then Francis had a collection of related um yeah here they are so there there's some tests that are related to execution tests and then there's three that are

all um Francis uh dropped all of these on Friday night I believe that were related to the cell proof related or sorry the cell proof calculation. Um so there are three different changes to the builder spec the execution API the beacon API and then he also opened two issues one in prism and one in G that were uh proof of concept implementations of this.

Um, so I don't know if we want to discuss these individually or as a as a larger effort. I think as a suggest make sense. Yeah, I would I would just going to say the same thing. Yeah, sorry. What was that? I was going to say exactly the same. Yeah, I think we should discuss the whole like self-p proof computation as a as a bucket. Okay, great. Um, is there someone that would like to start that? I don't know, Francesca or Pari. Oh,

sorry, Jimmy. Uh, do either of you have enough context to be able to speak to it? Yeah, I mean I can't speak to it in general, but I so my my senses here like really uh the most important thing that we can do on this call is for uh Yale clients to like talk to each other about this and and come up with like what they actually like about you know should there be this uh version by should there not be this version by about um yeah I mean all of the things that have been discussed in the um um in the sound proof respects uh this word like there's various like kind of small parts to this that people have had strong opinions about it seems so I

think yeah it would be more helpful for you guys to just like hash it out and and decide the way forward because we have all these like half uh done specs where we can't really like move forward without just decisions uh on some of these things. Uh yep. So if if anybody minds I can just start blasting my questions. So my first question uh is uh one of the PR specifies that on the fork boundary we should just drop all of the uh all of the blocked transactions with previous side cars. Uh are there do

other clients have any concerns to it or is it just fine we'll just start accepting uh new uh version of sidec cars and like there will be no blobs included for I don't know for the first and second blocks just after the fork. So I think it's kind of a client policy thing. So if you want to the main challenge is just that around a fork boundary you have to kind of either you have to implement either the conversion to the new proof type and if you do this then like it's just additional work in the client but I mean if you do it then you can keep the content but obviously the content of the pool becomes a bit

useless uh around the fork because these transactions can no longer really be included into the um into the block. I mean there's a bunch of alternative designs as well. For example, you could convert the proofs I guess after building the block. So my suggestion would be maybe to remove this language uh from the specification and just to list like here are a couple of things how client can implement this but but not necessarily commit to any of them.

Yeah, it's it's like it's kind of an open-ended thing anyway. That's the thing. There's a lot of options. Okay. So, uh like I would be fine the spec containing just it is up to the client implementation to decide what to do with the existing blobs on on the for my Yep. Sorry. Uh just small question. Uh what do you guys think about uh uh P2P? Uh do you plan to decline uh to reject uh transactions of certain proof type uh before or after work?

Before I can answer that that question, can somebody tell me like like this wrapper version change is it somehow similar to like TX envelope how we went from like not having a version by and then we can just look at this version bite and decide how to decode the the rest of the network message. So it's kind of a similar idea. Yes, it's similar but not uh the same. So critically uh with transaction types the payload of the transaction is an opaque uh data item. But in this particular change where this this version bite is proposed is just something that while you are traversing the RLP structure to

decode um the transaction and the proofs you will first encounter this bite and then you can make a decision there while you're decoding uh what to expect next. But it's not it's not it's not as it's not handled the same way as transaction types are. And I don't really anticipate that we're going to be creating uh a lot more mechanisms like this. This is really just a way for us to like know at the at the decoding time what is the intended type of this proof.

But but but for all intents and purposes like the the types the underlying types are the same, right? It just what the proofs mean is different. Yeah. Yeah. I mean from what from what I understand since the cell proofs is just an extension of I mean it's it uses kind of the same data items. It's not exactly the same because you're going to have a bit more like the proofs are a bit larger for the cell proofs, right? Do you have a couple like it's just like it changes the number of items and so on.

Yeah. Yeah. There's there's more of them than there would be for KG proofs. Yeah. So this point point to a solution uh that we um for the other question that was raised how to handle this in P2P. So one thing is that in the P2P we have a mechanism for communic like for block transactions when we relay them on the P2P they are kind of pull only. So we announce that this transaction exists and then it's up to the client to pull it and um the client before it has downloaded this particular blob transaction it um has knowledge about the transaction type and the size of the transaction. So I think because the cell

proofs are larger than the um No, I guess it doesn't work because of call data. I was just thinking that maybe there would be a way for us to somehow um know ahead of time if this was an old or new blob transaction and then we could maybe avoid even fetching it. But I guess that's not really possible. So basically what we're going to be seeing is if uh around the time of the fork there's going to be some peers that will still announce some old version blob transactions and I guess these can just be dropped then uh but if you receive new type before the fork I mean you can um collect some of

them to build uh the f the very first block of the work for example. Yeah, I would say it's something that it's a bit like the activation of for example 1559 or other types of like basically like with other transaction types where like even before the fork it's technically okay to accept these transactions and turn them put them into the pool. It's just they cannot be included in the block. So, so let's consider both types. Uh, okay. Uh, and do not u like pigs spear for I wouldn't do that. No, because because the type

because the like like you said these like sending new type of uh proofs is useful for building the block after the fork. However, maybe you should uh put a bit of a cut off there like if you shouldn't be like first of all in the implementation you shouldn't really start sending these transactions before the fork because yeah I mean it's basically they're they're not really going to be used but also um if someone spams the pool like weeks before the fork with these types of transactions is also kind of bad right so we want to we have to apply some nuance there to figure out what's the correct period for rejecting it.

Yes. And actually after uh the fork uh we can uh not request it. It's like uh because of cold data there is no 100% uh uh we can't be 100% sure but uh if transaction size will be lower than minimal possible uh transaction size with style proofs then uh we can be sure that it's old format and not requested and yes and if that data length is uh uh is high enough then we are requesting it and it It can be transaction with cell proofs or al transaction with some cold

data but it will be I think uh almost never like that. Yeah. I I also think so the the optimization you mentioned there is quite good. So I think there is this cut off size where we we can definitely know it's not going to be a blob transaction with cell proofs and these we can just ignore. So the only ones that we would possibly fetch is are the ones that that that have a lot of call data. And then it's the question just if it's if it's a problem that we have to fetch these transactions. And I mean at some point after the fork like I don't know a month or couple weeks after we're going to have another release out of the clients anyways where we will just start rejecting these transactions I guess. So yeah, like the whole idea with this with

this version bite is just to make sure that we can handle these transactions outside of the fork schedule, but it's up to the clients to decide how how to handle them. Yeah. So uh uh on our side again any change uh that makes the decoding nonfork dependent is great and rapper version seems to accomplish just that. Todd is supportive.

All right. Uh so yeah, I'm happy to summarize all this and it seems like this is a pretty meaty isolated topic and I you know communicate the feedback back to Francis and uh let him know if there's any you know revisions that he should be making. But um sounds like this has been a pretty productive uh conversation. Any final thoughts on this topic before we move on? Great. Um, and then within the list of issues, there was one last one.

Kacia, I this, you know, again, just trying to like uh level set my understanding of where we're at with things. I believe this one had been open for a while. The beacon metrics. Uh, I don't know if there I see you on the call. I don't know if there's anything specific that you want to flag for this PR or if it's just uh sort of uh moving along as the project progresses. Yeah, I will. Um well, I am currently working on this PR. It is open because I am implementing these metrics in every client. So, it just takes time and uh we have also updates on the metrics. So uh the next update I think will be about

the validator custody um and backpill process. So we're going to start from tech with this metric and I will add it later. We'll see the result and I will add it later. So it is open because it's in progress. That is why it's open. We have also a peer dashboard um on the it pandops uh dashboards. U so yeah we have uh some clients on it like lighthouse, prism, tech, grand and I hope other clients will also join this dashboard.

Great. Thanks so much. Um, Pari going back to uh and Barnabas some of the we had this async document coming together Google doc last week uh talking about status of testing testing tooling. Um it I put a a table in the PR the agenda issue if there's anything that you want to speak to about status of new transaction format for spammer uh TX spammer pupool P2P testing tool or checkpoints.

Yeah. So I think for the spammer the last agreement was that the transaction format for the one submitting the transaction doesn't change. So that should be a closed issue. Um, for the peer-to-peer testing tool, we are still working on it. I think in the there's a telegram group where uh Sam is actively discussing what approach to use, but I don't think it would be ready in the next week at least. Um, and for checkpoint Z, we're mainly waiting for uh specs to freeze and once specs are freeze, then we have to update one or two dependencies and once the dependencies are updated, then we have checkpoint ZD as well. Um, but in the meantime, we should still be able to do

checkpoint sync uh via a beacon node. And if you need the password to access a beacon node API, just DM us and we can get that to you. Great. Thank you. Uh, Minhai, I see you on here from uh, Sunnyside. I don't know if there's anything that you want to mention about the root cause analysis and the I guess the max blob testing that you've been doing. I just dropped a link to your latest uh notion in the chat. Yeah. Can you guys hear me? Yes, we can. Yeah. Yeah. I am currently working on a news report that we will be completing in a few hours after the

call. But the gist of it is that uh we've been testing. So we noticed that the maximum block count that we noticed from last testing was about 10 blobs per block. So we were root causing it and in doing so we tried different combinations of uh clients. So when using only lighthouse and prism we noticed that the block counts could increase to uh 20 blocks per block. So then uh I also did the testing with other clients like uh Grand uh Loadar Teu and so on.

So that in those tests I found that some some specific uh consensus clients yielded in a lower blob count compared to the like the baseline test I did with Lighthouse and Prison. So uh that will be uh in the next report. And also when working with some of the execution clients the issue was that the TX poolool had was hitting some limits. So the ys were dropping blobs from the tx pool one after hitting a specific number of blobs. So I had to either increase the the expose size through the

configuration and I think this is something that we might want to consider when we decide to increase uh block blob limits in pure. Great. Thanks. Um and sorry did you mention when you thought you were going to publish this? Uh in a few hours. In a few. Okay. Great. Yeah. Um, and then I just dropped a image in the chat just like sort of a random scattering of testing related uh status updates that I'd seen. I don't know if pandops you or sunny side or uh steel I'm still trying to uh understand how to bucket sort all

these but yeah if anyone wants to pro provide updates on these I can give an update. Um so deploy at least one node with withhold column. We've already done this. We have roughly 20 30% of the network doing this. So it's been successful and so far we're able to withstand the flag itself or the chaos the flag causes. Um verify all nodes are properly configured. We've had all the client teams look through that and that is also working. Um premine node keys. Um maybe once I'm done Barnabas can give an update on that. Um him and Rafael have been looking into this. Um implement flag to advertise false uh CGC count. I don't think we have this right

now. Um I know im uh limitation issue. So that's the um database reworking that Manu was talking about earlier. So I would consider this done and um batch publishing column optimization. Last I know there's a go as well as if I'm not wrong a rust implementation. So I'm not sure if every client will be able to do this but yeah maybe the client teams would be able to talk about this. Yeah I can mention the premine node keys to ensure coverage across all column subnets uh is done. So from our side uh we have in mind all the keys that we need to create a 16 node network uh with each node uh covering eight subnets and

this will enable us to test if every single client is able to serve all subnets. Uh last week I've been working on uh setting these uh custom uh not keys in into the different client teams and uh right now we have it working for every single client. So we can do some local testing and figure out if everything is working as expected and then the next step of this would be uh to ensure validator custody is also working as expected. Great. Thanks Barners and Pari. Um there's a handful of other items on the agenda. Just one small ask

um if anyone wants to comment on the issue. We're trying to get a uh you know sort of a a single uh to-do list or source of truth of uh status for each of the clients pointed at uh just so that people have visibility across uh the different clients for setting up devets and testing and things of that nature. Um so yeah, I I I also went through the discord and I'm just trying to get a sense of Google's actively contributing. So yeah, feel free to comment on the issue. uh link to uh maybe an over a top level governing issue that or PR that kind of gives us a sense of your overall progress and how you're moving through

things, how how you're tracking things. Um and that would be very much appreciated. Uh there was one I guess fairly substantial discussion item uh which this is just kind of speaking to the overall schedule. Um, we can come back to this later, but I just wanted to get people thinking about it and then open the floor up for any pressing, more timely discussion items. Yeah, maybe I can talk about um DevNet 6 a bit. So, we normally maintain the spec

document. I've shared this earlier. Um, and thanks for collating all the PRs already. So, based on the decisions we have today, we'll uh pin a commit hash in the PR. Maybe we wait another day for whatever can be merged to get merged already. Um, and then we should have a proper fixed spec for Paset6. And the moment that there is an EL and CL able to interrupt, there will be a ktosis config there. And then we can um expand that config as we get closer to a deadnet. Does about like two week twoish weeks from now sound realistic to client

teams? I feel like if we set a deadline, it might help us at least get a sense of when we want to ship this. Is validator custody going to be required for DevNet 6? I think that's up for discussion. Ideally, yes. There's no reason not to like there's no reason for us to launch the net 6 without it in my opinion. So like that's one of the key features that we've been pushing for quite a while. The thing is validator custody is backwards compatible. So we can still launch it without and then add it as and when clients have it ready.

Uh, sorry. Yeah, Jimmy had his hand up. Do you want to go? Yeah. Um, so just to your point, um, it's validated custody is is backward compatible, so it could be, um, deployed anytime. Um, but I feel like the sooner we have a Dev Net6 running with um, like CL and E um, interrupt testing started, I think that would be ideal. Um if we have an implementation of E that is that is ready for um the self proof computation then um then I feel like we could start DevNet 6 and and all the features can can be deployed anytime after we launch the DevNet.

Yeah, that sounds good. Um maybe that should be the priority order. So if your client team has 10 issues, try and focus the ones that are more selfproof computation oriented first and then push the validator custody stuff towards the end so that we can still deploy the network. We can have it live. We can do the testing and then we can update validator custody as and when client teams are ready for it. Yeah, that feels like the fastest path forward. Great.

Um, and then in the chat there's some uh EIP related items. I'm aware of the like sort of primary PIDOS uh EIP uh ETH dreamer. Jimmy, you flagged for me that this was the draft version of the BO uh hardforks that is up for debate in terms of being included in the Fusaka update or not. Um, and then there's two others that Francis noted for me that are related to hardware and bandwidth and max blob flags. Um it it I'm just trying to get a a better understanding of the likelihood

of these three having uh shipping with Peros and if people could speak to the sort of uh impact that that would have on design decisions and uh ability to scale blobs with that hard fork. Yeah. Um yeah, I can talk about the um the BPO hard fork. Um so um Mark um if from from from our team has um created this PR and it's it's it like the reason why wouldn't have included is um it's like even if we if we even if we're able to get to 50 plus blobs in the test net

um it's still quite uncomfortable and risky to actually deploy this onto a mainet. So, so ideally we start with something that that we think would be um safe enough and then if we have the BP hard fork then we could schedule um a BP hard like pretty much immediately after the um after the um the um after the bullet fork um and I think Sam also mentioned that with that works an immediate quick hard fork that's we always talk about but it It it never works like that. And this would be sort of like the gas limit change, right? So we did have technically a gas limit change in between the last fork

and now. Um so the idea is that it would be in a similar vein. Yeah, I guess the so the main difference with the PB hard fork is it doesn't require um most of the boiler plate of a regular hard fork. Um I I think if I understood correctly like most of the things like the the uh the beacon block data structure and the gossip topic um those remain unchanged. The only thing that change is the is the block count. So everything else remains unchanged. So it should be a pretty small change for clients and should be able to to roll out pretty quickly. Yeah.

I would not compare uh gas limit change to a blob change because gas limit change can be literally rolled out by anyone at any time uh independent of any uh coordination and the blob change would require every single node to change their uh change their client to the latest version and to have a fork at a specific block height. So it's I I don't think this is compare comparable at all. Agreed. But the main reason why forks have been relatively slow at least on the CL side is they involve a complete new data structure and so on.

And BO forks is fundamentally different in that way. It makes it easier. So you could imagine we even just have a bo fork scheduled um with the pas fork saying that in 3 months increase it by x and you basically cancel it if it doesn't happen and that way you keep the same data structure you promise some it yeah it is growth only well I don't think it has to be um but yeah you will have some amount of coordination effort um the main reason it's being proposed is it feels a lot safer to do this as opposed to doing a massive bump at the pure das fork. So you could do a moderate bump, analyze how pure das is going on mainet and then do a bigger

bump. Yeah, if I remember correctly, the roadmap discussion that we had in person um with some of the EF people was I guess the hard fork with no this is speculation and I'll just kind of repeat it just you know some people are aware of this but do the hard fork with no increase in blobs just to observe all the changes and then two weeks later it was I believe to 2x the blobs, then two months later, 2x the blobs, two months later, 2x the blobs, um, and be able to observe it. If I'm understanding

correctly, the BO hard fork is a is a requirement to be able to like proceed in that manner. Is that correct? Has any of the client actually uh taken a look like how easy it would be to do this incremental increase? Yeah. Um, E3 has an implementation in Lighthouse already. But I think the hard part is we would need more CL clients to look at it cuz an implementation like every client team implements it differently. I think especially Nimbus has a lot of intricacies. So, it would be good to have them look at it and if this would actually work for them or not. Um, and also to Franchesco's point,

I do think pre-sheduling linear increase could also work. I think there's roughly like two or three different ways we could imagine getting to a higher blob count um safer. I think BO is going to be one option that exists. Just to be clear, we would also need to update the E for this, right? Because the also needs to know about the max and target. Yes. So, it's not really a soft fork after all.

We Yeah, I think it would still have to be a hard fork, but yeah, we have to check. I think increasing might be a soft fork. Decreasing might be a hard fork. I no but like the it comes all back to the gas calculation on the side like right now we have to hardcode the blob numbers in the yeah config in the genesis JSON file. So I would assume that we would also have to even if you do a linear change we would have to update the side as well. Yeah, I think the linear change might be harder, but I mean, yeah, we I think we have to look at it. For BPU, it's relatively

straightforward. It's essentially a new fork. Um, except it doesn't have all the complic complications of a fork. I think for a linear increase, it might be a bigger discussion. We don't really have a direct way on how this needs to be done and it would need implementations. Whereas with a BPO fork, it's literally define a new fork, call it BPO1, have a new blob schedule on the EL side, have a new config value on the CL side, and then ship it plus all the coordination, plus all the test nets.

Well, that's the benefit. There shouldn't have to be any test nets. It's all about analysis on mainet. So you ship pas on mainet and same way how we came up with the logic for 6ix9ine as safe on mainet like you can ship 6ix9ine on a devet it doesn't really tell you much cuz the way 6ix9ine behaves on a devet is different from mainet but we should still roll all the features out on a test net first which requires additional coordination so that's my point in one devet it's not as much as a full fork Okay, great.

Um, well, I think we're at time. Uh, thanks everyone for bearing with me today. This has really helped me kind of get my bearings and I will get the notes prepared and circulated here in the near future. And then I will see you next week and async