Post-Quantum Interop Call #41

Transcript

Shariq Naiyer: Where do you get this link from? Cause the link… the link in my calendar was in this link, I don't know that.

Parthasarathy Ramanujam: I have a local copy of the original invite, which had this.

Shariq Naiyer: Interesting, interesting. Oh, this is last week's link, like, as you can see from the title. I think they generated a link every week.

KaydenML: Yeah, the one, Thomas posted yesterday just leads to this, call to…

Gajinder Singh: So, has someone joined? How do we start this?

Shariq Naiyer: Yeah, I don't think he has joined, but I think he wants to join.

Thomas Coratger: Hello, everyone. Just waiting, other people are joining. I guess this is it. Okay, since everyone is there, like… Data starts the call. So, today, what we want to talk about is mostly, like, So first, we will do the usual update. On all of the clients, and then I will talk about, briefly, lean spec simplification and stabilization. And once that is done, we can talk about the problems that you currently have with DevNets, in terms of stability, scaling, all of the bugs that you encountered into the past week. My feeling with the recent message that I have seen. Is that, you have, like, much better stability, and you have, like, now better hardware stuff, so things are going better, but we can talk about this, so let us start first with the client updates.

With Zeam.

Gajinder Singh: Thanks, Thomas. So on DevNet 5 front, we were able to implement it, but, I think we have an issue over there, again, with multi-message aggregation taking too much time for the block proposal, which So, unless we drastically improve that, which basically changes… The entire game of how a block proposal might work, or… Or basically bumping up the specs for the block proposal, which we don't want, but I think, for now, we can part this conversation for a bit and focus a bit on DevNet4, where we are also trying to scale validators as well as subnets over there, and Partha did a lot of runs Try to get that working with 64 validators, 8 subnets, with each client as an aggregator on the subnet.

And, so he has been able to make some progress in the sense that there… so we did actually have a finalization event, which was not happening before. And all the validators are running in their… all the validators, which are not aggregators, they are basically running in shared machines, while aggregators are running in the bigger machines, and I still think that We still need to work a bit more on… the aggregator… aggregation heuristics, because we would want aggregate… aggregators to prioritize, the aggregates that is matching with the local view, which is something that we discussed on Zeam, that, how do we want to prioritize The aggregators, how we want to prioritize the aggregation so that We can, we can get the good aggregates out fast.

And, even with that, so there is, I think, still… Some work to be done, and some, Some, Optimizations to be done, and that is what we are focusing upon. On, on the front of Shadow Sims, we… are working with Camille on that, and we were able to… run Zeam and Qlean, in Shadow SIM along, using Lean Quick Start. So, Camille is, I'm sure Camille has done some runs of Shadow. using that, and I'm sure he will update us with his findings on it. But over time, I think we will, need to… again, allocate bigger server resource to run the Shadow SIM, so that we can have larger number of nodes and simulate

The latency from different regions. But even then, you can only do a small network, and… Then… You would still need to… run a bigger network. to make sure… We can basically get production ready, and that is something We'll talk about when we come to that point right now. We definitely have to cross quite a large number of hurdles before we reach that point. So, that is update, from our site.

Thomas Coratger: Great, thanks a lot. So, if I understood correctly from your summary, you mean, like, the DevNet 5 stuff are not going well for you due to the time it takes to aggregate the Type 1 or Type 2 messages, right?

Gajinder Singh: Yeah, because proposer has to do this work, and right now, I believe multi-message Aggregation over 9… payloads, right? Because there are 8, max attestations and 1 block signature. That is taking 4 or 5 seconds. Is my understanding. And even that basically needs to come to Like, less than, Half a second, right? So… There is a whole lot of gap that needs to be covered in that area, and this is even before we're a proposer needs to do its own compaction. For example, if it got multiple, aggregates, which it will get with the same registration data, it will need to compact those as well. So this does not include the time. And assuming those are limited in number.

and… That might… that… that time itself, from what I believe is also 0.3, 0.4 second on a… good machine. So even that time basically needs to come to, at least half of that, so that So that with all the compaction that a validator needs to do, it… it is able to propose the block. And and if it's not able to do, then obviously we will have to remove some of the compaction from the validator that it is trying to do for… For attestation aggregates, and basically say that we come to a previous, To the previous spec, where you can have same attestation data multiple times in the block, because if the validator is not able to compact them in time, because it also has to do a multi-message aggregation, then

Basically, we say that whatever aggregates a validator has, they choose it greedily so that they best cover, the entire, validator set. Or the best move, the justification. forward, right? So this is our algorithm that we choose at stations that move the justification and finalization forward. So… so we will basically have to come to this particular design, where we will say that, okay, validators will not do or do minimum compaction, and will only compact For, signature across, whatever it's packing as max registration aggregates and the block signature. So, at max 9… 9 compactions on a multi-message level, it will do. So we will even… so if… if we are not able to Speed up the basic compaction process.

then we will basically have to come to a different design. And, and the thing to… to basically be considerate… to have consideration about this compaction process is that, so… so… So we can have some constraints on what is the depth, or… so what is the fan out. That a validator, a proposer, is trying to compact when it's trying to basically compact, different aggregates of, say, attestation data. So we can… we can have some sort of, Constraint on that, but with that constraint, we basically need to make sure that It can happen in less than… 200 millisecond, and then give some 200 milliseconds again for… For multi-aggregate, multi-message aggregation. So, I think… there is a whole lot that needs to be covered in terms of performance by the underlying libraries, and then when we run the underlying libraries, each of the clients will have to basically also do their own optimizations. For example, Zeam is facing, some…

FFI overhead, which is… which we are trying to figure out and trying to optimize. So that… that might also need to be figured out by the client. So we… we need to have A particular comfortable, space of… Having this competition done in, in a very small amount of time.

Thomas Coratger: I think… thanks a lot for the detailed stuff. Let's just go to the next client, Ream?

Shariq Naiyer: Hi guys. So, we've been working on implementing the DevNet5 spec, and we've been working on debugging and pushing fixes for the issues Bata has brought up in running the DevNets. On the… one of them was our client was crashing, and it was because the minimum number of signers needed to be, more than at least 3, otherwise, there was an internal crash from the multi-sec. So currently, I will be demoing this further, but currently, setting the minimum number of sign as required to 3 and skipping other attesta- skipping other aggregations, has seemed to work, temporarily. On the Hives front, On the high front, we have added tests for multiple subnets. Client drop across multiple subnets, we have added tests for aggregation on all clients. And…

Oh, currently, the Hive is using the DevNet 5, the latest, lean spec, commit, although we're working on splitting it out into DevNet 4 and DevNet 5. we would like to propose that we continue the DevNets on DevNet 4, as that's a lot more stable, till we are a lot more stable, before moving to DevNet 5. That's all from our…

Thomas Coratger: Thanks a lot. Emil, you wanted to say something?

T. Wambsgans: Yeah, if, if there are some edge cases, typically, like, you need at least 3 centers or stuff like this, please let me know, it's probably easy to fix, yes, and it, so it's likely a breaking change. But, we can say, like, together we have a breaking change at this date for DevNet 5. And maybe also, if we do breaking change. we'll be able to integrate all the latest optimizations from the main branch, so it will likely make the DevNet 5 faster. So, probably something we want to do, maybe, like, in the coming days. And so please let me know if there are these little edge cases that are annoying, we can likely fix them and put the performance upgrades at the same time.

Shariq Naiyer: Okay, I'll reach out to you regarding the issue we were having, in a… after the call.

T. Wambsgans: Perfect.

Thomas Coratger: Great. Qlean? Okay, London?

Mihir Faujdar: Yep, hi Thomas. Hi, everyone. So, basically since last time I gave an update, so basically, in Lantern, the new LPPDP library has been integrated. As well as the new SSZ library, and we have seen improvements for Hive, especially the gossip tests, they're all passing. And as well as, after we added the Hive integration, most of the tests are passing. So, basically now, what we're focusing on is, improving client interop tests for, multiple subnets. That's where the issues are facing, and have recently, implemented support for DevNet 5. So, like, the goal is to fix the client interop tests and, do some testing for DevNet 5 image. Thanks.

Thomas Coratger: Thanks a lot. It's Lambda.

Pablo Deymo: Yes, hello. We found, that… we found out that the station processing was falling back. On the latest certified checkpoint, when another certified was processed, So, we fix… We fixed that, and, we… open the PR in green spec that was merged. And we put some test vectors for that. We'll really publish a fixed image. And such as organis to the same. We merged an improvement to the blog building algorithm in Lambda. It shows faster finalization. Or non-finality, when there are multiple attestations to choose from. Since it better prioritize, which ones to include. We are looking… Into porting this to a spec, or documenting this to better share with other clients?

On the other side, we implemented a website spec PR changes. And started running DevNets in our server. We are finalizing a stable in a single subnet DevNet, but block billing almost consistently takes more than an interval, as I think Rajinder said. We're not debugging and looking for performance improvement on blog building. Regarding, what we mentioned about execution layer integration, we are running… we are running a DevNet with egg nodes. each node with an F lambda on, And one executioner node, and it's analyzing one. Oh, that's from our side.

Thomas Coratger: Thanks, thanks a lot for… for your summary and, like, for the PR on LeanSpec. This is super valuable to discover such, like, tricky… Biggs. Let's go to Gim.

Bankat | gean: Hi everyone. So, quick update from Gean's side of things. On DevNet 4. The consensus score here is currently stable. That is our state transition. fork choice justification finalization are all aligned with the inspectors vectors at our current pin. We completed an async aggregation refactor aimed at improving, aggregation performance, and we are now seeing aggregation latency within the same range as other clients, too. And then on lean spec part side of things, we stayed closely and aligned with, the upstream. all DevNet 4 bug fixes merged into the spec so far have corresponding changes and updates on Gean, including the most recent change, which was the latest justified fix that was flagged by IT Lambda team. Which we also putt alongside, a goal regression test, too.

While continuing to stabilize and optimize DevNetwork, we are also making progress towards DevNet 5 readiness. So, the current plan is also to target, DevNet 5 interop alongside, Ethlambda, Zeam, and other clients that are currently implementing, DevNet 5. So, that's just the updates on Geanside. Thank you.

Thomas Coratger: Thanks a lot. Lighthouse? Grundine.

Mercy Boma Naps-Nkari: Hello, good afternoon. So, we've been running, our own local, DevNet to figure out, cases of, client, the OM issue, but so far, we've not been able to still replicate the issue Pata was, referring to. But also, we are trying to, later today, run with 4 subnets. And I will keep an update on that, and then… we have, our DevN5 implemented, but, the same issue at blog building, and I'm still trying to debug and also figure out how to make better optimization. Thank you.

Thomas Coratger: Great, thanks a lot, and Lynn. Then, did I miss anyone? Okay, cool. So, let me talk briefly, like, about what we are doing on the spec side. So I have worked, like, pretty much, like, full-time on the spec for the past week. We have, like, some internal rounds of discussion internally at DEF on how we can improve the spec, make it more readable, make it simpler, all of this. So that is basically what I am doing right now. I am trying to go 5x5, from the bottom to the top, so I started with the type stuff. in order to simplify everything, I checked all of the comments, basically, and I… at least I tried. to check all of the lines in order to see if we have, like, potential bugs, stuff that we didn't catch, or anything like this. I also added, like, a bunch of tests for the types only for now, so that is the SSD types.

the UIN types, the slot types, and all of this stuff. So I am pretty happy right now with the state of the types folder, and so now I am going to the top. So today, I have worked on the XMSS implementation to make it as simple and clear as possible, so I have tried to minimize the number of helpers, for example, the number of additional files that are useless, a number of abstractions that are useless as well. So I have tried to simplify basically everything, and also to add test coverage to prevent bugs in the future. So that is something that I will continue to work on in the coming days. Of course, on the side, we are still maintaining, like, Plonky3,

With a couple of PRs that has been merged, but not something, necessarily, like, relevant to you. We, like, had also, a conference last week in Zurich, to present, to, like, more institutionals and bankers, what we are doing in the lean content side of things. So yeah, that is basically the update that we have. Maybe, Emil, you want to talk about LeanVM and the current status on what you're doing, and then we can talk about DevNet 4 stability, stuff, and everything like this. So, please, Emily, feel free to update.

T. Wambsgans: Yeah, on my side, I've been mainly working on… specification, mainly improving a lot the PDF documentation for LeanVM, and yesterday, finally, we have a Python implementation of the verifier. It's, when you exclude Poseidon and finite field arithmetic. It's 1,200 lines of pure Python, which is cool. Maybe below 1,000 soon. And the next step is to improve the documentation of the ZKDSL, like the high-level ZKDSL, and then maybe try to formally verify the error constraints, at least for the execution table, stuff like this. That's it for me, thanks.

Thomas Coratger: So yeah, maybe then we can talk about, like, the… stuff that we encountered, like, with DevNet 4. So, to summarize briefly, my understanding is that we are not, like, running DevNet 5 yet, because we are still debugging DevNet 4. Last week, we talked about, like, possible, like, the fact that the aggregation was super slow, possibly due to the fact that we are using, small machines. With very small capacities. Pretty far from what we are benchmarking on right now, using… with LeanVM. So, as far as I understood, you worked on this pretty evilly this week, and I have seen, like, a message, I think it was yesterday, when you finally have stability in a four-suspect DevNets.

So, maybe, Partha, like, you want to comment on this? Like, the… the stuff that has been sold, like, I am also curious on… on understanding what were the blockers for this. Was it, like, client bugs, or… Maybe hardware requirements, or anything?

Parthasarathy Ramanujam: Yeah, it was a combination of, everything, hey, everyone. So, as you described, I've been running these multiple subnet devnets. The last run that I ran yesterday was a 4-subnet devnet, and today I ran an 8, subnet version of it. So, there have been several updates by various clients over the past week, which has actually helped in improving the… stability of the network. The main, issue right now, even after we have narrowed down on the configuration for aggregators, is that, the mean, aggregation time for, the clients that I've noticed is between 1.2 to, 2.8 seconds for aggregation. I don't yet have the aggregation metrics for REAM, because there's some issue there. Lantern is probably the quickest. It completes, aggregation in 0.3 to 0.4.

seconds, mostly. But, I need to add that the way these metrics are captured differs across each client implementation. So, the objective I want in this meeting is probably agree to a common way in which we measure. Ethlambda measures only the aggregation part of it. Zeam, does it by measuring the number of workers, which are triggered, when, an FFI is called to consume the the Lean Multisig library, and Lantern has a different way to do it. So if we can agree on what we ought to measure, that would be great. The other, we have to target P95, for all clients for aggregation to probably come within one second, because that would help us, achieve better stability. Given that a slot interval is just 0.8 seconds, if we can complete aggregation 95 percentile, within 1 second, then maybe

the network would be more stable. At the moment, it is not. P95, I believe, is between 3 to 4 seconds, and it differs for each client. So…

Thomas Coratger: I have maybe, like, like, quick question, so… Yeah, you are running 4 DevNets, 4 subnets, sorry, right?

Parthasarathy Ramanujam: No, 8, 8, 8 subnets. right now.

Thomas Coratger: 8 subnets, okay. Yeah. And so, per subnet, you have one aggregator? One aggregator?

Parthasarathy Ramanujam: One per subnet, 1 aggregator, 8 clients per subnet. 8 nodes per subnet, and it's a homogeneous subnet, so a given subnet will have all clients of the same type.

Thomas Coratger: Okay. That's clear. And when you said, like. 1.2 to 2.8 seconds for the aggregation. Is it, like, the entire pipeline? So, going from all of the row signatures to the… single proof? Is it what you measure there, or something specific?

Parthasarathy Ramanujam: Thanks, buddy. Yep. I mean, I just measure the aggregation time across multiple slots, and then take a mean out of it. So, each client has implemented it slightly differently. So, from what I've seen, some cases when the number of raw signatures to aggregate is low, aggregation completes within 0.5 to 0.6 seconds. In few cases, they take a longer time. There could be various reasons for it, maybe because the process that was supposed to aggregate was holding for a mutex on the data. I don't know. I mean, this is, again, my experience is based on Zeam, so I'm giving you a perspective there, but it's for clients to… Probably debug us further and detect what's causing the delay.

But…

Thomas Coratger: At least for Zeam, like, the time you measure for aggregation, let us say 2 seconds, for example, this is the time between you have all of the raw XMSA signature up to the time where you have the final proof that.

Parthasarathy Ramanujam: Exactly, that's critical. That's correct.

Gajinder Singh: So, I think what is more useful would be… so, from the start of the interval. Till what time you are able to produce an aggregate. So, I.

Parthasarathy Ramanujam: the X.

Gajinder Singh: That will include everything, whatever a client needs to do, and could be a more holistic way to measure it, but of course, clients can break down on their level that, okay, this is what each of their segments are consuming and can use it for the optimizations. So that is something that can be done. Also.

Thomas Coratger: I think…

Gajinder Singh: We also need to, figure out How many, what is the attestation coverage by each of… these clients, for example, you know, are you packing all the 8 attestations, or are you packing more? Or you should not be packing more, so that is something to be looked at if there is an 8-node subnet. And you are packing more. And if you are packing less, then why you are packing less? So… so we basically need to make sure all of these things, and we then also need to make sure, we also need to figure out how many attestation data are you aggregating, right? Because each attestation data will take its own time. So… so how clients are doing it needs to…

So they also need to indicate all these numbers, that what is it actually they are doing when they are doing an aggregation? So if, for example, you are doing an aggregation and you are just processing one registration data. And saying that this is, my time to aggregate, then that is not same across every run, because other… Others might be also, producing aggregates for other attestration data. So we basically need to Standardize this a bit across all clients, so we have So we can figure out what's happening in the network.

Parthasarathy Ramanujam: Yeah, so that brings me to the other question that, Toma, you raised earlier. How do you want to deal with these metrics definitions in the spec? Because you said that you were simplifying the spec, and you want to know whether these are debug-specific metrics, but it'll really help us stabilize the DevNet, so In my opinion, it'll help if we could merge it to the spec, and we could use the spec as the source of truth, and all clients can ensure we are conformant to what is defined there.

Thomas Coratger: I agree. I have this PR in mind, like, yeah, this is definitely on my merge list. Right now, since I am simplifying a bit all of this. I would probably wait a bit, like, a couple of days, but yeah, I would do that. So, if I am correct also, like, you have 8 devnets with 8 nodes per… 8 subnets with 8 nodes per subnet, that means 64 total, validators, right?

Parthasarathy Ramanujam: That's correct, yeah.

Thomas Coratger: And, do you know right now if… What would be the limit to scale? For example, 100, 200, 500.

Parthasarathy Ramanujam: So, that's more a question on the budget and more for Larisclas and Jihoun, but I think if we achieve good stability where the finality continues to progress stably in this setup, then we can always scale to maybe 128 or 256 as well. that wouldn't be a problem. But we should probably do this in DevNet 4 before we move to DevNet 5, because the hardware requirements for DevNet 5 would be a lot more than what we are doing here. And just to add on to that, one thing that I've noticed on clients such as Grandine, Ream. Qlean as well, that, I mean, we have for the regular validator nodes, we run it on a shared server, so there's a per container memory, cap of 3.5GB.

And, that is causing quite a few clients to occasionally crash with out of memory. If the clients can work to stabilize that, that'll be really helpful, in, probably, stabilizing the current DevNet setup. I know it's not a hard or lean spec-specific requirement, but given the constraints we have. on the, DevNet setup, it would be really helpful, I appreciate if clients can work towards this. So…

Thomas Coratger: I… I am not… sure that, like, just for your question of adding more validators before we move to DevNet 5, I am not sure that we should do this, because the setup that we have right now in DevNet 4 is pretty far from reality. Because we don't publish, like, one proof per block on-chain, so… We are still missing one part of the aggregation pipeline in total. we are still in a mock configuration, let us say, while the DevNet 5 is much more realistic, because at the end, we end up with a single proof per block on-chain, so that is much closer to what we want at the end, so maybe…

Gajinder Singh: I think…

Thomas Coratger: An alternative idea would be to wait for DevNet 5-like stuff to stabilize before doing a broader validator set.

Gajinder Singh: Right, I think, just to echo what Camila has also written in chat, so those… so we need to come up with an acceptance criteria for DevNet, right? And say that, no, we accept it. And to have an acceptance criteria, we also need to have the hardware criteria, where we say that the validators. Will basically run on this particular hardware, and the aggregator will run on that particular hardware, because if you just bump up aggregator hardware, then probably you can do things in less amount of time. So we need to come up with acceptance criteria based on some particular hardware where we target it, and with regard to DevNet 5, So we still have a few steps to go over there, because seems like even multi-message aggregation is not at that particular point where you can have a proposer, you can expect a proposer to

just aggregate 9 payloads, right? So, we basically need to get it To get it… to get that, bench under… 0.4 second, I think that makes sense, to basically be able to say that, okay, we are in a position to run DevNet 5, and, that obviously can't be on a node which is basically, some 24 gigs and, some 16 or 30 cores, right? So… so… so we need to fix, what the… Validators, hardware. requirement is for acceptance of these DevNets, and once we do that, I think then we can start to… have an acceptance criteria for each of these dev labs.

Thomas Coratger: Yeah, then, like, just, I'm saying something, and then Mercy, like, you can add, like, your comment. I have no, like, specific target about hardware requirement to say, like, okay, we should do this or this, but the only things that I know is that all of the lean VM benchmarks that Emil is doing right now are, like, on an M4 Max. So, my guess…

T. Wambsgans: And just also on, the benchmarks on, on, Hetzner server, AX42U, that costs 50 bucks a month. And the performance is, like, half the performance of, M4Max.

Thomas Coratger: Yeah. My guess would be that we should have, like, a machine pretty close to this if we want, at least to have, like, some numbers that are satisfying, I would say.

Parthasarathy Ramanujam: So, the machine configuration that Emil shared, that has a one-time setup fee of 227 euros, so… I might have to see if there's something else that we could acquire which would give us similar performance, but the readily available ones are what I already shared with you, Emil. If we can probably benchmark on that, it'll be easier, because, Hetzner would only provide these servers for, spinning up. It's very difficult to acquire the specific machine that you said, and, I don't think we would be in a position to pay a one-time setup fee multiple times for all of those machines.

T. Wambsgans: Yeah, also, I think the DevNet4 is, like, using Multisig version, like, 2 months ago, so probably if we do an incoming braking change, I expect the performance to… to be boosted quite a lot, I hope.

Parthasarathy Ramanujam: Okay, yeah, that sounds cool.

T. Wambsgans: And also, there is something… like, for the clients in Rust. It turns out you can make the generation of the proofs, like, 20% faster, or even more on the HNR server I use by switching to a custom memory allocator. So, it's… probably, like, I don't know, potentially, because it's not entirely stable, so, not sure if you want to use it in product, like, even in the testnet, but still, just to let you know that there is, at least on the main branch, there is the possibility to use a custom memory allocator that gets you 20% performance increase on average, just to let you know. But it likely requires to use LeanVM as a Rust dependency. Not sure if that's something you can do using ZIG or, like, FFI stuff, I don't know.

Parthasarathy Ramanujam: So, on Zig, we do create a wrapper around mean multi-slig, and then link that as a static library, so I still think it should be possible, as long as we retain the same I mean, build configuration that you have on lean market slate.

T. Wambsgans: I mean, I proceed about, yeah, pursue it.

Thomas Coratger: Mercy, sorry for, like, you waited a lot.

Mercy Boma Naps-Nkari: Yeah, so, based on the, the server issue, because whenever Embata reports something and we try to run it internally, we don't get the… the same, issue he points out. So, I feel like, the server also can contribute to this, because there is really no way to figure it out when, we can't kind of, like, reproduce the same thing locally on our own servers.

Thomas Coratger: And so you mean… when you mean you have different eruos, you mean you have, like, less euros, or more euros, or just simply different ones?

Mercy Boma Naps-Nkari: So, when we tried running all the clients together, that was a problem, but when we picked Some certain clients, and then run it to the same, number of validators and subnets. It's worked fine.

Gajinder Singh: So I think it should just be possible to provide Mercy with access for the grand lien service path, alright?

Parthasarathy Ramanujam: Yeah, that should be possible. The logs should also be there on Grafana, but Masji, if you need access to the servers, let me know, I can provide you access, you can inspect the logs directly there. But as I said, in DevNet 4, the regular validators are… multiple validators run on the same server, so that's why we have the hard cap of, 3.5GB per container.

Thomas Coratger: Okay. Okay, and so… so if I understand correctly, right now, in DevNet 4, we are pretty much stable in terms of finality. How long did you run, like, the DevNet part? It was, like, a couple of minutes, or a couple of hours?

Parthasarathy Ramanujam: No. So, we… the latest one was run for close to 12 hours. We had finality until 56 lot number. After that, several containers crashed, and there have been several issues with each client. I've reported them. And when we lost Finality. I think it's still running, or I have stopped it. I would be restarting the DevNet with the new images, at least from Zeam. If other teams are publishing one, do let me know. I'll restart later today evening, and see if there's any improvement or security there.

Thomas Coratger: Okay.

Gajinder Singh: Supply. So, we are a bit away from basically saying that we are stable on DevNet 4, on multiple subnets. Need to keep running it for a while.

Thomas Coratger: Yeah, yeah. Okay, so let us focus on this, let us not… let us forget for now, like, DevNet 5 stuff, let us try to reach stability for DevNet 4.

Gajinder Singh: Yeah, so DevNet5, I think a mail can still work to basically bring down the multi-message. Time down, because we still need that to happen, right? So…

Thomas Coratger: Yeah, yeah, yeah. Yeah. Okay, so that is pretty clear. So, yeah, so… if I summarize, like, we will have, like, part of our, like, team, let's say, working on lean spec simplification, so basically I am not touching anything that is, picking. I am just doing cosmetics and stuff, so I don't touch the logic at all. So that will be one bucket of work. The second bucket of work will be, like, you folks working on DevNet for stabilization, running DevNets, identifying issues, trying to figure out what's happening. And then maybe there is, like, one other bucket of work. about syncing how we can make DevNet 5 running with the current stages that we have.

Yeah, that may be 3 buckets of work that we can continue working on. Until, next week.

Gajinder Singh: I think we should also hear from Camille what his update is regarding, Shadow Sims, because that ties into the same.

Thomas Coratger: Sure. Of course. Please, Camille, feel free to.

Kamil Salakhiev: Yeah, thanks. Yeah, we kind of started the discussions first, like, what server do we need to run large-scale simulations? Yeah, I think… I don't remember if I reported this in the PQ interrupt chat, but yeah, it's like… I think I tested with Zeam and Qlean, and it was, like. third node, if it's not an aggregating node, we were, like, consuming 400MB in at least shadow sims, and if it's an aggregating node, it's, like, 600MB. So basically, if we can afford 64 gigs in a server, we should be capable of running Like, 100 notes simulation, so that's even… and we can even have some margin in there, so that, yeah, for sure, it's, like, we have enough memory for…

For the simulations. Yeah, then, yeah, we already have Qlean client that, well, it works under Shadow. Then, I also tested with Zeam, it also runs under Shadow. Yeah, we also have, like, this, topic in PQInTrop channel for Shadow Sims, I think. I don't remember, did I share there? I think there should… yeah, there is a link… there should be a link to… if not, yeah, I'll share it later. There should be a link to the branch in Link Quick Start, where, yeah, we can basically… anyone can run, shadow simulations, if you… if you want. The… yeah, they can run either natively or, in Docker, in ARM, so if you have macOS, for example, yeah, you can also run shadow simulations.

What I am trying to do now is, yeah, first, yeah, like, we have these discussions, like, yeah, what… how network would behave, like, if we say we have this or that, performance, so… I'm trying to create a fuzzer that would basically run continuously, for the… I don't know, gonna… Let me transfer the screen. Yeah, it's… Let me know if you cannot see it. Because I'm under 42 VPN, so it might be slow. Anyway, so basically, what I'm trying to do is that the fuzzer is running continuously, and we can set up, like. I don't know, the durations of each simulations, like, how many nodes we expect to have, like, how many subnets, how many aggregators, so basically it's father randomly selects some values.

Yeah, we can… it has some seed value so that we can reproduce it if we end up with some errors. We can play with, like, signature aggregation rates, or, recursive aggregation rates, and so on, like, also play with a portion of each client, like, zoom Qlean, yeah, we can, for example, say. So that, yeah, each run has its own setup. And then, yeah, each round would produce some kind of, like, statistics, like, what was the… yeah, how we were finalizing blocks within the slots for… and also how the blocks were propagating in each client, how attestations were propagating. Yeah, it's currently… I'm working on visualizations, and then, yeah, I won't… this cache of… I'm still working on it, it's, like, it's not ready. But it's gonna look something like this. So, basically, we have these runs, so that we can, like, choose, either previous run.

And check, like, what was the block propagation, and yeah, what were the distribution of nodes that we had during this run. For example, here we had only Zeam. Yeah, I'm using the region distribution to kind of set latencies between nodes, so that we basically have a… basically, I have a JSON, so that… with what is the latency between each region, and then, yeah, I'm assigning the node to one of the regions. And, yeah, also assigning, like, some bandwidth to each node, and so on and so on. So basically, all these parameters, yeah, we can just randomly assign, and then, yeah, we can, like, check whether the simulation was stable, for example, see, like, what was the research propagation coverage for each slot.

Now, what was the block propagation? And so on. And if there were some errors, yeah, we just can go and download the log files, and yeah, check deeply if, yeah, what was happening. That's the idea, just you have some kind of… running simulation, once it's ready, yeah, I'll test on my… Local server, it's like… it actually has 64 gigs. So, yeah, and then if everyone's happy, yeah, we can deploy it somewhere and keep it running. The good thing about the shadow simulations is that, well, we can just keep it running on a single machine, but we are not competing for the same resource in terms of, like, if we have… because basically the nodes… well, the simulation is deterministic, and nodes are kind of the exchange who is kind of…

Who is being the… who is running, basically, the… who's allocating the… who's taking over the resources right now, so that, yeah, no two nodes are running the aggregation in parallel. What we need from clients now is, well, because Shadow is also kind of, because of the… all these deterministic stuff and so on. Yeah, we… it would be great if clients, for example, had flags, like, we could say, for example. But for these flags, they just disabled, but if we can put, for example, signature aggregation rate flag to be, like, 1000 signatures per second, and then when we actually perform an aggregation. We're kind of putting artificial sleep. in the note, and that way, it kind of… because in shadow, if we don't do this, the aggregation, yeah, basically the timestamp before and after the aggregation would be the same, so that it would… it would look like the aggregation happened instantly, which is not the case.

So that, yeah, if we had these artificial slips, then we can play with the different aggregation rates. We can even go further, I think, and yeah, we started doing this work with Ruslan and Collin, that, yeah, we can also even play with sizes of signatures, of sizes of aggregations, and if we just replace real XMSS with some kind of fake… XML says so that no real aggregation is happening, because it doesn't really matter for simulation, but yeah, we can play with It would even.

Thomas Coratger: And so, here, like, what we are saying, basically, is that I understand, like, what you are mentioning, that you can play with a bunch of parameters to simulate various scenarios with different nodes, different signature time, and everything. So, like, Gajinder, like, was your idea to simulate more DevNet 5 stuff, with, like, multiple message stuff? with this kind of simulation tool before going to real-world validation through client and stuff? Was it, like, your ID?

Gajinder Singh: Yeah, I think this can be… actually be used, for DevNet 5 simulation, because since this is simulated time, so you can, for example, take 3 seconds to aggregate and still log 0.4 seconds on the shadow, right? So, you can use this, actually, to very well simulate how we are interoping on DevNet 5. So, in that regard, yeah, I think we can definitely do that, and move forward on DevNet 5 interop in the meantime, when we are all trying to bring down the real times down. So, in that sense, yes, we can definitely use this for DevNet 5 same runs.

Thomas Coratger: Okay, so that is very clear. Let us continue, like, on that direction. Maybe, like, someone else has something else to do, or are we done? Okay, so perfect, guys. Thanks a lot for all of the work and for the updates. See you next week.

Parthasarathy Ramanujam: Thank you.

T. Wambsgans: Bye.

Thomas Coratger: Bye-bye.

Mihir Faujdar: Right?

Pablo Deymo: Alright.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.