Post-Quantum Interop Call #48

Transcript

Thomas Coratger: Hello, guys.

Shariq Naiyer: How's it going, Thomas?

Thomas Coratger: Yeah, fine. And you? Very nice, man. Cool. So, yeah, let's get started. So, by the way, I think that maybe not last… next week, but just after, like, the small August break, maybe we'll change a bit the format of this, by… maybe doing more presentation and keeping the debugging stuff for async, so… I need to check, but I think this is beneficial for everyone, for us also to do some… sometimes some presentation, maybe on a bi-weekly basis. On anything that is new, for example, like binary field techniques, or… Anything like this. But yeah, for today we will keep the same format, so let us go for client updates first. For the Zoom team.

Parthasarathy Ramanujam: Thanks so much. So, last week, we spent most of the time fixing bugs that were discovered. in both, are run, as well as the one, DevNet that is run by Ethlambda. I believe that's addressed most of the outstanding issues that we had on Zeam. But, I mean, we are not able to. achieve proper justification or finalization when we run a larger DevNet with all clients. Justification stalls after 12,000 slots, and finalization around 9,000 slots, because, The P95 for, most of the clients right now, apart from a couple, are still, probably 4 seconds. For both block production as well as aggregation, and aggregation signatures are not getting delivered on time, which is, resulting in this stall once, the definite starts to progress beyond a certain time.

And yeah, we spent, the last week, implementing ETH P2P and doing an interop with Ethlambda as an additional transport layer, and it works fine. Hopefully, once, Ethlambda is ready, we could probably replace, ETH P2P's gossip completely with, ETHP2P, and then do an Interop on that. Yeah, that is what we've been working on. Nutting ends. Thank you.

Shariq Naiyer: suite.

Thomas Coratger: Cool, yeah, please go ahead, Ream.

Shariq Naiyer: Okay, so from our end, similarly, we have been focusing on the bugs, that were just covered on, our, our dev nets, as well as Parthas and Azeems and Etambles DevNets. after all the optimizations, I suggest, Partha, we try running on your DeemZeneath Lambda DevNet, and see if, we're still getting… Because on my needs LambdaDevnet, we finalized up to 45… roughly 45,000 slots, and the finalization gap is pretty tight from head. And… And the only issue… the issues that was… the issue that was happening was the REAM nodes died because of out-of-memory issues. So, from our end, the issues we are seeing are out-of-memory issues on Ream that are something we are actively fixing, debugging and fixing.

But… but yeah, I suggest… but after the optimization, I find the DevNets to run fairly stable.

Parthasarathy Ramanujam: How many DevNets… I mean, how many subnets are you running your DevNet with? And it's just 3 clients, I think it's okay. It's mostly… I'm running 8 subnet DevNet, and as you say, just as you experienced, ream crashes with out-of-memory after running for some time. But the Interop issue is not necessarily with just Ream now. We're having issues with other clients as well. So, just curious, how many subnet DevNet did you run?

Shariq Naiyer: So, it's 4 subnets, and…

Parthasarathy Ramanujam: Right.

Shariq Naiyer: 24 to 6, 6 clients per subnet.

Parthasarathy Ramanujam: Okay. I'm running an 8 subnet DevNet, and yeah, that's what happens. Probably a smaller one, it's progressing much longer.

Thomas Coratger: Can you repeat the numbers? Like, you said 6 DevNets? with… s… Can you repeat?

Shariq Naiyer: subnets,

Thomas Coratger: 16.

Shariq Naiyer: No, wait, no. 6 clients per subnet and 4 subnets. We ran 6 Eats… 12H Lambda nodes and 12 Ream nodes. I can… yeah, and we found… Justification up to 44,946 slots, and finalization up to 44,941 slots. This is one of the longest DevNets I've run. The head slot is at 262,143, but all the nodes have Died at this point. So, we… we still have to figure out the memory issues that we're having. Apart from that, we integrated with Shadow, and we've been running some small shadow simulations. I wonder if other teams are having an effort of running shadow simulations? If so, I would love for them to include Ream, and we'll include you guys in our shadow simulation.

Oh, but that's about it for me.

Thomas Coratger: Pardon. Thanks, thanks a lot for… for the update. Zen, clean, I am not sure if you are here, because I have seen the messaging in the Telegram channel, so… Maybe not here. Lambda, then?

Mihir Faujdar: Yep. Thanks, Thomas. Hi, everyone. So, since the last update, we've just been focused on ensuring stability of Lambda in the DevNets being maintained by the Lambda and Zeam team. So… Currently, our EMIT supports the two optimizations that were used by the Et Lambda team, which is, building the block an interval earlier, and avoiding proof compaction. But we're still looking at implementing other optimizations, provided that they don't break compatibility. In the demos that are running. And we're also looking at getting additional servers in the coming weeks, so we can start running multi-subnet DevNets, because currently we're still, have been running on a single server. But yeah, that's our update from my side. Thanks.

Thomas Coratger: equal, East Lambda?

Pablo Deymo: We were working on the aggregation improvements. Our aggravators start, early, once it receives two-thirds of attestations. And it uses a similar scoring algorithm as block building to prioritize Which ones to aggregate recursible in a second round? We fixed our tick duration, login port time, and validation count metrics. On ETP2P compatibility, we are working on implementing the rest of things for the EP2PRS library. We port from the Go spec. We are implementing ALPN stream model. frame length, peer identity. The goal is to have it interoper with the model… with the POC of, Zeam. We… Are also working on an alternative execution client iteration.

that inverts ethrex, our execution client, into Ethlambda. That's an idea we took from RIM. Instead of calling the… the Engine API with the JSON RPC. And we have been fixing some type issues that were causing flaky or failed tests. That's it. Thank you.

Thomas Coratger: And thanks a lot, Gean.

Bankat | Gean: Yeah, thank you. Hello, everyone. So, over this stretch, our main focus has been making sure Gean runs smoothly on DevNet 5 multi-client runs. So, a good part of that work involved, addressing a few rough edges that made Gean appear on LD during runs. And then digging deeper into why it wasn't keeping pace with other clients under certain conditions. I think one of the significant issues we identified was a nasty case where EIP stall a request indefinitely. And then, quietly killed in its block sync while the node still looked alive and kept, attesting, so we've worked on that. We've also improved, check, checkpoint sync behavior. So, nodes starting from checkpoint sources that are temporarily unavailable will now retry for a while instead of, immediately giving up. That was what was happening initially. So, alongside these fixes, we've…

Consumer general, housekeeping and operational improvements. to make the clients more resilient and easier to, operate. So, other than that, we keep on, okay, we've been also keeping an eye on Hive, and making changes where necessary, and then pushing our updates. So, I think that is it from the gene side for now. Thank you.

Thomas Coratger: Cool. Thanks a lot, Ramdeen.

Mercy Boma Naps-Nkari: Hello, good afternoon, Chil. on our side, we made some optimization to, our client, and then we also had some couple of EPF fellows who are currently contributing to grinding and Their focus will be on DB, and then, the, high failures we've been saying. Then, on releasing a new image by tomorrow, we should have a new image with, some optimization, and then the lean spec fixes that we made. Thank you.

Thomas Coratger: Great. So, like, on the general side, from a research and spec perspective, so, again, on the spec, I still didn't touch. anything so that you guys can… can still, like, do the job of debugging and improving and scaling things. In the meantime, we have worked quite a lot on binary feed techniques. And the proving system. So, basically, right now, we are not touching that much. the original LeanVM, that was with Postalian and Koala Bear, so this is, like, just on hold for now, because we are focusing on getting the performance right, from LeanVMB, that is the binary version. the binary field version of LeanVM, so Emil is working on that. He wasn't able to join today, but I can give a brief update of what we are doing.

So, yeah, we are basically doing kind of parallel work in the sense that we have, like, one bucket of work on LeanVMV that is using binary fields plus, traditional hash functions, such as Blake right now. And we are also trying to improve, in the same time the BNU64 library of the ex-irreducible folks, in order… because this library is very laser-focused on binary field, and so we are trying to reach the same performance as Flock, and even to improve them, and to include also aggregation inside this proving system. So, it is in a very good shape, because we have a lot of open source participants there. we are still quite far from the target numbers, in the sense that, if I recall correctly, the latest numbers for the recursion, so the 2 to 1 proof with BLINVMB was, like.

2.5 seconds or something like that. Maybe it has improved a bit in the next couple of days, but last week it was in this order of magnitude, while on the classical LimVM it was, like. 500 milliseconds, I think, something like that. So, we have roughly, like, a 4X slowdown there for now. It keeps improving day after day. We have a lot of optimization in the pipeline, so we are mainly working on that. Also, like, Just, like, a note on the fact that binary field-proofing systems are much more complex. Than, the classical prime field proving system, in the sense that you… You manipulate a lot of different stuff with the bits, and so you are able to do much more fine-tuning optimizations, and so we manipulate bits everywhere, and so this leads to a very complex system at the end.

from, like, a formal perspective, but of course, from an API perspective, this doesn't change anything. You will have, like, exactly the same function, like, just to prove and verify and things like this. So that is about the research topic. So, yeah, I am curious about what you guys think about, like, the potential, like, the improvements, blockers, and stuff like this. if I am pretty correct, right now you are simulating with, like. couple of subnets for the DevNets, and inside each of these subnets, you are simulating with just a handful of validators, so at the end, roughly, like, you are, for all of your simulations, less than 50 validators overall, or something like this. So, yeah, I don't know what's… what are the blockers for further skating, what are your thoughts, and all of this. From what I understood, going into the Telegram message, I can understand that basically the problems are kind of the same that we discussed in the past.

Couple of weeks, in the sense that, the recursive aggregation is pretty slow, and so you have both hardware limitation, and also, like, algorithmic limitation in the sense that sometimes this does blow up. Due to this slowness, so do you guys have more thoughts about this, or steps that you want to share?

Mega | Lambda: We in Atlanta were, worked on some tooling to, To better measure the latency of block proposals and attestations, arrival, and all of that. M… I can show it. Here for a moment… Yeah. This is using an API that we ported from the Beacon… from the Beacon API. Yeah. It's the… I have this open here, the 8th V1 events. Basically, you subscribe to that endpoint, and the node starts sending you events. Here, we made a centralized collector that listens for these events and timestamps. each of those. And this front end here shows the arrival time. This is from…

The start of a SLOT to the end, right here. And the colors are for blocks, attestations, and aggregations. These four are the aggregators, that's why they received it immediately, but… The rest of the notes… him receiving, like… Around that interval.

Thomas Coratger: Okay, so what you can… like, if I am correct, in this graph, like, on the left, in blue, you have, like, the block.

Mega | Lambda: Yeah.

Thomas Coratger: or in green, you have the attestation time, and in yellow, you have the aggregate time, and what is shown is that you have, like, kind of a big, big variations in aggregation. This is what it shows.

Mega | Lambda: Yes, there are some particularity, some particulars here in Lambda that Maybe some, like, the yellows you can see here at the end are probably second aggr… like, a second aggration from that aggregator, and the first one arrived on time here in this interval. But yeah, there's… some… there's some variants here. Also, with blocks here, you can see that there's a big group here that arrives on time, but then you have some that enter on the next interval. That's probably due to a high attestation count, or something like that.

Thomas Coratger: But this, like, this part that you just mentioned is not linked at all with the LeanVM, right? The, the blue, the blue dots that are in the green…

Mega | Lambda: This? Oh, yeah, yeah, probably, it's… it's probably, like, due to aggregation being slow.

Thomas Coratger: And so you mean aggregation is very slow, and so it delays the block production, and so it goes to the green? This is what you mean?

Mega | Lambda: the… I mean, the difference between the two groups is probably here's more… here there's more attestation, so there's more aggregation work.

Thomas Coratger: Okay.

Mega | Lambda: I have… I see. Just to…

Thomas Coratger: Just to be sure, like, on the yellow side. This means that you have a huge difference in terms of aggregation times, right? You have some aggregates that… take a reasonable amount of time that are on the left, while some other on the right are very, very slow. This is what it means.

Mega | Lambda: Yes, that's probably due to… Regulatory aggregation.

Thomas Coratger: Yeah, so basically all of the points on the right are a problem, right?

Mega | Lambda: Yeah, as I mentioned, some of these… I think… I think the answer is more nuanced. we still have… I still have to, like, differentiate here between, Agates for the current slot… slot votes. And maybe there weren't enough boats to upgrade, so we started doing emergency aggrations, or… Irrating attestation that… aren't… Like, the ones that have to arrive in this slot. Hmm… But yeah, the ones on the right, at first glance seem to be a problem.

Thomas Coratger: Because, yeah, I am thinking out loud here, but, like, we have different options for that. Basically, in LeanVM right now, if I am correct, like, on the DevNet, we are using the canonical branch of LeanVM, so that is basically… the branch with the canonical, Linux MSS signatures, so it's with the big signatures that is 3 kilobytes, approximately, if I remember correctly. But we have another branch of LeanVM. Not the LeanVM binary, but the classical leanVM. That is not using this canonical signature. It is a much more… a much smaller signature. something around 1KB or something like that, and we know that this one is much faster to prove. So, I am thinking that here it should, solve a lot of your problems that you observed there. But the problem that we had with this version is that there is no proof about this.

And so, especially with Poseidon, we didn't want that much to use it, at least for our testing, before we had a theoretical proof from the cryptographers. So, that is the reason why we didn't use that up to now. In LeanVM, it will be much more friendly, because since we are using Blake, the trade-offs are completely different, and so it is much more acceptable to use the sub-MTU version of XMSS, that is a small XMSS, instead of the big one. But maybe if… All that being said, if all of your problems still exist. maybe we should check with, Emil in order to migrate to the smaller, to the smaller signature stuff, so that you will be able to use the other branch.

the other branch of LeanVM, and so maybe this will solve a lot of your problems, not sure.

Mega | Lambda: Do you have any numbers for the, like, the speedup with that branch?

Thomas Coratger: I am not entirely sure, because we had a lot of evolution, and we should ask Emil precisely about the bench. But if I remember correctly, there was, like, a huge speedup. I think this is… because right now, we are… am I correct that you are not using the main branch of LeanVM? You are using a specific branch, right?

Mega | Lambda: Yeah, we are using Deb.5, but I believe it's just made with some… with some additions.

Thomas Coratger: I am not sure. Okay, we should… we should check that async with… with EL, the difference between the branches. But yeah, maybe this is also something that we can check, in order to see if Even if we don't have a proof. We can check this, in order to see if this improves a lot. on it. Mercy, yeah, you have your hand raised.

Mercy Boma Naps-Nkari: Yeah, so, I wanted to ask each client about the various optimizations that was, proposed. to understand which ones they implemented, and then what the data point looks like. Although on each Lambda site, they already showed a data point for their own optimization proposal, which We picked option 1 and option 2, so… On our last call, we did… I did raise this up, and then the Zoom team said they will show, get a data point to… Oh… of their notes running the optimization they had. So, I want to know if client teams has already implemented the Zoom proposal, and, if there is any available data points to look into.

Parthasarathy Ramanujam: To answer your question, Macy, we didn't… on the Zeam side, we didn't implement the proposal to change the slot time. That proposal is not there, but everything else, that was already proposed by Ethlambda is there on Zoom. So that's the version I'm running on DevNex as well. if… I mean, I don't have metrics to share based on this performance, because I don't have confirmation yet, apart from Zeam and EAT Lambda, which other client has implemented these proposals. Once I have that confirmation from everyone. I can run a DevNet and then share, How this optimization has helped.

Shariq Naiyer: We have, so… oh yeah, Marcy, go ahead.

Mercy Boma Naps-Nkari: No, sorry, go on, go on, sir.

Shariq Naiyer: So, from ReamSai, we have implemented the early bird optimization, and we have removed Type 1 proofs. If there's any… so if you run the DevNet with Reams E meet Lambda, we can benchmark those, those two optimizations. And, if there's any one… any other minor ones you want us to include in this run, we can… just, like, shoot me a text, and we shall get that in. There is the proposer, skipping the proposer signature, I believe that was an optimization, but I don't know how much hold that waits, I don't know if the client teams have implemented that. So those are the two, the early bird, start building the block early, and the… removing the Type 1 compaction, the Type 1 proofs from, Ream has been implemented.

Mercy Boma Naps-Nkari: So, if I understand you correctly, you only implemented the Ethlambda proposal?

Shariq Naiyer: Yes, two of them from the Lambda proposal.

Mercy Boma Naps-Nkari: What about anyone from the, zinc proposal?

Shariq Naiyer: I think there was some overlap between some of them, and some of them was, like, 8-second slots. we… we didn't… I looked into it, I tried them out, but, like, right now, I… we have… So, I ran a… I ran a workflow in AI to, benchmark stuff, given both the proposals, and what it found… what it found with the Ethlambda proposal had, better numbers, and I stuck with that.

KaydenML: We also made a PR, fixing our tick interval metric to be, continuous, and then, we made a fix for the 128-slot rearg issue Mega had mentioned.

Mercy Boma Naps-Nkari: Sorry, I'm going around. What about Eats Lambda? Do you… have you guys already implemented the, Zeam proposal?

Mega | Lambda: The part about, Increasing the slot… the slot time? No, but… We implemented, We changed our aviation algorithm to do more than one… aggregation per, during the… The aggregation interval, which would… is similar to the… to the, two rounds aggregation that, seemed proposed.

Mercy Boma Naps-Nkari: Okay, sorry I'm dragging this further, because looking at the proposal, I… I have a theory of some gaps that hasn't been thoroughly mapped out. Which has to do with, past finalization idea of this split, target vote, so… I think it's going to change 3FS meaning justification from a specific source target to, a shared, source recognition. So, I don't see a safety argument for that yet, or how it should be implemented, or to avoid that. So, since If we have a data point that cancels out this year with these gaps, it would be nice to have, so that we can also start implementing it, because I think we are not really looking at how this is going to affect, 3FS Mini finalization.

Parthasarathy Ramanujam: I mean, VFS Mini, we are not achieving 3 EFS Mini anyway because of, late aggregates being costed, so… I think unless we address the slow gossip of aggregates. We won't be able to validate what view… you are…

Mercy Boma Naps-Nkari: Sorry, my, my point, my point is about the split targeted… target vote. It changes it. from specific source targets to EIP shared recognition, that's what I'm trying to point out.

Parthasarathy Ramanujam: Can you post that on Telegram now? Because I'm not clear which proposal this is in reference to, and whether… I'm not sure.

Mercy Boma Naps-Nkari: This is the Zeam prop… one of, one of the Zeam… it's, it's on, the HackMD, IShell post-it. It's the, what's it called? This split target foods, 1. I don't have the link up.

Parthasarathy Ramanujam: Okay.

Mercy Boma Naps-Nkari: Yes, the one Ansha posted. Or is that… we're not looking into that at all?

Thomas Coratger: Yeah, I think that you can, like, Partha, just as a reference, maybe… Two calls before this one. this was mentioned in the call issue on GitHub. We had, like, two improvement proposals, one from the East Lambda team, and one from.

Parthasarathy Ramanujam: Yeah, I understand, but there were overlaps between both the proposals, and I believe all teams that I've heard till now have only implemented those that, where there is an overlap between the two. So the 8 slot, one has not been implemented, and I'm not sure if the one that Mosi is referring to right now has been implemented, even on the Zeam side. I'll have to double-check, so… I'll go back and verify the code and come back to you. I see, but I'm fairly certain that, Ethlambda and Ream have not implemented that yet till…

Mercy Boma Naps-Nkari: But you just confirmed that you implemented it, but the only one you left out was the one that increased, this timing to 8 seconds slots.

Parthasarathy Ramanujam: Correct. So I was under… if it is an overlap with the one that Ethlambda has already proposed, then it has been implemented, but I was under presumption all the proposals from… by the Zeam team were already, proposed by Ethlambda, apart from the increasing, the time slot. If it isn't the case, then I may be wrong, I'll have to look at the code and come back to you.

Mercy Boma Naps-Nkari: Oh, okay.

Thomas Coratger: our place like this. So, I… I will ask… I will pause the question on the… on the Telegram channel, so that Emil can see if we… maybe we can switch the branch of LeanVM just as an experiment to see if it solves some of your problems within the DevNet, just checking. Like, I have no confirmation for now, but I will check that. And, no, otherwise, like, we should continue simulating stuff and hopefully increasing the number of validators. In any case, please reach out on the Telegram channel if you have any issues or block error or stuff like this. And if we don't have any other question or topic to discuss, that's it for me. Yeah, I'm Mihir?

Mihir Faujdar: Yep, I… from my side, just have one quick question,

Thomas Coratger: Yeah, so…

Mihir Faujdar: for… ETH P2P implementation. It seems like we are replacing LIP2P with EP2P. And it's the goal that… Doing that will improve, stability of DevNets, or is that focused, performance?

Thomas Coratger: Yeah, this is a good question, like, indeed, like, we had, like, this, this proposal, I think that you mean the repo from Raul that is called ESP2P, this is what you mean, right?

Mihir Faujdar: Yeah.

Thomas Coratger: So, yeah, we had this that was presented in Cannes for ZCC, by the peer-to-peer team within the Ethereum Foundation. kind of as a potential future replacement of ETH P2P. But for now, like, the peer-to-peer team inside the EF is evolving a bit. So, yeah, I am pretty sure that we will have to change our peer-to-peer stack to take into account, like, the Post-Quantum size of objects and stuff, but I have no, like, perfect answer, nor, like, a timeline for this, so I don't really know, but I hope that we can experiment about this soon, yeah.

Parthasarathy Ramanujam: Yeah, so just to add on to what you said, Tama, at the moment, what Ethlambda and Zoom have done is that we have written our own code in both zig and Rust, respectively, of what, Raul has done, and we have this as a separate transport layer, so it's not a… Original replacement for gossip sub completely. But what zeem and Ethlambda does is it, transports the same, attestations and aggregations on two layers, both using lib p2P as well as ETH P2P. One of the suggestions that, I shared in the group with Ethlambda was probably try out replacing gossip directly with ETH P2P instead of lib P2P. And we don't know, I'm sure, the P2P team would have done a benchmark to see how, what's the performance improvement, but this is purely experimental, or just… I wanted to experiment with ETH P2P, so I built it, and I believe

Ethlambda want to do the same thing, and we've just interrupted between the two clients ourselves, and It's working, but we have not compared it against, how… what's the improvement that it provides when compared to your P22.

Thomas Coratger: Okay, yeah, okay, okay. And I encourage you to reach out to Camille, for example, who is in ZF, and he's probably more familiar than me, for sure, about all of these topics, so I encourage you to reach out to him.

Mihir Faujdar: Yep, right, that's clear. Thanks so much.

Thomas Coratger: Okay, so, yeah, thanks a lot, guys, and see you next week.

Shariq Naiyer: Thank you guys, have a good one.

Parthasarathy Ramanujam: Thank you.

Thomas Coratger: Bye-bye.

Pablo Deymo: new.

KaydenML: Bye-bye.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.