Post-Quantum Interop Call #43

Transcript

Thomas Coratger: Hello, everyone. Okay, so… Let's get started. Today, I think that this will be a light call in some sort, because I don't think that we have a huge number of updates since we are… checking DevNet 5 and interop for this DevNets. So, what I have in mind is, like, first client updates, as usual, then… We can do a brief spec update, especially on the features that have been touched this week. For example, the testing infrastructure. And stuff like this. So we can discuss that, we can, as always, do an update on LeanVM.

And maybe, at the end we can discuss potential blockers, question, feature around .NET 5, so anything you want, so… Let us start first with the client updates. Zeam, first.

Gajinder Singh: Hello, everyone. Thanks, Toma. So, as we discussed last time, we moved to DevNet5, we merged the DevNet 5 PR after Doing some testing. with ourselves and with other clients. We also tried to run a final DevNet for multi-subnet. network with other clients, but seems like all the other clients seems to be on different, different commits for LeanVM, and the interop was not happening. So, basically, then, we totally discarded that effort. So now we are focused on DevNet 5. We have already tried interop, with multiple, multiple clients on DevNet 5, and, We observed that there was justification and finalization. This was for single subnet, and now we have started efforts for, Multiple subnets, runs. And, that is what we are currently doing. So we'll… we are… we will be currently running on 4 subnets.

We have already tried running it with Xen. Nodes, and the finalization and justification seems to be happening all good, but we are seeing that Some of the messages… are getting published late, and this is what we are debugging and figuring out. And in the meantime, we also will start multiple subnet runs with other clients as well.

Thomas Coratger: Okay, great, thank you. Rim?

Shariq Naiyer: Hey, how's it going, guys? So, on the REAM side, on the DevNet 4 side, on the DevNet 5 side, we… we were running DevNets with, 5 nodes on one subnet on Ream, and that seemed to work quite well. However, when we tried running intraop against other clients, there was, tons of, incompatibility. Some of that was coming from LeanVM, but I'm sure that will be addressed, over this week. Some of it may have already been addressed, so I'll test, soon. On the DevNet 4 side, I've been running, 24 validator… 24-node DevNet. So we've got some infrastructure, running, which allows us to run 24 nodes. Currently, I ran only Lean. And after tons of debugging, I was able, debugging and fixing issues. We had, we had some issues with memory that we had to deal with, where our memory would expand very quickly. So after addressing those issues, things seemed to justify well, but finalization still happens in bursts, so I'm currently debugging that. As soon as Ream is stable on these 24-node, 4 aggregator, 6-node

per subnet, config. We'll start adding other clients, and, and scaling that even further. That's all, that's all from our…

Thomas Coratger: Great, thank you. clean?

Ruslan Tushov: Whoa. So, Qlean has released a DevNet5 image. We tested interop with Zeam and its Lambda, however, we couldn't reach finalization between Qlean and Qlean. And also, as I mentioned in chat, locally, I couldn't get Ethlambda and Zeam for analyzing DevNet 5 to I tested only two nodes in a minimal scenario. Thank you.

Thomas Coratger: Great, amazing, land here. It's lambda?

Pablo Deymo: Hello, hello. We have streamed some of the changes Camille shared with us. We're fixing an issue with Rox TV reaching the descriptor limit. We're also bumped the LeanVM to the latest commit in DevNet 4. We ported the lean spec PR egg33 to East Lambda that rejects attestations not on the same chain. We fixed a discrepancy we had in the… with the spec. Now we are reflecting blocks with invalid attestation bits. And we find… we found, a perf regression on the 4 LeanVM, and fixated with the help of Emil. And we are running an interop DevNets in our server to find interrupt issues. We will share some results, we also share some of them, and we are also continuing our work on the execution layer integration with our client ethrex.

Thank you.

Thomas Coratger: Great, thanks for the update, Guin.

Bankat | gean: Yes, thank you. Hello, everyone. So, over the past week, we've been focused on getting fully aligned with DevNet 5, while also keeping track of DevNet 4 updates and maintenance on the side. So, for DevNet 5 so far, we've been able to land a lean VM bump on Gean, and also, get… Ultra. to deeds. We deleted. as head of, LeanSpec actually included the, I think, Choice SNS. So, currently for Gean, we are trying to stabilize DevNet 5 for the next run. We are currently working through the reports from Pata on the DevNet5TG channel, and also the logs from the last run. So, a fix is already scoped and ongoing, and ongoing for that.

So, once that fix is ready, what we do is we'll continue to run JIN alongside, DevNet 5 ready clients, while we continue tracking, lean spec changes and the rest. So, I guess that's basically it from the JIN side. Thank you.

Thomas Coratger: Great, thank you. Lighthouse? Grandine?

Mercy Boma Naps-Nkari: Sorry.

Thomas Coratger: Okay.

Mercy Boma Naps-Nkari: Good afternoon.

Thomas Coratger: Oh, yeah.

Mercy Boma Naps-Nkari: So…

Thomas Coratger: Yeah, go ahead, please.

Mercy Boma Naps-Nkari: We are currently testing, our DevNet 5 changes, and we'll publish an image very soon, so thank you.

Thomas Coratger: Great, thank you. So, yeah, now I can start with an update on the… on the lean spec. So, as described for the past couple of weeks, and probably again in the next couple of days or weeks, I am working on improving the lean spec, make it clearer, make it simpler. So, to give you a bunch of information about this, I have continued to work on different files, last week. I think, like, one is particularly interesting, so maybe I can share my screen. Let me share my screen, that would be… yeah. So awesome. Yeah, can you see my screen?

Okay, you can see my screen?

Shariq Naiyer: Yep.

Thomas Coratger: Thank you. Perfect. So… I added a bit of consolidation on the test infrastructure, so the purpose was, Was on multiple factors, let's say. So, first I noticed that we had a bunch of duplications of tests, that were, like, in… that all of these tests that were previously PI tests. And so that was testing, basically, all of the aggregation, attestation, block, checkpoints, fork choice rules, state, and all of this. With pretty difficult-to-maintain, setup function, like this one, for example, which was… which was, like, maybe duplicated over multiple files. But there was a different point about this. First. We had this test there, but we probably had an equivalent duplicate test. In the test vectors for you to consume, so that was kind of a duplicate effort.

Also, it was, like, not consumable directly by you as client teams, so this was a nice test to test the… if we take this one, for example, this was a nice test to confirm that the spec is good, but… In… Any sort, you cannot, like, consume this, so it was very difficult for the client teams to be in that equation with those tests, because they are not consumable directly by the clients. That was different points. And also a lot of boilerplate code to maintain, and a lot of stuff that was very difficult to check. So what I did is that I completely removed all of the spy test. So… and instead, what I have done is that I have I've done a bit.

like the test infrastructure on the… on the test vectors, with a bit more test vectors, not that much, because we already had a wide coverage, but more. So that, now, all of the spec tests can be consumable by the client teams. We have less duplicate boilerplate code to set up the test. And, everything is at the same place. I also added, like, a rule that is in the… in the agent description. To, document those tests. So, if you apply this with a basic cloud code, you will end up with exactly the same structure always. So, with a given, that is a bit of the context of the test. Then, when, that is, like, the… what we want to test, and the action of the test, let us say. And the then, which is, like, the result that we expect at the end of the test.

And so… And I remove all of the internal comments inside the test, so that for each test, we have exactly the same… always the same structure of comments, so that this is pretty readable, and With uniformity, let's say. Also, I did another interesting PR. I think… no, it was in the same one. It was a big PR. And in this one, what I did is that I… Shit. Let me… let me go directly to the next page. Because I noticed also that the time to run the CI was horrible. Like, basically, it was, like,

tenths of minutes, I think it was 30 minutes for the test pro… test configuration. And it was, like, it was, like, 1 hour for the production configuration. on Maine, and so this was pretty terrible. And so, yeah, I had it, basically, this decorator there. To… that is there only where we want to test the real crypto. That is, we are… a bunch of tests, mostly those who are, like, in this file, in this folder, where we want explicitly, like, to test the cryptography. So we want to test that the proofs are handled correctly, and that the verification of the proof are handled correctly. For the rest of the test, in fact, this is… pretty much not needed at all to do that, and we can mark everything. So, as you can see, we don't have the decorator there.

And so that means that for all of those tests, where we want, for example, for this one, we just want to test a basic state transition. And so, in fact, we don't need the proof, we can just mock it. And so, what I did is that I just, in this case, I just mock the proof. So that the proof is mocked, and the verification is mocked as well. So that the only thing that we… we test is the state transition, or the fourth choice, or anything else. But, so, we just limit exclusively to a couple of tests, the real cryptography. And this way, if you look at the… I have a problem with the production configuration that I should check, but if you, check, for example, like, the…

the test feature on any past commit of today. Now, this is super, super short. Like, for example, let's just go to this one… Yeah, here you can see, like, it is just 4 minutes. 4 minutes for all of the test vectors, and before it was 40 minutes, or something like that, so… It's much better due to this, because we skip all of the aggregation work and all of the rest work. Everywhere. And so, yeah, we are able to do much faster test vectors. So, yeah. That was one goal, so now this will be ended like this. Also, I will continue, of course, like, to… To simplify the spec and all of this. I don't know if you have seen this, let me share, like, a bit more my screen.

Before, we had a big, spec File, where we have basically all of the spec. Before, when you went to there, you had, like, a massive spec.py file, but there. since couple of days, I think, we… We, like, we scoped everything within different files. So, for example, you have a broad production file, or you have a Fortress file, this one is a bit long, but if you go to the state transition inside this file, you just have the state transition related stuff. So you have all of the state transition function, plus the helpers, but nothing else. And if you go to the fault choice, then you have everything related to the fault choice and nothing else.

So, that is a bit better, so that you can think about the logic in a separate way. So that is about the spec. About… the cryptography side of things, as always, like, we have continued to maintain the Plonky3, a lot, extensively. We have tried we have tried to conduct different audits with LLMs, especially with the new cloud model yesterday, in order to try to find new optimizations, new bugs, new stuff. We have also Onyur, who is working with us on a grant. I don't know if I have mentioned that in the past, but he's trying to implement some Bitcoin-related stuff within LeanVM. So, his first objective is to try to implement SHA-256. inside LeanVM as a precompile or something, so he's working with us on the arithmetization of SHA, on some lookup constraints and stuff like this.

So yeah, kind of a lot of cryptography work there. Maybe, Emil, you want to share, like, an update on LeanVM's side of things?

T. Wambsgans: Yes, hello everyone, thank you. I guess the main, nice stuff from this week was the fact that everyone can now benefit from the fancy ZK allocation stuff without having to set up a global allocator, which was, I guess, painful. Sorry about this. So now it should be much better for everyone. And, on my side, I'm starting to work a little bit on, making LeanVM ZK in the real sense of the term. That's it for me, thank you everyone.

Thomas Coratger: Okay, great, so… maybe now, if you have some stuff that you want to discuss about DevNet 5 and things like that, don't hesitate. In the spec, I am trying to clarify everything and find potential bugs, but since couple of weeks, we didn't have any new features to to the specs, so yeah, I think it's a good time to consolidate everything, and to simulate properly, to test properly things, and things like that. Yeah, Mercy?

Mercy Boma Naps-Nkari: So, I have a question for the, based on the WNet 5 for the spec. So, while building one of our blogs? So, it takes about, let's say, 2 minutes because of this Nike that signs, Overall, the attestation packs inside it, so… the provider is sort of doing, 64X the work it does for a single per slot aggregate. So, in the spec, it says a block can carry up to 8 distance, at the station data entries, so that's, like, the constants there. So, but… It doesn't really say if a producer has to, like, include it, or is it okay if we just ship, fewer… Per block, let's say 2 or 3, so the prover finishes inside the 4-second, slot.

Sure, I don't know if I was audible enough.

Thomas Coratger: I… I think that you mean that in the spec, we have a hard, like, hard limit of 8, Numbers of different attestation. And you are asking if this is okay, if we do less. So that the prover has less work to do, and finish early. This is… this is your point?

Mercy Boma Naps-Nkari: Yeah, that's my question, yeah.

Thomas Coratger: Yeah, I think this is totally fine, yeah. I think this is just a maximum, but this is not a… yeah, just a max.

Mercy Boma Naps-Nkari: Let's stop. It's… it's not, like, as a hardcore requirement.

Thomas Coratger: No, no, I don't… in my mind, at least, this is a max. Okay.

Mercy Boma Naps-Nkari: Thank you.

Thomas Coratger: Yeah, Pata, maybe, like, you had a point about the metrics stuff that you just shared.

Parthasarathy Ramanujam: So, I mean, just continuing over the DevNet 4 testing from last week, I couldn't complete it because each team had bumped the LeanVM version and we didn't achieve interop, so I stopped that and have been testing since this morning. directly on DevNet 5. I've identified a few issues which I've reported already. So, right now, I'm running 32 node Zeam only, 4 subnet testnet, and it's progressing well, but we have, identified areas for improvement within Zeam. But one thing that, was outstanding from DevNet4, which I would, request all teams to implement is this metric, because in the previous runs, we noticed several clients were missing their aggregation slot, or they were not publishing the aggregates on time.

So, including this metric would help us isolate what is causing that and, maybe improving the performance of the aggregator role as well. I'm not sure if we might require a similar metric for regular validators, but we'd probably look into that once we have a DevNetify run with proper I mean, observability. The other thing I'd like the teams to keep in mind is that although we are running regular validators on a machine with 16GB of RAM, I believe only 12GB will be available for the container, because we have other stuff running on the same server, like, Node Exporter, Prometheus, and Loki as well. So if… when you're doing your testings locally, ensure your container limit is 12 gig, because only then

We can, I mean, observe the same kind of behavior on the long-run engagement as well. So, that's… That's all I had. Thank you.

Thomas Coratger: Okay, great, thanks. So if we don't have any singles to discuss. I think, yeah, that the… we have basically one important point. for the next week, and maybe the next couple of weeks, is to stabilize on DevNet 5, and to be able to simulate with a larger number of validators. So, yeah, I guess that you guys know… you know what to do, like, in terms of stability, metrics, performance, testing, and stuff like this, I think this would be our main goal. So, yeah, Camille?

Kamil Salakhiev: Yeah, just regarding a larger number of validators. I… I am trying to also test that, yeah, I'm still kind of working on that, but just quickly can share, some… Updates if we can. just a few things that I identified. Basically, I was running 64… I believe it is Lambda nodes. Yeah, a few things that I've noticed is, first, we, like, consume a lot of bandwidth on… Aggregation topic, even though this is the setup where we have, like, 30… oh, sorry, 64 nodes, single subnet, and only single aggregator. Nonetheless, yeah, because of how Gossip Sub works, a single node was receiving, like, say, 60 megabytes Of data, sometimes, per slot. So that means, like, there were a lot of duplicate messages.

that's… yeah, we discussed that in P2P Team as well, it's… Yeah, there might be some… two weeks, like, to Gossip sub that we might need to enable, maybe just change parameters. For example, sometimes this could happen because of, like, I have messages. Yeah, so I'll be playing around to see if we can reduce this. And also with 64 notes, I think… Yeah, the block propagation was pretty slow. Basically, in a setup where we have, like, majority of nodes running EIP7870 requirements, so basically 50 megabits per second, yeah, we see that sometimes, basically, the block propagation latency… Yeah, it's pretty slow, around 2 seconds for, yeah, P95. propagation. And not much less for P50. So, yeah, that's just one thing that also might make sense to consider, maybe in future, maybe changing the intervals durations, maybe for some upcoming DevNets, because now it's, like, 800 milliseconds per reach.

phase of the slot, and yeah, if… maybe it makes sense to change that. I don't know. Just here doing this… And.

Thomas Coratger: Do you… you mean to increase the slot time as well, or like…

Kamil Salakhiev: Well, that would be the easiest solution, just to increase the slot time, but maybe, yeah, if… Yeah, I need to get more numbers, but maybe if, say, attestations propagate fast, aggregations propagate slow, or blocks are not propagating very fast. Maybe, yeah, we just can take this into account, so reduce interval for decisions propagation and, yeah, increase for… Block propagation, or for, yeah, aggregation propagation. Things like that. But maybe, yeah, easily… the easiest thing is block propagation… well, sorry, like, just increase the slot from, say, I don't know, 4 seconds to 6 seconds, for example. then we… then I believe network would be more stable, because now, Yes, I'm… yeah, I don't have this beautiful… oh, no, if I can share it…

Now, what I see is, Yeah, sometimes we have pictures like this, so basically this is the finalization, so we see it's not smooth, like… It's not finalizing every slot. There is just a bump. Or sometimes it's smoother, but still… Yeah, we ideally should finalize. Every slot, and this is not happening every time. So, yeah. basically, we should either… we might consider, I don't know, playing around with intervals duration, increasing… maybe the slow duration, or… I don't know, changing the requirements, I mean, in terms of bandwidth, for node. Yeah, things like that. But yeah, now it's easier to simulate. And yeah, here I also did not yet… play with, propagate… oh, sorry, with aggregations, so here we assume, like, any aggregation is just, instant. Yeah, that's also another thing to consider.

Yeah. That's it from my side.

Thomas Coratger: Okay. Yeah, thank you. That's interesting, like, I think that, yeah, once we have a bit more stability in DevNet 5, In terms of debugging and testing, we can go back, maybe, with those numbers and use simulator in order to fine-tune a bit the things and see how we can increase the number of validators in a safe way, maybe increasing the start time or… or anything. Yeah, Merci?

Mercy Boma Naps-Nkari: Sorry, one last question. I wanted to ask if client teams build Type 2 multi-message aggregates at, Block, sign time. Because I want to know how they measured the workload for it on its distance attestation entries. That's the point.

Thomas Coratger: I don't know, maybe, like, Zeam, since you mentioned that you have already started in TerraOps, maybe you have an answer to this question?

Parthasarathy Ramanujam: I haven't observed that metric yet, probably I'll have to come back to you on that one. Let me take a look and come back. If you can just ping me the actual metric that you're looking for, I'll respond to you later.

Mercy Boma Naps-Nkari: Okay, thank you.

Parthasarathy Ramanujam: Also, a question to you, I don't have the latest DevNet 5 image for Grandine, so I couldn't test the interop, if you can ping me that as well.

Mercy Boma Naps-Nkari: Yeah, we'll publish an image, we're still testing locally.

Parthasarathy Ramanujam: Okay, that's… thank you.

Thomas Coratger: Great. Any other questions, or comments, or anything like this? Great. Thanks, everyone. And, see you next week, and don't hesitate to ping me on the chat or anything. See you. Bye-bye.

Gajinder Singh: Bye, everyone. Bye-bye.

Pablo Deymo: Nope, you know.

Bankat | gean: Grace.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.