Post-Quantum Interop Call #35

Transcript

Thomas Coratger: Hello, hello.

Shariq Naiyer: How you doing, Tomas?

Thomas Coratger: So, yeah, Will cannot make it this week, so let's start. Let's start first with client updates. So, Zim first.

Gajinder Singh: Hi, Ramal, thank you. And let's start with our update. We were able to achieve Interop for devnet 4 over the weekend. And I believe, we… Zoom is working well with Lantern and EthLambda, and over multiple subnets as well. So, that's fantastic news. On… on, on this front, Zim was mostly, busy in fixing the issues. That we encountered, while doing the interop. And we are working on stability and optimizations. Which… there are some few low-hanging optimizations that we are going for, and then we'll dig deep into the performance, issues that might… that are coming up, or that might come up. And, basically, our aim is to make sure that, devnet 4… Onwards, Zee becomes one of the most stable clients, participating in the network.

And, that is what our focus is on devnet 5, me, we, we, so I and Anchilla are basically… discussing on what could be the devnet 5 spec. Basically, what we really want to do is aggregate all the signatures into one signature, but as we have seen in some spec discussions and PRs and rewards, that That we will need to repack at a station. So, so we will basically need to figure out a way where we can reutilize… The signatures, in repacking of the attestations, even if we have a single aggregated signature. For some particular validators, because let's say your node was not able to… was not able to receive the aggregated attestations for some particular validators, and you got them in a… in a block.

And now, if you are a proposer, so you'll need to rebuild it and… You will need to repack, which is exactly the case that we discussed for the latest, justified, justified updates that we talked about in devnet for Channel itself. So, we are basically looking into that and, determining what is the best way to go forward, and basically discussing with Amel And you, over what should be the interface. for… Block-level aggregation of all the signatures. Yeah, and then our aim is to also produce a shadow-compatible build, so we did some, shadow-related, fixes, in the sense that to support Shadow, to support running on Shadow, and once we have a stable devnet for

a much more stable devnet for running, codebase and images, we will basically then try to do shadow simulations with Camel. Thank you.

Thomas Coratger: Okay, cool, let us talk about, like, the… the block syncs for devnet… for devnet 5, just after, maybe? or maybe next week, let's see, like, as we are running on stability for devnet 4. Maybe this call is not the best one to talk about it, but maybe the next one, but let's see. maybe with Emil, who is in VM discussion. Let's continue first on the client updates.

Shariq Naiyer: So, yeah, how you doing, guys? We've been working on getting devnet 4 up to spec, and we have been testing against, currently, we have been testing against Zim and Grandine, and we seem to be fin- we seem to be finalizing quite well. And, on the other hand, we have seen all the spec tests that have been added to the spec, and we're updating our spec test suite to, match, to match the devnet 4 spec. And our main focus is going to be devnet4 stability and, and testing, and, and just ensuring that we can, we can finalize at, pretty high slots. On a different note, we've also been working on a testing framework. We've been working on Hive. We've been integrating some of the clients on Hive.

I think we have Zim, we're slowly working on Lantern, adding Lantern, we have Reem, and we have ETH Lambda. Derek would like to give a quick, a quick, a quick spot on it, just a quick show of, tool on it, at the end, towards the end of the call.

Thomas Coratger: Yeah, perfect. Let us do that just after, like, client updates. clean.

Ruslan Tushov: Hello. Oh, we've… implemented devnet4, however, we didn't implement case separation yet, so… During testing, we used two same key players, and didn't test separate subnets yet. Also, when running interop tests, I noticed that, there is some Yes. slow penalty, and as I mentioned in chat, it seems that recourse fabrication it can be slow. I've attached a link to the… in the chat. If you can measure it on your machine, I would add its results to the README, too. Thank you, that's all.

Thomas Coratger: Great, thank you, Lantern.

Mihir Faujdar: Yep. Hi, everyone. So yeah, I've basically been preparing for devnet 4. We have implemented the specs and have been able to interop with Grandine in our local testing, but also good to see that we can interop with Zim. So that's great to see. As well as being working on adding tests, the new test vectors that have been added in, lean specs, just to ensure that we're compliant with all of them. I guess on the metric side, just waiting for the devnet4 metric to be merged, before adding support for it. Yep, that's your update, thanks.

Thomas Coratger: Great, thank you. It's Lambda.

Pablo Deymo: Hello. We finished working on the devnet4 support. The last week? This last week, we implemented the station count cap per block. Proof aggregation changes from the spec. We integrated the new SSZ and the STF standard intervention test from the spec. And we also updated to the latest version of lean spec Development 4 Branch. And we were running multiple interoperability devnets to debug issues. With some of our clients. And, we also started an internal initiative We work… started working on formalizing the state transition function and four-choice algorithm in lean for language. So we made a PR in our repo, adding the… Porting the is justified… shutifiable after function.

In… in lean form. So, that's our update. Thanks.

Thomas Coratger: Great. Thank you. Guin?

Shaaibu Suleiman: Yeah, hello. Hello, yeah, hello. Yeah, so, we've been working on… still working on, support for devnet 4, basically. We've, implemented, I think basically the support for dual keys, basically for, attestation. and also proposal, and also, recursive aggregation. We've, provided support for our latest updates, been made, I think basically for lean, multi-seq. And also support for annotated, validators. You know, config for, validators. And, we've tested, we've tested, the implementation we have so far. against, Zim and It Lambda, and it has been, coming up, pretty well. Locally, though, we've been doing that test, locally.

And I think our PR for devnet 4 only in Quick Start was, was merged through that. We could also participate, in the multi-client, interlock run for Defnet 4. So yeah, basically, that's it from us. Yeah. Thank you.

Thomas Coratger: Great, thanks a lot. Next, Lighthouse? Next… Grandine?

Mercy Boma Naps-Nkari: Good afternoon. So, excuse me. On the grounding side, we've implemented devnet4, and we're running locally to actually see If we have good enough stability for them to forwards, It's flamba, lantern, and tin, which everything has been me. Going all well, so… about the metrics, once the metrics PR get managed, we'll have our metrics ready also. Thank you.

Thomas Coratger: Cool, thank you, Nimbus? Okay, did I miss anyone? Don't think so. Okay, so let's go for spec and research update. So… On this side, basically, on the spec. We have implemented a lot of tests in the past weeks. No spec change, basically, but a lot of new tests have been added, especially test vectors for you to be able to consume. And to check with. I have added a lot of good first issues for external contributors to participate as well in this effort to diversify a bit the… The contributors that we have, and not only me and a couple of people pushing tests, but also, like, the broader community.

So we started by adding a bunch of very edge case, situations. Even if some of them should never happen, for me, this is cool to have tests for this to ensure that we are all on the same page with all of the behavior of the code. Then I have started yesterday to add much more basic test vectors that we didn't have before, for example, for SSD implementation, and now for networking stuff. to ensure that, even on these very basic primitives, we are on the same page. If this is not the case, this is not at all a problem. We should fix either on the client side, or on the spec side. This is not a problem, this is just a good news, because we'll be able to catch bugs much more easily.

Next, we should work on some kind of visualization tool to have a list of the client and the tests they pass and they don't pass, so that we can have, like, a clear visualization. I know, Marci, that you will send me a message about this. Thanks a lot for… for taking this, like, for thinking about this. I will answer you, today. Mmm… That is, on the… on the pure spec update, we will continue like this to try to reach a stability point and add more tests. For now, I am a bit concerned about the duration of the tests, especially the test vectors. I think this is mainly due to the recursive aggregation that we have added in devnet 4.

causing much more aggregation to happen, and so much more time than we had before. We multiply something by 2 or 3, the duration of the test vectors, and this could become a problem. If we have much more tests, I don't want to inflate the duration that much. I should check, maybe. Then I think that we have, like, the production test vectors. on main not passing, but I think this may be due to not regenerate… due to not… to us not regenerating the production keys after the latest post agent update. I think this is… Probably the root cause. Because, test configuration of the test vectors pass… properly. So that is one thing. That is on the spec side. On more, like, research and cryptography side,

We are making very good progress on PromQ3 and LeanVM. Emil will talk about LeanVM just after, but on the PromQ3 side of things, we are implementing much more things. We're in… is in the process of being fully integrated. We have also some SEM check optimization in the pipeline. with at least two additional PRs to be merged in Plonkey 3, and probably some upcoming one with a new paper coming on ePrint in the coming days. So that is basically… it's about the research, yeah. Emil, if you want to talk about LeanVM.

T. Wambsgans: Hello, thank you. Yes, simply, some… some check simplifications, basically, batching previously that there was one sum check for each Airtable, and now we batched them. And so, in some sense, the lean VM now is only, like, one commitment with wear, one GKI error for logup, one… some check for error validity, and one we're opening, and that's it. So basically, it becomes simpler conceptually, and also more recreation-friendly. And maybe, I don't know, about the… The, the fact that the… recursive aggregation is slow. I think that's indeed a problem for the tests. What do you think? Maybe, couldn't we, like, run to server to have faster continuous integration than the default GitHub one, maybe, I don't know. And if we have a dedicated server.

one valuable aspect would be to be able to have AVX512, which is not available on GitHub. And typically, for a lean VM, I know that sometimes there are some commits that breaks AVX512, and since I'm not on this machine, it's hard to detect it, so having, a fast, continuous integration on all the targets would probably be valuable for the spec, for lean VM, for everyone. So maybe we should invest some time and resources into it, probably.

Thomas Coratger: Yeah, yeah, sounds good. Yeah, definitely, yeah, that should be an option. There are, like, definitely different options. Either these, or maybe for some… For some specific case where we don't want to test precisely the aggregation, but we want to test some other properties, we could just mock the aggregation and have, like, some random values, because this is a test where we don't want to test aggregation, and just reserve the aggregation for a couple of tests, where we want to test aggregation precisely, that could be also an option. I should check.

T. Wambsgans: That makes sense. Actually, like, we could say that you only have A few tests is, as you said, a few tests On the spec, and, on the lean VM side, it's okay if the CI is slower, so we could have much more tests here, probably, yeah.

Thomas Coratger: Yep, On the RIM side, you mentioned that you wanted maybe to show us some tooling integration or something? Maybe this is the right one?

Shariq Naiyer: Yeah, so, one thing is, we were discussing visualization of the spec test in all the clients. We wanna use the same… we're trying to get this visualize, the tool that Derek is going to show, to, to do that visualization as well. I'm gonna pass over this to Derek right now, and he can present, what we are working on.

Derek Sorensen: For sure, hello? Just make sure I can be heard while I just had a fresh install of Zoom. I think it's saying I can be heard. Perfect. Yeah, so, I'm just gonna go ahead and share my screen, if that's alright, but basically… It's a visualization… at the end of it, it's a visualization tool, but… Just for a, maybe a high-level overview for anyone who isn't familiar with Hive, essentially, it's a… A test harness, and it's… the way that it's ran is, It's done through Docker, and so at the beginning of the test, for example, we'll have, like, a test to run with a client, Zim, as we're running Zim, and… What we're trying to do in this test.

is test, client RPC compatibility. And so, essentially, all it is at the end of the day is, a bunch of lambda functions for different tests. And so it'll go through… a bunch of different tests, to test different functionalities for, in this case, fork choice, checkpoints, and state for RPC. And, what it'll do is run through those tests, and for each test, it will start up a Docker image of Zim, run through the Zim test, and then… produce that to… a workspace, so it'll dump all of the logs, and so we can see, like, here's our Zim logs, and then we have Reem, Lantern, and, ETH Lambda. And right now, we are in the process of getting a Lantern devnet 4, but just for the sake of demonstration,

We can see the visualization tool. Move this out of the way. There we go. Okay, so… For example, we can look at… the, Lantern run, and so we can see RPC compatibility and Lantern devnet 3. And we can see, right now, when we run through it, there's two fails, and we can see why. And so… We can see all of the tests here, and view all the tests, and if nothing happens in the test, then we'll just say no log output. But for each test, we can actually view all of the… the Docker logs, and so if we want to see, for example, the health healthy, just to check if the RPC Health Endpoint is working properly.

We can check, see… the whole startup process in Lantern, gone back, and see, of course, it returned it, and the test went well, and… When something goes wrong. So, for example, right now, this test was just picking up. There's a OX prefix in the log, and so it happened to not return. an OX prefix, and if… If we needed more in-detail information about it, we could, of course, go to the Lantern log and view through that. But this is just done through HiveView, which is an existing tool in Hive. And the idea is that… Right now, I've just done individual tests for these first few, just to show that, okay, we can look at the individual tests, and some of them right now are failing. So, for example, the one, I believe, in ETH Lambda, I'll have to take a look at the test itself.

I think it's just a race condition that this shouldn't be failing, but I'll have to fix that later on. But, for example, the one, if we look at, like, Zim, I believe that's one of the same problems, but it'll just pick up inconsistencies, and so, like, right now, in the, health endpoint, instead of returning, like, the lean RPC, it'll turn… right now, it returns, like, Zim API, which I suppose we could… make more lenient, depending on how consistent we want that to be. But at the end of the day, the idea is we run all of the clients all at once for each suite, and so if you go back to the code… here we go,

there is, suites that you can specify, and so right now, we have the RPC compatibility suite that we've specified as a tool, and as we build onto Hive, this can be more like interoperability tests, and so I believe, Colby will be working on, like, right now in Reem, we have a matrix CI test, where it'll go through Reams compatibility with each client, but ideally in Hive, then we make it so that each client runs against each other client, to see if it can finalize, and, like, stuff like checkpoint sync, and late compatibility. But… Just for, like, a…

Thomas Coratger: Just maybe a stupid question, but in the test that you have shown in the screen. where they come from? Like, natural to understand, like.

Derek Sorensen: Oh, for sure.

Thomas Coratger: These are, like, specific, specs that you define yourself, because I think these are not spec tests or something.

Derek Sorensen: Yeah, so these specific tests, were built on just going through the behavior, like, either explicit or implicit behavior from lean spec, so I guess an example of that would be, lean spec, what it wants to do is return, like, OX, prefixed roots, and so that's one of the tests, is literally just, if I can find… all the way at the top here… One of the tests is just, does it hex encode the root? Does it have that… Ox… encoded root, should be OX prefix. Just implicit behavior.

Thomas Coratger: And… Maybe this should be tests that we add, as well as test vectors inside the spec, no?

Derek Sorensen: Yeah, for sure, we could eventually migrate this to, like, test vectors inside the spec as well. This was maybe more so just done as, like, proof of concept and something that could be done either way, but yeah, it could migrate either way, where we also, I believe, would want to take…

Thomas Coratger: In this case, in your tooling right now, if we do that, if we migrate everything as test vectors inside the spec, so that all of the client teams can consume, and then in your tooling that you present there, you will just test, basically, all of the test vectors of the spec, correct?

Derek Sorensen: Yeah, this would go for testing everything, and then making it easily visualizable, and ideally, what we'd want to do is, just have some sort of long-running high-vis instance that kind of just goes through each test, and as clients update their… their image, it will then be like an immediate feedback to say, okay, does this image do anything to change the test results? But ideally, what we do is, for each run, in each suite, we'd run all clients all at once, so this is a run, for example, where I ran, in the… there's a command here. I ran Xem, ETH, Lambda, and REAMS, comma separated in the, command line here. But what it'll do is then…

run, for example, all we have right now is the RPC compatibility Suite, but it'll run through each one individually, it'll test them against the RPC compatibility Suite, so see, we can see, like, each RPC compatibility test, ETH Lambda, you scroll down later, and then it's Reem, and a little bit lower is Zem. And then as we add a different suite, which would just be added right in here, we could add, like, the matrix test, for example, where we just take, like, Xem against ream, two zems, 1 ream, and then two reams, one Zem, and so on. And… and for each client, we could do that, and so it could really cover all the interoperability bases, just to make sure that

As, like, a basic start. everyone can finalize with everyone else, in every combination. That's, like, every basic combination. Obviously, we could make them more complex combinations, but there's probably diminishing returns as to how long those tests would take to run. As opposed to just actually doing, like, a full devnet run. But yeah, that's… that's kind of the idea of, of Hive. If there's any questions as well, I could answer those.

Thomas Coratger: That's great. Like, personally, I love it. I don't know about other things, but… I love it.

Derek Sorensen: Perfect, sounds great. Yeah, we'll be continuing to update as time goes on. I believe this week we'll try to get the, kind of, the matrix tests to go on, and perhaps we could present later on to kind of give an idea of what that looks like as well, but yeah, that sounds good.

Shariq Naiyer: And, and when he says…

Gajinder Singh: Suspended.

Shariq Naiyer: Someone who's speaking. I don't know if I'm speaking.

Gajinder Singh: Yeah, yeah, so, yeah, this is great, and we will aggressively integrate with Hive. And make sure that, you know. This is sort of one of the steps that everyone needs to achieve when they come before and come to an interop point. So, I think this is an excellent step, and hopefully all teams will contribute to writing tests for Hive.

Derek Sorensen: Awesome, sounds great. Oh, one thing I should also mention, I know that it was… maybe a concern last interrupt call, to make sure that any test… so, for example, right now in our individual tests, there are cases that we want to test that are, post-genesis, and so… the best way that I've found to make this most consistent is what we'll actually do is spin up the lean spec client itself, and so… For our, certain post-Genesis tests right now will actually test against the given client that we're testing. So, for example, we're testing Zim, or something like that. Zim would show up, and we'll insert it in here as Zim, but we'll actually run it against the lean spec client, so…

To get, like, this example's just like a… Maybe a very, like, niche edge case, but, should there be a difference in how the checkpoint shape is Genesis versus post-Genesis, but essentially what we'll do is spin up the lean spec client with 3 validators, so it can get to a point where there's some finalization achieved, and then we'll checkpoint sync the given tested client. To then get that new checkpoint that's finalized. And this way, it's as unbiased as possible to just run specifically against lean spec for very, like, in-depth. Specific spec tests, I think as we go on, we can… there'll probably be more applications of when this would be a good use of using the lean spec client itself.

But yeah, I think that's probably also important to note.

Gajinder Singh: So, for most of the tests, lean spec client should be the source of truth, but maybe in some scenarios. Lean spec, client might not be, optimized enough to produce some particular test, and in that case, I guess, we'll figure out, how to go about and, run that test. But, most likely, I think test will be a series of hard-coded RPC calls. In that sense, I think we'll be able to sort of circumvent, the issues that lean spec client might not be production. Grade, but, yeah, I sort of… because what I know is that, Currently, Hive uses GET as the backend to… Orchestrate everything. But hey, we want that, that lean spec should be that, source of truth.

Derek Sorensen: Yeah, yeah, 100%, I agree. So yeah, we can… In Hive, it'll show runtimes, and that is one thing, like, it takes a little while to spin up, but I think that's something that we'll tackle as we go to see if we can kind of get things more optimized, but I do agree that, yeah, lean spec is ideal as source of truth, but if it is insufficient, then, yeah.

Thomas Coratger: Yeah, but in any case, this is what I respond to the message of Colby just in the chat, but in any case, I am sure that sometime you will find or discover some tests that are not in lean spec, for example, for the peer-to-peer behavior, or for the syncing, or for some risk. Rec, RESP, Exchange, and stuff like this, so please, in this case, just… Send me a link with those tests that… so that we should add them as well in the spec. So that all of the other client teams can benefit from this test. So, this is totally possible that you will discover tests, that we should add in the spec.

Derek Sorensen: For sure, yeah, that sounds good. Alright, I'll go ahead and end the screen share.

Thomas Coratger: Yeah, so thanks, thanks a lot for this presentation, this is, this is amazing. now, maybe, about metrics, Katya, if you want to add some stuff, anything?

Katya Riazantseva: Yes, hello, everyone. So, PR is open. I'm working currently on the dashboard update itself. If you have any comments, please take a look into this PR, so if does it cover all these… all we need for, measuring devnet4? So that would be great if someone just start to look into it, and how it is… it can be applied to their code, so maybe we'll have some more updates for this. If not, I will… to the maximum, I will merge it on Friday, maybe. I mean, not… not later than Friday, so this week, If we cover something interesting, we can add to this. And regarding running devnet4, maybe Parse, you will share the updates?

Parthasarathy Ramanujam: Yeah, thanks, Kathy, I can do that. I mean, we've had a stable run of 3 clients with a single subnet running yesterday. Today, I managed to get, I mean, multiple subnets, I think 2 subnets running with 3 clients, and then after a while. it's stalled, I'll probably, Include the additional clients today, and then share the logs as soon as we have something working. That's true.

Thomas Coratger: Great. Anything else that anyone wants to add? In any case, like, as a summary, I think that we are in a stability period, let us say, where, from what I understood, and from what I have in mind, all of the client teams want to stabilize on devnet 4, trying to debug, add more tests. This is also our status right now in the lead spec, where we are trying to stabilize pretty much everything. And had a lot of Cisco rage. Yeah, and on the R&D front, we are constantly improving. The cryptography. to be noted that beginning of May, we will have this cryptography event in Rome, so we will have hopefully… a bunch of events there, and so more news to share about new cryptography and R&D development.

So, yeah, that's it. If you have nothing else, we can wrap up. Thanks a lot, everyone. See you next week. Bye-bye.

Katya Riazantseva: Thank you, boy.

Mihir Faujdar: Thanks for Eric.

Pablo Deymo: My wife.

T. Wambsgans: Thanks, buddy.

Shariq Naiyer: Have a good one, guys.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.