L1 zkEVM Breakout #8

Transcript

Ignacio Hagiopan: Hello?

Gary Schulte: Good morning.

Ignacio Hagiopan: Let's wait for maybe 2 or 3 extra minutes for people to join and can start.

Maxim Menshikov: Bye.

Ignacio Hagiopan: Hey, Maxim, how are you doing?

Maxim Menshikov: I'm fine. Yeah. And you?

Ignacio Hagiopan: Oh, very good, yep.

Maxim Menshikov: Great.

Chetany Bhardwaj: Hello, everyone.

Ruben: Hey, guys.

Ignacio Hagiopan: Hey, Ruben, how are you doing?

Ruben: Good, yeah.

Ignacio Hagiopan: All good. Waiting for an extra minute before starting.

Ladislaus: Hi, folks.

Maxim Menshikov: Bye.

Chetany Bhardwaj: Amy.

Ignacio Hagiopan: Okay, so I think we are ready to start. Let me show my screen… What's the con… Okay. So… oh, I should probably have changed this to number 8, I think. I'm sorry, I messed up a Hicks here. Let's see if this works… Are you seeing my screen?

Ladislaus: Yep.

Ignacio Hagiopan: Okay. Yes, so this is, L1CG VMware Cloud Call number 8, I should put here August, I'm sorry for that. So, the agenda today, it's, yeah, so having some highlights news, then doing some project updates. And then we have Chetany, from, the PandaOps team, internship, that he will be presenting the work that he has done there, which will be pretty cool stuff, on the more infrastructure side. And then we will have time for Q&As, or whatever other topic people want to… to… to raise. So the first highlight that I wanted to share is that Maybe you've already seen this, published, but the EF has a new blog post. Where, at least within DF, we try to come up with some kind of tier list for Hegota,

This is considering most of the cluster… protocol cluster teams, so client teams, research teams, engineering teams, different projects. So here on the left, you can see, like, some preview of the blog post, and it's a link here, which are the slides, links later. And as a highlight on, like, relevant EIPs, mentioned in these blog posts. It's, ACD25, got an A, ish, score. The main argument here is trying to upstream, all these, execution specs. branch that we have that describes what the stateless guest program should do, so it's basically Are quite agnostic. execution spec branch, doesn't really point out to any kind of zkVM or programming system, it's just, like.

Fully defining what a stateless block execution should do, and which kind of checks. Must, happen. Interestingly… interestingly, like, DCIP is, I'm sorry, this branch is not really changing the state transfusion function at all, because that's kind of the whole point of Ruben Blocks. It's kind of… invasive, in a way, in the internals of the testing framework, because we have to add some new fields, like stateless input bytes, and stateless output bytes, the transition tool to support these kind of things. Yeah, this is kind of the main, argument for trying to include the EIP in Hegota. It's not really about… Using proofs for testing or something like that. So yeah, this is kind of the main argument. The other EIP related to ZK EVMs is the EVMification EIP, championed by Kev. So the TLDR of this EIP, if you are not aware of it, is mainly trying to,

transform some precompile that… some precompiles that are not really heavily used into EVM bytecode. So this indirectly means that, for… at least for zkVMs, it will be much… Natural, way of price… repricing them. But of course, this is, like, an indirect repricing for users, so it has some, like, friction on that side. But yeah, I mean, if you balance correctly which ones you target, yeah, it might be worth it. The catch here is that… Yeah, we have to, like, formally verify this bytecode transformation of existing precompiles. We have to decide if, like, EL clients have to run the bycodes or not. And if it is not the case, this has some implications on maybe, like, correctly computing the gas costs, so this has a C, at least from the EF perspective.

And the last, EIP related to zkVMs is the other one that I'm championing, which is Block, BlockHash. running block hash using, like, storage from EIP 2935. This will simplify a bit of stateless input for guest programs, because now we can remove that field and just basically resolve block hash execution from storage and call it a day as any other kind of storage access. This Gotta be here, more like, if there's enough capacity in Hegota to really put this in, it would be really nice. I also have some link that I can share with some UX impact, analysis of, like… because this is, like, an indirect reprising of BlockCh, right? Because it's now signature and SLOAD,

So yeah, that's kind of the first highlight. The other highlight is that, we have a new repository called EIP8025, Where I shared the link here. Many of you have already received this link from from me, via DM, probably if you're an EL or CL, core dev. So, in this repository, we try to do two things. The first one is write this ACDMD file. Which tries to write, more precisely why we think 8025 would be useful to be included in Hegota. Mainly because, we felt that there was a lot of confusion on what 8025 really means compared to mandatory proofs. So, we tried to, yeah, kind of be precise on the argument, instead of simply going to each person and re-explaining things. We thought it was really useful to simply put everything in one place.

We also have this ACD Q&A section, where we collect some questions that we have received before from people. With the answers, and we also had other questions that we were never asked, but we think they are important. So hopefully this can, help, teams to inform themselves about what… where to put the D25 into the… prioritization for Hegota. And the other main document is progress.md, where the, the goal here… Was to have some kind of… engineering landing page for anyone trying to understand what's going on with zkVM progress on Anlua. So this document has, like. Where to find the progress on CL branches, which are the tooling that we are building, some quick TLDR of what these tooling are doing. I kind of collected all the,

breakout calls, and the repo already has all the slides. So, if you are, like, let's say, a heavy LLM user, and you want to give, like, a super concrete hopefully efficient context for, hey, I want to understand some particular angle of what's going on here, probably this is a good landing page, and this will be maintained, and… Yeah. Hopefully this is also useful for people to understand what's going on. Okay, so, those were the main highlights for my site. I have some updates on project number 1, 2, 3, and 6. So, as a refresher, Marcin is not part of the team anymore, so he was leading the standardization part. of the zkVM project. Now that's a bit more owned by Cody, and the rest of the team, of course.

Cody is out of office for this week, so he won't be presenting anything today. He will be probably folding most of his updates in the next breakout. So yeah, I'm trying to grow a bit of… some things here. So, the first thing is that, on the specs and test side, we have released, version 0.8x. There are a couple of releases that I think many of you already know about, but we have simplified the stateless input Regarding chain config and things like that. We now have finally found, working way of filling, east fixtures with an external transition tool, which we use GetH for now for that. We did some fixes, so yeah, just check out, like, the latest release, which is basically based on the LabsLab definite dates.

And I also, released this test zkVM benchmark. Thing, which is, like, the first one, out there. which has official East… Benchmarks for 103… 10.30 and 60 million gas. I'm planning to keep doing these releases, so we have that, like, the test zkVM releases, we are, like, the test fixtures, and we have the test zkVM benchmark releases, which are the benchmark fixtures. And all this released from the e-execution specs repository, so there's, like, a single source of truth for all these kind of things. I'm not sure if we will keep using GetH or not for filling these benchmarks. I know that now Ethrex is able to also fill these things. And it might simply depend on how each client is, implementing the latest specs, right? Because

Filling benchmarks should be done with an execution client that's up-to-date with the specs. But yeah, I will try to keep these both releases kind of in sync. And we can also consider, like, other gas limits, right? This is just, like, a first experimentation. As another update, we also have this new R2 bucket with stateless input for Glamsterdam 8, so if you want to have some stateless input for your guest programs, on this live dev nets. You can simply go to this link. This is similar to previous DevNets, so nothing new here. Some warning on this particular case of devnet 8. I don't know if you are following the latest ACDE or ACDC discussions. There was a recent, let's say, consensus-breaking spec change that, will be still…

happening in these devnets, and I think… they will try to sync that all ELs are ready for being changed at the same time, without spawning a new devnet for that. So, maybe at some point, I will reset this R2 bucket, so we don't have, like, a bunch of logs With this spec change. blogs without it, because that might be confusing, but in any case, I think people usually will rely on the latest blogs anyway, so, just to keep it clean, at some point, you might see that this is reset and start, from scratch, let's say, or from the latest blocks, after all, like, all ELs move to the new specs. I'm touching here a bit more on the standardization, topic.

Since COVID is not, is out of office this week, just wanted to mention one particular, point. We have been discussing a bit something that Lukasz from the Nethermind team has, raised. Regarding some potential U.S. vector that guest programs can have. Mainly because, for example, whenever you're using HashMap or any kind of data structure that requires some randomness. to have, like, uniformly distributed keys or something like that, you kind of need some kind of good seed for randomness. And since guest programs are kind of pure functions, and we don't have any kind of random precompile in zkVMs. There's a fair question to say, okay, what… what is the kind of seed that we can use in guest programs to… to really protect against this? Because…

With FOCIL, now, like, block construction is not really totally… in the control of the block builder, which is good for censorship resistance, but this also means that this opens the door for, like, some proper manipulation, let's say, from a potential attacker using FOCIL to bias Guest programs to worst-case scenarios on this front. So, there's, like, an issue where we are exploring a bit the options. Some people have been chiming in there with With ideas, so here's the link. Cody already created some potential, new precompiled proposal. This is, like, pretty fresh, so… If you want to chime in, you can also write your comments here. So yeah, I would say that if you read the issue thread discussion, you will have enough context, and

There are, like, multiple options, and feel free to leave your comments there. Regarding some tooling repos, we are basically… Finishing some integration of client updates on this latest… test CDVM release, mainly regarding this last-minute, spec change. For example, that 8. And we are so happy to say that now our guests have fully switched to pulling ELFs directly from guest program… guest themes releases. We had, as an exception before, ethrex and REST were compiled inside era guests. This wasn't really nice, because at the end of the day, we want… Teams to really own releasing the ELF that will be really proven.

We don't want to get really in the way of that in any kind of shape or form. So, ethrex and Reth had work into having a CI pipeline that really do these releases, as Besu did originally. So, this also simplifies a lot of this repository. If we want to add a new guest program, it's mainly just editing some JSON file that points to the, a release made some… by some team and It will automatically run a bunch of tests, east, fixture or sound. Those kind of stuff, kind of for free, for, like, a last-minute check before integrating. And yeah, that's everything from my site. Happy to have two. Answer any questions. And if not, we can continue with someone else presenting.

Let me see the chat if I'm missing something. Okay, that's gonna seem to be the case. So, I think the next… thing would be the Consensus layer updates. Maybe, Francesco, you have some presentation? Let me see, Francesco is here…

Derek Sorensen: Francesco wrote our team chat that he… his bike broke down, so he's in a taxi. He'll, join late.

Ignacio Hagiopan: Okay, okay, so we can leave it for later. So maybe… okay, Cody's out of office, so maybe, Han? You have some updates on the infrastructure side?

Han Jian: Sure. Can you see my screen?

Ignacio Hagiopan: Yes.

Han Jian: Okay, cool. So, how do I give some update on the approval infrastructure. So, we have run some benchmarks on the new suits, and with the OpenVM v2.1 and zkV1.2. And the result can be found instinct. It's using the prover to do the benchmark. And we also, did some new features on the prover and the benchmarker front end. So now, when running the benchmark king, we… the benchmark will scrap the matrix from the node exporter and a DC2N exporter during each proof. So, we can see the matrix of each proofs. And also the scrap details in per-test model. So you can see the… at which phase, perhaps the GPU utilization is lower, or the… Or the…

GPU is sorted by the power, so to see if there's anything wrong. And also, during each proofs, we… The benchmarker will collect the proving pipeline trace. So you can see at which point the… probably the proverb is idle or not. And on the trace, you can also see the cost composition. And also the… Also the benchmarker… Front-end is added with the cost analysis. Like, now, all of the three of the ZKVNs have the cost estimation, tools. That you can… you can, given a guess and input, you can estimate how much the cost will be. And in the front end, it shows the cost composition and cost by category, so you can see that

Which kind of category has more cost, without doing the really proving. And also the correlation with the proving time. And you can see that some of the categories Obviously, the outliers. So, when optimizing your guest. We shouldn't use the cost for this category, but might really need to do the proving. And yeah, that's the update, and feel free to check the new frontline with these new metrics, and appreciate for any feedbacks. Thanks.

Ignacio Hagiopan: Thanks, Han. Yeah, it is pretty awesome. This will be super useful. I think both for, guest program teams, and for zkVMs, so if… NAC EVM has any feedback on, on this, will be great. This is also, like, circling back to some open proposals we had for cost estimation. Staff, we are kind of trying to explore this by ourselves, mainly Han on all this, if… how much we can simply use whatever is available, and maybe Repropose a new standard that is a bit more flexible, or… maybe simply decide to not have standards about, yeah, cost estimation, or, yeah, how much we can link cost estimation with real proving times and things like that. And I also think, like, a lot of Han's work will eventually

be linked with EVM gas, repo, which is a tool by Maria that was used in Amsterdam for repricing. Probably for that, we will use simply proving times instead of any kind of cost estimation. But yeah, this is awesome work, and yeah, thanks a lot of Han for this. This is any question for Han? I don't see anything in the chat, but… Okay, let me see if… Francesco is here, I think it's not yet. So, maybe we can continue with, Chetany's presentation about his internship work with EthPandaOps.

Chetany Bhardwaj: I'll start sharing my screen, just a second. Is it visible? Can you see the screen?

Ignacio Hagiopan: Yes.

Chetany Bhardwaj: Alright, so my time at EthPandaOps started with a question. So the idea is to make sure, can we prove Ethereum mainnet in real time on our own infrastructure? We already had a proving stack then, and eventually 4 machines with a total of 16 RTX 5090s. My focus was on deployment and observability, getting the stack running, measuring the full lifecycle, and understanding what happened when and if it failed. The goal was to answer something we could demonstrate with the sustained mainnet runs, instead of one-off runs. So, the main tool, that I built is Professor. It follows a standard Beacon API and sends proof requests through zkBoost, which is the proof coordinator. So testing does not depend on a specialized consensus client pool, which also takes off the overhead of maintaining a specialized fold for, optional proving.

So it keeps history per proof in a SQLite database, it also reconciles pending proof if it disconnects, and exposes state timings with trace links. Getting there also meant improving the surrounding stack, so I added configuration validations and assertions that, exercise actual proving and verification. So, on zkBoost, I work on separate witness for HQ weight, and proving times, and carrying trace context across different services. These changes made it possible to pick a late or a failed block and ask exactly where the time went, instead of reconstructing every request from scattered logs. So… One of the zIS grants showed why this visibility mattered. There was a cancellation raise that left our proving workers stuck for 21 and a half hours, even though the services still showed they were active.

But the fleet had lost usable capacity, which means nothing was being proved, but also there was nothing being logged as an error, or if there was something stuck, there was no notification of it. So we saw requests lasting minutes, with only a few seconds of actual proving work. Measuring queue weight, made that distinction clear. I reproduced, the failures in attended runs, added recovery handling, and enforced a minimum worker capacity for this, zk proving. In a later validation run, the exact same, class of wedge, it recovered in 72 seconds. Those numbers come from separate, runs, one without the fix, and one after the fix. So, 21 and a half hours compared to, 72 seconds. And they also show failure detection and recovery, are part of whether a proven system can keep up with the chain.

The strongest run, it used OpenVM v2.0, and also the first version of Axiom Edge with all of 16 CPUs. It ran unattended for about 72 hours, checking against the chain afterwards. 99.4% of real blobs had proof inside the measured 12-second budgets, basically a slot time. And median end-to-end latency was 6.26 seconds, and P99 was 11.15 seconds. Her end-to-end starts when we observe the block, and ends when the proof resolves. So, it's not just the proving time, it is exactly when a node on gossip would discover the block, which is also how we were doing it. So, but it also excludes gossiping and signing, which are also not part of, 8025, and it does not claim time from exact slot start. This was measured result in a deployed setup. The remaining misses gave a specific list of infrastructure gap.

This is the run over time, with one dot per completed proof. The lower line is the median within 15-minute window. It stayed around 6 seconds for the whole run. The upper line is P99 within each window, so it moves around more than the full run P99 we just saw. The 5 recorded failures were 3 witness timeouts and 2 proof fetch results. So, checking afterwards, it's also found out the two blocks missing from the request histories. Basically, two of these errors were also during Beacon API outage. And this distinction between successful request and complete chain recovery is exactly why persistent follow-through tracking matters. So once proof fitted comfortably inside the budget, witness delivery became the biggest constraint. Most fetches were quick, but the number… a small number of those took much longer. The slowest end-to-end request had proving times and unusually slow witness fetches.

Witness fetching explained 63% of the increase from median to P99 end-to-end latency. 3 requests out of these, of this long run with OpenVM. They hit a witness timeout before they ever could reach the prover itself. So, the next useful improvement was execution client and witness delivery path alongside work inside the zkVM. So, I'd like to see the work taken forward in three directions. First, would be closing the coverage gap, account for every observed block, recover missing work, and persist verification outcomes. This is something that Professor does in some shape even today, but making it more resilient, especially for recovery in case of a failure, would be the next step. Second, improved witness delivery and endpoint failover. So, slow fetches do not consume the timing margin. So, the witness, fetching errors we saw.

Third would be making a deeper analysis during a live run, so we could correlate blocks with backend, actual proof IDs, workers, which the ZKVMs or the distributed coordinators give to us. and the prover phases afterwards. Much of what we saw in this presentation was done after the run and manually, a shared approach across proving stacks. would make both incident investigation and comparison, comparative analysis within ZKVMs, proving stacks, kg programs much, much easier. And the broader output of my work is a setup that lets us measure performance, explain failures, and Validate fixes against, actual mainnet. So, I'd like to thank everybody for this opportunity, especially, Eith Pandorf's team, and Potuz, Stephan Barnabas, for the guidance and trust, and to Han, Ignacio, and Kev for their

insights and work, especially on the ZK side of things. Professor is available with the link on the slide, and Oh, thank you very much for your time.

Ignacio Hagiopan: Awesome. Thanks a lot for the presentation, and also for your… internship work in EthPandaOps has been great collaborating with you, and all this is super, super useful. I… I, like, more on your, like, latest points on conclusions, I pretty much agree that… on two main things. First one is, having more observation… observability around all these proving pipeline is… Yeah, super useful to understand bottlenecks or, idle, times or things like that. And on the witness generation part, I think we need to explore a bit more, how block building and witness generation can be pipeline. The way that we have things today are mainly framed in the 80-25 context, where we kind of wait for the block, and then we generate the witness by re-executing

The blog, so we have a lot of extra work there that I think that, for mandatory proofs. probably the BlockBuilder will be pipelining the block building with the generation, so we can kind of hide a lot of that cost. I think we have to do some more in-detail analysis around this, on some particular ELs. Blockbuilding, logic. But I think it should be pretty much doable. Yeah, just as a high-level comment, I think Francesco wants to talk about something about this.

Francesco Risitano: Yeah, I just had a few questions on that. I missed the start, so I was wondering how many GPUs were used for each run. And secondly, in regards to Witness Gen. Which, method were you using? Were you using the debug method or the new payload witnesses? SSV method. Or, yeah, what was being used?

Chetany Bhardwaj: Okay, yes. So the configuration that we were using, it was with, 16, total, 1590s, RTX 5090s across 4 machines. So, each machine had 4 of these GPUs. And for witness, it was using the old debug method instead of the SSZ one, because that was also being worked back then when I was doing these actual runs. Yep, this is, the configuration that we had around GPUs. It was 16 GPUs, but they were not on a single node.

Francesco Risitano: Right, that makes sense. So, I guess, in the context of an optimized witness fetch mechanic, that Should, in theory, reduce proving time… pretty significantly, right? At least in the worst case, I believe I saw witness fetching, and the tail was up to, like, 6 seconds, is that correct?

Chetany Bhardwaj: Yeah, the worst case were actually more than that. They were close to 9 to 11 seconds, so the 99 plus was there. So this is something that, we had discussed, so using this new SC, flat, witness. But for the runs and the time I had during the internship, I just, couldn't get to work on that. But if we have some time towards the end of the call, I could maybe give a demo for Professor and how it actually works, and maybe it will help everybody around here.

Francesco Risitano: Okay, yeah, one more question. In terms of, like, intuition from working on…

Chetany Bhardwaj: this.

Francesco Risitano: project, like, how much… optimization potential, do you think still remains, based on, like, the… the orchestration and architecture, as opposed to, you know, the proof system side of things?

Chetany Bhardwaj: Yep, yep, exactly, that's, that's, to question exactly what… where my work was. So, I think… Personally, I think there is more that we can, churn out of optimizing infrastructure, rather than proving… I mean. Of course, there are optimizations being done there, even Ladisk on OpenVM 2.21, they have accelerated things by a lot, but the developments over there are parts less, so let's say it's a per version, we are getting a second or two on the… good results, like, reduced, while, let's say, just getting the tail of execution witness would be much longer. So that is, like, let's say… it's more of a client development thing, but in just terms of intra, so… More sophisticated, distributed,

workload distribution is something that could really help, so OpenVM is more optimized for proving, but what we had in their, cluster, orchestrator was that they were basically just using a regular, lowest index, free that the node with the lowest index that was available, they would give the workload to that. So what ended up happening was the initial load, so the first few nodes, were taking most of the workload. So, again, it was a very primitive, version of the orchestrator, but that's, like, one thing I can remember from top of my head. So there are a lot of, stuff, especially on infrastructure side, so let's say just vicinity within the provers is a big thing, so if you could have all the provers connected

With very low latency, that is one thing. If it's a single machine, you do not even need an orchestrator, so you could potentially save out a few hundred of milliseconds to a couple of seconds on a good run. So these are the places I see from intuition we could improve, just from infrastructure perspective.

Francesco Risitano: Okay, that's interesting. Do you have, could you… oh, you shared the slides. Do you have numbers in regards to latency, like, communication latency? Because… I believe I've heard some conflicting points on that, where it's, in some cases, it's kind of negligible, but seems from your opinion, or the study that you've done, it's not negligible.

Chetany Bhardwaj: Yeah, I mean, it was a general observation for our infra. I'm afraid I'll have to confirm with the team if I can share the details, but it was also what… it depends on what you're working on. So, let's say if you're working on a Kubernetes orchestrated setup, it would mean the nodes will have more latency than, let's say, it's a four nodes on a single LAN. So, it also really depends on what exactly is your infrastructure set up, right? So, if it is across the internet, one provider, let's say you're running locally, one is on the other side of the world, or maybe you are using some cloud providers, it could really depend on that as well.

So I… for our hands, for our purposes, it was within the same land, so they could communicate pretty fast with one another, and that is how I designed the thing. So… But if the nodes are distributed, so especially within, instead of worker-to-worker, if the coordinated to worker latency is high, so I'll be specific here. Since it's not worker to worker, two workers within ZKVM, they are not coordinating with one another, but only the, orchestrator, basically, provided by these zkVM vendors, that is what communicates with each worker, so… If that in itself has some latency, and what kind of network it is on, that is more, Impactful to this latency. But if they are very close, if they have very low latency, if they are ideally on the same network, it probably wouldn't even matter much. You cannot squeeze much out of that.

Great, thank you.

Ignacio Hagiopan: Cool. Is there any more questions for Chetany? I see some coming from Yi.

Chetany Bhardwaj: I'll clear it up. Again, the point is, it's not material until it is. for… if it's on a local 10Gbps LAN, you are not going to notice it, but if it is across infra, so that is why most of my work was on distributed infra, but there are limits to how much we can push it. So if it is truly distributed in terms of location, in terms of how they connect to each other, it could be significant. It is not normally significant in your case, like, you're telling it around 10 Gbps, let's say they are on the same switch. No, it's not significant at all.

Francesco Risitano: But I wonder if there's, potential for overhead from the likes of serialization, and things of that nature, as opposed to necessarily just the network bandwidth. I'm not sure.

Chetany Bhardwaj: And could you explain, what, what, serialization of exactly what you're talking about here?

Francesco Risitano: So if you need to serialize, information that you're sending over the wire between workers or the orchestrator, I have no idea if that's actually valid, but, in… in terms of, the… observations that were made when investigating, execution witness fetching. Json serialization was a substantial overhead in regards to, witness fetching latency, so… Yeah, not… I'm not suggesting that's necessarily the case, but it could, contribute in some instances if… If, yeah, if there is serialization taking place,

Chetany Bhardwaj: Yes, so the witness we used was not serialized, it was, again, the… the debug one, but there is a lot of serialization, so the Beacon API, we take the serialized, Beacon block, deserialize it, then get the execution payload out, then serialize it again for zkBoost. For what we call as a proof request ID, essentially. So there is a lot of serialization and deserialization even in this process, but it is quite negligible, at this point, but Yes, if we could just optimize execution witness generation, it could probably give us good outputs from there. Again, if we have the time, I will be happy to show a floor dashboard showing exactly how it would look in a real run, so let's see about that at the end of the call.

Or we could connect, after the call as well.

Ignacio Hagiopan: Cool. Yeah, I think we will gain more experience on this, like, whole end-to-end proving whenever we maybe start proving, you know, if we'll be lumps around the net 8. end-to-end, or maybe whenever we switch to testnets for Glamsterdam. We will have… we can, like, re-measure everything and, just be sure about, like, where are these bottlenecks, and how, pipelining block building with… with next generation will help, or can hide this kind of, bottlenecks there. Okay, so we have, like, one remaining presentation from Francesco. Are you… do you have some slides to present?

Francesco Risitano: Yep. Can you see my screen?

Ignacio Hagiopan: Yes.

Francesco Risitano: Cool. So yeah, in terms of, progress on the CL, Starting with the consensus specs, there's been a… small refactor to the, to the specs, primarily around, kind of, the data model and, the messages that are sent over the wire, so… I've linked the PRs here in the… in the slides, which I'll share shortly. But maybe we can dive into the… into the specs themselves. Can you… Can you see the specs? I'm not sure if,

Ignacio Hagiopan: Yes, we got speed.

Francesco Risitano: Yeah, okay, cool. Yeah, so the… The primary type that has been introduced is the execution-proof envelope. And, this is essentially the type that's sent over the wire. Proof data remains the same, proof type remains the same, but instead of keying based on the new payload request route, we key on the beacon block route. And this just makes, kind of integration and association with the block far simpler, because it aligns with the data model that is, you know, extensively used throughout the rest of the client. And then, secondly, on the P2P interface, There's been a small change in regards to, In regards to, how we verify payloads that are, execution proofs that have been received.

So… I guess… The main change here is, that we actually do the proof engine verification inside of the, the gossip handler. And the reason that we do this is because, Like, attribution of invalid proofs. Needs to be more… needs to be stricter than when you receive, say, a beacon block or, an execution payload. And the reason for that is, the number of… entities or validators that can produce execution proofs is much larger, whereas typically for a single slot, there's one proposer and one builder. So, even if you do receive an invalid beacon block or execution payload. and gossip it exclusively after verifying the signature, and not actually verifying the block, which is typically done. It's gossiped just after signature verification. The impact of that is pretty minimal.

But in the context of execution proofs, where you can have many different validators producing proofs, many of which can be invalid. And if you're only, gating, like, gossip on the signature verification, then you can have spam on the network of You know, lots of invalid proofs, essentially. So this is essentially a more conservative approach. We still need to do some benchmarking because, this will have a tangible impact on, proof dissemination latency. Because ultimately, you introduce around 80 milliseconds or so of verification latency. We still need to get some more extensive benchmarks there, but, yeah, ultimately, we're kind of moving in the direction of being more conservative, at least for optional proofs. Obviously in the context of mandatory proofs, the, the number of entities that can produce a proof per

slot. Reduces to 1. So, in that case, it, you know, we can kind of revert to only verifying the signature. And then… And then, and then gossiping immediately. So, in that context, you know, latency should be improved somewhat. And I guess there's also work being done in regards to erasure-coded broadcast, of execution payloads, and, I guess the implication is that if that dissemination strategy is successful and it reduces latency, then we would likely use the same strategy for execution proofs. So, yeah, that's all on the specs, they've been merged into master, so, yeah, please do take a look. our intention is to try and get a, DevNet running on Gloas. Gloas is reach and stability, so, we'd like to join either DevNet 8 or, maybe a DevNet following that. So, I'm currently doing some work on Lighthouse.

So yeah, Lighthouse have been merging PRs upstream, and in parallel, I have been doing some work on the EF ACT Lighthouse, repository. So, the first PR is essentially just, was just kind of aligning the data model to introduce the execution, proof envelope. And then secondly, Han has got some C bindings for the various different, zkVM verifiers, so… Yeah, I've essentially just implemented a wrapper around the, the, the proof verifiers, and, yeah. I'll, either… intend to upstream this, or kind of just ping the Lighthouse devs to see if they would rather have a PR from this, or they want to implement it themselves, that's also fine, but,

yeah, we just want to kind of push on and try and join, yeah, join the DevNet. One thing to note is that in regards to reducing, the burden or workload on Consensus clients' teams, the ask now is to only implement support for proof verification. That's mandatory, so to speak. And proof generation support is optional. We will support, proof generation in our fork of Lighthouse, but, yeah, I kind of get the impression that, client resource is constrained at the minute, so, we will try our best to kind of, you know, limit the overhead that 8025 has on client devs, and kind of eat the… the additional burden ourselves. In any case, in terms of proof generation.

one client is probably good enough, two is ideal, so if there are client teams that are willing to support on ProofGen, that would be great, but as I say, it's not mandatory. Yep, that's all from me.

Ignacio Hagiopan: Great, thanks a lot, Francesco. There was a question from Enrico, but I think that you already answered it in your presentation some seconds after he asked, yeah. Okay, any further questions for Francesco? Okay, We have 5 minutes left. I don't know if anyone else wants to raise some other points. If that's not the case, we can wrap it up.

Chetany Bhardwaj: I mean, if there is nothing, I'd be happy to show what we have in terms of just seeing a proof flow, if people would like to see that.

Ignacio Hagiopan: Sure, feel free. Sweet.

Chetany Bhardwaj: I'll share my screen again. So, I hope it is visible.

Ignacio Hagiopan: Yes.

Chetany Bhardwaj: Logan. So, I'm running… this is a local DevNet, using Ethereum package. The, the… Professor integration is not publicly out. I haven't had the time to debug a lot of things, but it's just, running two instances of zkBoost, so one is rust node, another one is, SP1. And, to complement that, there are two professor nodes running for each of those. So, again, this is just a more definite, but we have an idea what exactly Professor does, so… So these are the live blocks. You could plug in a current Beacon API, and it would start picking up mainnet, definite, or whatever you want to throw at it. So, let's say I want to see what exactly… so you can basically see the cadence first, so what exactly is happening, the minimum turnaround time, the maximum, how many failures were there, and such, within the last, so you can have a rough idea.

You can also filter things, so let's see what failed, what completed, what is currently in flight, and you can sort them and filter them according to different things. So, let's say this is the proof you want to see, how much time did it take to generate witness, what was the through time, what was the generation. Again, the zkVM is also mocked right now, so this is a mocked zkVM backend. But the cool stuff here is, let's say you can take the trace ID for this exact slot and block, and… Go to yours. tempo, or basically a profana, whatever you're running, and once you are running Professor along with it, you can see exactly where the time is going. So, this build request is, as I said earlier, it is the deserialization, serialization, and that stuff, for BeaconBlock right now.

Again, these are DevNet empty blocks, so that is why everything is so fast. But you can see, basically, how much time did it take to fetch the block? My professor, build the request, submit it to zkBoost. And… How much time did ZK Boost take? And this is basically where we hit the wall right now, because most ZKVMs and their distributed coordinators do not give visibility into that, but we can, again, as shared earlier, we can rebuild a lot of that from the logs and other metrics that they give us. I was able to do a lot of that. The screenshots you saw were mostly coming from that, but this is basically the view you could get.

If you run Proving with Professor, and… So this is another one, there would be something that would have taken a lot of time, so there is reconciliation somewhere around there. We could do… So, end-to-end, high to low, and… Sort by end-to-end, I don't know. So this was a proof that was reconciled, so you can see source, how it was fixed. If something failed, you could also see that. We could artificially do that, just to show things. So, the SP1 one should, now, latest one should be… failure and… If it… we start that again, it would start by itself. So… and even if proof is a fail, so as you saw in reconciliation, it would fetch the blocks that it missed.

And, submit request for the incoming ones. Right now, it does not… if it misses the Beacon block by itself, it does not reconcile that, because old proofs are really not that helpful. But if it submitted a request, and zkBoost got it, and proof SL got, there was some issue with it, it got down, or it got destroyed. once it starts back, it can reconcile the proofs. So, this is basically what we can do with proof as a result of today. Yeah, that was all. Thank you.

Francesco Risitano: Quick question, have you… have you got a, Kurtosis network parameter file merged into Ethereum package.

Chetany Bhardwaj: No, unfortunately not. I've been busy, so I haven't got proof as a, like, put into the Ethereum package. This is local. It is in plans, that is why I have it on local. There are some debugging that I need to do. It's something to do with the blob increase and BPOs. So, once that is fixed, you'll probably have it out, but I just haven't had the time to do it, In the recent few weeks. But yes, this was a Kurtosis running locally.

Ignacio Hagiopan: Amazing. Oh, this is super cool. Plus one on all this observability stuff, being piped into Grafana and… And, like, filtrating, on, like, a distributed system. This is pretty awesome. Okay, so, we are at time. Thanks, everyone, for the presentation and discussions, and see you next time. Thank you.

Chetany Bhardwaj: one.

ben bencik: Goodbye.

Kevaundray Wedderburn: Right.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.