Evaluating Client-Side Proving Systems

Transcript

[music] [music] Okay. Uh, let's start. Um, hello everyone. My name is Alex and I work at the client side proving team of the access layer of Ethereum Foundation. And uh of course uh we today we're going to talk about zero knowledge proving and um how many of you are ZK developers or have u any experience with ZK? Okay great I think it's the right audience for today. So but we in particular are talking today about the client side proving. So let's first uh

discuss what are the peculiarities of client side proving and how it is different from like general zik proving. So first of all of course when we're talking about client side we mean the devices that um most of us own uh most of the time uh and have them at hand. So it's either a phone or maybe a laptop and uh for such devices as we can imagine uh the compute is quite limited because they don't have as powerful uh CPUs as servers and uh definitely don't have GPU capabilities as as powerful as servers like they don't have CUDA for example. Um [snorts]

another very important constraint in client side is RAM. Uh because again uh RAM is quite limited especially on smartphones. For example, in the beginning of 2025, our team did a survey of mobile hardware. And what we found out is that at that moment, an average iPhone had around 6 GB of RAM uh available and an average Android phone had around 4 GB of RAM. And of course it also means it doesn't mean that you can actually use all that six or all that four GB because you still need to leave some for the operating system for other applications. [snorts] And uh what is

also interesting about client side proving is that bandwidth and u storage may become uh constrained because for some of the zero knowledge proving systems uh just to start proving you need to download quite a sizable proverifact uh maybe like a Z key like uh proving key in case of growth 16 uh that could be up to several hundreds of megabytes depending on your circuit and of course doing that um while the user is on a meter data connection it may be not the best choice from the user experience perspective.

So um we have been running these client side proving benchmarks for a while and uh you can find the source codes for those benchmarks at that repository there. And uh we present the results at this eth proofs page dedicated to the client side proving benchmarks. So please don't confuse with the zkm benchmarks that also leave in e proofs and um yeah we will get back to this page in a while. First of all I want to talk about uh how we run the benchmarks.

So we choose [snorts] a cloud instance that is like uh an good enough approximation of the client hardware. And in this case we want a bare metal machine and u a good choice was Apple M1 which we can have like whole of it not not virtualized and uh yeah it has [snorts] 16 GB of RAM and eight cores. Um we run those benchmarks on a variety of workloads and we will talk about those workloads uh in more detail soon. And of course we are measuring the timing, the proving time, the verification time and

uh RAM footprint uh and various sizes. So um yeah uh all those measurements they are related to the constraints of the client side proving. So uh as I mentioned before uh RAM is a particularly uh important constraint and uh the size of the proverifs that we need to distribute to our users alongside with our app and of course the times.

So now let's talk a little bit about the workloads that we are uh running benchmarks on. So there's a variety of workloads, a few hashing functions and uh ECDSA uh elliptic curve uh uh signature proof. Um uh hashes are the classic hashes and uh let's say the hashes that are more specialized for zk and uh the classic hashes are shot 256 and catch check. So those workloads are uh heavy in bitwise operations because they weren't designed for ZK specifically and it turns out that those bitwise operations they pose

a significant bottleneck for most of the proving systems um that operate on uh prime field elements instead of bits. So um those workloads also come from some of the real world world uh applications. They're mostly um let's say [snorts] web two uh for example different digital credentials where the uh the hash function is shot 256 or uh even from the Ethereum itself where catch was chosen as the main hash function before um before ZK was a big thing let's say so uh get check was

chosen because of its security But uh as it turned out it's not very ZK friendly later. So that's why these workloads are important. And um ECDSA is of course widely used in uh signatures. So it's uh the signatures uh on Ethereum and also Bitcoin and uh in various identity applications and like pass keys. So uh how ECDSA relates to ZK is also um uh not that simple because uh ECDSA should be done uh on in the field uh of your your curve the

curve that you're working with which is uh defined by some standard like let's say it's a pass key so it's a sep 256R1 and if your proving system is actually instantiated over a different field. Uh and sometimes you don't really have a choice of field in your proving system. Then uh what is happening is that you are uh forced to do a so-called foreign field arithmetic. So you are doing the arithmetic of your ECDSA field uh in a field of your proving system. And that's why this is also a significant challenge.

And uh we also benchmark uh the Poseidon family of hash functions and they are uh the opposite of the previous two because they were created specifically for ZK. Um they were made ZK friendly from the beginning by uh making them operate on the prime field elements instead of bits. uh and uh you also in many cases you have a freedom of field choice. So you can just choose the field for your application that is the same as the field of your proving system and in these cases uh using Poseidon is very natural. Yeah. Recently there uh have

been some concerns regarding Poseidon security and like in the Ethereum ecosystem maybe we can say it's being phased out a little bit but still it's widely used in very many privacy applications um on Ethereum like shielded pools and zk applications so [snorts] it is still um quite important so what we will do today is we will take a look at how you can use benchmark data for um um choosing a proving system for your particular application. And so we will work with some example requirements,

example specifications of hypothetical u zero knowledge applications and then we will work with the ETH proofs uh dashboard data to select uh proving system that best fits those requirements. Um, and of course, um, we may want to do it in a in a way that is better than just looking at the numbers and trying like to find the system that's like best uh, according to all criteria. And there is a very nice method that can help us to uh, do that.

The method is called a parto frontier named after the scientist who probably first um applied it and uh the method is quite simple. So uh usually you have a some kind of tradeoff when you're designing um your system [snorts] and for simplicity we can talk about a tradeoff between just two parameters although of course in many cases there are more than two but let's look at this um simplest case of a tradeoff between two parameters.

Um we plot the points in two coordinates for each of the system and then we see if a point is dominated by any other point and by dominated we mean if any other point has a value that is like better for this parameter. For example, in this case, we are plotting proving time against uh proof size. Uh and uh we want them to be as low as possible. So here lower is better. So like an ideal proving system should be somewhere here.

Uh and what does mean for a system to be dominated by others? For example, if we take a look at jolt, uh we can see that in both uh dimensions in proving time and in proof size there are systems that are better. But then if we take [snorts] a system like Bretenburgg for example, we can see that it is only dominated by a couple of systems in the proof size but uh not in the proving time um uh at the same time. So this is a frontier point and uh by uh having this kind of

plot we can find out what is the frontier here and uh the points on the frontier uh represent the best tradeoffs that we can get. So um we discard everything that is like doesn't belong on the frontier and then uh we are choosing from these frontier points uh according to to the further criteria that we have. So for example if we need the best proving time here then we'll probably go with binas and if we need the best proof size we'll go with [snorts] groth 16. But if we want a good compromise then we can choose u

ratenburgg and the best compromise u as you may guess lies around the like uh diagonal here. Yeah. So now um let's talk about the dashboard of eth proofs and I will try to maybe to open it here also. Okay.

Doesn't work. Yep. Yep. So on this dashboard uh on the top right we see the different workloads that we'll be selecting according to the assignment and for those workloads that have uh varying uh input sizes for the benchmark we can also select input sizes here and we see that the charts are changing accordingly.

Then uh we also have we can compare individual systems by just selecting them under the charts um or deselecting all of them. We'll just reset the chart to its initial state. Then we have a system properties table where uh it is not a one row per measurement but it's a one row per system and you can explore um those properties of the systems that are not measured by the benchmarks but it's more like the some qualities of the system like security whether the system is zk or not whether the system is postquantum

or not and maybe some of its internal cryptographic components but that is not going to be the focus of our uh workshop today and uh more importantly you can filter this table. So for example, if you're only interested in systems that have the actual zero knowledge property and they're also postquantum, then you can filter the table uh on the top left. And then finally um maybe the most important part for especially for those who are willing to use their agents today for the workshop is the full uh measurement table that you have at the

bottom where there is a one row per measurement across all the workloads like without any [snorts] separation. And here you have this button this called export all that you can press just to download the CSV with all the measurements on your machine. So um I suggest you to do that now if you're looking at this page cuz uh later you can give it to your agent and just tell it to um plot uh [snorts] various uh graphs like a parto frontier for example. It will be convenient to um for accomplishing the tasks.

And uh maybe now uh it's a good time to answer any questions if you have them. Okay. No questions. No. Okay. Let's

continue. Yeah. So we have talked briefly about uh various filters that we have on the dashboard and now uh we are ready to start with the group assignments. So um let's divide uh ourselves into four groups. Um I suggest maybe to try to divide by by just uh by rows like this. So maybe two um two or three seats on here on the left is a group A. Then uh these two three seats group B uh C and D. So those

who are willing to um participate and discuss may also maybe turn to face each other and uh let's take a look at our tasks. So um I didn't print anything and I hope you can just um take a picture using your phone. Uh this is like a um kind of like a short version and then I will um show all those individual cards with like a detailed task. So please let's let's do it like group by group. So first uh group A uh yeah let's first let's check if we are properly divided

into groups. So those who believe that they are in a group A uh raise your hands. Okay. One, two, three. Okay, that's not bad. Uh, now please group B, raise your hands. Okay, great. Now C. Okay, and D. Okay, great. And now uh a more detailed task for group A is here. So, please uh take a picture of this.

Now, please, group B, take a picture of your task. Now see Okay. And and D. So um

let's do this for um maybe 10 minutes. I hope that's going to be enough. So, let's check back in 10 minutes. And uh also during the those assignments um you may uh consider that no system satisfies the requirements. that is also possible. Then uh what we're going to do in the end is that a representative from each group would uh tell the result and uh maybe talk about what could be an

alternative like the second best system if it exists. Uh and uh also maybe which actual benchmark you would run uh that suits your application better. And uh once again uh for those of you who have your laptops, I strongly suggest to try playing with the Parto Frontier.

[clears throat] Uh and the easiest way of course is just to instruct your agent to plot the Parto Frontier according to those tradeoffs. uh that you need to account for according to those limitations for each of the tasks. But also uh one quick note is that sometimes um you can have a situation where you have a system that is so good that it dominates all other systems. So for example, you can imagine if there was a system that has um proof size that is lower than draw 16 here and also a

proving time that's faster than binius. So that system would live somewhere here on the plot and in this case your paretto frontier is just one point. So it's just uh um an easy situation where you don't need to think about the tradeoff. So I think in one of the tasks you will experience that situation. Oh yeah, and maybe it flashed fast, but

for all groups uh we require the system to be uh fully zero knowledge because we are considering it for like privacy applications an actively maintained system because we don't want to work with duplicated ones. Uh yeah, so don't please don't forget about these requirements. And another small note on parto

frontiers um since we are working with such like a wide variety of systems that some of them are optimized more let's say for smaller proof some are optimized for proving time uh the units uh I mean like the measured values uh span across a wide range. So it is more convenient to uh plot on against a logarithmic axis instead of linear because like as [snorts] you can see for things like proof sizes the values are varying in in the range of thousands and uh for proving time in the range of hundreds.

So yeah, please use a log log. Okay, it's roughly 10 minutes. So, uh,

how are the groups doing? Uh, do we have solutions for group A? Oh, so you don't need to clone the repo. Yeah, you uh can simply download the CSV from the dashboard and then just uh use the measurement data from the CSV. So, please do not run the benchmarks that is not necessary. I hope I sorry that I didn't make it clear. Yeah.

>> Uh yeah, that's true. But I'm I'm asking if you're I'm just checking if everyone is ready so we can have the discussion. So how about group B? Okay. Okay. So you're ready. See ready system. >> Yeah. For that use case that was specified on in the task. >> Okay. That's uh and okay. So let's uh discuss uh although yeah group A we we also have the result. So let's take a look at the cheat sheet of a sort. So if group A actually try to plot the Petta frontier

uh this is an interesting case where you can actually plot uh proving time against proving time and in those two axis you're using uh proving time for two different measurements which is uh ECDSA and SHA 256 because in your application you had a task that resembles like a ZK ID application usually Usually in ZK ID you have some credential that is signed and that credential is first hashed and you are signing the hash. [snorts] That's why here we need both uh SHA and uh ECDSA benchmarks and um it's quite clear what the frontier is. But then uh you need to

apply those uh other requirements limitations that were in the task. Specifically, there was a 32 kilobyte proof size limit and uh 512 MGB of RAM. And you're basically choosing among uh two frontier points. And it's quite easy to see uh which of them satisfies those uh limits by just looking at the measurement data in the ETH proofs dashboard. So we will see that uh it filters out the proof kit in this case.

Okay, let's take a look at group B. Uh here um we are interested here. We have a workload that is like homogeneous. It's just shot 256. It's an anonymous membership kind of a proof. And here we are plotting proving times against uh our constraints and we have again same kind of constraints we are constrained in RAM and proof size.

>> [snorts] >> So uh here for RAM it's exactly that special case where the whole parto frontier is actually a single point and for proof size we have two points but um after we [clears throat] filter according to uh the criteria the the requirements of the task we will see that only beas actually satisfies uh the task. for C. Uh the application is yeah also something like private data

and uh here we again we plot the same two para frontiers uh against the proving time against RAM and proof size. And here again we have this special case where uh Jolt is dominating one of them completely. And then uh even though uh for proof size we have a rather sizable frontier but we still filter out jolt as you said according to those other uh requirements.

And then for group D looks like this. Uh group D had that kind of like private commitment application with Poseidon. And here uh after all the filtering we will see that uh circum growth 16 would be the system of choice. Now I'm curious. Has anyone tried to plot the actual PTO? Okay. Uh would is what you got like similar to Okay, that's great. Now what we are going to do is that we're going to change the requirements a little bit

for each of the tasks. So we'll uh relax the requirements in some cases uh stricten them a little bit and do one more uh 10-minut assignment just to see what is changing under the changing requirements. And I want to remind that if you come to

conclusion that no system satisfies the requirements that is also one of the possible outcomes. Okay, I think that for the second

assignment probably you uh didn't need as much time because you already had those para frontiers for those of you who actually plotted them. So we can just check the results. Uh how about group A? Yeah, that's correct. Uh it fits the proof size limit and uh yeah, we have a bit more relaxed uh RAM requirement.

So that is correct. How about group B? Yeah, that's correct. you uh don't have uh proof system that satisfies the requirements and it's mainly because BU 64 uh even though looking good but it is configured for just 96 bits of security not uh 128 so it cannot satisfy the requirement how about group C yeah that's correct uh because uh Jolt has a larger proof size. So it exceeds the new limit and the Benberg fits within uh the REM limit. And now how

about group D? Yeah, that is proof kit. It uh it is uh both ZK and post quantum at the same time. It has a 128 bit security. Uh yeah, so that is correct. And we still have a few minutes to um yeah maybe to discuss uh first of all that even though we have these benchmarks um when you're designing your application of course uh you need to uh run those benchmarks that are specific to your application. Like if we come

back to group um if we come back to group a assignment for example uh that is like a zkid kind of application where in your circuit you are supposed to do two different things. It's uh both CDSA and SHA 256. uh you eventually need to uh benchmark something uh something like that and uh in like in group B's case uh if this is a Merkel tree then it should be actual Merkel tree not just like a hashing benchmark uh so um the data in the

dashboard is like uh for your guidance but not uh yeah [snorts] uh not to be used as like a absolute source of truth. Um also um what I can say is that Pareto frontier method is very useful in any kind of optimization task where you have uh tradeoffs and uh yeah even if you have many tradeoffs you can still uh break it down into groups of tradeoffs uh between two of the parameters.

Um so it's very useful just beyond the beyond zk beyond benchmarks [snorts] and uh if you have any other questions please uh I I'm ready to take them. Okay. If uh there are no questions then uh thank you for the participation.

[music]