Benchmarking Mainstream ZKP Development Frameworks

Transcript

foreign [Music] how are you feeling in a few hours that'll be the after party woohoo yeah hi I'm junda I'm the co-founder and the CTO of seller Network you know our Crosshair Bridge has you know transferred over 14 billion dollars of assets and our users really love it fast and secure and we do believe in the trust-free omnichann future so that's why today I'm super excited to share what we have done with some of the you know zero knowledge benchmarkings and I hope you know other developers you know and the people who

are interested in this field can learn a little bit from there yeah so in computer science you know we are all familiar with you know all kinds of benchmarks some of our CPU respect and the 3D Mark for your graphics card so you can get your headshots faster than your opponent and uh lean park right so the scientific Computing the result is actually used to rank the top 500 you know HP Series right and of course the tpcc for database and just stream for you know JavaScript webassembly some may ask we are still in such an early age of zero knowledge what's the point of benchmarking right and that's exactly what I try to

emphasize here right I don't want anyone leave this talk and take the wrong impression right we are never intended to try to squeeze those 0.1 percent of performance against right now instead what with this Benchmark what we want to do as a contribution to the community is to make informed decisions right there are so many projects out there they they believe in the technology they are interested but what they have to build something and then the first thing they run into is oh there are like 20 Frameworks out there which one should I choose right so that's exactly what our Benchmark tried to achieve here say oh there are circuits you can write directly there are compilers of VMS you

can use kind of right upper like higher level languages or if your you know use case is really you know time critical you might just have to go with the hardware accelerator directly right and on the other side understand the pros and cons again this is a huge design space different Frameworks different algorithm they choose their own like design choices in this space sometimes your use case doesn't care about the latency but sometimes your use case might care a lot about proof size right so all this kind of you can think about the different variables applied to different Frameworks when we started to do something I'll explain a little bit later on we also have the same confusion

as any kind of new developers coming to this field from the other side We Believe The Benchmark will also be very helpful for the framework developers so who are writing this code to basically provide this you know apis or zero knowledge proving Frameworks to The Wider Community why you know during our interaction with them after we have the first initial run of data almost all of the framework come back to us and say hey there's the easiest thing we can do to make it quicker to make the memory footprint like 20 smaller so I think that's kind of the feedback maybe when they were developing this they didn't say oh that might be a shortcut I just take but then when we kind of present The Benchmark to

them they say hey there's a some low hanging fruit I can make the the whole framework so much better right I think that that's kind of the power you know we believe the Benchmark can enable and of course for the wider Community we want to have more people rallying behind you know um the zero knowledge progresses right imagine someone some news come and say hey there's a new framework in town that's like beat all the other Frameworks right so that kind of makes the heist line get more people excited about the progress of the technology okay now you might ask you you come from a bridge background what exactly are you doing with zero knowledge right exactly exactly because we are in this High

risky Bridge Business you I'm sure everyone has read enough news about multi-sake Bridge being broken yeah that's why we said we have to look at something let's get rid of this additional trust right so that's why we keep talking about trust free the only thing you trust is the math and the code just like Bitcoin since the beginning right and um if you're a developer right now all the zero knowledge the same based thing is already working right we are not just hand waving say oh the thing will come it's actually already announced everything was published two days ago we have the full very detailed documentation to explain each step what we are doing with the zero knowledge and also a very well written easy to use API

as Insider SDK so what you can do exactly is from uh say ethereum or any other evm compatible chain you can prove the storage value you can prove the transaction you can even prove the receipt so which of course for the developers out there the events like for example when you do a swap a D5 trade the the final outcome number only lives in the receipt right so I think that's kind of the power of it of course because we're coming from a bridge background this is the kind of very trivial to build arbitrary arbitrary message passing Bridge across the change right again the power here is that there is no third party in the middle you need to

trust the proof will be generated based on ethereum you know Merkel tree you know where we're talking a little bit more in detail later but on a high level there's no one can even fake or like manipulate the data because the zero proving their knowledge proving give you the confidence the whole thing is correct and the the layer by layer data that's already built into ethereum consensus give you the guarantee the data is correct right so there is no the data Integrity is untouched so you know we have been coming to this talking with many the both the layer one layer two Roll-Ups as kind of supporting their chance you know with seller say Bridge we already have like almost 50 different chances as our partners but

now we're looking more interested into with this new dimension of data what we can enable the adapts developer to do right so here is one simple example if you ever done any trade on centralized exchanges you know they have VIP program you know if you kind of a big heavy Trader you get a discounting fees I would say five years already passed and even today right now if you go out there there is not a single desk can do this why because the history data is missing while the history data is missing because it's so expensive to put everything on chain so that's why we believe with the you know storage slot zero knowledge proof and also the

receipt transaction proof it trivially for any you know decks to enable oh Whenever there is a trade coming unlike today you always treat them like a brand new address right think about that let's like Unthinkable unbelievable if you're coming from a web tube background every single if a transaction happens even if you're like a loyal user you will say hi nice to meet you it's my first time right it's actually no because your your own chain smart contract doesn't know even though thousands of transactions may already happen but it's not simply not available some project they have been very

successful on chain a and now they want to move on to chain B they don't have to start from scratch right they can literally carry some of the things again trust free to the new chair okay there are many other use cases I won't I wouldn't have enough time to go into but I hope you know what I explained earlier we'll get to know the developers out there really excited to say oh this is kind of can enable completely completely new use cases but before we get too excited we still have to face the question how do we start to build right that's exactly what we did we said let's go out there and find all the you know the uh the zero knowledge proven Frameworks out there let's see right let's see how they perform how you

know if it's even doable because you know honestly speaking last year this time I am I'm self is kind of what they call ZK Skeptics right I I saw the papers I saw some of the numbers I just don't think it can be practical right so it is indeed one small step we only did share 256 hash the reason being you know we choose this of course number one our brevis were heavily depends on that why because ethereum use it and it's of course widely used in many other you know blockchains and another interesting Factor if you are in this field you might heard about like zero knowledge friendly hashes we choose sha-256 exactly because it's not friendly so

there is like a non-trivial proven work to do otherwise every framework will give you results within like 10 milliseconds there's no point to compare right and uh oh by the way if you scan this is a valid QR code it's called Pantheon or if you scan that it goes to our blog post Pantheon that's you know the Benchmark for the zero knowledge proven Frameworks and of course to be thorough and complete we vary the input data size and also to try to you know keep the variable as small as possible we write the native circuits using each of the Frameworks okay this is just the first step we know there are many exciting great work being

done right now in the field for higher level compilers IRS and VMS that will definitely be our next step and for the hardware we choose two right one is of course the Linux server mostly Focus the focus is on thanks focus on the the multi-core to look at the parallelization but per core frequency is lower and the other is the Mac because there is another hope that we do not want the proving to be too centralized otherwise everyone will run their approver in AWS right so the so the decentralized thing in the end will just be a myth okay it's okay you do not understand every word on this it's also okay if you do

not understand any of this on these slides the high level takeaway is just like we have guest cars you know diesel cars electric cars flying cars it just show you there are so many different choices out there in this field the math is getting more and more exciting and literally every month there will be new paper coming out right so here's the first result right as I mentioned earlier the complexity are all in in this field people talking about the number of constraints that's very critical because it kind of give you the up a sense of upper bound how how big the mass problem it is right no matter how fast your Hardware is if the if the problem itself is like 10 times more expensive you still need to solve a 10

times harder problem right and of course it's not really very hard to get apple to Apple comparison but what we have tried you know for the people who are more familiar with all this indeed oh for the same scheme they could actually generate different you know number of um constraints that kind of even has a 20 difference that's also a surprise to us because initially we thought if it's the same say I want to say as the the results will be similar but it turns out different framework might have their own secret sauce and also of course if you look at even for the similar plunky ones some of the them with heavy optimization they can literally achieve like an order of magnitude uh you know reduction in complexity that's very hiding right

and now the oh that's the problem with animation I need to go back to proving time yeah we try our best to do Apple to Apple comparison and the the the plot is actually drawing in logarithm scale so the time if you go to the top right is already lacking several hundreds of seconds right for for the bigger size of uh don't don't worry don't try to read I have a simple one side one slide summary so and this is the memory consumption why we care also care about memory consumption is that people talk about decentralized a lot and there is a one fundamental thing if your Hardware requirement is too expensive naturally it will kind of have this tendency to be

more and more centralized right so we so that's why we say oh we would actually like to have the framework even able to run on my Mac Mac right MacBook Pro or even a MacBook Air or even a Raspberry Pi right it's like a low low under low powered you know embedded devices okay this might be one of the most interesting uh you know if you do not remember anything else this this should help you um uh there is one you know obvious winner in terms of speed but there's comes at the trade-off the proof it's it generates is not succinct so that 800 kilobytes and independent of what size of the data we throw at it so this

actually might still work for some use cases but for for like for for example for us for brevis we have to verify the final proof on chair is simply a straight no right there is no way we submit like 800 kilobytes of code I don't even think ethereum even today is a product right the claim is uh unlimited or like only constrained by by uh the block the gas limit but actually there is another much lower limit uh hard-coded in gas we found that the hard way yeah and among the Planck Frameworks uh as I mentioned the the new framework called bujam from ZK sync team with their heavy optimization it achieves really amazing both fastest and also with the smallest you know memory

footprint and for growth 16 it was kind of the the original uh you know or the first batch of the ZK Frameworks uh the ganark from consensus and rapist Arc they have you know pretty good parallelization so that's why we remember in the beginning I mentioned we specifically tested a multi-core architectures yeah and for people using Halo 2 which is of course another very popular very popular you know ZK framework uh in Rust it does perform in our test right with the lookup tables approach it does perform better than Alternatives but it also comes at the bot when the data size is small the additional setup overhead

actually overweighs the performance scan one more minute I think I will try to summarize really quickly with with this one you might notice there's a Nova line but I didn't even mention that because Nova is like similar to a flying car it's really hard to compare Apple to Apple with the previous framework so it's unfair if we just draw the line and call it a day right so we actually did something more Deep dive with Nova to to surprise recursive and you know parallelization and we actually found something very interesting with Nova and we are in close contact with the author to kind of figure out the author himself is also surprised we didn't expect that and the the first step time keeps going down no

matter what and the final compressed time actually has kind of this stair-like Skyrock right uh let's basically tell you to now try to make each Nova step too complicated right so I think you know again this is absolutely the first step the beginning of that and we welcome the entire Community everything what I just talked about including brevis is open source you know on GitHub you know with our circuit test circuits uh Benchmark scripted scripts and uh brevis SDK and documentation right so if you're a developer or you are interested to say hey I have to deal with some other uh elliptic curves feel free to write your own Benchmark and you know share with

the community yeah that's out of my talk thank you [Applause] actually we don't have much time but we you can like ask one question thank you um do you know I had benchmarks for sha256 I myself am trying to start research on a benchmarking different DK EVMS so I was just wondering what kind of operations would you consider to be the most suitable uh uh benchmarks in that direction because like some do you get the question right so so you so yeah

let's see if I understand your question correctly so basically my understanding is that because the event we're mostly working on a little bit higher level uh it's very hard for you to to say directly to to to to to to evaluate session 256 itself right yes because it's because they have like this large execution Trace right right wondering yeah uh my intuition right my intuition for that so that's why you know for for shadow 56 it's more like the the the the building blocks out there so with circuits is like with the the circus plank whatever it's it's it's it's a better Benchmark case but if you say you are anyway looking at a higher level I

would actually suggest go with some concrete use case let's say you're approving a MPT ethereum commercial Miracle Patricia tree right that's kind of the whole thing and it actually will be much easier to write supposedly using a VM or language higher level languages right and that's where you compare Apple to Apple right and it's actually more close to use case and the developer will be very interested because everyone if they want to do a live client they have to verify that um do you think Shaw 256 is a is a good circuit to Benchmark against because plunkish schemes with lookup tables are going to have an edge just because look up tables right

um uh yes and no right you're so sorry I did I did yes that's why I mentioned we didn't just pick one data point let's say I call it there I said yeah no it's always you know the reality has more nuances in there even the Halo 2 I explained earlier with lookup table when the data is larger it's faster when the data is small the lookup table table setup overhead actually causes more time so I think shot 250 you know there's no perfect on the ideal Benchmark case right we picked Char 256 honestly number one we need it in brevis right we like I mentioned we have to implement aziki like land and for all the slot value verification we literally have to verify

many many times yeah so there's no way to run it and two it is like widely used and people say yeah at least we get a some idea it's not the full idea it's you know that's why we welcome the community contribution to like you can feel free to contribute to other benchmarks or like share algorithm using mix more you know sense to you right that's our hope okay thank you [Applause] [Music] [Music]