Episode 355 - Coordinating Provers with Norbert from ZkCloud

Transcript

[Music] Welcome to Zero Knowledge. I'm your host, Anna Rose. In this podcast, we will be exploring the latest in zero knowledge research and the decentralized web, as well as new paradigms that promise to change the way we interact and transact online. [Music] This week I chat with Norbert, head of product at ZK Cloud, which was previously Galot. We dive back into the topic of prover networks, mapping out the different actors in the ZK proving supply chain. We discussed the different ways in which teams building prover marketplaces build the economics of

these systems and the trade-offs each of these models offer. We then discussed the ZK cloud system and what he thinks large-scale industrial-grade proving will look like in the future. Now, before we kick off, I do want to just let you know about the ZK Summit. ZK Summit 13 is coming up. It's happening on May 12th in Toronto. If you've never been to a ZK Summit, I definitely recommend checking it out. This is the spot to find out about the latest research, the newest applications, to find out who are the most important players in ZK today. It's also a wonderful way to get to know sort of the larger ZK community. The application to attend is now open. Links as always are in the show notes and I hope to see you

there. Now Tanya will share a little bit about this week's [Music] sponsor. ZK is finally easy with Noir, the fastest growing zero knowledge programming language. Built privacy preserving apps without any ZK experience. Apps today all look alike, especially across chains. Noir gives you simple privacy tools to build something new. Noir is rustbased, backend agnostic, and already used by leading ecosystems like Starkware, World, and Aptos. Aztec Labs is running a 4-week program, Noir Hack, with 200K in funding and grants. You'll get hands-on support from a dedicated team of 10 Devrils, weekly deep dives, partner workshops, and a demo day where you'll pitch to judges from Bass, A16Z Crypto, IOSG

Ventures, and Paradigm. Applications close soon. Sign up now at noirhack.com. That's noirhack.com. N oi rhk.com. And now here's our [Music] episode. Today I'm here with Norbert, head of product at ZK Cloud, which was previously called Galot. Welcome to the show, Norbert. Hi, Anna. Thank you very much for having me. I'm excited to have you on. So, we've actually been in touch quite often because ZKV, my other project, has been working with ZK Cloud.

ZKV is right now an operator, both a validator improver on the permissioned early network firestarter and ZK cloud. Cohen from my team I know speaks to you quite regularly, but he is the one doing most of the work. I think you'll hear from this interview. I still have a lot to learn. I'm so glad that you're on the show to share with us a little bit like the state of prover networks, how they've evolved over time, what you guys are doing specifically. Yeah, I'm looking forward to it. Thanks a lot. Actually, we're very happy to be working with you guys. Yes, I've mostly been in touch with Cohen because as you mentioned um you are one of the prover node operators on our scalable permission network firest starter. But

then as we get closer to the mainet launch and we started uh validator on boarding I think ZKV is also participating there. So super happy to be working with you guys in different fronts. Yeah. Very cool. Norbert I think you and I met actually in KCO. You had joined uh on behalf of what was at the time the Galot team as a sponsor. We did the ZK hack KCO event. And I just I just really remember this t-shirt that you printed which had this basically Gaylord from the ZK hack team had done a bunch of stickers and one of them had this dragon and you managed to grab it and print it and it's quite it's quite a stunning shirt. I've seen it a few times

around since then. Yeah. I mean, people have been coming up to me and uh asking me about that shirt, whether we do have some, you know, a couple of samples that could be distributed because it's it's been really popular. And I also hear funny thing, some of those shirts are still walking around in Krakco on random people because the local people love it so much. I don't know how it landed there. Yeah. Uh but it's it's a funny story. Nice. So what I want to do with today's episode is really look at prover networks. I think we have covered this previously on the show and so we'll dig up any old episodes where we kind of introduced the concept but I think it's

really exciting to have you on to do a bit of a snapshot of like where we're at today with prover networks. We hear a lot about a lot of teams that are launching prover networks or partnering with prover networks and I think trying to understand what they are how they've evolved how different teams are kind of interacting with them will be really helpful. So let's start by going way back in time to the beginning of prover networks aka 3 years ago four years ago I don't think it's that long ago actually somewhere around there. Yeah 2021 possibly. So yeah, I want to talk to you a bit about like what were those early prover networks or prover marketplaces as they were called back then. Yeah, so to be honest with you, I was uh just coming to the blockchain

space back then. Uh so I may have a little smaller grasp on on that era of proving basically. But I do know that um basically the first ones were Mina's marketplace which they used to call snark marketplace maybe even today. And then uh I think actually nil was also starting off creating a proof market which then uh and then they pivoted to building an L2 based on ZK sharding but that's a different story regarding the the snark marketplace what I remember um having heard and having read is that um so it was mainly generating proofs for the

Mina blockchain. Yeah. And um what was very interesting and very eye opening also for us particularly is that um some entities started offering proofs in that marketplace um for free and it just basically killed the competition, killed economics, basically it killed the marketplace and obviously there are ways to come around that. Um however it was a a really important moment where and I think architecturally it's it's a very important topic how you want to distribute proving because auctions have their advantages but also as we've seen with this marketplace they are gameable

and there can be situations where uh large entities could could easily gain monopoly. On the other hand, other approaches may have their own um benefits and advantages and happy to dive deeper uh on that later on. Cool. Yeah, at ZKV actually also ran I guess it was approver on the snark marketplace. Um but I remember like we were participating early on but I think eventually like you just described first I think there wasn't enough demand for proofs because it was just for the blockchain itself not for applications living on the blockchain yet later they released the the language and then you could create applications and then you needed a lot more proofs so it started

to make more sense but back then it was just the recursive proof that like kind of powers the Mina blockchain being created and it just wasn't enough demand and then like you said like someone you know people came in with zero fees zero cost and it sort of tanks the marketplace. I would mention one other team that I believe like that I think has a prover network of some kind which is IO because they're running hardware provers as part of their offering as well but as far as I know they're only proving the Alo network. Y I think it was last year where we started to hear maybe maybe the year before actually where we started to hear this sort of story of of prover

networks, prover marketplaces coming back where teams were talking also about like decentralized prover networks where you'd have multiple provers potentially proving multiple L2s or anyone who would need these sort of proof requesters, people who needed ZKPs. And so it stopped being sort of this ver this vertical integration where it was often like a blockchain plus their own prover network and more like this prover network like true network as a as a service proving to me multiple parties.

Tell me a little bit about because I think galot zk cloud comes from that era. Exactly. So tell me a little bit about what the thinking was there. That's a very interesting era. I think in it was in summer 2023. Um I was in uh in Paris ECC and actually it was talk rule talking about the different ways to approach proving for example for an L2. So currently it's it's a well-known fact that basically the largest proof demand is coming from ZKL2s um validity rollups mainly and there's been quite some talk about whether they should build an enshrined prover network

meaning an in protocol prover network and bring that network entity beside the sequencer into the protocol bake it in and uh progress that way or go with a third party proof provider who could optimize for proving but then you know the thing is that it's the go to market is much easier with a third party network because you don't need to build everything yourself. On the other hand, it's very important to ensure that whoever you outsource your proving to does not introduce unnecessary or unhealthy dependencies or maybe even you know you build a nicely decentralized validator set but then you have a centralized prover who is you know introducing trade-offs to your network

that you've built. I mean this is not the situation with the current L2s because most of them are running in a centralized fashion and exploring they're running their own right. Yeah. Um and and like they do uh sequencing except for Tao obviously because uh that's a based roll out but uh most of the L2s are doing both sequencing and proving for themselves. There are some um that are um already actively exploring decentralization of sequencing governance proving etc etc and also some others like Aztec for example who want to go decentralized from day one. Yeah.

So both on their test net and um on their main net and I think also ZKV has been um involved in sequencing and and proving of uh the Aztec test net. Yep. That phenomenon of rollups. I mean it's true now nowadays there are these rollups that are hopefully going to launch with that decentralized but when they first launched all of them or most of them were quite centralized. So centralized sequencer, centralized prover. I always wondered where do those actually happen? Like where like you talk about it being centralized, but is it like AWS? Is it like hardware? Do they have distributed provers at least or distributed sequencers but all controlled by them? Do you know this? I

mean I do have some information about different teams. Most of them were basically renting large clusters from Google or or Amazon. Okay. So it was a very kind of like one of the approaches uh dominant there. And then um I do know of some teams who had private engagements with u GPU farms or uh you know data center operators and um renting compute from a couple of different places. But what's very important I think and that's already leading or pointing towards the dynamics changing is that when you rent your own machines and you run your own proving you need to rent machines to cover your

redundancy or else you will have you know an extended uh and prolonged finality in times of transaction peaks etc etc. So it's very interesting to see the different approaches but it's also very interesting to see that um basically over time the industry started leaning toward third party proving and not anymore thinking about building their own in protocol approver networks. Oh and then third party proving obviously has a couple of advantages that uh are are worth mentioning. It's the paper proof model versus you know you rent your servers and they are sitting idle if there is no proof to be

generated or or if transaction volume are low that's burning a lot of uh runway for the different teams. So basically the paper proof models that are available through prover marketplaces or prover networks or even some um centralized proof providers out there um they come in very handy. What I would add on top is that, and this may even be a larger chunk of the cost, but saving the idle time for all these teams may actually be a larger saving than just making proving the cost per proof cheaper. So, that's an interesting dynamic there.

Interesting. I feel like we should name some of the actors, sort of the roles that exist in this. I'm going to call it supply chain or chain of actors that are creating proofs. We can use maybe the L2 as an example. So we've mentioned this like sequencer and prover. In this case, the way that I think this is often described in the prover network context is the L2 is a proof requesttor. It needs proofs. In the case that we just described where that L2 is running its own provers, it is also the proof provider or the prover networks though kind of live in between that, right? So you have proof requesters, could be an L2, could be

something else, and then you have this prover network which can create proofs for the proof requester. But they're often as as I understand it almost like bringing in these prover machines from different places in order to satisfy. Is that how you would define a prover network? Yes, I think that's that's more or less what a proven network is like where they source the compute capacity from. That may differ though uh pretty significantly because in that sense I' I'd rather maybe uh call the bucket proof providers or like proving services. Okay. And as you said, so there is the proof requesttor. Um there is a proving service which needs to

source compute from somewhere and then uh the supply chain progresses with the verifier uh and the settlement of the uh of the proof itself. But regarding proof networks and and marketplaces, I mean the marketplace itself already hints at anyone who has hardware either through renting or having their own clusters on prem. they could come in, they could submit bids in the marketplace to win the right for different proving jobs. But then in a prover network that's a little bit different I think because um and it's particularly because the prover networks may have different workload

allocation mechanisms. And what I mean by that is a prover network can be decentralized and permissionless allowing anyone to come in and contribute compute and then still use an auction mechanism to win proving jobs and to distribute the different jobs to um certain providers um certain node operators in the network. However, with for example Gavlot and Zigg cloud, we actually build our network in a way that we randomly assign proving workloads to proven node operators. So in that sense it's it's a permissionless network that we are building. Anyone can join who meets the minimum hardware requirements

and then the randomness based selection ensures neutrality, minimizes centralization risks, etc., etc. what's there and what's important I think for all these L2s even though most of them are running in a centralized fashion now and exploring the way to decentralized both sequencing and the proving on the other hand it's going to be a very important decision I'd say what kind of proof provider they select because the architectures of those providers may come with significant tradeoffs in certain cases the proof provider the way you've defined behind it. This could be a prover network. Could be also a direct prover, right? Like proof provider is

anyone who's creating proofs or proofs are being generated somehow by this entity. Yes. But when we go to the prover network like what do you call the provers that feed the just the prover network? I usually call them prover node operators. Okay. Okay. They are the ones with the actual hardware plugging into the prover network um and contributing the compute. Got it. The prover node operators like I have heard and I think maybe we're far off from this but I've heard like the proposition that one day maybe you anyone can be a prover node operator. You could prove from your phone. You could prove from your laptop.

It seems like and also because we're in it, we're experimenting with this as well. So right now it seems like we're pretty far off from that. Like the prover node operators tend to need to actually have pretty significant hardware and like you're you're not running this on sort of like a random laptop at this point. Yeah. So I think that there are different approaches and it's absolutely possible that there will be certain teams who will take the approach and allow any type of consumer grade hardware even you know browser based proving or mobile phones to come in and contribute compute. The approach for example we've taken with ZK cloud is that uh we have to have a a certain minimum hardware requirement to be met

because that's part of our optimization strategy uh from different angles. Since we use randomness to distribute workloads in our network we need to ensure that um everybody has a uniform minimum capacity. you know if you use a let's say um a consumer grade laptop and I have a huge cluster of you know we cannot be in the same random selection in that sense so it's important for us to have some uniform uh capacity for all operators in our network but also on the other hand it helps us to break down costs and to um you know further disperse the cost of proving because one large machine could actually process

multip multiple totally independent smaller proving jobs um you know further dispersing the operational costs on the other hand also it may happen that you need multiple of those large capacity machines even to complete certain jobs for example Ethereum block proving and when you want to bring it down to uh like Ethereum's block time you need maybe 50 or 100 GPU nodes at minimum to be able to get to that level. So I think the approaches are different. There will definitely be uh networks there. I think Nexus um recently had a couple of test nets where they were actually exploring

people contributing through browserbased proving and that's an interesting story obviously the architecture for that and how they orchestrate workloads you know breaking a larger workloads into tiny chunks. Yeah, you have networking overhead there and then aggregating all those tiny proofs generated in a browser or on different machines um to come to the final single proof. I think that's um that's a different challenge. So basically the different approaches have their own architecture challenges and their own viability. I think I want to kind of continue on that like reframing of what the prover network or

prover marketplace like the role that it plays because the one I des like what I described kind of at the start was this idea of this prover network which had a bunch of like you called them prover node operators. So different teams maybe with a standardized prover capacity all running these nodes and making proofs and then I always pictured that like the prover network would be servicing like multiple networks like multiple L2s for example multiple proof requesters but we've seen a lot of the prover networks come online or like at least be proposed with like one proof requesttor customer in mind sometimes

from the same company. So you'd have like I'm a ZKVM. I'm creating lots of proofs. I'm also going to run a prover network where other people can help satisfy those proofs, create those proofs for this proof requester which is me. And so there's almost like this link between those two things. This was something that I felt we heard a lot last year. Would you say teams are moving away from that or moving towards that? like how is that relationship changing and how does how is actually like ZK cloud maybe a little bit different from that that's a very good observation and it is indeed true that most of the prover networks like yeah if we step back maybe 18 months it was autumn 2023 I think um that's when the

initial architecture design of of glo was released and basically largely it kicked off kind of like this proveret craze where early 2024 different teams started um coming in and announcing they're building a proof market or a prover network. I think in March yeah it was March last year when uh we deployed our first version of the network. Basically this wasn't a scalable network but it already allowed universal proving. And by universal proving I mean is that anyone could come in and deploy their own prover program on the network and have proofs generated. Currently I mean this is just a side track but if I

may uh mention that currently universal proving is used for you know teams supporting different provers and whatever they integrate proofs can be generated on their networks for those. In our case, universal even goes a bit further because we allow you, we don't just support any proof system from our perspective, but we allow anyone to bring their own prover program, deploy it and generate proof. So that's very unique in that sense. But coming back to the initial topic, sorry I don't want to distract too much. Coming back to the initial topic, I think it's a very interesting observation. So many of the teams that had some kind of product like

log ranch uh their co-processor initially or succinct was building the the zkvm or risk zero zkvm um all of them started to offer proving services for risk zero it was bonsai which was basically a centralized service they've provided but then others started um announcing different poor networks and I think it's a very viable thing that they want to you know and it's a very natural thing that they want to service those people who come in with demand uh that they want proof generated on those CKVMs or on those products. Now some of these teams were actually

planning to go more universal and open up proving and support other ZKVMs or ZKVMs potentially. Uh but I also know of a couple who actually decided to go more focused and optimized for what they what their own product is. So it's kind of like uh I think we've got the proven networks that have been launched by teams that have some kind of product and they may be a little more focused on satisfying the proof demand for that. And then you've got some more universal uh networks supporting all different ZKVMs and ZKV EVMs out there. I think with Kevlot and

ZK cloud, we went even one step further um where we just offered anyone to package and deploy their own program easily and then just start generating proofs on our network. So it's basically just as simple as referencing the prover ID that they deployed on the network, sending the inputs, defining the the amount of hardware to be allocated to that particular job and then off it goes. Um once the the proof has been generated, they can they can fetch the proof. So it's even one step further and I think that's a that's a key differentiator there. But yeah, the dynamics are interesting on how uh how the different prover networks have been

evolving and also I think the marketplaces seem to be interesting because marketplaces are in my head a marketplace you know it includes the word market so it's naturally an auctionbased platform. Basically everyone puts up an offer for how expensive a proof would be and then the proof requesttor can choose. Yes, exactly. So, it's a very good point you bring up and I think it's very interesting how also the the different workload we call these workload allocation mechanisms work and I think that's also an interesting topic. Maybe we could touch up on that later. Uh proof mining which is a little bit similar to to Bitcoin mining where

you've got a race, everybody's racing for the job and whoever generates the proof first gets rewarded. However, this is never going to be the cheapest option because you know someone has to pay for all that uh redundant and I could even say wasted computation involved in the process. Then there are order book based mechanisms where there are basically bids and asks. How much do proof providers ask for generating a certain proof and how much proof requesters are willing to pay for a certain proof? And when there is a match then uh the workload is allocated and then uh the

proving job can start. Obviously there can be situations where there is no match for certain amount of time and for fairly time constrained workloads that's not necessarily the best option. Then you've got these different auction mechanisms uh where it could be a simple reverse auction where basically there is a proving job that you want to have proven and uh you accept bids from operators or from different prover networks with a quote for how much they are willing to actually generate that proof. And usually the the proving since it's a compute heavy task and it's not a

cheap task to be performed it's very costsensitive and actually that was one of the key reasons to start developing Gavlot back then. So the founders of Gavlot wanted to have one a very cheap proof provider an order of magnitude cheaper than the traditional web to cloud compute providers. Plus they wanted to have a decentralized alternative because we've seen what happened in web 2 with the the comput space. There are very few large providers that dominate the space and this is something that um since we are in the blockchain space we have this ethos of decentralization and all the benefits and and advantages that come

with it. landing with um the same centralized setup uh in terms of you know compute heavy tasks would not be an optimal scenario at all. So that's why building a decentralized network that is also providing super cheap proofs for any user because we've seen that um the cost of proving it did not make projects economically viable for quite some time and as soon as we see that we hit maybe an order of magnitude drop in in proving costs those projects and you know implementing ZK in those projects actually becomes realistic and you know it's not a bottleneck anymore. more and then there is randomness coming back to

the workload allocation mechanisms and and this is how you guys do it right yeah that's how that's our the decision that we have made we've seen what happened with the snark marketplace also we know that um like we've analyzed a bunch of different um auction mechanisms starting from a very simple reverse Dutch auction to more complex mechanisms unfortunately whatever sophisticated architecture you manage to figure out 80 90% of the allocation still depends on the cost of proving and so that's a dominant factor and until we are not able to solve that auctions are very easily gameable meaning any large entity uh with a lot of resources could come in

and just under bid everyone else and make it unprofitable for them and basically run them out of the market and then be able to put their prices up. Yeah, exactly. So an auction and that that's a very key point I think an auction can be a way to find you know to let the market find uh the cheapest cost for uh proving but as soon as it's gamed and some dominant entities emerge they can actually introduce unhealthy economic dynamics and start influencing the cost of proving a lot. Obviously go to market is much easier if you let the market decide the cost of proving because you don't need to build and

figure out the entire economics yourself. We've taken a very different approach. So we wanted to be very independent in terms of uh any underlying economics. That's why we went with a very minimal Cosmos-based L1 that is managing our workload allocation and the economics. And basically this way we had a clean slate. We don't depend on the economics of Ethereum or an L2 or any other network that we would be building on top of. So with this L1 we've designed the economics in a way that it makes sense for node operators and it ensures profitability but then it provides super cheap proving for all the users. And um with randomness the extra

benefit that you get is you can introduce actually backup mechanisms and fallback mechanisms into your protocol. While with an auction for example if I win the auction and I don't deliver the proof then you need to start the auction process all over. So it's it's very hard to build by default these fallback mechanisms or you know backup mechanisms because you just need to start it all over. It's a onetoone match one requesttor matched to one proof provider. But with with our network and the the with randomness being used for workload allocation, we can actually initially select more than one prover kind of secondary or you know third

provers, backup provers to come in and then we can also as as a last resort open up the proving job for anyone to come in from the network. First come first serve. Whoever delivers the proof first uh they will get rewarded. So in a sense there are multiple layers of execution guarantees that can easily be built on top of this architecture that we've designed. So I think longterm this is the the safe solution for L2s or you know for proving jobs where actually the finality of the chain or the progress of the chain depends on those proofs arriving. So where livveness is the most important. Yes. Exactly. This is

interesting. And so I mean you've defined these different kind of economic approaches, the proof mining speed, the order book, the auction, the random. You are identifying the random one as this is potentially an ideal case for livveness if it's very critical that like you can't have a case where the proofs don't come in or there isn't relevant party ready or they're down or something like you need the redundancy. But are there other cases where an auction is makes more sense or where an order book or a proof like sort of proof mining uh model makes more sense? I'd say the mining um wouldn't it be cheap for the end user? No, I I think actually it would be more expensive

because okay with with mining and with all those parties racing and only one getting rewarded, they would expect those rewards to be higher and at the end of the day the it would be the user who is paying you know um those those higher fees. So I think okay that would land with with a quite high uh proving cost. Auctions are a good way um to let the market find the optimal cost and maybe it can you know be a good tool to find kind of low proving cost as well.

On the other hand, we like our bet is that as soon as you can design an economic model that is cheap enough, you can uh just bake it into a protocol which ensures that nobody can actually influence the cost of proving over time. So you bake it into the protocol and you can just do a credibly neutral selection through through randomness basically. I think overall I do believe that randomness would be the the right long-term approach even though parties are not necessarily rational. So let's say you've just you just have a cluster of GPUs for example and you want like in

a rational world you would want your operational costs to be covered plus earn something but very interesting thing is uh and this is what I heard about alo proving for example when Alo was on testn net actually many different operators were doing proving even though they were only partially compensated for their operational costs. So the market is not necessarily or or the operators are not necessarily rational in that sense. What we have been focusing on is to build a network that is suitable and optimal for maybe 98% of the use cases.

M and for that uh we bake into the protocol you know the the characteristics of decentralization and permissionlessness all the high livveness sensor resistance etc that comes with it and uh by designing user focused economics we ensure that they get cheap proofs so I think that's a more long-term approach rather than a quick go to market and then let the market figure out the economics and the cost of proving For the proof requesttor side of things though, wouldn't it be cheaper to do the auction for them? For the proof requesters, not necessarily. I mean, we will see uh as soon as um some of the

teams are planning to launch test nets. Some of them have already launched uh their test nets. Um we are a couple months ahead of the of the industry, I think, because we launched our version one network first. So basically I'm looking forward to seeing all these quotes and you know where the cost of proving lands on the different marketplaces versus for example our economics. So currently what we see is that we are about 95% cheaper what compared to for example AWS or GCP which is which is great. On the other hand, um technically an auction can be cheap, but the risk of not lending with the

cheapest proof is always there because you are dependent on who bids to your request and what those bids are. you can then wait for a cheaper offer. But then you know if you are not time constrainted then maybe you can wait 10 minutes or 30 minutes more or you know maybe even just one or two minutes more. But if you are time constrained and you need you need to have a proof generated by some time to ensure your finality then maybe you just need to go what you have there as bids and it may not be the cheapest possible option at all specifically and yeah we haven't talked about what if a dominant entity starts emerging and you know starts controlling

those offers in a large portion of the of the incoming proof requests and I think uh one important thing regarding this is predictability of the proving cost. So for example, for an L2 to have the same proof for X amount of dollars and the same proof an hour later for double the price is not necessarily the most sustainable approach. But that's what they are going to have to accept if you know in an auction there are no other biders or there is no cheaper bidder. While with ZK cloud, you see we have the hourly cost of our machines, 84 cents for a for a double GPU machine,

double490s. And then if you only need one of those GPUs and you need it for 3 minutes, then that's what you pay and that's going to be, you know, it's baked into the protocol. So that's what you pay today and that's what you pay in a month's time for the same workload. So in that sense kind of the sustainability or the predictability of the proving costs is a bit different based on the different you know architectures as well or design decisions as well. I like that you just defined those different kind of options though. I want to talk a bit about hardware and I mean cuz the minute you're talking about provers and you even mentioned sort of proof mining hinting mining proving it's compute intensive. It's not proof of stake where

it's like a game. I mean, a lot of the prover networks might have a proof ofstake validator set underneath which are more like an economic game with stake and and what have you, but proving does seem like a bit of a throwback to mining except that you're doing useful work, right? Like you're you're running computation with the goal of creating a thing that is actually being used. Whereas mining is you're running computation on sort of useless puzzles in order to win a competition that secures a network, right? But there's no like the the output of that computation is not used towards something other than the security of the network. Tell me if you think like is proving I mean we already hear about AS6 and and we've

done actually episodes from like the ZK prize with some of the ZK prize winners um where we talked about like accelerating FPGAAS and getting A6 in the in the works. So yeah, how does how does hardware play a role and how is this related or like is it similar to mining? Are we kind of recreating a mining situation? I think you're right there because proof generation is is kind of like a hybrid workload where you do need a large amount of compute for you know for most of the large workloads obviously for for tiny ones not necessarily but like for for proving an L2 proving a block or a batch of blocks um you need quite significant hardware sometimes tens or or maybe at higher

TPSs even hundreds of of GPUs or or machines. So in that sense it does resemble mining and some people also started calling proof generation kind of proof mining. Mhm. Both require substantial compute resources but these resources and the compute itself serves very different purposes. So for Bitcoin mining the mining uses proof of work to to secure the network and to achieve consensus. So redundancy is kind of the in an integral part uh of network sec security basically and you pointed that out very well. I think for proof mining we generate these validity proofs or zero knowledge proofs uh to make any

data or computation verifiable and redundancy there increases the cost significantly. I don't want to uh repeat what we've talked about earlier, but it actually comes back to proof racing ending up being one of the most expensive ways to do proving even though it could make sense for certain use cases where you cannot even afford the time of a secondary prover to come in and do the proving for you in case the first one fails. So if you want to have redundancy and actually uh customizable redundancy is also something that we have in the network meaning if you want to be super sure and you know uh receive

the proof uh at first shot then you could ask two or maybe even three different nodes to do the same job and if some of them don't deliver that's absolutely fine the others will serve you the the proof that you need. So that's something that we also offer in our network. I think that's that's one of the execution guarantees for those very sensitive um cases. So proof racing or proof mining would be quite an expensive uh approach. I think uh and also then a very interesting thing would be redundancy versus economic security because if you don't have redundancy then how do you know or how do you

ensure that the selected prover the one prover that is working on your job will actually deliver and this is like we are already I don't want to distract and and I'll come back to hardware in a sec but that's also a very interesting thing so by default you could use stake and then slash the prover if they don't deliver. There could be a financial bond maybe even but then you already have economic commitment in the form of of the compute hardware which may be actually very expensive. So you know the entry barrier for anyone to come in and join proving may be extensively high if you add staking or like a large financial bond

on top and um you know there could be ways uh to come about this for example we haven't really seen reputation systems we haven't really seen performance or reputation based rewarding for example or proving capacity and availability constantly on node operator s and if you know that they do have the capacity and they are available then it could make up for part of the the economic security. So a combination of these I've been thinking a lot about this and uh we've been chatting a lot about this internally as well. So how can you ensure that the proving market is accessible for people and not overly you know not get

overburdened by different economic commitments that they need to make but on the other hand you also ensure liveness you ensure proof delivery because the finality of a chain may depend on that. Um yeah it's a it's a very interesting topic I could talk a lot about that sorry for for for distracting the topic. I want a quick clarification though because you're saying that like the ZK cloud model you're not doing full redundancy it's not like everyone it's not proof racing you're not all running for one proof but how many machines then like cuz you but you also have redundancy built into it so is it almost like a percentage of the network will be running for any particular proof like how do you choose how many by default we allocate one but

if your proof request is actually very sensitive then you could configure your request in a way that there can be two or three allocated initially and doing the same job. I see. So it's it's up to you. Obviously it will increase the cost. Um but yeah um so it's it's something that's possible for you in the network. Yeah. But it's a very good point regarding hardware. I think that's also a super interesting topic and the impact of hardware on the proving industry because currently what I see is there are some teams like Polygon ZKEVM or Aztec Aztec's Barrettenber prover that use purely CPUs but we do see that

most of the ZKVMs and most of the other um uh prover programs they use GPUs and there are again some teams that actually bet on GPUs being sufficient and they do kind of software hardware code design focusing on GPUs and the improvements coming uh over the next generation of GPUs. But then there are some teams um I think SCIC or irreducible could be one of them. They're more hardware oriented.

They're coming from hide hardware. Yeah. Yeah. who have built FPGAAS for example and these these types of things on the other hand it's not easy uh because you would need I think alo A6es for ALO mining you know you cannot reuse it for other purposes basically and that again may be a bottleneck being more universal may mean that you can accept jobs from you know for very different uh proving workloads so your hardware utilization could actually be much higher in a sense. So there could be advantages and tradeoffs, but but there is the one in the middle, the FPGAs. Could you use those? You could, but then you may land

with a little bit less performance than than A6s. On the other hand, the one of the most interesting developments is, for example, fabrics, VPUs, these verifiable programming units. Those are, as far as I know, they should be coming out in the next few months. maybe even less. And those are basically custom silicon that is purely optimized for all those arithmetic operations included or involved in in ZK proving and everything being processed on these custom chips.

uh because when we talk about GPU acceleration, usually most of the operations run on the GPUs, but there are some that are still requiring certain amounts of CPUs and then the communication and the the the memory requirement. There can be different constraints there. But with these VPUs, it seems that they are able to process every single operation on these custom chips super fast. and the cost performance ratios that we are seeing if those benchmarks that have been published do hold they can shake up the space quite a lot I think okay uh because I can mention maybe a couple of numbers for example compared to

4090s one single uh VPU card compared to 1490 could increase performance by five 6x for many of the risk zero operations like the operations that the risk zero ZKVM uses. So, and as soon as we start getting there, the operational costs, the cost versus performance, it's going to drastically change if indeed those benchmarks hold. Yeah. So I'm really curious to see uh when they come out how they really perform on actual real world workloads like like I've been talking to Jeremy in CEO of Risk Zero in uh Denver and he said they are making 3 4x

performance improvements purely on their software every quarter. Wow. Now you add a 5x or maybe I don't know 3x 5x 10x who knows improvement on the hardware side on the hardware. Exactly. Then for example, things like real time Ethereum block proving become really tangible and really a reality even now or maybe in the next couple of months what we couldn't have imagined maybe a year ago or even 6 months ago. Are there other hardware companies? Cuz like I remember at ZK11 Justin Drake announced and shared I think he had it like a a ZKP ASIC but I don't know it was by Cizzix I

think but I don't know like what what was that for was it for a particular proving system? Unfortunately I don't remember that by heart. What I do know is that uh uh these VPUs for example are reprogrammable and you can add multiple different proof systems uh to be supported natively. So that's kind of like the performance of an ASIC with the programmability of an FPGA and that's going to be a gamecher I think. Interesting. You sort of mentioned the different proving systems. This has kind of gotten me now curious. You you said that because it's somewhat universal, you can almost like deploy the proving system of your choice and then provers

will prove the way that you need to prove for that proving system if I'm if I understood it correctly on ZK cloud. Yeah. So what are the differences there? like maybe just share with people like what what is the difference between a plunk I guess a plunk system or groth 16 system in terms of proving what are the different things happening and I guess if you're mostly using GPUs you can do all of them because you can adjust but the minute you get into these what did you call them vpas or something vpu um the customer what's that stand for actually um verifiable programming unit if I'm not mistaken okay okay but yeah like would that interfere with that flexib capability. That's a very good point. So yeah, in our network basically

obviously we integrate most of the well-known and you know very popular provers for example you know ZKVMs that are out there or the ZKVMs support for different L2s like ZK sync polygon linear um Aztec and scroll and all those others. On the other hand, we do not want to be the bottleneck ourselves in terms of do we have the capacity to integrate your custom prover if let's say you took some uh existing one you made your optimizations and you want that um to generate proofs for you at a third party provider because you don't want to rent the hardware or you

don't want to build your the hardware yourself. So for that purpose basically we built a very flexible network where as long as you can compile your prover into a a Linux binary um and as long as your prover runs in a container you can very easily deploy it on galot on sorry ziki cloud and um and just send proofs to be generated. So basically coming back to your point um we are going to support VPUs and obviously um when it comes to prover node selection and like workload allocation we are going to take into consideration which prover node is able to support that workload and

generate the proof for you. And actually the way we work is that when you submit the workloads to to ZK cloud, you need to define the amount of uh CPU cores, the amount of memory, um the amount of GPUs that uh you want to be allocated to your job. So I guess in the future there would be another parameter. Do you want VPUs to be allocated to your job? And then you would know as a requesttor that you want GPUs and that performance is sufficient or um your your prover program is supported by the VPU. So you can even have a couple of cards thrown onto the job and have it generated even faster and maybe you pay a little more

for those particular jobs. Yeah, it's it's going to be interesting to see where we land in the cost because Oh, it's faster. Exactly. So we might think that it costs more because those are custom silicon, but then if you only need 1/5 of the time or one of the time, you actually may end up very um cost effective there. Interesting. But are are there other teams doing these things? I think as far as I know, irreducible is planning to build a custom silicon. If I'm not mistaken, they already have FPGAAS, but kind of like a programmable chip, ASIC type chip for ZK. And also I heard about Ponos Technology, uh, a team based in

Switzerland who are looking to build something like that, but they are super early if I'm not mistaken. Okay. Interesting. There's one part of the sort of proof journey that we haven't really talked about and I I don't think it's something that your project really deals with, but I do want to just sort of like wrap up that journey. We talked about sort of the proof requesttor. So, this is a an entity that needs proofs made. Could be an L2, could be a ZKVM, could be an application that uses ZK. They need proofs. We have the proof provider as you you've defined which could be a prover network prover marketplace or it could be an individual prover machine going directly to create that proof but then what happens to the

proof like somewhere it has to be verified I guess like what we've sort of talked about the first half of this journey but yeah where where do they go and is it actually the prover network or prover machine that sends it there I think there can different approaches there. So if you take the example of an L2, then the sequencer of an L2 who builds a certain block and needs to has to have that proven. So they they commit the block to the to the L2's smart contract on Ethereum, for example, and then they send this workload to to a marketplace or a network. they get a proof generated and then the proof has

to be sent to the verifier contract also on L1. If it verifies correct and you know the the state transitions uh in the proof match the state transitions in the initial commitment then it gets settled and and finality is reached. Now there could be different ways that L2s want to interact. Maybe some of them want the proof uh to be sent back to the sequencer and then they send it to to L1. Other teams may want the proof to be directly sent to L1 to be settled. In that way we basically our initial design is uh you send a proof request and as

soon as it's it's ready you can download the proof but we are also uh working to be able to support um like sending the proof directly to for example a smart contract on Ethereum. I mean that that could be also done through um to an adapter maybe. uh but yeah there are different approaches and and that's basically the last step and I'm not very deep into verification and like the proof aggregation part because many of the teams say there are teams that focus purely on that but I'm sort of trying to figure out like would you then send proofs their way and then they aggregate it into single proofs and then verify it like as a batch is that sort of how one

of these flows could work yes I think that that could be one of the flows where they receive multiple proofs. So these proofs don't go to L1 directly. They receive multiple proofs at the at the proof aggregator. Maybe they aggregate 10 20 of those different types of proofs into a single one and then have it settled on on Ethereum or some other network. Now the interesting thing for aggregation is that you may also need compute capacity for those jobs. So a proof aggregator in theory and we are also um exploring this with a with a few of them they could actually outsource the aggregation workload to some network like for

example ZK cloud we are very well fit to do that. Uh so in that case the proofs to be aggregated would be input would be the inputs to our network. Oh weird. we do the aggregation uh with with you know um all the compute resources in the network send the final one to them which they can settle but I think there are other approaches as well um whereas some networks may actually verify the proofs themselves kind of offchain and then just send a commitment to Ethereum saying yes there exists a valid proof uh for this state transition and sending the commitment might be much cheaper than actually sending the proof

and you know having that verified on L1. So there are different approaches but I don't know a lot about that aspect. I feel like I'm due to dive back in with one of those teams like one of the proof verification teams because like the first time I heard about it was when Nebra came on and originally Schumo and Nebra had been more of like a prover network and then had moved sort of up the stack to be more of the the verification aggregator and that was sort of the first time we started talking about that. He also was I think the person who coined the ZK supply chain that I've sort of hinted at here with the different actors. Um but his focus is more just like proof requesttor proof generator verification and focusing on that third step. But yeah,

it seems like there's there's a lot of teams doing that right now. I mean I remember for ZK Summit for one of the ZK summits we got a bunch of applications from and they were almost identical and I was like wow what's you know what this boom of verification folks. So yeah, it's probably due to I'm probably due to to jump back in. Yeah, there are a few very good teams that are building that. Cool. So I think we have covered this through the the episode, but I do want to just sort of wrap on the why of the prover marketplace sort of summarizing what we've shared because I I think you have actually said what the benefits of it are, but like why decentralized this proof provider role? what are all the

benefits that one can get from them? I think we might have talked about this partially earlier but usually decentralization serves two purposes livveness and also security but in case of ZK security is basically a given because of the math the underlying math so you don't necessarily need decentralization for that aspect but to strengthen livveness and to streng strengthen properties um such as for example censorship ship resistance or I could imagine a situation where some block of some L2 includes transactions touching certain you know

accounts on different sanction lists whatever so there could be a censorship element there even though may not be very likely it may rather happen on the sequencer side as soon as it's decentralized because they are the ones building the blocks but uh as we have seen with um with Ethereum's block production sometimes it's the builder censoring sometimes it's the relay that's censoring sometimes it's actually the block proposer uh the Ethereum validator that is censoring so in that aspect I could imagine something like this happen and because of that decentralization is not important for the sake of decentralization but really for for what additional benefits it

brings in terms of high livveness sensor resistance We've we've talked about these. It's a very important decision for any project that wants to outsource proving what kind of provider or you know what kind of way or path they choose when they are going out um selecting uh a provider. I think Zach from Aztec has been vocal about this uh maybe a month or two ago saying that you build a nicely decentralized network but then any proof provider out there should actually have the same properties permissionlessness decentralization censorship resistance or else they are going to introduce tradeoffs to the

network that you have built and I think this is a key element uh that has not been talked about a lot we've recently released a blog post on that because exactly for that reason did we build ZK cloud as an independent L1 um as a permissionless network with the most neutral workload allocation mechanism in the form of randomness you know with all the all the properties for minimizing the risk of any type of centralization as much as possible minimizing the risk of control over proving fees through baking economics into the protocol So it's kind of like it was very good to hear him talking about this and being vocal about this and a year and a half

ago when the initial design came out it was actually designed in a way that it facilitates what he was talking about to ensure that networks don't need to accept tradeoffs actually there exists a solution and there may be others as well later on uh I don't know that where you don't need to accept tradeoffs in terms of your decentralization or uh the other properties. I don't know if I could summarize that properly, but yeah, that that would be my kind of like my answer. Cool. Well, thank you so much for coming on the show, Norbert, and exploring this, you know, evolving proofer marketplace space. Um, it's been really fun to explore that, to tease it out. You can

tell like in talking about it this year that there have been findings, there have been some understanding like some some experiments have been run or at least have been proposed and now people are really thinking about these setups at a deeper level. So, it's been really fun to talk about it. Thank you very much for having me. I've really enjoyed the chat. Cool. I want to say thank you to the podcast team, Henrik, Rachel, and Tanya, and to our listeners. Thanks for listening. [Music]