Episode 247: Exploring Penumbra with Henry de Valence

Transcript

[Music] welcome to zero knowledge I'm your host Anna Rose in this podcast we will be exploring the latest in zero knowledge research and the decentralized web as well as new paradigms that promise to change the way we interact and transact online [Music] this week Guillermo and I catch up with Henry de valence from penumbra in this we discuss his thoughts on the requirements for adoption of privacy systems how these ideas led him to develop penumbra we then dig in to help a number aims to use privacy features not as nice to haves within a system but

rather as essential components that offer a new paradigm for how to think about defy in a multi-chain ecosystem quick disclosure the ZK validator was an early investor in penumbra but before we kick off I want to share an announcement from one of our recent ZK Summit Partners anoma anoma has released a new white paper that better describes their system and architecture it also shows how all the awesome cryptographic libraries they've been developing fit together find this paper at inoma.net papers We'll add the link in the show notes and be sure to check it out I also want to highlight that there is a fresh batch of jobs over on the ZK jobs board if you're looking for a new opportunity to work with the best teams in ZK be sure to check out their job postings I'll add the link in the show notes as

well now Tanya will share a little bit about this week's sponsor today's episode is sponsored by Elio Elio is a new layer 1 blockchain that achieves the programmability of ethereum the privacy of zcash and the scalability of a roll-up if you're interested in building private applications then check out Elio's programming language called Leo Leo enables non-cryptographers to harness the power of zkps to deploy decentralized exchanges hidden information games regulated stable coins and more visit Leo lang.org to start building that's Leo dashlang.org you can also participate in Elio's incentivized testnet 3 by downloading and running a snark OS node no signup is necessary to participate for questions join their Discord at

alio.org forward slash Discord so thanks again Elio now here is Anna and guermo's chat with Henry from penumbra [Music] today we're going to be talking about penumbra with Henry de valence I'm here with Guillermo we're going to be covering this project the day after ZK Summit 8 we're all a little tired but I do want to welcome you both to the show thanks it's great to be here thank you that's great to be back so Henry you gave a great presentation yesterday at the summit I want to talk about penumbra but before we do that you've been on the show before and there's been I assume like quite a story since then it's we recorded in 20 like April 2020 I would

love to hear a little bit about like what has happened since then what have you been up to uh yeah so I think at the time um I had been working on one of the projects to do privacy preserving contact tracing and that eventually I think got subsumed by the Apple Google version of that protocol I went back to working on zebra which is the zcash foundation's zcash full node implementation and then towards the end of uh 2020 I sort of felt like it was time to kind of move on and have a little bit of a break and then in in the first part of 2021 sorry the years of the pandemic are

all like mashed together but in the in the first part of 2021 I had some time off you know there's only so many Hills that you can like hike up before you go back to thinking about blockchains and then that's kind of how I started thinking about the problem space that eventually evolved into penumbra and kind of reflecting on the experience of spending say you know four years working on various kinds of ZK tooling infrastructure trying to build all the necessary technical Legos to build the kinds of private systems that I wanted to exist in the world and yet at the same time looking around at the ecosystem and

seeing well there's like all of the projects that people actually use and then there are the private projects and this Venn diagram has like an uncomfortable resemblance to just two distinct circles um and as somebody who's right like if you if you spend all of your time like actually like you know you're sort of pouring your soul into these technical artifacts the idea of oh I'm I'm just kind of eating all this stuff out into the world and it is never really getting used um it's not not a great feeling and so um so yeah I had a process of of thinking through sort of what would it look like to have some kind of theory of of change

for for how people would actually start using private blockchains and what is the reason that people aren't using existing privacy tools and is there some way that you can kind of hook these things together and find something where you get like actually good Market fit for a privacy product and what I mean by that is it's not just oh I'm I'm trying to build this thing and it's private and people will use it because it's private like how do you break out of this mode of oh I want to build a private X which means like it does this is a thing it does X it's like

kind of worse at it but it's private so like maybe people will want to use it right like that's not really a compelling product pitch how do you get to you know we're building a better X and the reason that it's better is because it has these privacy features and we don't have to spend time arguing about like oh is privacy a human right you know I I believe in that but I don't want to have a I I don't think it's workable to build technical products that are like based on on that as a pitch you have to build something that's actually like better for what people want to do it sounds pragmatic it's like looking at Human Nature we can wish that people cared about privacy or we can

make it so useful so so much better to actually use these private systems anyways yeah I mean I I think that's I would push back on that framing a little bit in a way that I think is hopefully interesting um which is I think when you look around there's actually a lot of evidence that people do really care about their privacy but when people make choices like this is the whole reason that I think like the idea of like revealed preference is a lot of the time what do you mean by that so revealed preference is the idea that like oh if you want to know like what somebody's preference is then like instead of asking them for their opinion you just like look at what they do because that's sort of like revealing their like true you know

choice or preference or whatever but what that misses is is that when people make choices they make those in some specific contexts and the entire like technical infrastructure of our society has been carefully crafted so that those choices cause people to choose to do things that benefit various corporate entities right like the choice of oh do I want to have Google reading all of my emails right if somebody's just asking me that you know abstractly of course no I would rather not that they not do that but the scope of that choice in Practical terms is do I want to never

communicate with anyone via email because every other person that I would want to talk to already has a Gmail account and so they're going to be getting all of the email anyway so I don't actually have a meaningful choice about my privacy you know based on sort of what email provider I use or something I mean there's a whole thing too right like people have been trying to self-host email lately but because of like a bunch of reasons like spam first but also secondarily like Google Microsoft Etc will just like reject your emails out right if you're self-hosting unless you do like jump through all of these like crazy hoops because otherwise it just like assume it's spam so even like anything not to be like you know the

Microsoft Defender but like also likely it is right like like there's also an interesting story that I think is relevant to to blockchains or other decentralized systems about email where you have this sort of purely decentralized protocol where there's no real kind of economic model and then you have this spam problem which you can think of as sort of like this resource usage abuse issue and the way that that has actually been solved is that email is a centralized system that is like dressed up in a decentralized protocol Google just reads everybody's emails and figures out what the spam is

and then bends it and that's why email works and and I think part of the challenge of building a decentralized protocol in general is how do you account for enough of the economic logic of the system internally to the the logic of the system itself so that if it's successful you don't have a situation where some monopolist has to come in and close that comments and then centrally manage the resources to like take account of that sort of lack of economic model so I mean I think the problem with like email and it's naive Incarnation right is now this is just a restatement of your points but

there's a mismatch between economic incentives of like Bad actors and like economic incentives of good actors right like you are it is very easy for someone to kind of like maybe maybe that's a weird term but maybe dos the comments um simply because it is like very simple very cheap to do so right while the cost burden of that kind of externality right it's very expensive for a user because like sifting through you know 5000 pages of like spam is usually like particularly costly for any one individual yeah I think that the the phenomenon is more General than that though too because another example that I think about is GitHub where you have git which

is this decentralized system a decentralized protocol and although it is uh at the protocol level decentralized in practice the way that git works is that everybody has a GitHub account and GitHub hosts all the repos and I think part of that story is that the social infrastructure of doing code collaboration requires like various kinds of technical infrastructure like hosting like issues having some kind of system to keep track of proposed changes that isn't just like people emailing each other on a mailing list and because those aren't part of the system

they end up being provided by Microsoft who pays to maintain all of that infrastructure so that they can sort of have ownership over this like developing community do you feel like that to and obviously there's been proposals to decentralize that but it's just so enormous like a an Endeavor that it's it's very slow to happen I think it would be good probably better you know if if somebody makes like a better thing that would be great it is a lot of work um I know that like for instance like radical yeah is trying to build um some alternative but rebuilding like social infrastructure is a lot of work

you know that's this whole tangent digression that's hopefully interesting to the listeners but where we got onto this right was this idea of how do you build a system that can provide private infrastructure but with some kind of self-sustaining feedback loop that actually drives people to want to use that system as a as a product privacy is actually really interesting in that it has very strong Network effects right the the larger the anonymity set is the better the privacy or even like the more people participating in private systems even like say it was like email or something then you don't have that kind

of like or maybe it's the opposite if there's more people participating in the very non-private it reduces the privacy of all even those who are being careful about it so I'm guessing the opposite is true if more people are careful about privacy then generally that privacy not even the set but just like there aren't the level of surveillance that you would have otherwise yeah right so by the way just a note to The Listener we are recording this as mentioned the day after the ZK Summit eight so if we are slow AKA if I am slow I apologize so yeah but but also more generally right like the more people that use a system which is anonymous by default the harder it is a pinpoint any one individual person

so this is this is another notion of like anonymity as well so that was one kind of line of thought so we'll just sort of pause that and we'll see how these intersect in a minute the second line of thought that I had at the time was thinking about why um people weren't really using um zcash specifically um since that was what I had been working on and I got to this perspective of of kind of concluding that although payments are held up and have been held up you know since Bitcoin was created right it's like a peer-to-peer electronic cash system the whole idea is like we're gonna do payments and that's

like the simplest possible thing it's like the MVP for building uh blockchains and if we can do payments that means we're on to something and if we can't do payments that means we're failing at the most basic thing and my conclusion was actually that that is completely backwards and that payments are actually like the hardest possible thing to build uh and the reason for that is that the the Simplicity of a payment transaction oh I I have some Ledger there's value in this account or address and it goes to this other address and I'm just like you know there's numbers moving around in the database is actually a totally

superficial Simplicity and the reason is that the economic transaction is always bigger than the transaction that gets recorded on chain so whenever somebody makes a payment there's always two sides of that flow right you have some flow some good things changed exactly that's right and you don't even see the goods yeah even in the case of you know for instance like a donation like you can still think that there's some kind of like abstract flow in the opposite direction like somebody is you know getting good feelings about contributing to some project or something and so you have a situation where the technical part of the system is

recording only half of the full economic transaction and so in order to to use that usefully you have to have this whole parallel social infrastructure of where that counter flow is happening off chain and when you're trying to bootstrap that you're running into the sort of Technology with the strongest possible Network effects in human history which is money right um and so it actually feels to me like can you do payments on a blockchain is I think going to be sort of like the final boss of this like Cypher Punk dream

where like you know blockchains take over the world or something and in the meantime chasing down payments is kind of this like Moby Dick like white whale of like this is the thing that you know and all these other projects have like you know failed to do this but mine will be totally different and you know whatever every point that the ship just gets wrecked you sort of talk is are you saying something like you need to be building or bringing that value more on chain the other side of that transaction before you could really accomplish this payments would it work more in digital Goods in a way so the the thing that I I was was thinking about at the time was was actually reflecting on um seeing the defy summer happen you

know over the last at the time it was like over the preceding six or eight months and the thing that I think is really interesting about all of these D5 protocols is that okay although what those tokens are is kind of limited the types of transactions that people were making with defy systems were ones where the entire economic logic of that transaction was included in the blockchain transaction like when somebody makes a uniswap trade on ethereum they don't need to have like some kind of counterparty that they've arranged with off chain to do blah blah they just say okay this is a thing that

is going to be useful to me I've decided that I want to make this trade and now I'm going to do it and what's really powerful about about that is that if you're trying to get adoption you've created a scenario where individual people can decide to start using the system without having to pre-coordinate with anybody else I think this is actually the key thing that makes payments so hard it's that what are we building when we're building blockchains we're building this coordination tool but if you can't do the thing on chain directly if you have to do off-chain coordination then it's like oh before I can use the coordination tool I first have to somehow coordinate before I can coordinate totally and I think that that

makes it much harder to get adoption whereas if you can start with something like you know some kind of defy application where people are able to get Direct Value out of transactions that they can make themselves that's a much better way to bootstrap a user base because once all those people are on the system then you know maybe actually payments do kind of work right if you look at where crypto payments are really successful like there's a lot of use of crypto payments by crypto people but that's because they're all already in the system and so the thought was like okay could you could you have some kind of like defy functionality that would let you build a private system where individual

people could make a choice to start using it without having to First sort of pre-coordinate with some other counterparty how would they even find that person right and the place where those two lines of thought kind of converged was thinking about okay of all the defy stuff is there an application where where having privacy is actually this kind of like killer feature where you can actually be better as a as an ex because you have control over information disclosure and in that case uh or in in that sort of framing trading is a really really interesting use case

because every Market is also a market in information if you think that like the per you know one of the purposes of a market is to determine prices then you know that market activity is some kind of like decentralized computational process where people are sharing information whatever like it's it's it's a really like fundamental part of the Dynamics of that system and so if you if you're looking out at all this like defy trading where everybody has to reveal everything to everybody at all times it's so obvious that like that can never be the best system because it's just completely missing this whole like giant part of like what

makes that such a like deep and interesting Dynamic I mean market dynamics I mean um are you thinking so when you say this it definitely makes me think of like strategies of certain groups being exposed and then I guess people just copying those strategies over and over again why is a private system better I I think that the the idea of sort of a market as being a marketing information is such a fundamental thing that you can actually see that in a lot of different types of behavior so one really obvious example is like all this you know front running um fundamentally that's about somebody sort of revealing information before they can do their action and then that

being exploited on the strategy front right exactly as you mentioned if somebody has some kind of strategy where they've you know done a bunch of statistics and figured out what their estimation of the prices is if somebody else can just like clone all of that without actually having to do the work themselves that kind of limits the incentive of to do that kind of work and there's other cases like for instance with market makers generally a market maker wants to stay flat with regard to price movements right their goal is basically trying to make a little bit on every trade and not get rolled over by being wrong and so if

somebody can figure out okay I've carefully watched this Market Maker's activity I can kind of reverse engineer what their strategy is another thing is instead of cloning it if you can predict like what the like two percent of times they're going to make a wrong prediction are then you as a Trader can like roll in a huge order right when you think they're going to do a mispricing and that can like totally wreck their whole economics right yeah this is a very famous term called adverse selection is the fact that um if information is kind of known ahead of time and can be used they can also be exploited which of course is we know from Mev is a is a very is I mean Mev in in many ways uh

like on chain is like a market and information where not only do you have kind of a statistical notion of information but you have a very very concrete information about you know exactly what someone's going to do and when it'll get included in a block and then you can do things to trade against it so privacy here is a very interesting um idea right because essentially if you could somehow I mean this is people have been thinking about this for a very long time not quite in this framing but just more generally right it's like if you can prevent this from happening this information leakage from happening in the first place then congratulations you've kind of like achieved you've kind of cut out a huge part of this kind of you know people using knowledge that is kind of silly that it gets

leaked uh in order to trade against a certain person or something like that do you think it would become very transparent system has created a whole range of new strategies and attitudes and ways of doing it in the private system is it a is it sort of a return to previous Market types or is it it in itself a brand new playing field with like brand new ways that people would interact on it I think are you thinking about this yeah I think so the the thing that I think is really interesting is that and that this is kind of a whole other tangent that also comes in you know we'll get to at some point when we talk about the the lateral section in the paper uh but

speak about this another kind of interesting and underappreciated aspect of this is that if you try to build a private Market the problem isn't just okay how do we make everything private like how do you just like make like every piece of this whole system private and it's all completely ZK and blah blah blah well actually nobody wants to have a market where literally every piece of information is private because like you want to know like how much liquidity is there like what kind of Trades are there hap like are happening in general what is the aggregate state of the market what is the price of the market right so there's a very there's a much more kind of subtle and interesting problem of privacy which is how do you have privacy

for individual actions so that you know some individual Trader doesn't have their specific trade front run or some Market maker can have their strategy concealed but you still have some kind of aggregate information about what is you know the total trading volume what is the aggregate set of all of the liquidity that's available how is that moving around so that all the participants in the system are able to make choices based on that information to go back to your question of is this just a kind of return to tradition or is this something genuinely new I think that there's a possibility for private defy

to build this kind of hybrid Best of Both Worlds uh situation where you can keep some of the confidentiality of you know people being able to do trading without leaking their strategy but also you're in this totally open and permissionless environment if you look at the way that traditional trading works on like a like a centralized exchange the entity that's running the exchange actually you know sees everything that everybody is doing all the time and just like doesn't tell that to other Traders like hopefully right but also now you have like the nyac or something which is this you know like centralized entity that gets to like gatekeep like

people's access to price feeds right like you can't even like get prices without paying a bunch of money to some data provider whereas in a blockchain system like there is a much greater degree of kind of permissionlessness uh in terms of access to the system so I think now would be a really good time to talk about penumbra uh not just in generality but in specificity yeah I mean I think we've given those some some really I mean just the thinking and sort of the path that you were on in kind of intellectually that leads up to this more tangible project tell me a little bit about the beginning of penumbra like where where does this start yeah so with all that you know just kind of like

hold up like context you know somewhere in your mind the the kind of how do we actually Manifest this is what kind of became penumbra the thought is if we want to have the genuinely private decentralized info Financial infrastructure that that we think is important to exist in the world can we build a private trading system that lets people do things better because it's private and use that to like bootstrap to systems sort of creation it's use its adoption Etc so that's how I started on the project at the time I was kind of in this like awkward period of

um Visa related limbo oh yeah it's a us just like never issued me a work permit because they're yeah I think you remember this yeah so we don't have to get into this whole thing like the U.S immigration system is just like very bad and just dysfunctional um do we think Canada's better um I've never been on the receiving end of it so who knows I just found out today that Henry and I are both Canadian that's cool but yeah we don't know what our immigration well he has a point actually it's not I heard it's actually I heard you could do it if you don't live in Canada yeah I have no idea anyway and and so at the time I was kind of stuck in this position where it's like okay well I can't technically you know work on anything but I can just

sort of like do research publish ideas uh put them out into the world and sort of see what ripples come back and so I had started writing up kind of this like design sketch as a kind of a way to like occupy time until you know someday uh when I would get some kind of clarity wait I kind of remember this this is uh like when we were talking about the Privacy results actually differential privacy no prior to that this was like oh yeah this was like the first half of 2021 that's right yeah that's right yeah yeah okay um so that was mainly a way to kind of stay occupied um and sort of actually set the course on

building in public it turns out that if you just post a bunch of ideas on the internet sometimes people will read them and sometimes those people will have a technical background that is like totally different from your own and because of that they can tell you things like oh actually you should do X instead of Y or you should go read this paper um or whatever right um and and so that was like pretty cool and and quite rewarding and then eventually over the summer you know that kind of situation got resolved and by that point I had kind of spent enough time thinking about this that I realized

that I really wanted to uh actually make this real and and so then you know started trying to get get it together to to build a team and try to like incubate this project that got underway for Real uh about last September so we're almost exactly a year into the development of the project and since then we've continued to iterate on the the design and we've also built most of an implementation cool I think what we do need to hear is what is penumbra so we've talked like and now we know it as sort of an org like from what we talked about before the markets what

is it today because also I do want to say like and we'll add maybe links to this I remember when you first started on the project there was like the private validator sets there's like a lot of different places where privacy was being explored where did you land yeah um so probably the the succinct way to put that is uh penumbra is a fully private proof of stake L1 and inside of that blockchain there's one multi-asset shielded pool that records all of the value in the system we have a shielded staking system that gives privacy to delegators but you

know still accountability and transparency for the validators and we have uh integration with IBC so any IBC compatible chain whether that's a cosmos chain or any other place where IBC has been implemented they can transfer assets into penumbra as those assets come in there are recorded in the shielded pool so the Privacy boundary of the system is actually the boundary of the chain and once you have those assets inside of penumbra we have a shielded Decks that allows people to do either sealed input batch swaps on the kind of

Market taker side so when somebody submits a swap they say I want to trade you know this amount of this token for this other token the input is encrypted and batch together with all the other swap intent in that block executed at a common clearing price and only the sort of aggregate batch total of what the the trading flow for that entire pair is is revealed so you can have long-term privacy even for individual trade amounts that's kind of on the market taker side which is one sort of point in this trade-off between how much privacy do you get versus how much control over

your trade execution do you get the other trade-off point that we have is on the market maker side where we have all concentrated liquidity and the concentrated liquidity positions themselves are public but they're created out of this shielded pool and they're returned back into the shielded pool and the positions themselves are immutable so it's not like you know on ethereum where there's some account and it like owns certain positions and it can change them over time it's that you know a position is created and then maybe the position will be closed and later another Position will be created but there's not any way to link together

what the history of those positions are and especially in a context where a market maker has some kind of more complex trading strategy they're likely going to be saying okay I have this this is my desired trading function I'm going to approximate that using various like smaller individual positions and which positions correspond to like which Market maker or their strategy is not revealed maybe you could try to estimate do some kind of statistical guessing but really what you see is just the aggregate of here all the liquidity positions and that's kind of the like Market view of what prices

are but you don't get to see any individual market makers can I check something when you talk about the different liquidity positions having sort of this immutability is that similar to like the nft like thing we see in uniswap V3 like is it sort of unique yeah and it and maybe it's not an nft per se but like where it's like it's it's in itself it's not just the collection of tokens in a setting it's like produces a new token of some kind or some sort of representation yeah it's a it's an nft and like the technical sense and not in the like monkey picture yes different words we need different words uh but yeah so so similar to to univ3

um except very private or parts of it being private so the the difference is that that the actual LP data and the that forms that nft that is public because that's part of what the market state is right what's private is who controls that position and where the funds for that position came from so unlike in say ethereum where everything is sort of going back to some account that controls it on penumbra all of these positions are just you know they're more tokens that are in the shielded pool so the data that's revealed when someone creates a liquidity position is okay there's a transaction it spent some funds from the

shielded pool uh it put them into a a position you know what what the details of that position are is public but that position is then controlled by this Bearer token which is an nft and that nft is also recorded in the shielded pool interesting just like any other asset wow so like those are mixed in sort of helping to even grow that anonymity set like more types of tokens yes okay all you can see is like effectively okay there was a transfer of like you know 100 atom and like you know 20 usdc or something into this position and that's it right I mean overall it's a it's like

a very interesting architecture because even even on the lp side you well you don't get privacy of the positions themselves you get privacy and that like you get this notion of kind of unlinkability there's like a statistical notion of linkability right like if I knew your strategy ahead of time I could maybe try to unscrew whatever you've been doing but I would have never know that it was actually you other than like by knowing ahead of time that I knew what you were gonna do and like kind of it's a little bit worse than that um if from a privacy standpoint and like one way that that you could have um linkability between positions is like you know suppose somebody closes a position and then in the next block they

open a new position that has you know roughly the same amount of capital but like the bounds are like slightly shifted over somebody could infer that actually that's probably the same person and this is why any kind of private system that doesn't have privacy of amounts is actually kind of dangerous from a from a privacy standpoint because if you're not careful about like what your amounts are uh and like what information you're revealing with the the amounts you can potentially leak a fair amount of information but but the thought is that how is that how why that seems like counter-intuitive no well like by not having the information you leak the information oh no sorry if the system

reveals the amount that actually reveals quite a lot of information about what the activity is I see so and even an even sillier example which like has happened uh I think in some cases is um people have tagged the end of like certain so for example you know you have a bunch of like decimal places in a token right and what you do is let's say you know at the 15th decimal place you put one two three four five and so someone who isn't really paying a lot of attention so like you know let's say I've given you we're all doing a bunch of shielded things I give you whatever 10.000 one two three four five penumbra and you're like oh cool I have like 10

penumbra because a lot of you know wallets will not show you the 15th through 18th decimal place um what can happen is if you then so you know every let's say everything is completely private and then you you then go uh and you know for some reason or another uh unshield that specific value because let's say you've transferred it out of penumbra into some other chain um I'm immediately gonna know that it was you because you didn't notice this like small end of this like you know value tagged at the very end right you like I now know it's you because by you know the probability that anyone else is going to have exactly that you know end like little decimal tail place is pretty low

um so and this is actually a common thing yes wouldn't there be Gatsby's somewhere yeah so well it depends on how it's paid right but but like if you were just to withdraw for example let's say you're paying gas fees and and Adam and you know you're withdrawing the number to a separate place you you end up with this like extra little tidbit at the end um I think some people have used as a fingerprinting technique at some point but I don't recall who yeah I guess like the way that I would frame this just like as a kind of like General like intuition on why this is a problem is if you think of like okay how much entropy is revealed how much information is my transaction revealing if that is including publishing an amount like and if you have like you know 32 64-bit

amount that's a lot of bits of data that you could potentially leak Henry the differential privacist right um but but coming back to the the liquidity position side of the decks the thought process is that there's a kind of a fundamental trade-off between how much control you have over the execution of your liquidity on the one hand and your privacy on the other hand and although for sort of normal users I think it's unreasonable to expect like oh you should just like you know carefully think about your amounts and never mess up um for somebody who's running a kind of actively managed Market making system

they're having to think about okay what is the information in the market what prices am i setting and so on and so I think it's like a little less unreasonable to have you know as in that part of the system the idea that like you know maybe you decide that you should be thinking about sort of like what information your trades are revealing leaking yeah um and so that's the reason that I phrase is sort of being these are sort of two trade-off points on privacy versus execution control where if you just want to do a swap and have the most amount of privacy possible you just submit it as a swap if you're doing Market making then like you kind of need

to get good right right even in the case of the swaps though you still you know unless you have a lot of people swapping you still have this notion of differential privacy right like even just even if you just batch everyone's trades together right if I also know something like let's say there aren't that many people trading with the you know the market there's like you and someone else right I can reasonably infer if I know something else about you I could reasonably infer that how much you traded or something of a like but it's much harder yeah right because there's very little public information the only information that's made public is um kind of what everyone traded in Aggregates as opposed like any one individual person

one of the ideas that we've been thinking about that on on the client side actually is the idea of having some kind of UI for doing trading where a user can specify their time preference where they can on some kind of slider between like I want this to happen right away versus I want this to happen over you know the next like five minutes or the next hour or the next eight hours whatever and have the uh client side UI AI be able to automatically prepare a kind of randomized subdivision of their like meta Swap and submit that in

individual unlinkable transactions at like randomized sub-intervals so wait is there actually essentially implementing a time-witted average pricing mechanism but it's randomized but yeah well that's for randomization right well that's pretty sick the thing that's really cool about this is that for the user you actually get sort of two benefits at once number one is that you get some kind of hedge against execution risk like maybe you get batched in a swap with some like whale that's trading in the opposite direction or the same direction yeah the same direction and that like wrecks your price but since you did this like randomized average price mechanism it only affected a small part of your Trading

um and you also get this privacy benefit that even if you know you end up you get unlucky nobody else was trading in the same block as you and the batch size is a batch size of one well okay you're only revealing this randomized sub amount that that reveals with much less information about your your total trade intent right so but it the the interesting part is this goes back to the previous conversation right is that this whole notion requires like the notion of privacy right requires having more people interacting with the system so it's just like oroboros uh for like you know you have to have people in the system in order to have privacy in order to like ideally also have more people in the system in order to have privacy yeah the way that I would put that which is actually like you know specifically

intended to to nerd snipe Guillermo damn it is that um when you're optimizing for privacy oh no this is not like a convex optimization problem it's not wait something isn't a convex optimization problem and I never would have guessed and um and so the the the but the reason that that is like relevant is that um the local Minima is not the global Minima or maximum yeah you know depending on which way you frame the optimization problem if you're trying to get the most privacy maybe you decide oh I'm going to build a bunch of um pieces of this system so that every action is as private as possible and

it's impossible for anybody to leak any information but then by doing so I've sort of I've locally optimized for making all the transactions private but I've curtailed the functionality of the system so much that now it has no users and so it's like oh I have very good privacy within a very small anonymity set whereas another set of engineering trade-offs might have slightly weaker privacy properties for each individual transaction but then cause the system to be much more useful and have a much larger set of users and so you maybe have like slightly weaker privacy but like within a much larger set and so on balance that's actually better that's the what I mean by it's sort of this

like very um like complex optimization problem and one way that that plays out with the the swaps is we're planning to have penumbra do the batching at Block Level resolution so when you submit a swap the trades that will be batch batched together with are just the trades in that block and if there's no other trades in that block then you're in a batch size of one and the reason for that is although that's you know from a privacy standpoint maybe it would be more optimal if you said oh you have to wait for like you know five minute intervals or hour long intervals

or something but in that case I think it becomes a lot less useful as a trading product and in that case you end up with okay well there's this like weirdo like privacy project that's only only used by the like you know weird nerds who care about privacy and then you end up being you know private within a much smaller set whereas if you build something that is intended to be like you know best as a trading system then you can do things on the the client side like that kind of like randomized average pricing mechanism that kind of mitigates the risk of the individual batch being you know batch of one or something um while preserving the system for

people who just want to do trading because ultimately we want penumbra to be providing just like better execution even if you don't care about the Privacy part at all you should still be using penumbra to do trading because the fact that the market making allows private strategies means that you're going to be able to have market makers with like much better Capital efficiency and therefore like lower spread better prices for for Traders interesting I actually now I'm curious to ask you like you're in the cosmos ecosystem are you competing with osmosis

that's a spicy question I I honestly I don't think it's that spicy um I mean okay like no not really uh like osmosis is a DEX but it is also a lot more than a decks like osmosis is this sort of whole like defy hub for Cosmos and so just pausing on the cosmos part and just looking at privacy projects generally there's a lot of privacy projects that are trying to build some kind of like general purpose private system with different kind of takes on what that means for penumbra we're not trying to do that we're trying to build like one useful private

application we are an app chain that's a decks and we're doing one thing which is private trading and private Market making and with that kind of in mind I think that that probably clarifies like why I don't really see this as being competitive with osmosis um if anything there's this kind of like hybridization because we're both in this um interchain ecosystem I think there's going to be a lot of interaction between users who use both penumbra and osmosis yeah there's all these things that osmosis does as this like defy Hub that penumbra is just like not going to do and it's actually because of IBC and because of the existence of other chains like osmosis or like Juno or like the

cosmos Hub or whatever that it's possible for us to have this like very narrow differentiated product strategy and that's another lens that I think is interesting to compare sort of the state of the ecosystem today versus say you know four or five years ago right if you wanted to build an L1 four years ago well your L1 is going to be this like isolated little universe so you'd better make sure that like you know you have a programmability story that you have like all of these pieces of like what it means to be an L1 you're going to have to go and do like all this like Biz Dev to get it listed on all these exchanges

you have to like do all this stuff and all of that work is just repeated over all of these different projects and the thing that's really cool about IBC and having a a really solid cross-chain communication mechanism is that now individual projects can be differentiated while still having the sovereignty that comes with being their own chain right so we get the freedom to design a chain with like the data model that's compatible with privacy with all of the trade-offs that we think are important to make but we get to do so as part of an ecosystem right like we're

not trying to build an ecosystem we like live in a society of other blockchains and our users are going to interact with them and so I don't really see it as being you know particularly sort of this kind of like zero-sum competition or anything like that I actually want to follow up on this so you said you're IBC enabled but are you actually using the cosmos SDK as the foundation or did you have to like create a lot of this on your own and then just use the IBC modules um so yeah maybe it's helpful to give a little bit of context for people who are not like you know totally causing us most nerds the I think a useful analogy for the cosmos SDK is that it's sort of like the like rails for blockchains in

that if you want to like do a blockchain project you can like generate this kind of opinionated scaffold of you know reasonable implementation of various blockchain stuff and then you can write your own extra if you want to do like one custom thing you can add that in as a module and like all the other stuff is is written for you um the cosmos SDK sits on top of tender mint which is the consensus engine and so there's this separation between the consensus layer that's determining what was in the blocks and what the blocks are and the application Level that's determining what the meaning of those transactions are right so to Tender man a transaction is just like a blob of

bytes that gets passed to the application and then it's the application that gets to decide what to do with it and if you're if you're trying to build a blockchain that's like mostly the same as other chains but does like oh this is our thing that we do differently yeah yeah then I think that's actually a really reasonable approach right but for us the data model of a shielded chain is just so different from a transparent chain uh and the whole state model is different we have all these different design concerns so for us we use tender mint as this kind of off-the-shelf consensus piece but everything on top of that is our own custom application it's

written in Rust it is built around this kind of Baseline of the shielded pool as our kind of fundamental store of value which is quite different from say having accounts um and uh we have our own IBC implementation that uses that's written in Rust that uses some of the IBC libraries from informal but our IBC State machine is is integrated with our app and you had to redo parts of that then like that's a lot of work I guess the journey I suspect to say the least well yes

um so the good news the good news is very good at long pauses I gotta say no he's got a good dramatic pause yeah um so the the good news is that uh we were actually the the first um non-cosmos SDK chain to open a connection an IBC connection to the cosmos Hub this summer so we got one of our test Nets I mean it required some amount of like manual poking but it did work um and so that was a pretty cool milestone so the way that IBC works is that you have two chains and they're trying to

communicate with each other the way they do that is each chain is going to run a light client of the other chain so it can understand and verify its counterparty's consensus date and then also has to have a way for what the state commitment that's in the block header is actually a way to understand what that's actually committing to and be able to read messages that were sent by the other chain so that kind of factors into three separate pieces the first piece is the consensus specific light client for us because we're using tender mint we didn't have to do anything on that we just because we're another one you could reuse so and that's that's good because that's the

part of the counterparty chain that actually has to run code the second part is there's a standardized way to express like a generic Merkel proof so that regardless the idea anyway is that regardless of what type of like Merkle tree the counterparty chain is using there should be some way that you can just like plug in the proof um and then the third part is that there's some kind of standardized system of like how you encode messages into your chain state so that your counterparty chain can read them and I think actually the IBC spec is like for all of the like warts and you know we know where the bodies are buried and

whatever like is actually remarkably difficult to build a generic protocol when you only have one implementation and although there were some like weird you know quirks this is like they're like Bob Ross like you know there's no mistakes it's just happy accidents we'll just add another layer of hashing in here to make the code happy um it is actually part you know it was possible for us to build our own custom chain and submit proofs of what its state was to the cosmos Hub which you know clearly is like not code we control and like that code that's already running was able to

um you know understand what our state is and I think that's a really cool accomplishment that we were able to just kind of permissionlessly plug into that there is in IBC though there is a message passing that layer as well right like I guess the question here is like would you be able to do something on one chain that interacts into penumbra like under the hood with the setup that you have yeah so there's once you have this kind of message passing layer there's various kind of application Level stuff that you can put on top of that one simple application which is the one that we're aiming to support first is just token transfers another application that is very cool is

interchain accounts and also interchained queries so this is a way that you can for instance have on one chain some kind of verifiable query result of data on another chain oh okay that's the interchange queries interchain accounts is where you can have an account on one chain directly control an account on another Chain by like passing messages back and forth so those are like kind of more advanced features and we're not planning to implement them initially but in the longer term it's something that we're thinking about and there's also some potential like quite cool twists that we could do on those so one idea that we've just kind of thought

about is building an implementation of interchain accounts where instead of a normal setup where you just have two chains you know each with their own account set and here's my account on chain a and I'm controlling this account on chain B on penumbra we don't have accounts but what we could do potentially is have a way for the penumbra chain itself to have a pooled account on each of its counterparty chains so we would only have control over a single account and then track who controlled what assets within that account internally to penumbra inside of the shielded pool wow so that you could

it would all come from one account on that other chain but you would actually be able to direct where it's actually going oh interesting you could do like um actions on other chains uh from penumbra yeah wow without having to sort of reveal exactly you know like you could have for instance like the penumbra account on the cosmos Hub yeah yeah and there's some atoms in it and somebody can make a transaction on penumbra that's like oh I'm using my like Bearer token that represents control of like 100 atoms worth of this account and I'm staking them to like the occlusion validator wow and that's my action and like that action conclusion yeah I'll say yeah why

not CK validator it's important to have like decentralized yes yes of course um this is really some big brain stuff I don't know I love it but yeah the the I think there's a really cool kind of opportunity for for penumbra when we're implementing these kind of interchain standardized uh features where it's like how can we implement this in a way that is both compatible with the standard but also is like doing something that's kind of unique for the capabilities that we have from penumbra wow Switching gears uh maybe a little bit from from slightly I guess we were

talking about IBC recently but but from the philosophical part to kind of how how do you enable like private transactions with global state is already in itself not an obvious thing right and you talked a little bit about this in your you know presentation in CK Summits but you know could you describe exactly like how does penumbra work right it feels like if it were so easy I feel like everyone would have you know made a private decks but in fact uh as we referenced as we referenced on many yeah previous episodes uh building in private decks is really damn hard so and and you know I guess how did you how did you manage it how did you like I mean I want to be clear it's not done yet we're

still working through it but but even having a reasonable plan for it I think is non-trivial so I think um probably a good way to kind of frame the problem is is by thinking about the kind of basic sort of state and data models of blockchains and how we end up building private chains and this is totally Independent by the way of any sort of specific details about the proving system whatever that's a whole awesome piece of engineering but it's kind of orthogonal to this issue which is like what is the data how do you represent State on the Chain so if you think about like a transparent chain

like ethereum you've got this like big ball of global state which is like here is everything that is on ethereum and a transaction has some kind of description of these are some State changes that we're going to make to that state and you have the nodes they come to consensus on those transactions and then to execute them each transaction basically takes a lock on the entire world and does whatever changes it's going to make all the nodes execute those and then they've sort of progressed from like state to state to state and so the the whole Paradigm is around Global

mutable state in order to build a shielded chain okay we're trying to use like snarks for something right how do we do that first you have to fragment all of the application State and kind of rewind to this more utxo style model like in Bitcoin where instead of having one giant ball of global State you have immutable composable State fragments this is like the sort of cool way to think about utxos is that each transaction output is like one fragment of the chain State and when I have a transaction that has some list of input

txos and produces some new outputs what I'm doing is I'm composing together those State fragments and then producing a new set of State fragments and those are going to get included in the chain and although in Bitcoin you know that's a just a simple kind of value transfer application you could do a lot more complex types of State uh in in principle okay so why would we want to have this kind of fragmented State the reason is that this way we can move all the state off chain so instead of having this like big like Merkle tree of State fragments we're going to replace all of those State fragments with cryptographic commitments to the

state and that way the on-chain data is just this big tree of commitments that don't reveal anything about the data that they're committing to and now in my transaction I can have instead of okay I'm taking these three pieces of State combining them together and making you know these two new ones um I can do a ZK proof of I have honestly formed a state transition that consumes certain States and produces some new States and now the details of that state transition are completely private all the chain C's is like okay

here are some new state fragments that were produced and also here are some serial numbers of some other fragments that were consumed right and this is from a privacy Point standpoint this is like okay great we've like solved the problem because we've moved all the execution off chain we're just supplying verifications that the execution was correct and so the chain doesn't like learn anything but what's the what's the kind of problem here why aren't we all using private chains then then if this solves the problem and the big thing that you lose is the ability to access shared States right it turns out that actually having like public shared state is like kind of

the whole point of having the blockchain going back to the kind of big sort of philosophical discussion at the beginning and the point about not wanting to have a technology where you have to coordinate off chain before you can use the coordination tool the reason that everybody loves to have shared State on chain is that it means that they only have to coordinate with the chain they don't have to go find some counterparty oh yeah um and that's why losing shared state is a problem but why exactly did we lose the access to Shared State the issue is that when somebody is submitting a transaction on a transparent system because the execution actually happens

on chain some pieces of that execution's inputs can have a late binding so if you think of the kind of overall State transition that's created by some transaction there will be like various inputs to that computation and various outputs and some of those inputs are going to be specified by the user when the user is creating and signing the transaction but others won't be and so for example with a swap right somebody is going to sign over you know I'm sending this amount of tokens to the amm and I am hoping to get like this type of token in return but

another input to that computation is what is the current state of the amm what are the reserve quantities and you need those inputs in order to be able to compute the final outputs and it's because the execution is actually happening on chain that someone can create a transaction where they filled in like some of the pieces and then they've left some of the other pieces as kind of like okay well this will be filled in by like whatever the chain state is when I actually get to be executed right and you can also use this as a perspective to on kind of the whole like Mev games and like transaction ordering and stuff like if you're signing a transaction where there's like

you know all take you know whatever the current thing is then like obviously there's going to be some kind of game about like can I rearrange your transactions that you get an unfavorable input right but that's really kind of the the key property that lets you build useful applications is having some way to interact with the public shared State and if we think of what's happening when somebody makes a private transaction because they've moved the execution off chain and they're only supplying a proof they actually it's a like a ZK proof is a proof of knowledge so they actually have to have like complete exact knowledge of the state transition that they're proving about

and so they have to sort of they have to seal all of the inputs and all of the state that that Transit State transition Touches at the point that they make the transaction and that would appear to block people from Ever interacting with the shared state in any meaningful way yeah fundamentally yeah exactly the the big problem is when things are fully private right is when you have the shared state right like knowledge of that shared State should not be required directly in order to like perform the action you want to refer and often it's not known at the time of creation I mean again just a restatement of what

you were saying but that this is a huge problem with markets right whenever you submit an order to a market you often won't know the price of the order lands at I mean unless you you can do many things but the point is or you won't know the time when it lands or like things like that which is not inherently possible to do with at least zero knowledge proofs although there might be some weird homomorphic encryption stuff you can do but so what did you do what do you do so the thought process is that really the sort of fundamental issue there is how do you mediate concurrent access to Shared State the problem is oh all these people who are making their transactions doing their execution on their own device and then submitting that to the

chain the problem is while they they're all kind of trying trying to concurrently access or modify some kind of shared State and we don't have really a way to coordinate that so you know we can't make it happen and if you look at you know not blockchains but just like systems programming in general you can think of the approach to accessing shared State on a chain like ethereum it's like okay we have like one big Global mutex and we've like put all of the data under this lock and like every transaction acquires the log does stuff and then releases it and if you just think about systems programming design if your goal is oh I want to build this kind of high performance like scalable

concurrent system the architecture that you use to do that is not like I put all of the state in like one giant lock that's how you get the bad programs um the bad programs generally speaking right and so another approach would be you could try to do some kind of more fine-grained locking you could either build that into the transaction mechanism or you could do some kind of like you know very fancy ethereum implementation that's basically like the like out of order x86 core but for ethereum where we're just kind of like secretly go in and figure out all the data dependencies and then you know give you this like as if execution through a heroic amount of engineering effort and like that's a thing that I'm sure is

gonna happen just because of like the total weight in that ecosystem I think it's kind of already happening actually there was a paper about this somewhere recently for Microsoft research right but anyways but but another just from a systems design perspective another approach to handling concurrency which is actually better is uh an actor model with message passing so the idea is instead of like oh different parts of the program are all trying to like you know access the same data and like share state or whatever instead you say like we're going to divide the program up into these like little actors that all execute independently and they'll communicate with each other through

message passing and that way you don't need to have any kind of like locking or synchronization because everything is just like passing messages to everything else what would that look like in the context of this specific problem of how do we share a state on chain it would look like instead of your transaction having some code that makes a call to some smart contract and gets the result your transaction would send a message to a Smart contract describing something that you wanted to do and then eventually that would execute and you would get a response from the smart contract asynchronously so you're

moving from a synchronous execution model where every transaction gets to lock the world to an async model where a transaction gets to send messages to an application and then some computation will eventually be performed and in that case you can have a really cool kind of like batched processing model where each smart contract is going to execute once per block but on input all of the messages that all transactions sent during that block and then it can perform you know whatever kind of application specific logic it wants to determine how to process those messages so for some applications like swaps right you can actually agglomerate all

of those messages into a single batch trade and executed it once and now you've got this a huge performance speed up because instead out of having to execute all those trades individually you're only executing one trade per block and so you can be considerably more sophisticated about how you actually do that execution but more generally for other applications you could also have some kind of app specific logic that like chooses like for an auction protocol you could sort the messages by bid or something like that and the challenge now is like okay I've like sent a message to a contract like what do I you know what does my transaction actually doing and the thing that's kind of interesting is that what this model lets you do is

it it lets you retain the setup where each user has their own state for their own account balance that is theirs and they're interacting with this public state by sending messages to it and so they can now have a kind of asynchronous programming model for their own side of that contract where they do some part of the computation and now they have to wait they send out some message and they have to wait for a response and you end up having a model that looks a lot like futures or some kind of async programming on the user State side where for instance if I'm

doing a a swap I'm going to make a transaction where I've submitted my swap inputs I've like burned those input tokens and now I need a way to like you know hang out while I wait for that swap to execute but be able to later demonstrate to the chain oh I actually did participate and the way to think about that is you need a way to take all of the kind of intermediate state of that execution and somehow freeze it so you can pick it up later which is exactly what happens when you do async programming with Futures every time you have to wait for a message to come in that's a different sort of intermediate

point of this state machine and the like really cool thing that we do in in penumbra is we have a way to miracleize all of that intermediate state in a snark friendly way into the asset ID of an nft that we record in the shielded pool um so like if you if you've done like rust programming for instance you may know of this like idea of like oh you have like ownership and you can only you can have these like move semantics for your state machines so you know that you can't like accidentally do the same state transition twice well actually if you have like you know a database that's recording quantities of financial assets

and you turn every intermediate state of your future into like a distinct you know financial asset then like you can reuse that infrastructure to track the execution of the program right um so the way that the swap works is when I make a swap I'm going to burn my input funds that I'm committing to the swap I meant to myself a swap nft whose asset ID so that's you know one unit of an asset ID which is a snark friendly hash of the trading pair my input amounts the address that I'm going to Mint the notes to in the end and that way I'm verifying that all those things

are consistent in the proof that I make when I submit my swap and now as soon as I see that that transaction has been included in a block I'm also able to see what the results of the batch output are because the the actual trade is going to be executed at the end of that block and now I can in a second transaction I can spend my swap nft that recorded the details about my participation in the Swap and now I'll make a proof for the second half of this state transition where I now have the results of the trade execution as a public input and I

can prove that I'm privately minting the correct Pro rata share of that batch output and that I'm sending it to the correct address that I signed over in the in the first part of the transaction and the thing that's really cool about that actually is that it means that you don't actually even need to have a separate signing step for this secondary like claiming uh part of the the execution because there's no way for the funds to be you know diverted to some other things yeah it's literally just your user agent kind of submitting a transaction that like finishes off to change to your private part of the state

right cool it's it's very yeah the the notion is quite interesting right it's like separating out intermediate steps of a state machine right and then enumerating the intermediate steps in a very explicit way such that once a computation is completed which you know in order for you to do a swap you depend on data that you did not know at the time right namely how much was actually traded in the output to then you know you can you can then use this like intermediate step to then complete the computation as if it would have happened kind of like purely on chain the other thing that I think is really cool about it too is that it actually although this is a like designed around privacy I think this actually has really cool

implications for scalability too because if you think about the sort of sum total of the execution that it's going to happen if there's you know a whole bunch of Trades because we're explicitly modeling here's the part of the computation that is only touching the per user State and here's the part of the computation that's touching the shared State as like distinct Concepts the execution of all the per user State changes is actually happening off chain on the user's device and the execution of the shared State changes is happening in one go in a batch where you can get like hyper efficient batch

processing in you know in the in the case of the swaps is sort of the optimal thing because you can just batch all the swaps together and execute only a single trade but in general like batch processing lets you share work between items in the batch and the extent to which that's true depends a little bit on your task but it's like very rare that you have like no sharing possible right so you get to have like hyper efficient public computation without having to have contention on the state access and the private side which is the per user data is like free from a scalability perspective because it's all happening on the the user's device and the other thing that's different from

like a ZK roll-up right is that each user's proofs that they're forming are only about their own State transitions that's right so that's a small amount of proving to do it's not like you have to appoint some sequencer to do this like giant proof of everybody's State transitions all at once you're Distributing the computational work out to the edge the actual end users and then you're enabling the part of it that is necessarily shared to execute much more efficiently right so you covered a lot of this in the talk at CK Summit 8 as well so I'll add a link to that in the show notes if anyone wants to to see that it's a good one I want to say a big thank you for coming

on and sharing with us all of this kind of your journey and the kind of ideas that led to penumbra a lot of the thinking that you have around constructing it and some of these pretty cool Innovations findings like had anyone else done that has anyone else tried just uh I don't think I think this is this is new but I the thing is that this although I've described it in this kind of Fairly general terms it's like oh here's this kind of idea of uh this whole Arc system architecture the way that we got there actually was not by starting with we're going to build this like generalized programmability thing and in terms of what we're actually focused on building we're still focused on we're going to do private

swaps and we're going to make them good but what's interesting is that by focusing on that one specific problem the solution that we came up with as like oh here's a way that we can solve it in this case actually I think generalizes to a much more interesting State model than we would have got if we'd started from this kind of abstract like oh let's build this kind of like generalized thing but without having like a really specific concrete use case very cool thank you so much for for uh coming on with us and explaining I guess what do you call it the state model of uh for yeah so I guess the penumbra State model let's let's uh put a little trademark TM motion somewhere in there um thanks again yeah it's been great so

thanks again I also want to say a big thank you to the ZK podcast team Tanya Henrik and Rachel and tour listeners thanks for listening thank you