Episode 289 - Penumbra Catch Up with Finch, Erwan and Jen

Transcript

foreign [Music] welcome to zero knowledge I'm your host Anna Rose in this podcast we will be exploring the latest in zero knowledge research and the decentralized web as well as new paradigms that promise to change the way we interact and transact online [Music] this week I chat with members of the penumbra team Finch erwan and Jen we dive into how the CK powered blockchain Works under the hood how ZK is used to offer new paradigms for staking voting and trading what kind of cryptography penumbra is using how the decks works

and more now a quick disclosure I'm an investor in penumbra through the ZK validator and this is a follow-up and catch up to our earlier episode which was an introduction to penumbra that we recorded last year right around the zero knowledge Summit 8 event so this will be an amazing catch-up now speaking of the zero knowledge Summit I did want to let you know that if you're planning on joining our upcoming event that is zk10 happening in London on September 20th be sure to get your application in now and if you've been accepted and have received the link to buy the ticket from Eventbrite be sure to grab this ASAP only folks who've bought the ticket will be guaranteed entry into the event so hope to see you there also keep your eyes on the ZK jobs board there you'll find new job postings from Top teams

working in ZK including some of our sponsors for the ZK Summit we've been thinking about the ZK jobs board as a neutral ground for teams to share some of their new openings to the larger ZK community so if you are a team looking for new Talent be sure to check it out and if you're looking for your next job opportunity be sure to check out the latest job postings links are in the show notes now Tanya will show a little bit about this week's sponsor [Music] anoma's first fractal instance namata is launching soon namata is a proof-of-stake L1 for interchain acid agnostic privacy namata natively interoperates with fast finality chains via IBC and with ethereum via a trustless two-way bridge for privacy namata deploys an upgraded version of

the multi-acid shielded pool circuit otherwise known as masp which allows all assets fungible and non-fungible to share a common shielded set this removes the size limits of the anonymity set and provides the best privacy guarantees possible for every user in the multi-chain the masp circuit's latest update enables shielded set rewards directly in the shielded set a novel feature that funds privacy as a public good follow namata on Twitter at namata to learn more and join their community on Discord discord.gg forward slash nomada so thanks again anoma and now here's our episode [Music] so today I'm chatting with Finch erwan and Jen from penumbra welcome to the show all of you hello good to be here

hey so we had Henry from penumbra on the show last year and he introduced us to the project and many of the things that were being built at the time but like a lot of those early stage projects a lot of I'm sure change and evolved over this year and so I'm really excited to meet with all of you to find out what's new uh what are you working on what challenges have possibly been solved what challenges remain open yeah and just get sort of a lay of the land of what's going on in penumbra I do want to start off with the intro to each of you so if you can just say a few words about maybe where you got started and why you're working on what you're working on I think that would be really cool to hear erwan so on my end uh I'm a

core engineer at penumbra the way I got started it's kind of funny uh I happen to have like a bunch of interest in cryptography doing the market design in privacy and got massively note sniped by the projects and you know started like contributing and eventually like just proceeded to work full time on it had you been working on something in this field before or was it like a very different space not really I have been lurking for a while uh born in Anon capacity I guess but yeah no this is uh this is essentially like the first uh the first time I work uh publicly on a crypto

protocol and yet it seems like you doubt you dove in sort of on the deep end this this is not a beginner project I don't think no no no yeah I've had a long interest in in the field and so um I was familiar with a lot of the work that uh Henry for example had had produced I think mainly the the Ed 25519 library that he and others wrote in Rust so this is uh this is basically how I came back to work on penumbra got it Finch tell us a little bit about what got you started yeah so um a bit I guess my my work adversary is coming up in something uh

like a few months so I've been here for about two years um since just up at the beginning Jen is a slightly more senior than I am I I had known Henry before he got started on penumbra and uh so he told me about how he was working on this project and it was very exciting to me um I had sort of like drifted into working on uh security and privacy Sideways from having a background in programming language design it seemed like a really exciting thing to be a part of so I signed up and since then uh my my role is also core engineer most most of us have that role

um because in the beginning it really wasn't possible to differentiate who was working on what um I've I've touched a lot of parts of the system I focused a lot on the design of our core data structures especially like the tiered commitment tree that's used for holding commitments to both notes as well as other pieces of private state and I've also worked on our re-tooling of the jellyfish Merkel tree which is borrowed from the DM project um I guess yeah metastm like the Libra now Libra then DM then dead and now back from the dead oh

um because we're using it and a whole bunch of other projects have cottoned on to the fact that this is actually a really great design for a Content addressable Merkle tree so we're using it and trying to um uh help centralize the ecosystem around this particular tree Finch I want to ask you were you working with Henry before you sort of said you knew Henry but were you working in this field before joining penumbra I was working not with Henry um but I was working at another blockchain privacy company before I joined penumbra um but then before that I had never really thought much about cryptocurrency at all

um it was uh I I had thought quite a lot about privacy um but it was a pretty quick lateral jump it's been really exciting yeah so you were coming more from that angle from the privacy in not from the like blockchain on yes I learned I learned quite a lot uh in my first couple of years in the industry that's for sure Jen tell us a little bit about yourself yeah so I think I'm similar in that I came from the Privacy World previously so I was working on sort of tour adjacent projects um so previously I worked on a whistleblowing platform called secure drop which uses toward-in services so if

a source wants to uh share something with journalists in a secure and Anonymous way that was the tool it would use so I worked on that for maybe five years previous to this so no cryptocurrencies involved um but I was drawn to penumbra I knew Henry also uh through kind of the the greater tour web I guess and I thought that the Privacy goals were important and the technology is really interesting and we get to use roster which is also quite sick something like yeah what do you what parts of the stack are you working on today these days I mostly work on the uh the core cryptographic Primitives and

implementing the proofs uh that doing the snark ceremony those are the pieces that I'm mostly working on would you put yourself more on the research front or on the engineering front or both I would say both yeah okay like part of the work is you know like many kind of cryptography people in cryptocurrencies taking uh paper and implementing it and so it's sort of like research engineering eh cool applied cryptographer yeah it's the rare role that everyone is always hiring for erwan we didn't really hear from you what part of the stack you're working on so what yeah what parts would you say you're focused on lately I've been focusing on the tax engine anything ranging from okay like

how do you go from a shopping time to an outcome for the user that's uh that's positive so how do they get the trade that they want um so that's the main part I've been I've been focusing on um that sort of like joins with like my interest about the cryptography and Market design because one of the ways I've learned about penumbra was that I was very interested in that concept called frequent batch auctions um okay which is like this concept that like oh what if like instead of having this rates towards latency and all this money thrown uh having like the slaves and sabotage game what if instead we badged all the trades together and executed all of those at discrete interval so switching from like a continuous vision of time to a more

discrete one and in traditional Finance this isn't really I guess a super easy things to do because well time is continuous and if you decide that your exchange is going to run this on this microstructure model then you have to sort of make sure that everyone else in the world is also running on the same model because otherwise well someone else can just run a continuous exchange and now you're you're nice microstructure is sort of exhaust of all the toxic flow coming from those places in blockchains however what's cool is time is discrete we produce blocks at regular intervals and it makes complete sense that the market

microstructure would map cleanly into this model of a discrete clock and that's really like when uh when I read the the protocol spec for penumbra that it clicked for me and I thought that was really interesting and incidentally it solves like a large number of kind of ux and fairness problems that uh arise in crypto So reading this I was like okay like you can choose you can design your your decentralized exchange to not have batch uh swaps not have batch execution and decide to execute every trade against the entire State one by one or you can just patch them together and kind of have finer control over okay like remove sequencing and ordering as a as a thing

that validators or block producers need to uh be very involved in so that's one of the aspects I I found really interesting uh but yeah that's mostly been working on the decks you just mentioned also this concept of like intense yeah the the intent in penumbra is kind of like simple uh you submit a trade you submit a swap and your intent this didn't let many intent to express your intentions to get the best possible execution possible um period so the the role of the text engine is to take a bunch of those intents of hey I want to swap this amount for that amount and work with the liquidity that is available work with all those different knobs and levers to

produce the best outcome for the user hmm so I want to actually do an intro to penumbra I feel like you've started to introduce a part of it which is that Dex but why don't we take that step back I mean even though you know I'm going to obviously add the link to the episode we do with Henry I think it's valuable to the listeners of this episode to learn a bit more about the project in that sort of high high level way so what is penumbra so penumbra is a an L1 where a shielded Cosmos Zone built on top of the tendermint now uh Comet uh bft stack um we are built entirely in Rust and uh the way that penumbra works is that when

you do an IPC transfer into penumbra this Shields the funds as they're transferred in and an IBC transfer out of penumbra unshields the funds to the destination so everything that happens inside of penumbra is the zone of privacy um we've essentially aligning the the boundary of the chain with the boundary of your expectation of your privacy of action and then within the shielded pool itself this is a multi-asset shielded pool so all of the value in the the pool is within a single anonymity set regardless of denomination we have the shielded decks which everyone has been talking

about and working on a great deal that allows users who are on the market taker side to get really really strong anonymity for the swaps that they're performing and on the market maker side for liquidity providers to preserve their Alpha to preserve their secrecy of their strategies because they're different micro positions are unlinkable from one another within this model would people using penumbra then like they're coming from outside of penumbra they go in they use it as a DEX and then do they leave or is there are there other things that they can do in the zone itself currently penumbra is the

decks shielded staking shielded governance delegator voting is also shielded using some of the same mechanisms but there's no general purpose programmability for penumbra 1.0 and I say that with an intent to a 2.0 or 1.5 yeah well there's there's a lot in the road map looking forwards like penumbra when it launches is not going to have this thing that we call Flow encryption um which we originally had on the roadmap and that uh allows the aggregation of swaps and the aggregation of delegator votes to be encrypted to a threshold key of all of the validators

to provide even more enhanced uh indistinguishability of these different actions that's uh on the roadmap for pin number 1.5 after relaunch and then for penumbra 2.0 or 3.0 there's all these really cool ideas about general purpose programmability using an async batch oriented model for smart contracts um but that's a bit more like I was going to say in the clouds actually um Henry gave a talk about this at ZK Summit eight um talking about some of the possibilities in the future for async uh models of computation and how that fits

into ZK well I'll add that we'll we'll dig that up and add it to the show notes along with your talk Finch from the same event which hopefully we'll get a chance to talk about as well absolutely okay so this is really great you actually just mentioned a part of this project that I wasn't sure was still in the works and that's this I think you called it is it shielded stake or is it shielded delegation shielded staking or shielded delegation I think like those are both sort of uh synonymous at least in my view yeah I can describe a little bit more yeah please do because that just as an aside like ZK validator when we started zero dollar validator we had our dream was to be able to do some sort of like private delegation like

that you know more in the actual you know proof of stake stack using ZK using privacy and I remember going this is a little aside but I remember like before we even started the thing before we started CK validator I went and talked to Sunny and I was like can we just make Pro like Cosmos private delegation like really like very naive so then he had to explain like how this entire thing worked much better to me and it was very clear that at least on existing systems it's not a small add-on and so when penumbra came around it was very exciting because it was like that was part of the proposal and it's really nice to hear that that's still uh the plan because it's it is so unique to penumbra the fact that you'd have any

sort of privacy and and how you're you're staking and actually like the way that validators would operate or the way that like you know if you have tokens and you want to participate in the consensus how it how it looks but yeah tell us more about it so on penumbra our staking model is shielded staking for delegators and transparency and accountability for validators so if you're a delegator then it is not possible to link your different delegations to one another and it's not possible to see which address has delegated how much to which different validators the because addresses don't appear on chain period except with a

tiny asterisk that is for inbound IBC transfers but that's the technicality that doesn't actually factor in here um so when you delegate to a validator instead of having a globally readable Ledger record that delegation so that rewards can be minted to your address transparently instead it's an exchange of your unstaked tokens your number tokens for a specific delegation token that is unique to each validator and then when you choose to undelegate at some point in the future the exchange rate between that delegation token and

the penumbra staking token has been adjusted by the chain and so you get back more penumbra staking tokens than you put in proportionate to what the staking rewards would have been during that period of time and that's your like whereas most transparent systems will have like daily emission almost of rewards in this case you as the end user only see that once you take it out that's right that's right and you could partially undelegate you could at any time choose to partially undelegate uh only the amount that represents the emission that would have happened up until any particular Point um essentially you can think of it as like on penumbra staking rewards

compound automatically and what this also means this design for privacy also gives us first class liquid staking just as a consequence of how it's built because each one of these delegation tokens is itself something that can be moved around um can be sold can be traded can be moved out of penumbra over IBC there's a lot you can do then with this mechanism that just isn't possible without further Machinery if you have a conventional sticking mechanism like the cosmos SDK and if I remember correctly from that previous interview I did I think don't those tokens also add to the anonymity

set almost like they just add an extra class of tokens to mix with the other tokens also kind of like further adding I don't want to say noise but kind of like yeah stuff that makes it harder to decipher who's who and what's you know what's connected the more things that happen in the shielded pool the more powerful the shielded pool is in terms of its ability to protect people's privacy yeah um so everything is mixed in there um when when you're a liquidity provider you get back a shielded nft that represents your capability to then manipulate the position that you've put onto the decks when you are participating in swapping uh there's a

commitment that goes into the exact same uh commitment tree that represents your claim on the outputs of that swap everything uses the same mechanism of the shielded pool and so the anonymity set is compounded by every kind of activity that could happen when it comes to stuff like the private governance shielded governance I guess shielded is this private voting is there a different mechanism are you using anything like Macy or anything like those constructions that have been created specifically for voting or is it bespoke and it just uses the the properties that already exist in the system it's um bespoke to penumbra um we're not doing anything particularly

special um we the the way that uh private voting works is that you to you know as a as a validator you don't have privacy in voting let me first make that clear a validators votes We Believe should be attributable to the validator so that people can place their delegations with validators whose on-chain voting record they actually agree with um but as a delegator you should have privacy when you're voting you shouldn't have to choose between participating in governance and giving up your privacy so Del alligators proving that you have a particular voting power for a particular proposal looks like proving that you have spent Authority for your delegation

tokens to a particular validator or rather that you did at least have spent Authority at one point in the past when the proposal was first created because the delegator can switch their vote from the validator if they want right like they if they don't want to follow and just like Let It passively Be voted by the validator they could I mean this is the same in all Cosmos SDK governance as far as I know and at least the ones that we we're active on you can always split with your validator if you want to um and but yeah that was always sort of the the question here about Lake day some sort of leakage like if you split from the validators vote then I guess can't you put together that that delegator had delegated to that

validator in a weird way because it's like all of a sudden it's like oh there's 10 000 tokens worth of voting power missing here's a delegator that's voting that amount and this this validator has gone down by that amount like is there any sort of way that you can connect it statistically yes and the fewer number of delegators there are the easier it is to do yeah so imagine that there's only two delegators to a particular validator then you know when one of them has chosen to vote otherwise because you can simply see like this like we saw how much a particular delegator delegated say they delegated all at once like the worst case scenario is you can see the amount of a

particular delegation because flow encryption doesn't exist yet then you can see they chose to differ from the validator so you can identify that the same person who delegated voted differently from the validator but you don't get any more information about them you don't know what their address is you don't know about the rest of their on-chain activity you can't link that with anything else so it's limited in impact and with flow encryption this gets even harder to do because it means that we batch together all of the delegations and undelegations over a particular Epoch and only the sums are decrypted by the threshold of validators at the epoch boundary and similarly for

voting we batch together all of the votes and decrypt only the totals of the votes so that you can't see intermediate tallies which would reveal more fine-grained statistical information that you could use to try and de-anonymize people got it it and actually just to you sort of already said this but the separation where like the validator is transparent and accountable the delegators those who have the tokens who are staking that's where that privacy lives I think that's a really important and it strikes a nice balance because I think some of the issues when we originally talked about like private proof of stake was oh but how do you keep these validators accountable in that case like they can be malicious or they could be doing

something weird and not paying out their rewards in certain systems where you have to manually do that there's like all these things that would go wrong if the validators were actually anonymous but the delegators themselves because everything is so transparent in most of the systems everyone's actions are revealed yeah and I mean this is a little bit of a political perspective but um it seems to me like privacy for individuals but transparency for institutions is a good way of organizing a society and I think the validator as being a lot more like an institution than an individual and you can override a validator's vote by knowing how they voted in a particular uh proposal before

you yourself cast your delegator vote so that transparency there is necessary because otherwise you don't know like how critical is it that I override my delegated Vote or not mm-hmm you just mentioned that the cosmos SDK or the cause you were using like tendermint but now it's called Comet I don't know if I said this right but what is this a penumbra thing or is this someone else's work this is not a penumbra thing this is a a rebranding of tendermint the like organization and application and all that jazz got it um okay so tenderness new names is the same thing or what yeah new name same software um tender mint is now called Comet bft

um and it's this it's the same project it's the same architecture but there's I I'm not sure exactly what was uh behind it yeah some sort of issue and so there is the original tender mint repo but I think most of the activity it seems is happening in the comet bft World got it yeah this sounds like a little bit of Cosmos like drama reasoning maybe but yeah we are simply going to use Comet bft because that is the one that is going to ship photo extensions very soon which is what we need in order to enable blow encryption for penumbra one point whatever which we're very excited about

after our main net release I want to ask a little bit about IBC and penumbra because IBC is also I mean this is a transparent system and you mentioned that like when you're outside of this system and you're coming in there is this one point where it's transparent I guess because you need an address to give to whatever other chain where you're sending on penumbra that would be transparent what happens after that good question so it's different in different directions so first looking at an outbound IBC transfer from penumbra this looks very similar to an IBC transfer onto any other chain or from any other chain

penumbra invokes the appropriate methods in the IBC State machine so that the relayer knows that it should tell the other chain that this amount has been withdrawn out of penumbra and all is well in the other direction deposits into penumbra need to specify an address on penumbra and this is the only place in the penumbra architecture where an address is actually recorded in the chain State at all everything else you see no trace of anybody's address on chain but in this case you do need to be able to put an address there so that the chain can Mint new notes that are spendable by the recipient of the

transfer which is why and I think Jen you can speak more to this we have a key hierarchy that allows you to generate essentially unlimited numbers of completely unlinkable addresses so that each time you do an inbound IPC transfer you use a fresh one and they're not linkable to each other but they share the same spend Authority so if you wanted to after you IPC transfer into a particular address you could immediately move it to another address or just spend it from there and that activity is not linkable can you explain what the key hierarchy part of this is or how that works yeah so we have keys that have different capabilities that are all derived from a root a seed phrase similar to other

chains and our key hierarchy is very similar to zcash because penumbra is based on mostly zcash sapling and so one of the nice features of zcash sapling is that they have what are called Diversified addresses which are addresses that are all different when you share you know an address to Alice and an address to Bob they are unlinkable between Alice and Bob um however they all share the same spend Authority which is a nice privacy feature this might be like a bit of an odd question here but like when I think

of the zcash model with the utxos and the notes yeah at the same time like is penumbra than utxo at its core or is this like there's something underneath and you're running sort of an emulation of utxo on top it is as at its core utxo using notes just like zcash the main difference is that we allow arbitrary assets so a note has typed a value where the type is a specific asset and that's also private so it could be one of the nfts that Finch has mentioned could be the delegation token for a specific validator um or the penumbra token and that's not something that is uh public

can you have other types of assets non-penumbra assets also in here and those would also be their own type I guess yeah so presumably um wants people IBC transfer in you know their favorite token then that'll be in their print number Zone and will be a new asset cool how then though does the decks work with that like so is the decks a separate program that's sort of like just tapping into the utxo like to to use it to read things but it's doing things with it like I I guess I always I always have in my mind the idea that like a utxo model is very like unprogrammable that you can't do that much with it other than transfer and I know that's changed like I know teams like Elio would say it's changed teams

like Aztec will say but like you know I know that there's variations on that but yeah I'm just curious how does the decks interact then with this um that's uh that's a really good question at a high level when we design and built the decks there was like this this constraint of we are working on a permissionless protocol where people can as potential said IBC Assets in and out so we don't have control over the space of assets that are present in the decks people can open positions close positions we can specify bounds on the positions but really if if a trade happens to be expensive to process or if an error occurs during the execution step of the decks

it's hard to attribute where the error actually came from who's responsible for maybe like a spike in processing cost so we really had to to treat that as a first class concern of building something that's robust enough and that is basically like too efficient to trick into wasting compute and the way this connects into okay like we have those shielded notes we have those different components part of penumbra is really core to the application architecture that we converge onto which are different components that stack on top of each other that have a specific sort of like generic Behavior specified into a trade

that essentially splits uh Behavior into three steps doing stateless checks doing stateful checks and finally doing some processing uh some sort yeah so I think one of the critical um things to recognize that differentiates penumbrance decks from other dexes out there like on ethereum is that penumbra's decks is built in to the protocol level so just as at the protocol level a spend exists um a swap exists on the protocol level got it so does it have a category in the utxo part like does it have its own field almost yeah so the different kind of protocol level things that you can do in penumbra are called action so spend

is one of them there is a swap which is when you're sort of signaling your intent I would like to swap uh one per number for one you know wrapped Bitcoin or whatever it is um and then there is a swap claim action that allows you to collect the outputs of that swap once it has been executed against the decks and there has to be an intermediary step there has to be this two-step dance of Swap and swap clean because when you submit a swap you don't know what the clearing price is going to be every everybody in that block who submits a swap is going to get the same price it's going to be the best price that's available given the liquidity that is in that block but what you get back when you submit a swap is

effectively a handle that says when we know the price I can then prove that I had this chunk of the total swap volume and then get out the proportionate output of all of the swaps across that particular trading pair and that's sort of going back to that that you're batching it because the whole trade is is not just you it's like a lot of similar intents I guess bundled together that's right and that's where the Dex engine comes in because it's the Dex engine's job to take all of those swap in the the swap intents aggregate them together by the trading pair that they're associated with and then do routing over the graph of all of the liquidity that's currently on offer in

order to optimize the best possible price for those aggregate swap totals I guess that's my picture of the high level of what everyone has been Deep In The Weeds of optimizing what does it mean to optimize that everyone what do you have to do so at a certain level it's designing the Vex engine as we call it to make sure that even though we can't attribute processing complexity to a specific user action because it might have happened in the past everything is patched some things are shielded even though we can't do all of this that you would do in a normal sort of like trading engine it's kind of this you engineer things so that

a lot of the complexity a lot of the processing is done is front loaded at certain steps of the decks and really what is on the hot path of determining okay how much inventory am I going to Route over the specific route what is the impact on the marginal price doing that entire thing of like past search routing and the combination of the two that's actually like executing trade on the text doing all of this has to sort of be regimented in two phases where you are going to build indices a prone that are going to help you prune a part of the state that are going to act as juristics of like okay like this is where I should Orient my path search for example even though you know the liquid quarries permissionless anyone can IBC

deposit assets or Bridge assets anyone can create positions so you have this liquidity graph where vertices in the graph are assets so for example ethereum Bitcoin whatever like any asset you want payback and edges between those vertices are going to be positions and stepping back again really that liquidity graph is a multi-graph so between two assets you can have many positions each position in penumbra is this constant Price tool or a m in a way that is kind of akin to a limit order with some caveats so for example really what you do when you open an equity position on penumbra you specify a price you specify an inventory and then until

you decide to close or withdraw the position the position is going to stay on the order book and when it gets filled automatically start quoting the other side of the book so each of this position has a linear cost but routing over different hubs has a convex one because all of a sudden you have to compose with two different uh sort of like price points uh over two different parts of the route and doing this in a permissionless context is it's kind of a really interesting problem actually because you have to step by step iteratively decide okay this is the optimal amount of flow to Route over this given route and then

step back a little and say okay now that I have altered this view of the state how do I uh how do I decide what is the next step how do I decide what are what is the next best route what are the next positions that I should consume and doing this just naturally over like you know sort of like a knife storage implementation would be excessively expensive so what we did was build this very neat storage uh and caching layer that lets us do speculative execution at different steps of the processing sort of like life cycle of a batch trade and so we can actually like say okay what happens if we route all the stuff in

that direction what happens if we do it in that other direction doing this like very efficiently and at the same time have sort of like this escape hatch to revert to a known good state if anything bad occurs but of course in in and of itself that wouldn't be sufficient because we need to bound the amount of work that we do processing trades so in addition to this there is like different layers and this goes back to this phased execution thing where we do path search we don't consider the entire search space instead we use indices that we build up front at when we write and update positions into the the Dex State we maintain those indices

and using those indices as heuristic heuristics we can trim the search space figure out an estimate of like the best couple rats available and using this pipe that to a second phase of actually filling or executing that is going to be in charge of managing that sort of like convex cost function and convex problem of deciding how much flow I should route so that I don't go above a certain like Mark like a certain price essentially a certain we call it the spell price so this goes back to like this comebacks problem which also has a has a computer science equivalent in that sort of like Niche called Network flows where you

have to maximize the amount of flow through a network but minimize the marginal price increases that routing any additional units causes so that's that's part of the the way we addressed those challenges another way is also choosing a very simple but very versatile liquidly primitive which is the constant Price Market maker the console price Market maker you specify a price you specify an amount of inventory on either side of the the pair that you're that you're quoting and that's it uh there's it's a linear function it means it's very easy to compose new functions with it it's very easy to have a precise approximate a precise trading

function I mean you keep mentioning convex convex and all I can think is like Guillermo works on that um have you worked with Guillermo at all on this is this anything related to the work that he's sort of talked about on the show absolutely absolutely yeah we had a really great collab uh with Guillermo um so a lot of this replication Market making work is actually a completely pioneered by the day I say the seminal work of Guillermo Anguirus of Alex Evans and Talon Chita in their replicating market makers paper and uh totally this is this is actually I think one of the cool instantiation of This research work into a soon live TM engineering project where

yeah you have this simple liquidity primitive you have those quote unquote consistent payoff functions how do you approximate that with this and so that's what we that's part of what we worked on so you just mentioned though this Pub like the storage you have some sort of storage state or something like what I was thinking as you were talking about this it's like if you have these intents and you want to batch them how long would it take would you almost like need to make the intent and then wait like a few days for them to sometimes happen because you're waiting for other things to be batched along with them is that why you need the storage I'm I'm sort of asking two questions here so maybe let me break it in one is would you need to wait a long time could you end up

waiting a long time in this system to get your trade fulfilled and then the second one is is that held in storage is that why you have storage it depends on the tax state but typically the expectation for a user would be to send a swap and it's fulfilled in that block right the only reason why it would not be failed is that for example you were executing against an empty order book but much like you know classic kind of boring centralized exchanges um when you do a market order there uh there's not really like the intent is a little bit loose because the intent is aggregate me at any price but uh you would typically be executed at any price right in penumbra the difference is your intent is just it took me at the best

possible price and you have this sort of like guarantee that the Dex engine will actually look at the entirety of the liquidity graph not just the pair you're looking at but the entirety of the liquidity graph and find all the best available liquidity and execute at the clearing price for that block and but still pretty immediately like it would be great yeah absolutely I think the finale should be five seconds essentially okay so you're not saying the order here the intent is not I want this for this much and I will wait until I can get it for this much oh that's that's right yeah that would be for example a limit order which you can also Express uh you can a swap oh yeah absolutely yeah so penumbra is more like uh it it's more a decentralized order

book than it is uh say like an automated Market maker if you're a market maker or just a swapper or like Trader you have fine control over the like the trading functions of the the shape of the liquidity you want to provide say like I want to provide only like like the the current prices on like some third-party reference Market and as a as a just Trader you can also specify hey I want to either execute a market order having that notion and that guarantee that the Dex engine will actually look at the entirety of the liquidity graph including the synthetic liquidity so for example if you say hey I want to swap BTC for eth you're not just going to be executed on the

btc-eeth pair you're going to be executed perhaps on the BTC USD USD if there as well so you get your trade can take many different routes and actually the extension will gather all those different routes and fetch all the best positions and provide you with the output at like the best the best price and we have a this active Market maker bot on our test net that kind of like does a little bit of what I talked about previously with active Market making um it's called Osiris it was written by my colleague Chris and what Osiris does is it sort of like implements this simple Market making

strategy of looking at like a third-party reference Market quoting the on the penumbra order book the different assets at those like you know reference prices and maintain liquidity that way um what that means is on the test net when you submit a swap you're going to be executed both against you know active market makers like Osiris but also against you know passive market makers or just users who are like posting limit orders much like a central limit order book except it's decentralized I want to go back though to the public State storage system you need that because you need somewhere to hold these orders in place as they're being executed or like what's it for so every blockchain needs a way to record its state and uh most of

the time this is done on a block by block basis where you accumulate in memory the set of changes that happen over the course of that block and then you atomically commit here are all of the changes that occurred during this block because it might be that something failed or didn't make sense or whatever and like when you when end block happens you commit what your state is and on penumbra we've developed this bespoke storage infrastructure that is meant to allow us to do all sorts of cool stuff with the state so that we can enable this speculative execution of the decks Etc what that looks like is at the base level we have roxdb which is a

Facebook originated Fork of the level DB database that is a key value store on top of that we have this bespoke storage layer that lets us make in-memory copy on write Forks of the state so that you can say I would like a copy of the state and anything I do to modify it is not going to be committed down to the storage layer until I say please commit this and then you can Branch those Forks arbitrarily many times so you can be like 12 speculations into the future and only when you finally say yes this is what I want to actually change about the world does it collapse down and the like Quantum wave function coheres or

whatever and you get a final actually committed durable state so what the Dex engine uses this for is when you're doing path exploration you might say suppose I tried executing this trade across this path what kind of price would I get and this looks exactly like actually executing the trade across the path except that the state that you're working off of is this ephemeral copy on Right Fork of the actual state that exists only in memory and only when you've found the best possible path you say great now commit these changes down we've tried to package this up so that it's an independent component because we

think that other chains might be able to use this for their own use cases so there's a rust library that we've published with this what's it called um I believe it's called penumbra storage right now it's in our giant mono repo somewhere cool so how does the cryptography that you've been building and working on how does that actually interact with this decks so one of the ways that it interacts with the decks is as we mentioned previously the decks is built into the protocol layer and so the Swap and the claiming of outputs of a swap are protocol level actions and so when you claim the outputs of a swap for

example you need to prove that you are claiming the right amount so we have executed a whole batch of swaps together in that block and then you are allowed to take out just the part that corresponds to your uh inputs and so you we prove in zero knowledge that that's one of the places that we use uh zero knowledge proofs that that has been done correctly which zkps are you actually using we use uh Groth 16 proofs at this point and do you think there'll be a move at all to something more modern or anything sort of like let's stay on the true intestine side of things yeah so for mainnet it's going to be growth 16 okay proofs and

the reason for that was just that we are a fair fairly small team and we can get really good out of the box performance using existing libraries by using growth 16 and so we will be doing a setup ceremony we just had Lucas Meyer join us and so he's been working on uh the snark ceremony that we're going to be doing in soon TM but maybe a couple months real cool at some point in the future it is possible that we migrate to a different system are you sort of like on the lookout for those systems then like with all the activity and new systems coming online I mean there's also been I mean since growth 16 According to some narratives Planck would have been like uh also

quite battle tested well-versed although there's lots of different variations of it you know maybe the Halo 2 stuff like are you are you sort of avoiding that because at this moment because of its newness and like maybe lack of tools are you skeptical if it's security like is there I'm just curious if you stay stick to graph 16 because of ease or because of security at this point it's primarily for ease so a lot of the development work that's been done over the last year are implementing like rank one constraint system gadgets to use with growth 16 and so there is some work to migrate to some another system that's using like a planckish arithmetization um but like Halo 2 would be you know

very high at the top of the list like there were definitely closely following other proof systems um for a possible migration in the future like for number of 1.5 or or two could use a different proof system you sort of mentioned you get you have to build some of these like custom Gadget like very specific to the system but if you do like how intense is pulling out a proving system and sticking in a new one do you have to hire new people almost I don't think we would need to hire new I mean it wouldn't hurt but I don't think we would need to hire new people um it would just be so well like one of the differences about the number is that we have also implemented um some of our own crypto Primitives and

so like we have our own elliptic curve library and so we have to implement whatever gadgets we need for the proving system um and so there's a little bit of additional work there but I don't think it it's a massive lift to change where else are like ckps being used throughout the system so you mentioned the one this one case in the decks but yeah like and I guess we can assume in voting but yeah I'm just curious like where exactly how many are there yeah yeah so uh you're right in voting so in delegate to voting we use the ckp there um and then for the multi-asset shielded pool uh similar to Z cash spends and output so minting new notes or also have

zkps associated with them and then for uh on delegating to make sure that you're applying so if the validator was slashed while you were delegated to it then we ensure that you are applying that penalty as you convert back to penumbra do you also use a zkp when you're going from that like transparent IBC input state to into the system there is not a zero knowledge proof there okay there's no separate action because on the inbound Direction you have the chain transparently like when you do an IVC transfer from another chain

you on the other chain initiate the transfer then the relayer does an action as a penumbra transaction that says to penumbra here's an incoming transfer and then penumbra transparently mints notes to the specified address which then means that those notes are part of the shielded pool but because that address is a one-time use ephemeral address then whoever controls those notes then moves them around throughout the shielded pool as usual and then in the other direction the person who wants to transfer out says I'm going to spend these notes

which releases value that you can then spend on an ics20 withdrawal that allows you to send outwards are they just all like completely separate systems it's triggering something within a private system but it's not like there's no transfer into it there's no visible like does it just stay then in that transparent penumbra account I'm sort of trying to follow it so like you're coming in through the IBC there's some sort of like it hits somewhere that like yes there's been a transfer in that is visible and then an action happens in the shielded pool at the same time the shielded pool uh if you're talking about the perspective of the penumbra node software the shielded pool has the

authority to Mint new notes within itself okay and so when there's an incoming transfer it can say by Fiat here's a new note that should be considered to be unspent and belonging to the spend authority of this address that is the one that's being deposited into so it just it simply does that and then you the recipient of the note are all ready to spend that note if you feel like revealing a little bit of my ignorance about IBC transfers in this question because what I actually am picturing and this is because of the way that we usually use IBC is to take you take a token you send it over IBC it lands in the other place and then you

see it in an account but what you're saying like is the re-emergence of that token on the penumbra side when when it goes to penumbra does it just not appear in that transparent thing at all it just sort of like triggers something so you never have it like land on the penumbra side visibly as a token it becomes a utxo on the penumbra side in the same shielded pool as everything else so if you if you think about the zcash model where there's the shielded set and the transparent set yeah and I think I might have a bit of that model in my mind so I'm picturing like it lands transparent and then you move it in but you're saying no no it immediately goes right into shielded

imagine that except that penumbra is the shielded pool and the transparent pool is the entire rest of Cosmos okay okay so an IBC transfer is like a t to Z transfer in Z cash um and and IBC transfer out of penumbra is like a z to T transfer and then everything else inside of penumbra is like C to Z I think I got it I think that actually that's really helpful on how maybe for the end user working with like what will actually happen when you're using and like if you're sending something over IBC for the end user it should actually look very similar to the user experience on any other chain yeah they'll just see

like oh I sent tokens into penumbra and it's in my wallet yes exactly okay but under the hood it's just like there's there's like if you're looking on chain if you're looking at some sort of Explorer I mean it we would not see other people's trans like transfers in once it reaches the sort of gate of penumbra like once the IBC transfer is hitting penumbra it sort of like goes into the shield you see yeah you see where it lands and you see how much because uh the amount and the denomination are actually recorded in the ledger so a block Explorer could show you uh like a ticker of all of the incoming and outgoing IBC transfers but it could only say here's this ephemeral address that you'll never see again and

have never seen before okay and so in in some sense we we've we've got a little bit of a toy block Explorer for penumbra that we haven't put a ton of effort into but we'd like to at some point TM uh take that and make it a little bit more full featured because the funny thing about a block Explorer for a fully shielded chain is how much it can't show you so it's very funny to look at this and be like well I sure don't see anything like although you would see as a user you'd have your own sort of personal dashboard somewhere which which could track your transactions and your trades I'm guessing like you'd want to be able to

see that yourself absolutely and so we've been building a web wallet um that lives as a browser extension um which allows you to see your fragment of the global public encrypted State as if you could it sort of provides you this this prism that you can see into the darkness but only for the part that's actually relevant to you and so you can see with your transactions you can behave almost as if you are living in a transparent account model blockchain that's the that's the user experience goal that we want to level up to um it should be no more difficult and no

slower Etc than doing something on a normal Cosmos SDK chain you shouldn't have to think about the fact that it's all shielded nice other sort of crypto Primitives or like like you sort of mentioned you built the elliptic curve that you're using but like is there other things that are like built by penumbra for this particular system well another change we made from uh zcash in the key hierarchy is we use Poseidon the Poseidon it's not currently hash function in a few places and so we built um a Poseidon library and a general purpose library for generating Poseidon parameters for whatever project and

field they want to use so that's all publicly available and has been audited why would you not have used existing implementations was there nothing sort of up to the level you needed to feel secure yeah like I think when we first started um maybe two years ago because doing the crypto primitive is one of the first things we had to do um when we were doing the Poseidon parameter generation for our field we saw that that the there was sort of a few kind of Python scripts that had sort of been kind of hackily modified um over the preceding months but we wanted to have something implemented directly from the paper in Rost

um that was general purpose uh and and solid and I think other I think filecoin has something similar now that I think was developed around the same time and maybe other projects have done that since interesting so we're almost at time and I want to hear a little bit about kind of what's coming up you know you'd sort of mention some of the roadmap things but yeah share with us some exciting features that people can maybe look out for or maybe just like the ways you can see penumbra being used in the wild so one of the really exciting things that we have been plotting um is this thing that we're calling narsal and narcel comes from the realization that every pin number transaction is effectively a micro

roll-up can retaining only the data about that specific transaction in essence a penumbra transaction is two things it's a commitment to the state change that that transaction affects and it is also an encrypted package that allows the person who initiated the transaction and also the recipients who interact with its outputs to be able to decrypt the relevant information for themselves however if you are the only person who is interacting with your own transactions say you're a big Market maker or something like that you are doing a lot of sending things to yourself over and over again oh yeah and you don't need to

actually post on chain and pay for it in fees all of the encrypted data blobs that are addressed to yourself you already know your own data but if you were to lose that data then you would lose spend capability over everything that you had done which would be pretty catastrophic so we're building this tool called narsal which is a personal self-hosted roll-up chain that you can use to do threshold key custody for your penumbra activity and also allows you to save on gas costs by avoiding actually placing on chain the self-encryption of

all of the transaction data that's addressed only to you so what are you excited about erwan what's coming up on your side uh there's a bunch of things but I think um for me it's not really a specific product feature but having having penumbra run um and operational and be sort of like this hopefully agent of of changes is very exciting to me I think our philosophy speaking for myself but I think we have a short Norm around this is building something that's privacy preserving but also really useful and a Dax for example if it's a good product what it means is that it has good liquidity when you trade on it it has the correct and good prices and it's

there when you need it and that combined with okay like actually also like it's not just convenient it also like protects your your Trail it you know guarantees that like what you do isn't completely observable by the entire world at any time forever to me that makes me really excited because that's that's something that helps a lot of other things come about a lot of other comments emerge when they couldn't before a lot of um specific you know maybe types of organizations to to emerge as well um so I think I think it will be good capital G and I'm excited for that nice Jen what about yourself in terms of the cryptography like you've sort of

mentioned you know at least for this first version we're gonna you're gonna stick with growth 16 but like are there still pieces you have to build is there anything kind of coming up uh that you'd like to see included I think the next big thing um that's coming up right before mainnet that we would like to have people participate in is uh this knock ceremony um and so that will be something that you know the more contributions uh the better and so that's going to be really cool nice nice cool well thanks to all of you for coming on the show sharing with us an update on penumbra kind of doing a little bit of a deeper dive than we did last time to be honest into the inner workings what you're thinking about when it comes to the decks what that actually

offers to the to the end user the private delegation which is obviously like a highlight for me yeah thanks so much thanks for having us happily thanks for having us cool and I want to say thank you to the podcast team Henrik Rachel and Tanya and tour listeners thanks for listening thank you