ZKPodcast: From Warp Sync to SPREE with Polkadot's Rob Habermeier

Transcript

[Music] welcome to zero knowledge a podcast where we explore the latest in blockchain technology and the decentralized web the show is hosted by me Anna and me Fredrik [Music] in this episode an assistant with Rob from parody technologies to talk about his early work at parody how some of that is led into his work at polka dot and some of the exciting features he's discovered along the way before we start we want to say thank you to this week's

sponsor trail of bits trail of bits is offering affordable packages for young technology companies who are serious about their security it's often hard to hire security experts it can be expensive and tracking down the right talent could be time intensive still your team may have security questions right now and the code you're committing may need that extra review trailer bits has a service that can help Crytek provides continuous assurance for your Theor and smart contracts making sure they're safe and functional if you would like security help from this very experienced security team integrated into every pull request during your development cycle fill up the signup form on Crytek dot io now on to our episode with Rob hammer Meyer so today

I'm sitting with Rob hammer Meyer from parody hey Rob Hawaii how you doing pretty good now this is not the first time that you're on the podcast Rob you've been on before but this is the first time that I get to interview you which is pretty cool and today what we're going to talk about is a little bit of your background how you started at parody the work you did there your work on polka dot a little bit of an update around polka dot and substrate and then hopefully we can get into some like new ideas you've been discovering as you progress in this development so why don't we start with like where do you come from and how did you get here okay well I'm glad to be back now I can talk all about myself

yeah so so Who am I and where do I come from yeah so I'm Rob I am like I guess my official title is the the co-founder of polka dot I come from the DC area my background is you know originally it was sort of computer security that's what I was super interested in I got into like coding game programming optimization kind of stuff as well and you know the the security stuff is really what got me into crypto but then you know I was I was into this programming language called rust and that sort of the intersection of those two things is what brought me into the the etherium world and in touch with the the folks at Paradis who are using both

of those things that was sort of my entrance into the crypto space what was a connection there between like gaming and rust cuz it's not like a lot of games written in rust so why would that kind of stuff have led you to rust oh yeah so that didn't lead me to rust the computer security stuff is so I would do these things called CTF like capture the flags and the goal there is like you know there's two kinds of CTF so one kind is where it's literal capture the flag and you're playing against other teams and the goal is you know everybody has like a server and you're trying to attack each other's server and steal each other's flags and the other kind is where you're just solving some challenges and every team has like you know is trying to solve the same challenges and that's where I

learned how to do a lot of different kinds of exploits like a return-oriented programming smashing the stack that kind of stuff that you can do when you exploit like programmer errors that are dealing with low-level memory and that's in like 2013 or 2014 no 2013 I stumbled on the rust program language which said hey look all of this stuff is not possible in rust and it's still just as fast as C++ or C it was way different back then it was super unstable as really early days but as somebody would actually you know learned how to carry out some of those kinds of exploit sit really appealed to me I was like oh [ __ ] I could actually you know write something that's really fast I like to write things that go fast but I don't have to worry as much about things going wrong cool then I had this

background of computer security and I was super into cryptography and math like when I went to go study I started to do a bachelor's in math rather than computer programming because I just thought that that's like that's more general dad you know that teaches you how to solve problems so that that made it like a really easy entrance into the crypto world over the course of 2015 you know when when aetherium was really starting to to come up was when I ended up getting into the crypto world it made it super smooth because I had you know all the background I needed in cryptography and math and programming what did a theorem look like to you when you first stumbled on it yeah that's a good question

I think one thing that appeals to a lot of programmers in particular is the entire aspect of automation where at some point you could just set things in motion and they just go right and to me a theory I'm looked like a great platform for setting up things that just run now I think my perspective on that has changed quite a lot because when you deploy things at that scale that affects so many people it's never going to just run right there's always going to be some kind of intervention but at the time what really struck me was like wow

you can just program anything any kind of application to anybody all over the world can just jump in and they don't have to you know send their data over to Mark Zuckerberg or anything like that so I thought that was really cool that's really what got me most into etherium you sort of mentioned that you were like into cryptography but that was that like a hobby thing was that something you were actually studying I mean we had Shawn Bowen some time ago and history is so interesting where he like dropped out of high school to work with his friend who was a cryptographer like he's just super into it so I'm curious like what how did you really get into that oh no no I'm not that cool so well thank you yeah but I'm not that cool so I I was

more of like a hobbyist cryptographer was like oh this you know this is pretty neat you can you know I was learning about all the programs I got into to steganography which is the process of hiding payloads within things like images so that you know I could I could say like hey and I check out this funny meme and like you know if the government is spying on us they could see like oh you know he's just sending Ana pic that's you know funny like ignore that but actually hidden inside the pixels like I've slightly altered it so that it contains a secret message Wow so that kind of stuff I got more into like an overarching view of cryptography and privacy and things like that you know understanding on a low enough level how this stuff worked but I wasn't doing

stuff like hacking on elliptic curve schemes or whatever Shawn was doing so nowadays though you get a chance to work you're working like pretty directly with researchers and like cryptographers a lot do you feel like are you absorbing a lot from them now do you feel like your cryptography work or understanding is sort of like at an all-time high is it something you focus on now or is it more like just is it more implementation yeah my approach has changed over the past couple of years I'm very much interested in cryptography and I think if I had no other projects or obligations I would be like one thing that I find really

interesting is the the zero knowledge stuff like I first learned about that at the Stanford blockchain conference in 2017 and I saw a presentation by Eli Ben Sasse on about Starks he was like what do we call it do we call it Starks and we need to show hands and he was okay there starts and after that I was like wow this is so cool I went up to talk to me was like how do I actually get into learning that and you know from there I went and studied up a bunch on elliptic curves and finite fields and you know projective affine spaces that kind of stuff brushed up on the number theory and I got really into it I think I would be doing that at some point but my main goal is to not get lost in details is I'm trying to accomplish something very

specific with polka-dot and my goal is to always know just enough about everything you know you know I understand how public key cryptography works at a decently low level but I'm not a cryptographer right but I can build protocols that construct crypto economic incentives using that kind of knowledge you know I like to have a sort of a broad base of tools to draw on where you can build something more complex more powerful out of all of it when you first join parity though you weren't working on polka dot you were working on something else you were doing what were you doing there well when I first heard parody polka dot didn't exist there's F Corps back then charity didn't exist exactly when I first heard parody parody didn't exist

and the first things that I did were one I built warp sync for parody theorem so the ability you know it's it's it's a very practical feature but essentially the idea is to have the entire network of nodes periodically generate some snapshot of the entire state of the block chain and then disseminate this via BitTorrent like mechanism to any new nodes who want to sync you know at the time it was really fast you know you could get synced in like 30 seconds from the whole network now the theorem network is like 20 or 30 times larger and it takes you know a good time longer to actually get synched even with warp sync but it was you know it was a nice introduction to the the world of consensus algorithms in

practice and synchronization securely so what you had done is actually may you'd sped up the syncing process at the time by like a crazy amount but now just because of the size of the chain it's like does this does a warp sync still help or is it like too big for that to even be noticeable now well warp sync in principle is still viable but it's going like this is what a lot of the etherium one point decks efforts are on sort of to do further research into I guess adaptive warp and fast sinks like death's approach was fast sync which is not to do a compressed snapshot of the state but rather you know you have this rout of the state tree that's in every header and just to download this

recursively from your peers you download the whole tree state tree node by node but it's way slower you have to do tons of round trips you know it's like you know once you get one node in the tree then you fetch its children you'd fetch those children's so it's like it's a lot of round trips so the idea is okay what if we do something that's a hybrid of both right you have snapshots periodically but you're downloading portions of the state tree as well that have updated since that snapshot so you don't do some app shots quite as often or you don't do some shots that cover absolutely everything well you do snapshots that cover most of the stuff and then you sort of fill in the gaps adaptively and it's much more difficult to do but it can be done so in principle yeah warp sink stuff like that

is I think it's absolutely necessary for the challenges that aetherium is currently facing which is what do you do when a blockchain gets heavy like most of what you would put down on paper when you design protocols for blocks it's like yeah yeah it's gonna keep running like people are gonna be able to store all that information but after a while it does add up and this is where you run into those very practical engineering problems it's part of the I guess that divide between research and implementation that you see so much in this space where the researchers will say yeah you know it can be done and then and the engineers are like well I guess right but we have to write a lot of code to keep it running but even the

engineers can't always predict like I mean I think you've probably had this really interesting front row seat to see a blockchain grow really big kind of even though I'm sure that people were working on predicting a lot of that stuff like to see it in action all of these problems must emerge no definitely a lot of it is also that these engineering efforts involve oftentimes massive restructuring of large code bases hmm and you can identify what needs to be done but the actual work to reorganize to get things in a state where you can actually start to build out those new features is also something

that needs to be accounted for so oftentimes it's not only that though engineers won't see what needs to be done in the future but they'll kick the can a little further down the road you know it's like okay we can postpone that a little bit a little bit a little bit because it's just going to be such a monumental effort and resources you need a lot of resources to do that but you know software projects don't don't scale just by adding more people unto them so if there's only so many people you can have poking at one piece of code at a time before the you know your conflicting with too many people's work text should be productive that reorg idea like we've talked a number of times on the podcast about like maintenance but reorg is a different deal because that's like recreate you have to start

kind of from the ground up and rebuild the entire thing right oh yeah like you can run into kind of shat like world-shattering assumptions being broken you know yeah I mean not quite on that level because you you always are aware that these things sort of will happen and you'll write your code sort of in a way where you can accommodate those kinds of changes but then when you actually get to that point you'll realize well it wasn't ready that's one of the reasons we you know we had a lot of lessons learned with parity aetherium and we're still building it but we've spent now a good three or four years just building parity aetherium and we know what it takes to like write a pretty robust block to Incline now we've

taken a lot of those learnings into us with substrate where we can say well everything is really modular and flexible and pluggable and that means that you know you don't make assumptions about like okay this is where the consensus engine is used this is exactly where the network fits in it's just you know you you sort of let all the pieces control whatever parts of that infrastructure they need to we started we started with like what you worked on at parity or Igor at the time in in regards to aetherium but before I want to move on soon to poke it on and substrate but before I do that I am curious what else have you worked on the main thing before polka dot I mean this was wild we were already planning out polka dot because we were you know we

were investigating ideas like before stateless clients were named that we were exploring things like stateless clients and different light client protocols because we wanted to see what's the viability of putting light clients for other blockchains into a blockchain so one of the things that I was researching and developing at parity in the early days was a light client protocol so just building that out from from scratch over the course of 10 months doing the research on like what's most efficient designing the protocol and protocol messages so it's a different protocol than what GAF uses which is le s what le S will give you is the ability to ask a peer many requests

of the same type in one packet so you know if I want to get twenty different state tree nodes I can ask one peer for all of them and if they have all of them and they're on still return all of them what I wanted to do with pip was you know minimize round trips for things like light clients latency is super important like if you talk about user interface after you get past 100 milliseconds things are really annoying so I wanted to do pipelining in single packets where basically I could say give me the header with block number 1 million then from the state route in this header proved to me the account with this addresses balance and storage route and then prove to me the storage at this key so you could do all of that in one packet

you're doing all these like pipelining where you know you have unresolved outputs of prior requests being fed into further requests but all within one round-trip and it can all be done super fast on the on the other side you could even do smart contract calls and stuff and that was really nice because it really minimized the network latency you just mentioned two acronyms so what was gets le s and what does it stand for light aetherium sub protocol I think and yours is PIP yeah pip which stands for it's the parity light protocol got it with the the I in light is the high okay got it got it because we know how to do acronyms cool so I want to talk about substrate and I want to talk about polka

dot one thing to note those we do have an entire episode with Gavin that we recorded in November where he went into the detail of like the history of polka dot and a lot of the core ideas so I don't think we have to do that today necessarily we don't have to necessarily cover all of it but I think it would be good to reintroduce what it is what those two things are substrate and polka dot but also I think what I want to lead into after is like how does light client work lead into that how do they connect because I think there's kind of maybe some new insight I can glean from that I think I said earlier that polka dot is a heterogeneous multi chain system right so that's the that's what it says on the tin there is there is a lot more to it

than that and there's a lot of motivation behind it I think it's a lot easier to start with fee motivation briefly I'm sure Gavin tin too a ton of detail in his episode he's really good at explaining those kinds of things as well so if you want a super deep dive and polka dot that's the thing to check out but basically what it boils down to is the idea that right now blockchains all the blockchain platforms that we've seen so far you know that we had seen at around 2016 or 2017 we're all trying to do everything and be everything that they wanted to be they Bank they wanted to be the platform for a computation they wanted to secure all the decentralized applications and they wanted to do it all you know on one single chain that they were the layer

one and what we realized is that you know for instance with the smart contract paradigm the way that you have to charge fees introduces some inefficiencies so for instance the the gas metering right if I want to write an EVM smart contract I'm breaking it down into byte code and that byte code is little instructions like okay add a number subtract a number let's do some multiplication let's write this to storage or read this what's the author those kind of pieces of byte code it's like really really small granular instructions and if you wanted to do something like build the decks that has an order book you're gonna have to combine hundreds or thousands of these or imagine if you try to do some kind of elliptic curve cryptography verification it's tons of math right and the way that gas works is that before every single

one of these op codes you're stopping charging little fee if they have enough to pay the fee the gas cost then you continue so this is a huge inefficiency you do tons of extra work whenever you try to compute something with smart contracts but they're very very general you can kind of like write anything with smart contracts if you're willing to pay a fee so what we realized with polka dyes that you can rather for specific purposes why not build a specialized chain for that purpose where you say you know you don't have to do transactions in terms of these little op codes but you rather say this is the transaction for making and order in the order book for a decks or for storing a file on a

file exchange service or for querying an Oracle on an Oracle service and then you only have to pay a fee once that's vastly more efficient not to mention the fact that you know for specialized services you don't have to deal with fitting into it the smart contract platforms representation of storage you can make storage formats for the data that's kept on chain that is more efficient to the problem domain that you're trying to address so this is one principal specialization is great for optimizing problems but there's caveat when you specialize you lose the flexibility to represent many different problems you lose the network effect if you have a smart contract system all the smart contracts can talk to each other so this is where a need for interoperability

arises you know you have a network of many chains which are specialized and very efficient for certain problem domains one problem domain is smart contracts because the problem that smart contracts are really good at solving is being general and then you have the ability for all of these chains to communicate and that's basically what polka dot is I mean I think that that kind of contrasts the specialization which could lead to you know maybe faster times more optimized computational capacity versus this general approach which would be any you wouldn't have to fit into a certain box you could do a lot more but then you're gonna be stuck with like long wait times and as you mentioned before kind of like

this gas issue oh it's it's it's pure engineering like this is sort of a research versus practicality approach because you know in research you would say yeah you know smart contracts can do every computation but in practice we see that well there's some constant factor there and their analysis that you know we can get down we can make that low we can make things more efficient what I don't fully get though in the interoperability kind of explanation you just gave though it's like what would you be like creating different parts of a program somehow with these different chains like sort of the example you gave before about the smart contracts with like certain was it opcodes and like each one requiring a gas transaction in this case would you have entire chains

dedicated to certain pieces of that like with the chain just with the chain only do something in a kind of program I think it's much more likely that chains will occupy key infrastructure positions and the examples that I mentioned before of storing files for apples there's also stuff for like shielded transactions and fast transfers plasma things like that rather than pay gas costs for all the opcodes that it would take to represent those constructs you can say well you know it's a linear overhead to make an order in the order book so we're gonna charge you a fee relative to the size of

your order something you know I don't know exactly how that's so that's an example that came out of nowhere but you know in principle you get the idea which is that you can levy fees once for things that would take levying fees many times in a smart contract system got it so you also have substrate which is connected to polka dot but is separate from coke polka dot what substrate yeah so polka dot is a very broad project says if there are specialized chains that fulfill some you know constraints on how they are executed and how the blocks for them are produced then we can connect them and we can build this network out of all of them but of course those chains still need to be built and

the tool for doing that that we've made is called substrate right so the idea is make a it's like a blockchain toolkit all those things that are actually pretty tricky to deal with in writing a blockchain know like the database sink consensus all this stuff the RP sees everything except for what the blockchain is actually supposed to do which is typically as a developer as a development team building a new blockchain that's actually what you care about right you don't want to get bogged down and like debugging some database issue or some consensus fault for ages and ages you just want to make the block chains transactions do what you want it to do and that's what substrate is for it's essentially to let people focus on the part of the blockchain that they want to build the purpose of substrate

is also to allow someone to create a blockchain that would be easily kind of implemented into the polka dot system right like it would have all the same it's probably written in the same language it's like easy to fit in right so I wouldn't say that language is actually particularly important in polka dot these block chains that we connect are called pair chains and their execution is defined by web assembly code web assembly is sort of like the thing that's been coming forever to kick JavaScript off the throne of the web so it's actually a very portable low-level bytecode sort of like you know what the EVM was aiming to be but much more ubiquitous and web assembly is targeted by many different

languages so you can write programs in rust C C++ go like there's many many different languages typically in the low-level spectrum that can compile to web assembly so you know you technically could write a pair of chain in any of these any of these languages but substrate yes to answer your question you know without going off on a nitpicky tangent is yes it's made to plug well into polka dot but the projects are are complementary like you can write something with substrate that doesn't use polka at all and it would be nice and we like it if you plugged your chain into polka dot but you don't have to we have it like we sort of approach it in a couple different ways so basically what it boils down to is substrate is completely generic over its consensus

engine doesn't make any assumptions about what is actually providing security for your chain so you know you have you can do your typical consensus implementations like proof-of-work you know we have ones that we've invented called or babe and Grandpa you know we have implementations for all of those you could do a tender mint implementation for substrate but there's also one other consensus engine we've implemented for substrate which is what we call polka dot piggyback consensus that's the idea that you can write a chain in substrate that draws its security from the consensus of another chain right so it's just piggybacked on to it and that means that it's you know it's not its own sovereign chain that's how you would create a substrate chain that deploys on to polka dot the other

thing that we've done with you know while we're on the topic of pluggable consensus is we've set things up we haven't implemented this code yet but it's a feature that's on the roadmap for substrate is the idea of swappable consensus hot-swappable consensus the idea that not only can you to choose what consensus engine you deploy your chain with but while the chain is running you can change the consensus mid-flight whoa what would that be used for that's wild so one of the things that we've done when designing substrate is really try to put as much flexibility in the hands of project developers as possible and that means that there's no platform blocking from polka dot for instance that means that if you know if you want to be a pair of chain you don't have to be a pair of chain forever

or if you start as a sovereign chain one that provides its own security you don't have to stay a sovereign chain forever so that's sort of the goal of swappable consensus you can start as a sovereign chain imagine imagine this is the lifecycle of a pair of chain right so the chain is launched it has its own validator set its own security and it's providing a service on its own but you know it's not part of the polka dot network but then some people from the project decide to buy a pair of chain slot in the open auction when one comes up and they get a pair of chain slot for six months or 12 months after that point they can trigger a swap of consensus to the polka dot piggyback consensus engine

so that their chain which was previously sovereign will become a pair of chain it's technically a hard fork but it's a hard fork that you don't need to distribute a new version of a client for like that's typically what happens with a hard fork is that you know you have to say okay everybody update you got to update before this day if you don't do it you're just gonna be off on your own chain and that's useless you're useless update right we were doing something completely different which is that with swappable consensus your substrate client will just support a bunch of different kinds of hard Forks that it can do and everybody's clients knows how to do these consensus hard Forks at least to the consensus engines it supports so you won't need anybody to update their client specifically for this hard fork so they'll hard fork

become a pair of chain live as a pair of chain for a while maybe after a couple of years they'll decide we don't want to be a pair of chain anymore then they can swap back to sovereign consensus of some kind and cease to be a pair of chain could you use this also when you said swappable consensus though I thought like oh you could switch between like proof of work improve mistake or so they just fight for fun oh absolutely like without it necessarily interacting with no polka dot yep so it's but you also said mid-flight is that sort of like what like while the blockchains lieth post-launch okay yeah no absolutely you don't have to use this in a way that's connected to poke not at all like we the framework that we've built is for navigating between arbitrary consensus engines what we've designed we haven't

coded it yet that's planned for you know a later version of substrate but this is the roadmap for a substrate and you know that is how it fits into our vision with polka dot the idea for instance that pair of chain developers can deploy their pair of chain before polka dot maintenance even launch right so they don't have to wait for polka dot main net to be launched to actually start delivering their product and then when polka dot maintenance launched they can attach to it cool is this done by like so you sort of mentioned the the clients don't have to update but does that mean that there's like an auto update pushed is it's sort of like sort of like your client is pre ready for certain types of

hard Forks now imagine if I want to swap from consensus engine a to consensus engine Z but consensus engine Z was sort of newly introduced then you would kind of have to wait for people to update their substrate versions right or they're you know that note written in substrate version but imagine if consensus engine Z has been part of the substrate catalog of consensus engines for a good while or the project developers want to develop their own consensus engine consensus engine X they think okay you know at some point we're going to want to fork to this consensus injured but we don't know when then they can disseminate a client update that contains consensus engine X and after that point they can do the hard fork

whenever they want so it's not like they have to do an emergency update we're gonna hard fork right now it's just like you know they can plan ahead for it and if substrate has a large suite of consensus engines that means that block chains are going to support these kinds of hard Forks across almost every client of every block chain when you're implementing with substrate are you basically down will you be having a sort of a large selection of consensus mechanisms that are already kind of written into the client itself and you can choose which ones you use depending on certain things and then like and that would already be in the client itself so wouldn't be about downloading a new version but rather triggering a new version like yes maybe

I'm a bit unclear on like the difference between a downloading of a new client and updating of so yes yeah so downloading and updating are basically the same thing in this case you would have to update right but the fact that these consensus engines are sort of pre bundled or that you know they are just part of the standard substrate suite of consensus engines or that developers might have the foresight to include them well before the fact means that at the time that you actually decide to trigger the switch basically everybody has already updated hmm you don't have this situation where it's like okay emergency update time we're gonna do a hard a hard fork you can just sort of decide at the drop of a hat you know via via any

process whatsoever let's change the consensus engine you don't have to do you know anything formally to say okay but then it's going to be a big drag because we have to distribute this client update you can just say you know issue some kind of signal in the chain via you know a governance system or something like that that says okay switch and at that block it'll just switch well you know for anybody who's updated their client in the last like six months oh cool like as long as the consensus mechanism has been and that's sort of what you were saying there where it's like if the if that can if there's a new consensus mechanism it isn't included it would first have to be included in a substrate bundle and then downloaded and then I mean doesn't necessarily have to be part

of the the suite of consensus engines that's included with substrate itself like it could be that the client developers of some project would say we have our special consensus engine and now that we've developed that we're going to include it in all of our releases from this point on and then at some point we're just going to trigger the switch I have a question how does this like going back to your background with like clients you sort of mentioned that before polka-dot you guys were already experimenting with these like clients you know how does that actually relate like is there a connection between the light client work and what polkadot is great is yes I think there's a huge amount of that in particular the stateless client concept

so now Alexia Knopf has sort of taken over that mantle for eath 1x so he's released a really nice blog post with like up-to-date statistics but at the time when I was doing this in towards the end of 2016 I also had my own graphs and things so the statistics that we were collecting on the etherion blockchain which was the best sample that we had at the moment of a blockchain that's actually doing significant computational work was to see how big the state proofs were when you wanted to execute an entire block so that is if I want to execute the block we have this whole state tree that contains you know all the accounts and smart contract data it said you know it's a tree of nodes what if we just

packaged up all the nodes that you need to execute that entire block how big is that package right and we found that for the etherium chain typically at that time it was you know in the range of a few hundred kilobytes but we could see that there was a trend going up because the state tree does get bigger and as it gets bigger it gets more inefficient - because it gets lower you know it gets deeper right gets deeper the more nodes that yes sort of leaves on the Merkel tree is that what it is or is it more entries in a it's not just about Leafs but it's about how many accesses you have to do it's like how many children do you have to traverse ok that's what I mean by it getting more inefficient there's like you know if you have a tree

with one item then lookup is always just you know one the one note yeah right but if you have to go for levels deep or five levels deep or six levels deep you have to go you know or you have to do more for every piece of state that you want to read and that makes the proofs more inefficient so even though you know the amount of pieces that you might read for a given block would stay the same for a given transaction per gasp let's say would stay the same it gets more inefficient those proofs get bigger because you have to punch through all those levels because the entire state tree is much bigger and this is all cryptographically proven because these state trees are Merkel trees right that's the state route in the etherium block header but this is

essentially how polka dot works polka dot is a system of stateless clients the validators of polka dot who are signing off on whether pair of chain blocks are good or bad those valid leaders are shuffled between pair of chains if you have ten pair of chains I as a validator from block to block might get shuffled from pair of chain one two pair chain five to three to seven you know over the course of just a few minutes even and that means that I as a validator don't have time or space to keep around the entire state of every single pair of chain we have nodes who are living on a single pair of chain called co-leaders those store the entire state of their pairing and they generate these stateless client proofs of execution for

new blocks to send to validators validators check those ensure that the proofs remain available and then sign off on them so the stateless client research was really pivotal in finding out whether the idea of polka dot was going to be viable I remember in 2016 I had a hacked version of the parity a theorem client that was just going through every single block in the chain and seeing how big is that proof and we could we looked at the graphs like there was this whole spurious dragon state bloat attack and you know those were out of out of you know out of the ordinary there was like thirty to fifty megabytes typically but mostly they were on the order of a few hundred kilobytes and that's where we saw okay for a blockchain that's seeing as much throughput as if you're a man with the

current state of Internet infrastructure this is actually doable and this is an architecture worth pursuing the co-leaders you just mentioned are they almost like archive nodes today like you're saying they they have the full state they have are they like a regular node are they like the full state like the state changes every what does it look every cent every block so therefore I mean we kind of mentioned this on a previous podcast where if you really want to see the history of the state you'd actually have to create snapshots at every block to see that and look back and that's an archive an archive node are the co-leaders related like are they more just like holding a regular node co-leaders are what we would call full notes okay yeah

you store the and when I say the entire state I meant the entire state of the head of the chain okay but nuts that's a short that's a shorthand because what I really mean because you know the consensus is not instant finality and you can you know have safety failures where you might have Forks even if it's instant finality what I really mean is the collator nodes keep state for all viable heads of the chain so what anybody might reasonably perceive to be the head of the chain so that's usually you know across a few different short Forks and back maybe a hundred or a thousand blocks and that's about how much state a typical full node would keep in the etherium Network as well that allows you to account for

reorganizations and things but when I say co-leaders have the full state they have the full state of the block that they are building whereas validators have absolutely nothing right they have nothing they have they are light clients of every pair of chain so that's you just mentioned to sort of actors within polka dot the validator and the collator originally in the paper there was like fisherman and nominators so has that changed a little bit I know even when we interviewed Gavin November he'd sort of talked about how there might be sort of cross action or like you'd be doing two actors jobs at Lee so where's it at now yeah well I think the like the four roles that we came up with are

convenient for economic analysis but in practice you would be doing more than one at a time simply because when you're performing one role you may be well positioned to perform another so in particular you mentioned nominators and fisherman I think of the of these two fishermen are more interesting so I'll mention what nominators are briefly but essentially they are there people who vouched for stagers who are validating right so validators staked their own coins or their own dots to validate and nominators say I trust that guy I want to get a portion of his rewards I'm gonna back him but if you get slashed I'm also gonna take a hit so aren't they just I mean nominators to me

sound a lot like delegate errs in other systems that we I mean just recently we had actually a few episodes on proof of stake and validator setups can I at least for Ken our listeners for this purpose just like combine those two is a nominator a delegator you're these different in some way I think it's it's like at a very high level they're similar but typically DPOs does not propagate slashing on to the delegator I see which is what we do and the other thing is we use an algorithm called partial fragments method it takes a voting algorithm from the 1800s that's very applicable this use case it has it happens to basically have nominators say which candidates they'd be willing to

support and the system will automatically apportion their stake across those different candidates they're nominated stick yeah this is not howdy pause typically works from the systems that I've seen so far there are delegation tools like coming out and there's some talk of like other maybe wallets offering this kind of stuff but it doesn't sound like in any way that it's automated or within the chain itself these are often like external pieces of software that you'd be running in order to that's right gate to the fragments but it takes into account not just your own preferences but everybody else's preferences and this is not something that's possible to do just by adding on a extra layer right of delegator tools anyway so those that's

that's nominators but then we have fishermen so what fishermen do is they're looking for misbehavior they want to see who [ __ ] up so earlier on I said validators are getting state proofs that they can check para chained execution with and they make it available and they vouch for it so the reason that they make it available is that then fishermen can download that proof of a blocks validity and check it and if it turns out to be bad then they can basically file dispute and anybody any valid who signed off on that block being good when it has now been proven bad by a fisherman will get slashed and the

fisherman will get a portion of that steak right they they have an incentive for doing something here they get rewarded the the hairier question is always what do we do if the validators say it was available but it's not like for instance with a block validity proof that never existed because the block wasn't valid so then we have availability games where fishermen if they can't seem to find the data to check whether a block was good or bad from a pair of chain they query they ask around and if they can't recover it we have a whole whole economic game around this then they'll issue another kind of dispute which leads to the offending validators being slashed as well but

it's it becomes more subjective at that point this is all automated this is very much just like I imagine it's built into software like you would if you were a fisherman running the fisherman software you just basically it would all be automatically happening for you and so the only way and a validator would ever actually do something incorrect would be like if it went offline or if it had injected the wrong code or something like that right in general if they would use the client software provided by honest developers they would not misbehave except in the case of bucks mm-hmm but we're dealing with adversarial BFT systems here where we're assuming up to one third of the validators are

malicious and collusive and running yeah alternative software so the the yeah like you know we don't we don't care what software they run we just talk we're just worried about like what are they signing and sending over the network right there they're running arbitrary evil software right you know in practice that's going to be hacks of the official clients official as those official clients get you know we further for the purpose of security analysis we don't really worry about what software exactly they're running we worry more about like what people do within a protocol and we are considering up to one third of validators being malicious right but we've got a system where validators are split into groups randomly shuffled across chains so even if the whole set of validators is only

1/3 malicious you could very easily get a pair of chain group that's majority dishonest right and that's a problem because if your threshold for inclusion of a pair of chain canada is 51% of the validators within that group signed off on it then that can be easily compromised by that kind of case where you have that many malicious validators in the system overall so what we do is we try to set up a system where it's very hard to finalize blocks where data isn't available and if you have one fisherman for every pair of chain and data is available you can always find out who lied and we also set it up so that even if a block is finalized or authored where data wasn't available but

you have some constant number of fisherman querying availability over all then you can always prove who failed to make data available or rather that the data is not available so I want to move on from the actors because I kind of want to talk about the general set up we've sort of mentioned this pair of chains different chains interoperability connecting them as I've understood it there's like overarching real a chain there's something that's like the security it's it's where the consensus is happening and the pair of trains connect to that and they get to share that security all of these actors are talking about exist in the real a chain

right well the cult leaders are sort of half and half ok co-leaders are you know they're their home is one pair of chain got it and fishermen likely to you know fishermen may may also be more likely to catch out bad pair of chain blocks because they're watching a specific pair of chain I see right and co-leaders being on a specific pair of chain are well equipped to be fishermen okay so would you say that it's it's often gonna be like a collator fisherman as the same actor very likely yeah I'm not saying that you have to be a co-leader to be a fisherman but if you're already running a collator and you've got a key setup that's signing things and making you money in the background you might as well have it fish for misbehavior now let's talk a

little bit about like these comparable solutions so recently I had sunny from cosmos on and we talked about the cosmos model which is also an interoperability protocol and it's also about like connecting different chains instead of having to sync the single chain that's a generalist sore smart contract chain having different sub or other chains connected what we talked about was this idea that like certain chains would require different security so for example something like a chain for crypto kitties wouldn't need the same security as a chain that was for a stable coin or something like this in the case of poccadot all of the security exists in the relay chain so this would

be a consistent security model despite maybe some chains not needing so much security so I'm just curious is that overkill like do you need do you need the same security on those two chains well one thing that's kind of hard to tell a lot of the time is how much security you're going to need in the future I mean imagine like very early-stage crypto kitties you know nobody would have really put much security on it but once it really took off it needed a lot of security because those kitties became pretty valuable and it's actually very very possible to 51% attack and make some money there right imagine if it was secured by a very weak chain this is

sort of the problem with like the bridging approach so for instance if you have you have many different chains which are all just bridged to each other and you let's say you have chains a B and C and a is the weakest and you have an asset coming from a bridged on to be and then bridged further from B on to C but then a is weak and it's 51% attacked then those tokens that you think you have on pair of chain B are actually garbage so it's not even just that you can like cut off connections directly to weak chains if you're worried about like garbage tokens that you know could be 51% attacked and pulled out from under you but you have to

allow all transitive connections to so that approach has some real problems because it want one thing that it does is sort of hurt the idea of economic abstraction so economic abstraction being the idea that you can treat all tokens equally in terms of they will all follow the interface of token which is that you can transfer you can receive you can approve you know someone else to make some transactions on your behalf things like that now if you use this bridged model you have to do basically due diligence on like you know imagine if you're trying to build multi collateral die it wouldn't work because you have to do due diligence on every

token that's going to be collateralized it's putting a lot of unnecessary burden on the user that's actually not going to be carried out correctly and I think that's a real like it's not the right way to approach the problem of different chains are going to need different security which I do agree actually is the case so the way that this fits into specifically the different chains need different security the way that this fits into our roadmap is the idea that polka-dot 2.0 will have hierarchical relay chains so you have a relay chain at the top that's the polka dot we've talked about right now you have a really chain with pair of chains attached to it and all of those pair of chains have the same level of security the security is split evenly across all of those pair of

chains but what if one of those pair of chains was itself a relay chain right then that pair of chain is taking the security it has from the top level relay chain and splitting it amongst its pair of chains so that's how you solve the problem of some chains will need less security than others in the polka dot hierarchy you just push it further down the tree of chains and it's going to take a bit longer for messages to propagate all the way up that path to the root of the tree and down some other branch of the tree that's the price you pay for having less security you're less prioritized you're less important but you're still protected against being 51% detect because you can't be 51% detect

with all the other chains in your group being 51% detect so if there's something connected to the sub relay chain in model what you just sort of said it would it take more time things to be secured or like what you sort of mentioned it would take longer it would take a longer to go up and down the thing so is that sort of the you have the same amount of security but it's slower or less security so there's a couple different things that are getting mixed here and I'll address them one by one so one it will have less security simply because it is easier to fifty one percent or thirty four percent attack the entire validator set of that sub

relay chain because they are simply not going to be as highly staked as the top level relay chain right so it is possible to do bad things on that religion that's one thing so it's it's it is less secure and that's why I say the the the solution to the problem of some chains won't need as much security as others is to push them further down the tree of relay chains now the other thing that was questioned was does that mean that the blocks will be will take longer to be produced or finalized so they'll be produced just as fast but they will be slightly longer to finality because finality flows from the root of the top level relay chain which means

let's say we have top level really chain you have a really chain below it and then you have para chain a connected to it so you have to submit a block four pairs in a it's included in the lower level relay chains block then that lower level relay chains block is included in the top level relay chains block then when that's finalized but you have to go through this extra step right before it's finally somehow referenced by hash in the top level religion that's that's what it takes before it can be finalized those block chains that require less security could they still go at the same speed or even like faster sure they can go faster it's just that they have a slightly longer leakin c2 finality and that's where the sort of the less security comes in like it's possible that for a period of time a false block

is there and then wouldn't need to be fixed well it's it's even moreso than that it's that the entire pair of chain there can be adversarially over just because that lower-level religion like we're assuming like the adversary has 33 percent of all dots right so that lower level relay chain isn't going to have a one-to-one correspondence in stake between the top level relay chain and the lower level one so the adversary that owns 33% of all dots can easily 34 percent attack that lower level through relay chain right and that's where you can cause problems by you know attacking all the pair of chains on that lower level relay chain at once that's I have one last question on this which is like

why would you create these chains for with less security in the first place like you sort of mentioned like they can be fast or the same speed they will have less security but is it is it like yeah I don't really I don't understand why why wouldn't you want all chains to have the same security well it's just more that different chains are going to experience different levels of demand and use right that's economic some of them are simply just going to be more broadly applicable and have a bigger market than others so that means that if you split security perfectly evenly between all chains it's not just that you're you know you're choosing to give more security to other chains it's that you're also taking away a bit of

security that you know some other chains might deserve a bit more so that's kind of the nice thing about the hierarchic ality approach is that you know you're chained depending on how much security it needs can slot itself in somewhere further away from the root of the tree of chains so one other thing about that I'm aware of about like polkadot is that you'd have to you have to I don't know if the word would be steak if the lock dots in order to like rent out a pair of chain slugs right yeah in this model do you imagine like more pair train slots or is it like cheaper pair chain slots yeah they would be way cheaper okay they because there's less demand for them like they're you know the the highest

most so the one thing that I didn't get to in the previous explanation is that it means that you'll have higher latency for message passing when you have a chain that's lower down in the tree of chains because you when I want to send a message to another pair of chain and I'm in like a sub sub sub chain right I have to send it to my real a chain the real a chain above that the real a chain above that and then if I'm going down into another part of the hierarchy have to go down down down and then eventually I will get my message over to the pair of chain that's in like you know another branch of the tree of chains but when you are closer to the root that means that you always have less distance to travel for

your messages and so this is sort of a premium place to be it's a good place to be for very heavily trafficked pieces of infrastructure and maybe a pair of chains that have high value but you could also imagine is that you know add on every relay chain you might have one or two different smart contract instances like you know an etherium like or something like edge where where you have webassembly smart contracts and you know having a smart contract live on the smart contract chain that's at the very top is going to be way harder to and the gas fees are going to be more expensive because it's way more trafficked but you're gonna be able to send messages from your smart contract that reach every other pair of chain fairly efficiently cool yep so what I'm kind of curious about like going back to your

background I'm kind of happy that we started this episode looking at like how you started and the stuff that you were trying out in the different pieces of the etherium client you were working on and how that kind of led you to develop a lot of skills and insights into how to do polka dot stuff now what I'm really curious about you've been working on this project for how long at almost three years we started we started in late June 2016 it's now mid June Wow 2019 are there any new ideas that you've picked up as you're kind of building this out one thing we've been thinking about a bunch lately as we you know we're murdered we're moving further and

further on with our iterations of getting closer and closer to what we originally wanted to ship which was basically a an ability to get security for a bunch of different chains and pass messages between them so the next question that follows logically I have that is how do we know that what the Matt what we get out of those messages is actually good and that's that's actually a super tough question to answer because you know you could have a pair of chain with its own governance right where it can actually change what its code does and you might know that okay at at the current time you know what I'm building my application I know

that messages that come from this pair of chain that say I burned some of this token please mint some correspondingly on this side or you know anything like that wear a pair of chain says I've done something how do you trust that when those pair chains might be able to you know change their code arbitrarily or if you want to deal with new pair of chains that didn't exist at the time when you deployed your app right you can't account for all future possibilities so this is where we've come up with a new concept it was originally put forward actually by a user on reddit and he framed it as smart protocols and we've taken it and refined it a bunch and we're calling it trust wormholes now but the basic idea is that for those things where you need to guarantee that a pair

of chain actually has accountably done something you don't rely only on the pair of chains code to do that but rather you can create something sort of like a smart contract sort of like a it's a trusted piece of code that any pair of chain shares and it controls a piece of state on that pair of chain so you know it might be for tokens for the token abstraction it could be an accounting a piece of accounting for all tokens that live on that pair of chain so for then when I want to transfer a token from perch and a to perish in B pair chain a actually tells the trust

wormhole for pair of chain a transfer the tokens from the trust wormhole of pair of chain a to the trust wormhole of pair of chain B and it the trust wormhole can actually be trusted because it's webassembly code is registered on the real a chain and it can't be subverted by any pair of chains governance so that means that the trust wormhole when it gets the command to transfer some tokens will actually the tokens and it will send a message that goes over to the trust wormhole of parish and businesses I have burnt the tokens and perish a and B will be able to recognize this as actually having had happened and then would you take that like how does that then go out of the trust wormhole into the pair chain so they're so trustworthy and they can talk

to their host pair of chain but nothing else but nothing else the trust wormhole of pair chain B cannot speak directly to pair chain a that's correct okay but the trust wormhole from pair of chain B can tell pair chain B hey I've received some tokens and this is sort of our way around that leaky abstraction thing that you get when you bridge a bunch of chains together where it's the chain that's the lowest security becomes the weak the weakest link this way even if chains are changing the way that they operate out from under you they can't pull the rug from under your feet they can't make your code accidentally accept a double spend and it doesn't apply just to tokens but tokens are just a like a

premier example of this but it sort of applies to all interfaces where you are expecting that some kind of accounting or execution has been done prior to an event being emitted so that's the generalization so that you know the way that you apply that to tokens is the accounting or execution before the event being emitted is that the tokens were actually burned before an event that says I burned the tokens it could be any event but it could be any event yeah it could be like I have done the work and then it will send a message that says I did the work and then somebody gets paid right on another pair of chain could you in these cases could you like run something within the trusted wormhole on one pair of chain and send the results

to the second one kind of like doing off chain computation or is it only for like a change in state or a message or something like this well you can I mean you can make the trust wormhole run as much execution as you want but it's always going to be on chain computation so these do trusted on Chane computation and not any kind of off chain work so it's always going to be part of the state transition of a pair of chain okay alongside the transaction execution something about this seems a little similar to T's or trusted what is it trusted execution environments but it's not exactly the

same cuz it's not in the hardware necessarily right this is like a like where does this where is this where does it exist I think I think that it's a it's a very good analogy so one analogy we've used to talk about polka dot and and and pair of chains is that pair of chains are like computers and polka dot is like the Internet okay right and if you think about okay how do I get trusted messages from a computer to another computer over the Internet te es or one approach to that because you have this trusted enclave that's actually signing things right so this is sort of the same principle applied to polka dot

how do I get a trusted message from one pair of chain to another over polka dot through a trust wormhole and the actual one acronym we're we probably will end up calling it this but trust wormholes sounds kind of cooler but we're thinking of calling it spree which is shared protected runtime execution enclaves which is an homage to trusted execution environments and an homage to the river that runs through Lin that's right that's right although we would say pay them yeah I'm sure that's right come through right yes yeah that's definitely where it comes from yeah so we're really good at acronyms at polka dot we've got babe which is our consensus algorithm

for the new blocks which is called blind assignment blockchain extension we've got Grandpa which is for the old blocks which is ghost based recursive ancestor deriving prefix agreement now you've got spree very nice once the last question about spree or these trusted wormholes how are they different from T's so you just sort of you've made the comparison of how they're similar but they're not really exactly the same and also teas have all these like issues so yeah so the nice thing about spree is that it's actually gonna work unlike tes but like they're they're like tea ease but if they did work and you mapped it onto polka dot the main reason being that teas draw their trustworthiness

from trusted hardware which attackers have clearly been able to break without too much effort you know via timing attacks are just bad cryptography or in all kinds of weird CPU tricks and meddling but with polka dot spree we're not really getting into anything that crazy you know it's just making sure that deterministic webassembly code that's registered on the real a chain for everybody to see that has been agreed upon by consensus to make sure that it's been run correctly and that's not a particularly difficult thing to do because it's all deterministic it's not as hard of a problem as building GE and

it also sounds like it's all like this will be open source it'll be Co it's cold it's not you're trusting in a chip manufacturer I guess the the parallel would be a spree module a trust wormhole which had a buck so you know one for accounting and tokens since that's the example I've used already might have a bug where it sends you can force it to send that I have burned the tokens message when it didn't actually burn the tokens but these are going to be very likely open source modules they're going to have to be very well audited before the governance would allow it to end up anywhere on the the relay chain and

since they are intended to be the underpinnings of secure interchange communication they will have to be very well checked and will be eventually open source and then of course this is it's not something that necessarily even the polka dot devs are going to write right it's a very it's a general construct where anybody can put forward a trust wormhole cool for instance you can have trust wormholes that run smart contracts and then you can give every pair of chain a little smart contract component as an idea is this a wish out into the world just a little idea cool well listen Rob I want to say a big thank you for coming

back on the podcast and this time speaking with me it's great to be back one day you'll speak to both being Frederick so I remembered when I actually was on the podcast last time I think it was like February 2018 sweet in the beginning you're also in Italy oh that's right oh that's so weird actually times I've been Italy in the last two years okay ended up on the journal vodcast trying to one of the hosting time well anyway thanks again for being on and I hope to have you back on soon and I also I sort of put this towards I asked Sonny this and I'm gonna ask this to you as well I would love to have you come back on sometime and actually do a deep dive comparison between the two protocols because I feel like there's a

lot to get into yeah I'm excited to see what kind of commonalities can come out I one thing that the Kosmos team has done a lot of work into is for instance how these kinds of cross chain communication are going to work and it's interesting to see if those same kind of protocols can be adapted onto polka-dot and will end up with a much larger interchain ecosystem encompassing both projects cool alright well then thanks again for being here good and to our listeners thanks for listening you [Music]